hachyderm.io is one of the many independent Mastodon servers you can use to participate in the fediverse.
Hachyderm is a safe space, LGBTQIA+ and BLM, primarily comprised of tech industry professionals world wide. Note that many non-user account types have restrictions - please see our About page.

Administered by:

Server stats:

8.9K
active users

#compression

1 post1 participant0 posts today

🚀 Compute Bigger with Cat2Cloud Compression! 🗜️

In this third video, we explore lazy expressions 😴—a compression-native tool to optimize operations ➕✖️➗ on your data using the Cat2Cloud web client. 🌐

💡 With lazy expressions, you can decompress and compute data only when needed, saving time and resources! ⏱️

👉 Try it out here: cat2.cloud/demo

🔗 Learn more at ironarray.io/cat2cloud 👀
#compression #datascience #ironArray #data

Missed PyData Global 2024? No worries! You can now catch with our recorded session on YouTube. 🎥✨

In my tutorial, I covered:

🧮 The basics of an NDArray object in Blosc2
⚡ How to leverage lazy expressions for efficient computation
🛠️ Efficiently computing UDFs within the Blosc2 engine
🌍 Accessing your data from the network: a gentle intro to Caterva2

🎬 Watch the video here: youtube.com/watch?v=aR-i_a3nGx

Don't miss out—learn, explore, and level up your data skills! 🚀

Excited to announce the release of the second cat2cloud introductory video! 🥳
Our software optimises file transfer ⚡ and access for server-hosted data 📅 . cat2cloud's compression-first framework enables users to minimise transfer times and storage requirements 🗜️.

In this video, you can see how easy and quick it is to manage file storage using the ubiquitous jupyter notebook format 🐍 !
Find out more (and see the first video) at ironarray.io/cat2cloud!
#API #datascience #compression #jupyter

This is Internet Gold.

> "... it is important to note that the compression algorithm used by lzip only discards the unimportant data. And if it was unimportant before, what makes it so important now? Huh? In fact, many users may find that compressing their entire file system and then restoring it will be a good way to learn what is truly important."

web.archive.org/web/2001060804

web.archive.orgLzip lossy compression

Brand new PEP by @emmatyping to add Zstandard to the standard library:
peps.python.org/pep-0784/

Will it make it in to 3.14 before the feature freeze on 2025-05-06? It'll be close but it's possible!

The PEP also suggests namespacing the other compression libraries lzma, bz2 and zlib, with a 10-year deprecation for the old names.

Join the discussion to give your support, suggestions or feedback:

discuss.python.org/t/pep-784-a

Python PEPs
Python Enhancement Proposals (PEPs)PEP 784 – Adding Zstandard to the standard library | peps.python.orgZstandard is a widely adopted, mature, and highly efficient compression standard. This PEP proposes adding a new module to the Python standard library containing a Python wrapper around Meta’s zstd library, the default implementation. Additionally, to a...
#PEP#PEP784#zstd

🚀 **Exciting News!** After 15 years of developing #Blosc/#Blosc2, we're thrilled to announce the beta program for Cat2Cloud! 🎉

- 🔄 Share complex data securely and effortlessly
- 🗜️ Access to the best compression algorithms available
- ⚡ Perform advanced computations directly in the cloud

...and more!

ironarray.io/cat2cloud

Join our beta program today and be among the first to experience the power of Cat2Cloud!

#DataScience #Compression #SaaS #CloudComputing #BetaProgram

Share Data Faster!⚡

Replied in thread

@sarahdalgulls Arduously long but one point struck me. The use of ambiguity in AI responses because it doesn't know what to put. In effect the reader sees the meaning in the sentence.

Does this mean that we are being gaslighted, and AI is not nearly as advanced as we think? Has it just learnt to produce text that is plausible to anyone and fits multiple interpretations, just like the tabloid astrology columns?

I don't think there's a simple answer to this. On relatively closed domains like writing computer language it definitely can produce real direct answers. On ambiguous social questions, perhaps it does produce ambiguous answers for us to project meaning on to?

I'm quite surprised that I hadn't noticed or thought of this possibility. I think I will ask some questions and look at the answers while asking myself "how would someone with a different worldview understand this?".

📢 🔥 Updated article on Blosc2: Compute with TB-sized datasets on your own hardware, within human timeframes!

Highlights:

🚀 Outperforms NumPy by 10x ~ 100x for large computations
💾 Maintains performance with datasets far exceeding physical memory
🐍 Integrates seamlessly with the Python data science ecosystem
💻 Works both in-memory and on-disk with minimal performance differences

Read more: ironarray.io/blog/compute-bigg

🤔 Ah, the latest #GroundbreakingResearch from the wizards of academia: "Parameter-Free KV Cache Compression" – because who doesn’t love another impenetrable acronym party 🎉? It's so cutting-edge that even the abstract needs an abstract. Time to 🤓 "compress" this into the recycle bin! 🗑️
arxiv.org/abs/2503.10714 #GroundbreakingResearch #ParameterFree #Compression #Academia #TechHumor #ResearchTrends #HackerNews #ngated

arXiv logo
arXiv.orgZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMsThe linear growth of key-value (KV) cache memory and quadratic computational in attention mechanisms complexity pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often incur irreversible information loss or require costly parameter retraining. To this end, we propose ZSMerge, a dynamic KV cache compression framework designed for efficient cache management, featuring three key operations: (1) fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) a residual merging mechanism that preserves critical context through compensated attention scoring, and (3) a zero-shot adaptation mechanism compatible with diverse LLM architectures without requiring retraining. ZSMerge significantly enhances memory efficiency and inference speed with negligible performance degradation across LLMs. When applied to LLaMA2-7B, it demonstrates a 20:1 compression ratio for key-value cache retention (reducing memory footprint to 5\% of baseline) while sustaining comparable generation quality, coupled with triple throughput gains at extreme 54k-token contexts that eliminate out-of-memory failures. The code is available at https://github.com/SusCom-Lab/ZSMerge.

🚀 Exciting Python-Blosc2 Update!
In 3.2.1 release, we've implemented the __array_interface__ protocol for our array containers, dramatically expanding interoperability with NumPy, CuPy and other array libraries.

This means our blosc2.jit decorator now works with virtually all NumPy functions! Compute with the full power of NumPy with either decompressed data... or compressed data without decompressing first.

Check out examples: github.com/Blosc/python-blosc2