Streaming datasets: 100x More Efficient

We boosted load_dataset('dataset', streaming=True), streaming datasets without downloading them with one line of code! Start training on multi-TB datasets immediately, without complex setups, downloading, no "disk out of space", or 429 “stop requesting!” errors. It's super fast! Outrunning our local SSDs when training on 64xH100 with 256 workers downloading data. We've improved streaming to have 100x fewer requests, → 10× faster data resolution → 2x sample/sec, → 0 worker crashes at 256 concurrent workers. Loading data, especially at the terabyte scale, is a major pain in any machine learning workflow. We suffered this while training SmolLM3, at one point we had to wait 3 hours before each run to download enough data. Streaming has always been possible in the datasets library, but large scale training with massive datasets remained a challenge. That changes today. We spent a few months improving the backend, focusing on streaming datasets to make it faster and more efficient. First things first: our changes are backwards compatible. You can still stream any dataset from the Hub with the same simple streaming=True flag. Thousands of AI developers around the world use datasets daily; they should just get improved performance with zero extra work. We focused on two phases: startup and streaming. 1. Startup: Persistent Data Files Cache and Optimized Resolution Logic. 2. Streaming: Prefetching for Parquet and Configurable Buffering. Together, these improvements can double your data throughput, allowing you to train faster and more efficiently. Hugging Face uses Xet: a dedupe-based storage which enables fast deduped uploads and downloads. These powerful new features landed in the datasets and huggingface_hub libraries. To take advantage of them, simply update your libraries: pip install --upgrade datasets huggingface_hub.