How does a robot learn from its own demonstrations without downloading terabytes?

A robot arm picks up a red cube. The cameras watch, the joints move, and every frame lands in a directory of Parquet and MP4 shards.

That directory is a LeRobot dataset. Over 90,000 such datasets already live on the Hub from more than 8,000 publishers. The new part is what happens next: the same shards sync into a Hugging Face Storage Bucket, a mutable, non‑versioned, Xet‑backed object store announced in March 2026. The bucket sits beside your dataset repositories in the same hf:// namespace and uses the hf CLI you already have.

The bucket is a working layer, not an archive

Versioned repositories turn every append into a commit and keep every revision. Collection wants the opposite: a place to write bytes and overwrite them in place. Buckets do exactly that. No IAM roles need configuration. No CORS rules apply. No upload service requires maintenance. Your agent records a LeRobotDataset in the same format LeRobot writes on hardware, then calls sync_dataset_to_bucket("/tmp/cube_pick", "my-org/robot-fave"). The sync writes to hf://buckets/{bucket}/{run_id} where run_id defaults to the dataset directory name. The same capability lives on DatasetRecorder.sync_to_bucket and stop_recording(bucket=...) if you drive an open recorder directly.

For the published artifact you still call push_to_hub(). Both hold the same format. The bucket is the layer you write to through the day; the versioned repository gets the versions you choose to publish.

Byte‑level deduplication changes the cost curve

Point two fixed cameras at an arm clearing the same table for eight hours. Most of what you record is pixels you already have: the same lighting, the same chassis, the same background across thousands of episodes. On a versioned repository changing one frame in a multi‑gigabyte video shard re‑uploads the whole file.

Buckets are backed by Xet, which deduplicates uploads at the byte level using content‑defined chunking. Chunk boundaries follow the content, so inserting a few bytes changes only the chunk it lands in instead of shifting every boundary after it. In Hugging Face's own measurements, content‑defined chunking reduces data transferred per upload by about four times across the Hub. On Enterprise plans billing is on the deduplicated footprint.

Their bucket benchmarks show what that looks like on a single file. Starting from a 500 MB upload, changing 1 % of the bytes and re‑uploading moved 5.5 MB. Changing 5 % moved 27.5 MB. Changing 10 % moved 55 MB. Without chunk‑level deduplication overwriting an object means sending all of its bytes again whether or not they changed.

The Strands Robots recorder uses LeRobot's shard layout. Episodes go into Parquet shards (data/chunk-000/file-000.parquet) and per‑camera MP4 shards (videos/observation.images.front/chunk-000/file-000.mp4), rolling to a new file only when the current one fills at LeRobot's defaults of 100 MB for data Parquet and 200 MB for video MP4. A sync after a day of recording uploads the new trailing shards plus the one partially‑filled shard that grew, rather than the whole dataset. Sync the same bucket again tomorrow and Xet handles the deduplication.

Streaming turns the dataset into a torch iterable

To train you point GPUs at your dataset. Download it first and those GPUs sit idle until hundreds of gigabytes finish copying. Streaming straight from the Hub works because of the shard layout: a batch becomes a few byte‑range reads over large shards rather than thousands of small fetches.

LeRobot's StreamingLeRobotDataset turns that into a drop‑in torch iterable. Strands Robots exposes it through stream_dataset():

reader = sim.stream_dataset(
    "my-org/robot-fave/cube_pick",
    repo_type="bucket",
    shuffle=False,
    max_num_shards=1,
    buffer_size=1
)
print(reader.num_episodes, reader.num_frames, reader.fps)
for frame in reader:
    frame["observation.images.front"]  # (3, H, W) tensor, decoded on the fly
    frame["observation.state"]         # joint vector from the Parquet shard
    frame["action"]
    break

Nothing lands on local disk except the small meta/ folder of schema, statistics, and episode index. Camera frames are decoded from the remote MP4 shards as you iterate; state and action come from the Parquet shards. That loop reads one frame at a time, which suits inspecting an episode. To train, pass the reader to a DataLoader and iterate batches instead. The streaming dataset shuffles internally through a bounded reservoir buffer, so video decoding parallelizes across worker processes.

If you would rather not write the loop at all, LeRobot's own trainer