Hugging Face Xet Storage Layer Enables Incremental Uploads
ORIGINAL / HF Buckets: Upload Only What Changed with Xet
Hugging Face's Xet storage layer uploads only changed data pieces instead of re-uploading entire files when data changes. For users frequently updating large datasets, this saves bandwidth and time, improving workflow efficiency.
01 ABSTRACT
The video demonstrates how Hugging Face's Xet storage layer uploads only changed parts of files. Specifically, after uploading a 191 MB CSV and appending 100 rows, only new data is transferred; after converting to Parquet and prepending 100 rows, only 8.6 MB of a 52.9 MB file is sent. The author finds this useful for engineers who regularly update datasets.
02 KEY FINDINGS
- Xet storage layer uses content-defined chunking to achieve incremental uploads, transferring only changed data chunks.
- In the demo, CSV appending and Parquet prepending operations result in slight data transfers, saving about 90% bandwidth.
- Using PyArrow 21+'s use_content_defined_chunking parameter optimizes Parquet file synchronization.
Return to the primary source
Read original ↗AI GENERATED SUMMARY / DISCOVERED BY HUGGING FACE