2025 · Data Engineering
Azure HDFS Sync
An incremental Azure Blob Storage download pipeline for geophysical simulation outputs.
- Azure Blob Storage
- Download missing HDF5 files
- Local simulation runs
Public engineering projectGitHub ↗
Problem
Repeated analysis of geophysical simulation outputs requires an organized local copy without re-downloading completed runs.
My role
Built a Python utility for retrieving and organizing simulation outputs.
Approach
- List simulation outputs using the Azure SDK for Python.
- Identify missing local HDF5 files and download only those files.
- Group the retrieved files by simulation run for inspection.
Results
- The repository documents the architecture and includes a sample-data workflow for trying the downloader.
Stack
- Python
- Azure SDK
- Azure Blob Storage
- HDF5