2023–2025 · Data Engineering
SkyCAST Data Pipelines
PySpark/Databricks pipelines feeding an airline-network planning platform at Amadeus.
- Passenger and network data
- PySpark on Databricks
- Planning datasets
Professional work (private codebase)Private code, no public repository
Problem
Network-planning and calibration teams needed passenger and network data integrated into reusable, trustworthy datasets, and a forecasting component that a Python-based team could inspect.
My role
Data Scientist (Data & ML Engineering) at Amadeus: built and maintained the transformations and reimplemented a forecasting component.
Approach
- Built PySpark transformations on Databricks integrating airline passenger and network data (T100, MIDT) into reusable datasets.
- Reimplemented a C# demand-forecasting component in PySpark and compared both implementations on identical inputs.
- Integrated a Transformer-based demand model (QSI) and analysed factor contributions across 6+ scoring dimensions.
- Documented calibration notebooks and Power BI dashboards for downstream users.
Results
- Reported difference between the C# and PySpark implementations: about 0.0005% on tested inputs (from CV).
Stack
- PySpark
- Databricks
- Azure
- SQL
- Power BI
- Jupyter