Data Engineering Intern, Incremental Data Pipeline for Medical Claims using Azure and PySpark
Built an incremental data pipeline for medical claims data using Azure and PySpark, supporting creation of structured datasets used for AI training. Simulated hourly ingestion from CSV sources and converted them into Parquet for reliable downstream consumption. Implemented surrogate key generation and fact/dimension structuring to enable consistent labeling and model feature usage. • Used PySpark and Azure Data Lake Storage to process a 1M-record dataset • Triggered Azure Data Factory pipelines for CSV-to-Parquet conversion using MD5-hash based events • Structured transformed data into fact and dimension tables in the sink layer • Managed archival and purge policies to optimize storage while preserving data needed for training