End-to-end data pipeline transforming Olist e-commerce data through Azure cloud services. Implements medallion architecture (Bronze-Silver-Gold) with multi-source ingestion, Spark-based processing, and OLTP-to-OLAP optimization for analytics-ready datasets.

Project based on application of azure databricks

This project implements a real-time data pipeline with Kafka, Spark, and MongoDB. It generates vehicle data using UXSIM, streams it to a Kafka broker, processes it with Spark, and stores raw and processed data in MongoDB. Queries analyze vehicle counts, speeds, and routes over specified periods.

This project performed data wrangling, analysis, visualization as well as machine learning prediction on a hypothetical music app's user churn with pyspark.

Sample analysis done using pyspark on parking violations issued for fiscal year 2017 using the databricks platform

ontains the code and examples for my article on Medium, which introduces the English SDK for Apache Spark, showcasing how to combine the power of Apache Spark with large language models (LLMs)

A repository concentrating on using High end parallel pipelines to perform ETL across various data sources

NBA shot predictions with PySpark and SparkML

1.This is an end-to-end ETL pipeline to ingest data from AWS S3, validate it, process transformations using PySpark, and load final outputs into MySQL fact and dimension tables. 2.Designed a star schema and created optimized data marts using PySpark SQL and parquet processing. 3.Automated S3 operations (download, upload, archival) and implemented a