Abdul Manan

• Design & maintain scalable Azure Databricks (PySpark), ADF, & Delta Lake ETL pipelines in ADLS. • Build vector search, embeddings, & RAG data pipelines supporting AI agents and LLMs. • Integrate real-time streaming using Azure Event Hubs & Apache Kafka for event-driven systems. • Collaborate with global teams to deliver Power BI datasets & maintain Azure DevOps/GitHub CI/CD.

• Developed Azure Databricks & PySpark pipelines, integrating multi-source ETL across ADF & ADLS. • Applied Spark optimization techniques, including AQE, broadcast joins, shuffle tuning, & salting. • Designed & maintained high-throughput data ingestion flows using Apache NiFi for low-latency ETL. • Built Kafka streaming pipelines with Delta Lake governance; managed CI/CD in Azure DevOps & GitHub.

• Built end-to-end Azure Data Engineering pipelines utilizing ADF, ADLS, Databricks, & Synapse. • Implemented Medallion Architecture (Bronze/Silver/Gold) for analytics-ready Power BI data. • Managed SQL-based ETL transformations across MySQL, Postgres, Oracle, MongoDB, & Apache NiFi. • Developed interactive BI dashboards and reporting models using QlikView, Qlik Sense, & Power BI.