Saurabh Kumar

Created data management dashboard to manage and visualize metadata of 130 PiBs collected data. Engineered a data grading framework using python to optimize data quality and lifecycle management. Implemented Analytics layer for decision support and established data pipelines for efficient ELT processes using dbt on snowflake. Identified and developed Data products for Finance data hub to enhance efficiency across the finance domain. Designed data products related to Supplier Invoices and Subscriptions etc.

Designed a cloud-agnostic data pipeline utilizing PySpark to predict retail product removals and enhance sales, ensuring cross-platform adaptability. Engineered streaming pipelines for ingesting data of various formats and loading it into structured tables. Developed a framework for data ingestion and transformation, optimizing resource utilization on Azure and GCP platforms. Oversaw the migration of on-premises systems to cloud infrastructure.

* Managed data contracts with source applications and utilized Azure EventHub streaming services for efficient data ingestion. * Developed streaming pipelines to ingest json files into tables within Azure Data Lake Storage Gen 2 (ADLS Gen 2). * Structured a framework for data ingestion and flattening processes, optimizing data transfer to Azure SQL Data Warehouse. * Generated dashboard to monitor and track all project processes, providing easy-to-access insights for performance monitoring.

Developed framework for comprehensive data quality checks. Orchestrated pipeline from S3 buckets to PostgreSQL tables, ensuring efficient data storage and retrieval using pyspark. Established data quality framework with PySpark, executing on a transient EMR cluster in AWS. Migrated SQL server procedures to Databricks jobs for enhanced data processing and analysis. Produced real-time Power BI reports to monitor ongoing processes via a user-friendly dashboard for stakeholders.

Migrated Informatica mappings to configurable spark pipeline, enhancing data processing capabilities for customer invoicing. Collaborated closely with clients to ensure precise requirement gathering and alignment with project objectives. Facilitated seamless data import and export between Oracle RDBMS and HIVE. Implemented SCD logic using Spark. Designed and developed generic mappings capable of executing dynamic SQL queries to efficiently process data for transaction analysis.