Gurpreet Singh

• Designed and optimised data pipelines using PySpark to process large-scale data from MongoDB, SQL Server, Oracle, various file formats (Parquet, Avro, ORC, and CSV), and Azure Blob Storage. • Collaborated with stakeholders like Bloomberg and Blackrock to fetch and process financial data. • Improved data pipeline efficiency by 30% through DataFrame caching and Spark optimisation techniques. • Developed automated data validation scripts, ensuring 99.9% data accuracy.

• Developed PySpark-based ETL workflows to transform raw data into structured insights for American Express. • Led data localisation projects, ensuring compliance with India-specific regulations. • Optimised Spark jobs by tuning configurations, data partitioning, and caching techniques, reducing execution time by 40%. • Collaborated with cross-functional teams to deliver scalable data engineering solutions.

• Designed and developed ETL pipelines using PySpark for clients including NatWest and Deutsche Bank. • Processed large volumes of structured & unstructured data (CSV, JSON, Parquet, Avro). • Enhanced pipeline performance by implementing Spark optimisations and caching techniques. • Developed PL/SQL scripts, stored procedures, and triggers for TCS’s Corporate Finance system. • Assisted Java development teams in writing optimized SQL queries for application integration.