Job Summary

Leads the data engineer team in the development of distinctive analytical and risk insights that deliver improvements in business results through the application of advance analytical techniques to complex data sets.

Job Qualifications

• 7+ years data engineering experience, including 3+ years in leadership or lead roles.
• Strong skills in Spark SQL, Python, PySpark, Databricks, and designing scalable pipelines on cloud architecture
• Expertise in dimensional modeling, star schema, SCD, and workload optimization.
• Experience with automated testing, CI/CD pipelines, and version control systems.
• Proven ability to optimize Spark jobs and manage cloud data platforms (AWS, Azure, GCP).
• Excellent communication and stakeholder engagement abilities. Ability to connect business and technical domain.
• Proficient with IDEs (e.g., VS Code) and GitHub or equivalent tools.

Job Accountabilities - Key Accountabilities

• Lead design and implementation of ETL/ELT pipelines using Spark SQL and Python on Databricks Medallion architecture, with robust dimensional modelling optimized for analytics workloads.
• Grow, manage and mentor a high-performing data engineering team aligned with modern best practices.
• Implement and manage CI/CD pipelines, version control, data quality framework and automated testing for data workflows.
• Drive Spark SQL and DataFrame performance tuning with partitioning, clustering, and join optimizations.
• Collaborate cross-functionally to align data engineering with business goals.
• Maintain technical documentation: ERDs, data lineage, schema specs, and metadata.
• Evaluate new tools and technologies to enhance data engineering capabilities.