Kamalakannan (Kamal) Palaniappan

Working on Proof of Concept to explore features in Microsoft Fabric, especially usage Onelake for existing Azure Data lakehouse. Guiding 5+ Data engineers in building Wearable health devices Streaming Data processing solutions to achive near realtime data processing. Had mentored and organized upskill training for 3 entry level trainers every year and also for new joiners in our team.

Developed 20+ Data pipelines using Azure Data Factory, Azure Databricks, Pyspark and SQL. Create 50+ data migration pipelines to move 500 TB of on-premise data to Azure cloud. Worked on Proof concept to capture Streaming data near realtime using Azure IoT device, Azure Event Hubs and ADLS. Trained in Power BI Desktop and Services and also Power Query Editor, Workspaces and Power BI Embedded Reports

Created ETL batch Data pipelines using Azure Data Factory, ETL cleaning and transformations using Azure Data bricks with Pyspark and SparkSQL. CI/CD with Azure DevOps

Lead team of 5 team members, assigning project as Stories in each sprint, Agile sprint planning, Story grooming, Story tracking in RTC, Requirements clarification with Client and onsite lead, prepare detailed design and code migration.

Developed ETL using pipelines using DataStage to process more than 4.5 billion rows. Data Analysis with SQL and Job execution with Unix Shell scripting.

Developed ETL Data Integration pipelines using DataStage. SQL for Data analysis and Job execution with Unix Shell scripting.