Kamalakannan (Kamal) Palaniappan

Failure Remediation Multi Agentic AI workflow, that can automatically fix pipeline failures, which results in 20% cost saving in maintaining pipelines. Created pipeline diagnostic report generation which saves 30% of effort in monitoring pipelines in Production. Processed Streaming sensor data from 50k wearable health device, which reduces the altering time 90% near realtime Working on Proof of Concept to explore features in Microsoft Fabric, especially usage Onelake for existing Azure Data lakehouse.

Developed 20+ Data pipelines using Azure Data Factory, Azure Databricks, Pyspark and SQL. Create 50+ data migration pipelines to move 500 TB of on-premise data to Azure cloud. Worked on Proof concept to capture Streaming data near realtime using Azure IoT device, Azure Event Hubs and ADLS. Trained in Power BI Desktop and Services and also Power Query Editor, Workspaces and Power BI Embedded Reports

Created ETL batch Data pipelines using Azure Data Factory, ETL cleaning and transformations using Azure Data bricks with Pyspark and SparkSQL. CI/CD with Azure DevOps

Lead team of 5 team members, assigning project as Stories in each sprint, Agile sprint planning, Story grooming, Story tracking in RTC, Requirements clarification with Client and onsite lead, prepare detailed design and code migration.

Developed ETL using pipelines using DataStage to process more than 4.5 billion rows. Data Analysis with SQL and Job execution with Unix Shell scripting.

Developed ETL Data Integration pipelines using DataStage. SQL for Data analysis and Job execution with Unix Shell scripting.