Owen Dyer

• Overseen the entire working operations of Data Modelling, ETL and Data Warehousing operations of Big Data. • Performed in-depth analysis of streaming on Kafka and Spark. • Coded in database languages - Presto, Hive, Spark SQL, MySQL, Oracle PL/SQL.

• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself. • Developed multiple POCs using Scala and deployed on the Yarn cluster, compared the performance of Spark with Hive and SQL/Teradata.

• Developed Spark Applications by using Spark, Java and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources. • Handled importing of data from various data sources, performed data control checks using Spark and loaded data into HDFS.

• Developed optimal strategies for distributing the web log data over the cluster importing and exporting the stored web log data into HDFS and Hive using Sqoop. • Design and develop ELT data pipeline using Spark App to fetch data from Legacy system and third-party API, social media sites.

• Developed optimal strategies for distributing the web log data over the cluster importing and exporting the stored web log data into HDFS and Hive using Sqoop. • Design and develop ELT data pipeline using Spark App to fetch data from Legacy system and third-party API, social media sites.