Responsibilities

• Use your skills and experience to design and develop our innovative big data platform
• Use big data frameworks such as Spark, Kafka, Redshift, Airflow, Flink, k8s, Hudi and so on to develop large scale data pipelines
• Lead, deliver and own high scale projects end to end
• Work closely with the data science team
• Lead technological initiatives and improvements while instilling development of “best practices”
• Develop processes and tools to monitor, analyze, maintain and improve data operation, performance, and usability

• At least 2 years experience in Scala/Java or equivalent language
• Proven experience in building, optimizing, and maintaining large scale big data pipelines using popular open-source frameworks (Kafka, Spark, Hive, Presto, Airflow, etc)
• Strong SQL abilities and hands on experience with SQL and NoSql DBs
• Experience with AWS cloud services such as EMR, Redshift, S3, Athena, Glue - advantage
• Experience in Python
• Experience in stream processing frameworks such as Flink
• Experience with scientific computing packages (numpy, scipy, pandas etc.) - advantage