MSD Data Engineer 

India, Maharashtra 

186179056

Job Description

Primary Responsibilities:

· Design, develop and maintain data pipelines to extract data from a variety of sources and populate data lake and data warehouse

· Develop the various data transformation rules and data modeling capabilities

· Collaborate with Data Analyst, Data Scientists, Machine Learning Engineers to identify and transform data for ingestion, exploration, and modeling

· Work with data governance team and implement data quality checks and maintain data catalogs

· Use Orchestration, logging, and monitoring tools to build resilient pipelines

· Use test driven development methodology when building ELT/ETL pipelines

· Develop pipelines to ingest data into cloud data warehouses

· Analyze data using SQL

· Use serverless AWS services like Glue, Lambda, StepFunctions

· Use Terraform Code to deploy on AWS

· Containerize Python code using Docker

· Use Git for version control and understand various branching strategies

· Build pipelines to work with large datasets using PySpark

· Develop proof of concepts using Jupyter Notebooks

· Work as part of an agile team

· Create technical documentation as needed

· Bachelor’s Degree or equivalent experience in a relevant field such as Engineering (preferably computer engg.), Computer Science

Required Experience and Skills:

· 4-8 years of relevant experience

· Good experience with AWS services like S3, ECS, Fargate, Glue,

StepFunctions, CloudWatch, Lambda, EMR

· SQL

· Proficient in Python, PySpark

· Good with Git, Docker, Terraform

· Ability to work in cross functional teams

Preferred Experience and Skills

· Any AWS developer or architect certification

· Agile development methodology

Current Contingent Workers apply

*A job posting is effective until 11:59:59PM on the dayBEFOREthe listed job posting end date. Please ensure you apply to a job posting no later than the dayBEFOREthe job posting end date.

A job posting is effective until 11:59:59PM on the day BEFORE the listed job posting end date. Please ensure you apply to a job posting no later than the day BEFORE the job posting end date.