Job responsibilities
- Develops, and maintain robust data pipelines and ETL processes to ingest, process, and store large volumes of data from various sources.
- Collaborates with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
- Optimizes and improves existing data systems for performance, scalability, and reliability.
- Implement data quality checks and validation processes to ensure data accuracy and integrity.
- Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal disruption.
- Stay up-to-date with industry trends and best practices in data engineering and incorporate them into our processes.
- Adds to team culture of diversity, equity, inclusion, and respect
Required qualifications, capabilities, and skills
- Formal training or certification on software engineering concepts and 2+ years applied experience
- Hands-on practical experience in system design, application development, testing, and operational stability
- Experience in developing, debugging, and maintaining code in a large corporate environment with one or more modern programming languages and database querying languages
- Demonstrable ability to code in one or more languages, such as, Python, Java or Spark
- Experience across the whole Software Development Life Cycle
- Exposure to agile methodologies such as CI/CD, Application Resiliency, and Security
- Emerging knowledge of software applications and technical processes within a technical discipline (e.g., cloud, artificial intelligence, machine learning, mobile, etc.)
- Hands on experience as a Data Engineer or similar role.
- Strong proficiency in SQL and relational databases (e.g., MySQL, PostgreSQL ).
- Experience with containerization and orchestration tools, especially Kubernetes.
- Proficiency with AWS services like EKS, EMR, Lambda, DynamoDB, and ECS .
Preferred qualifications, capabilities, and skills
- Knowledge of Hadoop, Spark or Terraform concepts and frameworks
- Familiarity with data warehousing solutions, especially Snowflake, and ETL tools
- Experience with Airflow or AWS MWAA
- AWS Certification