Job responsibilities
- Provides guidance to immediate team of data engineers on daily tasks and activities
- Sets overall guidance and expectations for team output, practices, and collaboration
- Anticipates dependencies with other teams to deliver products and applications in line with business requirements
- Manages stakeholder relationships and the team’s work in accordance with compliance standards, service level agreements, and business requirements
- Create a culture of diversity, equity, inclusion, and respect for team members and prioritize diverse representation
- Develop data strategy (and enterprise data models for applications), Manage data infrastructure (design, construct, install, and maintain large-scale processing systems and infrastructure), Drive data quality, Ensure data accessibility (to analysts and data scientists), Ensure compliance with data governance requirements, and Ensure business alignment (ensure data engineering practices align with business goals)
Required qualifications, capabilities, and skills
- Formal training or certification in software engineering concepts and 5+ years of applied experience
- Experience leading technology projects and managing technologists
- Proficient in all aspects of the data lifecycle
- In-depth knowledge of the financial services or ecommerce or Travel industry and their IT systems
- Advanced proficiency in atleast one programming language (preferably Python, alternatively Java or Scala)
- Advanced proficiency in atleast one cluster computing frameworks (preferably Spark, alternatively Flink or Storm)
- Advanced proficiency in atleast one cloud data lakehouse platforms (preferably AWS data lake services or Databricks, alternatively Hadoop), atleast one relational data stores (Postgres, Oracle or similar) and atleast one NOSQL data stores (Cassandra, Dynamo, MongoDB or similar)
- Advanced proficiency in atleast one scheduling/orchestration tools (preferably Airflow, alternatively AWS Step Functions or similar)
- Proficiency in Unix scripting, data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), big-data storage formats (Parquet, Iceberg, or similar), data processing methodologies (batch, micro-batching, and stream), one or more data modelling techniques (Dimensional, Data Vault, Kimball, Inmon, etc.), Agile methodology, TDD (or BDD) and CI/CD tools
- Able to coach team members in continuous improvement of the product
Preferred qualifications, capabilities, and skills
- Budgeting and resource allocation
- Vendor relationship management
- Proficiency in Snowflake
- Proficiency in IaC (preferably Terraform, alternatively AWS cloud formation)
- Proficiency in cloud based data pipeline technologies such as- Fivetran, DBT, Prophecy.io, etc.