Data Pipeline Design & Development
- Data Engineers are responsible for designing and building robust, scalable, and high-quality data pipelines that support analytics and reporting needs. This includes:
- ETL/ELT development using tools like Azure Data Factory, Databricks, and Snowflake.
- Integration of structured and unstructured data from various sources into data lakes and warehouses.
Cloud Platform Engineering
- They operationalize data solutions on cloud platforms, integrating services like Azure, Snowflake, and third-party technologies.
- Manage environments, performance tuning, and configuration for cloud-native data solutions.
Data Modeling & Architecture
- Apply dimensional modeling, star schemas, and data warehousing techniques to support business intelligence and machine learning workflows.
- Collaborate with solution architects and analysts to ensure models meet business needs.
Data Governance & Security
- Ensure data integrity, privacy, and compliance through governance practices and secure schema design.
- Implement data masking, access controls, and metadata management for sensitive datasets.
Collaboration & Agile Delivery
- Work closely with cross-functional teams including product owners, architects, and business stakeholders to translate requirements into technical solutions.
- Participate in Agile ceremonies, sprint planning, and DevOps practices for continuous integration and deployment.
Key Skills Required
- Programming: Python, SQL, Spark
- Cloud Platforms: Azure, AWS, Snowflake
- Data Tools: DBT, Erwin Data Modeler, Apache Airflow
- Governance: Data masking, metadata management, SOX compliance
- Soft Skills: Communication, problem-solving, stakeholder engagement
- Experience in Data Eng > 7 years