Key Responsibilities:
- Design, develop, and maintain robust data pipelines using GCP services such as BigQuery, Dataflow, Pub/Sub, CloudSQL, Cloud Composer (Airflow).
- Build and manage Airflow DAGs for orchestrating complex workflows.
- Write efficient and reusable Python scripts for data processing and automation.
- Develop solutions to move and transform data across GCP projects securely and efficiently.
- Collaborate with data scientists, analysts, and other engineers to ensure data availability and quality.
- Optimize data workflows for performance, scalability, and cost-efficiency.
Required Skills & Qualifications:
- Minimum 4 years of experience in data engineering, with at least 2+ years on GCP.
- Strong expertise in BigQuery, Dataflow, Pub/Sub, CloudSQL, Composer/Airflow.
- Proficient in Python for scripting and automation.
- Experience in designing and implementing data pipelines across GCP projects.
- Familiarity with CI/CD practices and version control (e.g., Git).
- Excellent problem-solving and communication skills.
Nice to Have:
- Experience with Terraform or other IaC tools.
- Knowledge of data governance and security best practices on GCP.
- Exposure to real-time data processing and streaming architectures.
What we look for :People with the ability to work in a collaborative manner to provide services across multiple client departments while following the commercial and legal requirements. You will need a practical approach to solving issues and complex problems with the ability to deliver insightful and practical solutions. We look for people who are agile, curious, mindful and able to sustain positive energy, while being adaptable and creative in their approach.