Job Responsibilities:
- Write efficient Python and SQL code to extract, transform, and load (ETL) data from various sources into Databricks.
- Perform data analysis and computation to derive actionable insights from the data.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Ensure data quality, integrity, and security across all data processes.
- Develop Optimized solutions for performance and scalability. Monitor and troubleshoot data workflows to ensure reliability and efficiency.
- Document data engineering processes, methodologies, and workflows. Communicating analytical findings to senior leaders through data visualization and storytelling
Required qualifications, capabilities and skills
- 3-5 years of Develop, implement, and maintain data pipelines in Databricks to support data integration and analytics.
- Develop, implement, and maintain data pipelines in Databricks to support data integration and analytics. Write efficient Python and SQL code to extract, transform, and load (ETL) data from various sources into Databricks.
- Perform data analysis and computation to derive actionable insights from the data.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Ensure data quality, integrity, and security across all data processes.
- Optimize data pipelines for performance and scalability.
- Monitor and troubleshoot data workflows to ensure reliability and efficiency. Document data engineering processes, methodologies, and workflows.
Preferred qualifications, capabilities and skills
- Certification in Databricks preferred.
- Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
- Knowledge of machine learning and data science concepts.
- Experience with data visualization tool Tableau will be plus