Job responsibilities
- Design, build, and maintain efficient data pipelines using Spark, Python, and Databricks.
- Assist in migrating data from Omni-AI to Databricks, ensuring accuracy and consistency throughout the process.
- Work closely with data scientists and analysts to understand data needs and deliver effective solutions.
- Identify and implement improvements to data workflows, enhancing performance and reducing latency.
- Implement and monitor data quality checks to ensure reliable and accurate data processing.
Required qualifications, capabilities, and skills
- Formal training or certification on software engineering concepts and 3+ years applied experience
- Must hold a relevant AWS certification, demonstrating proficiency in cloud-based data engineering solutions.
- Strong experience in using Spark and Python for developing and maintaining data pipelines.
- Solid understanding of Databricks, including its architecture, features, and integration within data workflows.
- Proven experience in data engineering, including data modeling, ETL processes, data integration, and pipeline optimization.
- Ability to troubleshoot and resolve complex data-related issues effectively.
- Excellent communication skills with the ability to work effectively in cross-functional teams.
Preferred qualifications, capabilities, and skills
- Migration experience from On-prem to AWS
- Exposure to any cloud technologies