About the role:
- This role involves managing data ingestion processes on the Databricks platform, requiring expertise in Data Lake ingestion, ETL/ELT design, and CI/CD practices.
- You will design, implement and manage data ingestion processes from a variety of source systems, ensuring best practices are followed and tailoring approaches to each system.
- You will be responsible for developing, maintaining, and optimizing ETL/ELT pipelines while ensuring adherence to security, compliance, and PII management standards.
- Additionally, this position requires knowledge of system integration tools and effective data pipeline management.
- You'll implement automated monitoring to maintain pipeline infrastructure reliability.
- This role also includes contributing to Agile sprints and ensuring adherence to SDLC best practices.
- The ideal candidate will apply ingestion best practices across diverse source systems, tailoring approaches as needed.
About you:
- You should have 5 to 8 years of experience in data engineering, ingestion pipelining, and ETL/ELT,
- You should have proficiency in SQL and programming languages like Python or PySpark, as well as experience with Databricks and cloud platforms such as AWS.
- Hands-on expertise with Spark/Scala, ETL/ELT development using tools like Apache Airflow, and ingestion tools such as Dell Boomi is required.
- A solid understanding of data quality guidelines, CI/CD pipelines, and Agile sprint management is also necessary.
- Familiarity with Git and version control systems is expected.
- Experience in leading small, high-performing teams of data engineers is essential.
- Product subscription business model domain is a plus.
- Knowledge on Kong-API and Kinesis is a plus.
We work hard to embrace diversity and inclusion and encourage everyone at McAfee to bring their authentic selves to work every day. We offer a variety of social programs, flexible work hours and family-friendly benefits to all of our employees.
- Bonus Program
- Pension and Retirement Plans
- Medical, Dental and Vision Coverage
- Paid Time Off
- Paid Parental Leave
- Support for Community Involvement