EY EY-GDS Consulting-AI DATA- Azure Databricks-Senior 

India, Karnataka, Bengaluru 

27664947

How you will contribute:

• Develop and maintain scalable data pipelines, in line with ETL principles, and build out new integrations, using AWS/Azure native technologies, to support continuing increases in data source, volume, and complexity.
• Define data requirements, gather, and mine data, while validating the efficiency of data tools in the Big Data Environment.
• Lead the evaluation, implementation and deployment of emerging tools and processes to improve productivity.
• Implement processes and systems to provide accurate and available data to key stakeholders, downstream systems, and business processes.
• Partner with Business Analysts and Solution Architects to develop technical architectures for strategic enterprise projects and initiatives.
• Coordinate with Data Scientists to understand data requirements, and design solutions that enable advanced analytics, machine learning, and predictive modelling.
• Mentor and coach junior Data Engineers on data standards and practices, promoting the values of learning and growth.
• Foster a culture of sharing, re-use, design for scale stability, and operational efficiency of data and analytical solutions.
  

Minimum Requirements/Qualifications:

• Bachelor's degree in engineering, Computer Science, Data Science, or related field
• 5-9 years of experience in software development, data science, data engineering, ETL, and analytics reporting development
• Experience in building and maintaining data and system integrations using dimensional data modelling and optimized ETL pipelines.
• Experience in design and developing ETL pipelines using ETL tools like IICS, Datastage, Abinitio, Talend etc.
• Proven track record of designing and implementing complex data solutions
• Demonstrated understanding and experience using:
  • Data Engineering Programming Languages (i.e., Python, SQL)
  • Distributed Data Framework (e.g., Spark)
  • Cloud platform services (AWS/ Azure preferred)
  • Relational Databases
  • DevOps and continuous integration
  • AWS knowledge on services like Lambda, DMS, Step Functions, S3, Event Bridge, Cloud Watch, Aurora RDS or related AWS ETL services
  • Azure knowledge on services like ADF, ADLS, etc.
  • Knowledge of Data lakes, Data warehouses
  • Databricks/Delta Lakehouse architecture
  • Code management platforms like Github/ Gitlab/ etc.,
• Understanding of database architecture, Data modelling concepts and administration.
• Handson experience of Spark Structured Streaming for building real-time ETL pipelines.
• Utilizes the principles of continuous integration and delivery to automate the deployment of code changes to elevate environments, fostering enhanced code quality, test coverage, and automation of resilient test cases.
• Proficient in programming languages (e.g., SQL, Python, Pyspark) to design, develop, maintain, and optimize data architecture/pipelines that fit business goals.
• Strong organizational skills with the ability to work multiple projects simultaneously and operate as a leading member across globally distributed teams to deliver high-quality services and solutions.
• Excellent written and verbal communication skills, including storytelling and interacting effectively with multifunctional teams and other strategic partners
• Strong problem solving and troubleshooting skills
• Ability to work in a fast-paced environment and adapt to changing business priorities
  

Preferred requirements:

• Master's degree in engineering specialized in Computer Science, Data Science, or related field
• Demonstrated understanding and experience using:
  • Knowledge in CDK
  • Experience in IICS Data Integration tool
  • Job orchestration tools like Tidal/Airflow/ or similar
  • Knowledge on No SQL
• Proficiency in leveraging the Databricks Unity Catalog for effective data governance and implementing robust access control mechanisms is highly advantageous.
• Databricks Certified Data Engineer Associate
• AWS/Azure Certified Data Engineer
  

EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.