Job responsibilities
- Provides recommendations and insight on data management, governance procedures, and intricacies applicable to the acquisition, maintenance, validation, and utilization of data
- Designs and delivers trusted data collection, storage, access, and analytics data platform solutions in a secure, stable, and scalable way
- Defines data back-up, reconciliation and archiving strategy
- Creates functional and technical documentation supporting best practices
- Advises junior engineers and technologists
- Evaluates and reports on access control processes to determine effectiveness of data asset security
- Adds to team culture of diversity, equity, inclusion, and respect
Required qualifications, capabilities, and skills
- Formal training or certification on software engineering concepts and 5+ years applied experience.
- Experience across the data lifecycle
- Experience in building Data frameworks, working with Data lakes.
- Experience with Batch and Real time Data processing with Spark or Flink
- Working knowledge of AWS Glue and EMR usage for Data processing
- Experience working with Databricks
- Experience working with Python/Java, PySpark etc.
- Working experience with both relational and NoSQL databases
- Experience Normalising and delivering data for Reporting and Dashboarding purposes.
Preferred qualifications, capabilities, and skills
- Advanced experience with SQL (e.g., joins and aggregations)
- Experience with Batch and Real time Data processing with Spark or Flink
- Working knowledge of AWS Glue and EMR usage for Data processing
- Experience working with Databricks
- Experience working with Python/Java, PySpark etc.