Finding the best job has never been easier
Share
You will be immersed in a substantial business challenge, tasked with creating architecture, designing documents, constructing logical and physical data models, and building intricate ETL pipelines. All of this will be accomplished using a suite of AWS tools such as Lambda, Redshift, S3, DynamoDB, Glue, and Hive. Additionally, you will be an integral part of a team dedicated to supporting the ISS organization in maintaining and facilitating easy access to our definitive source of metric data.Key job responsibilities
• Construct intricate ETL pipelines utilizing Datanet (Redshift) and Craddle (Spark Sql ).
• Create and manage datasets within AWS Dynamo DB, AWS S3, and Amazon Redshift.
• Develop complex ETL workflows employing AWS Glue.
• Orchestrate step functions through Python / Scala.
• Implement and enforce rigorous data quality standards and data governance policies, ensuring data accuracy, consistency, and security. Establish procedures for data validation, cleansing, and lineage tracking.
• Continuously monitor and optimize data pipelines and processes to enhance performance and efficiency. Identify bottlenecks and implement enhancements to expedite data processing and minimize latency.
• Collaborate closely with data analysts and visualization experts to facilitate data-driven decision-making. This includes designing and maintaining data reporting and visualization solutions.
• Demonstrate profound expertise in AWS data services, including Amazon S3, Glue, EMR, Redshift, Athena, and more. Select the most appropriate services for specific data engineering tasks and leverage their capabilities effectively.
• Develop and maintain mission-critical applications extensively used by sellers worldwide.
• Have good knowledge on scripting using either Python or Scala
- 5+ years of data engineering experience
- Experience with data modeling, warehousing and building ETL pipelines
These jobs might be a good fit