Rapid7 Staff Big Data Software Engineer 

United Kingdom, Northern Ireland, Belfast 

957739546

About the Role

Technologies you will work with:

• Trino

• Iceberg

• Parquet

• Spark

• Airflow

• Kafka

• AWS services such as Glue, S3, EKS

In this role, you will:

• Analyse and optimise distributed SQL queries to improve performance

• Suggest optimisations to our data pipelines

• Provide recommendations for efficient partitioning strategies and schema designs

• Conduct performance tuning for the data pipelines and queries

• Develop performance monitoring strategies and tools

The skills you'll bring include:

• 5+ years of hands-on software engineering experience, with a specific focus on database query optimization

• Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design

• Experience in continuously monitoring and optimising data pipelines for performance and cost-effectiveness

• Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams

• Skills to analyse and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements

• Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation

• Excellent verbal and written communication skills.

• Strong, creative problem solving ability.

Nice to haves:

• Trino/Presto data-mesh

• AWS, Terraform, Kubernetes

• Java

• Kafka