Role Description:
As a Machine Learning Scientist, your work will focus on theandof generative AI systems. You will develop and fine-tune Judge LLMs to assess model outputs across a variety of tasks, design robust evaluation frameworks for agentic workflows, and build scalable pipelines for synthetic data generation. The team also plays a critical role in multilingual evaluation, enabling GenAI applications to support market expansion across all supported languages.
Key Job Responsibilities and Duties:
Develop and apply state-of-the-art techniques for evaluating generative AI systems, with a focus on agent workflows, multilingual output, and task-specific Judge LLMs.
Design and implement scalable evaluation pipelines, including synthetic data generation and benchmarking for model quality, relevance, and consistency..
Optimize and maintain Judge LLMs to assess outputs across dialog systems, Q&A, and trip planning use cases.
Conduct in-depth data analysis to define and track evaluation metrics, validate label quality, and explore performance across different languages and user scenarios.
Ensure the reliability, efficiency, and scalability of evaluation tools and frameworks in both offline and online environments.
Collaborate closely with ML engineers to integrate evaluation components into production pipelines, supporting continuous improvement of GenAI applications.
Work cross-functionally with product, research, and analytics teams to align evaluation strategies with business goals and user impact.
Qualifications & Skills:
Advanced knowledge and experience in Computer Vision and Natural Language Processing, engineering aspects of developing ML and GenerativeAI models at scale.
Experience designing and executing end-to-end research and development plans and generating impact through large-scale machine learning model development. Preferably evidenced by peer-reviewed publication, patents, open sourced code or the like.
Relevant work or academic experience (MSc + 4 years of working experience, or PhD + 2 years of working experience) , involved in the application of Machine Learning to business problems.
Masters degree, PhD or equivalent experience in a quantitative field (e.g. Computer Science, Engineering Mathematics, Artificial Intelligence, Physics, etc.).
Experience on multiple machine learning facets: working with large data sets, model development, statistics, experimentation, data visualization, optimization, software development.
Experience collaborating cross functionally in the development of machine learning products (e.g. Developers, UX specialists, Product Managers, etc.).
Strong working knowledge of Python, Java, Kafka, Hadoop, SQL, and Spark or similar technologies. Working experience with version control systems.
Excellent English communication skills, both written and verbal.
Successfully driving technical, business and people related initiatives that improve productivity, performance and quality while communicating with stakeholders at all levels
Leading by example, gaining respect through actions, not your title. Developing your team and motivating them to achieve their goals. Providing feedback timely and managing your key team performance indicators
Booking.com’s Total Rewards Philosophy is not only about compensation but also about benefits. We offer a competitive , as well unique-to-Booking.com benefits which include:
Annual paid time off and generous paid leave scheme including: parent, grandparent, bereavement, and care leave
Hybrid working including flexible working arrangements, and up to 20 days per year working from abroad (home country)
Industry leading product discounts - up to 1400 per year - for yourself, including automatic Genius Level 3 status and Booking.com wallet credit
Application Process:
Let’s go places together:
This role does not come with relocation assistance.
Pre-Employment Screening
If your application is successful, your personal data may be used for a pre-employment screening check by a third party as permitted by applicable law. Depending on the vacancy and applicable law, a pre-employment screening may include employment history, education and other information (such as media information) that may be necessary for determining your qualifications and suitability for the position.
