Salesforce Evaluations Operations Manager 

United States, Indiana, Indianapolis 

926266961

Job Category

Program & Project Management

Job Details

As the Evaluations Operations Manager for the Digital Success Data & AI Team, you will be accountable for executing and analyzing opportunities for our performance baselines across Answer Quality, Latency, Instructions Validation and other Agentforce capabilities for our Agentforce on Salesforce Help.This role is also accountable foranalyzing these baselines for new opportunities and innovations to increase overall performance.As an Evaluations Operations Manager, you will work with various partner teams including Support Delivery, Engineering, AI Operations, Data Science and more to help educate and inform our Agentforce on Help performance roadmap.The Evaluations Operations Manager helps to contribute and build out our repository of LLM Judge Prompts, Ground Truths, Golden Pairs, and other synthetic testing inputs, as well as providing feedback for automated testing improvements, and capabilities; both custom and native Agentforce capabilities.
Your Impact

• Maintain a regular baselining program for Agentforce performance using synthetic and automated tooling.

• Analyze results to help identify opportunities and innovations to our AI model, implementation, strategy and solutions.

• Build documentation and business review materials to inform partners, stakeholders, and leadership.

• Support a feedback loop of automated testing, LLM Judge Prompts, and Ground Truth / Golden Pairs for ongoing testing sophistication.

• Perform on demand evaluations as needed based on special projects or out of cycle evaluation requests.

• Support a calibration and review process for manual assessment of synthetic testing to ensure quality of automated testing.

Required Skills

• Knowledge Generative AI Technology and an understanding of Generative AI functionality principals. Specifically Agentforce, Data Cloud or similar applications.

• Expertise in establishing consistent frameworks for qualitative and quantitative data analysis.

• Program or product management experience (entry-level is okay)

• Experience with data story telling, and communicating results to various audiences.

• Strong written and communication skills; specifically where maintaining regular communications such as newsletters, blogs, agendas or other programmatic communication.

• Strong reading comprehension skills that support the interpretation of generative AI responses and their overall quality inline with Salesforce calibrated requirements.

• Experience working with Agile teams and Agile methodologies.

If you require assistance due to a disability applying for open positions please submit a request via this.

Posting Statement

For Washington-based roles, the base salary hiring range for this position is $101,900 to $140,200. For Illinois based roles, the base salary hiring range for this position is $101,900 to $140,200. Compensation offered will be determined by factors such as location, level, job-related knowledge, skills, and experience. Certain roles may be eligible for incentive compensation, equity, benefits. More details about our company benefits can be found at the following link: