AI Agent Evaluation Engineer

Canada (Remote); United States (Remote)

first seen Nov 13, 2026updated Nov 12, 2026

When this one is worth applying to

Assisted apply Apply on Cresta's posting

Assisted apply drafts a resume, a cover letter and screening answers for this posting from your Source of You record, and drops figures it cannot find there. Applying always happens on the company's own page.

The posting, in Cresta's own words

archived Nov 13, 2026

3 skills recognized in this posting, underlined below. Sign up free to see which ones your record already covers →

Cresta unlocks the true potential of the customer experience, turning every conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive conversation intelligence to drive revenue and efficiency gains across every channel. The world’s leading companies, including United Airlines, Cox Communications, and Marriott, use Cresta to power world-class customer experiences every day. Born from the Stanford AI Lab, Cresta has raised more than $270 million from the world’s leading investors, including a16z, Greylock, and Sequoia. Cresta’s leadership includes some of the leading minds in AI today. Our CEO, Ping Wu , founded and led Google's Contact Center AI and Vertex AI platforms before joining Cresta to build the future of AI-driven customer experiences. Over the next few years, AI is going to redefine how people all over the world interact with businesses every day. Come build that future at Cresta.

About the role:

Cresta’s AI Receptionist answers the phone for small businesses that would otherwise miss the call. It handles millions of conversations a month today, making sure no call goes unanswered and helping owners complete the front-office tasks that keep their business running, and it does so across hundreds of customers. Whether those conversations are actually good is the single most important question in the business, and it deserves a full-time owner. That’s you. As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across every conversation, and you are the person the company trusts when the question is “how good is it, really?” The work combines hands-on conversation analysis, rigorous evaluation design, and engineering execution: reading and annotating real conversations to learn how the agent actually fails, building the automated evaluation systems (LLM-based judges, golden sets, regression suites) that measure those failures at scale, and keeping those systems honest by calibrating them against human judgment. Fixing what you find is welcome but not the core of the job; an accurate, prioritized picture of agent quality is.

Read the full posting ↓

Evaluation Design & Quality Definition

What We're Looking For:

3+ years of professional experience in software engineering, applied ML, or data-intensive research roles, including at least 1–2 years working hands-on with LLM-based systems in production. Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience. Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite. Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails. Tool-agnostic; you will work in our in-house tooling. Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state. You can read a trace and tell what the agent was doing and why. Strong programming skills in Python and experience with LLM APIs; able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required. Able to write a rubric that independent reviewers would score the same way, and to measure whether they do. Able to design a test from an open-ended quality question: you decide what to measure and how, rather than executing a predefined plan. Willing to do non-scalable work weekly, both yourself and by directing our QA partners: listening to calls, labeling conversations, building a failure taxonomy from scratch. Professional fluency in at least one language other than English, sufficient to evaluate multilingual agent behavior directly. Strong written communication; you author the guidelines others follow. Note on Our Hiring Process We highly value your time and engineering craftsmanship. To ensure a streamlined and objective process, the first step for qualified applicants following an initial resume review is a self-paced CoderPad technical assessment. This allows you to showcase your practical coding and problem-solving skills in a real-world environment before moving into conversational interview rounds.

AI Agent Evaluation Engineer

The posting, in Cresta's own words

About the role:

Evaluation Design & Quality Definition

Conversation Analysis & Annotation

Scalable Evaluation Systems

Coverage & Edge Cases

Reporting & Cross-Functional Collaboration

What We're Looking For:

Perks & Benefits:

Compensation at Cresta: