Evals make selection pressure executable

Torres defines AI evals as methods for measuring whether an AI product or workflow is performing well, and describes them as a feedback loop that can help teams maintain quality and catch issues before they reach users.

For the customer-facing Interview Coach, Torres says she released the product in beta to obtain traces, reviewed hundreds of cases, logged mistakes, and categorized those errors into candidates for evals. She describes a trace as including user input, system prompts, tool calls, intermediate steps, and final LLM responses.

The article describes four common eval types: golden datasets, code assertions, LLM-as-a-Judge, and customer feedback. Torres prefers deterministic code assertions when possible because they are fast and cheap, and uses LLM-as-a-Judge when the error requires judgment because those evals are more expensive and require additional alignment work.

Torres describes combining a cheap code assertion with an LLM-as-a-Judge so only cases that cross a structural threshold are sent to the judge, reducing how often the more expensive evaluation must run.

The article recommends establishing a baseline, changing a prompt, context, model, decomposition, or orchestration pattern, running the same inputs through the variant, applying the evals, and comparing the new scores with the baseline. Torres says this process may require multiple experiment variants before an improvement is found.

For customer-facing products, Torres describes a Python test harness that calls the LLM service for each input, logs outputs, transforms those outputs into the inputs required by each eval, runs the evals, and scores the full run. She also describes using Claude Code sub-agents and Python scripts to run evals for personal workflows.

The article ends its practical loop by instructing teams to collect a baseline, design and run an experiment, compare the results, and repeat the cycle until the desired outcome is reached.