Practice Observations
Observations of specific engineering use cases, encountered problems, and reported practices extracted from the evidence corpus.
18Evidence
18Assessed
0Pending
39Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for practice observations. Assessed evidence may legitimately contain zero observations.
18 practice observations · verification
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Cursor | Turn completed agent work into reusable improvements for future workflows. | Learnings from completed tasks disappear unless they are reviewed and encoded into durable workflow artifacts. | The Reflect skill runs three parallel reviewers, synthesizes durable learnings, checks whether they should be encoded structurally, and requires approval before skill edits. | learningverification |
| Cursor | Keep verification evidence valid as agent-authored pull requests change. | A verification verdict can become stale when the pull-request head changes. | pstack records verification verdicts against a pull-request number and head SHA and voids the verdict when the head SHA changes. | verification |
| Cursor | Coordinate multi-day programs containing many stacked pull requests and subagents. | Parallel work needs explicit ownership, bounded context transfer, durable state, and independent verification. | pstack defines coordinator, optional sub-coordinator, worker, and verifier roles; structured state files; queue events; and worker briefs. | contextcoordinationverification |
| Teresa Torres | Determine whether a change to an AI workflow actually improves its behavior. | Anecdotal testing cannot reliably distinguish an improvement from a regression. | Establish a baseline, change prompt/context/model/decomposition/orchestration, rerun the same inputs, apply evals, and compare scores. | learningverification |
| Teresa Torres | Turn failures observed in real AI usage into regression protection. | Production failures remain isolated incidents unless they become reusable evaluation criteria. | Review traces, log and categorize recurring mistakes, and turn those failure categories into eval candidates. | learningobservabilityverification |
| Teresa Torres | Run judgment-based evaluation without paying its cost for every case. | LLM-as-a-Judge evaluations are more expensive than deterministic assertions. | Use a cheap code assertion first and send only threshold-crossing cases to an LLM judge. | economicsverification |
| Infobip | Delegate implementation to fresh coding-agent sessions with sufficient task context. | Fresh sessions need goals, dependencies, relevant code paths, success criteria, and validation methods without relying on conversational history. | Research and planning produce validated artifacts, and task definitions carry goals, dependencies, code paths, acceptance criteria, and validation methods. | contextverification |
| Uber | Determine whether agent automation is economically efficient. | Token or request spend alone does not show how much useful engineering work was delivered. | Uber tracks outcome-denominated cost together with workload-specific quality signals and delivered volume. | economicsverification |
| Uber | Choose models for specific managed software-engineering agent workloads. | General model benchmarks do not establish the best quality, reliability, and economics for a particular production workload. | Uber builds benchmarks from each managed agent's real work and evaluates cost per completed task, output quality, and reliability. | economicsverification |
| Cursor | Review large volumes of concurrently agent-produced code. | A single review mechanism can become a verification bottleneck as swarm output grows. | Cursor uses stacked review agents with different review lenses. | coordinationverification |
| Spotify | Run many background coding-agent sessions with controlled execution and verification. | Autonomous sessions need an execution environment with trusted tools and a reliable quality gate. | Spotify runs Honk sessions in Kubernetes with trusted tools and CI verification. | executionverification |
| OpenAI | Run more concurrent Codex coding tasks than an engineer can comfortably supervise as separate sessions. | Engineers lost track of work and debugged stalled agents beyond roughly three to five simultaneous sessions. | Symphony moves task assignment and workspace lifecycle into an always-on orchestrator where open tasks receive agents in isolated workspaces and humans review outcomes. | coordinationexecutionobservabilityverification |
| Spotify | Determine whether an autonomously generated change is ready to become a pull request. | Agent completion does not establish that the generated code works or satisfies the task. | Spotify selects independent deterministic verifiers from repository contents and requires relevant verification before a pull request can be opened, then applies an LLM judge to the diff. | verification |
| Spotify | Automatically repair an agent-generated change that fails quality evaluation. | A verifier or judge veto would otherwise leave failed work for a human to recover. | Judge vetoes are fed back into the coding-agent loop, and Spotify reports that the agent course-corrects about half of vetoed cases. | learningverification |
| Bun | Use a coding agent to fix bugs in the Bun repository. | An agent-authored fix needs evidence that it resolves the bug rather than merely changing code. | Bun's Claude Code bot opens pull requests with tests that fail against the earlier system-installed Bun and pass against the fixed debug build, then responds to review comments. | executionverification |
| Spotify | Make autonomous background coding runs more predictable. | Additional dynamically fetched context and tools increase dimensions of unpredictability. | Spotify limits tools and hooks, constrains Git and Bash operations, and exposes standardized verification. | contextexecutionverification |
| Spotify | Improve the likelihood that a background coding task ends in a usable change. | Broad tasks without concrete success criteria are harder for agents to complete predictably. | Spotify uses tests, concrete examples, preconditions, and smaller task scope as verifiable end states. | verification |
| Spotify | Operate background coding agents predictably while agent and model components evolve. | Agent execution needs formatting, linting, diff evaluation, logs, traces, and the ability to change agent or model components without rebuilding the surrounding workflow. | Spotify built a pluggable CLI harness with MCP-based formatting and linting, LLM-as-judge diff evaluation, GCP logs, MLflow traces, and swappable agents and models. | executionobservabilityverification |