Practice Observations
Observations of specific engineering use cases, encountered problems, and reported practices extracted from the evidence corpus.
18Evidence
18Assessed
0Pending
39Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for practice observations. Assessed evidence may legitimately contain zero observations.
39 practice observations
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Cursor | Turn completed agent work into reusable improvements for future workflows. | Learnings from completed tasks disappear unless they are reviewed and encoded into durable workflow artifacts. | The Reflect skill runs three parallel reviewers, synthesizes durable learnings, checks whether they should be encoded structurally, and requires approval before skill edits. | learningverification |
| Cursor | Keep verification evidence valid as agent-authored pull requests change. | A verification verdict can become stale when the pull-request head changes. | pstack records verification verdicts against a pull-request number and head SHA and voids the verdict when the head SHA changes. | verification |
| Cursor | Coordinate multi-day programs containing many stacked pull requests and subagents. | Parallel work needs explicit ownership, bounded context transfer, durable state, and independent verification. | pstack defines coordinator, optional sub-coordinator, worker, and verifier roles; structured state files; queue events; and worker briefs. | contextcoordinationverification |
| Cursor | Choose whether a task should use multi-agent orchestration or a plain agent. | Coordination ceremony can cost more than it saves on work small enough for one agent. | The Orchestrate playbook says not to orchestrate when one agent can finish inside the task budget and uses a task-scale threshold for orchestration. | coordinationeconomics |
| Teresa Torres | Determine whether a change to an AI workflow actually improves its behavior. | Anecdotal testing cannot reliably distinguish an improvement from a regression. | Establish a baseline, change prompt/context/model/decomposition/orchestration, rerun the same inputs, apply evals, and compare scores. | learningverification |
| Teresa Torres | Turn failures observed in real AI usage into regression protection. | Production failures remain isolated incidents unless they become reusable evaluation criteria. | Review traces, log and categorize recurring mistakes, and turn those failure categories into eval candidates. | learningobservabilityverification |
| Teresa Torres | Run judgment-based evaluation without paying its cost for every case. | LLM-as-a-Judge evaluations are more expensive than deterministic assertions. | Use a cheap code assertion first and send only threshold-crossing cases to an LLM judge. | economicsverification |
| Infobip | Continue or reconstruct engineering work across agent sessions. | Important state disappears when it exists only in conversational history. | The workflow externalizes durable state into Git, task state, logs, and other artifacts. | contextlearning |
| Infobip | Carry a coding task from research through planning, task definition, and implementation. | One growing context across the workflow becomes noisy and allows upstream errors to compound into implementation. | Infobip structures work into four phases and repeatedly writes, selects, compresses, and isolates context between phases. | context |
| Infobip | Delegate implementation to fresh coding-agent sessions with sufficient task context. | Fresh sessions need goals, dependencies, relevant code paths, success criteria, and validation methods without relying on conversational history. | Research and planning produce validated artifacts, and task definitions carry goals, dependencies, code paths, acceptance criteria, and validation methods. | contextverification |
| Uber | Reduce the cost of high-volume agentic engineering work. | Using an expensive primary model for every bounded subtask makes scaled execution more costly. | A primary model decomposes and evaluates work while lower-cost subagents execute bounded tasks. | coordinationeconomics |
| Uber | Determine whether agent automation is economically efficient. | Token or request spend alone does not show how much useful engineering work was delivered. | Uber tracks outcome-denominated cost together with workload-specific quality signals and delivered volume. | economicsverification |
| Uber | Find expensive behavioral patterns in live agent sessions. | Repeated unnecessary tool use and other poor session behaviors can silently consume compute. | Uber uses live cost counters and a session-analysis dashboard that identifies 16 anti-pattern classes with financial impact and targeted remediation. | economicslearningobservability |
| Uber | Choose models for specific managed software-engineering agent workloads. | General model benchmarks do not establish the best quality, reliability, and economics for a particular production workload. | Uber builds benchmarks from each managed agent's real work and evaluates cost per completed task, output quality, and reliability. | economicsverification |
| Cursor | Integrate changes produced simultaneously by many agents. | Conventional merge behavior accumulated more than 70,000 conflicts in an earlier run. | Cursor uses neutral third-party agents for merge conflicts and a purpose-built version-control system at higher commit throughput. | coordinationexecution |
| Cursor | Coordinate hundreds of agents building a substantial software system concurrently. | Homogeneous agents make conflicting architectural decisions and spend context on both global planning and local implementation. | Cursor separates planner agents that decompose and delegate goals from worker agents that execute bounded pieces. | contextcoordination |
| Cursor | Keep independently produced implementation decisions coherent across a swarm. | Workers can make locally reasonable but globally incompatible design decisions. | Cursor uses reconciler agents to resolve conflicting design decisions. | coordination |
| Cursor | Review large volumes of concurrently agent-produced code. | A single review mechanism can become a verification bottleneck as swarm output grows. | Cursor uses stacked review agents with different review lenses. | coordinationverification |
| Spotify | Coordinate agent-generated migrations across thousands of repositories while directing scarce human attention. | At fleet scale, teams need to know which changes were created or merged and which require human intervention. | Fleetshift schedules and orchestrates Honk changes and tracks migration progress, pull-request state, and changes needing human attention. | coordinationobservability |
| Spotify | Run many background coding-agent sessions with controlled execution and verification. | Autonomous sessions need an execution environment with trusted tools and a reliable quality gate. | Spotify runs Honk sessions in Kubernetes with trusted tools and CI verification. | executionverification |
| Spotify | Give coding agents organizational context about software components before and during engineering work. | Agents need component ownership, documentation, and responsible-team information that is not necessarily present in source files. | Spotify exposes Backstage's software catalog to agents through MCPs and command-line tools. | context |
| Google Cloud | Run agent jobs that continue for hours or days and survive infrastructure or human interruptions. | A long-running agent cannot depend on one process or session remaining alive for the duration of the work. | Google Agent Executor uses durable event logs and snapshots to support execution, resumption, and distributed deployment. | executionobservability |
| OpenAI | Run more concurrent Codex coding tasks than an engineer can comfortably supervise as separate sessions. | Engineers lost track of work and debugged stalled agents beyond roughly three to five simultaneous sessions. | Symphony moves task assignment and workspace lifecycle into an always-on orchestrator where open tasks receive agents in isolated workspaces and humans review outcomes. | coordinationexecutionobservabilityverification |
| Cursor | Execute multiple parts of an engineering task concurrently. | Parallel agent work needs decomposition, isolation, and dependency-aware composition to avoid interfering changes. | Cursor Multitask decomposes work into asynchronous subagents, uses worktrees to isolate concurrent changes, and parallel plans to preserve task dependencies. | coordinationexecution |
| Spotify | Find concrete source locations that require changes during a fleet-wide migration. | Knowing the downstream consumer does not identify the actual repository usages that need modification. | Spotify uses Codesearch queries to locate migration targets across its GitHub Enterprise landscape. | context |
| Spotify | Identify repositories affected by a downstream dataset migration. | The migration system must know which repositories consume the changed dataset before an agent can modify them. | Spotify uses Backstage endpoint lineage to identify downstream consumers. | context |
| Spotify | Apply a dataset migration across thousands of downstream consumers. | Repository-scale changes need coordinated scheduling and pull-request execution across a large fleet. | Spotify combines Honk, Backstage, Codesearch, and Fleet Management to target and orchestrate the migration. | coordinationexecution |
| Gloaguen et al. | Provide coding agents repository-specific instructions, workflow guidance, and tool information. | Agents need repository-specific knowledge, but extra context can be redundant and increase exploration and inference cost without improving task resolution. | The evaluated CTXbench repositories use developer-committed repository context files that agents read and follow during coding tasks. | contexteconomics |
| Lulla et al. | Provide coding agents repository-specific guidance while they execute pull-request tasks. | Agents otherwise need to infer repository-specific instructions during execution, which can increase completion time and generated output. | The evaluated repositories provide a version-controlled root AGENTS.md file as repository-level context for the coding agent. | contexteconomics |
| Spotify | Run background coding agents with predictable behavior. | Greater execution flexibility increases unpredictable behavior. | Spotify constrains permissions, available binaries, and execution through sandboxing and surrounding infrastructure. | execution |
| Spotify | Determine whether an autonomously generated change is ready to become a pull request. | Agent completion does not establish that the generated code works or satisfies the task. | Spotify selects independent deterministic verifiers from repository contents and requires relevant verification before a pull request can be opened, then applies an LLM judge to the diff. | verification |
| Spotify | Automatically repair an agent-generated change that fails quality evaluation. | A verifier or judge veto would otherwise leave failed work for a human to recover. | Judge vetoes are fed back into the coding-agent loop, and Spotify reports that the agent course-corrects about half of vetoed cases. | learningverification |
| Bun | Use a coding agent to fix bugs in the Bun repository. | An agent-authored fix needs evidence that it resolves the bug rather than merely changing code. | Bun's Claude Code bot opens pull requests with tests that fail against the earlier system-installed Bun and pass against the fixed debug build, then responds to review comments. | executionverification |
| Spotify | Give different background coding agents instructions they can execute effectively. | A single prompt style did not work equally well across agent architectures. | Spotify tailored prompt structure to the agent, using strict step-by-step prompts for its homegrown agent and outcome-oriented prompts for Claude Code. | context |
| Spotify | Make autonomous background coding runs more predictable. | Additional dynamically fetched context and tools increase dimensions of unpredictability. | Spotify limits tools and hooks, constrains Git and Bash operations, and exposes standardized verification. | contextexecutionverification |
| Spotify | Improve the likelihood that a background coding task ends in a usable change. | Broad tasks without concrete success criteria are harder for agents to complete predictably. | Spotify uses tests, concrete examples, preconditions, and smaller task scope as verifiable end states. | verification |
| Spotify | Execute software migrations across a large repository fleet with a background coding agent. | Deterministic transformation declarations do not cover every migration, while repository targeting, pull-request lifecycle, review, and production merging still need coordinated infrastructure. | Spotify retained Fleet Management for repository targeting and pull-request lifecycle while replacing transformation declarations with natural-language instructions executed by a coding agent. | coordinationexecution |
| Spotify | Turn developer requests from Slack or GitHub into background coding work. | The background coding agent needs sufficient task information before it can independently produce a pull request. | An interactive agent gathers task information and hands a resulting prompt to the background coding agent. | contextcoordination |
| Spotify | Operate background coding agents predictably while agent and model components evolve. | Agent execution needs formatting, linting, diff evaluation, logs, traces, and the ability to change agent or model components without rebuilding the surrounding workflow. | Spotify built a pluggable CLI harness with MCP-based formatting and linting, LLM-as-judge diff evaluation, GCP logs, MLflow traces, and swappable agents and models. | executionobservabilityverification |