Evidence-backed practices
Emerging Practices
Evidence-backed engineering practices emerging across public engineering organizations, shown with the use case, problem encountered, reported response, and Selection conditions behind each observation.
47Evidence
47Assessed
0Pending
165Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for emerging practices. Assessed evidence may legitimately contain zero practice observations.
165 evidence-backed practice observations
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Anthropic | Add and optimize history, blame, search, and graph capabilities without changing clients or materializing every index eagerly. | Precomputing all repository metadata increases write amplification and couples new query capabilities to client formats. | Anthropic defines recursive derived-data types that can be computed eagerly or lazily and optimized behind a stable service interface. | economicsexecution |
| Anthropic | Serve ephemeral agents and CI jobs operating continuously against a very large monorepo. | Complete local repositories and offline-capable clients transfer and maintain data that online automated clients rarely need. | Anthropic moves history, diff, blame, commit, merge, and rebase operations to a server while clients retain bounded caches and fetch only requested data. | contexteconomicsexecution |
| Anthropic | Let CI, agents, and internal tools query repository state without checking out or maintaining a repository. | Every consumer independently cloning and maintaining repository data multiplies compute, storage, transfer, and operational work. | Scion exposes diff, ancestry, blame, merge-base, blob, and related operations through horizontally scalable APIs. | contexteconomicsexecution |
| Anthropic | Accept very high rates of concurrent changes without making every commit contend on global repository state. | Packfile processing, locking, and shared-reference movement place unrelated work on the same coordination-critical path. | Scion creates incremental server-side changesets without packfiles or locks and separates changeset creation from the small number of supported reference updates. | coordinationeconomicsexecution |
| Anthropic | Maintain human control when generated change volume makes uniform line-by-line review increasingly expensive. | Agents can produce locally plausible code that violates system architecture, uses brute-force access paths, or over-engineers hypothetical concerns. | Engineers review every change request with increasing emphasis on intent, architectural invariants, system-level tradeoffs, and rejecting invalid solution patterns. | coordinationverification |
| Anthropic | Integrate hundreds or thousands of changes while tests execute against a rapidly advancing main branch. | A globally locked merge queue serializes change and cannot keep pace when main advances substantially during each test run. | Anthropic invests in post-merge and pre-deployment automated testing, accepts concurrent integration, and uses agents to repair resulting conflicts. | coordinationexecutionverification |
| Anthropic | Observe subtle operational behavior across independently scalable source-control components. | Fixed alerts can miss novel or weak signals in component telemetry and logs. | Anthropic assigns persistent superfan agents to individual components to inspect telemetry and logs alongside conventional alerting. | coordinationobservability |
| Atlassian | Expand an agent tool surface without allowing tool and context metadata to consume the task budget. | Adding products and tools can increase prompt overhead before an agent begins the requested work. | Atlassian benchmarks matched Jira and Confluence work across MCP versions and redesigns context delivery to reduce token use while expanding the available tool surface. | contexteconomicsobservability |
| Atlassian | Operate agent access across many enterprise products and external AI clients. | Distributed integrations make it difficult to enforce data policy, constrain connectivity, and reconstruct agent actions. | Atlassian centralizes domain, permission-scope, IP, data-protection, and data-loss-prevention controls with organization-wide audit logging. | coordinationobservabilityverification |
| Atlassian | Allow agents and teammates to edit shared organizational documents concurrently. | An agent can overwrite a teammate's newer change when it acts on stale document state. | Atlassian uses version-aware, section-level edits that pause on concurrent changes and expose a preview before publication. | coordinationobservabilityverification |
| Atlassian | Let external AI clients act on enterprise work without gaining authority beyond the invoking user. | A shared agent integration can become an authority-escalation path if tool access bypasses existing identity and permissions. | Atlassian authenticates with OAuth 2.1 and PKCE and evaluates every request against the invoking user's existing permissions. | executionverification |
| Atlassian | Govern a broad MCP surface containing read, write, and destructive operations. | Treating all tool calls alike prevents clients and administrators from applying stronger controls to higher-impact actions. | Atlassian separates read, write, and destructive actions, scopes access by toolset, and enables explicit confirmation for higher-risk operations. | coordinationverification |
| GitHub | Scale repository reads and writes independently under bursty agent and CI workloads. | Using complete local repository copies for both durability and serving means that adding read replicas also adds overhead to every write. | GitHub separates authoritative repository data in durable object storage from elastic compute workers that cache and serve requests. | economicsexecution |
| GitHub | Accept frequent concurrent pushes from developers, automation, and agent fleets. | Coordinating every part of a push across durable replicas makes write latency and throughput depend on work that does not require global agreement. | GitHub narrows synchronous coordination to the reference update while allowing object storage, connectivity validation, and secret scanning to proceed independently or in parallel. | coordinationexecution |
| GitHub | Increase automated repository throughput while retaining organizational control. | Higher agent activity can outrun governance if throughput replaces review, protection, auditability, or approval mechanisms. | GitHub treats branch protection, required review, audit logs, repository visibility, and human approval as constraints the new architecture must preserve. | coordinationobservabilityverification |
| GitHub | Maintain heavily written repositories without slowing live pushes and fetches. | Compaction and garbage collection compete with request handling when they execute on the same hosts. | GitHub moves compaction and garbage collection to separate workers operating directly against durable storage. | executionobservability |
| Astekin et al. | Decide whether an engineering task needs non-agentic, single-agent, or multi-agent execution. | Selecting a complex agent architecture without a lightweight baseline can hide whether cooperation contributes any task-specific benefit. | Evaluate an explicit ladder from direct model query through single-, dual-, and multi-agent configurations under matched task conditions. | coordinationeconomicsverification |
| Astekin et al. | Introduce multi-agent cooperation for a bounded software-engineering workflow. | Additional agents impose systematic execution costs while accuracy gains vary by task. | Begin with a lightweight task-aware baseline and adopt multi-agent workflows only when comparative evaluation demonstrates a benefit that justifies the added cost. | coordinationeconomicsverification |
| Astekin et al. | Deploy heavier agent workflows whose latency and energy depend on the execution platform. | A configuration that appears viable on high-end server hardware may be too slow or inefficient on a developer workstation. | Measure the same agent configurations across representative hardware and include runtime environment in the selection decision. | economicsexecution |
| Astekin et al. | Choose an agent configuration when outcome quality, response time, and resource use all matter. | Ranking only by task accuracy can select a configuration whose small benefit carries disproportionate latency and energy costs. | Compare configurations on a joint accuracy, latency, and energy Pareto front rather than optimizing one metric in isolation. | economicsobservabilityverification |
| Astekin et al. | Select models and prompting strategies across different engineering task families. | One model or prompt strategy does not remain optimal across code, vulnerability, debt, and log-analysis work. | Treat model and prompt choice as task-specific experimental variables rather than global defaults. | executionverification |
| Cursor | Distinguish a production regression from expected effects after a deployment. | Operators must connect a changed signal to one of several deployed changes and decide whether the movement is intentional. | Follow the change through deployment and compare its planned signals with a pre-deploy baseline to identify localized or unintended regressions. | contextobservabilityverification |
| Cursor | Detect exploit paths introduced by a pull-request change. | Local pattern matching can flag superficial risks while missing vulnerabilities that depend on data flow, authorization, dependencies, or broader codebase context. | Analyze every pull request against whole-codebase context and return severity, attack path, explanation, and a proposed fix for each finding. | contextverification |
| Cursor | Define how a particular code change will be verified after deployment. | Generic dashboards and alerts may not cover the risks, intended effects, or instrumentation gaps introduced by a specific diff. | Read the pull-request diff before merge, draft an editable plan of expected effects and risk signals, and flag where telemetry cannot determine success. | contextobservabilityverification |
| Cursor | Respond to a suspected deployment regression with authority appropriate to organizational risk. | A single remediation policy is either too passive for low-risk cases or too autonomous for consequential production changes. | Configure the rollout bot to escalate to the author, pause progressive delivery, or prepare a revert pull request for human approval. | coordinationexecutionverification |
| Cursor | Preserve source-location guidance while reducing context added by large file reads. | Numbering every source line adds substantial repeated context when agents read tens of thousands of lines. | Include line numbers at a sparser interval that remains sufficient for model citations, then verify quality before rollout. | contexteconomics |
| Cursor | Delegate work when fresh subagent context saves more than coordination costs. | Isolated subagents can reduce inherited context but may duplicate work or pursue tasks invalidated by the parent's progress. | Avoid blanket instructions to delegate exploration and constrain alternate-model selection to cases directed by the user or harness. | contextcoordinationeconomics |
| Cursor | Keep a broad tool set available without placing every definition in every model request. | Most specialized tools are used in a minority of conversations, but static definitions consume context throughout every turn. | Keep high-frequency and flow-critical tools in static context and load lower-frequency MCP and built-in tool definitions only when needed. | contexteconomicsexecution |
| Cursor | Optimize context construction for the real distribution of agent tasks. | Offline evals can overrepresent hard tasks and miss the token, latency, and behavior effects seen in ordinary production traffic. | A/B test context configurations on production traffic while tracking token use, cost, latency, tool-call errors, agent usage, and quality. | contexteconomics |
| Cursor | Avoid repeatedly sending harness instructions that current models no longer need. | Legacy behavioral guidance remains in every turn after models learn to follow simpler tool definitions. | Re-test system instructions across model families and remove directions whose absence does not degrade production behavior. | contexteconomics |
| Cursor | Reuse unchanged prompt prefixes as an agent conversation grows. | Variable setup mixed with stable instructions and tools prevents effective prefix-cache reuse across turns. | Place explicit cache breakpoints after stable request layers and move user- and request-specific setup behind those boundaries. | contexteconomics |
| Will Larson | Prevent an agent from executing project work without a defined direction or observable outcome. | A task backlog can appear actionable even when the project lacks an explicit goal or a way to measure progress. | Start each project loop by auditing the goal definition and measurement tools, and iterate with the operator to create missing prerequisites. | contextobservability |
| Will Larson | Give an agent enough project state to reason beyond one assigned task. | Goals, measures, work state, and implementation context held in private human knowledge cannot regulate an automated project loop. | Represent goals and approach in a Notion RFC, measures in Datadog or Snowflake, and shared work state in a Linear project that the agent can audit. | contextcoordinationobservability |
| Will Larson | Advance a broad engineering goal across a changing set of tasks. | A task-scoped agent cannot identify missing work, reconcile changed issue state, or choose the next non-blocked action. | Review metrics and issues, create newly identified tasks, update state, execute non-blocked work, and select the next action from current project state. | coordinationexecutionobservability |
| Will Larson | Keep agent execution aligned when project understanding or operating results change over time. | Continuing to drain an old task list can optimize against a stale description after the project or its environment has changed. | Continue to the next task only when the project description is fresh; otherwise restart the loop from goal and measurement assessment. | contextlearningobservability |
| Will Larson | Implement one project across frontend, backend, infrastructure, and data repositories. | Repository-scoped checkouts make coordinated cross-repository agent changes difficult to manage. | Maintain about ten independent local workspaces, each with a checkout of every repository, and operate agents at workspace rather than repository scope. | contextcoordinationexecution |
| Atlassian | Retrieve several related components, tokens, or guidance records during one agent task. | Agents made repeated context round trips, accumulating avoidable lookup latency. | Add a batch command and improve fuzzy search so agents can consolidate related context queries. | contexteconomicsexecution |
| Atlassian | Choose how a coding agent should retrieve structured engineering context. | Equivalent knowledge can produce different end-to-end outcomes when invocation, instructions, and output formats differ. | Hold the agent, model, tasks, and questions constant; vary only the context-delivery path; then record pass rate, task time, tokens, tool calls, and transcripts. | contexteconomicsobservabilityverification |
| Atlassian | Serve context repeatedly during an agent task without paying process or package startup costs on every lookup. | An npx-based command added more than 15 seconds per query and compounded when agents performed multiple lookups. | Provide a pre-installed CLI path for the default environment while retaining the portable npx invocation as a fallback. | economicsexecution |
| Atlassian | Expose one governed knowledge source to agents with different tool access. | Some agent environments provide a shell but no MCP client, while some sandboxes cannot access a terminal. | Generate multiple interfaces from shared handlers and data, prefer the benchmarked CLI path, and fall back to MCP or bundled references when required by the environment. | contextexecution |
| Cursor | Continue a migration across hundreds of pull requests without requiring equally intensive review of every change. | Human review becomes the throughput constraint when every repetitive agent-produced change receives the same level of scrutiny. | Establish a safe approach, review early pull requests closely, and reduce review intensity as repeated fixes hold up. | learningverification |
| Cursor | Make discoveries from one agent execution available to later agents working on the same body of work. | Session-local agents repeatedly rediscover codebase procedures, architectural context, and user preferences. | Synchronize project files containing research, artifacts, codebase knowledge, and preferences across every cloud and local machine used by the Project. | contextlearning |
| Cursor | Act on new pull requests, CI failures, bug reports, schedules, or Slack signals without waiting for a new prompt. | Prompt-driven agents stop observing the environment after an individual task completes. | Subscribe the coordinator to schedules and external events so it can initiate recurring maintenance and follow-up work. | executionobservability |
| Cursor | Keep long-running work active while still executing tests or operations that require a developer's machine. | Laptop-bound execution limits duration and parallelism, while some tasks still require local state or hardware. | Run the Project persistently on a cloud computer and let its coordinator start a local agent when machine-specific testing is needed. | coordinationexecution |
| Cursor | Direct a feature, migration, or recurring maintenance program that outlives one agent session. | A developer otherwise has to repeatedly plan work, start agents, relay context, and integrate results across a long-running body of work. | Create a persistent Project whose coordinator plans and delegates while separate worker agents implement and test bounded tasks. | coordinationexecution |
| OpenAI | Keep agents operating reliably across sessions that span hours, days, and multiple context windows. | Long-running work requires context continuity, recoverable execution, files, code execution, and preserved intermediate results. | Use a managed, versioned agent harness with automatic context compaction and a durable execution environment. | contextexecution |
| OpenAI | Parallelize complex work across independent subproblems without implementing a custom coordination layer. | Applications otherwise need to build and observe delegation, separate subagent contexts, parallel execution, and result aggregation themselves. | Delegate independent work to parallel subagents with separate contexts while a main agent coordinates and combines their results. | coordinationexecutionobservability |
| OpenAI | Let an agent use a large tool environment without loading every tool definition or raw result into context. | Large tool catalogs and result sets consume tokens and dilute the context needed for the task. | Load relevant tools on demand, run related calls programmatically, and return only relevant combined results to model context. | contextexecution |
| Cursor | Turn completed agent work into reusable improvements for future workflows. | Learnings from completed tasks disappear unless they are reviewed and encoded into durable workflow artifacts. | The Reflect skill runs three parallel reviewers, synthesizes durable learnings, checks whether they should be encoded structurally, and requires approval before skill edits. | learningverification |
| Cursor | Keep verification evidence valid as agent-authored pull requests change. | A verification verdict can become stale when the pull-request head changes. | pstack records verification verdicts against a pull-request number and head SHA and voids the verdict when the head SHA changes. | verification |
| Cursor | Coordinate multi-day programs containing many stacked pull requests and subagents. | Parallel work needs explicit ownership, bounded context transfer, durable state, and independent verification. | pstack defines coordinator, optional sub-coordinator, worker, and verifier roles; structured state files; queue events; and worker briefs. | contextcoordinationverification |
| Cursor | Choose whether a task should use multi-agent orchestration or a plain agent. | Coordination ceremony can cost more than it saves on work small enough for one agent. | The Orchestrate playbook says not to orchestrate when one agent can finish inside the task budget and uses a task-scale threshold for orchestration. | coordinationeconomics |
| Meta | Convert expert corrections into durable improvements to agent knowledge and reasoning. | Manually tracing dependencies, editing interdependent files, and proving that a correction does not break existing behavior can take weeks. | Diagnose feedback, compile minimal edits with parallel impact analysis, perform independent review and structural checks, then submit a validated pull request for expert approval. | coordinationlearningverification |
| Meta | Execute expert reasoning procedures without loading every instruction and source into each agent turn. | Flat instructions and broad retrieval introduce mixed-relevance context that consumes tokens and degrades attention. | Separate knowledge from composable reasoning recipes and load only the instructions and files required for each analytical phase. | context |
| Meta | Verify that a proposed knowledge or reasoning change fixes the reported failure without losing existing behavior. | An automatically drafted correction can solve one case while introducing regressions elsewhere in the institutional knowledge system. | Run blind targeted replay and domain regression tests, retry failed compilations, and add every landed failure case to the permanent regression suite. | learningverification |
| Meta | Give an agent durable organizational positions, terminology, and routing rules without encoding them in model weights. | Retrieving raw documents forces the agent to re-derive institutional reasoning on every run, producing slow and inconsistent analysis. | Distill frequently used organizational knowledge into versioned files with explicit boundaries, dependencies, consumers, and deterministic routing. | contextlearning |
| Teresa Torres | Determine whether a change to an AI workflow actually improves its behavior. | Anecdotal testing cannot reliably distinguish an improvement from a regression. | Establish a baseline, change prompt/context/model/decomposition/orchestration, rerun the same inputs, apply evals, and compare scores. | learningverification |
| Teresa Torres | Turn failures observed in real AI usage into regression protection. | Production failures remain isolated incidents unless they become reusable evaluation criteria. | Review traces, log and categorize recurring mistakes, and turn those failure categories into eval candidates. | learningobservabilityverification |
| Teresa Torres | Run judgment-based evaluation without paying its cost for every case. | LLM-as-a-Judge evaluations are more expensive than deterministic assertions. | Use a cheap code assertion first and send only threshold-crossing cases to an LLM judge. | economicsverification |
| Infobip | Continue or reconstruct engineering work across agent sessions. | Important state disappears when it exists only in conversational history. | The workflow externalizes durable state into Git, task state, logs, and other artifacts. | contextlearning |
| Infobip | Carry a coding task from research through planning, task definition, and implementation. | One growing context across the workflow becomes noisy and allows upstream errors to compound into implementation. | Infobip structures work into four phases and repeatedly writes, selects, compresses, and isolates context between phases. | context |
| Infobip | Delegate implementation to fresh coding-agent sessions with sufficient task context. | Fresh sessions need goals, dependencies, relevant code paths, success criteria, and validation methods without relying on conversational history. | Research and planning produce validated artifacts, and task definitions carry goals, dependencies, code paths, acceptance criteria, and validation methods. | contextverification |
| Uber | Reduce the cost of high-volume agentic engineering work. | Using an expensive primary model for every bounded subtask makes scaled execution more costly. | A primary model decomposes and evaluates work while lower-cost subagents execute bounded tasks. | coordinationeconomics |
| Uber | Determine whether agent automation is economically efficient. | Token or request spend alone does not show how much useful engineering work was delivered. | Uber tracks outcome-denominated cost together with workload-specific quality signals and delivered volume. | economicsverification |
| Uber | Find expensive behavioral patterns in live agent sessions. | Repeated unnecessary tool use and other poor session behaviors can silently consume compute. | Uber uses live cost counters and a session-analysis dashboard that identifies 16 anti-pattern classes with financial impact and targeted remediation. | economicslearningobservability |
| Uber | Choose models for specific managed software-engineering agent workloads. | General model benchmarks do not establish the best quality, reliability, and economics for a particular production workload. | Uber builds benchmarks from each managed agent's real work and evaluates cost per completed task, output quality, and reliability. | economicsverification |
| Destefanis and Aste | Decompose a software change across agents with dependent outputs. | Each contribution can be locally correct while a convention spanning two owners remains unresolved and causes the integrated system to fail. | Identify cross-agent interfaces explicitly and assign one agent responsibility for validating each end-to-end contract. | contextcoordinationverification |
| Destefanis and Aste | Understand how a multi-agent coding run coordinates beneath its final output. | Pass rates and aggregate tokens hide which agents communicated, which files carried shared state, when coordination happened, and where it failed. | Represent agents and files as nodes and log messages, reads, and writes as timestamped, costed edges linked to the run's verified outcome. | coordinationobservabilityverification |
| Destefanis and Aste | Choose between direct messaging and persistent shared artifacts for an agent team. | One-to-one messages repeat shared information, while mandatory files add overhead when the work already communicates through file interfaces. | Measure the task's emergent coordination network and require shared files only where they replace message-heavy many-to-many traffic. | contextcoordinationeconomics |
| Destefanis and Aste | Evaluate multi-agent configurations whose tasks leave teams freedom to self-organize. | Identical configurations under a pinned model can produce substantially different coordination networks across sessions. | Run each configuration repeatedly, compare matched sessions, and report distributions or cross-session ranges rather than treating one run as representative. | observabilityverification |
| Destefanis and Aste | Evaluate coding agents against hidden tests or reference solutions. | Agents may search outside their intended workspace for grading material, contaminating scores and revealing weak containment. | Remove access to hidden evaluation artifacts, substitute detectable placeholders, and repeat key conditions under sealed execution. | executionobservabilityverification |
| Destefanis and Aste | Introduce a coordinator or orchestrator into a multi-agent workflow. | Naming an agent as coordinator in its prompt may create no hub, authority, routing behavior, or reliable outcome improvement. | Inspect the interaction graph and comparative outcomes to verify that the proposed coordinator actually performs a structural integration role. | coordinationobservabilityverification |
| Cloudflare | Let sites identify and selectively admit legitimate agents. | Sites cannot reliably make policy decisions when machine actors are indistinguishable, spoofed, or treated as generic bots. | Cloudflare presents Web Bot Auth and PACT as cryptographic identity and delegated trust mechanisms for agents. | coordinationexecution |
| Cloudflare | Help agents find resources and help resource owners understand whether agents can discover them. | Human-oriented search and visibility mechanisms do not directly expose what agents can find or how visible a resource is to them. | Cloudflare presents AI Search for agent-oriented retrieval and AEO for measuring visibility to models and agents. | contextobservability |
| Cloudflare | Let agents compensate resource owners directly for content, APIs, or excess resource consumption. | Advertising and seat-based business models do not map cleanly to software actors that consume resources without rendering human interfaces. | Cloudflare presents x402-compatible payments, agent wallets, monetization gateways, and receipts as transaction rails for agent interactions. | coordinationeconomicsobservability |
| Cloudflare | Let agents consume web content without paying to parse human-oriented presentation markup. | HTML intended for humans consumes bandwidth, tokens, and context-window capacity that does not contribute to the agent's task. | Cloudflare presents Markdown for Agents as a server-side representation optimized for machine consumption. | contexteconomics |
| Cloudflare | Let agents perform actions on websites without reverse-engineering the DOM. | DOM parsing, guessed form fields, and synthesized clicks are brittle interfaces for machine actors. | WebMCP lets sites register explicit tools and schemas that agents can invoke using the user's existing session and state. | coordinationexecutionobservability |
| Cursor | Integrate changes produced simultaneously by many agents. | Conventional merge behavior accumulated more than 70,000 conflicts in an earlier run. | Cursor uses neutral third-party agents for merge conflicts and a purpose-built version-control system at higher commit throughput. | coordinationexecution |
| Cursor | Coordinate hundreds of agents building a substantial software system concurrently. | Homogeneous agents make conflicting architectural decisions and spend context on both global planning and local implementation. | Cursor separates planner agents that decompose and delegate goals from worker agents that execute bounded pieces. | contextcoordination |
| Cursor | Keep independently produced implementation decisions coherent across a swarm. | Workers can make locally reasonable but globally incompatible design decisions. | Cursor uses reconciler agents to resolve conflicting design decisions. | coordination |
| Cursor | Review large volumes of concurrently agent-produced code. | A single review mechanism can become a verification bottleneck as swarm output grows. | Cursor uses stacked review agents with different review lenses. | coordinationverification |
| Stripe | Allow coding agents to provision and configure infrastructure without exposing production by default. | An agent that goes off-script can create direct production impact if credentials and environment access are too broad. | Use named environments with isolated credentials and default agents to the development environment rather than production. | coordinationexecution |
| Stripe | Move from code generation to runnable software without requiring a human to configure each provider dashboard. | Provisioning infrastructure, connecting services, and managing credentials remain adjacent bottlenecks even when an agent can write the application code. | Expose infrastructure provisioning through a structured agent skill or embedded CLI and connect it to multiple providers through delegated, scoped authority. | contextcoordinationexecution |
| Stripe | Continue multi-day or multi-week agent-built software projects without rediscovering project state on every session. | Session-local agents lose project context and repeatedly reacquire state before they can act. | Carry project context across sessions so the agent can return to the same project with persistent state. | context |
| Stripe | Let agents consume infrastructure while constraining runaway resource cost. | An autonomous workflow can overspend on a single model, database, hosting, or other provider. | Set service-specific spending limits and expose unified current and historical cost across providers. | economicsobservability |
| Spotify | Coordinate agent-generated migrations across thousands of repositories while directing scarce human attention. | At fleet scale, teams need to know which changes were created or merged and which require human intervention. | Fleetshift schedules and orchestrates Honk changes and tracks migration progress, pull-request state, and changes needing human attention. | coordinationobservability |
| Spotify | Run many background coding-agent sessions with controlled execution and verification. | Autonomous sessions need an execution environment with trusted tools and a reliable quality gate. | Spotify runs Honk sessions in Kubernetes with trusted tools and CI verification. | executionverification |
| Spotify | Give coding agents organizational context about software components before and during engineering work. | Agents need component ownership, documentation, and responsible-team information that is not necessarily present in source files. | Spotify exposes Backstage's software catalog to agents through MCPs and command-line tools. | context |
| Google Cloud | Run agent jobs that continue for hours or days and survive infrastructure or human interruptions. | A long-running agent cannot depend on one process or session remaining alive for the duration of the work. | Google Agent Executor uses durable event logs and snapshots to support execution, resumption, and distributed deployment. | executionobservability |
| OpenAI | Run more concurrent Codex coding tasks than an engineer can comfortably supervise as separate sessions. | Engineers lost track of work and debugged stalled agents beyond roughly three to five simultaneous sessions. | Symphony moves task assignment and workspace lifecycle into an always-on orchestrator where open tasks receive agents in isolated workspaces and humans review outcomes. | coordinationexecutionobservabilityverification |
| Cursor | Execute multiple parts of an engineering task concurrently. | Parallel agent work needs decomposition, isolation, and dependency-aware composition to avoid interfering changes. | Cursor Multitask decomposes work into asynchronous subagents, uses worktrees to isolate concurrent changes, and parallel plans to preserve task dependencies. | coordinationexecution |
| Spotify | Find concrete source locations that require changes during a fleet-wide migration. | Knowing the downstream consumer does not identify the actual repository usages that need modification. | Spotify uses Codesearch queries to locate migration targets across its GitHub Enterprise landscape. | context |
| Spotify | Identify repositories affected by a downstream dataset migration. | The migration system must know which repositories consume the changed dataset before an agent can modify them. | Spotify uses Backstage endpoint lineage to identify downstream consumers. | context |
| Spotify | Apply a dataset migration across thousands of downstream consumers. | Repository-scale changes need coordinated scheduling and pull-request execution across a large fleet. | Spotify combines Honk, Backstage, Codesearch, and Fleet Management to target and orchestrate the migration. | coordinationexecution |
| Cloudflare | Provide many engineering clients and agents with governed access to models and internal capabilities. | Direct client-to-model integrations make attribution, permissions, model changes and cost control difficult to manage consistently at organizational scale. | Route AI clients through a shared proxy Worker and AI Gateway so identity, permissions, model catalog configuration, usage attribution and routing can be changed centrally. | coordinationeconomicsexecutionobservability |
| Cloudflare | Make organizational engineering standards directly usable by coding and review agents. | Standards remain weak selection pressure when they live only in human-readable documents disconnected from agent execution and review. | Distill engineering standards into structured Codex rules and an agent skill, then cite those rules during automated review and use them in multi-agent compliance audits. | contextlearningverification |
| Cloudflare | Give coding agents repository-specific operational context across thousands of repositories. | Agents otherwise infer ownership, architecture, build commands, testing conventions and standards from incomplete repository exploration. | Generate AGENTS.md from Backstage metadata, repository structure and Engineering Codex rules, open it as a merge request for owner review, and flag it for refresh when repository changes make it stale. | contextlearningverification |
| Cloudflare | Increase the chance that agent-generated changes already fit repository and organizational constraints before human review. | Agent outputs remain plausible but locally wrong when context acquisition, execution and standards enforcement are disconnected. | Wire Backstage context, repository AGENTS.md, centralized tools and Codex-backed AI review into one engineering path; Cloudflare reports that the resulting first draft is now usually close enough to ship. | contextcoordinationeconomicsexecutionlearningobservabilityverification |
| Cloudflare | Review every merge request on a standard CI path without applying the same review cost and expertise to every change. | Uniform generic review does not match the risk profile or expertise required by different engineering changes. | Classify merge requests by risk tier and delegate review to specialized agents for code quality, security, standards compliance, documentation, performance and release impact. | coordinationeconomicsverification |
| Cloudflare | Control inference cost while retaining stronger reasoning for high-risk engineering work. | Using the most capable model for every review task makes high-volume agentic workflows unnecessarily expensive. | Route documentation and lower-risk review workloads to lower-cost Workers AI models while reserving frontier models for security-sensitive and architecturally complex reviews. | economicsexecution |
| Cloudflare | Evaluate whether documentation restructuring changes agent cost and task speed. | Standards conformance alone does not demonstrate that the environment actually improves agent efficiency. | Run the same agent architecture against multiple technical documentation sites and compare token use and time to the correct answer. | contexteconomicsobservability |
| Cloudflare | Reduce the exploration required for agents to find precise technical documentation. | Large or HTML-heavy documentation forces iterative search, fragments context, and increases token and latency cost. | Restructure documentation into bounded agent-readable Markdown paths and context-sized product directories. | contexteconomics |
| Cloudflare | Assess whether a web environment exposes the information and capabilities agents need to operate efficiently. | Agent compatibility remains difficult to improve when discoverability, content access, permissions, and callable capabilities are implicit or unmeasured. | Score sites across Discoverability, Content, Bot Access Control, and Capabilities using machine-checkable standards. | contextexecutionobservability |
| Cloudflare | Turn an environmental readiness gap into an actionable implementation task. | A readiness assessment has limited value if teams must independently determine how to repair every failing condition. | For each failed readiness check, generate a prompt that can be given to a coding agent to implement the missing support. | executionlearning |
| Cloudflare | Measure how widely agent-friendly environmental traits are spreading across the Web. | Individual site assessments do not show whether agent-compatible traits are becoming common across the wider environment. | Scan 200,000 high-traffic domains and publish standards-adoption data in Cloudflare Radar with weekly updates and API access. | learningobservability |
| Cloudflare | Keep long-running agent work recoverable without maintaining an always-on process for every agent. | Deploys, restarts, and idle periods can destroy in-memory work or make one-runtime-per-agent economics impractical. | Give each agent addressable persistent state and run checkpointable work that can recover after the runtime restarts and hibernate when idle. | economicsexecution |
| Cloudflare | Delegate a bounded task without sharing the parent's entire context and mutable state. | Subagents need independent context and tools while the parent still needs a defined mechanism for delegation and result collection. | Create isolated child agents with their own state, model, memory, and tools, connected to the parent through typed streaming RPC. | contextcoordinationexecution |
| Cloudflare | Run agent-authored code with only the capabilities required by the current task. | A general-purpose execution environment gives generated code unnecessary ambient authority, while one fixed sandbox is inefficient for every task. | Escalate through workspace, isolated code, package, browser, and full-sandbox tiers while granting network and resource capabilities explicitly. | economicsexecution |
| Cloudflare | Continue agent work across long conversations, interruptions, and session branches. | Long-running work exceeds model context windows and loses continuity when conversation state exists only in the active process. | Persist full tree-structured session history, compact context non-destructively, and expose searchable history plus updateable context blocks. | contextobservability |
| Meta | Keep repository context useful as code and dependencies change. | Stale context can misdirect coding agents and may be more harmful than missing context. | Periodically validate paths, detect coverage gaps, rerun quality critics, and automatically repair stale references. | contextlearningobservability |
| Meta | Help coding agents locate the right files, commands, conventions, and dependencies before editing a proprietary codebase. | Without repository-specific tribal knowledge, agents repeatedly explore the codebase and can make plausible but subtly wrong changes. | Generate compact context files containing quick commands, key files, non-obvious patterns, and cross-references instead of exhaustive repository summaries. | contexteconomics |
| Meta | Answer cross-repository impact questions without repeated exploratory searches. | A change in one subsystem can propagate across repositories, making dependency discovery expensive and error-prone. | Pre-compute dependency indexes and data-flow maps so agents can retrieve affected relationships directly. | contexteconomics |
| Meta | Prevent generated repository guidance from becoming a new source of agent error. | AI-generated context can be incorrect, stale, overly broad, or contain invalid references. | Run multiple rounds of independent critics, prompt tests, integration checks, and file-path validation before relying on generated context. | observabilityverification |
| Meta | Create structured AI-readable repository knowledge across a large multi-repository system. | Manually documenting thousands of files and hidden engineering conventions does not scale. | Orchestrate specialized explorer, analyst, writer, critic, fixer, tester, and gap-filling agent roles to pre-compute repository context. | contextcoordination |
| Stripe | Determine whether an agent-generated integration actually works after the agent declares completion. | An agent can mistake a plausible intermediate result or expected-looking error for task success. | Use deterministic API and automated UI graders, including inspection of created service-side objects, to validate the final system independently of the agent's own completion judgment. | observabilityverification |
| Stripe | Measure whether coding agents can complete end-to-end API integration work rather than isolated code-generation tasks. | Function-level benchmarks can miss the planning, state management, browser use, database work, and glue code required by real integrations. | Evaluate agents in complete coding environments with repositories, databases, scripts, test credentials, browsers, and real API interactions. | contextexecutionverification |
| Stripe | Help coding agents verify API migrations and integrations against meaningful runtime behavior. | Testing with nonexistent or unrealistic data can produce technically valid errors that an agent misreads as proof that the implementation works. | Provide realistic test credentials and state, and grade behavior against valid end-to-end outcomes rather than accepting error responses as verification. | contextverification |
| Stripe | Improve an agent workflow after observing recurring execution or recovery failures. | Without replayable tasks, changes to prompts, skills, tools, or documentation cannot be compared against the same engineering problem. | Use replayable benchmark environments as a test bed for prompt, skill, browser-tool, and documentation interventions, then measure whether task performance changes. | learningobservabilityverification |
| Gloaguen et al. | Provide coding agents repository-specific instructions, workflow guidance, and tool information. | Agents need repository-specific knowledge, but extra context can be redundant and increase exploration and inference cost without improving task resolution. | The evaluated CTXbench repositories use developer-committed repository context files that agents read and follow during coding tasks. | contexteconomics |
| Lulla et al. | Provide coding agents repository-specific guidance while they execute pull-request tasks. | Agents otherwise need to infer repository-specific instructions during execution, which can increase completion time and generated output. | The evaluated repositories provide a version-controlled root AGENTS.md file as repository-level context for the coding agent. | contexteconomics |
| Dae-Kyoo Kim | Assess whether an agent configuration generalizes beyond one favorable engineering task. | A configuration can appear effective on one task or domain while performing poorly elsewhere. | Evaluate specialization strategies across multiple lifecycle tasks and application domains using structural and semantic artifact-validity criteria. | learningverification |
| Dae-Kyoo Kim | Evaluate whether a new specialization strategy improves an agentic engineering workflow. | Changing the model and orchestration at the same time makes it difficult to attribute outcome differences to agent configuration. | Hold the underlying LLM constant while comparing orchestration granularities so configuration effects can be evaluated more directly. | observabilityverification |
| Dae-Kyoo Kim | Design agent boundaries for tasks that depend on information accumulated across a development phase. | Overly narrow task boundaries can discard useful continuity between related activities. | Use phase-level specialization where contextual continuity across related tasks improves artifact validity, such as operation identification or test design in the study. | contextcoordination |
| Dae-Kyoo Kim | Choose how narrowly to specialize agents across a software-development workflow. | Increasing role specialization does not consistently improve artifact validity across all engineering tasks. | Compare task-specialized, phase-specialized, and process-generalist configurations and choose specialization granularity according to the task rather than assuming narrower roles are universally better. | coordinationlearningverification |
| Spotify | Run background coding agents with predictable behavior. | Greater execution flexibility increases unpredictable behavior. | Spotify constrains permissions, available binaries, and execution through sandboxing and surrounding infrastructure. | execution |
| Spotify | Determine whether an autonomously generated change is ready to become a pull request. | Agent completion does not establish that the generated code works or satisfies the task. | Spotify selects independent deterministic verifiers from repository contents and requires relevant verification before a pull request can be opened, then applies an LLM judge to the diff. | verification |
| Spotify | Automatically repair an agent-generated change that fails quality evaluation. | A verifier or judge veto would otherwise leave failed work for a human to recover. | Judge vetoes are fed back into the coding-agent loop, and Spotify reports that the agent course-corrects about half of vetoed cases. | learningverification |
| Bun | Use a coding agent to fix bugs in the Bun repository. | An agent-authored fix needs evidence that it resolves the bug rather than merely changing code. | Bun's Claude Code bot opens pull requests with tests that fail against the earlier system-installed Bun and pass against the fixed debug build, then responds to review comments. | executionverification |
| Spotify | Give different background coding agents instructions they can execute effectively. | A single prompt style did not work equally well across agent architectures. | Spotify tailored prompt structure to the agent, using strict step-by-step prompts for its homegrown agent and outcome-oriented prompts for Claude Code. | context |
| Spotify | Make autonomous background coding runs more predictable. | Additional dynamically fetched context and tools increase dimensions of unpredictability. | Spotify limits tools and hooks, constrains Git and Bash operations, and exposes standardized verification. | contextexecutionverification |
| Spotify | Improve the likelihood that a background coding task ends in a usable change. | Broad tasks without concrete success criteria are harder for agents to complete predictably. | Spotify uses tests, concrete examples, preconditions, and smaller task scope as verifiable end states. | verification |
| Shafin et al. | Generate complete classes through a structured multi-agent software-development process. | Direct code generation does not separate requirements, design, implementation, and verification responsibilities, making it difficult to study how process structure changes outcomes. | Assign Requirement Engineer, Architect, Developer, and Tester agents distinct responsibilities and pass artifacts sequentially between them with feedback-based refinement. | contextcoordination |
| Shafin et al. | Use verification results to refine generated implementation code before the workflow finishes. | Generated implementations can fail executable tests even after requirements and design stages have completed. | Have a Tester agent create and execute tests, emit a test report, and allow the Developer agent up to three bug-fix cycles from that feedback. | coordinationexecutionverification |
| Shafin et al. | Identify which parts of a multi-agent development process materially change correctness and code quality. | A complete multi-agent workflow can change outcomes without revealing which development activity caused the effect. | Remove Requirement, Design, or Testing one stage at a time and compare each configuration with the full workflow and RawPrompt baseline. | learningobservabilityverification |
| Spotify | Execute software migrations across a large repository fleet with a background coding agent. | Deterministic transformation declarations do not cover every migration, while repository targeting, pull-request lifecycle, review, and production merging still need coordinated infrastructure. | Spotify retained Fleet Management for repository targeting and pull-request lifecycle while replacing transformation declarations with natural-language instructions executed by a coding agent. | coordinationexecution |
| Spotify | Turn developer requests from Slack or GitHub into background coding work. | The background coding agent needs sufficient task information before it can independently produce a pull request. | An interactive agent gathers task information and hands a resulting prompt to the background coding agent. | contextcoordination |
| Spotify | Operate background coding agents predictably while agent and model components evolve. | Agent execution needs formatting, linting, diff evaluation, logs, traces, and the ability to change agent or model components without rebuilding the surrounding workflow. | Spotify built a pluggable CLI harness with MCP-based formatting and linting, LLM-as-judge diff evaluation, GCP logs, MLflow traces, and swappable agents and models. | executionobservabilityverification |
| Yin et al. | Identify where an agent architecture spends its inference budget. | Aggregate token totals do not reveal whether cost is concentrated in planning, reflection, execution, or editing. | Break token consumption down by workflow stage to expose architectural cost patterns and optimization opportunities. | economicsobservability |
| Yin et al. | Compare software-engineering agent frameworks before selecting one for repeated engineering work. | Task success alone can hide long trajectories, repeated corrections, or prohibitive token cost. | Evaluate agent frameworks on effectiveness, execution efficiency, and token or monetary overhead rather than ranking them only by final-task success. | economicsobservabilityverification |
| Yin et al. | Decide whether richer multi-agent orchestration is worth its operational complexity. | Additional roles and coordination loops can increase reasoning steps, corrections, latency, and resource consumption without a proportional outcome gain. | Measure trajectory length and correction behavior alongside effectiveness so coordination overhead is visible when comparing agent architectures. | coordinationeconomicsexecution |
| Yin et al. | Choose between lightweight and more elaborate agent orchestration for a software-engineering task. | Agent quantity is an unreliable proxy for execution quality or efficiency. | Compare reasoning depth, coordination strategy, feedback integration, effectiveness, and cost instead of assuming that adding agents improves the workflow. | coordinationeconomicslearning |
| Google Cloud | Scale multi-architecture rollout without destabilizing production services. | A migrated job may crash-loop or exhibit unacceptable throughput after rollout even if repository checks passed. | Use automated continuous health monitoring to detect unhealthy Arm jobs and evict them for later offline tuning and debugging. | executionobservabilityverification |
| Google Cloud | Migrate tens of thousands of applications when direct coordination with every application owner is impractical. | The coordination cost of scheduling and executing owner-by-owner migrations becomes prohibitive at warehouse scale. | Use programmatic change generation, automated verification and rollout systems, and AI repair agents to minimize required application-team involvement, escalating difficult cases to owners. | coordinationeconomicsexecution |
| Google Cloud | Repair build and test failures encountered while migrating applications across instruction-set architectures. | A migration can reach repository-specific build or test failures that do not lend themselves to a simple mechanical edit. | Invoke an AI migration agent when a build or test fails and let it iteratively edit files, execute the failing check, and continue until verification succeeds or the agent gives up. | contextexecutionverification |
| Google Cloud | Evaluate a migration agent against known real repository fixes before relying on wider production use. | Synthetic coding tasks may not represent the failure modes encountered in a warehouse-scale migration. | Select historical non-AI migration commits that can be cleanly rolled back, recreate the pre-fix state, and measure whether the agent can repair the same failures. | observabilityverification |
| Google Cloud | Separate repair strategies for build failures and test failures while preserving one migration workflow. | Build and test failures require different repair loops, and a test repair may itself encounter a build failure. | Use an outer orchestrator that repeatedly calls specialized build-fixer and test-fixer agents, with the test-fixer able to invoke the build-fixer as needed. | coordinationexecutionverification |
| Google Cloud | Allow an agent to adapt its next repair action to the result of the previous engineering command. | A repair loop needs current build and test evidence rather than relying only on the initial task description. | Execute each selected tool and attach its output back into the agent context before the next reasoning and tool-use step. | contextexecutionverification |
| Google Cloud / DORA | Absorb higher change volume produced by AI-assisted development without losing delivery stability. | Faster code production can expose downstream weaknesses and increase instability when feedback is slow or controls are weak. | Use strong automated testing and fast feedback loops as control systems around AI-assisted change. | executionverification |
| Google Cloud / DORA | Improve the usefulness of AI-assisted engineering beyond generic model knowledge. | General-purpose AI lacks organization-specific repository, documentation, and workflow context. | Invest in connecting AI tools to internal context so engineering assistance is grounded in company-specific information. | context |
| Google Cloud / DORA | Keep rapidly produced changes recoverable when AI-assisted work introduces regressions. | Higher change velocity increases the need to isolate, inspect, and reverse unsafe changes. | Maintain mature version-control practices as a safety net for accelerated AI-assisted development. | executionverification |
| Google Cloud / DORA | Scale AI-assisted engineering across teams with shared capabilities and guardrails. | Local AI productivity does not automatically translate into reliable organizational outcomes when teams lack a strong execution platform. | Invest in a high-quality internal platform that provides shared engineering capabilities and supports reliable AI-assisted workflows. | executionverification |
| Google Cloud / DORA | Keep accelerated AI-assisted work aligned with useful product outcomes. | Faster engineering output can move a team in the wrong direction when the problem and user outcome are unclear. | Use explicit user-centric focus to provide direction for AI-assisted engineering work. | context |
| OpenAI | Make delegated agent execution resemble the repository's real engineering environment. | An agent can produce misleading results when dependencies, tooling, or environment setup differ from the environment in which the code is normally built and tested. | Configure the agent sandbox with the repository's dependencies, tooling, and setup so execution matches the real development environment as closely as practical. | contextexecutionverification |
| OpenAI | Integrate agent-generated code into normal software-delivery workflows. | Passing checks and agent-generated evidence do not guarantee that a generated change is safe or correct for production use. | Require human review and validation of agent-generated code before integration and execution. | verification |
| OpenAI | Delegate repository changes to a background coding agent without sharing mutable execution state across tasks. | Concurrent coding tasks can interfere with one another or with a developer's local workspace when they share execution state. | Run each delegated coding task independently in its own isolated cloud environment preloaded with the target repository. | contextexecution |
| OpenAI | Give a coding agent repository-specific navigation, testing, and convention guidance before it edits code. | A generic agent prompt does not encode how a particular repository is organized, tested, or expected to be changed. | Store repository-local AGENTS.md files that tell the agent how to navigate the codebase, which commands to run, and which project practices to follow. | context |
| OpenAI | Review background agent work without trusting only the agent's natural-language summary. | A reviewer needs evidence that the claimed edits and checks actually occurred before accepting an asynchronously produced change. | Return citations to terminal logs and test outputs alongside the generated change so reviewers can trace and validate the agent's actions. | observabilityverification |
| Cursor | Delegate bounded engineering work that can return as a reviewable artifact. | Investigations, small fixes, and medium-sized changes consume developer attention even when the work can be performed asynchronously. | Use background agents for small fixes, investigations, and first drafts of medium-sized pull requests. | execution |
| Cursor | Keep humans in control of remotely executing coding-agent tasks. | A long-running remote task needs supervision and a way to recover when the agent needs correction or human intervention. | Expose per-agent status and allow the user to send follow-up instructions or take over execution at any point. | coordinationobservability |
| Cursor | Execute several software-engineering tasks without serially occupying one foreground coding session. | A single interactive agent session limits how much independent engineering work can proceed at once. | Run multiple coding agents in parallel, with each agent operating in its own remote environment. | coordinationexecution |
| Airbnb | Finish residual migration work after automation stops being economical. | Some files remained failing after 50-100 retries despite expanded context and repeated tuning. | Stop further automation tuning for the final long tail and complete those files manually using the failing automated refactor as a baseline. | economicsexecutionverification |
| Airbnb | Recover automatically when an LLM-generated migration step fails validation. | A first attempt frequently leaves validation errors that require another targeted correction. | Retry failed steps with dynamic prompts containing the validation errors and latest file version, up to a configurable limit. | executionlearningverification |
| Airbnb | Migrate complex tests with project-specific setup, indirection, and conventions. | Retrying alone stopped being effective when the model lacked the repository context needed to resolve complex test behavior. | Expand prompts with selected source files, related tests, passing project-local examples, migration guidance, and validation failures, reaching roughly 40K-100K tokens and up to 50 related files. | contexteconomics |
| Airbnb | Improve completion across the remaining long tail without tuning one file at a time. | Repeated failures clustered around common issues that required changes to prompts or migration tooling. | Sample representative failures, tune prompts and scripts, verify the change on the sample, then sweep the remaining corpus again. | learningobservabilityverification |
| Airbnb | Migrate thousands of tests while preserving test intent and code coverage. | An LLM-generated refactor cannot be treated as complete until repository-specific validation succeeds. | Break migration into per-file refactor and validation states and advance a file only after the preceding check passes. | executionobservabilityverification |