Evidence-backed practices
Emerging Practices
Evidence-backed engineering practices emerging across public engineering organizations, shown with the use case, problem encountered, reported response, and Selection conditions behind each observation.
47Evidence
47Assessed
0Pending
165Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for emerging practices. Assessed evidence may legitimately contain zero practice observations.
72 evidence-backed practice observations · verification
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Anthropic | Maintain human control when generated change volume makes uniform line-by-line review increasingly expensive. | Agents can produce locally plausible code that violates system architecture, uses brute-force access paths, or over-engineers hypothetical concerns. | Engineers review every change request with increasing emphasis on intent, architectural invariants, system-level tradeoffs, and rejecting invalid solution patterns. | coordinationverification |
| Anthropic | Integrate hundreds or thousands of changes while tests execute against a rapidly advancing main branch. | A globally locked merge queue serializes change and cannot keep pace when main advances substantially during each test run. | Anthropic invests in post-merge and pre-deployment automated testing, accepts concurrent integration, and uses agents to repair resulting conflicts. | coordinationexecutionverification |
| Atlassian | Operate agent access across many enterprise products and external AI clients. | Distributed integrations make it difficult to enforce data policy, constrain connectivity, and reconstruct agent actions. | Atlassian centralizes domain, permission-scope, IP, data-protection, and data-loss-prevention controls with organization-wide audit logging. | coordinationobservabilityverification |
| Atlassian | Allow agents and teammates to edit shared organizational documents concurrently. | An agent can overwrite a teammate's newer change when it acts on stale document state. | Atlassian uses version-aware, section-level edits that pause on concurrent changes and expose a preview before publication. | coordinationobservabilityverification |
| Atlassian | Let external AI clients act on enterprise work without gaining authority beyond the invoking user. | A shared agent integration can become an authority-escalation path if tool access bypasses existing identity and permissions. | Atlassian authenticates with OAuth 2.1 and PKCE and evaluates every request against the invoking user's existing permissions. | executionverification |
| Atlassian | Govern a broad MCP surface containing read, write, and destructive operations. | Treating all tool calls alike prevents clients and administrators from applying stronger controls to higher-impact actions. | Atlassian separates read, write, and destructive actions, scopes access by toolset, and enables explicit confirmation for higher-risk operations. | coordinationverification |
| GitHub | Increase automated repository throughput while retaining organizational control. | Higher agent activity can outrun governance if throughput replaces review, protection, auditability, or approval mechanisms. | GitHub treats branch protection, required review, audit logs, repository visibility, and human approval as constraints the new architecture must preserve. | coordinationobservabilityverification |
| Astekin et al. | Decide whether an engineering task needs non-agentic, single-agent, or multi-agent execution. | Selecting a complex agent architecture without a lightweight baseline can hide whether cooperation contributes any task-specific benefit. | Evaluate an explicit ladder from direct model query through single-, dual-, and multi-agent configurations under matched task conditions. | coordinationeconomicsverification |
| Astekin et al. | Introduce multi-agent cooperation for a bounded software-engineering workflow. | Additional agents impose systematic execution costs while accuracy gains vary by task. | Begin with a lightweight task-aware baseline and adopt multi-agent workflows only when comparative evaluation demonstrates a benefit that justifies the added cost. | coordinationeconomicsverification |
| Astekin et al. | Choose an agent configuration when outcome quality, response time, and resource use all matter. | Ranking only by task accuracy can select a configuration whose small benefit carries disproportionate latency and energy costs. | Compare configurations on a joint accuracy, latency, and energy Pareto front rather than optimizing one metric in isolation. | economicsobservabilityverification |
| Astekin et al. | Select models and prompting strategies across different engineering task families. | One model or prompt strategy does not remain optimal across code, vulnerability, debt, and log-analysis work. | Treat model and prompt choice as task-specific experimental variables rather than global defaults. | executionverification |
| Cursor | Distinguish a production regression from expected effects after a deployment. | Operators must connect a changed signal to one of several deployed changes and decide whether the movement is intentional. | Follow the change through deployment and compare its planned signals with a pre-deploy baseline to identify localized or unintended regressions. | contextobservabilityverification |
| Cursor | Detect exploit paths introduced by a pull-request change. | Local pattern matching can flag superficial risks while missing vulnerabilities that depend on data flow, authorization, dependencies, or broader codebase context. | Analyze every pull request against whole-codebase context and return severity, attack path, explanation, and a proposed fix for each finding. | contextverification |
| Cursor | Define how a particular code change will be verified after deployment. | Generic dashboards and alerts may not cover the risks, intended effects, or instrumentation gaps introduced by a specific diff. | Read the pull-request diff before merge, draft an editable plan of expected effects and risk signals, and flag where telemetry cannot determine success. | contextobservabilityverification |
| Cursor | Respond to a suspected deployment regression with authority appropriate to organizational risk. | A single remediation policy is either too passive for low-risk cases or too autonomous for consequential production changes. | Configure the rollout bot to escalate to the author, pause progressive delivery, or prepare a revert pull request for human approval. | coordinationexecutionverification |
| Atlassian | Choose how a coding agent should retrieve structured engineering context. | Equivalent knowledge can produce different end-to-end outcomes when invocation, instructions, and output formats differ. | Hold the agent, model, tasks, and questions constant; vary only the context-delivery path; then record pass rate, task time, tokens, tool calls, and transcripts. | contexteconomicsobservabilityverification |
| Cursor | Continue a migration across hundreds of pull requests without requiring equally intensive review of every change. | Human review becomes the throughput constraint when every repetitive agent-produced change receives the same level of scrutiny. | Establish a safe approach, review early pull requests closely, and reduce review intensity as repeated fixes hold up. | learningverification |
| Cursor | Turn completed agent work into reusable improvements for future workflows. | Learnings from completed tasks disappear unless they are reviewed and encoded into durable workflow artifacts. | The Reflect skill runs three parallel reviewers, synthesizes durable learnings, checks whether they should be encoded structurally, and requires approval before skill edits. | learningverification |
| Cursor | Keep verification evidence valid as agent-authored pull requests change. | A verification verdict can become stale when the pull-request head changes. | pstack records verification verdicts against a pull-request number and head SHA and voids the verdict when the head SHA changes. | verification |
| Cursor | Coordinate multi-day programs containing many stacked pull requests and subagents. | Parallel work needs explicit ownership, bounded context transfer, durable state, and independent verification. | pstack defines coordinator, optional sub-coordinator, worker, and verifier roles; structured state files; queue events; and worker briefs. | contextcoordinationverification |
| Meta | Convert expert corrections into durable improvements to agent knowledge and reasoning. | Manually tracing dependencies, editing interdependent files, and proving that a correction does not break existing behavior can take weeks. | Diagnose feedback, compile minimal edits with parallel impact analysis, perform independent review and structural checks, then submit a validated pull request for expert approval. | coordinationlearningverification |
| Meta | Verify that a proposed knowledge or reasoning change fixes the reported failure without losing existing behavior. | An automatically drafted correction can solve one case while introducing regressions elsewhere in the institutional knowledge system. | Run blind targeted replay and domain regression tests, retry failed compilations, and add every landed failure case to the permanent regression suite. | learningverification |
| Teresa Torres | Determine whether a change to an AI workflow actually improves its behavior. | Anecdotal testing cannot reliably distinguish an improvement from a regression. | Establish a baseline, change prompt/context/model/decomposition/orchestration, rerun the same inputs, apply evals, and compare scores. | learningverification |
| Teresa Torres | Turn failures observed in real AI usage into regression protection. | Production failures remain isolated incidents unless they become reusable evaluation criteria. | Review traces, log and categorize recurring mistakes, and turn those failure categories into eval candidates. | learningobservabilityverification |
| Teresa Torres | Run judgment-based evaluation without paying its cost for every case. | LLM-as-a-Judge evaluations are more expensive than deterministic assertions. | Use a cheap code assertion first and send only threshold-crossing cases to an LLM judge. | economicsverification |
| Infobip | Delegate implementation to fresh coding-agent sessions with sufficient task context. | Fresh sessions need goals, dependencies, relevant code paths, success criteria, and validation methods without relying on conversational history. | Research and planning produce validated artifacts, and task definitions carry goals, dependencies, code paths, acceptance criteria, and validation methods. | contextverification |
| Uber | Determine whether agent automation is economically efficient. | Token or request spend alone does not show how much useful engineering work was delivered. | Uber tracks outcome-denominated cost together with workload-specific quality signals and delivered volume. | economicsverification |
| Uber | Choose models for specific managed software-engineering agent workloads. | General model benchmarks do not establish the best quality, reliability, and economics for a particular production workload. | Uber builds benchmarks from each managed agent's real work and evaluates cost per completed task, output quality, and reliability. | economicsverification |
| Destefanis and Aste | Decompose a software change across agents with dependent outputs. | Each contribution can be locally correct while a convention spanning two owners remains unresolved and causes the integrated system to fail. | Identify cross-agent interfaces explicitly and assign one agent responsibility for validating each end-to-end contract. | contextcoordinationverification |
| Destefanis and Aste | Understand how a multi-agent coding run coordinates beneath its final output. | Pass rates and aggregate tokens hide which agents communicated, which files carried shared state, when coordination happened, and where it failed. | Represent agents and files as nodes and log messages, reads, and writes as timestamped, costed edges linked to the run's verified outcome. | coordinationobservabilityverification |
| Destefanis and Aste | Evaluate multi-agent configurations whose tasks leave teams freedom to self-organize. | Identical configurations under a pinned model can produce substantially different coordination networks across sessions. | Run each configuration repeatedly, compare matched sessions, and report distributions or cross-session ranges rather than treating one run as representative. | observabilityverification |
| Destefanis and Aste | Evaluate coding agents against hidden tests or reference solutions. | Agents may search outside their intended workspace for grading material, contaminating scores and revealing weak containment. | Remove access to hidden evaluation artifacts, substitute detectable placeholders, and repeat key conditions under sealed execution. | executionobservabilityverification |
| Destefanis and Aste | Introduce a coordinator or orchestrator into a multi-agent workflow. | Naming an agent as coordinator in its prompt may create no hub, authority, routing behavior, or reliable outcome improvement. | Inspect the interaction graph and comparative outcomes to verify that the proposed coordinator actually performs a structural integration role. | coordinationobservabilityverification |
| Cursor | Review large volumes of concurrently agent-produced code. | A single review mechanism can become a verification bottleneck as swarm output grows. | Cursor uses stacked review agents with different review lenses. | coordinationverification |
| Spotify | Run many background coding-agent sessions with controlled execution and verification. | Autonomous sessions need an execution environment with trusted tools and a reliable quality gate. | Spotify runs Honk sessions in Kubernetes with trusted tools and CI verification. | executionverification |
| OpenAI | Run more concurrent Codex coding tasks than an engineer can comfortably supervise as separate sessions. | Engineers lost track of work and debugged stalled agents beyond roughly three to five simultaneous sessions. | Symphony moves task assignment and workspace lifecycle into an always-on orchestrator where open tasks receive agents in isolated workspaces and humans review outcomes. | coordinationexecutionobservabilityverification |
| Cloudflare | Make organizational engineering standards directly usable by coding and review agents. | Standards remain weak selection pressure when they live only in human-readable documents disconnected from agent execution and review. | Distill engineering standards into structured Codex rules and an agent skill, then cite those rules during automated review and use them in multi-agent compliance audits. | contextlearningverification |
| Cloudflare | Give coding agents repository-specific operational context across thousands of repositories. | Agents otherwise infer ownership, architecture, build commands, testing conventions and standards from incomplete repository exploration. | Generate AGENTS.md from Backstage metadata, repository structure and Engineering Codex rules, open it as a merge request for owner review, and flag it for refresh when repository changes make it stale. | contextlearningverification |
| Cloudflare | Increase the chance that agent-generated changes already fit repository and organizational constraints before human review. | Agent outputs remain plausible but locally wrong when context acquisition, execution and standards enforcement are disconnected. | Wire Backstage context, repository AGENTS.md, centralized tools and Codex-backed AI review into one engineering path; Cloudflare reports that the resulting first draft is now usually close enough to ship. | contextcoordinationeconomicsexecutionlearningobservabilityverification |
| Cloudflare | Review every merge request on a standard CI path without applying the same review cost and expertise to every change. | Uniform generic review does not match the risk profile or expertise required by different engineering changes. | Classify merge requests by risk tier and delegate review to specialized agents for code quality, security, standards compliance, documentation, performance and release impact. | coordinationeconomicsverification |
| Meta | Prevent generated repository guidance from becoming a new source of agent error. | AI-generated context can be incorrect, stale, overly broad, or contain invalid references. | Run multiple rounds of independent critics, prompt tests, integration checks, and file-path validation before relying on generated context. | observabilityverification |
| Stripe | Determine whether an agent-generated integration actually works after the agent declares completion. | An agent can mistake a plausible intermediate result or expected-looking error for task success. | Use deterministic API and automated UI graders, including inspection of created service-side objects, to validate the final system independently of the agent's own completion judgment. | observabilityverification |
| Stripe | Measure whether coding agents can complete end-to-end API integration work rather than isolated code-generation tasks. | Function-level benchmarks can miss the planning, state management, browser use, database work, and glue code required by real integrations. | Evaluate agents in complete coding environments with repositories, databases, scripts, test credentials, browsers, and real API interactions. | contextexecutionverification |
| Stripe | Help coding agents verify API migrations and integrations against meaningful runtime behavior. | Testing with nonexistent or unrealistic data can produce technically valid errors that an agent misreads as proof that the implementation works. | Provide realistic test credentials and state, and grade behavior against valid end-to-end outcomes rather than accepting error responses as verification. | contextverification |
| Stripe | Improve an agent workflow after observing recurring execution or recovery failures. | Without replayable tasks, changes to prompts, skills, tools, or documentation cannot be compared against the same engineering problem. | Use replayable benchmark environments as a test bed for prompt, skill, browser-tool, and documentation interventions, then measure whether task performance changes. | learningobservabilityverification |
| Dae-Kyoo Kim | Assess whether an agent configuration generalizes beyond one favorable engineering task. | A configuration can appear effective on one task or domain while performing poorly elsewhere. | Evaluate specialization strategies across multiple lifecycle tasks and application domains using structural and semantic artifact-validity criteria. | learningverification |
| Dae-Kyoo Kim | Evaluate whether a new specialization strategy improves an agentic engineering workflow. | Changing the model and orchestration at the same time makes it difficult to attribute outcome differences to agent configuration. | Hold the underlying LLM constant while comparing orchestration granularities so configuration effects can be evaluated more directly. | observabilityverification |
| Dae-Kyoo Kim | Choose how narrowly to specialize agents across a software-development workflow. | Increasing role specialization does not consistently improve artifact validity across all engineering tasks. | Compare task-specialized, phase-specialized, and process-generalist configurations and choose specialization granularity according to the task rather than assuming narrower roles are universally better. | coordinationlearningverification |
| Spotify | Determine whether an autonomously generated change is ready to become a pull request. | Agent completion does not establish that the generated code works or satisfies the task. | Spotify selects independent deterministic verifiers from repository contents and requires relevant verification before a pull request can be opened, then applies an LLM judge to the diff. | verification |
| Spotify | Automatically repair an agent-generated change that fails quality evaluation. | A verifier or judge veto would otherwise leave failed work for a human to recover. | Judge vetoes are fed back into the coding-agent loop, and Spotify reports that the agent course-corrects about half of vetoed cases. | learningverification |
| Bun | Use a coding agent to fix bugs in the Bun repository. | An agent-authored fix needs evidence that it resolves the bug rather than merely changing code. | Bun's Claude Code bot opens pull requests with tests that fail against the earlier system-installed Bun and pass against the fixed debug build, then responds to review comments. | executionverification |
| Spotify | Make autonomous background coding runs more predictable. | Additional dynamically fetched context and tools increase dimensions of unpredictability. | Spotify limits tools and hooks, constrains Git and Bash operations, and exposes standardized verification. | contextexecutionverification |
| Spotify | Improve the likelihood that a background coding task ends in a usable change. | Broad tasks without concrete success criteria are harder for agents to complete predictably. | Spotify uses tests, concrete examples, preconditions, and smaller task scope as verifiable end states. | verification |
| Shafin et al. | Use verification results to refine generated implementation code before the workflow finishes. | Generated implementations can fail executable tests even after requirements and design stages have completed. | Have a Tester agent create and execute tests, emit a test report, and allow the Developer agent up to three bug-fix cycles from that feedback. | coordinationexecutionverification |
| Shafin et al. | Identify which parts of a multi-agent development process materially change correctness and code quality. | A complete multi-agent workflow can change outcomes without revealing which development activity caused the effect. | Remove Requirement, Design, or Testing one stage at a time and compare each configuration with the full workflow and RawPrompt baseline. | learningobservabilityverification |
| Spotify | Operate background coding agents predictably while agent and model components evolve. | Agent execution needs formatting, linting, diff evaluation, logs, traces, and the ability to change agent or model components without rebuilding the surrounding workflow. | Spotify built a pluggable CLI harness with MCP-based formatting and linting, LLM-as-judge diff evaluation, GCP logs, MLflow traces, and swappable agents and models. | executionobservabilityverification |
| Yin et al. | Compare software-engineering agent frameworks before selecting one for repeated engineering work. | Task success alone can hide long trajectories, repeated corrections, or prohibitive token cost. | Evaluate agent frameworks on effectiveness, execution efficiency, and token or monetary overhead rather than ranking them only by final-task success. | economicsobservabilityverification |
| Google Cloud | Scale multi-architecture rollout without destabilizing production services. | A migrated job may crash-loop or exhibit unacceptable throughput after rollout even if repository checks passed. | Use automated continuous health monitoring to detect unhealthy Arm jobs and evict them for later offline tuning and debugging. | executionobservabilityverification |
| Google Cloud | Repair build and test failures encountered while migrating applications across instruction-set architectures. | A migration can reach repository-specific build or test failures that do not lend themselves to a simple mechanical edit. | Invoke an AI migration agent when a build or test fails and let it iteratively edit files, execute the failing check, and continue until verification succeeds or the agent gives up. | contextexecutionverification |
| Google Cloud | Evaluate a migration agent against known real repository fixes before relying on wider production use. | Synthetic coding tasks may not represent the failure modes encountered in a warehouse-scale migration. | Select historical non-AI migration commits that can be cleanly rolled back, recreate the pre-fix state, and measure whether the agent can repair the same failures. | observabilityverification |
| Google Cloud | Separate repair strategies for build failures and test failures while preserving one migration workflow. | Build and test failures require different repair loops, and a test repair may itself encounter a build failure. | Use an outer orchestrator that repeatedly calls specialized build-fixer and test-fixer agents, with the test-fixer able to invoke the build-fixer as needed. | coordinationexecutionverification |
| Google Cloud | Allow an agent to adapt its next repair action to the result of the previous engineering command. | A repair loop needs current build and test evidence rather than relying only on the initial task description. | Execute each selected tool and attach its output back into the agent context before the next reasoning and tool-use step. | contextexecutionverification |
| Google Cloud / DORA | Absorb higher change volume produced by AI-assisted development without losing delivery stability. | Faster code production can expose downstream weaknesses and increase instability when feedback is slow or controls are weak. | Use strong automated testing and fast feedback loops as control systems around AI-assisted change. | executionverification |
| Google Cloud / DORA | Keep rapidly produced changes recoverable when AI-assisted work introduces regressions. | Higher change velocity increases the need to isolate, inspect, and reverse unsafe changes. | Maintain mature version-control practices as a safety net for accelerated AI-assisted development. | executionverification |
| Google Cloud / DORA | Scale AI-assisted engineering across teams with shared capabilities and guardrails. | Local AI productivity does not automatically translate into reliable organizational outcomes when teams lack a strong execution platform. | Invest in a high-quality internal platform that provides shared engineering capabilities and supports reliable AI-assisted workflows. | executionverification |
| OpenAI | Make delegated agent execution resemble the repository's real engineering environment. | An agent can produce misleading results when dependencies, tooling, or environment setup differ from the environment in which the code is normally built and tested. | Configure the agent sandbox with the repository's dependencies, tooling, and setup so execution matches the real development environment as closely as practical. | contextexecutionverification |
| OpenAI | Integrate agent-generated code into normal software-delivery workflows. | Passing checks and agent-generated evidence do not guarantee that a generated change is safe or correct for production use. | Require human review and validation of agent-generated code before integration and execution. | verification |
| OpenAI | Review background agent work without trusting only the agent's natural-language summary. | A reviewer needs evidence that the claimed edits and checks actually occurred before accepting an asynchronously produced change. | Return citations to terminal logs and test outputs alongside the generated change so reviewers can trace and validate the agent's actions. | observabilityverification |
| Airbnb | Finish residual migration work after automation stops being economical. | Some files remained failing after 50-100 retries despite expanded context and repeated tuning. | Stop further automation tuning for the final long tail and complete those files manually using the failing automated refactor as a baseline. | economicsexecutionverification |
| Airbnb | Recover automatically when an LLM-generated migration step fails validation. | A first attempt frequently leaves validation errors that require another targeted correction. | Retry failed steps with dynamic prompts containing the validation errors and latest file version, up to a configurable limit. | executionlearningverification |
| Airbnb | Improve completion across the remaining long tail without tuning one file at a time. | Repeated failures clustered around common issues that required changes to prompts or migration tooling. | Sample representative failures, tune prompts and scripts, verify the change on the sample, then sweep the remaining corpus again. | learningobservabilityverification |
| Airbnb | Migrate thousands of tests while preserving test intent and code coverage. | An LLM-generated refactor cannot be treated as complete until repository-specific validation succeeds. | Break migration into per-file refactor and validation states and advance a file only after the preceding check passes. | executionobservabilityverification |