Evidence-backed practices
Emerging Practices
Evidence-backed engineering practices emerging across public engineering organizations, shown with the use case, problem encountered, reported response, and Selection conditions behind each observation.
47Evidence
47Assessed
0Pending
165Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for emerging practices. Assessed evidence may legitimately contain zero practice observations.
66 evidence-backed practice observations · execution
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Anthropic | Add and optimize history, blame, search, and graph capabilities without changing clients or materializing every index eagerly. | Precomputing all repository metadata increases write amplification and couples new query capabilities to client formats. | Anthropic defines recursive derived-data types that can be computed eagerly or lazily and optimized behind a stable service interface. | economicsexecution |
| Anthropic | Serve ephemeral agents and CI jobs operating continuously against a very large monorepo. | Complete local repositories and offline-capable clients transfer and maintain data that online automated clients rarely need. | Anthropic moves history, diff, blame, commit, merge, and rebase operations to a server while clients retain bounded caches and fetch only requested data. | contexteconomicsexecution |
| Anthropic | Let CI, agents, and internal tools query repository state without checking out or maintaining a repository. | Every consumer independently cloning and maintaining repository data multiplies compute, storage, transfer, and operational work. | Scion exposes diff, ancestry, blame, merge-base, blob, and related operations through horizontally scalable APIs. | contexteconomicsexecution |
| Anthropic | Accept very high rates of concurrent changes without making every commit contend on global repository state. | Packfile processing, locking, and shared-reference movement place unrelated work on the same coordination-critical path. | Scion creates incremental server-side changesets without packfiles or locks and separates changeset creation from the small number of supported reference updates. | coordinationeconomicsexecution |
| Anthropic | Integrate hundreds or thousands of changes while tests execute against a rapidly advancing main branch. | A globally locked merge queue serializes change and cannot keep pace when main advances substantially during each test run. | Anthropic invests in post-merge and pre-deployment automated testing, accepts concurrent integration, and uses agents to repair resulting conflicts. | coordinationexecutionverification |
| Atlassian | Let external AI clients act on enterprise work without gaining authority beyond the invoking user. | A shared agent integration can become an authority-escalation path if tool access bypasses existing identity and permissions. | Atlassian authenticates with OAuth 2.1 and PKCE and evaluates every request against the invoking user's existing permissions. | executionverification |
| GitHub | Scale repository reads and writes independently under bursty agent and CI workloads. | Using complete local repository copies for both durability and serving means that adding read replicas also adds overhead to every write. | GitHub separates authoritative repository data in durable object storage from elastic compute workers that cache and serve requests. | economicsexecution |
| GitHub | Accept frequent concurrent pushes from developers, automation, and agent fleets. | Coordinating every part of a push across durable replicas makes write latency and throughput depend on work that does not require global agreement. | GitHub narrows synchronous coordination to the reference update while allowing object storage, connectivity validation, and secret scanning to proceed independently or in parallel. | coordinationexecution |
| GitHub | Maintain heavily written repositories without slowing live pushes and fetches. | Compaction and garbage collection compete with request handling when they execute on the same hosts. | GitHub moves compaction and garbage collection to separate workers operating directly against durable storage. | executionobservability |
| Astekin et al. | Deploy heavier agent workflows whose latency and energy depend on the execution platform. | A configuration that appears viable on high-end server hardware may be too slow or inefficient on a developer workstation. | Measure the same agent configurations across representative hardware and include runtime environment in the selection decision. | economicsexecution |
| Astekin et al. | Select models and prompting strategies across different engineering task families. | One model or prompt strategy does not remain optimal across code, vulnerability, debt, and log-analysis work. | Treat model and prompt choice as task-specific experimental variables rather than global defaults. | executionverification |
| Cursor | Respond to a suspected deployment regression with authority appropriate to organizational risk. | A single remediation policy is either too passive for low-risk cases or too autonomous for consequential production changes. | Configure the rollout bot to escalate to the author, pause progressive delivery, or prepare a revert pull request for human approval. | coordinationexecutionverification |
| Cursor | Keep a broad tool set available without placing every definition in every model request. | Most specialized tools are used in a minority of conversations, but static definitions consume context throughout every turn. | Keep high-frequency and flow-critical tools in static context and load lower-frequency MCP and built-in tool definitions only when needed. | contexteconomicsexecution |
| Will Larson | Advance a broad engineering goal across a changing set of tasks. | A task-scoped agent cannot identify missing work, reconcile changed issue state, or choose the next non-blocked action. | Review metrics and issues, create newly identified tasks, update state, execute non-blocked work, and select the next action from current project state. | coordinationexecutionobservability |
| Will Larson | Implement one project across frontend, backend, infrastructure, and data repositories. | Repository-scoped checkouts make coordinated cross-repository agent changes difficult to manage. | Maintain about ten independent local workspaces, each with a checkout of every repository, and operate agents at workspace rather than repository scope. | contextcoordinationexecution |
| Atlassian | Retrieve several related components, tokens, or guidance records during one agent task. | Agents made repeated context round trips, accumulating avoidable lookup latency. | Add a batch command and improve fuzzy search so agents can consolidate related context queries. | contexteconomicsexecution |
| Atlassian | Serve context repeatedly during an agent task without paying process or package startup costs on every lookup. | An npx-based command added more than 15 seconds per query and compounded when agents performed multiple lookups. | Provide a pre-installed CLI path for the default environment while retaining the portable npx invocation as a fallback. | economicsexecution |
| Atlassian | Expose one governed knowledge source to agents with different tool access. | Some agent environments provide a shell but no MCP client, while some sandboxes cannot access a terminal. | Generate multiple interfaces from shared handlers and data, prefer the benchmarked CLI path, and fall back to MCP or bundled references when required by the environment. | contextexecution |
| Cursor | Act on new pull requests, CI failures, bug reports, schedules, or Slack signals without waiting for a new prompt. | Prompt-driven agents stop observing the environment after an individual task completes. | Subscribe the coordinator to schedules and external events so it can initiate recurring maintenance and follow-up work. | executionobservability |
| Cursor | Keep long-running work active while still executing tests or operations that require a developer's machine. | Laptop-bound execution limits duration and parallelism, while some tasks still require local state or hardware. | Run the Project persistently on a cloud computer and let its coordinator start a local agent when machine-specific testing is needed. | coordinationexecution |
| Cursor | Direct a feature, migration, or recurring maintenance program that outlives one agent session. | A developer otherwise has to repeatedly plan work, start agents, relay context, and integrate results across a long-running body of work. | Create a persistent Project whose coordinator plans and delegates while separate worker agents implement and test bounded tasks. | coordinationexecution |
| OpenAI | Keep agents operating reliably across sessions that span hours, days, and multiple context windows. | Long-running work requires context continuity, recoverable execution, files, code execution, and preserved intermediate results. | Use a managed, versioned agent harness with automatic context compaction and a durable execution environment. | contextexecution |
| OpenAI | Parallelize complex work across independent subproblems without implementing a custom coordination layer. | Applications otherwise need to build and observe delegation, separate subagent contexts, parallel execution, and result aggregation themselves. | Delegate independent work to parallel subagents with separate contexts while a main agent coordinates and combines their results. | coordinationexecutionobservability |
| OpenAI | Let an agent use a large tool environment without loading every tool definition or raw result into context. | Large tool catalogs and result sets consume tokens and dilute the context needed for the task. | Load relevant tools on demand, run related calls programmatically, and return only relevant combined results to model context. | contextexecution |
| Destefanis and Aste | Evaluate coding agents against hidden tests or reference solutions. | Agents may search outside their intended workspace for grading material, contaminating scores and revealing weak containment. | Remove access to hidden evaluation artifacts, substitute detectable placeholders, and repeat key conditions under sealed execution. | executionobservabilityverification |
| Cloudflare | Let sites identify and selectively admit legitimate agents. | Sites cannot reliably make policy decisions when machine actors are indistinguishable, spoofed, or treated as generic bots. | Cloudflare presents Web Bot Auth and PACT as cryptographic identity and delegated trust mechanisms for agents. | coordinationexecution |
| Cloudflare | Let agents perform actions on websites without reverse-engineering the DOM. | DOM parsing, guessed form fields, and synthesized clicks are brittle interfaces for machine actors. | WebMCP lets sites register explicit tools and schemas that agents can invoke using the user's existing session and state. | coordinationexecutionobservability |
| Cursor | Integrate changes produced simultaneously by many agents. | Conventional merge behavior accumulated more than 70,000 conflicts in an earlier run. | Cursor uses neutral third-party agents for merge conflicts and a purpose-built version-control system at higher commit throughput. | coordinationexecution |
| Stripe | Allow coding agents to provision and configure infrastructure without exposing production by default. | An agent that goes off-script can create direct production impact if credentials and environment access are too broad. | Use named environments with isolated credentials and default agents to the development environment rather than production. | coordinationexecution |
| Stripe | Move from code generation to runnable software without requiring a human to configure each provider dashboard. | Provisioning infrastructure, connecting services, and managing credentials remain adjacent bottlenecks even when an agent can write the application code. | Expose infrastructure provisioning through a structured agent skill or embedded CLI and connect it to multiple providers through delegated, scoped authority. | contextcoordinationexecution |
| Spotify | Run many background coding-agent sessions with controlled execution and verification. | Autonomous sessions need an execution environment with trusted tools and a reliable quality gate. | Spotify runs Honk sessions in Kubernetes with trusted tools and CI verification. | executionverification |
| Google Cloud | Run agent jobs that continue for hours or days and survive infrastructure or human interruptions. | A long-running agent cannot depend on one process or session remaining alive for the duration of the work. | Google Agent Executor uses durable event logs and snapshots to support execution, resumption, and distributed deployment. | executionobservability |
| OpenAI | Run more concurrent Codex coding tasks than an engineer can comfortably supervise as separate sessions. | Engineers lost track of work and debugged stalled agents beyond roughly three to five simultaneous sessions. | Symphony moves task assignment and workspace lifecycle into an always-on orchestrator where open tasks receive agents in isolated workspaces and humans review outcomes. | coordinationexecutionobservabilityverification |
| Cursor | Execute multiple parts of an engineering task concurrently. | Parallel agent work needs decomposition, isolation, and dependency-aware composition to avoid interfering changes. | Cursor Multitask decomposes work into asynchronous subagents, uses worktrees to isolate concurrent changes, and parallel plans to preserve task dependencies. | coordinationexecution |
| Spotify | Apply a dataset migration across thousands of downstream consumers. | Repository-scale changes need coordinated scheduling and pull-request execution across a large fleet. | Spotify combines Honk, Backstage, Codesearch, and Fleet Management to target and orchestrate the migration. | coordinationexecution |
| Cloudflare | Provide many engineering clients and agents with governed access to models and internal capabilities. | Direct client-to-model integrations make attribution, permissions, model changes and cost control difficult to manage consistently at organizational scale. | Route AI clients through a shared proxy Worker and AI Gateway so identity, permissions, model catalog configuration, usage attribution and routing can be changed centrally. | coordinationeconomicsexecutionobservability |
| Cloudflare | Increase the chance that agent-generated changes already fit repository and organizational constraints before human review. | Agent outputs remain plausible but locally wrong when context acquisition, execution and standards enforcement are disconnected. | Wire Backstage context, repository AGENTS.md, centralized tools and Codex-backed AI review into one engineering path; Cloudflare reports that the resulting first draft is now usually close enough to ship. | contextcoordinationeconomicsexecutionlearningobservabilityverification |
| Cloudflare | Control inference cost while retaining stronger reasoning for high-risk engineering work. | Using the most capable model for every review task makes high-volume agentic workflows unnecessarily expensive. | Route documentation and lower-risk review workloads to lower-cost Workers AI models while reserving frontier models for security-sensitive and architecturally complex reviews. | economicsexecution |
| Cloudflare | Assess whether a web environment exposes the information and capabilities agents need to operate efficiently. | Agent compatibility remains difficult to improve when discoverability, content access, permissions, and callable capabilities are implicit or unmeasured. | Score sites across Discoverability, Content, Bot Access Control, and Capabilities using machine-checkable standards. | contextexecutionobservability |
| Cloudflare | Turn an environmental readiness gap into an actionable implementation task. | A readiness assessment has limited value if teams must independently determine how to repair every failing condition. | For each failed readiness check, generate a prompt that can be given to a coding agent to implement the missing support. | executionlearning |
| Cloudflare | Keep long-running agent work recoverable without maintaining an always-on process for every agent. | Deploys, restarts, and idle periods can destroy in-memory work or make one-runtime-per-agent economics impractical. | Give each agent addressable persistent state and run checkpointable work that can recover after the runtime restarts and hibernate when idle. | economicsexecution |
| Cloudflare | Delegate a bounded task without sharing the parent's entire context and mutable state. | Subagents need independent context and tools while the parent still needs a defined mechanism for delegation and result collection. | Create isolated child agents with their own state, model, memory, and tools, connected to the parent through typed streaming RPC. | contextcoordinationexecution |
| Cloudflare | Run agent-authored code with only the capabilities required by the current task. | A general-purpose execution environment gives generated code unnecessary ambient authority, while one fixed sandbox is inefficient for every task. | Escalate through workspace, isolated code, package, browser, and full-sandbox tiers while granting network and resource capabilities explicitly. | economicsexecution |
| Stripe | Measure whether coding agents can complete end-to-end API integration work rather than isolated code-generation tasks. | Function-level benchmarks can miss the planning, state management, browser use, database work, and glue code required by real integrations. | Evaluate agents in complete coding environments with repositories, databases, scripts, test credentials, browsers, and real API interactions. | contextexecutionverification |
| Spotify | Run background coding agents with predictable behavior. | Greater execution flexibility increases unpredictable behavior. | Spotify constrains permissions, available binaries, and execution through sandboxing and surrounding infrastructure. | execution |
| Bun | Use a coding agent to fix bugs in the Bun repository. | An agent-authored fix needs evidence that it resolves the bug rather than merely changing code. | Bun's Claude Code bot opens pull requests with tests that fail against the earlier system-installed Bun and pass against the fixed debug build, then responds to review comments. | executionverification |
| Spotify | Make autonomous background coding runs more predictable. | Additional dynamically fetched context and tools increase dimensions of unpredictability. | Spotify limits tools and hooks, constrains Git and Bash operations, and exposes standardized verification. | contextexecutionverification |
| Shafin et al. | Use verification results to refine generated implementation code before the workflow finishes. | Generated implementations can fail executable tests even after requirements and design stages have completed. | Have a Tester agent create and execute tests, emit a test report, and allow the Developer agent up to three bug-fix cycles from that feedback. | coordinationexecutionverification |
| Spotify | Execute software migrations across a large repository fleet with a background coding agent. | Deterministic transformation declarations do not cover every migration, while repository targeting, pull-request lifecycle, review, and production merging still need coordinated infrastructure. | Spotify retained Fleet Management for repository targeting and pull-request lifecycle while replacing transformation declarations with natural-language instructions executed by a coding agent. | coordinationexecution |
| Spotify | Operate background coding agents predictably while agent and model components evolve. | Agent execution needs formatting, linting, diff evaluation, logs, traces, and the ability to change agent or model components without rebuilding the surrounding workflow. | Spotify built a pluggable CLI harness with MCP-based formatting and linting, LLM-as-judge diff evaluation, GCP logs, MLflow traces, and swappable agents and models. | executionobservabilityverification |
| Yin et al. | Decide whether richer multi-agent orchestration is worth its operational complexity. | Additional roles and coordination loops can increase reasoning steps, corrections, latency, and resource consumption without a proportional outcome gain. | Measure trajectory length and correction behavior alongside effectiveness so coordination overhead is visible when comparing agent architectures. | coordinationeconomicsexecution |
| Google Cloud | Scale multi-architecture rollout without destabilizing production services. | A migrated job may crash-loop or exhibit unacceptable throughput after rollout even if repository checks passed. | Use automated continuous health monitoring to detect unhealthy Arm jobs and evict them for later offline tuning and debugging. | executionobservabilityverification |
| Google Cloud | Migrate tens of thousands of applications when direct coordination with every application owner is impractical. | The coordination cost of scheduling and executing owner-by-owner migrations becomes prohibitive at warehouse scale. | Use programmatic change generation, automated verification and rollout systems, and AI repair agents to minimize required application-team involvement, escalating difficult cases to owners. | coordinationeconomicsexecution |
| Google Cloud | Repair build and test failures encountered while migrating applications across instruction-set architectures. | A migration can reach repository-specific build or test failures that do not lend themselves to a simple mechanical edit. | Invoke an AI migration agent when a build or test fails and let it iteratively edit files, execute the failing check, and continue until verification succeeds or the agent gives up. | contextexecutionverification |
| Google Cloud | Separate repair strategies for build failures and test failures while preserving one migration workflow. | Build and test failures require different repair loops, and a test repair may itself encounter a build failure. | Use an outer orchestrator that repeatedly calls specialized build-fixer and test-fixer agents, with the test-fixer able to invoke the build-fixer as needed. | coordinationexecutionverification |
| Google Cloud | Allow an agent to adapt its next repair action to the result of the previous engineering command. | A repair loop needs current build and test evidence rather than relying only on the initial task description. | Execute each selected tool and attach its output back into the agent context before the next reasoning and tool-use step. | contextexecutionverification |
| Google Cloud / DORA | Absorb higher change volume produced by AI-assisted development without losing delivery stability. | Faster code production can expose downstream weaknesses and increase instability when feedback is slow or controls are weak. | Use strong automated testing and fast feedback loops as control systems around AI-assisted change. | executionverification |
| Google Cloud / DORA | Keep rapidly produced changes recoverable when AI-assisted work introduces regressions. | Higher change velocity increases the need to isolate, inspect, and reverse unsafe changes. | Maintain mature version-control practices as a safety net for accelerated AI-assisted development. | executionverification |
| Google Cloud / DORA | Scale AI-assisted engineering across teams with shared capabilities and guardrails. | Local AI productivity does not automatically translate into reliable organizational outcomes when teams lack a strong execution platform. | Invest in a high-quality internal platform that provides shared engineering capabilities and supports reliable AI-assisted workflows. | executionverification |
| OpenAI | Make delegated agent execution resemble the repository's real engineering environment. | An agent can produce misleading results when dependencies, tooling, or environment setup differ from the environment in which the code is normally built and tested. | Configure the agent sandbox with the repository's dependencies, tooling, and setup so execution matches the real development environment as closely as practical. | contextexecutionverification |
| OpenAI | Delegate repository changes to a background coding agent without sharing mutable execution state across tasks. | Concurrent coding tasks can interfere with one another or with a developer's local workspace when they share execution state. | Run each delegated coding task independently in its own isolated cloud environment preloaded with the target repository. | contextexecution |
| Cursor | Delegate bounded engineering work that can return as a reviewable artifact. | Investigations, small fixes, and medium-sized changes consume developer attention even when the work can be performed asynchronously. | Use background agents for small fixes, investigations, and first drafts of medium-sized pull requests. | execution |
| Cursor | Execute several software-engineering tasks without serially occupying one foreground coding session. | A single interactive agent session limits how much independent engineering work can proceed at once. | Run multiple coding agents in parallel, with each agent operating in its own remote environment. | coordinationexecution |
| Airbnb | Finish residual migration work after automation stops being economical. | Some files remained failing after 50-100 retries despite expanded context and repeated tuning. | Stop further automation tuning for the final long tail and complete those files manually using the failing automated refactor as a baseline. | economicsexecutionverification |
| Airbnb | Recover automatically when an LLM-generated migration step fails validation. | A first attempt frequently leaves validation errors that require another targeted correction. | Retry failed steps with dynamic prompts containing the validation errors and latest file version, up to a configurable limit. | executionlearningverification |
| Airbnb | Migrate thousands of tests while preserving test intent and code coverage. | An LLM-generated refactor cannot be treated as complete until repository-specific validation succeeds. | Break migration into per-file refactor and validation states and advance a file only after the preceding check passes. | executionobservabilityverification |