Evidence-backed practices
Emerging Practices
Evidence-backed engineering practices emerging across public engineering organizations, shown with the use case, problem encountered, reported response, and Selection conditions behind each observation.
47Evidence
47Assessed
0Pending
165Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for emerging practices. Assessed evidence may legitimately contain zero practice observations.
48 evidence-backed practice observations · economics
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Anthropic | Add and optimize history, blame, search, and graph capabilities without changing clients or materializing every index eagerly. | Precomputing all repository metadata increases write amplification and couples new query capabilities to client formats. | Anthropic defines recursive derived-data types that can be computed eagerly or lazily and optimized behind a stable service interface. | economicsexecution |
| Anthropic | Serve ephemeral agents and CI jobs operating continuously against a very large monorepo. | Complete local repositories and offline-capable clients transfer and maintain data that online automated clients rarely need. | Anthropic moves history, diff, blame, commit, merge, and rebase operations to a server while clients retain bounded caches and fetch only requested data. | contexteconomicsexecution |
| Anthropic | Let CI, agents, and internal tools query repository state without checking out or maintaining a repository. | Every consumer independently cloning and maintaining repository data multiplies compute, storage, transfer, and operational work. | Scion exposes diff, ancestry, blame, merge-base, blob, and related operations through horizontally scalable APIs. | contexteconomicsexecution |
| Anthropic | Accept very high rates of concurrent changes without making every commit contend on global repository state. | Packfile processing, locking, and shared-reference movement place unrelated work on the same coordination-critical path. | Scion creates incremental server-side changesets without packfiles or locks and separates changeset creation from the small number of supported reference updates. | coordinationeconomicsexecution |
| Atlassian | Expand an agent tool surface without allowing tool and context metadata to consume the task budget. | Adding products and tools can increase prompt overhead before an agent begins the requested work. | Atlassian benchmarks matched Jira and Confluence work across MCP versions and redesigns context delivery to reduce token use while expanding the available tool surface. | contexteconomicsobservability |
| GitHub | Scale repository reads and writes independently under bursty agent and CI workloads. | Using complete local repository copies for both durability and serving means that adding read replicas also adds overhead to every write. | GitHub separates authoritative repository data in durable object storage from elastic compute workers that cache and serve requests. | economicsexecution |
| Astekin et al. | Decide whether an engineering task needs non-agentic, single-agent, or multi-agent execution. | Selecting a complex agent architecture without a lightweight baseline can hide whether cooperation contributes any task-specific benefit. | Evaluate an explicit ladder from direct model query through single-, dual-, and multi-agent configurations under matched task conditions. | coordinationeconomicsverification |
| Astekin et al. | Introduce multi-agent cooperation for a bounded software-engineering workflow. | Additional agents impose systematic execution costs while accuracy gains vary by task. | Begin with a lightweight task-aware baseline and adopt multi-agent workflows only when comparative evaluation demonstrates a benefit that justifies the added cost. | coordinationeconomicsverification |
| Astekin et al. | Deploy heavier agent workflows whose latency and energy depend on the execution platform. | A configuration that appears viable on high-end server hardware may be too slow or inefficient on a developer workstation. | Measure the same agent configurations across representative hardware and include runtime environment in the selection decision. | economicsexecution |
| Astekin et al. | Choose an agent configuration when outcome quality, response time, and resource use all matter. | Ranking only by task accuracy can select a configuration whose small benefit carries disproportionate latency and energy costs. | Compare configurations on a joint accuracy, latency, and energy Pareto front rather than optimizing one metric in isolation. | economicsobservabilityverification |
| Cursor | Preserve source-location guidance while reducing context added by large file reads. | Numbering every source line adds substantial repeated context when agents read tens of thousands of lines. | Include line numbers at a sparser interval that remains sufficient for model citations, then verify quality before rollout. | contexteconomics |
| Cursor | Delegate work when fresh subagent context saves more than coordination costs. | Isolated subagents can reduce inherited context but may duplicate work or pursue tasks invalidated by the parent's progress. | Avoid blanket instructions to delegate exploration and constrain alternate-model selection to cases directed by the user or harness. | contextcoordinationeconomics |
| Cursor | Keep a broad tool set available without placing every definition in every model request. | Most specialized tools are used in a minority of conversations, but static definitions consume context throughout every turn. | Keep high-frequency and flow-critical tools in static context and load lower-frequency MCP and built-in tool definitions only when needed. | contexteconomicsexecution |
| Cursor | Optimize context construction for the real distribution of agent tasks. | Offline evals can overrepresent hard tasks and miss the token, latency, and behavior effects seen in ordinary production traffic. | A/B test context configurations on production traffic while tracking token use, cost, latency, tool-call errors, agent usage, and quality. | contexteconomics |
| Cursor | Avoid repeatedly sending harness instructions that current models no longer need. | Legacy behavioral guidance remains in every turn after models learn to follow simpler tool definitions. | Re-test system instructions across model families and remove directions whose absence does not degrade production behavior. | contexteconomics |
| Cursor | Reuse unchanged prompt prefixes as an agent conversation grows. | Variable setup mixed with stable instructions and tools prevents effective prefix-cache reuse across turns. | Place explicit cache breakpoints after stable request layers and move user- and request-specific setup behind those boundaries. | contexteconomics |
| Atlassian | Retrieve several related components, tokens, or guidance records during one agent task. | Agents made repeated context round trips, accumulating avoidable lookup latency. | Add a batch command and improve fuzzy search so agents can consolidate related context queries. | contexteconomicsexecution |
| Atlassian | Choose how a coding agent should retrieve structured engineering context. | Equivalent knowledge can produce different end-to-end outcomes when invocation, instructions, and output formats differ. | Hold the agent, model, tasks, and questions constant; vary only the context-delivery path; then record pass rate, task time, tokens, tool calls, and transcripts. | contexteconomicsobservabilityverification |
| Atlassian | Serve context repeatedly during an agent task without paying process or package startup costs on every lookup. | An npx-based command added more than 15 seconds per query and compounded when agents performed multiple lookups. | Provide a pre-installed CLI path for the default environment while retaining the portable npx invocation as a fallback. | economicsexecution |
| Cursor | Choose whether a task should use multi-agent orchestration or a plain agent. | Coordination ceremony can cost more than it saves on work small enough for one agent. | The Orchestrate playbook says not to orchestrate when one agent can finish inside the task budget and uses a task-scale threshold for orchestration. | coordinationeconomics |
| Teresa Torres | Run judgment-based evaluation without paying its cost for every case. | LLM-as-a-Judge evaluations are more expensive than deterministic assertions. | Use a cheap code assertion first and send only threshold-crossing cases to an LLM judge. | economicsverification |
| Uber | Reduce the cost of high-volume agentic engineering work. | Using an expensive primary model for every bounded subtask makes scaled execution more costly. | A primary model decomposes and evaluates work while lower-cost subagents execute bounded tasks. | coordinationeconomics |
| Uber | Determine whether agent automation is economically efficient. | Token or request spend alone does not show how much useful engineering work was delivered. | Uber tracks outcome-denominated cost together with workload-specific quality signals and delivered volume. | economicsverification |
| Uber | Find expensive behavioral patterns in live agent sessions. | Repeated unnecessary tool use and other poor session behaviors can silently consume compute. | Uber uses live cost counters and a session-analysis dashboard that identifies 16 anti-pattern classes with financial impact and targeted remediation. | economicslearningobservability |
| Uber | Choose models for specific managed software-engineering agent workloads. | General model benchmarks do not establish the best quality, reliability, and economics for a particular production workload. | Uber builds benchmarks from each managed agent's real work and evaluates cost per completed task, output quality, and reliability. | economicsverification |
| Destefanis and Aste | Choose between direct messaging and persistent shared artifacts for an agent team. | One-to-one messages repeat shared information, while mandatory files add overhead when the work already communicates through file interfaces. | Measure the task's emergent coordination network and require shared files only where they replace message-heavy many-to-many traffic. | contextcoordinationeconomics |
| Cloudflare | Let agents compensate resource owners directly for content, APIs, or excess resource consumption. | Advertising and seat-based business models do not map cleanly to software actors that consume resources without rendering human interfaces. | Cloudflare presents x402-compatible payments, agent wallets, monetization gateways, and receipts as transaction rails for agent interactions. | coordinationeconomicsobservability |
| Cloudflare | Let agents consume web content without paying to parse human-oriented presentation markup. | HTML intended for humans consumes bandwidth, tokens, and context-window capacity that does not contribute to the agent's task. | Cloudflare presents Markdown for Agents as a server-side representation optimized for machine consumption. | contexteconomics |
| Stripe | Let agents consume infrastructure while constraining runaway resource cost. | An autonomous workflow can overspend on a single model, database, hosting, or other provider. | Set service-specific spending limits and expose unified current and historical cost across providers. | economicsobservability |
| Cloudflare | Provide many engineering clients and agents with governed access to models and internal capabilities. | Direct client-to-model integrations make attribution, permissions, model changes and cost control difficult to manage consistently at organizational scale. | Route AI clients through a shared proxy Worker and AI Gateway so identity, permissions, model catalog configuration, usage attribution and routing can be changed centrally. | coordinationeconomicsexecutionobservability |
| Cloudflare | Increase the chance that agent-generated changes already fit repository and organizational constraints before human review. | Agent outputs remain plausible but locally wrong when context acquisition, execution and standards enforcement are disconnected. | Wire Backstage context, repository AGENTS.md, centralized tools and Codex-backed AI review into one engineering path; Cloudflare reports that the resulting first draft is now usually close enough to ship. | contextcoordinationeconomicsexecutionlearningobservabilityverification |
| Cloudflare | Review every merge request on a standard CI path without applying the same review cost and expertise to every change. | Uniform generic review does not match the risk profile or expertise required by different engineering changes. | Classify merge requests by risk tier and delegate review to specialized agents for code quality, security, standards compliance, documentation, performance and release impact. | coordinationeconomicsverification |
| Cloudflare | Control inference cost while retaining stronger reasoning for high-risk engineering work. | Using the most capable model for every review task makes high-volume agentic workflows unnecessarily expensive. | Route documentation and lower-risk review workloads to lower-cost Workers AI models while reserving frontier models for security-sensitive and architecturally complex reviews. | economicsexecution |
| Cloudflare | Evaluate whether documentation restructuring changes agent cost and task speed. | Standards conformance alone does not demonstrate that the environment actually improves agent efficiency. | Run the same agent architecture against multiple technical documentation sites and compare token use and time to the correct answer. | contexteconomicsobservability |
| Cloudflare | Reduce the exploration required for agents to find precise technical documentation. | Large or HTML-heavy documentation forces iterative search, fragments context, and increases token and latency cost. | Restructure documentation into bounded agent-readable Markdown paths and context-sized product directories. | contexteconomics |
| Cloudflare | Keep long-running agent work recoverable without maintaining an always-on process for every agent. | Deploys, restarts, and idle periods can destroy in-memory work or make one-runtime-per-agent economics impractical. | Give each agent addressable persistent state and run checkpointable work that can recover after the runtime restarts and hibernate when idle. | economicsexecution |
| Cloudflare | Run agent-authored code with only the capabilities required by the current task. | A general-purpose execution environment gives generated code unnecessary ambient authority, while one fixed sandbox is inefficient for every task. | Escalate through workspace, isolated code, package, browser, and full-sandbox tiers while granting network and resource capabilities explicitly. | economicsexecution |
| Meta | Help coding agents locate the right files, commands, conventions, and dependencies before editing a proprietary codebase. | Without repository-specific tribal knowledge, agents repeatedly explore the codebase and can make plausible but subtly wrong changes. | Generate compact context files containing quick commands, key files, non-obvious patterns, and cross-references instead of exhaustive repository summaries. | contexteconomics |
| Meta | Answer cross-repository impact questions without repeated exploratory searches. | A change in one subsystem can propagate across repositories, making dependency discovery expensive and error-prone. | Pre-compute dependency indexes and data-flow maps so agents can retrieve affected relationships directly. | contexteconomics |
| Gloaguen et al. | Provide coding agents repository-specific instructions, workflow guidance, and tool information. | Agents need repository-specific knowledge, but extra context can be redundant and increase exploration and inference cost without improving task resolution. | The evaluated CTXbench repositories use developer-committed repository context files that agents read and follow during coding tasks. | contexteconomics |
| Lulla et al. | Provide coding agents repository-specific guidance while they execute pull-request tasks. | Agents otherwise need to infer repository-specific instructions during execution, which can increase completion time and generated output. | The evaluated repositories provide a version-controlled root AGENTS.md file as repository-level context for the coding agent. | contexteconomics |
| Yin et al. | Identify where an agent architecture spends its inference budget. | Aggregate token totals do not reveal whether cost is concentrated in planning, reflection, execution, or editing. | Break token consumption down by workflow stage to expose architectural cost patterns and optimization opportunities. | economicsobservability |
| Yin et al. | Compare software-engineering agent frameworks before selecting one for repeated engineering work. | Task success alone can hide long trajectories, repeated corrections, or prohibitive token cost. | Evaluate agent frameworks on effectiveness, execution efficiency, and token or monetary overhead rather than ranking them only by final-task success. | economicsobservabilityverification |
| Yin et al. | Decide whether richer multi-agent orchestration is worth its operational complexity. | Additional roles and coordination loops can increase reasoning steps, corrections, latency, and resource consumption without a proportional outcome gain. | Measure trajectory length and correction behavior alongside effectiveness so coordination overhead is visible when comparing agent architectures. | coordinationeconomicsexecution |
| Yin et al. | Choose between lightweight and more elaborate agent orchestration for a software-engineering task. | Agent quantity is an unreliable proxy for execution quality or efficiency. | Compare reasoning depth, coordination strategy, feedback integration, effectiveness, and cost instead of assuming that adding agents improves the workflow. | coordinationeconomicslearning |
| Google Cloud | Migrate tens of thousands of applications when direct coordination with every application owner is impractical. | The coordination cost of scheduling and executing owner-by-owner migrations becomes prohibitive at warehouse scale. | Use programmatic change generation, automated verification and rollout systems, and AI repair agents to minimize required application-team involvement, escalating difficult cases to owners. | coordinationeconomicsexecution |
| Airbnb | Finish residual migration work after automation stops being economical. | Some files remained failing after 50-100 retries despite expanded context and repeated tuning. | Stop further automation tuning for the final long tail and complete those files manually using the failing automated refactor as a baseline. | economicsexecutionverification |
| Airbnb | Migrate complex tests with project-specific setup, indirection, and conventions. | Retrying alone stopped being effective when the model lacked the repository context needed to resolve complex test behavior. | Expand prompts with selected source files, related tests, passing project-local examples, migration guidance, and validation failures, reaching roughly 40K-100K tokens and up to 50 related files. | contexteconomics |