Evidence-backed practices
Emerging Practices
Evidence-backed engineering practices emerging across public engineering organizations, shown with the use case, problem encountered, reported response, and Selection conditions behind each observation.
47Evidence
47Assessed
0Pending
165Observations
100%Coverage
Assessment coverage measures evidence records explicitly assessed for emerging practices. Assessed evidence may legitimately contain zero practice observations.
50 evidence-backed practice observations · observability
| Company | Specific use case being solved | Problem encountered | Reported practice | Selection condition |
|---|---|---|---|---|
| Anthropic | Observe subtle operational behavior across independently scalable source-control components. | Fixed alerts can miss novel or weak signals in component telemetry and logs. | Anthropic assigns persistent superfan agents to individual components to inspect telemetry and logs alongside conventional alerting. | coordinationobservability |
| Atlassian | Expand an agent tool surface without allowing tool and context metadata to consume the task budget. | Adding products and tools can increase prompt overhead before an agent begins the requested work. | Atlassian benchmarks matched Jira and Confluence work across MCP versions and redesigns context delivery to reduce token use while expanding the available tool surface. | contexteconomicsobservability |
| Atlassian | Operate agent access across many enterprise products and external AI clients. | Distributed integrations make it difficult to enforce data policy, constrain connectivity, and reconstruct agent actions. | Atlassian centralizes domain, permission-scope, IP, data-protection, and data-loss-prevention controls with organization-wide audit logging. | coordinationobservabilityverification |
| Atlassian | Allow agents and teammates to edit shared organizational documents concurrently. | An agent can overwrite a teammate's newer change when it acts on stale document state. | Atlassian uses version-aware, section-level edits that pause on concurrent changes and expose a preview before publication. | coordinationobservabilityverification |
| GitHub | Increase automated repository throughput while retaining organizational control. | Higher agent activity can outrun governance if throughput replaces review, protection, auditability, or approval mechanisms. | GitHub treats branch protection, required review, audit logs, repository visibility, and human approval as constraints the new architecture must preserve. | coordinationobservabilityverification |
| GitHub | Maintain heavily written repositories without slowing live pushes and fetches. | Compaction and garbage collection compete with request handling when they execute on the same hosts. | GitHub moves compaction and garbage collection to separate workers operating directly against durable storage. | executionobservability |
| Astekin et al. | Choose an agent configuration when outcome quality, response time, and resource use all matter. | Ranking only by task accuracy can select a configuration whose small benefit carries disproportionate latency and energy costs. | Compare configurations on a joint accuracy, latency, and energy Pareto front rather than optimizing one metric in isolation. | economicsobservabilityverification |
| Cursor | Distinguish a production regression from expected effects after a deployment. | Operators must connect a changed signal to one of several deployed changes and decide whether the movement is intentional. | Follow the change through deployment and compare its planned signals with a pre-deploy baseline to identify localized or unintended regressions. | contextobservabilityverification |
| Cursor | Define how a particular code change will be verified after deployment. | Generic dashboards and alerts may not cover the risks, intended effects, or instrumentation gaps introduced by a specific diff. | Read the pull-request diff before merge, draft an editable plan of expected effects and risk signals, and flag where telemetry cannot determine success. | contextobservabilityverification |
| Will Larson | Prevent an agent from executing project work without a defined direction or observable outcome. | A task backlog can appear actionable even when the project lacks an explicit goal or a way to measure progress. | Start each project loop by auditing the goal definition and measurement tools, and iterate with the operator to create missing prerequisites. | contextobservability |
| Will Larson | Give an agent enough project state to reason beyond one assigned task. | Goals, measures, work state, and implementation context held in private human knowledge cannot regulate an automated project loop. | Represent goals and approach in a Notion RFC, measures in Datadog or Snowflake, and shared work state in a Linear project that the agent can audit. | contextcoordinationobservability |
| Will Larson | Advance a broad engineering goal across a changing set of tasks. | A task-scoped agent cannot identify missing work, reconcile changed issue state, or choose the next non-blocked action. | Review metrics and issues, create newly identified tasks, update state, execute non-blocked work, and select the next action from current project state. | coordinationexecutionobservability |
| Will Larson | Keep agent execution aligned when project understanding or operating results change over time. | Continuing to drain an old task list can optimize against a stale description after the project or its environment has changed. | Continue to the next task only when the project description is fresh; otherwise restart the loop from goal and measurement assessment. | contextlearningobservability |
| Atlassian | Choose how a coding agent should retrieve structured engineering context. | Equivalent knowledge can produce different end-to-end outcomes when invocation, instructions, and output formats differ. | Hold the agent, model, tasks, and questions constant; vary only the context-delivery path; then record pass rate, task time, tokens, tool calls, and transcripts. | contexteconomicsobservabilityverification |
| Cursor | Act on new pull requests, CI failures, bug reports, schedules, or Slack signals without waiting for a new prompt. | Prompt-driven agents stop observing the environment after an individual task completes. | Subscribe the coordinator to schedules and external events so it can initiate recurring maintenance and follow-up work. | executionobservability |
| OpenAI | Parallelize complex work across independent subproblems without implementing a custom coordination layer. | Applications otherwise need to build and observe delegation, separate subagent contexts, parallel execution, and result aggregation themselves. | Delegate independent work to parallel subagents with separate contexts while a main agent coordinates and combines their results. | coordinationexecutionobservability |
| Teresa Torres | Turn failures observed in real AI usage into regression protection. | Production failures remain isolated incidents unless they become reusable evaluation criteria. | Review traces, log and categorize recurring mistakes, and turn those failure categories into eval candidates. | learningobservabilityverification |
| Uber | Find expensive behavioral patterns in live agent sessions. | Repeated unnecessary tool use and other poor session behaviors can silently consume compute. | Uber uses live cost counters and a session-analysis dashboard that identifies 16 anti-pattern classes with financial impact and targeted remediation. | economicslearningobservability |
| Destefanis and Aste | Understand how a multi-agent coding run coordinates beneath its final output. | Pass rates and aggregate tokens hide which agents communicated, which files carried shared state, when coordination happened, and where it failed. | Represent agents and files as nodes and log messages, reads, and writes as timestamped, costed edges linked to the run's verified outcome. | coordinationobservabilityverification |
| Destefanis and Aste | Evaluate multi-agent configurations whose tasks leave teams freedom to self-organize. | Identical configurations under a pinned model can produce substantially different coordination networks across sessions. | Run each configuration repeatedly, compare matched sessions, and report distributions or cross-session ranges rather than treating one run as representative. | observabilityverification |
| Destefanis and Aste | Evaluate coding agents against hidden tests or reference solutions. | Agents may search outside their intended workspace for grading material, contaminating scores and revealing weak containment. | Remove access to hidden evaluation artifacts, substitute detectable placeholders, and repeat key conditions under sealed execution. | executionobservabilityverification |
| Destefanis and Aste | Introduce a coordinator or orchestrator into a multi-agent workflow. | Naming an agent as coordinator in its prompt may create no hub, authority, routing behavior, or reliable outcome improvement. | Inspect the interaction graph and comparative outcomes to verify that the proposed coordinator actually performs a structural integration role. | coordinationobservabilityverification |
| Cloudflare | Help agents find resources and help resource owners understand whether agents can discover them. | Human-oriented search and visibility mechanisms do not directly expose what agents can find or how visible a resource is to them. | Cloudflare presents AI Search for agent-oriented retrieval and AEO for measuring visibility to models and agents. | contextobservability |
| Cloudflare | Let agents compensate resource owners directly for content, APIs, or excess resource consumption. | Advertising and seat-based business models do not map cleanly to software actors that consume resources without rendering human interfaces. | Cloudflare presents x402-compatible payments, agent wallets, monetization gateways, and receipts as transaction rails for agent interactions. | coordinationeconomicsobservability |
| Cloudflare | Let agents perform actions on websites without reverse-engineering the DOM. | DOM parsing, guessed form fields, and synthesized clicks are brittle interfaces for machine actors. | WebMCP lets sites register explicit tools and schemas that agents can invoke using the user's existing session and state. | coordinationexecutionobservability |
| Stripe | Let agents consume infrastructure while constraining runaway resource cost. | An autonomous workflow can overspend on a single model, database, hosting, or other provider. | Set service-specific spending limits and expose unified current and historical cost across providers. | economicsobservability |
| Spotify | Coordinate agent-generated migrations across thousands of repositories while directing scarce human attention. | At fleet scale, teams need to know which changes were created or merged and which require human intervention. | Fleetshift schedules and orchestrates Honk changes and tracks migration progress, pull-request state, and changes needing human attention. | coordinationobservability |
| Google Cloud | Run agent jobs that continue for hours or days and survive infrastructure or human interruptions. | A long-running agent cannot depend on one process or session remaining alive for the duration of the work. | Google Agent Executor uses durable event logs and snapshots to support execution, resumption, and distributed deployment. | executionobservability |
| OpenAI | Run more concurrent Codex coding tasks than an engineer can comfortably supervise as separate sessions. | Engineers lost track of work and debugged stalled agents beyond roughly three to five simultaneous sessions. | Symphony moves task assignment and workspace lifecycle into an always-on orchestrator where open tasks receive agents in isolated workspaces and humans review outcomes. | coordinationexecutionobservabilityverification |
| Cloudflare | Provide many engineering clients and agents with governed access to models and internal capabilities. | Direct client-to-model integrations make attribution, permissions, model changes and cost control difficult to manage consistently at organizational scale. | Route AI clients through a shared proxy Worker and AI Gateway so identity, permissions, model catalog configuration, usage attribution and routing can be changed centrally. | coordinationeconomicsexecutionobservability |
| Cloudflare | Increase the chance that agent-generated changes already fit repository and organizational constraints before human review. | Agent outputs remain plausible but locally wrong when context acquisition, execution and standards enforcement are disconnected. | Wire Backstage context, repository AGENTS.md, centralized tools and Codex-backed AI review into one engineering path; Cloudflare reports that the resulting first draft is now usually close enough to ship. | contextcoordinationeconomicsexecutionlearningobservabilityverification |
| Cloudflare | Evaluate whether documentation restructuring changes agent cost and task speed. | Standards conformance alone does not demonstrate that the environment actually improves agent efficiency. | Run the same agent architecture against multiple technical documentation sites and compare token use and time to the correct answer. | contexteconomicsobservability |
| Cloudflare | Assess whether a web environment exposes the information and capabilities agents need to operate efficiently. | Agent compatibility remains difficult to improve when discoverability, content access, permissions, and callable capabilities are implicit or unmeasured. | Score sites across Discoverability, Content, Bot Access Control, and Capabilities using machine-checkable standards. | contextexecutionobservability |
| Cloudflare | Measure how widely agent-friendly environmental traits are spreading across the Web. | Individual site assessments do not show whether agent-compatible traits are becoming common across the wider environment. | Scan 200,000 high-traffic domains and publish standards-adoption data in Cloudflare Radar with weekly updates and API access. | learningobservability |
| Cloudflare | Continue agent work across long conversations, interruptions, and session branches. | Long-running work exceeds model context windows and loses continuity when conversation state exists only in the active process. | Persist full tree-structured session history, compact context non-destructively, and expose searchable history plus updateable context blocks. | contextobservability |
| Meta | Keep repository context useful as code and dependencies change. | Stale context can misdirect coding agents and may be more harmful than missing context. | Periodically validate paths, detect coverage gaps, rerun quality critics, and automatically repair stale references. | contextlearningobservability |
| Meta | Prevent generated repository guidance from becoming a new source of agent error. | AI-generated context can be incorrect, stale, overly broad, or contain invalid references. | Run multiple rounds of independent critics, prompt tests, integration checks, and file-path validation before relying on generated context. | observabilityverification |
| Stripe | Determine whether an agent-generated integration actually works after the agent declares completion. | An agent can mistake a plausible intermediate result or expected-looking error for task success. | Use deterministic API and automated UI graders, including inspection of created service-side objects, to validate the final system independently of the agent's own completion judgment. | observabilityverification |
| Stripe | Improve an agent workflow after observing recurring execution or recovery failures. | Without replayable tasks, changes to prompts, skills, tools, or documentation cannot be compared against the same engineering problem. | Use replayable benchmark environments as a test bed for prompt, skill, browser-tool, and documentation interventions, then measure whether task performance changes. | learningobservabilityverification |
| Dae-Kyoo Kim | Evaluate whether a new specialization strategy improves an agentic engineering workflow. | Changing the model and orchestration at the same time makes it difficult to attribute outcome differences to agent configuration. | Hold the underlying LLM constant while comparing orchestration granularities so configuration effects can be evaluated more directly. | observabilityverification |
| Shafin et al. | Identify which parts of a multi-agent development process materially change correctness and code quality. | A complete multi-agent workflow can change outcomes without revealing which development activity caused the effect. | Remove Requirement, Design, or Testing one stage at a time and compare each configuration with the full workflow and RawPrompt baseline. | learningobservabilityverification |
| Spotify | Operate background coding agents predictably while agent and model components evolve. | Agent execution needs formatting, linting, diff evaluation, logs, traces, and the ability to change agent or model components without rebuilding the surrounding workflow. | Spotify built a pluggable CLI harness with MCP-based formatting and linting, LLM-as-judge diff evaluation, GCP logs, MLflow traces, and swappable agents and models. | executionobservabilityverification |
| Yin et al. | Identify where an agent architecture spends its inference budget. | Aggregate token totals do not reveal whether cost is concentrated in planning, reflection, execution, or editing. | Break token consumption down by workflow stage to expose architectural cost patterns and optimization opportunities. | economicsobservability |
| Yin et al. | Compare software-engineering agent frameworks before selecting one for repeated engineering work. | Task success alone can hide long trajectories, repeated corrections, or prohibitive token cost. | Evaluate agent frameworks on effectiveness, execution efficiency, and token or monetary overhead rather than ranking them only by final-task success. | economicsobservabilityverification |
| Google Cloud | Scale multi-architecture rollout without destabilizing production services. | A migrated job may crash-loop or exhibit unacceptable throughput after rollout even if repository checks passed. | Use automated continuous health monitoring to detect unhealthy Arm jobs and evict them for later offline tuning and debugging. | executionobservabilityverification |
| Google Cloud | Evaluate a migration agent against known real repository fixes before relying on wider production use. | Synthetic coding tasks may not represent the failure modes encountered in a warehouse-scale migration. | Select historical non-AI migration commits that can be cleanly rolled back, recreate the pre-fix state, and measure whether the agent can repair the same failures. | observabilityverification |
| OpenAI | Review background agent work without trusting only the agent's natural-language summary. | A reviewer needs evidence that the claimed edits and checks actually occurred before accepting an asynchronously produced change. | Return citations to terminal logs and test outputs alongside the generated change so reviewers can trace and validate the agent's actions. | observabilityverification |
| Cursor | Keep humans in control of remotely executing coding-agent tasks. | A long-running remote task needs supervision and a way to recover when the agent needs correction or human intervention. | Expose per-agent status and allow the user to send follow-up instructions or take over execution at any point. | coordinationobservability |
| Airbnb | Improve completion across the remaining long tail without tuning one file at a time. | Repeated failures clustered around common issues that required changes to prompts or migration tooling. | Sample representative failures, tune prompts and scripts, verify the change on the sample, then sweep the remaining corpus again. | learningobservabilityverification |
| Airbnb | Migrate thousands of tests while preserving test intent and code coverage. | An LLM-generated refactor cannot be treated as complete until repository-specific validation succeeds. | Break migration into per-file refactor and validation states and advance a file only after the preceding check passes. | executionobservabilityverification |