Seven agent frameworks showed distinct effectiveness, coordination, and cost trade-offs across three software-engineering task families
Source → Observed → Interpretation → Model implication
A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Tasks
View source →The study evaluates seven general-purpose agent frameworks selected because they can be applied across all three studied tasks and are publicly available: AgentOrchestra, OWL, SE-Agent, Trae, GPTswarm, OpenHands, and SWE-Agent.
The three task families are evaluated with established public benchmarks. The benchmark datasets described by the paper contain 1,200 software-development prompts in SRDD, 115 labeled vulnerability instances in LLM-SmartAudit, and 300 program-repair instances in SWE-bench Lite.
The authors evaluate agent behavior along three dimensions: effectiveness of the final result, efficiency of the execution trajectory, and overhead measured by token consumption and associated monetary cost.
The paper characterizes overall agent effectiveness as moderate rather than uniformly high. OpenHands shows the most balanced software-development quality, GPTswarm reaches 77% accuracy on the vulnerability-detection evaluation, and only a subset of evaluated agents repair about half of the program-repair issues.
Execution efficiency varies substantially across frameworks and tasks. SE-Agent with three iterations exhibits especially long step sequences, while AgentOrchestra exhibits the longest correction sequences. The authors associate AgentOrchestra's long trajectories with coordination overhead.
The study reports that a lighter SE-Agent configuration with one iteration can perform competitively in software development and concludes that efficiency depends less on agent quantity than on reasoning depth, coordination strategy, and feedback integration.
Resource use also differs by task and framework. Software development is the most expensive task family in monetary terms, AgentOrchestra is the most resource-intensive configuration in the reported comparison, and GPTswarm is reported as the most cost-efficient framework overall.
Stage-level token analysis shows different cost shapes for different agent architectures. Multi-agent workflows spend a larger share on planning and reflection, while single-agent workflows concentrate more of their cost on execution and editing stages.
The authors make the experimental results publicly available and frame the study as a comparison of practical framework trade-offs rather than a claim that one architecture dominates across all software-engineering tasks.
The study supports Apparition because multiple agent frameworks repeatedly perform code-centric tasks through iterative reasoning, tool use, execution, and feedback rather than acting as single-pass text generators. Its stronger contribution is to Selection and Cooperation. Framework fitness varies across task type, execution trajectory, correction behavior, and token cost, so useful operation cannot be inferred from agent capability alone. The comparison also sharpens Cooperation: additional agents and richer orchestration do not automatically improve fitness. Coordination itself consumes execution steps, reasoning cycles, and tokens, and a lighter configuration can sometimes rival a more elaborate one. Cooperation is therefore better treated as an engineering trade-off whose benefit must exceed its coordination and verification cost, not as a monotonic advantage of adding agents.
REFINES. The source supports C01 by documenting meaningful automated software-engineering task execution across multiple agent frameworks. It refines C02 because the frameworks exhibit different effectiveness, efficiency, and cost profiles across tasks, making viability dependent on execution, verification, feedback, and economic conditions. It refines C03 because the comparative results show that multi-agent or heavier orchestration can incur material coordination overhead and that agent quantity alone does not determine efficiency. The evidence is consistent with cooperation being useful only when the gain from differentiated reasoning or feedback exceeds the additional coordination cost.
What this does not establish
- The paper is an arXiv preprint and the evidence record does not establish that the reported results have undergone independent peer review or replication.
- The benchmark evaluations are not sustained production deployments, so the source does not establish long-term reliability, maintainability, user impact, or organizational adoption of the evaluated frameworks.
- The seven frameworks differ in architecture, prompts, tooling, reasoning strategy, and other implementation details. The comparison therefore does not isolate agent count or cooperation as the sole causal explanation for performance differences.
- The three task families use different benchmarks and success metrics, so results should not be interpreted as directly interchangeable measures of one universal notion of software-engineering fitness.
- Token consumption and monetary cost depend on model choice, caching, pricing, and experimental configuration; the reported cost ranking is not a timeless economic ordering of the frameworks.
- The study does not establish that any multi-agent framework is consistently fitter than a useful single-agent alternative after controlling for model, compute, prompts, tools, and task distribution.
- The source does not establish persistent differentiated cooperation that forms a higher-order engineering object under C04.
Under matched models, tools, task distributions, and compute budgets, when does the additional success from specialized or multi-agent cooperation exceed the extra coordination, verification, latency, and token cost it introduces?