Agentic development: what the real evidence says and what must change in your team before adopting it

In July 2025, METR published the study that became the favorite argument of every AI-in-programming skeptic: experienced developers took 19% longer to complete their tasks when they were allowed to use AI tools, even though they themselves estimated afterwards that they had gone 20% faster.

In February 2026, the same authors published something far more interesting: they are changing the design of the experiment because it no longer measures anything reliable. The main reason? A significant number of invited developers refused to take part if they had to work without AI.

That arc —from "AI makes you slower" to "we can no longer measure it because nobody wants to work without it"— is the best possible introduction to agentic development. Because it describes precisely where we are: adoption is massive, the evidence on productivity is ambiguous, and the variable that does predict outcomes is not the model you use, but the engineering system you put it into.

This article is about that.

What is agentic development, really?

Agentic development is the use of AI agents that run cycles of plan, act with tools, verify the result, and iterate, within a bounded budget and under human review. The difference from a copilot is not the quality of the generated code: it is that the agent can call tools, read the result of that call, and correct itself.

A copilot suggests. An agent runs a closed loop. That distinction changes where the risk sits.

With autocomplete, the worst case is a bad suggestion you discard in two seconds. With an agent, the worst case is a four-step chain where the error in step one propagates and the final result looks plausible. That is why the two pieces that really matter in an agentic system are not the prompt or the model, but the output validator and the step budget.

In practice, an agent shows up at four moments of the work: breaking a request into tasks, writing or modifying code, running tests or checks, and correcting after feedback. The only one of those four that most teams have not solved is the third. And without the third, the other three are text generation with extra steps.

Do agents make teams faster?

The honest answer is that nobody has measured it well yet, and you should be suspicious of anyone who claims otherwise in either direction.

METR's July 2025 study was the most rigorous attempt to date: a randomized controlled trial with 16 experienced developers across 246 real tasks in their own repositories, with an average of five years of experience on those projects. The result was that allowing AI use increased completion time by 19%, when the developers themselves had forecast a 24% reduction before starting. The confidence interval, however, was wide: between +2% and +39%.

What happened afterwards matters more than the headline. METR started a new experiment in August 2025 with a larger group of developers and more recent tools, and concluded that the data offered an unreliable signal of the current effect, mainly because a growing number of participants refused to join the study rather than work without AI. They also cut compensation from 150 to 50 dollars per hour, which introduced its own selection bias, and they found that time per task stops being measurable when someone operates several agents in parallel.

The revised numbers point in another direction, though with enormous uncertainty: for the subset of original developers who took part again, the estimate moved to an 18% speedup with an interval between -38% and +9%, while among newly recruited participants the estimate was 4%, with an interval between -15% and +9%. Intervals that cross zero are not evidence of anything conclusive.

There is one figure in all of this that is solid and that almost nobody cites: the gap between perception and measurement. The developers in the original study believed they had sped up by 20% while the stopwatch said the opposite. That means any claim about productivity based on internal surveys —"the team says it's going faster"— does not count as evidence. If you are going to justify an investment in agentic tooling, you need to measure cycle time and production failure rate, not feelings.

Why do agents amplify a team's problems instead of solving them?

Because they do not change the system, they accelerate it. The 2025 DORA report on AI-assisted development, based on nearly 5,000 professionals, sums up its central finding in one sentence: AI does not fix a team, it amplifies what is already there.

The technical detail is more useful than the metaphor. DORA found that greater AI adoption is associated simultaneously with more delivery throughput and more delivery instability, and that the time saved in code creation is frequently reallocated to auditing and verification. In other words: the bottleneck does not disappear, it moves downstream. It goes from "writing" to "reviewing".

That has a concrete operational consequence that is rarely planned for. If your team generates three times as many pull requests and keeps the same review capacity, you have not multiplied speed: you have created a queue. And a saturated review queue is exactly the environment where mediocre code gets approved.

DORA's conclusion is that the greatest return does not come from the tools, but from the quality of internal platforms, the clarity of workflows, and team alignment. Translated into an architecture decision: before asking which agent to adopt, ask whether your repository has a test suite you trust enough to let something automated use it as the source of truth.

If the answer is no, the agent is not your next investment. The tests are.

I have seen this pattern on my own site, without needing a large team. Publishing an article here means touching four or five files: the page, the entry in articles.ts, the JSON-LD, llms.txt, the category filter. Delegating that integration to an agent works well and takes minutes. The problem is that the result always looks finished: the page compiles, the build passes, the site renders. The failures I have caught while reviewing —a slug registered in the wrong array, a schema field copied from another article without adjusting it— are not detected by any test I have today. The generator saves me the mechanical work. I am still the verifier, and that is where the real time goes.

It is the same constraint I apply when building AI agents for clients: the ceiling is not set by what the agent is capable of generating, but by what can be verified before it reaches production.

What does a controlled agentic loop look like in Next.js?

A minimal but real agentic loop has three elements that a toy example does not: a validated output contract, a tool registry with typed arguments, and a step budget. Without all three, it is not an agent: it is a model call inside a loop.

First, the contract. The model does not return free text: it returns one of three possible shapes, and anything else is rejected before executing anything.

import { z } from "zod";

export const DecisionSchema = z.discriminatedUnion("type", [
  z.object({
    type: z.literal("use_tool"),
    tool: z.enum(["check_inventory", "calculate_shipping"]),
    args: z.record(z.string(), z.unknown()),
    reason: z.string().max(240),
  }),
  z.object({
    type: z.literal("answer"),
    answer: z.string().min(1),
    confidence: z.number().min(0).max(1),
  }),
  z.object({
    type: z.literal("give_up"),
    reason: z.string().min(1),
  }),
]);

export const TOOLS = {
  check_inventory: {
    input: z.object({ sku: z.string().regex(/^[A-Z]{3}-\d{4}$/) }),
    run: async ({ sku }: { sku: string }) => {
      // The real database query would go here.
      return { sku, available: 12, warehouse: "CCS-01" };
    },
  },
  calculate_shipping: {
    input: z.object({ sku: z.string(), destination: z.string().min(2) }),
    run: async ({ destination }: { sku: string; destination: string }) => {
      return { destination, days: 3, cost_usd: 8.5 };
    },
  },
} as const;

Notice the detail almost everyone skips: the tool's arguments are validated separately from the general contract. The model saying "I want to use check_inventory" does not mean the SKU it proposes is valid. That second validation is what stops a hallucination from reaching your database.

Now the loop. Note that there is no infinite recursion and no "the agent decides when to stop": there is an explicit budget.

import { NextResponse } from "next/server";
import { DecisionSchema, TOOLS } from "./_contract";

const MAX_STEPS = 4;

export async function POST(request: Request) {
  const { task } = await request.json();
  if (typeof task !== "string" || task.length < 3) {
    return NextResponse.json({ error: "Invalid task" }, { status: 400 });
  }

  const trace: unknown[] = [];
  const observations: string[] = [];

  for (let step = 0; step < MAX_STEPS; step++) {
    const raw = await askForDecision(task, observations);
    const decision = DecisionSchema.safeParse(raw);

    // The contract fails: nothing is executed, the error goes back to the model.
    if (!decision.success) {
      observations.push(`Output does not match the schema. Fix it and retry.`);
      trace.push({ step, result: "invalid_schema" });
      continue;
    }

    const d = decision.data;

    if (d.type === "answer") {
      trace.push({ step, result: "answer", confidence: d.confidence });
      return NextResponse.json({ status: "ok", answer: d.answer, trace });
    }

    if (d.type === "give_up") {
      trace.push({ step, result: "gave_up" });
      return NextResponse.json({ status: "no_solution", reason: d.reason, trace });
    }

    // Second validation: the concrete arguments for this tool.
    const tool = TOOLS[d.tool];
    const args = tool.input.safeParse(d.args);

    if (!args.success) {
      observations.push(`Invalid arguments for ${d.tool}.`);
      trace.push({ step, result: "invalid_arguments" });
      continue;
    }

    const output = await tool.run(args.data as never);
    observations.push(`${d.tool} returned: ${JSON.stringify(output)}`);
    trace.push({ step, result: "tool", tool: d.tool });
  }

  // The budget ran out without a conclusion: that is also a valid result.
  return NextResponse.json({ status: "budget_exhausted", trace }, { status: 200 });
}

What matters in this example is not the code itself, but three design decisions:

1. The agent never executes an action that has not passed two validations. The general schema and the specific arguments.
2. Exhausting the budget is a legitimate state, not an error. An agent that always returns something is an agent that sometimes makes things up.
3. The trace is always returned. Without a trace there is no debugging, and an agentic system without observability is a black box writing to your production.

What makes a repository suitable for agents?

There is a concrete standard that answers half the question. AGENTS.md is an open format designed as a README for agents: a predictable place to leave the context, the build steps, the tests, and the conventions an agent needs and that would clutter a README meant for humans. Since its publication in August 2025 it has been adopted by more than 60,000 open source projects and by frameworks such as Codex, Cursor, Gemini CLI, GitHub Copilot and Jules, and the format was donated to the Agentic AI Foundation under the Linux Foundation. Claude Code reads CLAUDE.md instead, and the usual solution is to import AGENTS.md from there.

That such a simple format became a standard says something: the real problem was not the model's capability, it was that nobody was telling it how the project gets built.

The other half of the answer has no standard and depends on you. A repository suitable for agents needs:

• Deterministic commands. An npm test that sometimes fails due to flakiness turns the verifier into noise.
• Fast tests. If the suite takes 20 minutes, the agent cannot iterate; it can only guess.
• Strict typing. TypeScript in strict mode is the cheapest error detector you can hand an agent.
• Seeded test data. An agent that cannot bring up the environment cannot verify anything.
• An explicit human review budget. If you generate more PRs than you can review, you have made the process worse, not better.

None of those five points is new. They are all engineering best practices from fifteen years ago. That is exactly DORA's conclusion: AI does not reward whoever adopts it first, it rewards whoever already had their house in order.

What should a team do before adopting agentic development?

Three things, in this order.

First, measure the current state honestly. Cycle time, production failure rate, average PR review time. If you do not have that baseline before introducing agents, you will never know whether they worked — and we already know that the team's perception does not count as evidence.

Second, fix the verifier before the generator. Tests, types, CI. An agent without a reliable source of truth produces code that looks correct, and "looks correct" is the most expensive category of bug there is.

Third, narrow the scope. Agents perform where the work is structured: repetitive migrations, test coverage, mechanical refactors, CRUD scaffolding — much of the mechanical work in a custom management system falls right there. They perform poorly where the problem is ambiguous or the context lives in someone's head. Starting with the latter is the fastest way to conclude that "this doesn't work".

Agentic development does not replace engineers. It redistributes the work: less writing, more defining boundaries and more reviewing. The teams that come out ahead will not be the ones that automate the most, but the ones that have built the verification system that makes that automation safe.

Frequently asked questions

What is agentic development?

Agentic development is the use of AI agents that run cycles of plan, act with tools, verify the result, and iterate, within a bounded step budget and under human review. It differs from assisted code generation in that the agent does not just produce text: it calls real tools — running the tests, reading files, querying an API —, reads the result of that call, and corrects itself with it. Without that verification phase it is not agentic development, it is autocomplete inside a loop.

What is the difference between an AI agent and a chatbot?

A chatbot holds a conversation and returns text: everything it produces still has to be executed by a person. An AI agent has access to tools and acts on real systems — writing a file, running a query, executing a command —, chaining several of those steps and reading the result of each one. That is why the risk is different: the worst output of a chatbot is a wrong answer you discard; the worst output of an agent is a wrong action already executed. That is the reason an agent needs automatic validation of its outputs and an explicit iteration limit, and a chatbot does not.

Do AI agents really increase developer productivity?

It depends on the context, and today there is no clean measurement that supports a universal claim. METR's controlled trial (July 2025) found a 19% slowdown across a small set of tasks, with highly experienced developers working in their own repositories and a wide confidence interval; in February 2026 the researchers themselves redesigned the experiment after identifying methodological limitations. The 2025 DORA report, covering nearly 5,000 professionals, measured something different and compatible: greater AI adoption is associated at once with more delivery throughput and more delivery instability. The most solid figure is the least cited one: the developers in the original study believed they had sped up by 20% while the stopwatch said otherwise, so internal perception surveys do not count as evidence.

What is AGENTS.md and what is it for?

It is a Markdown file at the root of the repository that works as a README for AI agents: build commands, how to run the tests, code conventions, and the context an agent needs and that would clutter a README meant for humans. It exists so the agent knows the project's commands instead of deducing them, which is the most common reason an agent performs badly in an unfamiliar repository. It is an open format adopted by more than 60,000 open source projects and donated to the Agentic AI Foundation under the Linux Foundation. Claude Code reads CLAUDE.md instead, and the usual practice is to import AGENTS.md from there so you do not maintain two documents.

What characteristics does a repository need to benefit from agentic development?

Deterministic build and test commands — a suite that fails intermittently turns the verifier into noise —, fast tests so the agent can iterate instead of guessing, strict typing as the cheapest error detector, seeded test data so the environment can be brought up, and real human review capacity for the additional volume of changes. None of those five conditions is specific to AI: they are engineering best practices from fifteen years ago. If the test suite is not trustworthy, the agent has no source of truth and its verification is worth nothing.

When is it not worth using AI agents in a project?

When the source of truth is missing or when the problem is ambiguous. Specifically: if the test suite is unreliable or does not exist, if the environment cannot be brought up reproducibly, if the team does not have the capacity to review the extra volume of changes, or if the context needed to solve the task lives in someone's head and is not written down anywhere. Agents perform in structured work — repetitive migrations, test coverage, mechanical refactors, CRUD scaffolding — and perform poorly in ambiguous design decisions. Starting with the latter is the fastest way to conclude that this does not work.