Agents Lab
An experimental multi-agent software-engineering environment built to study orchestration, validation and correction. Specialized AI agents take a requirement from project discovery to review on a sandboxed copy of a project; the interesting part is not the agents but the controls around them.
Personal R&D project: not a product, not production-ready and not an autonomous developer. The source code is private, and only sanitized results are shown here. Runs and evidence can be walked through during an interview.
Why I built it
To learn, hands-on, how to control agents: how to coordinate specialized roles, run a multi-step development workflow and keep its state, execute tools safely, validate the work, handle revisions, collect evidence, and see where agentic systems fail.
Research question
Not “can an AI write code?”, but: how do you coordinate several specialized agents so that planning, implementation, QA, revision and review form a controlled engineering workflow?
Language-model agents report work they have not done: a file "updated" that never changed, tests "passing" that never ran, a review written without reading the code.
A multi-agent development workflow is only useful if each claim can be checked, if agents cannot act outside their role, and if a run cannot loop or spend without limit.
Architecture
How a run moves through the system
- 01Requirement & projectSandboxed copy
- 02Project discoveryCode, not a model
- 03ArchitectProposes the design
- 04PlannerOrdered tasks, acceptance criteria
- 05Tech LeadChooses task order
- 06DeveloperOnly role that writes
- 07QARuns approved tests
- 08ReviewerReads, approves or rejects
- Correction loopsA QA failure sends the work back to the Developer and QA validates it again; a Reviewer rejection leads to a correction and a new review. Both stop at a revision limit.
- 09Completed or failedDecided by evidence gates
Across every stage
- Orchestrator & run stateEnforces stage order and persists the state of the run
- Tool executionEach role gets only its tools; every call emits an event
- Request & token budgetsChecked before every model call
- Workspace isolationChecked paths, no shell, allowlisted test commands
- Tool-failure recoveryErrors return to the agent; limits end the run cleanly
- Evidence & eventsEvery claim is checked against recorded tool evidence
- Static & browser validationLint, tests and build, and checks in a real browser, kept apart
- A Python core exposes REST and WebSocket APIs; a React and TypeScript interface shows the run as it happens.
- Agents run on the OpenAI Agents SDK. The model provider sits behind an adapter; one provider is operational today.
- The orchestrator owns the run state and decides which role may act. Each agent receives only the tools its role allows: read, list, search, write or run tests.
- Every tool call goes through a workspace sandbox and emits an event. Events flow to an event store, a projection and, over WebSocket, to the interface.
- Static checks (lint, tests and build) and checks in a real browser with Playwright run separately, against the product the Developer built.
- A run never touches the original project: it works on a sandboxed copy.
Agent roles
- Project discovery
- Deterministic code, not an agent. Reads manifests and documentation and produces a project profile for the other roles.
- Architect
- Proposes the design. Cannot write code.
- Planner
- Turns the design into ordered tasks with dependencies and acceptance criteria.
- Tech Lead
- Coordinates and chooses the order of tasks. Never implements; code validates its choice before using it.
- Developer
- The only role with write access to the workspace.
- QA
- Runs tests, only from an allowlist of commands defined in code. Cannot edit files.
- Reviewer
- Read-only. Every finding must quote text that exists in the reviewed file.
Execution workflow
- Hybrid orchestration: code enforces the invariants of the workflow (stage order, which role may act, when a run may complete) and the Tech Lead agent only proposes the order of tasks.
- The state of each run is persisted, and every stage transition and tool call is recorded as an event, so a run can be inspected after it ends.
- Request and token budgets are checked before every model call, so a call that would exceed the limit is never sent. Each role also has a turn limit, and revisions are capped.
- Execution and presentation are separate: the interface renders a projection of events and never drives the run.
Validation and correction
Evidence before verdicts
- A failed tool call is returned to the agent as a result, so it can correct course instead of ending the run.
- Hitting a turn or budget limit ends the run with a structured failure that says which limit was reached, instead of hanging or overspending.
- QA failures and Reviewer rejections can send the work back to the Developer for correction, up to a revision limit. The run stops if revisions make no progress.
- Missing evidence is treated as blocked, not as a rejection, so the Developer is never sent to change code that may already be correct.
Lead decision
Evidence gates, enforced in code
The first real run failed because agents reported work they had not done. The fix went into code, not into prompts. A task passes only with evidence from the tools: write receipts with file hashes for the implementation, a real test run whose exit code overrides the QA verdict, and a recorded read of every relevant file for the review. A final gate checks that nothing changed after QA.
Discovery as deterministic code
Understanding the project is done by code that reads manifests and documentation, not by a model. It is predictable, testable and costs no model calls.
One joint QA and review pass
QA and review run once after all tasks instead of after each task, because test-writing tasks depend on earlier implementation tasks.
Limits taken from real runs
Turn limits were set from what real runs actually needed, plus a margin, rather than guessed.
No automatic passes
A task can count as already satisfied only with a stated reason and real evidence.
What I have learned
- A false PASS is the failure that matters most: a run that reports success on broken work is worse than one that stops. The evidence gates exist for that reason.
- The opposite failure matters too. In the benchmark campaigns, most failed runs were false FAILs: correct products that the system’s own browser QA rejected.
- Static evidence and browser evidence are different things. A run whose tests passed could still fail its build and render a blank page, so lint, tests, build and in-browser checks are judged separately.
- Tool and QA errors must be recoverable: returned to the agent as information, not allowed to end the run.
- Portability is part of reliability. Running a project’s test commands on Windows needed its own fix.
- Revision loops need a limit and a notion of progress, or they spin without improving anything.
- Most orchestration failure modes, such as roles running out of turns, only appeared in real runs, not in designs or in tests with a scripted model.
- Browser and visual QA is the hardest part to make deterministic. The latest changes compile browser checks from the plan instead of leaving QA to write them.
- Only validation outside the system, by an independent checker, makes a result credible.
Trade-offs
- Hybrid orchestration gives the Tech Lead less autonomy than a fully agent-managed run, in exchange for invariants an agent cannot argue its way around. A deterministic pipeline and a manager mode were also built; hybrid is the default.
- Evidence gates are strict: a run can stop for missing evidence even when the work was right. That is preferred over accepting claims that cannot be verified.
- A joint QA and review pass keeps task dependencies simple, but a problem is found later and can cost a larger revision.
- Strict budgets and turn limits protect cost and prevent loops, but a limit set too low stops a legitimate run. That happened on a larger project.
- Tasks run one at a time: state and evidence are easier to reason about, at the cost of speed.
- No shell and allowlisted test commands make execution safer, but QA can only run what was configured.
Current limitations
- Browser QA still rejects correct work too often; that is the main open problem.
- Only one model provider is operational.
- It is a single-user tool that runs on a local machine, with no authentication.
- An interrupted run cannot be resumed, tasks run one at a time, and there is no git integration.
Current R&D status
- Experimental R&D: a lab for studying orchestration, validation and correction loops. It has not met its own graduation criteria and is not production-ready.
- It is measured with a Basic Capability Benchmark: nine small tasks at three levels, each verified by an independent checker. Graduating requires at least two of three tasks per level, zero false PASS and at least one successful correction loop.
- The benchmark runs have exposed real limits in QA and orchestration, above all false FAILs: correct products that the system’s browser QA rejected. The changes made in response, deterministic browser checks and cost control, have not been benchmarked yet.
- Earlier, a real end-to-end run on a small existing project completed and was confirmed by an independent check afterwards. Runs on a larger real project have reached QA but have not completed.
Live project
The public site is a read-only replay of one real recorded run, a small benchmark task that the system completed. The agents are not running live, and the page calls no model.
agentslab.abrahamguerra.dev