How One Video Agent Iteration Grew into a Harness Method

A while back I was tuning a video agent for one specific visual style.

The task was not the broad "make it generate videos", but making it reliably produce one class of style: take a reference brand's Instagram / Reel visual language, swap in another brand or product, keep the composition, lighting, color grade and camera rhythm — and never let the reference brand's logo, packaging or text leak into the final frame.

This task is easy to misjudge.

Looking at any single output, it's just "not bad" or "doesn't look like it". But a few iterations in, the problems become hard to describe: some versions got more stable but the style went flat; some versions looked more like the reference video but product recognition got worse; some fixes looked reasonable, then broke previously good cases in the next round.

I did not set out to build a methodology.

I only wanted to record each round's results so we could align on generation quality in the dev meeting: what exactly changed this round, which cases improved, which regressed, what to fix next.

Only later did I realize that the thing being optimized was never one prompt, nor just some code — it was the entire feedback system around agent iteration.

Process log: it started with "aligning on results"

The first step was humble: have the coding agent sync results into a Feishu doc after every test run.

Those docs were never paper-style records or reports for external readers. They were more like a results board inside a development collaboration.

One kind of doc did horizontal comparison: the same batch of cases, and what each produced under different versions, strategies and generation pipelines. The point wasn't what the table looked like — it was that everyone in the meeting could point at the same row and say: this case improved, or this case regressed.

The other kind did vertical unrolling: one case from user input, through material scraping, style analysis, product analysis, material arrangement, prompt generation, tool calls, to the output. It spread out the agent's intermediate artifacts, so problems no longer landed only on the final cut.

Both kinds of record solve the same problem: don't let the agent understand what happened last round from conversational memory alone.

A video agent's failure is rarely as simple as "the final cut looks bad". It may have grabbed the wrong reference at the material stage, failed to get the real SKU at product analysis, or turned "reference the composition" into "replicate the reference product" at the prompt stage.

Without a trace, a coding agent will happily patch only the render layer.

Shared context: the Feishu doc becomes external memory

A few rounds in, the doc's role changed.

It was no longer just a results board, but the surface where the coding agent, the sub agent and I all worked.

The coding agent wrote test results into it; the sub agent did independent review in the doc; I added human judgment alongside — "this version looks closer to the reference style, but the product subject got weaker", "this is not a bad case, this roughness is exactly what the style should keep", "this fix direction is wrong, it drags everything toward a templated commercial".

At that point the doc carried three roles.

Shared Context: every participant works around the same batch of cases, the same round of outputs, the same set of comments.

External Memory: judgments persist across rounds instead of depending on one conversation window.

Review Surface: comments attach to a concrete paragraph, table or result, never floating as a vague "it looks bad".

This step gave me a more concrete understanding of what a harness is.

A harness is not just the script that runs tests. For an agent workflow, it also includes the history the agent can read, the cases it can reuse, the outputs that can be reviewed, and the channel that carries review results back into the next round of changes.

Independent review: the sub agent is more objective — not objective truth

Later I added sub agent review.

Its value is the independent context.

A coding agent that just changed the workflow tends to explain results along the lines of its own patch. Having just added a stronger product constraint, it reads "the product is clearer" as progress; but what I actually care about may be that with the product clearer, the reference brand's casual, offhand feel is gone.

The sub agent enters from another context: it sees only the test doc, the outputs and the review instructions. It didn't participate in the round of changes, so it more easily names side effects the coding agent hasn't noticed.

But don't deify the sub agent.

It is still a subjective reviewer. It may lean toward clearer, more complete, more commercial outputs; it may judge noise that a style is supposed to keep as a defect. So it is not the final referee — it is a context-isolated AI reviewer.

Which is why human calibration comes later — and even iterating the sub agent's own review prompt.

Three review layers: fixed scripts, AI, human

Around this point I remembered the harness-engineering articles I'd read.

What triggered me wasn't a particular term but a judgment: reliable agent review can't run on one kind of signal.

The view, as I remembered it, was that review inside a harness needs at least three things working together: fixed scripts, AI review, human review.

That mapped exactly onto what I was doing.

Fixed scripts own the hard rules not worth debating. Whether a file was generated, whether the video link is reachable, whether the duration is in target range, whether the output contains plainly forbidden subtitles, watermarks, brand text.

AI review owns high-throughput semantic judgment. Whether the style is close to the reference, whether the product got wrongly replaced, whether the camera rhythm clearly drifted, what regressed in this version against the previous one.

Human review owns direction and trade-offs. Whether a particular "roughness" is a failure or the very texture this style needs; whether a rule should enter the next round or stay as an observation on this case.

That's when I realized I wasn't making the coding agent "iterate a few more rounds".

I was building it a small harness: run cases, save traces, launch reviews, read comments, change code, return to the next round of cases.

Judge calibration: the AI reviewer gets iterated too

The sub agent's judgment is useful, but must not be presumed reliable.

In a video-style task, an AI reviewer easily and consistently leans certain ways: clearer, more complete, more like an ad, less noise. In ordinary quality evaluation those leanings might be virtues; in some style-matching tasks they steer the system off course.

So I started treating the sub agent as a component that needs iterating.

Human comments don't just patch missed judgments — they calibrate the judge's direction:

• Which style traits does it misjudge as defects?
• On which dimensions is it fairly stable?
• Does it over-reward clarity, product share of frame, or a commercial feel?
• Does its review prompt need constraints like "preserve the lo-fi texture"?

That is Judge Calibration.

It doesn't need precision, recall and agreement metrics from day one, but the idea is already there: the AI reviewer's output also gets sampled, down-weighted and corrected — it doesn't feed straight into the coding agent's basis for changes.

Boundary knowledge: from comments to conditions

The first version of the loop showed its problem quickly.

Feishu comments work for humans but are still too loose for a coding agent.

When I write "this looks more like a commercial now", a human understands that may bundle over-lighting, a staged product feel, too-stable camera work, too-clean post. The coding agent may seize exactly one of those and turn it into "reduce product clarity".

That is the grounding gap between natural-language comments and code changes.

So, following existing harness practice, I began organizing problems by the logic of "accumulating boundaries".

A boundary is not "this is bad" — it is a judgment with conditions attached:

• When the goal is keeping the Instagram offhand feel, overly clean studio lighting pushes the output toward a staged commercial shoot.
• When the reference material shows the source brand's product, only composition, light, grain and color may be extracted; the target prompt must not replicate the source product's shape.
• When product-image analysis hasn't obtained a sufficiently real SKU, continuing to render risks generating a fictional product — better to fail fast.
• When the subject is enlarged to strengthen product recognition, style transfer degrades into a product hero packshot.

In the methodology written up later, this is Boundary Knowledge.

Its job is upgrading a bad case from "that output was poor" to "this class of condition fails". What the coding agent reads is no longer an emotional comment but something much closer to a rule.

Structured boundaries: cutting semantic loss between agents

Accumulating natural-language boundaries alone still leaks.

The same sentence — "the product got shoved into the scene" — might mean the product takes too much frame, or the lighting doesn't match, or the product has no relation to the person's action. A human fills the gap from context; a coding agent doesn't always fill it correctly.

So boundaries get structured, gradually.

A sturdier boundary carries at least:

• category: style, product recognition, material selection, prompt constraint, camera rhythm, or safety rule.
• condition: under what condition the problem appears.
• failure: which goal it damages.
• evidence: which cases it links to.
• severity: must-fix or observation.
• confidence: from a human, the sub agent, or both in agreement.

This is not about turning writing into tables.

It is about making the coding agent guess less. Especially when agents pass information through documents, structured fields cut down the odds of "I thought you meant a different problem".

Two signals: boundaries alone underperform

At first, following the harness line, I mainly accumulated boundaries.

Right direction, mediocre results.

The reason is simple: with only bad cases, the system knows where not to go — never what is worth keeping.

Every bad case tells the agent not to do something: don't let the product get too small, don't leak the logo, don't let the frame get messy, don't let the cuts jump. The more rules pile up, the more the output looks like a safety template.

But a video style isn't satisfied by merely "not broken".

Some things come precisely from risk: the handheld feel, accidental composition, slight overexposure, the un-commercial everyday air in the material, the irregular rhythm between cuts.

So I started making feedback bidirectional and systematic.

On one side, boundaries — recording "under what conditions it breaks".

On the other, exemplars — recording "what must be preserved".

Say a version doesn't maximize product display but holds the reference brand's tone of voice; a case carries a bit of noise yet is the closest to a real Instagram post; a prompt structure isn't clean but reduces the AI's default studio-shoot feel.

That is Good Case replay.

Good cases aren't there to prove the system is great — they give the coding agent a positive reference: next round, whatever you fix, don't fix these away.

Golden Set: stop fixing one thing by breaking another

Push the two signals forward and you arrive naturally at a Golden Set.

Once certain good cases are human-confirmed, the next round of changes must not break them. Otherwise the agent overfits the latest comments.

The moment I comment "the product subject is too weak", the next round may shove every case's product to frame center; the moment I comment "the reference style isn't strong enough", the source brand's elements may come roaring back.

The Golden Set makes the coding agent prove first: this round's fix did not break behavior that already stood.

It doesn't need full automation on day one.

The minimal version can be crude: replay a fixed batch of human-confirmed cases, compare whether key dimensions dropped. Once the workflow stabilizes, upgrade it into a Regression Gate: if the pass rate drops beyond threshold, this round's change doesn't ship.

That is the Golden Set + Regression Gate in my later methodology doc.

Active probing: from stumbling into problems to hunting boundaries

Early testing ran mostly random or natural cases.

That finds problems, but inefficiently. Many failures only surface by luck.

A boundary may not be "short transitions always break", but "short transitions break in slow-paced narration and are acceptable over fast-paced music". On random cases alone, that line can stay invisible for a long time.

So in the later methodology I extended this into Active Boundary Probing.

The Probe Agent's job is not ordinary case generation but constructing tests near existing boundaries:

• Nudge parameters around a known bad case to find the true critical point.
• Cross two boundary conditions and see whether a new failure appears.
• Deliberately generate cases where the judge has low confidence, forcing out the unknown region.

This part isn't a module I fully practiced at the time — it is the next step pushed out from the problem that "random cases are inefficient".

Convergence: don't let the loop become endless patching

Another question that became clear only later: when to stop?

Agent iteration slides easily into endless patching. Every round finds some problem; every round can change something. But continuing to change doesn't mean continuing to improve.

So the methodology needs a set of convergence signals.

They can be simple:

• Is the Golden Set stable?
• Is the count of new bad cases falling?
• Are new boundaries still growing fast?
• After each fix round, is a high enough share of old bad cases actually improving?
• Is agreement between the sub agent and human judgment stable?

I don't want to write this up as "we had a full dashboard at the time".

The accurate version: practice first exposed "you can't stop by feel", and the methodology later extended that into Convergence Metrics.

How the method evolved: from one practice to a set of engineering modules

This method wasn't designed up front.

It grew like this:

1. I recorded each round of video agent output to align on results in dev meetings.
2. The record docs became shared context, taking on external memory and review-surface roles.
3. I added a sub agent to do AI review in an independent context.
4. I found the sub agent had its own subjective bias, so human review became judge calibration.
5. I accumulated bad cases and boundaries along existing harness lines.
6. I found negative feedback alone underperformed, so good cases and exemplars joined.
7. I found fixes breaking previously good cases, hence Golden Set and Regression Gate.
8. I found random-case exploration inefficient, pushing on to Active Probing.
9. I found the loop needed a stopping condition, filling in Convergence Metrics.

This matters more than giving it a new name.

The name is dispensable; the modules should stay: Trace, Shared Context, AI Review, Human Calibration, Boundary Store, Exemplar Store, Golden Set, Regression Gate, Active Probing, Judge Calibration, Convergence Metrics.

Together they solve one problem: how the experience produced by each round of practice actually enters the next round of changes.

The minimal reusable version

Compressed to its minimum, I would keep six modules.

Case Trace

Every round's results must be revisitable. At minimum: case, version, input, output link, key intermediate artifacts and tool calls. For a video agent, the final video is not enough — you want the material analysis, the manifest and the render prompt.

Shared Review Doc

Comments must attach to concrete results. Don't say "this version is off" in chat — put the comment next to the specific case, the specific stage, the specific output.

Isolated AI Review

Let a sub agent that didn't participate in the change do review. It doesn't make the final call; it exists to find the side effects the coding agent may have missed.

Human Calibration

Human comments own direction, taste and trade-offs, while calibrating the sub agent's review bias.

Boundary / Exemplar Store

Confirmed bad cases enter the boundary store, confirmed good cases the exemplar store. The former tells the agent under which conditions things break; the latter tells it what must not be fixed away next round.

Golden Set / Regression Gate

After each round of changes, replay a batch of human-confirmed good cases. Guarantee no regression first, then explore new problems.

This minimal version already resolves much of the drift in agent iteration.

Active Probing, metricized Judge Calibration and Convergence Metrics can bolt on as next-stage reinforcements.

Key takeaways

• The shared doc is not an ordinary log — it is the shared context, external memory and review surface of agent iteration.
• Sub agent review's value is context isolation, but it still needs human calibration and its own iteration.
• Boundaries alone are not enough; keep exemplars and a Golden Set too, or the agent fixes itself into timidity.
• A harness grown from practice is, at heart, the design of a feedback system — not the coding agent running a few more rounds on feel.

References

Articles I explicitly recalled and checked against during the practice:

• OpenAI: Realtime Eval Guide
• OpenAI: Harness engineering, leveraging Codex in an agent-first world
• Cursor: Continually improving our agent harness

Theoretical references added while later discussing the methodology with agents:

• Reflexion: Language Agents with Verbal Reinforcement Learning
• Self-Refine: Iterative Refinement with Self-Feedback
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• EDDOps: Evaluation-Driven Development for LLM Agents
• Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks