On This Page
A green test can conceal a missing rule. In this illustrative case, an agent is asked to connect a store to an email platform. A test server accepts a contact update and returns a neat 200. The agent completes its task and the test passes.
The real destination has one extra rule: the custom attribute must be declared before a contact update can set it. The test server did not model that rule. Its green result proves something about the agent in the test world. It proves nothing about the missing rule in the live one.
This is the central bargain of simulation. It makes dangerous or expensive actions repeatable. It also creates a second system whose behavior can drift from the first.
A stand-in makes action tests possible
To test an agent that acts, give it an objective, an environment and an observable end state. A source discussion of simulated agent environments describes this structure in the context of browser and knowledge-work tasks. The environment lets a verifier compare what the agent left behind with a known-good result.
For an integration, the stand-in could expose a small set of destination operations: list fields, create a contact, update a contact and read one back. It could hold a fixture with a declared FNAME field and an absent CUSTOM_TIER field. A test could ask the agent to propose a mapping, then reject a write that uses CUSTOM_TIER before a definition exists.
The stand-in is useful because each run starts from the same state. You can change one condition at a time: a missing field, a permission denied, a stale schema version, an ambiguous identifier. You can compare the agent’s call sequence and resulting state across models or prompts.
That is stronger than a text-only question about what the agent would do. It tests what the agent actually called in a controlled world.
Give the fixture a state before and after each call, not only a canned response. If the agent creates a contact twice, the second call should expose the duplicate behavior you intend to test. If it reads a field after writing, the read should return the stored state. Otherwise an agent can pass by sending a plausible request while the test never discovers whether its sequence produced the intended record.
Fidelity is part of the test
The fair objection is: “The vendor has a sandbox; why build another?” Use a vendor sandbox when it supports the case you need and keeps the data safe. A local stand-in earns its place when you need deterministic resets, deliberate faults or a way to test calls the vendor sandbox cannot stage. It also costs maintenance. Every rule you model can be wrong or become stale.
The official Brevo contact API says contact attributes must be present in the account and values with the wrong declared type are ignored. A stand-in that rejects every malformed value would behave differently from that documented case. It might train the agent to expect an error that never arrives.
Write down which vendor behaviors the stand-in covers. Keep a small set of contract probes against the real API or its supported test environment. When a probe finds a difference, update the stand-in and add the case that exposed it. The simulation is a model of the vendor, not a replacement for the vendor.
The same discipline applies to time. A stand-in that makes every write instantly visible may miss a workflow that reads too soon. A stand-in that delays every write may invent failures the vendor does not produce. Model timing only where it affects the decision under test, and label the rest as outside the simulation. A passing test then has a precise meaning: this agent completed these actions under these modeled rules.
Tajo’s Brevo connector guide describes the same practical issue from the merchant side: a successful request can leave a field absent. A test is useful only if it checks the field, not merely the HTTP response.
A simulated destination tests an agent under modeled rules. Contract probes compare those rules with the live destination before a bounded preview.
Move from safe rehearsal to bounded evidence
A passing simulation should lead to a small live preview. Pick a few records whose expected destination states are explicit. Include a normal record, a missing optional field and a record whose consent state forbids marketing use. Verify what the destination actually stores. Do not infer the result from the request payload alone.
A synthetic preview and a destination check answer different questions. Tajo’s mapping preview tests a translation without destination writes. The proposed activity view below shows the later boundary: even an applied update still needs its destination state checked. Keeping that last step pending prevents a rehearsal from being mistaken for proof.
A finished write still needs evidence
Did the customer sync finish?
Write applied. Verification pending.
The destination state has not been checked yet.
Sync activity
- Request accepted
- Update applied
- Destination check pending
Example run · 14:32 UTC
What we know
- Provider response
- Accepted
- Write result
- Applied
- Destination readback
- Pending
A successful call does not prove the final state.
Proposed layout informed by Tajo's activity feed. This illustration adds a verification state; it does not claim that destination readback is already implemented.
The preview should have a narrow scope and a clear stop condition. A failure should prevent broad activation and point back to the exact mismatch: field definition, identifier, type, permission or timing. A passing preview does not prove every future record will behave the same, but it establishes that the contract works for the selected cases today.
This sequence separates two questions. The simulation asks, “Does the agent follow the designed rule when we control the world?” The live preview asks, “Does the designed rule fit this vendor account?” Neither question can stand in for the other.
The guide to reverse ETL explains why the final job repeats over many records. That repetition magnifies any mismatch left between the simulated and real contract.
Rehearse in a world you can reset. Then inspect the world you cannot.
Frequently Asked Questions
What can a simulated destination test?
It can test whether an agent chooses the expected calls, respects permissions, handles known errors and leaves the expected state within the simulation.
What can it not prove?
It cannot establish that its model of the vendor is complete. Undocumented behavior, timing and account-specific configuration can differ from the simulation.
What should follow a passing simulation?
Run a bounded preview or canary against the actual destination, then inspect the resulting state before broad activation.