The short version

In everyday terms. AI assistants are impressive right up until they misunderstand something. If a misunderstanding results in a change that has already been made to real records, you do not have a feature - you have a way of generating incidents with a friendly interface.

The fix is a single rule: the AI is allowed to suggest, never to act. It prepares a precise, structured request; ordinary software then checks that request against the same permission rules a human would face, and only then carries it out. A misunderstood instruction becomes a rejected suggestion instead of a corrupted customer list.

Why a business should care. This is what makes an AI feature launchable rather than permanently stuck in a demo. It also makes the behaviour auditable: every action has a record of what was asked, what was proposed, and what was allowed. Three related concerns get the same treatment - output is validated before it is stored, a second provider takes over when the first fails, and spend is tracked per request so the monthly cost is knowable in advance.

What follows is how that boundary is implemented, and what happens at each point it is tested.

The demo is always impressive. You say “add everyone who opened the March campaign to a new segment and tag them warm,” and it happens.

The question that decides whether it ships is different: what happens when the model misunderstands? If the answer involves an UPDATE that already ran, you do not have a feature. You have an incident generator with a friendly interface.

ECamp has a Gemini-powered agent for contact management, driven by natural language and voice. It runs against production data. It does so because of one architectural rule.

The rule

The agent never writes. It proposes.

The model’s output is not an action. It is a structured proposal, validated against a declared schema, subjected to the same permission checks a human request would face, and executed by ordinary application code that has no idea an LLM was involved.

class AddToSegment(BaseModel):
    action: Literal["add_to_segment"]
    filter: ContactFilter          # a constrained query, not raw SQL
    segment_name: str
    tags: list[str] = []

class DeleteContacts(BaseModel):
    action: Literal["delete_contacts"]
    filter: ContactFilter
    confirm_count: int             # the model must state how many it expects

Proposal = Annotated[
    Union[AddToSegment, DeleteContacts, ...],
    Field(discriminator="action"),
]

The model chooses among operations that exist. It cannot invent one, cannot compose raw SQL, and cannot reach anything outside the declared set. An intent that maps to no operation fails validation and becomes a clarifying question - which is exactly the right behaviour.

Note confirm_count on the destructive operation. The model must state how many records it believes it is affecting. If the count and the filter disagree by more than a small margin, the proposal is rejected before execution. A misunderstanding becomes a rejection, not a deletion.

The three checks between proposal and execution

Schema validation. Does this parse into a known operation with well-typed fields? Invalid output triggers a repair retry with the validation error included as context - models are markedly better at fixing a specific complaint than at getting it right on the second blind attempt. After a small number of failed repairs, it is an error, not an infinite loop.

Permission resolution. The proposal runs through the exact authorization path a human API request would. Same tenant scoping, same role checks, same subscription gate. The agent is not a privileged caller; it is a caller that happens to have been assembled by a model.

Blast radius. Any operation exceeding a threshold - record count, irreversibility, cost - requires explicit human confirmation with the effect described in plain language: “This will permanently delete 3,412 contacts across 2 segments.” Not the model’s summary of what it thinks it is doing. The application layer’s description of what will actually happen.

Together these mean the worst case of a misunderstood instruction is a rejected proposal or a confirmation dialog. Never a silent mutation.

Schema-validated extraction, everywhere

The same discipline applies outside agents. In NewsCrawl, the LLM extracts structured fields from article text, and the pipeline treats every response as untrusted input:

  • Parse against the declared schema. Reject on mismatch.
  • Retry with the validation error attached, bounded.
  • On repeated failure, record the document as extraction-failed and continue. One bad document must not stop a pipeline.

The alternative - parsing hopefully and writing what you got - produces a database where some fields are strings, some are strings containing JSON, and one is an apology. Every downstream consumer inherits that mess forever.

Providers are dependencies, so treat them like dependencies

An LLM API is a third-party network service with rate limits, latency spikes and outages. Nothing about it deserves special trust.

Fallback. Primary provider rate-limits or times out, the call routes to a secondary. The pipeline degrades in quality rather than stopping. Which requires that prompts and schemas are not tuned so tightly to one provider that they break on another - a constraint worth accepting up front.

Timeouts and circuit breaking. A hanging call holds a worker. Aggressive timeouts, and a breaker that stops sending to a provider that is failing consistently, so you fail fast instead of queueing into a wall.

Cost accounting per call. Tokens in, tokens out, model, cost, attributed to the document or campaign that caused it:

usage.record(
    provider=provider.name,
    model=provider.model,
    prompt_tokens=r.usage.prompt_tokens,
    completion_tokens=r.usage.completion_tokens,
    cost_usd=provider.price(r.usage),
    attributed_to=doc_id,
)

This is the least glamorous part and the one I would keep if I could only keep one. Without it, “is this feature worth its cost?” is a discussion. With it, it’s a query. It is also how you find the prompt change that quietly tripled your bill, in the week it happens rather than at the end of the month.

Idempotency, again

Retries mean the same extraction can run twice on the same document. Same rule as everywhere else in the system: key results on the content, upsert rather than insert, and a repeated call is a wasted API call rather than a duplicated row.

Cache on that key too. In a pipeline that reprocesses documents after a schema change, the cache is often the majority of the savings.

Where the intelligence actually goes

The pattern underneath all of this: the model supplies intent; the application supplies authority.

The model is genuinely good at turning “everyone who opened the March campaign, tag them warm” into a structured filter. It is not good at deciding whether the caller may do that, whether three thousand records is a reasonable blast radius, or whether the operation is reversible. Those are questions with correct answers your code already knows.

Keep the boundary sharp and you get the useful part of an LLM without handing it the keys. Blur it and you have built a system whose safety properties depend on a probability distribution.