Running engines in production

The reference shape has four parts. A deterministic engine (tables, formulas, rules) that is pure and tested. A rule store with history. A review queue where proposals wait for an expert. And an evaluation harness that replays past jobs and a held-out set on every change.

Language models attach at three points: intake (free text to structured inputs, with the structured result shown for confirmation), explanation (the narrative that goes with a proposal), and rule drafting (a correction in plain words to a candidate rule). Each of those outputs is checked by something deterministic before it matters.

This shape runs anywhere: a Next.js app on Vercel with Postgres, a Python service, or inside an existing system. The engine is a library; the queue and the ledger are tables; the model is an API call with a budget.

The queue is the expert's whole interface, so it has to be fast. One proposal per screen, the inputs on the left, the proposal on the right, and the diff from the last similar job in between. Accept is one key. Correct is a click on the line that is wrong.

Each correction opens a tiny form: what changed (pre-filled), the conditions (suggested from the job), and the why (required). The expert can narrow or widen the conditions. Submitting the form writes a rule and re-runs the proposal so they see the effect before moving on.

Batch the queue by job type so the expert's context stays warm, and cap sessions at thirty minutes. Reviews after that are worse than no reviews.

On every change to a table, a rule, or a model prompt, the harness replays the past-job set and the held-out set and reports agreement by job type. A change that lowers held-out agreement does not ship. This is the single discipline that keeps an engine from getting worse while looking busier.

Drift is when the world changes and the rules do not: a new cement vendor, a rate increase, a regulator's update. Watch for agreement falling on a job type that used to be stable, and for rules that have not fired in ninety days. Both are prompts for the expert, not alarms.

Rules conflict. Two rules can match the same job and disagree. Order them, show both in the proposal, and make the expert pick. Retire rules rather than deleting them, so the history explains past proposals.

Unattended use means a proposal goes out, to a customer or a crew, without a human reading it. The decision is per job type and it is the owner's, not ours. The inputs are: held-out agreement above a threshold (we suggest 95% for pricing, higher for safety-relevant quantities), a minimum number of reviews, and no open rule conflicts.

Even unattended, every proposal is logged with the rules that fired and can be sampled for review. A fraction of unattended proposals go back into the queue so the agreement number stays honest.

A customer-facing first pass is the common unattended case: show ranges, name the assumptions, and offer a human. The engine's job there is to make the first conversation shorter, not to close the sale.

Scope one engine at a time. The scope page names the decision, the inputs, the tables, the past jobs, the expert, and the definition of acceptable. It is one page because anything longer is two engines.

Cost is mostly the first six weeks. After that the running cost is hosting, model calls (small, because the model is at the edges), and the expert's review time, which falls as agreement rises.

The return is measured on the same dashboard the expert uses: proposals per month, minutes per proposal before and after, agreement rate, and for customer-facing engines, first-pass quotes that turned into jobs.