Two expert teams.
Zero hallucinated authority.
A pipeline where no model grades its own homework.
Consultants is our account-based analysis app: pick a team — Marketing or Legal, DACH jurisdictions — describe your case, and receive a sourced report as a PDF. The interface is deliberately small. The engineering lives under it, where a question passes through up to seven stages before a single sentence reaches you.
The rule that shaped everything: the final wording is never produced by the same family of model that drafted it, and the counter-check always runs on a different model than the synthesis. A pipeline that cannot grade its own homework is the difference between an AI demo and a product a paying client can put their name on.
- Status
- Live (v0.7)
- Kind
- PWA + pipeline
- Teams
- Marketing · Legal DACH
- Pipeline
- 7 stages (legal)
- Report
- PDF, on demand
- Stack
- TypeScript · SQLite
- 01 intake gateSafety layer 1: the brief is pattern-checked for injection before a job exists. Malicious input dies here, with a clear refusal.
- 02 inventory factsBestandsaufnahme: what does the client's own vault already know about this kind of case?
- 03 research sourcesCurated sources only — a legal opinion without sources is refused by the pipeline itself.
- 04 counter-check→different model reviewGegenprüfung runs on a DIFFERENT model family than everything after it. No self-grading.
- 05 synthesis draftThe draft exists now — but it may never ship. Two more text stages follow, none of them the author's model.
- 06 paraphrase→non-author model rewriteA different family rewrites. Detectable model style is a defect, not a feature.
- 07 proofread + gate→PDF shipUmlaut guarantee, jurisdiction check, safety layer 2 over the final text — then a PDF with an analysis ID on every page.
A question enters at the top and becomes a PDF at the bottom. Each stage has one job, its own model routing, and the ability to fail loudly rather than pass something wrong downward.
Brief Why the hard version was the point § 01
Anyone can wire an LLM to a form. That is not the product.
The tempting version of this product is a textarea, an API call, and a streaming answer. It demos well and fails on the second real question: it cites nothing, it cannot tell a German from a Swiss legal context, it inherits the tone of whatever model wrote it, and it trusts every word the user typed.
Consultants is the other version. Two expert teams instead of one chatbot. A pipeline of up to seven stages instead of one call. A safety layer that treats free text as hostile input — because in a legal product, it is. And a report you can hand to a client with your name on it, because every page carries an analysis ID and every claim a source.
Timeline How the product earned its guardrails § 02
Every rule below exists because something went wrong first.
We run agent fleets on our own business since 2024.
Business OS taught us the failure modes: agents guessing instead of searching, shared context drifting into contradiction, costs exploding when everything runs on frontier models. Consultants is that experience, productized for paying users.
The CEO's own vault leaked into a customer database.
An early indexing run scanned a personal memo folder. Twenty-eight internal case files — including a real legal matter — sat in the customer-facing knowledge base. No customer ever saw them; the audit that found them was ours. That is still the worst bug this product has shipped.
Per-team vaults, SQL-level separation, and a standing purge rule.
Each team now scans only its own roots; the cross-team query filters in the WHERE clause, not in the client. And the rule we wrote down: every new access separation gets a stock audit plus a purge — an index that only upserts never deletes its own mistakes.
The safety filter blocked honest questions.
The first injection patterns matched too broadly. 'Ist es verboten, …?' — a perfectly normal legal question — was refused as hostile input. A false positive costs the user their entire request; the angry message that follows costs us the user.
A corpus test is now mandatory for every pattern change.
Ten legitimate briefs that must pass and eight attacks that must not — run directly against the matcher on every change. Patterns are plain, explainable regex lists, deliberately no ML classifier: a black box would have blind spots and no audit trail.
The report answered a question nobody asked.
Topic detection read the curated sources, not just the user's brief. A real-estate question got the DSGVO opinion, because the source base was full of privacy material. The source inventory was classifying the request.
Routing reads only what the client wrote.
Every decision that classifies a request now runs on the brief alone. Curated material may inform an answer, never a routing decision — the lesson generalizes to anything that mixes retrieved context into control flow.
A report a client can put their name on.
Version 0.7 ships the full loop: jurisdiction inferred and persisted, agent roster with stage IDs in the footer, umlaut-guaranteed German, PDF rendered on demand from the database. The demo provider runs the staging pipeline; production routing is a configuration decision, not a rewrite.
Three of these five incidents never reached a user. We count them anyway — the pipeline is judged by what it refuses, and so are we.
Decisions What we chose, and what it cost § 03
Four calls that make this boring to demo and safe to run.
No model family writes the final text of its own draft.
Synthesis drafts, a different model family paraphrases, a third proofreads. The last word in a client report is never the first author's — detectable model style is treated as a defect. The routing enforces this in code: configure the paraphrase stage on the drafting family and the pipeline refuses to run at all.
Cost: three text stages where one would do, and model routing as a first-class config problem.
Bought: reports that read like a person wrote them and survive the 'which AI is this?' question.
Every user text is hostile until pattern-checked.
Free-text briefs are an attack surface: prompt injection, exfiltration attempts, instructions smuggled in through 'my case looks like …'. Stage one checks patterns before a job exists; stage two checks the final report before it saves. A compromised middle stage cannot leak the client's data out through the PDF.
Cost: false-positive discipline — a mandatory corpus test on every pattern change, because blocking an honest legal question is its own incident.
Bought: a legal product whose input surface behaves like a bank's, not a chatbot's.
No sources, no report. The pipeline refuses, not the user.
A legal opinion without citations is a hallucination with confidence. The pipeline itself refuses to produce one — a missing research stage fails the job visibly instead of letting the synthesis invent. The user gets an error and their credit back, never a confident guess.
Cost: hard dependency on curated source material per jurisdiction.
Bought: every claim in every report traces to something — and failures are loud.
A failed analysis refunds the credit. Automatically.
Quota systems usually punish users for infrastructure failures. Here every terminal failure path books a refund event — idempotent per job — and a restart button re-runs the analysis at no cost. The accounting truth is 'started minus refunded', not 'started'.
Cost: refund logic on every failure path and a second button in the UI.
Bought: users never pay for our outages — in a trust product, that is pricing, not courtesy.
Product The app itself § 04
Small interface. The engineering is under it.
Live screenshots from the running app. The process view is not a marketing graphic — it renders from the pipeline definition in the database, which is why it can never show a stage that does not exist.
Incidents What broke, and what it changed § 05
The pipeline is judged by what it refuses.
Cross-tenant leak
A migration ran months late in production — invisibly.
SQL migrations simply were not copied by the build: tables existed in production only because of earlier manual state. Discovered when a stage could not find its tables. Nothing was lost; everything was unverified.
What we had assumed: that a green build implies a complete deploy. The compiler copies code, not schema.
Fix → the build step now copies migrations and the boot log names every migration applied. A deploy that cannot prove its schema is treated as failed.
Quality gate
“The PDF is thin. I expected the answer.”
The founder's verdict on a report whose full reasoning lived in the summary field. Data existed — the opinion, the sources, the roster — but nothing propagated it to the visible surface. A PDF that requires the reader to trust the app is not a report.
What we had assumed: that data in the database equals content in the deliverable. The gap between the two is where products go to die.
Fix → the expert opinion now ships as a full prose section, the jurisdiction is inferred and persisted per thread, and every PDF page footer carries the analysis ID — so even printed pages find their way back.
Transfer What this means where you are § 06
If you are putting an LLM in front of clients, these are the parts that bite.
Self-grading is the default failure
Without deliberate routing, the model that drafts also reviews, paraphrases and ships. It will always approve its own homework. Separating authorship by model family is cheap; not doing it is what's expensive.
Treat free text as an attack surface
Your users will paste instructions, and some of them are not your users. Pattern checks at intake and before persistence — plain, auditable lists, no ML classifier — catch what alignment training misses.
Retrieved context must never steer control flow
The moment your source inventory influences routing, your knowledge base classifies the request instead of the client. Route on the brief, inform with the sources.
Fail loudly, refund automatically
A pipeline that fails visibly and returns the credit builds more trust than one that never errors and sometimes invents. In products that answer questions people act on, refusal is a feature.