Every knowledge page in Hindsight is a mental model that rewrites itself. This is the whole pipeline — the mode decision, the seven phases, every branch, and each of the four outcomes — traced against the code as it stands today.
A refresh is split in two on purpose. _execute_mental_model_refresh runs the entire pipeline and writes nothing — it returns a _MentalModelRefreshRun recording every decision it made. refresh_mental_model persists that run; dry_run_refresh_mental_model reports it and throws it away. There is deliberately no flag that makes the executor behave differently for a preview, because a preview that could be configured would stop predicting the refresh it exists to predict.
Seven phases, in order. Phases 1–4 are unconditional: even a pure full-mode refresh resolves scope, takes a snapshot, and calls reflect. The branching all happens in 5 and 6.
content_written replaces it. Note the dashed bypass: in full mode phase 5 does not run at all, which is why full-mode pages never exhibit delta-specific damage.Asking for delta mode is not the same as getting it. requested_mode comes from the trigger — knowledge pages get KNOWLEDGE_PAGE_DEFAULT_TRIGGER, which sets mode: "delta", so delta is the default for every page. Three gates then decide whether the request survives.
use_delta being true still says nothing about whether delta will succeed; three further fallback reasons wait inside phase 5.Gate 2 passes when last_refreshed_source_query is NULL, not just when it matches. That is deliberate: someone who hand-writes a page and then turns on delta mode expects their prose to be the starting point, not to be discarded by a one-time full rebuild.
This is where the interesting failures live. The branch needs a baseline document to edit, and it will try two sources in order before giving up.
structured_content is missing or fails validation, the baseline is re-derived from the stored markdown. Since create_mental_model never populates structured_content, that fallback is guaranteed to run on a hand-authored page's first delta refresh.Two guards stand between a candidate and the database, and then one last conversion.
final_content is blank the refresh fails. A small model, an exhausted retry ladder, or an over-eager answer-cleaning regex can all yield an empty answer, and writing it would destroy a working document.final_content is still the reflect candidate — and that candidate was synthesised under created_after, so it only covers memories newer than the last refresh. Writing it would silently delete everything the document knew from older ones and advance the watermark past it. The guard is keyed on the window rather than on each failure branch, so any future branch inherits it.final_structured is still None — which is every full-mode refresh and every failed delta — the candidate markdown is parsed so the next refresh has something to edit.That last step is the lossy one. It runs on every full-mode refresh, and until recently a markdown table it could not recognise became a paragraph joined with spaces — the whole table welded onto one line. Because that join is idempotent, the next refresh re-parsed the wreck as prose and re-emitted it, so the page could never recover.
The outcome literal decides everything about persistence. Two of the four raise MentalModelRefreshError after recording the failure for audit.
| Outcome | content | structured_content | watermark | raises | Why |
|---|---|---|---|---|---|
content_written |
written | written | advances | no | The normal path, for both full and delta. |
content_preserved_no_new_facts |
untouched | untouched | advances | no | Nothing new in the window. The watermark still moves so the empty window is not re-read. |
refresh_failed_empty_candidate |
untouched | untouched | held | yes | Almost always an upstream LLM failure. Holding the watermark means a retry re-reads the same window. |
refresh_failed_delta_not_applied |
untouched | untouched | held | yes | The candidate covers only the delta window, so it is not a document. |
A fifth failure can occur during persistence itself: when the trigger carries a response_schema, the stored content is projected into structured_output, and if that extraction fails the whole refresh is failed rather than persisting content with a stale machine-readable view beside it.
The delta-window guard only fires when created_after is set. A model with no last_refreshed_at would fall through and write the candidate as a full document — legitimate, because a candidate read over full history is a document. In practice the column is NOT NULL and defaults to creation time, so a delta refresh always has a window.
This is the diagram that matters for the design decision. The delta LLM never sees markdown — it is handed current_doc.model_dump_json() and answers in typed blocks. Markdown enters the structured world at exactly three points, and two of them are on the refresh hot path.
The open question is whether structured_content should become the source of truth, with markdown rendered on read. Here is the difference, and it is narrower than it first appears — because on the delta path Hindsight already works this way.
content column. search_knowledge_pages fuses BM25 over page name and content, and returns content snippets. A rendered form has to stay materialised, so the parser problem is traded for a projection-sync problem.create_knowledge_page accepts markdown from clients, and imports restore markdown bodies. Both still need markdown → structure.Neither approach repairs pages already damaged: a collapsed table is a single physical line, and recognising a table needs at least two. That backfill is its own piece of work regardless of which direction wins.