Grounded Strategy Intelligence Workspace

A strategic intelligence workspace over a national-scale strategy corpus, built to move an executive from diagnose → measure → decide - where every claim traces to the exact source slide, and where quality is measured by twelve automated gates behind a judge validated at Cohen’s κ = 0.881.

The most technically demanding system in this portfolio. Sole delivery: research, architecture, ingestion pipeline, retrieval, scoring methodology, evaluation harness, frontend, and hardened cloud deployment.

1. The problem

A government authority in the GCC held its multi-year national strategy across ~45 documents (~3,900 pages, ~2,100 unique content slides) spanning ~12 programme strategies, plus 8 international peer-country strategies for benchmarking. Predominantly slide decks and prose PDFs - not clean text.

Leadership could not answer its own basic questions: where is our strategy thin or missing? how mature is each programme against its own targets? where are peer countries ahead of us? what should we prioritise next - and prove it.

The properties that break a naive build

The framing insight that determined everything

This is a multi-label tagging and synthesis problem disguised as a search problem. “Give me the consolidated view per dimension” cannot be answered by top-k retrieval, because no single document is the answer - the content for one dimension is scattered across a dozen decks that each touch it partially.

Getting this framing wrong produces a plausible demo that fails on the actual question. Every architectural decision below follows from it.

2. Research phase - the decisions I made before building

I ran a formal research pass before committing to an architecture, and documented each decision with its evidence base and its rejected alternatives. This is the part I’d most want reviewed, because the highest-leverage engineering here was deciding what not to build.

Rejected: full GraphRAG / graph database

The instinctive answer to “interconnected strategy documents” is a knowledge graph. I rejected it, on four independent grounds:

1. Corpus scale is below the payback threshold. Full community-detection GraphRAG indexing is documented at $50–200 and ~45 minutes per 500 pages, with the sweet spot at >1k documents. At ~2,100 unique content slides (<1M tokens) the build never pays back.
2. Its core value is wasted here. Community detection discovers themes. My themes were given - the client’s own assessment framework. Paying a graph-construction cost to discover what I already had as an ontology is pure waste.
3. Published evidence shows it underperforms on the dominant query type. Graph approaches lose to vanilla RAG on single-hop and detail questions - which is most of what an executive asks.
4. Most decisively - multi-hop is reasoning-bound, not retrieval-bound. A large multi-document benchmark (72,880 pages, 6 languages) found multi-hop QA reaches only ~0.655 accuracy even when the ground-truth pages are handed directly to the model. A graph that improves retrieval therefore cannot fix multi-hop. Agentic query decomposition addresses it; a graph does not. This single finding redirected effort from graph construction to decomposition - and decomposition is what shipped.

Instead of building it, I gated it. Triples are extracted and parked at ingestion; a graph gets built only if evaluation shows >~20% of multi-hop questions failing at the retrieval stage. If that gate ever trips, the pre-selected approach is an ontology-grounded method (documented at +55% fact recall and +40% response correctness on fixed-ontology domains - which is exactly what an assessment framework is), with an incremental-update graph as the cheap alternative. Never community-detection-plus-graph-server for this corpus.

The gate never tripped. Parking the triples meant the eventual relationship view cost zero re-ingestion.

Rejected: pure visual / page-as-image retrieval

Attractive for slide-heavy corpora. Rejected because: the decks are born-digital with an exact text layer (so the expensive visual path buys nothing on text fidelity); off-the-shelf visual embeddings collapse on Arabic (~0.88–0.95 NDCG@10 English versus 0.04–0.07 Arabic, with model scaling giving only ~1.6×); citations must resolve to a precise page, which the text path gives cleanly; and the specialist visual-document embedders my research identified as best-in-class were not available on the internal gateway, making that path a self-hosting project for a capability I had already argued against.

I documented this as the build’s one honest capability gap rather than hiding it - and confirmed the chosen spine (text + LLM-generated diagram descriptions) fully covers the requirement.

Rejected: a note-taking app as the engine

A knowledge-management tool was proposed as the platform. My research separated two things people routinely conflate: a wikilink visualisation is not a knowledge graph (untyped edges, not queryable, degrades to an unreadable hairball past ~200 notes - useless at ~2,100 pages).

More seriously, I found a confidentiality disqualifier: the ecosystem’s AI plugins default to cloud APIs, with one popular plugin shipping cloud embeddings on by default and silently falling back to cloud on local timeout. For government data under a professional-services engagement, un-auditable cloud exposure ends the discussion. Verdict: valid as an optional human review surface over the same markdown (structured frontmatter plus live query views), explicitly not the retrieval engine and not the graph engine.

Chosen, with reasoning

I also live-probed the entire gateway model catalogue before choosing, rather than assuming availability - which is what surfaced both the specialist-embedder gap and the correct frontier-model configuration.

Operational research

I additionally pulled from production-RAG post-mortems, on the grounds that failures in this class of system are usually operational rather than model-quality. That informed born-digital-versus-scanned detection by image coverage and text-quality score rather than naive character count, and the decision to build an evaluation set on the real corpus before shipping anything.

3. Architecture

     Internal LLM gateway  (all model + embedding calls - confidential data never leaves the tenant)
                                        ▲
                                        │
  SOURCES                      ┌────────┴─────────┐                 SURFACES  (React 19 / Vite 8)
  pdf / pptx  ───ingest────▶   │  core package    │  ───read────▶  FastAPI  ──▶  diagnostic views
  (offline, batch,             │  ~19 modules     │   (read-only)             assessment views
   idempotent, resumable)      │  pipeline+stores │                           decision support
                               └────────┬─────────┘                           grounded Q&A (SSE)
                                        │                                     administration
                          ┌─────────────┴──────────────┐
                          │ Vector store → chunks      │  hybrid+BM25
                          │ SQL store    → fact cards  │  precise SQL
                          │ JSON         → artifacts   │  precomputed views
                          │ PNG          → rendered    │
                          └────────────────────────────┘

Two clean halves

Ingestion was offline and batch at first delivery - run once per corpus change. Serving is online and strictly read-only. This is what made the system demo-stable: nothing expensive happens on the request path, and the API is safe to run during a re-index or an evaluation pass.

The system has since evolved: ingestion now runs online from the deployed admin surface. Administrators upload documents themselves and configure custom prompting per run. The read-only serving path is unchanged.

The recurring architectural pattern: precompute → read-only endpoint → thin view

Anything expensive is computed offline into an artifact. The API is a thin read. The UI renders.

I chose this deliberately over both alternatives: a static page goes stale silently, and re-running the model per view is slow, costly, and non-deterministic (fatal for a demo, and worse for an executive who expects the same number twice). The artifacts additionally carry a corpus fingerprint, so every derived view knows when it is stale and says so in the UI.

Two stores, deliberately

Different questions need different engines, and forcing one to do both jobs would have been the wrong call:

Both embedded and file-based: no database server to provision, which fits a single-tenant deployment path from laptop to cloud. Final scale: ~10,389 cited fact cards and ~2,872 embedded chunks.

Core package as the single source of truth

~19 focused modules, one per stage or concern, from rendering and extraction through retrieval, scoring, verification, judging, and calibration. The API and every script import the package directly; no logic is duplicated in the HTTP layer.

A single config module holds models, thresholds, the programme → document-IDs map, and the benchmark registry. Consequence: “add a programme or a deck” is a data edit, and every derived artifact regenerates from it automatically.

Three cross-cutting decisions worth naming

A programme owns many decks, but a deck stays one unit. Citations resolve by (document_id, page), and page numbers collide across a programme’s decks - “page 44” exists in many documents. So I aggregate at the view layer but never merge decks into one document ID. This is precisely what kept aggregation clean when 11 enrichment decks were added later, and what the embedded document ID in every citation marker disambiguates. A bare page number would have produced silent false citations.

Benchmark contamination guard. The 8 international peer strategies carry a source='benchmark' tag and are excluded from diagnostic retrieval by default, and from domestic-only views entirely. Mixing primary and external corpora in one retrieval pool is a known top-tier RAG failure mode - so the guard was built before the first benchmark document was ingested, and is verified by test rather than assumed. Peer content surfaces only where benchmarking is explicitly intended.

Flag, never delete. Drafts and near-duplicates stay in the stores - nothing is lost and evaluation stays honest - and are suppressed from default retrieval, with gates proving the suppression holds.

4. The ingestion pipeline

Six stages, each idempotent, resumable, and independently gated.

SOURCE (pdf/pptx)
  → RENDER   page images @150dpi + native text layer + tables
  → EXTRACT  vision LLM → schema-validated page record
  → TAG      multi-label ontology dimensions / sub-dimensions
  → CHUNK    one chunk per page + structured fact cards
  → DEDUP    corpus-wide near-duplicate detection
  → INDEX    embed → vector store; load → SQL store

Render - one contract, two formats

PDFs render page images at 150 dpi with native text and tables extracted separately. Slide decks are converted to PDF via headless office conversion, then run the identical PDF path - so a 16:9 deck and an A4 prose book emerge identically shaped downstream.

Three decisions:

• Keep both image and native text. The vision model reads the image (capturing charts and diagrams) and receives the born-digital text layer as ground truth for exact spellings and figures. Font substitution during conversion therefore never matters - a real class of silent corruption, eliminated structurally.
• Private per-call conversion profile, because a shared profile lock is the documented failure mode when a desktop office app is open or conversions run concurrently.
• Format-agnosticism paid off unprompted: when benchmark material later arrived in two unexpected shapes (A4 prose books and a landscape deck), the same path handled both with zero code changes.

Extract - the heart of the pipeline

One vision-LLM call per page → one schema-validated record:

• page_type - title / divider / content / card / table / visual / sparse
• markdown - faithful text of all content
• diagram_description - diagrams, wheels and org-charts described as explicit relationships. This is what rescues the content native extraction destroys, and it is embedded alongside the text, so a governance map becomes retrievable.
• cards[] - the structured gold: {label, value, temporal, time_horizon, owner, unit, dimension, sub_dimensions}
• multi-label ontology tags · temporal_orientation · confidence
• quality_flags[] - placeholder / draft_option / draft_watermark / duplicate / Arabic RTL
• triples[] - subject-relation-object, parked for a possible graph

Every quantified fact carries its own temporal label, so “58% today” and “80% by 2033” can never blur. That one modelling decision is what lets the SQL layer answer “all 2033 workforce targets” exactly, what makes every number traceable to one page, and what later made evidence-gated assessment possible at all.

Anti-hallucination enforced at the source. The extraction contract is strict: be faithful to what is visible; never invent a number, owner, or target; if a value is a placeholder, a draft option, or illegible - flag it and reproduce it verbatim, do not fill it in. Draft content can be suppressed downstream precisely because it was honestly labelled at extraction rather than silently absorbed.

A prompt-contract bug worth recording. Ontology keys are now presented in quotes as the exact string to emit, with human-readable names in a clearly separated “for your understanding only” block. An earlier format placed the key and a look-alike label on the same line - and models copied the parenthetical instead of the key. A tagging failure whose root cause was prompt formatting, not model capability.

Dedup - flag, never delete

Near-duplicates detected corpus-wide at cosine ≥ 0.97 with union-find grouping and a deterministic canonical choice. The same context slide legitimately recurs across a programme’s current-state, gaps, and benchmarking decks; without this, it pads every answer and double-counts every aggregate view.

5. Retrieval and answer path

1. Decompose the question into 2–5 focused sub-queries - broad multi-part questions otherwise crowd out specific sub-facts. (This is the agentic decomposition my research identified as the real answer to multi-hop, in place of a graph.)
2. Hybrid search per sub-query - dense + BM25 with reciprocal rank fusion, filtered to domestic source, non-draft, non-near-duplicate by default.
3. Merge the union, then rerank the whole set against the original question - not against the sub-queries. The subtle but important step: sub-queries are a retrieval device, and the user’s actual question is the relevance target.
4. Synthesize under a strict citation contract: answer only from retrieved passages; cite every claim with its verbatim [programme · document · p.N] marker; distinguish current from target figures; never reproduce draft content; a number belongs only to the slide it appears on - no cross-slide attribution; and omit anything that cannot be cited.
5. Stream token-by-token over SSE, with citation pills resolving to the exact rendered source page.

Deliberately, no numeric confidence score is shown in chat. Instead the user gets clickable citations they can verify themselves - a verifiable artifact beats an unfalsifiable number.

6. The product surfaces - held back, deliberately

The workspace presents its intelligence through a set of purpose-built surfaces spanning diagnosis, measurement, decision support, grounded Q&A, document generation, and corpus administration - designed as one narrative arc rather than a pile of tabs.

The surface-level design is not described here. It is the client’s solution, and until I have explicit clearance to discuss it, the detail stays out of a public page - the same rule this whole portfolio follows: nothing published that I am not certain I am allowed to publish. What I can say is generic by 2026 standards anyway: there is a grounded, citation-first Q&A surface, and everything above it links back to the same source documents.

The sections below - the evaluation harness, the deployment constraints, and the engineering decisions - are my own methods and are described in full.

7. Evaluation - the part that makes it defensible

A RAG demo is easy; a RAG system you can defend in front of a government client is not. The difference is measurement.

I treated the pipeline as software under test: every stage has a metric, every metric has a threshold, nothing ships unless all gates pass. 12 gates, re-run on every corpus change, exiting non-zero on failure so it drops into CI or a pre-commit check.

Three properties make it credible rather than self-congratulatory:

1. Reference-free - no hand-written gold answers required, so it runs on every change without a subject-matter expert in the loop.
2. Non-circular - the retrieval test is built from data the retriever never sees.
3. The judge itself is validated - before its scores were trusted at all.

Results - 12/12 green

Why the faithfulness judge is trustworthy

The obvious approach - ask a model to “rate faithfulness 0–1” - is the known-unstable pattern: roughly 0.35–0.47 correlation with human judgement and ±0.3 swings run-to-run. It is a vibe, not a measurement, and shipping it would have meant reporting a number I couldn’t defend.

I implemented decomposed NLI instead: split the answer into atomic standalone claims (pronouns resolved, citation markers stripped), take a binary entailment verdict per claim against the retrieved context with a written reason, and score as supported ÷ total. One noisy float becomes a ratio of discrete, individually-auditable verdicts - the published methods report r ≈ 0.69–0.82 against humans this way, versus ~0.35 pointwise.

The rubric is strict by design: exact match on numbers, units and entities; “absence of evidence is not evidence”; truncated context counts as zero support.

And then I validated the judge - κ = 0.881

Measured the judge’s agreement with an independent model annotator on the identical binary task, across 102 labelled claims:

• Cohen’s κ = 0.881 - 95% CI 0.78–0.96, bootstrap, fixed seed so it reproduces
• Observed agreement 0.941

Cohen’s κ rather than raw agreement, because κ corrects for chance agreement - the rigorous choice, and the one a statistically literate reviewer will look for. On the standard scale (>0.80 = “almost perfect”) the judge is reliable.

This is the number that makes the 0.984 faithfulness score mean anything at all. Without it, 0.984 is one unvalidated model’s opinion of another’s output.

The retrieval gate - non-circular by construction, after rejecting the obvious metric

The dangerous failure in RAG is a specific fact whose source page is never retrieved: it can’t be cited, so the model omits it or invents it.

I had to reject the conventional metric first. The client’s demo questions are whole-deck syntheses (“readiness across six pillars AND initiatives AND challenges”). An independent oracle reading a deck marks nearly all of it relevant - 418 of 434 pages in one flagship document. Page-recall@10 is therefore structurally capped near 10/|deck| and can never reach 0.85 no matter how good retrieval is. The metric is mismatched to the question type; shipping it would have produced a permanently red gate that taught me nothing and eroded confidence in the whole harness.

What I built instead - pinpoint fact-probes: take a number-fact the extractor pulled at ingestion, ask for it directly with a deterministic templated query, and check that the fact’s own source page lands in the top-k. Mean hit@k across ~25 probes per document. 0.96.

Why it cannot be gamed:

• Probes are generated from the fact-card store - never from any retrieval result or answer. The retriever cannot influence what it is tested on.
• Gold is keyed by (document_id, page) tuples, never bare page numbers - “page 44” exists in many documents, so bare integers would produce false hits.
• Queries are templated from the card label - deterministic, no LLM variance inside the gate itself.
• Gold is intersected with currently-retrievable pages, so the gate never demands a page the system deliberately suppresses.

Retrieval adequacy for the synthesis questions is covered transitively by faithfulness, relevance, and citation-validity - which is the correct decomposition, rather than forcing one metric to do both jobs.

Source-anchored extraction faithfulness

Before any answer is ever generated, the extraction is verified: does each fact card actually appear on its own slide? Two-tier - deterministic match against the slide’s native text first, then a constrained judge for values rendered inside charts and images. This catches a model inventing a number at the source, before it can propagate anywhere downstream.

Two bugs measurement caught that a demo never would

The batch-overflow false-fail. A long answer decomposes into 60+ claims. Judging all of them in one model call overflowed the output limit, truncated the JSON, and - via a length guard - collapsed every verdict to 0. A catastrophic silent false-fail: the harness would have reported total failure on the system’s best answers. Caught only by the calibration run, fixed by judging in batches of 12. This is exactly the class of failure that exists right up until you measure.

The tagging prompt-format bug described in §4 - found because the ontology-subset gate exists.

Honesty as a product feature

One generated synthesis scores 0.896 - just under the 0.90 bar - and is flagged in the UI rather than buried. It isn’t wrong: the corpus genuinely reports conflicting baselines across two programmes, and the synthesis shows both. Showing the flag is the point. The bar is real.

The application also ships a documented honest-limitations register for stakeholder conversations: scores reflect what’s in the strategy documents rather than live operational systems; programmes aren’t scored outside their remit; peer comparison exists only where a fair KPI match does; recommendations are draft-quality intelligence with flagged items meaning review the evidence before acting; and coverage is not quality.

Quality held under 3× corpus growth

Scaling 12 → 31 → 35 documents re-ran every gate each time: recall@10 held at 0.96, faithfulness moved 0.982 → 0.984. That is how I know tripling the corpus didn’t regress quality, rather than hoping it didn’t.

8. Deployment

Production deployment onto managed cloud app hosting under genuinely restrictive enterprise networking:

• Private endpoint inbound, public network access disabled - reachable only from the corporate network
• Dedicated virtual network with outbound integration, so the app reaches the internal LLM gateway cross-region while remaining publicly unreachable
• In-region deployment for data residency, with the prior region retained briefly as rollback
• Reproducible smoke verification: every endpoint curl-checked against the private endpoint IP with explicit DNS resolution - including the SSE streaming path, and a specific check proving gateway egress from inside the locked-down network

Two infrastructure problems solved en route: an orphaned-subnet failure blocking VNet integration, and corporate-proxy TLS interception breaking the deployment CLI (resolved by pointing the tooling at the correct CA bundle).

Architecturally, productionising was cheap by design: the stores are files, the API is stateless and read-only, and ingestion is an offline batch producing the artifacts the server serves. Nothing had to be re-architected to deploy.

9. What makes this system different

Six things I’d put in front of a reviewer:

1. Research-led rejection of the obvious architecture. Rejecting GraphRAG on four independent evidenced grounds - and specifically the finding that multi-hop is reasoning-bound rather than retrieval-bound, which redirected effort into agentic decomposition - is the highest-leverage decision in the project. It saved a large build that published evidence says would have underperformed, while keeping the option open at zero cost by parking triples.

2. A judge I validated before trusting. Most RAG systems report an LLM-generated quality score. Almost none validate the scorer. κ=0.881 against an independent annotator, reproducible with a fixed seed, is what converts 0.984 from a claim into a measurement.

3. A metric I designed after rejecting the standard one on principle. Proving page-recall@10 was structurally incapable of passing on synthesis questions, then building non-circular fact-probes generated from a store the retriever cannot influence.

4. Scoring with an evidence gate. A high assessment is structurally impossible without solid documentation behind it: scores are hard-capped when evidence is thin, and no target is ever inferred that a document does not state. The assessment layer is defensible rather than plausible.

5. Generation defined by what it refuses to do. Deterministic ranking rather than model opinion; a dual-layer compliance gate; low-evidence output dropped rather than fabricated; estimates labelled as estimates because the data couldn’t support a real number.

6. Anti-hallucination as architecture, proven at zero. Flagged at extraction, suppressed at retrieval, gated at zero leakage, cited-or-omitted at synthesis, page-scoped so no number is attributed across slides. Four layers, all measured - not a post-hoc filter.

And the meta-point: the client’s trust problems drove the engineering. No invented comparisons, no fabricated targets, estimates labelled as estimates, the flagged sub-threshold synthesis, the limitations register. Each is a small refusal to overclaim, and together they are why the output is usable in front of government leadership.

10. Skills demonstrated

RAG architecture · comparative architecture research with cited rejection · GraphRAG cost/benefit analysis · agentic query decomposition · vision-LLM document extraction · hybrid dense+BM25 retrieval · reciprocal rank fusion · reranking · LLM-as-judge design and calibration · Cohen's κ / inter-annotator agreement · decomposed NLI faithfulness measurement · non-circular evaluation design · multi-stage agentic pipelines with structured hand-offs · deterministic scoring and ranking methodology · evidence-gated assessment design · near-duplicate detection (union-find) · entity resolution and drift mitigation · knowledge-graph construction and salience filtering · ontology design and multi-label classification · corpus contamination guards · staleness fingerprinting · embedded vector + SQL stores · model routing and cost tiering · FastAPI · SSE streaming · React 19 / TypeScript / TanStack Query / Zustand · Azure App Service · private endpoints and VNet integration · data-residency-constrained deployment