Media Intelligence Platform

Automated media monitoring across regional and global press: ingests, classifies, clusters into emergent themes, resolves bilingual entities, and generates branded intelligence reports - so analysts see the story, not a list of articles.

1. The problem

A consulting practice needed to track media coverage across nine practice areas and 24 industry sectors spanning GCC regional and global business press. The manual alternative - analysts skimming feeds - fails on three counts: it doesn’t scale, it produces article lists rather than narratives, and it silently misses cross-cutting stories that only become visible in aggregate.

Two problems specific to this region shaped the build:

• Bilingual coverage. The same entity appears in Arabic and English sources under different names. Naive entity extraction produces two disconnected nodes for one organisation, fragmenting exactly the signal you’re trying to see.
• Theme, not article, is the unit of insight. Twelve articles about one developing story should present as one theme with twelve sources - not twelve rows competing for attention.

2. The intelligence pipeline

Ingest → Classify → Cluster → Assign Themes → Analyze

Orchestrated as a single sequence, with every stage independently runnable - which matters enormously for debugging a multi-stage NLP pipeline and for re-running one expensive step without repeating the others.

Ingest

Pulls from 7+ configured sources via RSS and sitemap connectors, extracts full article text, and deduplicates on a SHA-256 hash of the normalised URL with a unique index - so the same story syndicated across outlets doesn’t multiply.

Classify

Generates 3072-dimension embeddings per article, then computes cosine similarity against hand-written reference texts for each practice and sector, assigning any that clear a threshold - top match at 0.25 with additional matches needing 0.35, capped at three practices; sectors at 0.30, capped at five. Multi-label by design - a story about telecom infrastructure investment legitimately belongs to several practices, and forcing a single label would destroy that.

Cluster - adaptive, not fixed-parameter

This is more sophisticated than a textbook kNN clustering, and the sophistication was forced by the data:

• Mutual k-NN at k=15, not raw k-NN - an edge survives only if each article appears in the other’s top-15. Mutuality is what suppresses the hub-article problem where one broad piece links to everything.
• Mode-dependent thresholds rather than one constant: ~0.68 for full-corpus runs, ~0.45 for gate-filtered runs, recalibrated because a filtered high-quality corpus has genuinely different similarity distribution than raw newspaper volume. A single tuned constant would have been wrong for one of the two modes.
• Union-find merging of overlapping neighbourhoods - the step that turns fragments into themes.
• Recursive mega-cluster splitting: any cluster exceeding a size ceiling is re-clustered at progressively tighter thresholds until it breaks into genuine sub-clusters where no single one dominates. Without this, one giant “business news” blob swallows the corpus and the theme view becomes useless.
• Weighted representative selection for LLM labelling - 0.7 semantic proximity to centroid plus 0.3 recency decay, so a theme is named from articles that are both central and current.

No approximate vector index is used, deliberately: at 3072 dimensions the standard index type caps out below that, so clustering does exact brute-force nearest-neighbour search. Correct at this corpus size, and documented as the reason rather than left as an omission.

Assign themes

Each cluster goes to an LLM to generate a theme name, summary, and keywords - so themes are human-legible rather than “cluster 7”.

Analyze

Per-article LLM analysis extracting sentiment, a 0–100 relevance score, strategic implications, key quotes, highlighted passages, and named entities with canonical English names.

The four-tier relevance funnel - the largest piece of logic in the system

The pipeline above is the original design. What the system actually became, after the product question shifted from “track all coverage” to “surface genuine thought leadership and strategic signal”, is a cost-ordered cascade that puts cheap filters in front of expensive models:

The ordering is the engineering: the expensive model only ever sees articles that survived three progressively more discriminating cheap filters, which is what makes running a frontier model over a daily news firehose economically viable at all. A schema.org type signal parsed from the raw page feeds in as a calibration prior, and gate prompts are version-tagged so a threshold change is attributable.

Separately, a MinHash/LSH near-duplicate detector (128 permutations, banded, 5-gram shingling, Jaccard threshold ~0.70 over a rolling window) sits alongside the URL-hash dedup - because syndicated wire copy is near-identical rather than URL-identical, and only one of those two mechanisms catches it.

Eleven operational scripts exist for re-gating, auditing and backfilling already-ingested articles against changed thresholds - the unglamorous tooling that makes threshold tuning an experiment you can run rather than a rebuild you fear.

3. Data model

PostgreSQL 18 with pgvector, across 14 tables. The schema is explicitly relational rather than document-shaped, because the valuable queries are all joins - which themes touch this practice, via which articles, mentioning which entities.

Notable modelling decisions:

• entities.canonical_id - the extraction model returns both the name as written and a canonical English form. One row is upserted per canonical name, any differing surface form becomes an alias row pointing at it, and article-entity links always target the canonical row, never the alias - so Arabic and English mentions converge on one node by construction rather than by a later merge pass.

  The honest limitation: the merge is exact (case-insensitive) string equality on the model’s own canonicalisation output. There is no embedding similarity or fuzzy matching behind it. If the model canonicalises inconsistently across articles - a person’s name with and without a title, say - those will not merge. That is a real fragility in the bilingual dedup story, and the fix (embedding-similarity clustering over canonical forms, or a canonical-profile registry) is known but unbuilt.

• Join tables carry data, not just keys - article_practices holds a relevance score, article_themes holds cluster distance, theme_practices holds article counts. The relationship strength is information.

• pipeline_runs logs every execution with per-step timing and counts, so a slow or failing stage is diagnosable after the fact rather than by re-running blind.

• Vector column on articles with cosine distance, enabling both classification and clustering off one embedding.

4. Surfaces

• Dashboard - coverage and signal across practices
• Insights - the insight cards produced by tier 4 of the funnel
• Theme view - emergent stories with their constituent sources
• Article detail - full analysis with entities and evidence
• Pipeline monitoring - run status, per-stage history, and a funnel view showing how many articles survived each tier
• Report generation - brand-styled intelligence reports produced from aggregated data via LLM, with PDF export
• Research sessions - ad-hoc deep dives that fetch and extract from partner URLs (including PDFs), cluster the results, and generate a focused report on demand; backed by their own tables
• Source administration - enable/disable and configure media sources

Two honest notes on the frontend. Entity data is fully modelled and populated in the database, but the network-graph and regional-map visualisations are not built - routes for a separate practices view, an intelligence view and a whitespace view exist only as redirects to the dashboard, the residue of features consolidated away. And the platform is not under version control in this working copy, which is a genuine process gap rather than a design choice.

5. Engineering approach

Monorepo, three workspaces: API (Hono + Drizzle ORM), web (Vite + React 19), and a shared types package used by both ends so the contract is defined once rather than duplicated. In practice that shared package is TypeScript interfaces only - compile-time safety, not runtime validation. Worth stating plainly: the boundary is typed, not validated, and adding schema validation at the API edge is an obvious unbuilt improvement.

Hono over a heavier framework for a lightweight, fast, edge-capable API layer; Drizzle for type-safe SQL that keeps the actual query shape visible rather than hidden behind an ORM abstraction - which matters when the queries are vector-similarity joins.

All model calls route through the firm’s internal gateway (Anthropic-compatible, cloud-routed) rather than public APIs.

6. What I’d highlight

• Bilingual entity resolution via canonical IDs - a small schema decision that determines whether the entire entity-network feature works or produces fragmented noise.
• kNN + union-find clustering - recognising that raw nearest-neighbour output needs a merge step to produce coherent themes.
• Theme as the unit of insight, which reframes the product from “feed reader” to “intelligence tool”.
• Shared Zod schema package - one source of truth for the API contract across a TypeScript monorepo.
• Observable pipeline - per-run, per-step logging designed in from the start, because multi-stage NLP pipelines are otherwise undebuggable.
• Prioritisation under stakeholder pressure: heat maps and white-space analysis were explicitly dropped in favour of opinion filtering and source control. Choosing what not to build is part of the work.

7. Skills demonstrated

multi-stage NLP pipeline design · vector embeddings and semantic classification · kNN clustering with union-find merging · bilingual entity resolution · PostgreSQL + pgvector · relational modelling for graph-shaped queries · Drizzle ORM · Hono · TypeScript monorepo with shared Zod contracts · React 19 / Recharts · network graph and geospatial visualisation · LLM report generation with PDF export · pipeline observability