Ask someone to deduplicate a document corpus today and there’s a good chance the answer is “embed everything and cluster.” It works. It also costs more than it needs to by a factor I find hard to justify, and it fails in a specific, embarrassing way: it will happily decide that two different match reports about two different football games are the same story, because in embedding space they nearly are.

The dedup pipeline in NewsCrawl is a cascade of four stages, ordered by cost. Each stage only sees what the previous one could not resolve.

Stage Catches Cost per comparison
1. Normalised URL hash Re-crawls, tracking-param variants One index lookup
2. Raw content hash Byte-identical syndication One index lookup
3. SimHash banding Near-duplicates, minor edits Banded lookup, small candidate set
4. BGE-M3 cosine Same event, independently written Vector search

The distribution is what makes it worth building. In my corpus, the overwhelming majority of duplicates are caught by stages 1 and 2 - each a single index lookup. The embedding comparison, which is the expensive one, runs on a small tail.

Stage 1: the URL, normalised properly

The same article reaches you as:

https://example.com/news/story-123?utm_source=twitter&utm_medium=social
https://example.com/news/story-123
http://www.example.com/news/story-123/
https://example.com/news/story-123#comments

Four URLs, one article. Normalisation before hashing:

  • Lowercase scheme and host; strip a leading www.
  • Drop the fragment entirely
  • Remove tracking parameters - the utm_* family, fbclid, gclid, mc_cid, and a per-source list for the ones that are site-specific
  • Sort the surviving query parameters, so ordering is not significant
  • Normalise the trailing slash consistently

Then sha256 the result and make it a unique index.

The one that costs people real money is not keeping a per-source parameter allowlist. Some sites put the article id in a query parameter. Strip it globally and you collapse an entire site’s catalogue into one document. Ask me how I know.

Stage 2: hash the content, not the page

Byte-identical content arrives constantly - syndicated wire copy, a source you crawl through two different index pages. But hashing the raw HTTP response is useless, because every response differs: a timestamp in the footer, a rotating ad slot, a CSRF token, a “related stories” block that reshuffles per request.

So the hash is over extracted content - headline plus body text after boilerplate removal - with whitespace collapsed, quotes and dashes normalised to canonical forms, and Unicode normalised to NFKC.

That last one is not optional for Bangla. The same visible text can be encoded with different codepoint sequences, and without NFKC two identical articles produce two different hashes. This is the kind of bug that hides for months because it silently under-detects: nothing breaks, you just quietly index duplicates.

Stage 3: SimHash, and why banding is the whole point

Now the interesting case. Two documents that differ by a corrected typo, an appended update, or a changed headline. Byte hashes are useless - one character changes the entire digest.

SimHash produces a 64-bit fingerprint where similar documents get similar fingerprints, and similarity is Hamming distance. Documents within about 3 bits of each other are near-duplicates in practice.

The problem: finding all fingerprints within 3 bits of a query is not an index lookup. Comparing against every stored fingerprint is a full scan, which defeats the purpose of putting a cheap stage before the expensive one.

Banding fixes this. Split the 64-bit fingerprint into 4 bands of 16 bits and store all four as separate indexed columns. By the pigeonhole principle, two fingerprints differing in at most 3 bits must share at least one identical band - 3 differing bits cannot cover 4 bands. So:

SELECT id, simhash
FROM   documents
WHERE  band_0 = $1 OR band_1 = $2 OR band_2 = $3 OR band_3 = $4;

That’s four index lookups returning a small candidate set, and you compute exact Hamming distance only on those. A scan of the entire corpus becomes a handful of index probes.

The tokenisation feeding SimHash needs care in a multilingual corpus. Word-level shingles work well for English and poorly for Bangla, where morphology packs more meaning per token. I use character n-grams for the Bangla path and word shingles for English, chosen per document by detected language, with thresholds tuned separately for each. One global threshold across both languages is a compromise that is wrong for both.

Stage 4: embeddings, for the case only meaning can settle

What survives to stage 4 is the genuinely hard case: two outlets covering the same event, written independently. No lexical overlap worth mentioning. Same story.

BGE-M3 embeddings in pgvector, cosine similarity, threshold tuned against a hand-labelled set:

SELECT id, 1 - (embedding <=> $1) AS similarity
FROM   documents
WHERE  published_at > now() - interval '48 hours'
ORDER  BY embedding <=> $1
LIMIT  10;

Two things make this survivable in production.

The time window. Semantic similarity without a temporal bound will match this week’s election story to last year’s. Restricting to a recent window is both a correctness fix and the reason the query is fast.

BGE-M3 specifically - because it is genuinely multilingual, which means a Bangla article and its English counterpart land near each other in the same space. A monolingual model per language would need a separate cross-lingual strategy entirely.

Every stored vector carries the model version that produced it. Comparing vectors across model versions produces confident nonsense, and a model upgrade should be a backfill you can run gradually, not a cliff you fall off.

The first version deleted duplicates. Within a week I had three different threshold bugs, and each one had permanently destroyed data.

Now every suppressed document is retained with a pointer to its canonical version, the stage that made the call, and the score at the time:

CREATE TABLE duplicate_links (
  duplicate_id  bigint PRIMARY KEY REFERENCES documents(id),
  canonical_id  bigint NOT NULL REFERENCES documents(id),
  stage         text   NOT NULL,   -- url | content | simhash | semantic
  score         real,
  linked_at     timestamptz NOT NULL DEFAULT now()
);

Three things this buys:

Reversibility. A bad threshold is an UPDATE away from being undone.

Observability. “How many stage-4 links did we create yesterday, and at what scores?” is the query that tells you a threshold has drifted - before someone reports it.

Ground truth. Every link is a labelled pair. Reviewing the ones near the threshold gives you the evaluation set you need to tune the next version.

What generalises

The pattern is not really about deduplication. It’s: when a decision can be made by several methods of wildly different cost, order them by cost and let the cheap ones absorb the volume. Reserve the expensive method for the residue that actually needs it - and keep the decision reversible, because the first threshold you pick will be wrong.