The problem

News crawling looks like a scraping problem for about a week. Then it becomes a distributed systems problem.

The moment more than one worker is fetching, four things go wrong at once. Two workers grab the same URL. A worker dies mid-fetch and its URLs are lost - or worse, locked forever. The same story, republished by six outlets with slightly different wording, enters the index six times. And an LLM call that costs real money gets retried on content that was already processed.

NewsCrawl is my answer to all four, built as a system I could actually reason about at 2am.

Frontier: leases, not locks

The scheduler is a PostgreSQL table, not a queue service. Workers claim work with a single statement:

UPDATE frontier
SET    lease_owner  = $1,
       lease_expiry = now() + interval '5 minutes',
       attempts     = attempts + 1
WHERE  id IN (
  SELECT id FROM frontier
  WHERE  status = 'ready'
    AND  next_attempt_at <= now()
  ORDER  BY priority DESC, next_attempt_at
  FOR UPDATE SKIP LOCKED
  LIMIT  $2
)
RETURNING id, url, source_id;

FOR UPDATE SKIP LOCKED is doing the heavy lifting. Two workers running this concurrently never collide - the second simply skips the rows the first has locked and takes the next batch. No Redlock, no external broker, no coordination service to operate.

The lease is what makes it crash-safe. A worker that dies never releases anything; a reaper sweeps expired leases back to ready with exponential backoff on next_attempt_at, so a systematically failing source degrades into slow retries instead of a hot loop. The cost of a hard crash is bounded by one lease interval.

I wrote up the full design in Leases Over Locks.

Deduplication that respects its own cost budget

The naive approach - embed everything, compare everything - is quadratic and expensive. So dedup is a cascade, ordered by cost, and each stage only sees what the previous one couldn’t resolve:

Stage Catches Cost
Normalised URL hash Re-crawls, tracking-param variants Index lookup
Raw content hash Byte-identical syndication Index lookup
SimHash banding Near-duplicates, edited headlines Banded lookup
BGE-M3 cosine Same story, independently written Vector search

Roughly nine in ten duplicates die in the first two stages, which cost a single index lookup each. Only genuinely ambiguous pairs reach the embedding comparison.

The important design decision: duplicates are linked, not deleted. Every suppressed document keeps a pointer to its canonical version and the stage that made the call. When a threshold turns out to be wrong - and thresholds are always wrong at first - the merge is reversible instead of a data-loss incident.

The processing plane

Cleaning, LLM extraction and embedding run as separate Redis Streams consumer groups. Each stage reads from its own stream, acknowledges on success, and writes to the next.

At-least-once delivery is the guarantee, so duplicate delivery is a certainty, not an edge case. Every handler is keyed on a deterministic content id and writes with an upsert. Failures get delayed retries with backoff; messages that exhaust their attempts land in a dead-letter stream with the full error context attached. A reclaim pass picks up messages pending longer than the stage’s visibility window - the stream equivalent of lease reaping.

The model layer

LLM extraction is treated like any other unreliable network dependency:

  • Schema-validated output. Every response is parsed against a declared schema. Invalid output triggers a repair retry, never an unchecked database write.
  • Provider fallback. When the primary provider rate-limits, the pipeline degrades to a secondary rather than stalling.
  • Cost accounting. Tokens and cost are recorded per call and per document, so “is this source worth crawling?” is a query, not a guess.
  • Versioned embeddings. Every vector carries its model version. Re-embedding on a model upgrade is a backfill, not a migration cliff.

Bangla and English run through the same pipeline. Getting that right meant choosing thresholds that survive very different tokenisation and morphology - BGE-M3 was picked specifically because it holds up across both scripts.

Fetching politely

Scrapy handles HTTP-first fetching; Playwright is the fallback for sources that only render on the client. Both share a Redis-backed politeness budget keyed per host, so the browser path can’t accidentally hammer a site the HTTP path was being careful with.

Every outbound URL passes robots and SSRF guards before a socket opens. Recrawls send conditional headers and treat a 304 as a free confirmation that nothing changed - the cheapest possible crawl.

What it demonstrates

The interesting parts of this project are all failure-mode design: what happens when a worker dies, when a provider rate-limits, when a threshold is wrong, when the same story arrives six times. The happy path was a weekend. Everything since has been making the unhappy paths boring.