In plain terms
NewsCrawl reads the news so a person does not have to. It visits news sites continuously in both Bangla and English, pulls out what each article is actually about, notices when six outlets have published the same story with slightly different wording, and keeps only one copy - with the others linked to it.
The result is searchable by meaning: you can ask about a topic in your own words and find the relevant coverage even if the article never used those words.
Why this shape of system matters beyond news: any business that monitors competitors, tracks regulation, watches its own press coverage or ingests supplier feeds has the same three problems - collecting reliably, not paying to process the same thing twice, and finding things later without knowing the exact wording.
The problem
News crawling looks like a simple data-collection problem for about a week. Then it becomes a distributed systems problem.
The moment more than one machine is doing the work, four things go wrong at once. Two machines grab the same article. A machine dies halfway through and its work is either lost or locked away forever. The same story, republished by six outlets, enters the index six times and clutters every result. And an AI call that costs real money gets spent again on content that was already processed.
Each of those has a direct cost - wasted infrastructure, wasted AI budget, and a product that shows the same story six times to a user who wanted one. NewsCrawl is my answer to all four, built as a system I could reason about at 2am.
Work that survives its own workers dying
Instead of adding a separate queueing service, the schedule lives in the main database. A machine claims a batch of work on a timed lease - the equivalent of taking a job from the board and writing your name and a return-by time on it.
UPDATE frontier
SET lease_owner = $1,
lease_expiry = now() + interval '5 minutes',
attempts = attempts + 1
WHERE id IN (
SELECT id FROM frontier
WHERE status = 'ready'
AND next_attempt_at <= now()
ORDER BY priority DESC, next_attempt_at
FOR UPDATE SKIP LOCKED
LIMIT $2
)
RETURNING id, url, source_id;Two machines running this at the same instant never collide - the second simply skips whatever the first has already claimed and takes the next batch.
The lease is what makes it crash-safe. A machine that dies never hands anything back, so a sweeper returns expired leases to the queue and tries them again later - with increasing delays, so a site that is genuinely down degrades into patient retries rather than a hammering loop.
Two practical wins from this: a hard crash costs one lease interval rather than a day of collection, and the architecture stays one system smaller. Not running a separate message broker means one less thing to license, monitor, patch and be woken up by. The full design is in Leases Over Locks.
Removing duplicates without paying to compare everything
The obvious approach - analyse every article with AI and compare every pair - gets expensive faster than it gets accurate. So duplicate detection is a cascade ordered by cost, and each stage only sees what the previous one could not settle:
| Stage | Catches | Cost |
|---|---|---|
| Normalised web address | Re-visits, tracking-link variants | Index lookup |
| Exact content fingerprint | Byte-identical republication | Index lookup |
| Near-match fingerprint | Edited headlines, small rewrites | Banded lookup |
| Meaning comparison | The same story, independently written | Vector search |
Roughly nine in ten duplicates die in the first two stages, each of which costs a single index lookup. Only genuinely ambiguous pairs ever reach the expensive comparison. In budget terms: the costly method runs on the small remainder where it actually changes the answer.
The important design decision: duplicates are linked, not deleted. Every suppressed article keeps a pointer to the version it was merged into and a record of which stage made the call. When a threshold turns out to be wrong - and thresholds are always wrong at first - correcting it is an adjustment rather than a data-loss incident. Reversibility is what makes it safe to tune the system in production.
The full method is written up in Four Stages of Deduplication.
The processing line
Cleaning, AI extraction and indexing run as separate stages, each reading from its own queue and handing on to the next. Failures retry with growing delays, and anything that exhausts its attempts is parked with the full error attached - visible to a human rather than silently discarded.
Because the delivery guarantee is “at least once”, the same instruction arriving twice is a certainty rather than an edge case. Every stage is written so a repeat produces the same result. A stalled message is picked up again after its window expires, so a stage that dies mid-work does not leave an article stuck forever.
The plain-language version: nothing is quietly lost, and nothing is quietly done twice. Details in At-Least-Once Is a Promise You Have to Keep.
Treating AI as a supplier, not an oracle
AI extraction is engineered like any other unreliable external service:
- Checked before it is trusted. Every response is validated against a declared structure. Malformed output triggers a correction retry - it is never saved and discovered later by a user.
- A second supplier ready. When the primary provider rate-limits, the pipeline moves to another instead of stalling. Availability stops depending on one vendor’s capacity.
- Costed per item. Tokens and spend are recorded per call and per article, so “is this source worth crawling?” is a query rather than a guess - and the monthly AI bill is predictable before it arrives.
- Upgradeable without a rebuild. Every stored meaning-vector records which model produced it, so upgrading the model is a gradual backfill rather than a stop-the-world migration.
Bangla and English run through the same pipeline. Getting that right meant choosing thresholds and a model that survive two very different writing systems - which is what lets one product serve a bilingual audience instead of two half-products.
More on the guardrails: Let the Agent Propose, Not Write.
Collecting politely
Fetching is HTTP-first, with a full browser used only for sites that refuse to work otherwise - browsers are dramatically more expensive to run, so they are a fallback, not a default. Both paths share one per-site rate budget, so the expensive path cannot accidentally hammer a site the cheap path was being careful with.
Every outbound request is checked against the site’s own rules and against safety guards before a connection opens. When re-visiting, the system asks “has this changed since I last looked?” and treats “no” as a free confirmation - the cheapest possible visit.
This is reputational as much as technical: a collector that behaves badly gets blocked, and a blocked collector collects nothing.
The result
- ~90% of duplicate content is removed by checks that cost an index lookup, keeping the expensive analysis for the cases that need it.
- A crashed machine costs minutes, not a lost day of collection and a manual clean-up.
- AI spend is visible per article, which turns “how much does this cost to run?” into a number.
- One provider failing does not stop the product.
- Bangla and English readers get one coherent picture instead of two disconnected feeds.
What it demonstrates
The interesting parts of this project are all failure-mode design: what happens when a machine dies, when a provider rate-limits, when a threshold is wrong, when the same story arrives six times. The happy path was a weekend. Everything since has been making the unhappy paths boring - which is exactly what a business is paying for when it asks for something reliable.