NewsCrawl
Reads the news in Bangla and English all day, removes the duplicates, and answers questions about what it found.
- ~90%of duplicates removed at near-zero cost
- 2 languagesBangla and English, one pipeline
- No lost workwhen a machine dies mid-job
- Per-articleAI cost tracked and visible
- Python
- FastAPI
- Scrapy
- Playwright
- Redis Streams
- PostgreSQL
- pgvector
- BGE-M3
- +4
