# Md. Musfiqur Rahaman — complete site content > Every case study and article from https://musfiqdehan.com, inlined as Markdown so an agent can read the > whole site in one request. The index version, with links and one-line summaries, is at > https://musfiqdehan.com/llms.txt. Generated 2026-09-30. Source of truth is the live site; URLs are canonical. --- # Projects ## NewsCrawl - URL: https://musfiqdehan.com/projects/newscrawl - Tagline: Reads the news in Bangla and English all day, removes the duplicates, and answers questions about what it found. - Context: Personal R&D platform (2025 - Present, status: live) - Primary discipline: backend - Stack: Python, FastAPI, Scrapy, Playwright, Redis Streams, PostgreSQL, pgvector, BGE-M3, Next.js, Docker, Prometheus, Grafana - Metrics: ~90% of duplicates removed at near-zero cost; 2 languages Bangla and English, one pipeline; No lost work when a machine dies mid-job; Per-article AI cost tracked and visible - Links: Source ; Live ### What I did - Collects news continuously in Bangla and English, so a reader or analyst sees one coherent picture instead of two separate feeds. - Recognises the same story republished by different outlets and shows it once - roughly nine in ten duplicates are caught by checks that cost almost nothing to run. - A machine crashing mid-job costs minutes of delay, not lost work - unfinished jobs return to the queue automatically. - AI spend is tracked per article, so "is this source worth reading?" is a number rather than an opinion. - If one AI provider rate-limits or goes down, the pipeline moves to another instead of stopping. - Search understands meaning, so a question finds the right article even when it uses none of the article's words. - Runs on the database it already had rather than adding a separate queueing service - fewer moving parts, lower running cost. ## In plain terms NewsCrawl reads the news so a person does not have to. It visits news sites continuously in both Bangla and English, pulls out what each article is actually about, notices when six outlets have published the same story with slightly different wording, and keeps only one copy - with the others linked to it. The result is searchable by meaning: you can ask about a topic in your own words and find the relevant coverage even if the article never used those words. **Why this shape of system matters beyond news:** any business that monitors competitors, tracks regulation, watches its own press coverage or ingests supplier feeds has the same three problems - collecting reliably, not paying to process the same thing twice, and finding things later without knowing the exact wording. ## The problem News crawling looks like a simple data-collection problem for about a week. Then it becomes a distributed systems problem. The moment more than one machine is doing the work, four things go wrong at once. Two machines grab the same article. A machine dies halfway through and its work is either lost or locked away forever. The same story, republished by six outlets, enters the index six times and clutters every result. And an AI call that costs real money gets spent again on content that was already processed. Each of those has a direct cost - wasted infrastructure, wasted AI budget, and a product that shows the same story six times to a user who wanted one. NewsCrawl is my answer to all four, built as a system I could reason about at 2am. ## Work that survives its own workers dying Instead of adding a separate queueing service, the schedule lives in the main database. A machine claims a batch of work on a **timed lease** - the equivalent of taking a job from the board and writing your name and a return-by time on it. ```sql UPDATE frontier SET lease_owner = $1, lease_expiry = now() + interval '5 minutes', attempts = attempts + 1 WHERE id IN ( SELECT id FROM frontier WHERE status = 'ready' AND next_attempt_at <= now() ORDER BY priority DESC, next_attempt_at FOR UPDATE SKIP LOCKED LIMIT $2 ) RETURNING id, url, source_id; ``` Two machines running this at the same instant never collide - the second simply skips whatever the first has already claimed and takes the next batch. The lease is what makes it crash-safe. A machine that dies never hands anything back, so a sweeper returns expired leases to the queue and tries them again later - with increasing delays, so a site that is genuinely down degrades into patient retries rather than a hammering loop. **Two practical wins from this:** a hard crash costs one lease interval rather than a day of collection, and the architecture stays one system smaller. Not running a separate message broker means one less thing to license, monitor, patch and be woken up by. The full design is in [Leases Over Locks](https://musfiqdehan.com/blog/lease-based-url-frontier-postgres). ## Removing duplicates without paying to compare everything The obvious approach - analyse every article with AI and compare every pair - gets expensive faster than it gets accurate. So duplicate detection is a cascade ordered by cost, and each stage only sees what the previous one could not settle: | Stage | Catches | Cost | | --- | --- | --- | | Normalised web address | Re-visits, tracking-link variants | Index lookup | | Exact content fingerprint | Byte-identical republication | Index lookup | | Near-match fingerprint | Edited headlines, small rewrites | Banded lookup | | Meaning comparison | The same story, independently written | Vector search | Roughly nine in ten duplicates die in the first two stages, each of which costs a single index lookup. Only genuinely ambiguous pairs ever reach the expensive comparison. **In budget terms: the costly method runs on the small remainder where it actually changes the answer.** The important design decision: duplicates are **linked, not deleted**. Every suppressed article keeps a pointer to the version it was merged into and a record of which stage made the call. When a threshold turns out to be wrong - and thresholds are always wrong at first - correcting it is an adjustment rather than a data-loss incident. Reversibility is what makes it safe to tune the system in production. The full method is written up in [Four Stages of Deduplication](https://musfiqdehan.com/blog/four-stage-deduplication-pipeline). ## The processing line Cleaning, AI extraction and indexing run as separate stages, each reading from its own queue and handing on to the next. Failures retry with growing delays, and anything that exhausts its attempts is parked with the full error attached - visible to a human rather than silently discarded. Because the delivery guarantee is "at least once", the same instruction arriving twice is a certainty rather than an edge case. Every stage is written so a repeat produces the same result. A stalled message is picked up again after its window expires, so a stage that dies mid-work does not leave an article stuck forever. The plain-language version: **nothing is quietly lost, and nothing is quietly done twice.** Details in [At-Least-Once Is a Promise You Have to Keep](https://musfiqdehan.com/blog/at-least-once-redis-streams). ## Treating AI as a supplier, not an oracle AI extraction is engineered like any other unreliable external service: - **Checked before it is trusted.** Every response is validated against a declared structure. Malformed output triggers a correction retry - it is never saved and discovered later by a user. - **A second supplier ready.** When the primary provider rate-limits, the pipeline moves to another instead of stalling. Availability stops depending on one vendor's capacity. - **Costed per item.** Tokens and spend are recorded per call and per article, so "is this source worth crawling?" is a query rather than a guess - and the monthly AI bill is predictable before it arrives. - **Upgradeable without a rebuild.** Every stored meaning-vector records which model produced it, so upgrading the model is a gradual backfill rather than a stop-the-world migration. Bangla and English run through the same pipeline. Getting that right meant choosing thresholds and a model that survive two very different writing systems - which is what lets one product serve a bilingual audience instead of two half-products. More on the guardrails: [Let the Agent Propose, Not Write](https://musfiqdehan.com/blog/let-the-agent-propose-not-write). ## Collecting politely Fetching is HTTP-first, with a full browser used only for sites that refuse to work otherwise - browsers are dramatically more expensive to run, so they are a fallback, not a default. Both paths share one per-site rate budget, so the expensive path cannot accidentally hammer a site the cheap path was being careful with. Every outbound request is checked against the site's own rules and against safety guards before a connection opens. When re-visiting, the system asks "has this changed since I last looked?" and treats "no" as a free confirmation - the cheapest possible visit. This is reputational as much as technical: a collector that behaves badly gets blocked, and a blocked collector collects nothing. ## The result - **~90% of duplicate content** is removed by checks that cost an index lookup, keeping the expensive analysis for the cases that need it. - **A crashed machine costs minutes**, not a lost day of collection and a manual clean-up. - **AI spend is visible per article**, which turns "how much does this cost to run?" into a number. - **One provider failing does not stop the product.** - **Bangla and English readers** get one coherent picture instead of two disconnected feeds. ## What it demonstrates The interesting parts of this project are all failure-mode design: what happens when a machine dies, when a provider rate-limits, when a threshold is wrong, when the same story arrives six times. The happy path was a weekend. Everything since has been making the unhappy paths boring - which is exactly what a business is paying for when it asks for something reliable. --- ## SuperMart - URL: https://musfiqdehan.com/projects/supermart - Tagline: Retail software with a full online store, a till that keeps selling when the internet drops, and one stock truth for both. - Context: Commercial SaaS product (2025 - Present, status: live) - Primary discipline: fullstack - Stack: Python, Django, DRF, PostgreSQL, django-tenants, Next.js, React, TypeScript, Tauri 2.0, Rust, SQLite, Docker, SSLCommerz - Metrics: Web + POS one platform for shop and online; Keeps selling when the internet drops; No double charges on retried or replayed sales; Per-shop stock, staff and takings - Links: Platform ; Store demo ### What I did - A complete ecommerce storefront shares the product catalogue with in-shop POS - one barcode, one price list, stock scoped per branch. - Shops keep selling during internet outages on the desktop till - queues never stop, and no sale is lost to a connection the business does not control. - Offline sales reconcile exactly once when the connection returns, so a customer is never charged twice and no takings go missing from the books. - Two cashiers scanning the last item in stock cannot both sell it - the second is told immediately instead of the shop discovering it at stock-take. - Each retail business runs in its own isolated data space; each shop sees only its own stock, sales and staff. - Permissions are described once and enforced everywhere - a new role takes minutes to define, and forgetting one locks a page rather than exposing it. - Subscription billing and local card payments through SSLCommerz, with duplicate payment callbacks settling the same invoice instead of creating a second one. ## In plain terms SuperMart is the software a shop uses to sell things and the software the head office uses to run the shops: the till at the counter, the online storefront customers browse at home, the stock behind both, the staff who work there, and the reports the owner reads on Sunday night. Its defining features are unglamorous and worth real money: **when the internet goes down, the till keeps working**, and **the website and the counter sell from the same catalogue** without two spreadsheets drifting apart. **Why it matters commercially:** in a shop, an hour offline is an hour of refused customers. Retail internet in Bangladesh is not something a shop owner controls, and "sorry, our system is down" is a sentence that sends people to the shop next door. Selling online without a separate inventory system means the owner is not reconciling web orders against a notebook at the back. ## The problem A point of sale is the least forgiving screen in business software. Every other page in a business system can show a loading spinner and be forgiven. The till cannot: there is a customer holding cash, a queue behind them, and a shop whose revenue for the next ten minutes depends entirely on whether the software agrees to take the money. SuperMart has to hold five promises at once, and they pull against each other: - **Separate businesses must not see each other.** Several retail companies share one platform. - **Separate shops must not see each other.** Inside one company, a cashier has no business reading another branch's stock or takings. - **Stock must not go negative.** Two cashiers scanning the last unit is a normal Tuesday, not a rare edge case. - **The till must keep selling offline.** Connectivity is not a guarantee, and lost sales are the one failure the product exists to prevent. - **Online and in-store must agree on what is for sale.** Split systems mean overselling the last unit on the web while someone is holding it at the counter. The last two promises are what make this genuinely hard, because selling offline and selling online both mean acting without perfect knowledge of the current truth. Everything below is the accounting of that tension. ## Web store and counter on one platform The ecommerce piece is not a separate product bolted on. Company-wide product data - names, barcodes, prices, images - lives once. Stock levels and sales belong to a branch. A customer orders online; fulfilment and deduction follow the same rules as a walk-in sale, with branch scoping and permission checks on every mutation. For the business, that means launching a web channel is turning on a surface the platform already understands, not exporting CSVs to a third-party cart. For engineering, it means checkout, inventory, and RBAC were designed once and reused - the same idempotency and locking story whether the client is a browser tab or a Tauri till. ## Keeping businesses and shops apart **Between businesses: a separate database space each.** There is a hard split between platform-level information - the subscription packages on offer, the list of customer companies - and each company's own operational data. A request resolves which company it belongs to from the web address it arrived on, and sees only that company's data for the rest of its life. Company identifiers are random rather than sequential, so a number in a web address reveals neither how many customers exist nor who came first. The rule that keeps this honest: **no direct links across the boundary.** The moment one company's records point at another's, the separation is decorative rather than real. Anything genuinely shared lives in the platform space and is read, never joined across. I wrote up the costs of this approach - more migrations to run, more connections to manage - in [Schema-per-Tenant Django](https://musfiqdehan.com/blog/schema-per-tenant-django-multi-tenancy). **Inside a business: per-shop scoping.** A retail company is a head office and several shops. The product catalogue is company-wide - a barcode means the same thing everywhere - but stock levels, sales and staff assignment belong to a specific shop. Every change that touches shop data passes an explicit permission check rather than a filter someone might forget: ```python def assert_user_branch_access(user, branch_id): scope = resolve_branch_scope(user) # None == head office, unrestricted if scope is not None and branch_id not in scope: raise PermissionDenied("PERMISSION_DENIED") ``` The threat is not hypothetical. A cashier who edits a shop identifier in a web request, or a desktop till whose data has been tampered with, must be refused by the server - not merely hidden by the interface. What the user sees is a convenience; the server-side check is the actual control. ## Permissions described once, enforced everywhere The naive approach to permissions is a chain of "if this person is a manager…". It works until the third customer asks for a role that does not exist yet - and by then the logic is scattered across two hundred screens. SuperMart instead keeps a single register of every capability in the product (`pos`, `inventory`, `reports`, `billing`, ecommerce modules) with a level for each (`view`, `edit`, `full`). Roles are built by composing those grants. The backend enforces it: ```python class SaleViewSet(ModelViewSet): permission_classes = [HasFeaturePermission.require("pos", "edit")] ``` …and the interface hides or shows the matching page from the same register, so a page, its menu entry and its data access all come from one declaration. Two business outcomes follow. Defining a new role for a customer takes minutes rather than a release cycle. And because anything not explicitly granted resolves to **no access**, forgetting to wire up a permission produces a locked page - a visible, reportable bug - rather than an open one nobody notices until it is a breach. ## Billing across the boundary Subscription billing is where multi-tenancy stops being an abstraction and starts being money. Platform-level records - packages, limits, whether a customer is on trial, active or overdue - live on the platform side. Each company's own payment gateway credentials live on theirs, because those are their secrets. Payments run through SSLCommerz, and payment notifications are treated as something that **can and will arrive more than once**. A duplicated notification settles the same invoice instead of creating a second one, which is the difference between correct books and a customer support conversation. Package limits are enforced in one place, so a limit applies whether the action came from the dashboard, the desktop till, the storefront, or the API. ## Checkout: the moment everything else exists to protect Taking payment is a single all-or-nothing operation, and the order of steps inside it is the whole design: ```python @transaction.atomic def checkout(*, user, branch_id, lines, payments, idempotency_key): assert_user_branch_access(user, branch_id) existing = Sale.objects.filter(idempotency_key=idempotency_key).first() if existing: return existing # replay - no second decrement levels = (StockLevel.objects .select_for_update() .filter(branch_id=branch_id, product__in=[l.product for l in lines])) # ... verify quantities, else raise InsufficientStock sale = Sale.objects.create(..., idempotency_key=idempotency_key) record_payments(sale, payments) apply_stock_movements(sale, levels) return sale ``` Two different business problems, two different mechanisms - and confusing them is the classic mistake: - **Locking solves two people at once.** Two cashiers, one unit of stock. The second transaction waits for the first, then sees the true quantity and fails cleanly with a clear message. The cashier tells the customer straight away instead of the shop finding out at stock-take. - **A unique request key solves retries.** The same sale submitted twice - a timeout, a double-click, a resend after a crash - resolves to the same single sale. Nobody is charged twice; stock is never deducted twice. Because stock is held per shop, one branch's busy checkout never slows another's. That is the payoff of the two-tier data model. ## The offline till The desktop till is a small, fast native application with its own local database. The choice was driven by what a shop actually runs: a modest computer that also has a browser and a spreadsheet open. The design rule is one sentence: **the till proposes, the server decides.** Online, the till is a thin window onto the platform. Offline, it serves what it already knows from its local copy and writes each completed sale - the full request, plus a unique key generated on the spot - into a local outbox marked as pending. Local stock figures shown to the cashier are a best guess for display; the server remains the authority. On reconnect, the outbox is drained in order and each pending sale is sent **unchanged**, original key included. The server's duplicate check turns every repeat into a no-op that returns the original sale. If an item genuinely sold out online while the till was offline, that row is marked failed with the server's reason and shown to the cashier - never silently dropped, because a deleted sale is a sale nobody can audit. Local data is tied to the specific business and shop and wiped when either changes, so one company's pending sales can never be posted under another's account. The full design - key lifecycle, crash windows, conflict handling and what I would change - is written up in [Selling With the Wi-Fi Down](https://musfiqdehan.com/blog/offline-first-pos-tauri-sync-outbox). ## The trade-off I accepted, deliberately **Staying open beats being perfectly accurate.** Offline stock numbers can be briefly wrong, and I chose that on purpose: the alternative was refusing sales whenever the till was uncertain, which is exactly the failure the product exists to prevent. The mitigations are honesty rather than cleverness - a visible offline indicator, an explicit list of sales that failed to sync which the cashier must resolve, and a fast refresh the moment the connection returns. The shop is never misled about what the system knows. ## The result - **Online and in-store** share one catalogue and one permission model instead of two systems that drift. - **Sales continue through internet outages**, so revenue depends on the shop being open rather than on the connection being up. - **Every offline sale lands exactly once** - no lost takings at close of day, no customer charged twice. - **Overselling is prevented at the moment of sale**, not discovered at stock-take. - **A new retail customer** joins the existing platform with isolated data and their own branding, no separate installation to maintain. - **New staff roles take minutes** to define, and a forgotten permission fails safely closed. ## What it demonstrates Five hard problems that all have to hold at the same time: customer separation enforced by the database rather than by discipline, permissions that compose instead of branching on job titles, correctness when two people act in the same second, a reliable agreement between a desktop till, an unreliable network and a server, and a web channel that does not fork the inventory model - the kind of work that decides whether retail software is trusted with the money. --- ## Wufud - URL: https://musfiqdehan.com/projects/wufud - Tagline: Hajj and Umrah agencies launch a branded booking platform on their own subdomain — with seat integrity, branch RBAC and finance in BDT and SAR. - Context: Multi-tenant pilgrimage booking SaaS (2025 - Present, status: live) - Primary discipline: fullstack - Stack: TypeScript, NestJS, MikroORM, PostgreSQL 17, Redis, BullMQ, Next.js, React 19, Tailwind 4, Traefik, Argon2, JWT, Docker - Metrics: 0% commission on pilgrim payments; 14-day trial with instant tenant provisioning; FOR UPDATE seat holds under concurrency; 2 gateways SSLCommerz and Stripe plus branch POS - Links: Live platform ; Demo agency ; Source ### What I did - Each agency runs in its own PostgreSQL schema — host and slug resolve tenant context before any query runs, so one client's pilgrims never appear in another's reports. - Seat booking uses row locks and database checks so concurrent pilgrims cannot oversell the last place on a package tier. - Payment webhooks are handled idempotently — duplicate or out-of-order provider events cannot double-charge or resurrect released seats. - Agencies collect online via SSLCommerz or Stripe, or record branch cash with recorder/approver separation for auditability. - Platform operator console provisions tenants, plans and features; agencies get custom domains, installments, stock issues and SAR/BDT vendor accounting on higher tiers. ## In plain terms **Wufud** (وفود — delegations) is software for Hajj and Umrah travel agencies. Each agency gets a **branded storefront** to publish packages, pilgrims browse and book online, staff run branches from a **dashboard**, and the **platform operator** sells subscriptions — not a cut of every seat sold. Agencies worldwide can start on a **unique subdomain** (for example `demo.wufud.musfiqdehan.com`) and graduate to a **custom domain** with automated TLS at the edge. Pilgrims pay in full or on **installments**; finance tracks **BDT and SAR** vendor costs; branch managers see only their office's bookings when RBAC demands it. **Why it matters commercially:** pilgrimage travel is high-trust and high-stakes — overselling a tier, losing a payment webhook, or showing one branch another's pilgrims destroys reputation fast. Wufud is built around those stakes, not around a generic CRUD booking form. ## The problem An agency selling sacred travel is really running: - A **catalog** of dated packages with tiers, room types and finite seats. - A **sales network** of branches that need isolated visibility but shared inventory rules. - A **treasury** that mixes online gateways, cash at a branch desk, installments, refunds and vendor payouts in more than one currency. - A **compliance story** where seat counts, passport data and payment state must stay consistent even when three pilgrims click "Book" on the last seat at once. Spreadsheets and WhatsApp threads do not survive the first busy season. Wufud is a multi-tenant platform where each agency is a **first-class tenant** with isolated data, not a row filtered by `company_id` that someone might forget in a report. ## Isolation by schema, not by hope The platform schema (`public`) holds tenants, domains, plans, subscriptions, platform roles and signup flows. Each agency's operational data lives in **`t_`** — a dedicated PostgreSQL schema selected per request from the hostname (platform marketing host, tenant subdomain, or verified custom domain) via async request context before MikroORM runs queries. For the business, that is the difference between "we are careful with WHERE clauses" and "the other agency's tables are not addressable from this connection". Custom domains and white-label storefronts follow naturally: the edge routes by host, the API resolves tenant, the ORM sets `em.schema`. ## Seat integrity under real concurrency Package tiers track **total, confirmed and held** seats. Creating a booking **locks the tier row**, verifies availability, creates the booking and hold, and increments held seats in **one transaction**. PostgreSQL also enforces `seats_confirmed + seats_held <= seats_total`. Confirmation, cancellation and hold expiry take row locks so late webhooks cannot resurrect seats that were already released. Integration tests deliberately race bookings for the last seat and concurrent hold expiry — the kind of failures that only appear during Hajj registration windows. ## Payments that survive the real world Agencies enable **SSLCommerz** and **Stripe** (plus stub/manual paths in development). Webhook handling locks the payment attempt, treats a successful attempt as immutable, validates amount and currency against the booking, and commits receipt updates in the same transaction. Duplicate or out-of-order events are absorbed by attempt state rather than by hoping the provider sends perfect ordering. Branch **manual payments** separate who recorded cash from who approved it, with request keys to avoid double submission. **POS** supports product and service sales and refunds for ancillary items (Ihram, bags, tags) without a separate till system. ## Who uses which surface | Actor | What they get | | ----- | ------------- | | **Platform operator** | Tenant lifecycle, feature flags, subscription billing, platform admin at `/admin` on the marketing host | | **Agency owner** | Packages, staff, branches, gateways, accounts and reports on the tenant host `/dashboard` | | **Branch staff** | Branch-scoped RBAC — bookings and collections visible only where their role allows | | **Pilgrim** | Storefront browsing, family grouping, online or installment pay, journey tracking | Host-based routing is part of the product: `wufud.musfiqdehan.com` is platform marketing; `demo.wufud.musfiqdehan.com` is a fully seeded agency (Nur Travels) with storefront and operations. Tenant pages do not silently fall back to platform marketing when the API is unavailable — failures are explicit rather than brand-confusing. ## Edge, workers and operations Traefik terminates TLS and splits **Next.js** from **NestJS** (`/api` to the API). **BullMQ** workers handle asynchronous jobs against the same Postgres and Redis. Compose files cover local development, CI test stacks, staging and production behind GHCR images — with staging/production startup rejecting weak or identical JWT secrets. Reporting aggregates in PostgreSQL with grouped joins; the README documents honest limits at very large scale and the incremental rollups, caching and read-replica path forward — architecture writing that treats "5 million users" as a design review, not a marketing bullet. ## The result - **Agencies launch in minutes** on a subdomain, with optional custom domain and no per-booking commission. - **Seats and payments stay consistent** under concurrency and messy webhook delivery. - **Branches and finance** fit how Hajj operators actually collect and disburse money in Bangladesh and Saudi contexts. - **One monorepo** (`pnpm` workspaces, shared contracts package) keeps API and frontend types aligned. ## What it demonstrates Wufud is a greenfield TypeScript SaaS: NestJS 11, MikroORM 6, Next.js with React 19, schema-per-tenant isolation, Argon2 and JWT auth, OpenAPI-documented `/api/v1`, Playwright tenancy acceptance tests, and business rules captured in dedicated docs — seat locking, webhook idempotency and currency rules — so the implementation stays accountable to operators who cannot afford silent data leaks or oversold packages. --- ## FitPulse - URL: https://musfiqdehan.com/projects/fitpulse - Tagline: Software that runs a whole gym chain - memberships, billing, staff, door access, and AI that reads ID cards at the front desk. - Context: Commercial SaaS product (2024 - Present, status: live) - Primary discipline: fullstack - Stack: Python, Django, DRF, PostgreSQL, React, React Native, Redis, Celery, WebSocket, Docker, Traefik, Gemini API, Tesseract, Pillow, Cloudflare, PgBouncer - Metrics: 10k+ people using it every day; Scan to fill member info from ID images; 0s downtime when shipping updates; 3 clients web, mobile and turnstile, one backend - Links: Platform ; Hello Gym demo ### What I did - Serves 10,000+ daily users across multiple gym businesses - growth absorbed through caching and query work rather than a bigger hosting bill. - Each customer company runs in its own isolated database schema and on its own branded web address, so one client's data cannot surface in another's reports. - Front desk scans a member NID, birth certificate, or emergency-contact NID; OCR and vision models extract fields, the app validates the result, and staff confirm before anything is saved. - A shared AI layer routes tasks across Gemini, DeepSeek, and local providers with encrypted multi-key failover, per-tenant quotas, and Tesseract fallback for Bangla and English IDs. - Members check in with a fingerprint on ZKTeco hardware gyms already own, via a LAN bridge that keeps cheap terminals compatible with a cloud product. - Web, mobile, and turnstile clients share one API contract - idempotent writes and additive interfaces so the hardest client shapes reliability everywhere. - New versions go live with no downtime; platform operators configure AI providers, models, and task routes from a dedicated Settings panel. ## In plain terms FitPulse is the software a gym chain runs its business on: who is a member, what they paid for, which branch they belong to, whether they came in today, and what the trainers are owed. Several separate gym businesses use the same platform at the same time, each with their own branding, their own web address, and data none of the others can reach. Members walk up to a fingerprint reader the gym already owned and are through the door in a second. The owner opens a dashboard and sees today's check-ins as they happen. At the front desk, staff photograph a national ID or birth certificate and the form fills itself - with badges showing what the AI suggested and what a human still has to confirm. **Why it matters commercially:** the gym replaces a spreadsheet, a card machine, and a lot of arguing about who came in when. Onboarding a member stops being ten minutes of transcription from a laminated card. And because every customer company runs on shared infrastructure, the platform can sign a new gym without buying anything new. ## The problem A gym chain is not one business. It is a head office, several branches, staff who belong to exactly one of them, trainers who float between two, and members who bought a plan at one location and expect it to work at another. Model that badly and every question the software asks the database needs a "…and only for this branch" clause that someone will eventually forget. Forget it once and a receptionist in Dhanmondi is looking at Gulshan's revenue - or worse, a rival gym's. That is not a bug report; that is a lost contract. Model it well and the separation is structural: impossible to forget, because it is not something anyone has to remember. FitPulse is a platform for that shape of business, serving a web dashboard, a member phone app, biometric hardware, and now document-driven workflows from one system. ## Keeping customers apart, in two dimensions **Between businesses: each gets its own database schema.** Every customer company gets its own separate space inside a shared database cluster. The web address a visitor arrives on - including a gym's own custom domain - decides which space is opened, for that request only. A missing filter cannot leak another company's data because the other company's tables are not visible at all. For the business, this is the difference between "we are careful" and "it cannot happen". It is also what makes custom-domain branding possible: a gym can put the product on their own web address without anyone building them a separate copy of the system. **Inside a business: branch scoping.** Within one company, branch access is enforced deep in the data layer rather than screen by screen. Screens ask for "the members I am allowed to see"; the system decides what that means based on the person's role and which branch they work at. A curious staff member editing the web address in their browser gets a refusal, not another branch's payroll. Permissions then resolve three ways at once: the subscription plan says which features exist, the role says which actions are allowed, and the record says whether it is in reach. Any one of them saying no means no. I wrote about the trade-offs of this approach in [Schema-per-Tenant Django](https://musfiqdehan.com/blog/schema-per-tenant-django-multi-tenancy). ## AI setup and member autofill Adding AI to a multi-tenant product is not "paste an API key in settings and hope". Gyms upload photos of government IDs - sensitive, regulated, and easy to get wrong. The platform needed a provider layer operators can trust, tenants can quota, and membership features can call without reimplementing failover every time. **Infrastructure in `apps/ai` (public schema).** Providers (Gemini, DeepSeek, OpenAI-compatible local hosts, Tesseract) register with encrypted credentials, model priorities, and task routes. The router tries keys in order, backs off on rate limits, disables bad keys, and falls back to OCR when a text-only model receives a vision task. Usage is logged per tenant; Redis enforces RPM and daily caps. Platform staff manage the lot from an **AI Setup** tab - kill switch, image policy, route ordering, tenant overrides - gated by `platform.ai_setup`. Tenants see quota and status under **Settings → AI** when their package includes `settings.ai`. **Member onboarding uses it, safely.** The Add Member flow accepts uploads or camera capture for member ID, birth certificate, and emergency-contact NID. Each image is stored as a private `MemberDocument`, normalized (EXIF stripped, WebP for photos), and passed to `member_document_extraction`. The response fills name, date of birth, gender, address, and NID fields where confidence allows. Fields the user already typed are never overwritten; low-confidence extractions show a warning. Webcam capture compresses to WebP client-side before upload. The model proposes; RBAC and the membership app decide what gets saved. That split - task registry, tool registry for later agentic work, bounded agent loop - is the same foundation the next AI feature will plug into without touching provider code. ## Talking to the turnstile Attendance is the feature customers actually judge the product on, and it runs on ZKTeco fingerprint terminals sitting on a gym's local network - not on the internet, and not reachable from a cloud service. The bridge is a small program that runs on the gym's own network, speaks to the device the only way the device understands, and forwards the results upstream. It stores events locally when the internet drops, avoids sending anything twice when the connection returns, and treats the device's own clock as unreliable. Every record carries both the time the device claimed and the time the server received it - because those disagree more often than you would expect, and payroll depends on knowing which one was used. That bridge is its own project - [ZKTeco LAN Agent](https://musfiqdehan.com/projects/zkteco-lan-agent) - and it exists for a commercial reason: the cheaper terminals customers already own cannot reach a cloud service at all, and telling a prospective client to re-equip every branch before they can start is not a sentence that closes a deal. The attendance data then feeds a rules engine covering late arrivals, leave, overtime and payroll - rules that differ per branch, per role, and occasionally per employee. The outcome for the customer is that the argument about someone's hours is settled by a record, not by memory. ## One API, three clients The platform has three consumers with almost nothing in common: a React dashboard, a React Native member app, and a turnstile bridge that cannot be redeployed on demand. Designing for the most constrained client made every write idempotent and every interface additive. The full story is in [One API, Three Clients](https://musfiqdehan.com/blog/one-api-three-clients). ## Holding 10,000+ daily users without a bigger bill Growth is only good news if serving it stays affordable. Scaling this platform was mostly about refusing to buy hardware first: - **Connection pooling (PgBouncer).** The application was exhausting its database connections long before it ran out of processing power - a queue forming at a door that was needlessly narrow. Pooling widened the door for free. - **Caching (Redis).** Dashboard figures that were being recalculated from scratch on every page view are now computed once and reused. Pages that felt slow became instant. - **Fixing the queries before adding capacity.** The usual culprits: reports asking the database a thousand small questions instead of one, and missing indexes on exactly the branch-and-date combination every report filters by. - **Live updates instead of constant polling.** The check-in feed pushes new arrivals to the screen rather than asking the server every five seconds whether anything has changed - less server load and a more responsive product at the same time. The full breakdown is in [Holding 10k Daily Users Before You Buy a Bigger Server](https://musfiqdehan.com/blog/holding-10k-dau-before-you-scale-up). ## Shipping without going offline New versions start up and prove they are healthy before a single member is sent to them; the old version is retired only afterwards. Nobody sees a maintenance page, and nothing has to happen at 2am. Going back to the previous version is redeploying the previous release - deliberately boring, because rollback should never require thinking. The result is cultural as much as technical: fixes and features reach customers in days instead of waiting for a safe window. ## The result - **10,000+ people a day** rely on it, and growth was absorbed by engineering work rather than a larger infrastructure bill. - **Member signup at the desk** is faster and less error-prone when ID images drive autofill instead of manual entry. - **AI is operable** - keys, models, routes, and quotas are platform-managed rather than hard-coded per feature. - **No downtime during releases**, so improvements reach customers continuously instead of in nervous batches. - **A new gym can be onboarded** onto the existing platform with its own branding and its own domain - no separate installation to maintain. - **Existing biometric hardware works**, which removed a capital cost from the buying decision. ## What it demonstrates This is the project where "full stack" is literal: the data design, the backend, the permission model, the management dashboard, the member phone app, the hardware integration, the deployment pipeline, and now a production AI layer with privacy and tenancy baked in. Every layer was shaped by the same constraint - a mistake made in one customer's account must never become a problem in another's. --- ## StyleDesk - URL: https://musfiqdehan.com/projects/styledesk - Tagline: The software a salon runs on — online booking, chair-side operations, payroll and an AI assistant guests can actually talk to. - Context: Commercial salon management product (2025 - Present, status: live) - Primary discipline: fullstack - Stack: TypeScript, NestJS, Next.js, PostgreSQL, pgvector, Redis, Google Gemini, Docker, TypeORM - Metrics: 3 surfaces marketing site, admin, member appointments; RAG grounded AI with streamed responses; 768-dim pgvector embeddings aligned to Gemini; One API finance, inventory, HR and booking unified - Links: Live ### What I did - Public storefront and admin console share one API — clients book online while reception runs appointments, payroll and inventory on the same data. - Booking bag supports multiple services and packages before scheduling, so one visit is one coherent checkout instead of repeated back-and-forth. - Biometric and iClock integrations feed attendance into payroll and leave — hours disputes settle against records, not memory. - RAG chat streams answers from admin-curated knowledge with pgvector retrieval; the namespace is server-owned so callers cannot pick another tenant's corpus. - Gemini is optional — without an API key the salon product keeps working and AI routes fail gracefully instead of taking the site down. ## In plain terms StyleDesk is what a hair, beauty and barber business uses to stay open for bookings while the back office keeps up. Clients browse services and packages on a branded site, drop what they want into a booking bag, choose a stylist and slot, and confirm — without a phone tag chain. Staff manage appointments, clients, inventory, invoices, attendance and payroll from an admin workspace on the same backend. An optional AI assistant on the public site answers questions about services and policies using **retrieval-augmented generation**: admins maintain curated knowledge, the system embeds it into PostgreSQL with pgvector, and Gemini answers with those excerpts — streamed to the browser through a Next.js proxy so provider keys never reach the client. **Why it matters commercially:** salons lose money when chairs sit empty because booking is friction, and they lose trust when front desk and website disagree on availability or price. One system for discovery, scheduling and operations removes both failure modes. ## The problem A salon is not a single queue of appointments. It is services with different durations, packages that bundle them, stylists with different skills, branch or sister-concern branding, retail inventory sold at the chair, and payroll rules tied to attendance — often with biometric clocks that speak vendor protocols, not HTTP. Bolt-on booking widgets rarely connect to inventory or payroll. Spreadsheets for hours and commissions break the moment someone works at two locations or takes leave mid-cycle. And bolting a generic chatbot onto marketing copy without grounding creates confident wrong answers — worse than no bot at all. StyleDesk treats booking, operations and finance as one product, with AI added only where retrieval and server-side guardrails can keep answers tied to approved content. ## One backend, two experiences The **NestJS API** owns auth, RBAC, appointments, clients, services, packages, finance, payroll, leave, inventory, notifications, mail, reports and device integrations. The **Next.js 16** frontend serves the customer-facing marketing and booking flow and a separate admin dashboard (`/admin`), calling the API from the browser and from server components via distinct base URLs in Docker so SSR never confuses internal and public hostnames. That split matters for performance and security: public pages can be cached and polished while admin screens stay authoritative against live data. ## Booking that matches how clients think The public site is organised around **services, packages and people**, not around database tables. Clients can add multiple line items to a **booking bag**, adjust quantities, and only then pick stylist and time — mirroring how people actually plan a visit (cut plus colour, or a package plus an add-on). Signed-in clients use **My appointments** to see upcoming visits. FAQs and footer navigation keep policy questions out of the reception phone line where possible. ## Operations beyond the chair Modules on the backend reflect a real salon operator's week: - **Appointments and settings** — configurable rules for how booking behaves on the floor. - **Clients and employees** — roles and permissions for who sees what. - **Finance and invoices** — invoice settings and billing aligned with services sold. - **Inventory** — stock tied to retail sold during visits. - **Attendance, leave and payroll** — biometric and iClock paths ingest punches; payroll consumes the result. - **Reports and dashboards** — management visibility without exporting to spreadsheets. The codebase documents its current shape honestly: production today is a **single-business installation** (one salon organisation per deploy), while AI tables carry tenant identifiers as scaffolding for a possible future hosted product — with namespace selection **fixed on the server** so a public chat client cannot point retrieval at someone else's knowledge base. ## Grounded AI, not a gimmick The AI path is deliberately narrow and production-minded: 1. Admins maintain HTML knowledge entries in `/admin/ai-knowledge`. 2. The backend strips HTML to text, chunks it, batch-embeds with Gemini, and stores vectors in PostgreSQL (768 dimensions, matching the chosen embedding model). 3. A user question embeds the same way; cosine search pulls the closest chunks **within the server-resolved namespace**. 4. **ChatService** combines recent session history with those excerpts, labels them as untrusted reference text in the system prompt, and streams Gemini output as server-sent events. 5. The **PublicChatWidget** mounts only when enabled in client env; throttling, message limits, session validation and disconnect cancellation protect the public route. If `GEMINI_API_KEY` is missing, the rest of StyleDesk continues to run — AI endpoints return a clear unavailable state rather than breaking unrelated pages. ## The result - **Clients book online** through a flow built for multi-service visits, not single-slot widgets. - **Staff run the business** from one admin surface tied to the same appointments and clients. - **Attendance and payroll** connect through integrations salons already use on the LAN. - **AI answers stay tied to curated knowledge**, with server-side namespace control and streaming UX. ## What it demonstrates StyleDesk is a full-stack product in TypeScript: NestJS modular boundaries, Next.js public and admin UX, PostgreSQL with pgvector for semantic search, Redis for cache and sessions, Dockerised local and production compose files, and an AI feature designed like a production subsystem — optional provider, explicit dimensionality, migration-backed vectors, and security thinking on the public chat path — rather than a demo glued onto a CRUD app. --- ## ECamp - URL: https://musfiqdehan.com/projects/ecamp - Tagline: Sends hundreds of thousands of emails without wrecking the sender's reputation - and lets you manage contacts by talking to it. - Context: Master's thesis project, running in production (2025 - Present, status: live) - Primary discipline: ai - Stack: Python, Django, DRF, Celery, Redis, AWS SES, AWS S3, Next.js, React, Docker, WebSocket, LangChain, Gemini API, Cloudflare - Metrics: Never twice no recipient double-emailed on retry; Pause / resume stop a campaign mid-flight; 3 layers of pacing protecting sender reputation; Talk to it contacts managed by language and voice - Links: Source ; Live ### What I did - Campaigns can be paused and resumed mid-send, so a mistake spotted after launch costs a few hundred emails instead of the whole list. - A retried or duplicated send never emails the same person twice - the single most damaging failure in bulk email. - Sending is paced at three levels so a large campaign cannot trip provider limits or damage a sender reputation that takes weeks to rebuild. - One organisation's send cannot consume another's capacity, so a big customer's campaign never starves a small one. - Bounces and complaints suppress addresses automatically, keeping lists clean and deliverability high without manual work. - Each organisation keeps its own encrypted sending credentials, so a provider outage or price change is a settings change rather than a project. - Contacts can be managed in plain language or by voice - the AI proposes the change, the application validates and applies it. ## In plain terms ECamp is the software a company uses to email its customers at scale - newsletters, announcements, onboarding sequences - and to see what happened afterwards: who opened, who clicked, who bounced, who complained. Two things make it different from a simple mailing tool. First, it is built for the failures that cost money: a campaign can be **stopped mid-send** the moment someone spots a wrong link, and a technical retry can **never email the same person twice**. Second, contacts can be managed by talking to it - "add everyone who opened the March campaign to a new segment and tag them warm" - with the AI preparing the change and the application deciding whether it is permitted. **Why it matters commercially:** in bulk email nothing can be taken back. One duplicated send makes a brand look broken; one careless burst can get a sending domain throttled for weeks, which quietly kills every campaign after it. ## The problem Sending one email is trivial. Sending two hundred thousand is a pacing, reputation and repetition problem where every mistake is permanent. You cannot un-send. And a burst that trips a provider's threshold damages a sending reputation that takes weeks to rebuild - during which every subsequent campaign lands in spam folders and nobody can tell you exactly why. ECamp is a platform built around those specific stakes. It also doubles as my Master's thesis project on AI-assisted campaign management. ## Not locked to one email provider Every organisation configures its own sending route - AWS SES or standard SMTP - and those credentials are encrypted and kept separate per organisation. The application never talks to a provider directly; it talks to a common interface. The business consequence is leverage: switching provider for price, deliverability or coverage is a settings change rather than a development project, and a provider outage becomes a failover instead of an incident. ## Pacing in three layers A single overall speed limit is not enough, because the real constraints exist at different scopes: 1. **The provider's hard ceiling** - the limit that must never be crossed, on pain of throttling. 2. **Per-organisation throughput** - so one customer's blast cannot consume the shared budget and starve everyone else. 3. **Per-recipient-domain pacing** - because large mailbox providers throttle bursts from a single sender, and the polite rate is well below the provider's own ceiling. All three are enforced with shared counters that workers consult before claiming a batch. A worker that cannot claim capacity steps aside rather than sitting idle, so it stays available for a campaign that does have budget. Layers two and three are what protect **deliverability**, the metric that quietly decides whether email marketing works at all. ## Delivery you can actually stop Campaigns run in batches, and each batch records what it is about to do before it hands anything to the provider. That record is what makes pause and resume real: pausing sets a flag every batch checks before starting, and resuming picks up from the recorded position instead of restarting the campaign from the beginning. Every send is keyed on the campaign-and-recipient pair. A retried batch that already sent will not send again. With two hundred thousand recipients, "try until it succeeds, but never repeat" is the only combination that ends well. **Concretely:** spotting a broken link two minutes after launch costs a few hundred emails and an apology, rather than the entire list and a lost quarter of goodwill. ## The feedback loop The email provider streams bounces, opens, clicks and complaints back into the platform. Hard bounces suppress an address immediately; complaints suppress it and flag the list it came from. Delivery status pushes to the dashboard live, so a campaign in flight is something you watch rather than refresh. This is list hygiene running automatically, and it compounds: clean lists keep deliverability high, which keeps future campaigns landing in inboxes. Automation sits on top - immediate, delayed and scheduled triggers - so a signup can start a welcome sequence with nobody in the loop. ## The AI layer, and the boundary that makes it safe This is the thesis part. Contact management is driven by natural language and voice: *"add everyone who opened the March campaign to a new segment and tag them warm."* The design decision that makes it safe to point at real customer data: **the AI never writes to the database.** It proposes a structured operation. The application validates that proposal, applies exactly the same permission checks a human request would face, and only then executes it. A misunderstood instruction becomes a rejected proposal, not a corrupted contact list. That boundary is the whole difference between an AI feature a company can put in front of customers and a demo you would never point at real data. The argument in full: [Let the Agent Propose, Not Write](https://musfiqdehan.com/blog/let-the-agent-propose-not-write). ## The result - **Nobody gets the same email twice**, however many times a batch is retried. - **A campaign can be stopped mid-flight**, turning a discovered mistake into a small one. - **Sender reputation is protected by design**, which is what keeps future campaigns out of spam folders. - **Lists clean themselves** from bounce and complaint signals, with no manual maintenance. - **Switching email provider is a settings change**, so pricing and deliverability stay negotiable. - **Non-technical staff can manage contacts by talking**, without anyone worrying the AI will delete something. ## What it demonstrates Careful backend work under a genuinely unforgiving constraint - actions that cannot be undone, at volume - with an AI layer treated as a proposer rather than an authority. It is the pattern I would use for any feature where a language model touches production data. --- ## ZKTeco LAN Agent - URL: https://musfiqdehan.com/projects/zkteco-lan-agent - Tagline: Removed a hardware purchase from the sales conversation - old fingerprint terminals now work with a cloud product. - Context: Open source, deployed at customer sites (2025 - Present, status: maintained) - Primary discipline: devops - Stack: Python, Poetry, PyPI, ZKTeco SDK, ADMS HTTP, SQLite, systemd, Raspberry Pi, Linux - Metrics: No new hardware customers keep the devices they own; Never lost attendance survives internet outages; Never doubled a resend cannot inflate payroll; Zero server change one code path for all devices - Links: Source ; PyPI ### What I did - Lets customers use fingerprint terminals they already own with a cloud product - removing a per-branch hardware purchase from the buying decision. - Attendance survives internet outages: events are stored locally and forwarded when the connection returns, so a dropped link delays payroll data rather than losing it. - Re-sent batches can never double someone's attendance, which matters because attendance becomes payroll and payroll becomes an argument. - Treats the device clock as unreliable - both the device time and the server time travel with every record, so disputes are settled by evidence. - Needed zero changes on the server, so every future fix to attendance applies to old and new hardware alike. - Installs with one command on an office laptop or a small Raspberry Pi and runs unattended, so a gym manager never has to maintain it. - Reports its own health upstream, so "is the Dhanmondi branch syncing?" is answered from a dashboard rather than a phone call. ## In plain terms Fingerprint attendance terminals come in two kinds that look identical on a wall. The expensive kind can talk to the internet by itself. The cheaper kind - the one most businesses already own - cannot; it only talks to a computer on the same local network and waits to be collected from. This is a small program that runs on a laptop or a tiny Raspberry Pi in the customer's office. It talks to the old device the only way the device understands, and then reports the results to the cloud service as though the device had been internet-capable all along. **Why it matters commercially:** without it, the sales conversation includes "and you will need to replace the terminal at every branch first". With it, the customer starts on Monday with the hardware they already have. It is the smallest project on this site and one of the most commercially consequential. ## The problem ZKTeco terminals come in two families that are completely different to integrate with. The expensive ones support **ADMS**: the device itself opens a connection to a cloud server and pushes attendance events as they happen. The cheaper and older ones - sold as *Ethernet* or *PC Connection* models - do not. They speak a private protocol on the local network and wait for a PC to come and collect. They cannot reach the internet, and no configuration screen will change that. That is a commercial problem before it is a technical one. [FitPulse](https://musfiqdehan.com/projects/fitpulse) is a cloud product, and its customers are gyms that already own the cheap devices. "Buy new hardware for every branch before you can use our attendance module" is not a sentence that closes a deal - it turns a subscription decision into a capital expenditure decision, with an approval process attached. ## The design decision that pays off for years The obvious implementation is a second server endpoint - one for modern devices, one for the agent, each with its own data format. I did the opposite. **The agent imitates the protocol, not the device.** It reports events in exactly the format the server already accepts: ``` [ ZKTeco device ] --TCP/4370, ZK protocol--> [ LAN Agent ] --ADMS HTTP--> [ FitPulse ] (Ethernet only) (laptop / Pi) (unchanged) ``` The server does not know, and does not need to know, whether a record came from a terminal that pushed it or an agent that fetched it. One way in, one set of validation rules, one place where attendance can be wrong. The payoff shows up in maintenance rather than in the first release. Every future fix to attendance handling applies to both families of hardware automatically, and there is no slow drift where new devices get a feature old ones never do. **The compatibility problem lives entirely at the edge, in the component that is cheapest to update** - which means supporting older hardware costs almost nothing to keep supporting. ## Never lose a punch The link from a gym's office laptop to a cloud service is not reliable, so this has the same problem the [SuperMart offline till](https://musfiqdehan.com/projects/supermart) has, in a different shape: events happen in the physical world before the server hears about them. Two rules carry the design. **Progress is tracked locally, and the device is never wiped.** These terminals offer a "read the logs, then clear them" operation, and clearing is the intuitive way to avoid re-reading. It is also the way to permanently lose a day of attendance the moment an upload fails *after* the clear succeeded. The agent instead remembers how far it has read, in its own records, and re-reads from there. The device's memory is left as a safety net. **Each event's identity is derived, not invented.** A punch is identified by the device, the person and the time it happened. Re-sending a batch after a timeout produces the same identities, so the server absorbs it as a no-op rather than doubling someone's attendance. That last point matters more than it looks: attendance flows into payroll, and a duplicated punch is a payroll error somebody eventually has to argue about with a member of staff. Together the two rules mean a dropped connection **delays** attendance without losing or duplicating it. ## The device clock lies Every terminal keeps its own clock, and those clocks drift, get reset by power cuts, and occasionally get set wrong by a well-meaning member of staff. The agent syncs the device's time when it connects, but it never rewrites history: every record carries **both** the device's timestamp and the time the server received it. Payroll rules choose deliberately between them, and when the two disagree badly the discrepancy is visible rather than silently resolved. Quietly "correcting" a timestamp is the kind of helpfulness that becomes a dispute about someone's wages six weeks later, with no record of what the device actually said. Keeping both numbers means the business can answer the question instead of guessing. ## Deploying to a machine you do not own This is the part that made it an operations project rather than an integration script. The agent runs on hardware in someone else's building, on a network you cannot see, maintained by people whose job is running a gym. That constraint drove every packaging decision: - **One-command install.** `pip install zkteco-lan-agent` and a config file, not a repository clone and a page of build steps. - **Runs on what is already there.** An office laptop that stays on, or a Raspberry Pi for sites that would rather not depend on a laptop. Both are cheap enough that a spare can sit in a drawer. - **Supervised, not started by hand.** Installed as a background service so it survives reboots and power cuts - because it will get both. - **Self-healing on the two failures that actually happen.** The device goes unreachable when someone restarts the network switch; the uplink goes away when the internet does. Both recover on their own, rather than generating an alert asking a gym manager to intervene. - **Diagnosable from a distance.** The agent reports its own health and progress upstream, so "is the Dhanmondi branch syncing?" is answered from a dashboard rather than a phone call to a receptionist. Each device is identified by serial number and mapped to a branch on the server, so one agent can serve every terminal on a site, and moving a device between branches is a settings change rather than a site visit. ## Security posture A program running on a customer's own network, holding a credential to a shared cloud platform, deserves a moment of paranoia. The credential is scoped to one customer and to attendance only - it cannot read member records, cannot change anything else, and cannot reach another customer's data even if the machine it runs on is compromised. Traffic goes outbound only, over an encrypted connection, so installing it needs no firewall changes and opens no new way into the customer's network. That is also what makes it an easy "yes" for an IT-conscious client. ## The result - **A hardware purchase disappeared from the sales conversation** - existing terminals work on day one. - **Attendance survives outages** rather than vanishing with them, which keeps payroll defensible. - **A re-send can never inflate someone's hours**, removing an entire category of payroll dispute. - **The server never changed**, so supporting old hardware adds no ongoing maintenance burden. - **Non-technical sites run it unattended**, and head office can see remotely whether a branch is healthy. ## What it demonstrates The unglamorous integration work that decides whether a product is buyable. A bridge that leaves the server untouched, guarantees on an unreliable link, an explicit position on untrusted clocks, and packaging that lets non-technical people run a background service on hardware nobody is watching. --- ## Data Preprocessors - URL: https://musfiqdehan.com/projects/data-preprocessors - Tagline: A public Python package other developers install - and the release process that makes it safe to depend on. - Context: Open source (2022 - Present, status: maintained) - Primary discipline: software - Stack: Python, Poetry, PyPI, Docker, GitHub Actions, Semantic Versioning - Metrics: 1 command is the entire release process; 3 channels published from one workflow; Automated version numbers and release notes - Links: Source ; PyPI ### What I did - Published a reusable text-processing library that other developers install and depend on, replacing copy-pasted helpers across projects. - Turned releasing into a single command, so publishing is never skipped, half-done or done differently under time pressure. - Version numbers and release notes are generated by automation, so users can always tell whether an update is safe to take. - Builds are reproducible, so the version tested is exactly the version shipped - no "works on my machine" between test and release. - Distributed to three channels from one workflow, so users install it however suits their setup. ## In plain terms This is a small library of text-cleaning tools that any developer can install in one command - and, more importantly, the automated process that publishes it. The library itself is unglamorous. What makes it worth showing is the release pipeline: one command produces the tested build, the correct version number, the release notes and the publication to three places. Nothing depends on a person remembering a sequence of steps. **Why it matters commercially:** manual release steps are where outages come from. Every step a human performs under time pressure is a step that eventually gets skipped, and the skipped step is discovered by customers. The habits proven here are the ones I now apply to deploying business software. ## The problem Every NLP project I touched started the same way: copy the tokenising, cleaning and splitting helpers out of the last one, tweak two functions, forget which version was the good one. Multiply that across a team and you get five slightly different versions of the same logic, each with its own bugs, none of them fixed once. `Data Preprocessors` was the decision to stop doing that. Build it once, in the open, and depend on it everywhere. ## The part that actually mattered Publishing changes your relationship with your own code. Once strangers can install it, a breaking change is no longer a refactor - it is an event with consequences for people you have never met, who will find out at the worst possible moment. That forced three habits I now apply to internal business systems too: **Version numbers that mean something, enforced by automation.** The version comes from the release itself, not from a file someone edits by hand and forgets. A breaking change means a major version bump, and the pipeline is what says so. Users can tell at a glance whether an update is routine or risky - which is exactly what a customer needs to know before accepting an update. **Release notes generated from the work itself.** Commit messages compile into the changelog automatically. Release notes become a build output rather than a chore, which means they are never stale and never skipped at 11pm on a Friday. **Reproducible builds.** Dependencies are pinned exactly, so the version built in the pipeline is the version that was tested. It removes the entire class of "it worked before release" investigations. ## The pipeline Pushing a tag is the entire release ritual: ``` git tag v1.2.3 && git push --tags ``` From there, automation runs the tests, derives the version from the tag, builds the package, generates the changelog from the commit history, publishes to two package registries, and pushes a container image so the tooling is usable without setting up Python at all. No step in that list is one I can forget, do out of order, or do differently at midnight than at noon. That is the entire point. Written up in full: [A Release Should Be One Git Tag](https://musfiqdehan.com/blog/tag-driven-python-releases). ## The result - **Shared logic lives in one place**, so a fix benefits every project instead of one. - **Releasing costs one command**, which means it happens often instead of being deferred. - **Users can judge an update's risk** from its version number alone. - **What was tested is what ships**, removing a whole family of release-day surprises. ## What it demonstrates Small library, disproportionate lesson. Release engineering is the difference between code that exists and code that other people can rely on - and the same automated, reproducible, one-command approach is exactly what I now build for multi-customer SaaS deployments, where the cost of a skipped step is measured in downtime rather than embarrassment. --- # Articles ## Never Stop the Old Container First - URL: https://musfiqdehan.com/blog/zero-downtime-blue-green-traefik - Published: 2026-07-30 - Topic: DevOps (15 min read) - Tags: Docker, Traefik, GitHub Actions, CI/CD, Zero Downtime, Deployment - Summary: A routine deploy costs about a minute of failed requests, because "update" quietly means stop-then-start. Here is the pipeline that ships to a single server without a customer ever seeing an error - no Kubernetes, no second load balancer, and a rollback that is the same path as the deploy. ## The short version **In everyday terms.** The standard way to release a new version stops the running one before starting the replacement. In between - thirty seconds, sometimes a minute - the site is simply down. On a brochure site nobody notices. On anything that takes money, that window will eventually catch a real payment. The fix is to start the new version alongside the old one, prove it healthy, move traffic across, and only then retire the old one. Nobody sees a thing. **Why a business should care.** This changes when teams are willing to release. Deployments that cause visible errors get scheduled for late nights and weekends, batched up, and postponed - which makes each release larger and riskier than the last. Deployments nobody notices happen on a Tuesday afternoon, which is how fixes reach customers in days instead of weeks. The rollback path is the deploy path, so going back is routine rather than an emergency. **What follows** is the whole pipeline - quality gate, delivery, container overlays and the proxy that makes the switchover free - on a single server, without Kubernetes. --- Every deployment guide starts with the happy path. This one starts with the gap. You have a container serving production. You push a release. Compose stops the old container, starts a new one, and waits for it to report healthy. Somewhere in the middle of that sentence is a window - thirty seconds, sixty on a bad day - where the reverse proxy has nothing to route to and every request in flight comes back a 502. For a marketing site that window is invisible. For anything transactional it is not. If the deploy window catches a checkout, a payment callback, or a desktop client flushing its offline queue, the failure is not cosmetic - it is a retry storm and a support ticket. This is the pipeline I use to close that window: GitHub Actions for the quality gate, rsync for delivery, Docker Compose for the containers, and Traefik as the thing that makes cutover free. No Kubernetes, no second load balancer, one VPS. ## The gap nobody budgets for The naive deploy is one line, and it is one line because Compose is doing something reasonable: ```bash docker compose up -d --no-deps api ``` Compose sees a changed image, so it recreates the container. Recreate means **stop, then start**. The proxy watches the Docker socket, sees the container disappear, and pulls it from the routing pool. There is nothing else in the pool. ```text Timeline of a naive recreate ──────────────────────────────────────────────────────── Old container ████████████████████ │ stop ▼ Gap ·············· │ healthy ▼ New container ████████████████████ Client sees: 200 200 200 ... 502 502 502 ... 200 200 200 ``` Two common workarounds do not work here. **`docker compose up --scale api=2`** is the textbook answer, and it fails the moment you set a fixed `container_name`. Compose cannot start two containers with the same name, and dropping the fixed name costs you every ops habit built on `docker logs api` instead of `docker logs project-api-1`. **A longer healthcheck interval** does not help either. The gap is not a health-detection problem, it is an ordering problem. The old container is already gone. The fix is to invert the order. **Never stop the old container until something else is already healthy and already registered with the proxy.** ## What zero downtime actually means Worth being precise, because the phrase gets used loosely. My working definition: > For the entire duration of a deploy, every client request receives a successful response from *some* version of the application. Note what that does *not* promise. It does not promise that all clients see the same version at the same instant - during the overlap window, two requests from the same browser can land on different builds. It does not promise atomicity across services. And it does not extend to your background workers or your database, which have their own answers. That looseness is the price. In exchange you get a deploy you can run at 2pm on a Tuesday instead of scheduling for 3am. ## The shape of the system Two repositories - an API and a web front end - each with its own workflow, both deploying to the same VPS and attaching to an external Docker network the proxy owns. ```text ┌──────────────────────────────────┐ │ Internet / clients │ └────────────────┬─────────────────┘ │ :80 / :443 ▼ ┌──────────────────────────────────┐ │ Traefik │ │ TLS, routing, middleware, LB │ │ network: edge │ └──────┬────────────────┬──────────┘ │ │ PathPrefix(/api) │ everything else ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ app-api │ │ app-web │ │ :8000 │ │ :80 (nginx) │ └──────┬───────┘ └──────────────┘ │ ┌────────────┼────────────┐ ▼ ▼ ▼ PostgreSQL PgBouncer Redis (+ background workers, recreated after API cutover) ``` Each layer has exactly one job in the zero-downtime story: | Layer | Job | |---|---| | GitHub Actions | Gate on lint, types and tests, then deliver and invoke | | rsync over SSH | Put the CI checkout on the server - no git remote on the VPS | | Docker Compose | Build images, run the candidate alongside the live container | | Traefik | Discover containers by label; balance across both during overlap | | Healthchecks | Gate promotion - the only thing standing between a bad build and users | | `stop_grace_period` | Let in-flight requests finish when the old container is drained | Secrets live only on the server. The `.env` files are never in git and never in the rsync payload - CI has no way to overwrite them, which is deliberate. A deploy pipeline that can rewrite production credentials is a deploy pipeline that will, once. ## Blue-green without Kubernetes Blue-green usually implies two full environments and a switch between them. On a single VPS that is unaffordable, and it is also more than the problem needs. What I actually run is blue-green scoped to **one service, for about ninety seconds**: | | | |---|---| | **Live** | The container serving production right now, under the canonical name | | **Candidate** | A second container from this deploy's image, under a `-candidate` name | The candidate comes up *beside* the live container. Both serve. Then the live one goes away and the canonical name is recreated on the new image. The candidate is scaffolding - it exists for the duration of the swap and is removed at the end. ```text Phase A - build Phase B - overlap Phase C - promote ──────────────── ────────────────── ───────────────── Traefik ──► live ● Traefik ──► live ● Traefik ──► live ● (new image) Traefik ──► candidate ● candidate: building (both healthy) candidate: removed ``` The candidate is defined in an overlay file, `docker-compose.deploy.yml`, layered on top of the production Compose file. It extends the live service and changes four things: ```yaml services: api_candidate: extends: { file: docker-compose.prod.yml, service: api } container_name: app-api-candidate image: app-api:${IMAGE_TAG} restart: 'no' labels: - traefik.http.routers.api-candidate.rule=PathPrefix(`/api`) - traefik.http.routers.api-candidate.service=api ``` `restart: 'no'` matters. A candidate that crash-loops should stay down so the healthcheck gate catches it, not restart until the timeout expires and confuses the diagnosis. ## One service name, two containers This is the part that makes the whole thing work, and it is three lines of labels. Traefik requires **router** names to be unique - two containers cannot both claim `api`. But routers and services are separate concepts, and nothing stops two distinct routers from pointing at the *same* Traefik service: ```yaml # on the live container traefik.http.routers.api.rule=PathPrefix(`/api`) traefik.http.routers.api.service=api # on the candidate traefik.http.routers.api-candidate.rule=PathPrefix(`/api`) traefik.http.routers.api-candidate.service=api # <- same service traefik.http.services.api.loadbalancer.server.port=8000 ``` Traefik resolves both containers into one service's server pool and round-robins across them. You get load balancing during the overlap for free, from the proxy that is already terminating TLS. ```text During overlap Request ──► Traefik service "api" │ ┌─────────┴─────────┐ ▼ ▼ app-api app-api-candidate (old image) (IMAGE_TAG = git SHA) ``` When the live container is stopped, Traefik gets the Docker event, drops that server from the pool, and sends 100% of traffic to a candidate that has already been warm and serving for a minute. There is no cold start at the cutover instant - which is the other half of why this is smooth, and the half that a DNS or symlink switch does not give you. ## The pipeline, end to end Each repo has one workflow on the `production` branch. | Event | What runs | |---|---| | PR into `production` | CI only - lint, types, tests, build | | Push to `production` | CI, then deploy | | `workflow_dispatch` | Manual re-run, which is also the rollback path | ```text push to production │ ▼ ┌─────────────────────────────────────┐ │ job: ci │ │ API: lint, format, types, │ │ migration drift check │ │ Web: eslint, tsc, unit, build │ └──────────────┬──────────────────────┘ │ needs: ci ▼ ┌─────────────────────────────────────┐ │ job: deploy │ │ concurrency: production-deploy │ │ cancel-in-progress: false │ │ │ │ 1. checkout │ │ 2. ssh-agent, load deploy key │ │ 3. rsync → DEPLOY_PATH │ │ 4. ssh: IMAGE_TAG=$GITHUB_SHA \ │ │ ./scripts/deploy/prod.sh │ └─────────────────────────────────────┘ ``` `cancel-in-progress: false` is not a detail. The default in most concurrency configs is to cancel the older run, and cancelling a deploy **mid-cutover** is how you end up with no live container and a half-promoted candidate. Deploys queue. They never race. ### Why rsync and not `git pull` The obvious pattern is to SSH in and `git pull`. I do not, for three reasons. Actions has already checked the code out - pulling again means putting a deploy key with repo access on the server, which is a credential I would rather not have sitting there. It also makes the server's state depend on a network call to GitHub at deploy time, adding a failure mode to a step that has enough of them. And a git working tree on a production box invites someone to fix a bug in place at 1am, which then vanishes on the next deploy with no record it existed. ```bash rsync -az --delete \ --exclude '.git/' \ --exclude '.env*' \ --exclude 'node_modules/' \ --exclude 'media/' \ ./ "${VPS_USER}@${VPS_HOST}:${DEPLOY_PATH}/" ssh "${VPS_USER}@${VPS_HOST}" \ "cd '${DEPLOY_PATH}' && IMAGE_TAG='${GITHUB_SHA}' ./scripts/deploy/production.sh" ``` Consequences worth writing on the runbook: the deploy directory is rsync-managed, not a git checkout. `--delete` means anything not in the repo is removed, so runtime state - uploads, `.env`, caches - must be excluded or mounted from outside the tree. And since there is no git history on the server, the deploy script writes the shipped SHA to `.deploy-revision`, which is the only way to answer "what is actually running right now". ### The tag is the contract ```text IMAGE_TAG = GITHUB_SHA ``` The candidate is built with that tag and the promotion step uses the *same* tag rather than rebuilding. This sounds pedantic and is not: if promote rebuilds, you have health-checked one artefact and shipped a different one. Same idea as [building a release artefact once and promoting it](https://musfiqdehan.com/blog/tag-driven-python-releases) rather than rebuilding per target. ## The API deploy, phase by phase | Phase | Action | Live API up? | |---:|---|:---:| | 0 | Clean the deploy tree, write `.deploy-revision` | yes | | 1 | `compose build api` | yes | | 2 | Run migrations as a one-off container | yes | | 3 | Start the candidate | yes | | 4 | Wait for candidate Docker health, ~300s timeout | yes | | 5 | Smoke test `https://example.com/api/health/` through the proxy | yes | | 6 | `docker stop -t 30` the live container | candidate serves 100% | | 7 | `compose up -d --no-deps api` on the new image | yes | | 8 | Wait for the canonical container to report healthy | yes | | 9 | Remove the candidate | yes | | 10 | Recreate background workers | yes | | 11 | Final smoke test | yes | ```text Time ─────────────────────────────────────────────────────────────► Live ████████████████████████████████████░░ stop │ Candidate ░░ build ░░ ████ health ██████████████░░ remove │ smoke │ promote Migrate ████ one-off │ │ Workers ░░ recreate ░░ Client traffic ───────────────────────────────────────────────────► ``` Two phases carry most of the weight. **Phase 4** is the gate. Docker's own health status is the signal, which means the healthcheck in the Dockerfile is load-bearing infrastructure, not decoration. A healthcheck that only checks the process is alive will cheerfully promote a container that cannot reach the database. Mine hits a readiness endpoint that touches the DB and the cache. **Phase 5** is the second gate, and it exists because Phase 4 can pass while the request still fails. The container being healthy on its own network is not the same as the request path working - labels can be wrong, the container can be on the wrong network, TLS can be misconfigured. Phase 5 goes out through the public hostname and comes back in through Traefik, which is the path a user takes. Only after both gates pass does anything stop. Everything before Phase 6 is reversible by deleting a container nobody was told about. ## The web deploy Same shape, fewer moving parts - no migrations, no workers. | Phase | Action | Live UI up? | |---:|---|:---:| | 1 | Build the image at `IMAGE_TAG` | yes | | 2 | Start the candidate | yes | | 3 | Wait for health | yes | | 4 | In-container smoke: `wget -S /`, expect 200, not a redirect | yes | | 5 | `docker stop -t 30` live | candidate serves 100% | | 6 | Promote canonical | yes | | 7 | Wait for health, remove candidate | yes | | 8 | Brief pause for label rediscovery, then external smoke | yes | The front end's smoke test is fussier about *content* than the API's. A static server will happily return 200 for a build that routed `/` to the sign-in page, or shipped an asset manifest referencing chunks that are not in the image. Checking the status code alone passes both. So the in-container check asserts the response is a 200 and **not** a redirect, before anything is drained. Routing stays unambiguous by keeping the API rule specific and the web rule broad: ```text PathPrefix(`/api`) → service "api" (higher priority) Host(`example.com`) → service "web" (everything else) ``` The web front end never gets a chance to swallow `/api` on a bad day. ## Migrating while the old code still serves Phase 2 runs migrations while the **old** version is still the only thing serving traffic. That is a deliberate choice with a hard constraint attached: > Between Phase 2 and Phase 6, the old code is running against the new schema. If that combination does not work, you do not have a zero-downtime deploy - you have a broken one with extra steps. Which sorts every schema change into two piles: | Safe during overlap | Needs a window, or a second release | |---|---| | Add a nullable column | Drop a column the old code still selects | | Add a table | Rename a column in one release | | Add an index concurrently | `NOT NULL` without a default while old writers omit it | | Backfill in a separate pass | Destructive rewrites only the new code understands | The discipline is **expand, then contract**. Release one adds the new column and writes to both. Release two reads from the new column. Release three drops the old one. Three deploys to rename a field feels absurd until the alternative is a maintenance window, and the whole point of this pipeline is not needing one. The migration step is also the last fully safe abort point. It runs before the candidate exists, so a failure there leaves the live container untouched and the deploy simply stops. On multi-tenant setups where migrations run per schema, this phase is the long pole - and worth [thinking about early](https://musfiqdehan.com/blog/schema-per-tenant-django-multi-tenancy), because a per-tenant migration that takes minutes changes the shape of the overlap window. ## Failing closed, and rolling back The deploy script traps errors, and the response depends on one boolean: has the live container been stopped yet? ```text deploy error │ ┌─────────────┴─────────────┐ ▼ ▼ DRAINED = 0 DRAINED = 1 (before phase 6) (after live was stopped) │ │ ▼ ▼ remove candidate do NOT clean up live never stopped log loudly, exit non-zero users unaffected operator checks canonical ``` Before drain, cleanup is safe and automatic - the candidate is removed and nobody outside the pipeline ever knew it existed. After drain, **the script deliberately stops helping**. An automated cleanup at that point could remove the only container serving traffic. Loud failure beats clever recovery when the blast radius is production. | Failure | Outcome | |---|---| | Build fails | Live untouched | | Migration fails | Live untouched, deploy aborts | | Candidate never healthy | Candidate removed, live untouched | | Smoke fails before drain | Candidate removed, live untouched | | Failure after drain | Rare, and the one case that pages a human | Rollback uses the same machinery, which is the point of building it this way. Point `production` at the last known-good commit and re-run the workflow - it blue-green deploys that SHA exactly like any other release, with the same gates. There is no separate, less-tested rollback path to discover is broken at the worst possible moment. What you cannot do is `git reset` on the server. The deploy directory is not a git repo, and treating it like one is how you end up with files from two releases. Roll back through the pipeline. If the deploy failed *before* drain, there is usually nothing to roll back at all. Live never changed. ## Proving it actually held Claiming zero downtime and measuring it are different activities. From any machine with network access, during a real deploy: ```bash while curl -sf https://example.com/api/health/ >/dev/null; do printf '.' sleep 1 done echo "FAILED at $(date -Is)" ``` The loop exits on the first non-success. A clean cutover keeps it running for the entire deploy. Run it in a second terminal while you watch the Actions log - it takes ten seconds to set up and it is the only evidence that matters. Afterwards, on the server: ```bash cd "$DEPLOY_PATH" cat .deploy-revision # matches the Actions run SHA? docker ps # canonical container healthy, no orphan candidate curl -sf https://example.com/api/health/ ``` The one drill worth running deliberately: break a candidate's healthcheck on purpose in a staging copy and run the deploy. Confirm it aborts at Phase 4 and the live container is still serving. A safety mechanism you have never seen fire is a safety mechanism you are guessing about. ## What this does not cover Being honest about the boundary is more useful than the claim. | Component | During an app deploy | Why | |---|---|---| | API | Up throughout | Blue-green plus proxy load balancing | | Web UI | Up throughout | Same | | Background workers | Brief restart | Recreated after cutover - queued jobs delay, they do not vanish | | PostgreSQL, Redis, PgBouncer | Untouched | `--no-deps` keeps the data plane out of the app recreate | | The proxy itself | Separate lifecycle | Traefik is not rebuilt on every app push | | Desktop or mobile clients | Separate pipeline | Different release cadence entirely | Workers are the honest gap. A brief worker restart is acceptable because a durable queue makes it a latency event rather than a correctness one - which only holds if the jobs are idempotent and the queue survives the restart. If that is not true of your workers, fix that before you worry about the API's 502s. It is the larger problem. The other gap is **cross-service atomicity**. Two repos deploy independently, so there is no instant where the front end and API switch together. If a release needs both, the API ships the new field first in a backward-compatible release, and the UI follows once it is live. That is release discipline, not infrastructure - no pipeline will save you from a breaking API change shipped in lockstep. ## The operator checklist **Once per server** - Docker and the Compose plugin installed, deploy user in the `docker` group - The external proxy network exists and the proxy is running on it - Deploy directories created, `.env` files present and **not** in git - Initial stack brought up once by hand, so the deploy script only ever swaps **Once per repo** - Secrets: host, user, SSH key, deploy path - Workflow on the production branch, with `concurrency` set to queue rather than cancel - Branch protection so the only way in is a reviewed merge **Every release** - CI green before merge - Migrations reviewed against the expand-only table above - Watch the phase log through cutover - Confirm `.deploy-revision` and the external health URL - Optional, and cheap: run the curl loop during the deploy **When something looks wrong** | Symptom | First thing to check | |---|---| | SSH fails in Actions | Host, key, firewall, user in the docker group | | Candidate never healthy | `docker logs app-*-candidate` - almost always config or DB reachability | | 502 after cutover | Container health, proxy labels, is it on the right Docker network | | Two versions serving forever | Orphaned candidate from a failed run - `docker ps` will show it | | Mystery files on the server | Something was edited in place; rsync `--delete` will remove it next deploy | ## The sequence is the guarantee There is no flag for this. Strip the pipeline down and what is left is an ordering rule and two gates: ```text gate on quality in CI → deliver code, build the new image (live still serving) → migrate compatibly (live still serving) → start candidate → health → smoke (live still serving) → drain live → promote → clean up → smoke again ``` Every step before the drain is reversible. The drain only happens after two independent checks agree the new version works, one of them through the same path a user takes. And the data services stay out of the recreate entirely. The rest is discipline: migrations that respect the overlap window, workers whose restart is a latency event rather than a data-loss event, and a rollback path that is the deploy path so it cannot rot. Get those right and deploying stops being an event you schedule around, which is worth more than the uptime figure. --- ## Selling With the Wi-Fi Down: An Offline-First POS in Tauri 2.0 - URL: https://musfiqdehan.com/blog/offline-first-pos-tauri-sync-outbox - Published: 2026-07-29 - Topic: Full stack (13 min read) - Tags: Tauri, React, SQLite, Django, Offline-First, Idempotency, PostgreSQL - Summary: A till that stops working when the internet does is a till that loses money. Here is the contract that lets a desktop point of sale keep selling through an outage and reconcile exactly once when the connection returns - without ever charging a customer twice or losing a sale from the books. ## The short version **In everyday terms.** A shop's internet goes down. The customer is standing at the counter with cash. The till has to take the money anyway - and when the connection comes back, that sale has to reach head office exactly once. Not zero times, which is missing revenue. Not twice, which is a customer refund and a very bad review. **Why a business should care.** In retail an hour offline is an hour of turned-away customers, and connectivity is not something a shop owner controls. Making the till work offline converts a problem the business cannot fix into one it does not have to care about. The catch is that a device selling without checking with head office can go out of step with it - so the design has to say clearly what happens when it does. **The trade-off, stated honestly:** stock figures shown offline may be briefly wrong, and that was chosen deliberately. Refusing sales whenever the till is uncertain is exactly the failure the product exists to prevent. The safeguards are visible rather than clever: an offline indicator the cashier can see, and a list of sales that failed to sync which somebody must resolve. **What follows** is the outbox-and-unique-key contract in full, including the crash windows and what I would change. --- The requirement arrived as one sentence from a shop owner: *"When the internet goes, the till still has to work."* That sentence is a distributed systems problem wearing retail clothes. The moment a client can commit a sale without the server, you have two writers, an unreliable link, and money on the line. Every classic failure mode shows up: duplicate writes, lost writes, stale reads, and the special horror of charging a customer twice for one basket of groceries. This is how the [SuperMart](https://musfiqdehan.com/projects/supermart) desktop POS handles it - a Tauri 2.0 shell around a React front end, a local SQLite database, and a Django backend that refuses to be lied to. ## Why a desktop app at all A PWA with a service worker can go offline. I chose a native shell anyway, for reasons that are operational rather than architectural: - **The machine is a shop counter PC**, often old, often running one thing all day. A Tauri binary is a few megabytes over a system webview instead of a bundled Chromium, and the memory difference is visible on the hardware these shops actually own. - **Local data wants a real database.** IndexedDB is workable; SQLite with a Rust process in front of it is better when the same rows are read on every keystroke of a product search. - **Hardware.** Receipt printers, cash drawers and barcode scanners are far less painful from a native process than from a browser sandbox. - **Credentials.** A token in browser storage is a token in browser storage. In Tauri it can live behind the Rust boundary, and eventually in the OS keychain. Tauri 2.0's permission model also means the front end declares what it may touch, so the React layer cannot reach the filesystem just because someone imported the wrong helper. ## Two databases, one truth The single sentence the whole design hangs on: > **The client proposes. The server decides.** SQLite on the till is a **cache and a queue**, never a source of truth. It holds two categories of data, and keeping them separate is what stops the design collapsing: | | Purpose | Authority | |---|---|---| | **Cache tables** (`products`, `stock_snapshot`, `customers`) | Let the UI work offline | Server. Overwritten on every pull. | | **Outbox table** (`checkout_outbox`) | Hold writes the server has not accepted yet | Client, until acknowledged. | The cache is disposable. If it is corrupted, wrong, or a week stale, the fix is to drop it and pull again. The outbox is not disposable - it is the only record of a sale that happened in the physical world but has not yet been told to the server. Those two things need different treatment, and merging them into one "local state" store is how offline apps end up losing sales. ## The outbox When a cashier completes a sale offline, the client does not "queue a request". It commits an **intent** - the complete, final payload the server will eventually receive: ```ts async function commitSale(basket: Basket): Promise { const payload = { branch_id: session.activeBranch.id, lines: basket.lines.map(toLine), payments: basket.payments, idempotency_key: crypto.randomUUID(), // generated exactly once, here client_committed_at: new Date().toISOString(), }; await db.execute( `INSERT INTO checkout_outbox (id, tenant_subdomain, branch_id, payload, status, attempts, created_at) VALUES (?, ?, ?, ?, 'pending', 0, ?)`, [payload.idempotency_key, session.tenant, payload.branch_id, JSON.stringify(payload), payload.client_committed_at], ); return renderReceipt(payload); // the customer leaves with their goods } ``` Three details in there are load-bearing. **The payload is stored whole, not reconstructed later.** If the outbox stored a foreign key to a local sale record and rebuilt the request at push time, then a schema change, a cache refresh or a partially-written row could change what gets sent. A serialized payload is a promise about exactly what will hit the server. **`idempotency_key` is generated once, at commit time.** Not at push time. Not per attempt. This is the entire safety property, and the most common way to get it wrong is to generate the key in the HTTP layer, where a retry helpfully makes a fresh one. **The row is inserted before the receipt prints.** If the app dies between the two, the customer has goods and no receipt - recoverable. The other order means the customer has a receipt for a sale that no longer exists anywhere. ## Draining it Reconnection is where implementations get clever and then get burned. This one is deliberately boring: ```ts async function pushOutbox() { const rows = await db.select( `SELECT * FROM checkout_outbox WHERE status = 'pending' AND tenant_subdomain = ? ORDER BY created_at ASC`, // strict FIFO [session.tenant], ); for (const row of rows) { try { const sale = await api.post('/pos/checkout/', JSON.parse(row.payload)); await markSynced(row.id, sale.id); } catch (err) { if (isNetworkError(err)) return; // stop; try the whole queue later if (isRetryable(err)) { await bumpAttempts(row.id); continue; } await markFailed(row.id, err.code, err.detail); // needs a human } } } ``` **Why FIFO and not parallel?** Throughput is not the constraint - a busy till produces a few hundred sales a day, not a few thousand a second. What FIFO buys is a cash-register ordering that matches the cashier's mental model, a debuggable log when something goes wrong, and no self-inflicted lock contention from ten of your own requests fighting over the same hot SKU. Parallel drain is an optimisation for a problem this system does not have. **Why does a network error abort the loop rather than skip the row?** Because if the link is down, every remaining row will fail too. Continuing just burns the retry budget and inflates `attempts` counters that later look like real failures. ## The server half of the contract An idempotency key on the client is a wish. It becomes a guarantee only because of what the server does: ```python @transaction.atomic def checkout(*, user, branch_id, lines, payments, idempotency_key): assert_user_branch_access(user, branch_id) existing = Sale.objects.filter(idempotency_key=idempotency_key).first() if existing: return existing levels = (StockLevel.objects .select_for_update() .filter(branch_id=branch_id, product_id__in=[l.product_id for l in lines])) verify_available(levels, lines) # raises InsufficientStock try: sale = Sale.objects.create(..., idempotency_key=idempotency_key) except IntegrityError: # Lost the race against a concurrent first attempt with the same key. return Sale.objects.get(idempotency_key=idempotency_key) record_payments(sale, payments) apply_stock_movements(sale, levels) return sale ``` The part worth dwelling on is that **`select_for_update` and the idempotency key solve different problems**, and it took me one production scare to internalise that they are not interchangeable. `select_for_update` is about **concurrency**: two cashiers, two different sales, one remaining unit. The row lock serialises them so the second sees the true quantity and fails cleanly instead of driving stock to -1. The idempotency key is about **duplication**: one sale, sent twice. No amount of locking helps, because the second request is perfectly valid in isolation. Only identity does. And the check must be backed by a **unique constraint**, not by `filter().first()` alone. That read is not a lock; two genuinely concurrent first attempts can both see nothing and both proceed. The `IntegrityError` branch above is the one that makes it correct - the loser of the race fetches the winner's sale and returns it. If your idempotency column has no unique index, you do not have idempotency, you have a race with good intentions. ## The crash window The question I get asked most: *what if the server commits, then the network dies before the client marks the row `synced`?* The client wakes up, sees `pending`, and posts again. The server finds the existing sale by key and returns it. The client marks `synced`. Nothing is decremented twice. This is precisely why the client is not allowed to be the authority on whether a sale succeeded. "I think it worked" is not a state you can build on across a partition. The server's uniqueness constraint is the arbiter, and the client's job is only to keep asking until it gets a definitive answer. The complementary failure - client crashes *before* inserting the outbox row - is a sale that never existed digitally. That is a real loss, and it is why the insert precedes the receipt and the confirmation UI. The window is narrow and, unlike a double-charge, it is visible to the cashier standing there. ## When the sale cannot land The genuinely hard case is not a network error. It is this: the till sold three units offline, and while it was offline another branch's transfer or an online sale consumed the stock. On replay the server correctly returns `INSUFFICIENT_STOCK`. There is no clever resolution here. The goods have physically left the shop. Options that all look tempting and are all wrong: - **Force the sale through and let stock go negative.** Now inventory lies, and every downstream report inherits the lie. - **Silently drop the row.** Revenue disappears with no audit trail. This is the worst option and also the easiest to write. - **Auto-retry forever.** Stock is not coming back. The queue jams behind a row that will never succeed. What it does instead: the row moves to `failed`, keeps the server's reason, and appears in a queue the cashier or manager has to clear - reconcile the stock discrepancy, void, or accept and adjust. **`failed` is a state, not a deletion.** An offline-first system that cannot show you what it failed to sync is asking you to trust it about the one thing you cannot verify. ## The security bits people skip Offline-first quietly widens the threat surface, and two of these only exist *because* of the local database: **Branch tampering.** The desktop payload carries `branch_id`. A modified client can carry a different one. So `assert_user_branch_access` runs server-side inside the transaction, checked against the user's role assignment - not against anything the client claimed. Frontend branch scoping is UX; the server assertion is the control. **Cross-tenant cache bleed.** Outbox and cache rows are keyed by tenant subdomain, and switching tenant or logging out wipes that tenant's rows. Without this, a pending sale committed under tenant A could be pushed with tenant B's `Authorization` and `X-Tenant-Subdomain` headers - a cross-tenant write, from your own client, that no server-side check would catch because the request would look entirely legitimate. **Replay from a stolen device.** The idempotency key that protects against accidental duplication also bounds deliberate replay: re-posting a captured payload returns the original sale rather than creating a new one. ## What I would change Three things, in the order I would do them: **Delta sync.** The pull is currently a full catalogue refresh. It is fine at a few thousand SKUs and will not be at fifty thousand. `updated_since` plus tombstones for deletes is the obvious fix; I have not needed it yet, and saying so is more useful than pretending the current design scales further than it does. **Dead-letter visibility.** `failed` rows are visible in the app but do not alert anyone. A `failed` row older than an hour is a business problem, and it should page rather than wait to be noticed. **OS keychain for tokens.** They currently live in Tauri's store. The keychain is the right home, and Tauri 2.0's plugin ecosystem makes it a contained change. ## The shape of it If you take one thing from this, take the split: **Optimistic on the client, pessimistic on the server.** The till assumes the sale will work, because the customer is standing there. The server assumes nothing, because it is the only component that can actually see the truth. The outbox is the bridge, and the idempotency key is what makes crossing it repeatable. Everything else - the SQLite schema, the Tauri commands, the retry policy - is implementation. That contract is the design. --- ## Decomposing a Monolith Without a Rewrite - URL: https://musfiqdehan.com/blog/decomposing-a-monolith-without-a-rewrite - Published: 2026-07-08 - Topic: Architecture (11 min read) - Tags: Architecture, Microservices, Django, Refactoring, Leadership, Docker - Summary: Splitting a large codebase is usually sold as a rewrite and delivered as a disaster. Here is how to do it piece by piece while the product keeps shipping to customers - plus an honest account of what independent deployment actually costs you in return. ## The short version **In everyday terms.** Most business software eventually becomes one large, tangled program where changing anything risks breaking something unrelated. The usual proposal is to rebuild it from scratch. Rebuilds routinely run over time, deliver nothing for a year, and end up reproducing the old bugs. The alternative is to lift out one well-chosen piece at a time while the product keeps running and customers keep getting updates. **Why a business should care.** The point is not "microservices". The point is that teams stop blocking each other, a busy part of the system can be given more capacity without paying to scale everything, and a failure in one area stops taking the whole product down. If none of those three is your actual pain, splitting up is an expensive answer to a question nobody asked. **What follows** is how to find the real seams, the data-ownership rule that keeps them honest, and the costs you take on the moment a simple function call becomes a network call. --- Every proposal to break up a monolith starts the same way: a diagram with six neat boxes and arrows between them. Every one I have seen fail did so for the same reason - the boxes were drawn around *what the team wished the system looked like*, not around the seams that were actually there. I have been doing this decomposition at TechForing on a SaaS platform that works and has customers. The constraint that shapes everything: **the product ships the whole time.** No rewrite, no freeze, no cutover weekend. ## Find the seams; don't invent them A seam is a place where the system is already nearly separable. You find them by looking, not by designing: - **Which modules import from which?** A module nothing imports from is a candidate. A module everything imports from is the shared kernel, and it is not going anywhere. - **Which tables are written by exactly one module?** Single-writer tables are a boundary. A table written by four modules is four modules pretending to be one. - **What has a different scaling shape?** Something CPU-heavy, or bursty, or slow, buried in a request path that is otherwise fast. That difference is a real reason to separate; "it feels like its own thing" is not. - **What changes on a different cadence?** A module deployed on every release alongside one touched twice a year are not the same component. Notice that all four are observations about the existing code. Draw the boundary where the system already wants to split, and extraction is mechanical. Draw it where you wish it split, and you spend six months fighting the dependency graph. ## Modularise first, distribute second The step people skip: **make it a module before you make it a service.** Enforce the boundary inside the monolith first. One package, an explicit public interface, no other module reaching past it into its internals, no other module touching its tables. This is free to do, free to undo, and it surfaces every hidden dependency you would otherwise discover mid-extraction with half the traffic already moved. If a boundary is painful to hold *inside* one process - where a violation is a one-line import - it will be significantly worse across a network. A module you cannot keep clean is not ready to be a service. It might just be a module that should stay a module, and that is a completely legitimate outcome. ## The data rule If you take one thing from this: **one service owns its tables. Nobody else reads them.** Shared database access is the failure mode that turns "microservices" into "a distributed monolith" - all of the operational cost of services, none of the independence. When two services read the same table, you cannot change its schema without coordinating a deploy, which was the exact problem you were trying to solve. Which means extraction is usually a data migration, and that is the part that actually takes time. The sequence that works: 1. Route all access to the target tables through the module's interface, still inside the monolith. 2. Move those tables to their own schema. Still one database; access is still through the interface. 3. Stand the service up against that schema. 4. Switch the interface from an in-process call to a network call. 5. Only now, if it is worth it, move the schema to its own database. Every step is independently deployable and independently revertible. Step 4 is the only one where the failure mode changes - and by then, everything else is already proven. ## Strangling the old path For the switchover, the strangler pattern: the new implementation runs alongside the old one, and a flag decides who serves the request. Start in **shadow mode** - the new path executes, its result is compared to the old one and logged, and the old result is what the user gets. This is where you find that the new service rounds a currency differently, orders a list differently, or handles a null the old code silently coerced. Reading those diffs for a week is the cheapest confidence you will ever buy. Then shift traffic by percentage. Then remove the old path - and *actually remove it*, because a dead code path guarded by a flag that has been at 100% for four months is a trap for the next person who reads the file. ## What you give up Anyone who pitches decomposition without this list is selling something. **A function call becomes a network call.** It can now be slow, time out, or partially succeed. Every call site needs a timeout, a retry policy and a decision about what to do when the answer never comes. **Transactions stop at the boundary.** Two writes across two services cannot be one atomic operation. You choose eventual consistency with compensating actions, or you keep them in the same service. I keep them in the same service far more often than the architecture diagrams suggest - **needing a transaction across a boundary is strong evidence the boundary is in the wrong place.** **Debugging gets harder.** A stack trace used to contain the whole story. Now you need correlation ids threaded through every call and logs you can query across services. Set that up *before* the first extraction, not after the first incident. **Local development gets harder.** "Run the app" used to be one command. Keep it one command - a Compose file that stands up everything - or you will pay for it in every new engineer's first week, forever. ## The organisational half Services fail for organisational reasons at least as often as technical ones. Two things I insist on now: **A service has an owner.** Not a committee. Someone accountable for its interface, its dependencies and its on-call. Unowned services rot in a specific way - nobody feels entitled to say no to a new dependency. **The interface is documented before it is built.** A short design note: what this service owns, what it exposes, what it depends on, what it explicitly does not do. Reviewing that document is dramatically cheaper than reviewing the pull request that implements the wrong thing, and it is where most of the mentoring actually happens. ## When not to do this The honest answer, and one worth saying out loud in the meeting: if your monolith is slow to deploy because tests are slow, extracting services will not fix that - you will have several slow test suites instead of one. Decomposition solves **independent deployment**, **independent scaling** and **independent ownership**. If none of those is your actual pain, a well-modularised monolith is a better system than a poorly-bounded set of services, and it is a fraction of the operational cost. The best outcome of a decomposition project is sometimes discovering, at step two, that you needed the modules and never needed the network. --- ## Leases Over Locks: A Crash-Safe Work Queue in PostgreSQL - URL: https://musfiqdehan.com/blog/lease-based-url-frontier-postgres - Published: 2026-06-18 - Topic: Distributed systems (10 min read) - Tags: PostgreSQL, Distributed Systems, Queues, Python, Concurrency - Summary: How to get a crash-safe job queue out of the database you already run, instead of adding another service to operate, monitor and be woken up by. Workers claim jobs on a timed lease, so a machine dying halfway through costs minutes of delay rather than lost work. ## The short version **In everyday terms.** Any system doing work in the background needs a way to hand jobs out to several machines without two of them grabbing the same job, and without work vanishing if a machine dies mid-task. The usual advice is to install a dedicated queueing product. The approach here uses the database that is already running. Think of it as a job board where you take a card, write your name on it and a time by which you will be finished. If that time passes with nothing handed back, the card returns to the board automatically and someone else picks it up. **Why a business should care.** Every additional piece of infrastructure has an ongoing cost: it has to be paid for, kept up to date, backed up, monitored, and understood by whoever is on call at 3am. Removing one from the design is a permanent saving. And because a crash returns work to the queue automatically, nobody spends a morning working out what was lost. **What follows** is the SQL that makes concurrent claims safe, the recovery process, retry backoff, and an honest list of when you should reach for a real broker instead. --- The first thing people tell you when you say "I'm using Postgres as a queue" is that you shouldn't. The second thing, usually, is a suggestion to add RabbitMQ. I did the opposite for [NewsCrawl](https://musfiqdehan.com/projects/newscrawl)'s URL frontier, and after a year of running it I would make the same call again. Not because message brokers are bad, but because the thing I actually needed was not message delivery. It was **coordinated claims over a mutable, prioritised work set** - and that is a database problem wearing a queue costume. ## What a frontier actually has to do A crawl frontier is not a stream of independent messages. It is a set of URLs where: - Priority changes after insertion. A breaking-news source jumps the line. - The same URL is scheduled repeatedly, on a cadence per source. - Failures need exponential backoff, not immediate redelivery. - You need to answer questions like "how many URLs are pending for source 12?" - a query, not a queue operation. - A worker crash must return its claims, not lose them. Model that in a broker and you end up rebuilding half a database on top of it: a side table for state, a separate store for scheduling, and a reconciliation job to keep them agreeing. Model it in Postgres and it's one table. ## The claim Everything rests on one statement: ```sql UPDATE frontier SET status = 'leased', lease_owner = $1, lease_expiry = now() + $2::interval, attempts = attempts + 1 WHERE id IN ( SELECT id FROM frontier WHERE status = 'ready' AND next_attempt_at <= now() ORDER BY priority DESC, next_attempt_at ASC FOR UPDATE SKIP LOCKED LIMIT $3 ) RETURNING id, url, source_id, attempts; ``` `FOR UPDATE SKIP LOCKED` is the entire trick. Without it, twenty workers running this concurrently serialise behind each other - each one blocking on rows the others have locked, so throughput collapses to single-worker speed under contention. With it, a worker that finds a row locked simply steps over it and takes the next candidate. Two properties fall out of that: **No worker ever waits for another worker.** Contention costs you a skipped row, not a blocked transaction. **No two workers ever get the same row.** The lock is held for the duration of the `UPDATE`'s transaction, and by the time it releases, `status` is `leased`, so the row no longer matches the inner `SELECT`. One subtlety that bites people: the `SKIP LOCKED` must be in the subquery that selects candidates, and the `LIMIT` must be there too. If you write it as a plain `UPDATE ... WHERE status = 'ready' LIMIT`, Postgres has no such form, and the naive rewrites re-introduce the blocking you were trying to avoid. ## Why a lease, and not a lock A held lock dies with its connection. That sounds like exactly what you want for crash safety - until you realise the work takes ninety seconds and you do not want a ninety-second transaction, or a connection held open for the duration of an HTTP fetch that might hang. So the claim transaction is short: mark the rows, commit, get out. What the worker holds afterwards is not a database lock but a **lease** - a row-level assertion that says "this is mine until `lease_expiry`." The consequence is that a crashed worker releases nothing. Its rows sit in `leased` with an expiry in the past, invisible to every other worker, forever. Which is why the reaper is not optional. ## The reaper ```sql UPDATE frontier SET status = 'ready', lease_owner = NULL, lease_expiry = NULL, next_attempt_at = now() + (interval '1 second' * least(pow(2, attempts) * 30, 21600)) WHERE status = 'leased' AND lease_expiry < now() RETURNING id; ``` A periodic sweep returns expired leases to `ready` and pushes their next attempt out by an exponentially growing delay, capped at six hours. Three things worth noting: **The backoff is applied by the reaper, not the worker.** A worker that crashed did not get the chance to record a failure. If backoff only happened on the explicit-failure path, a source that reliably kills workers would be retried in a tight loop forever. **`attempts` was already incremented at claim time.** Not at completion. This is deliberate: an increment at completion never runs for a worker that dies, so a URL that crashes workers would have an attempt count of zero forever and never age out. Counting attempts at claim time means "how many times have we tried this" includes the tries that ended badly. **The cap matters.** Uncapped exponential backoff schedules a retry after the heat death of the sun. Six hours means a source that was down for a morning recovers the same day. Set the lease interval to comfortably exceed your slowest legitimate task - I use roughly three times the p99 processing time. Too short and you get duplicate work from slow-but-healthy workers; too long and crash recovery drags. ## Priority scheduling for free Because the frontier is a table, priority is a column, and changing it is an `UPDATE`. That sounds trivial and it is exactly the thing brokers make hard. ```sql UPDATE frontier SET priority = 100 WHERE source_id = $1 AND status = 'ready'; ``` Boost a source and the very next claim reflects it. In a broker you would be republishing to a different queue, or maintaining priority queues you must poll in order. The index that makes this fast is the one that matches the claim's `ORDER BY`, restricted to the rows that can actually be claimed: ```sql CREATE INDEX frontier_claimable ON frontier (priority DESC, next_attempt_at ASC) WHERE status = 'ready'; ``` A partial index is the difference between scanning claimable work and scanning history. It also keeps the index small: completed rows, which dominate the table over time, are not in it at all. ## The three failure modes I actually hit **Dead tuple accumulation.** Every claim and every completion is an `UPDATE`, which in Postgres means a new row version. A high-throughput frontier generates dead tuples fast, and the default autovacuum thresholds - scaled to a fraction of table size - get lazier as the table grows. Set an aggressive per-table `autovacuum_vacuum_scale_factor` on this table specifically. I use `0.02`. **Long-running transactions blocking cleanup.** Vacuum cannot remove tuples still visible to an open transaction. One analytics query left open in a session pins the horizon and the frontier bloats. This is where the "don't use Postgres as a queue" advice originates - and it's really "don't leave transactions open," which was already true. **The lease interval as a hidden coupling.** When I added Playwright rendering, task duration tripled and the reaper started stealing work from healthy workers, which then finished and wrote results for URLs another worker was also processing. Fix: make the lease interval a property of the task class, not a global constant. Fast HTTP fetches keep a short lease; browser renders get a long one. ## When I would use a broker instead I am not arguing Postgres wins every time. Reach for a real broker when: - You need fan-out to multiple independent consumer groups - that is genuinely what streams are for, and it is why NewsCrawl's *processing* plane is Redis Streams even though its *frontier* is Postgres. - Throughput is beyond roughly a few thousand claims per second, where `UPDATE` churn becomes the bottleneck. - Producers and consumers are separate services that should not share a database. But when the work set is mutable, prioritised, queryable, and needs to survive a worker being killed mid-task - the database you already run, already back up, and already monitor is a genuinely good answer. The best part is what you don't get: no broker to operate, no second durability model, no reconciliation job that exists only because two systems disagree about what has been done. --- ## Four Stages of Deduplication: Spend Your Compute Where It Matters - URL: https://musfiqdehan.com/blog/four-stage-deduplication-pipeline - Published: 2026-05-27 - Topic: AI engineering (11 min read) - Tags: Deduplication, SimHash, Embeddings, pgvector, Multilingual NLP, PostgreSQL - Summary: The expensive way to find duplicates is to compare everything using AI. Ordering the checks cheapest-first removes roughly nine in ten duplicates with a simple lookup and saves the costly comparison for the small remainder - the same quality of result, at a fraction of the running cost. ## The short version **In everyday terms.** When you collect content from many sources, the same item shows up again and again in slightly different wording. Removing the repeats is easy to do badly: comparing everything against everything using AI works, but the cost grows far faster than the collection does. **Why a business should care.** This is a direct line on the infrastructure bill, and a direct line on product quality. Cheap checks - has this exact web address been seen, is this text byte-for-byte identical - catch the overwhelming majority for the price of a database lookup. The expensive comparison then runs only where it can actually change the answer. The same principle applies far beyond duplicates: it is how you keep AI features affordable at scale. **One decision worth stealing:** duplicates are linked, never deleted. When a threshold turns out to be wrong - and the first one always is - correcting it is an adjustment rather than a data-loss incident. **What follows** is the four-stage cascade in detail, with the thresholds and the failure modes. --- Ask someone to deduplicate a document corpus today and there's a good chance the answer is "embed everything and cluster." It works. It also costs more than it needs to by a factor I find hard to justify, and it fails in a specific, embarrassing way: it will happily decide that two different match reports about two different football games are the same story, because in embedding space they nearly are. The dedup pipeline in [NewsCrawl](https://musfiqdehan.com/projects/newscrawl) is a cascade of four stages, ordered by cost. Each stage only sees what the previous one could not resolve. | Stage | Catches | Cost per comparison | | --- | --- | --- | | 1. Normalised URL hash | Re-crawls, tracking-param variants | One index lookup | | 2. Raw content hash | Byte-identical syndication | One index lookup | | 3. SimHash banding | Near-duplicates, minor edits | Banded lookup, small candidate set | | 4. BGE-M3 cosine | Same event, independently written | Vector search | The distribution is what makes it worth building. In my corpus, the overwhelming majority of duplicates are caught by stages 1 and 2 - each a single index lookup. The embedding comparison, which is the expensive one, runs on a small tail. ## Stage 1: the URL, normalised properly The same article reaches you as: ``` https://example.com/news/story-123?utm_source=twitter&utm_medium=social https://example.com/news/story-123 http://www.example.com/news/story-123/ https://example.com/news/story-123#comments ``` Four URLs, one article. Normalisation before hashing: - Lowercase scheme and host; strip a leading `www.` - Drop the fragment entirely - Remove tracking parameters - the `utm_*` family, `fbclid`, `gclid`, `mc_cid`, and a per-source list for the ones that are site-specific - Sort the surviving query parameters, so ordering is not significant - Normalise the trailing slash consistently Then `sha256` the result and make it a unique index. The one that costs people real money is **not keeping a per-source parameter allowlist**. Some sites put the article id in a query parameter. Strip it globally and you collapse an entire site's catalogue into one document. Ask me how I know. ## Stage 2: hash the content, not the page Byte-identical content arrives constantly - syndicated wire copy, a source you crawl through two different index pages. But hashing the raw HTTP response is useless, because every response differs: a timestamp in the footer, a rotating ad slot, a CSRF token, a "related stories" block that reshuffles per request. So the hash is over *extracted* content - headline plus body text after boilerplate removal - with whitespace collapsed, quotes and dashes normalised to canonical forms, and Unicode normalised to NFKC. That last one is not optional for Bangla. The same visible text can be encoded with different codepoint sequences, and without NFKC two identical articles produce two different hashes. This is the kind of bug that hides for months because it silently *under*-detects: nothing breaks, you just quietly index duplicates. ## Stage 3: SimHash, and why banding is the whole point Now the interesting case. Two documents that differ by a corrected typo, an appended update, or a changed headline. Byte hashes are useless - one character changes the entire digest. SimHash produces a 64-bit fingerprint where **similar documents get similar fingerprints**, and similarity is Hamming distance. Documents within about 3 bits of each other are near-duplicates in practice. The problem: finding all fingerprints within 3 bits of a query is not an index lookup. Comparing against every stored fingerprint is a full scan, which defeats the purpose of putting a cheap stage before the expensive one. **Banding** fixes this. Split the 64-bit fingerprint into 4 bands of 16 bits and store all four as separate indexed columns. By the pigeonhole principle, two fingerprints differing in at most 3 bits must share at least one identical band - 3 differing bits cannot cover 4 bands. So: ```sql SELECT id, simhash FROM documents WHERE band_0 = $1 OR band_1 = $2 OR band_2 = $3 OR band_3 = $4; ``` That's four index lookups returning a small candidate set, and you compute exact Hamming distance only on those. A scan of the entire corpus becomes a handful of index probes. The tokenisation feeding SimHash needs care in a multilingual corpus. Word-level shingles work well for English and poorly for Bangla, where morphology packs more meaning per token. I use character n-grams for the Bangla path and word shingles for English, chosen per document by detected language, with thresholds tuned separately for each. One global threshold across both languages is a compromise that is wrong for both. ## Stage 4: embeddings, for the case only meaning can settle What survives to stage 4 is the genuinely hard case: two outlets covering the same event, written independently. No lexical overlap worth mentioning. Same story. BGE-M3 embeddings in `pgvector`, cosine similarity, threshold tuned against a hand-labelled set: ```sql SELECT id, 1 - (embedding <=> $1) AS similarity FROM documents WHERE published_at > now() - interval '48 hours' ORDER BY embedding <=> $1 LIMIT 10; ``` Two things make this survivable in production. **The time window.** Semantic similarity without a temporal bound will match this week's election story to last year's. Restricting to a recent window is both a correctness fix and the reason the query is fast. **BGE-M3 specifically** - because it is genuinely multilingual, which means a Bangla article and its English counterpart land near each other in the same space. A monolingual model per language would need a separate cross-lingual strategy entirely. Every stored vector carries the model version that produced it. Comparing vectors across model versions produces confident nonsense, and a model upgrade should be a backfill you can run gradually, not a cliff you fall off. ## The decision that saved me: link, don't delete The first version deleted duplicates. Within a week I had three different threshold bugs, and each one had permanently destroyed data. Now every suppressed document is retained with a pointer to its canonical version, the stage that made the call, and the score at the time: ```sql CREATE TABLE duplicate_links ( duplicate_id bigint PRIMARY KEY REFERENCES documents(id), canonical_id bigint NOT NULL REFERENCES documents(id), stage text NOT NULL, -- url | content | simhash | semantic score real, linked_at timestamptz NOT NULL DEFAULT now() ); ``` Three things this buys: **Reversibility.** A bad threshold is an `UPDATE` away from being undone. **Observability.** "How many stage-4 links did we create yesterday, and at what scores?" is the query that tells you a threshold has drifted - before someone reports it. **Ground truth.** Every link is a labelled pair. Reviewing the ones near the threshold gives you the evaluation set you need to tune the next version. ## What generalises The pattern is not really about deduplication. It's: **when a decision can be made by several methods of wildly different cost, order them by cost and let the cheap ones absorb the volume.** Reserve the expensive method for the residue that actually needs it - and keep the decision reversible, because the first threshold you pick will be wrong. --- ## At-Least-Once Is a Promise You Have to Keep - URL: https://musfiqdehan.com/blog/at-least-once-redis-streams - Published: 2026-05-06 - Topic: Backend (10 min read) - Tags: Redis, Streams, Reliability, Python, Idempotency, Queues - Summary: Retries are what stop a system losing work - and what make it do the same work twice. This is how to get the reliability without the duplicates: handlers safe to re-run, retries that back off, a dead-letter queue you can actually triage, and recovery for messages a crashed worker left behind. ## The short version **In everyday terms.** When a computer hands a job to another computer, it can guarantee the job arrives, or it can guarantee the job arrives only once - not both. Every serious queueing system chooses "arrives at least once", which means the same instruction will sometimes be carried out twice. The fix is not a better queue; it is writing each job so that doing it twice produces exactly the same result as doing it once. **Why a business should care.** This is the machinery that decides whether a retry sends a customer a second invoice, counts a shift twice on a payslip, or quietly drops an order nobody notices until the month-end reconciliation. Get it right and failures become delays. Get it wrong and failures become disputes. **What follows** is the production detail: idempotent handlers, delayed retries, dead-letter streams and reclaiming stalled messages. --- "At-least-once delivery" is written in documentation as a guarantee. Read it again as a warning: **your handler will run more than once on the same message, and the system is telling you in advance so you cannot claim surprise.** The processing plane in [NewsCrawl](https://musfiqdehan.com/projects/newscrawl) runs cleaning, LLM extraction and embedding as separate Redis Streams consumer groups. Here is what it took to make at-least-once actually safe. ## Consumer groups in one paragraph `XADD` appends to a stream. A consumer group tracks a delivery cursor and a **Pending Entries List** - messages delivered to some consumer but not yet acknowledged. `XREADGROUP` delivers new messages and moves them into the PEL. `XACK` removes them. Anything in the PEL is, by definition, work someone claimed and did not finish. The PEL is the whole reliability story. It is also the thing that quietly fills up when you get this wrong. ## Rule 1: idempotency is the handler's job, not the broker's There is no broker configuration that gives you exactly-once. Not in Redis, not in Kafka, not in SQS. The delivery layer can guarantee "at least once"; only the handler can make a redelivery harmless. The mechanism that works is a **deterministic key derived from the content**, not from the message id: ```python def handle(msg_id: str, fields: dict) -> None: doc_id = fields["doc_id"] stage = "embedding" # Content-derived key: the same document always produces the same key, # no matter how many times or through which message it arrives. key = f"{stage}:{doc_id}:{fields['content_hash']}" with db.transaction(): if db.exists("stage_results", key=key): redis.xack(STREAM, GROUP, msg_id) # already done - ack and move on return result = embed(fields["text"]) db.upsert("stage_results", key=key, result=result) redis.xack(STREAM, GROUP, msg_id) ``` Two details that matter more than they look: **The key is content-derived.** Keying on the Redis message id fails immediately - a reclaimed message gets a *new* id, so an id-keyed guard sees fresh work every time. **The `XACK` is after the commit, not before.** Ack first and a crash in between loses the message permanently. Commit first and a crash in between means a redelivery - which the guard at the top absorbs. Choose the failure you can recover from. ## Rule 2: retries need to be delayed, and delays need somewhere to live Redis Streams has no native delayed redelivery. If you handle a failure by not acking, the message sits in the PEL until something reclaims it - which means your retry cadence is your reclaim interval, for every failure, regardless of cause. I use a sorted set as a delay buffer: ```python def defer(msg_id: str, fields: dict, attempt: int) -> None: delay = min(2 ** attempt * 5, 900) # 5s → 15min ceiling ready_at = time.time() + delay redis.zadd(DELAY_SET, {json.dumps({**fields, "attempt": attempt}): ready_at}) redis.xack(STREAM, GROUP, msg_id) # release the PEL slot # a scheduler pass, every second def promote() -> None: now = time.time() due = redis.zrangebyscore(DELAY_SET, 0, now, start=0, num=100) for payload in due: if redis.zrem(DELAY_SET, payload): # atomic claim - only one promoter wins redis.xadd(STREAM, json.loads(payload)) ``` The `zrem`-returns-1 check is the concurrency guard. Multiple scheduler instances can run this loop; only the one whose `ZREM` actually removed the member republishes. And note that deferring **acks the original message**. A failed message that stays in the PEL is a message that will also be reclaimed, and now the same work exists in two places. ## Rule 3: a dead-letter stream, with the context attached Some messages will never succeed. A malformed document, a URL that 404s permanently, a schema the extractor cannot satisfy. Retrying those forever burns capacity that healthy work needs. After a bounded number of attempts, the message goes to a dead-letter stream - and the useful part is what travels with it: ```python redis.xadd(DLQ_STREAM, { **fields, "failed_at": now_iso(), "attempts": attempt, "stage": stage, "error_type": type(exc).__name__, "error": str(exc)[:1000], "traceback": tb[:4000], }) ``` A DLQ without the error context is a graveyard. A DLQ with it is a triage queue - you can group by `error_type`, notice that four hundred failures share one cause, fix it, and replay the batch back onto the main stream. Replay is just an `XADD` back to the source stream. Which works precisely because Rule 1 made the handlers idempotent. ## Rule 4: reclaim what the dead left behind A worker killed mid-message leaves an entry in the PEL that nobody will ever ack. `XAUTOCLAIM` sweeps them: ```python def reclaim() -> None: cursor = "0-0" while True: cursor, messages, _ = redis.xautoclaim( STREAM, GROUP, MY_CONSUMER, min_idle_time=STAGE_VISIBILITY_MS, start=cursor, count=50, ) for msg_id, fields in messages: delivery_count = pel_delivery_count(msg_id) if delivery_count > MAX_DELIVERIES: to_dlq(msg_id, fields, reason="poison") # kills the poison-pill loop else: handle(msg_id, fields) if cursor == "0-0": break ``` `min_idle_time` must exceed your slowest legitimate processing time, or you will reclaim work from consumers that are merely slow - and then two workers are processing the same message concurrently. When I added LLM extraction, p99 went from two seconds to forty, and my ten-second idle threshold started stealing work from healthy consumers. **Visibility timeout is a per-stage property, not a global constant.** The `delivery_count` check is the poison-pill guard. A message that crashes its handler will be reclaimed, crash again, be reclaimed again - forever, consuming a worker each cycle. Bounding deliveries turns an infinite loop into a DLQ entry. ## What I watch Four signals, and they are the same four for any stream system: - **PEL depth per group.** Rising means consumers are claiming faster than they finish. This leads every other symptom. - **Oldest PEL entry age.** A single stuck message that reclaim is not catching. - **DLQ arrival rate.** A step change here is a deploy that broke something. - **Stream length vs. consumed offset.** The genuine backlog, independent of consumer health. Queue depth is the metric that tells you about an incident before your users do. It is worth wiring to Prometheus on the first day, not the day after the first outage. ## The short version Redis Streams gives you durable delivery and per-group cursors. It does not give you correctness. Correctness is four things you build on top: 1. Handlers keyed on content, so redelivery is a no-op. 2. Delayed retries in a sorted set, so failures back off instead of spinning. 3. A dead-letter stream carrying enough context to triage and replay. 4. Reclaim with a per-stage idle threshold and a delivery-count ceiling. Skip any one and at-least-once becomes at-least-once-and-occasionally-wrong. --- ## Schema-per-Tenant Django: Isolation You Cannot Forget to Apply - URL: https://musfiqdehan.com/blog/schema-per-tenant-django-multi-tenancy - Published: 2026-04-14 - Topic: Architecture (12 min read) - Tags: Django, PostgreSQL, Multi-Tenancy, SaaS, RBAC, Architecture - Summary: Sharing one database between customers is one forgotten filter away from showing one client another client's data. Giving each customer their own separate space moves that safety from something developers must remember into something the database enforces - and here is exactly what it costs. ## The short version **In everyday terms.** When one product serves many customer companies, all of their data usually lives in the same database, kept apart by the software remembering to ask for the right customer every single time. That works until somebody forgets once. The failure is the worst kind in business software: one client seeing another client's records. The alternative is to give each customer their own separate space inside the database. Then the question is not "did we remember to filter?" - the other customer's records are simply not visible from where the query is running. **Why a business should care.** For products sold to other businesses, data separation is usually a contractual promise and a question on every security review. Structural separation is also what makes per-customer branding, custom web addresses and restoring a single customer's backup practical rather than theoretical - all of which are things a sales team can actually sell. **And what it costs, honestly:** every change to the data structure has to run once per customer, connections need more careful management, and this approach stops being sensible somewhere in the low thousands of customers. **What follows** is the full accounting - how it works, what breaks, and how to decide which model fits your product. --- There are three ways to isolate tenants in a SaaS product, and the industry mostly picks the first one because it is the easiest to start with. **Row-level.** One schema, a `tenant_id` column on every table, a filter on every query. Cheapest to run. One forgotten `WHERE` clause away from showing customer A customer B's data. **Database-per-tenant.** Total isolation. Also a connection pool per tenant, a migration run per tenant, and an operational burden that grows linearly with sales. **Schema-per-tenant.** One database, one connection pool, a PostgreSQL schema per tenant. Isolation comes from the search path, so the query does not have to remember anything. [FitPulse](https://musfiqdehan.com/projects/fitpulse) uses the third. Here is the honest accounting. ## How it works PostgreSQL's `search_path` decides which schema an unqualified table name resolves to. Set it per connection and the same ORM query hits different physical tables: ```sql SET search_path TO tenant_acme, public; SELECT * FROM members; -- tenant_acme.members ``` Django middleware resolves the tenant from the request host - subdomain or fully custom domain - and sets the path for the life of the request: ```python class TenantMiddleware: def __call__(self, request): tenant = resolve_tenant(request.get_host()) if tenant is None: return HttpResponseNotFound() with connection.cursor() as cursor: cursor.execute("SET search_path TO %s, public", [tenant.schema_name]) request.tenant = tenant try: return self.get_response(request) finally: # Connections are pooled and reused. Never hand one back dirty. with connection.cursor() as cursor: cursor.execute("SET search_path TO public") ``` That `finally` block is the single most important thing on this page. Django reuses connections across requests, and PgBouncer reuses them across *everything*. A connection returned to the pool with tenant A's search path still set will serve tenant B's next request against tenant A's tables. It is a silent, total, cross-tenant data leak, and it will not show up in any test that runs one request at a time. If you use PgBouncer in transaction pooling mode, `SET` does not survive the transaction boundary - you must set the path inside the transaction that uses it, or use `set_config(..., true)` for transaction-local scope. Getting this wrong produces intermittent failures that look like flakiness and are not. ## What you get **Isolation you cannot forget.** No `tenant_id` filter to omit. A junior developer writing `Member.objects.all()` gets exactly the current tenant's members, because the other tenants' members are in schemas that are not on the path. **Per-tenant operations.** Restore one customer without touching the others: `pg_dump --schema=tenant_acme`. Under row-level tenancy, restoring one customer from a backup is a delicate surgical extraction. **Honest query plans.** Each tenant's tables carry their own statistics. Under row-level tenancy, one enormous customer skews the planner's estimates for everyone, and small tenants get plans optimised for a distribution they don't have. **Custom domains for free.** The tenant is resolved from the host, so `app.customer.com` is a DNS record and a row, not an architecture change. ## What it costs I would be selling you something if I stopped there. **Migrations multiply.** A schema change runs once per tenant. At ten tenants, that is a script. At a thousand, it is a job with progress tracking, resumability, and a very clear answer to "what happens when tenant 400 fails halfway." The pattern that survives is: migrations must be **backward compatible and independently applicable**. Add a nullable column, deploy code that tolerates both shapes, backfill per tenant, then tighten the constraint. Never a migration that requires all tenants to be at the same version simultaneously - because during a rollout, they aren't. **Cross-tenant queries get hard.** "Total active members across all customers" cannot be one query. It is a loop over schemas, or an aggregate table in `public` that tenants write to. I use the latter - a small reporting schema updated on write - because looping over a thousand schemas for a dashboard is not a plan. **Connection pooling needs discipline.** Covered above, and worth repeating: every path change needs a guaranteed reset. **Postgres catalog pressure.** Each schema multiplies your table count. A thousand tenants with fifty tables each is fifty thousand tables, and `pg_catalog` starts showing up in query plans. It works, but `\dt` becomes a bad idea and autovacuum needs tuning. This is the real ceiling on the approach, and it arrives further out than most products ever get. ## Isolation inside a tenant One schema per customer solves *between*-customer isolation. It does nothing for the structure inside a single customer - and a gym chain is many branches with staff who must only see their own. That layer is row-level, and it belongs in the queryset, not the view: ```python class BranchScopedManager(models.Manager): def for_request(self, request): qs = self.get_queryset() if request.user.is_tenant_admin: return qs return qs.filter(branch__in=request.user.accessible_branches) ``` Views ask for "the members I can see." The manager decides what that means. Putting it in the view means every new view is a new opportunity to forget. ## Authorization is a three-way resolution The permission question in a SaaS product is never just "is this user allowed?" It's: 1. **Does the plan include this feature?** Subscription tier - a billing question. 2. **Does the role permit this action?** RBAC - an organisational question. 3. **Is this record in reach?** Branch scoping - a data question. Any one of the three denying is a deny. Keeping them as three separate, individually-testable checks is what stops "premium feature" and "manager permission" from getting tangled into a single boolean nobody can reason about six months later. The failure mode when they *are* tangled: a customer downgrades their plan, and a manager who still has the role keeps access to a feature they are no longer paying for. That's a revenue bug produced by an architecture decision. ## Would I choose it again For this product, yes - the customers are businesses whose data separation is a contractual expectation, tenant count is in the hundreds rather than the hundreds of thousands, and per-tenant restore has already been needed twice. For a consumer product with a million accounts, no. Schema-per-tenant at that cardinality is a catalog problem, and row-level tenancy with rigorously enforced query-layer scoping is the right trade. The question to ask is not "which is best." It's: **how many tenants, and what is the cost of a leak between them?** Those two numbers pick the architecture for you. --- ## Holding 10k Daily Users Before You Buy a Bigger Server - URL: https://musfiqdehan.com/blog/holding-10k-dau-before-you-scale-up - Published: 2026-03-19 - Topic: DevOps (11 min read) - Tags: PostgreSQL, PgBouncer, Django, Performance, Redis, Deployment - Summary: A platform went from a few hundred to 10,000+ daily users without a bigger server. Four fixes did all the work - and buying hardware first would have hidden every one of them while the monthly bill kept growing. ## The short version **In everyday terms.** When software gets slow, the reflex is to rent a bigger machine. It usually works for a few months, costs more every month afterwards, and teaches you nothing about why it was slow - so you do it again next quarter. **Why a business should care.** This platform absorbed roughly twenty times the users on the same hardware. Four causes were responsible, and none of them was a genuine shortage of computing power: a self-inflicted limit on database connections, reports asking the database a thousand small questions instead of one, missing indexes on exactly the search everybody performs, and dashboard figures being recalculated from scratch on every page view. **The lesson for anyone approving an infrastructure budget:** "we need a bigger server" is sometimes true, but it is worth asking which of these four it is first. Each was a day of work that has kept paying every month since. **What follows** is the diagnosis and fix for each, in the order they mattered. --- The instinct when a platform slows down is to make the server bigger. It works, briefly, and it teaches you nothing - which means you will be doing it again next quarter. [FitPulse](https://musfiqdehan.com/projects/fitpulse) went from a few hundred to over ten thousand daily active users without a proportional increase in hardware. Four things did the work, in order of how much they mattered. ## 1. Connection exhaustion, which is not load The first symptom looked like an outage: `FATAL: sorry, too many clients already`. CPU was comfortable. Memory was comfortable. The database was refusing connections. Django opens a connection per request per worker and holds it for the request's duration. With `CONN_MAX_AGE` set, it holds it *between* requests too. Multiply by gunicorn workers, multiply by application instances, and you cross Postgres's `max_connections` while the database is doing almost nothing. The tempting fix is raising `max_connections`. Don't. Every Postgres connection is a backend process with its own memory, and a thousand mostly-idle connections cost real RAM and real scheduler pressure while making the eventual failure worse. The actual fix is **PgBouncer in transaction pooling mode**. Applications connect to PgBouncer; PgBouncer maintains a small pool of real backends and hands one to a transaction only while it is running. Hundreds of application connections multiplex onto a few dozen real ones, because at any instant most "connections" are idle. Transaction pooling has rules you must respect: - **No session state.** Session-level `SET`, advisory locks, `LISTEN/NOTIFY` and `WITH HOLD` cursors do not survive, because the next statement may land on a different backend. If you use `search_path` for [multi-tenancy](https://musfiqdehan.com/blog/schema-per-tenant-django-multi-tenancy), it has to be transaction-local. - **`CONN_MAX_AGE = 0` in Django.** Let PgBouncer own pooling. Two poolers stacked on each other is a way to have the problems of both. - **Prepared statements need care.** Disable Django's server-side prepared statements, or adopt PgBouncer's prepared-statement support deliberately rather than by accident. This single change removed the failure mode entirely, and cost one small process. ## 2. N+1 queries, found rather than guessed Every ORM makes N+1 queries easy to write and invisible to read. A member list page that looked fine at fifty members issued four hundred queries at four hundred. The fix is `select_related` for forward foreign keys and `prefetch_related` for reverse and many-to-many relations. That part is well known. The part that actually mattered was **not guessing**: ```python from django.db import connection from django.test.utils import CaptureQueriesContext with CaptureQueriesContext(connection) as ctx: response = client.get("/api/members/?branch=3") assert len(ctx.captured_queries) < 12, f"{len(ctx.captured_queries)} queries" ``` An assertion on query count, in the test suite, per endpoint. It turns a performance regression into a failing test - which is the only mechanism I have found that keeps N+1s from coming back the moment someone adds a nested field to a serializer. ## 3. Indexes that match the queries you actually run Attendance is the highest-volume table in the product, and every report filters it the same way: one branch, one date range. A single-column index on `branch_id` finds all attendance for a branch - thousands of rows - and then filters them by date in memory. The index that matches the query is composite, and column order is not arbitrary: ```sql CREATE INDEX attendance_branch_time ON attendance (branch_id, checked_in_at DESC); ``` **Equality columns first, range columns second.** Postgres can seek to `branch_id = 3` and then walk the date range in index order. Reverse the order and it cannot. Add `DESC` when your queries are "most recent first," which for attendance is all of them - it lets the planner satisfy the `ORDER BY` from the index and skip the sort entirely. The other habit worth having: `EXPLAIN (ANALYZE, BUFFERS)` on the query as it is actually issued, with realistic parameters. A plan over a hundred rows tells you nothing about a plan over a million, and the planner switches strategies between them. ## 4. Stop recomputing what has not changed The gym owner's dashboard aggregated today's check-ins, active memberships, expiring plans and revenue. Four aggregate queries over large tables, on every page load, for every owner, all day. None of that data needs to be a second old. Caching it in Redis with a short TTL, invalidated on the writes that actually affect it, removed the largest single source of database load in the system. The subtlety is stampede protection. When a hot key expires, every concurrent request misses simultaneously and all of them recompute. Under load that is worse than no cache. A short lock around recomputation - first requester computes, everyone else waits briefly and reads the fresh value - is a few lines and prevents a thundering herd on your most expensive query. And the related change: **WebSocket instead of polling.** The live check-in feed used to poll every five seconds per open dashboard. Pushing on the event that already existed removed a constant, useless query floor that scaled with the number of screens open in the building. ## Deploys that don't count as downtime None of the above helps if every release is a gap in service. Rolling container swaps behind Traefik with health gating: the new container starts, its health check passes, traffic shifts, the old one drains. A build that fails its health check never receives a request. Rollback is deploying the previous tag. The constraint this puts on you is that **both versions run simultaneously during the swap.** Every migration must be compatible with the code on either side of it. Additive changes deploy freely; destructive ones become a three-step dance across releases - add, migrate, remove. It is slower to write, and it is the reason deploys are boring. ## The order matters Notice that hardware never appears. Connection exhaustion looked like load and was not. N+1s looked like slow queries and were a count problem. Missing indexes looked like table size and were a shape problem. The dashboard looked like traffic and was recomputation. Scaling up would have masked all four for a quarter, at a permanent cost, and I would have learned nothing about the system I run. --- ## One API, Three Clients: A Dashboard, a Phone and a Turnstile - URL: https://musfiqdehan.com/blog/one-api-three-clients - Published: 2026-02-24 - Topic: Full stack (10 min read) - Tags: API Design, Django, React Native, IoT, Offline Sync, WebSocket - Summary: What changes when one backend must serve a web dashboard, a phone app, and a fingerprint terminal that cannot be updated, cannot be trusted about time and is usually offline. The most awkward client turned out to be the best specification the API ever got. ## The short version **In everyday terms.** Most software is designed for one kind of user on one kind of screen. This system serves three at once: a manager on a dashboard, a member on a phone, and a fingerprint reader on a wall. The reader is the difficult one - it cannot be updated, its clock is often wrong, and it spends real portions of the day disconnected. **Why a business should care.** Designing for the most constrained client made the whole product better. Because the device might send the same event twice, every write became safe to repeat - which also protects the phone app when a member taps a button twice on a bad connection. Because the device cannot be updated, the system can only add to its interface and never remove, which means old versions of the mobile app keep working after a release. The awkward client paid for reliability everywhere else. **What follows** is the concrete design: the three clients' conflicting assumptions, why the hardware reshaped the contract, and how untrusted clocks are handled. --- Most API design advice assumes one consumer: a web frontend built by the same team, in the same sprint, deployable at the same moment. [FitPulse](https://musfiqdehan.com/projects/fitpulse) has three consumers with almost nothing in common. A React dashboard used by gym owners. A React Native app in members' pockets. And a ZKTeco biometric turnstile sitting on a gym's local network, which cannot reach the internet, cannot be redeployed, and does not care about your API conventions. The third one changed how I design APIs generally. ## Three clients, three different assumptions | | Dashboard | Mobile app | Device bridge | | --- | --- | --- | --- | | Network | Reliable | Intermittent | Frequently offline | | Update cadence | Deploy any time | Days to weeks (app stores) | Effectively never | | Payload tolerance | Large | Bandwidth-sensitive | Tiny | | Clock | Trustworthy | Mostly | Untrusted | | Auth lifetime | Session | Long-lived refresh | Machine credential | You cannot satisfy all three by designing for the easiest one and hoping. What you can do is design for the *hardest* one and let the others benefit. ## Versioning becomes real the day you ship a mobile app On the web, a breaking API change is coordinated: deploy backend and frontend together, done. On mobile, a user running a four-month-old build is a normal, common, permanent condition. There is no "deploy the client." So the rules I now hold to: - **Additive changes only, within a version.** New fields are fine; renaming or removing a field is a new version. - **Never repurpose a field.** Changing `status` from a boolean to an enum breaks every old client silently - they get a truthy string and take the wrong branch. - **Every response carries a shape the oldest supported client can parse.** New capability goes in new fields the old client ignores. The mobile app is what makes these rules non-negotiable, and the whole API is better for them. ## The device: assume nothing The turnstile speaks a proprietary protocol over the local network. It has no route to the cloud. The bridge is a small LAN agent that talks to the device locally and syncs upstream, and every assumption I would normally make had to be discarded. **The clock is untrusted.** Device clocks drift, get reset by power cuts, and are occasionally set by someone in the building who thought they were helping. So attendance carries two timestamps: `device_time`, what the hardware claimed, and `received_at`, when the server got it. Payroll uses one of them and the audit trail keeps both - because "we computed overtime from a clock that was ninety minutes fast" is a conversation you only want to have once, and you want records when you do. **The connection is not there.** The agent buffers locally and syncs when it can. Which means the API receives batches of events that are minutes or hours old, out of order, and possibly overlapping with a previous partial sync. **Retries are guaranteed, not possible.** An agent that times out mid-upload will resend. It has no way of knowing whether the server processed the batch before the connection dropped. That last point is the one that shaped the contract. ## Idempotency as a first-class part of the contract Every event carries a deterministic id derived from its content - device serial, user id, and the device timestamp: ``` event_id = sha256(f"{device_serial}:{user_id}:{device_time_iso}") ``` The write is an upsert keyed on that id. A batch delivered twice produces the same rows. A batch delivered half-way and then re-sent in full completes cleanly, with the already-written half absorbed. The endpoint returns per-item results, not a single status: ```json { "accepted": 47, "duplicates": 12, "rejected": [ { "event_id": "a3f…", "reason": "unknown_member" } ] } ``` A partial-success response is what lets the agent clear exactly the events that landed and keep retrying the ones that didn't. An all-or-nothing endpoint forces the agent to choose between resending everything forever or dropping data - and it will eventually choose wrong. This is the design the mobile app quietly benefits from too. A member checking in on a train, in and out of signal, hits the same idempotent path. ## Shaping responses without three codebases Three clients want three payload shapes. The dashboard wants members with their plan, branch and last visit embedded. The mobile app wants a fraction of that. The device wants an id and a name. Three separate endpoints means three implementations of the same permission logic, and eventually they disagree. Instead: **one resource, explicit field selection.** ``` GET /api/v1/members?fields=id,name,status&expand=plan ``` One queryset, one permission path, one place where [branch scoping and RBAC](https://musfiqdehan.com/blog/schema-per-tenant-django-multi-tenancy) are enforced. The serializer honours `fields` and `expand`, and the query layer uses the same information to decide what to `select_related` - so asking for less data also *costs* less, rather than fetching everything and discarding it at the edge. ## Real-time where it is worth it The dashboard shows check-ins as they happen. The first version polled every five seconds; with dozens of dashboards open across branches, that was a permanent query floor unrelated to whether anything had happened. WebSocket fan-out replaced it. The event already existed - the device sync writes it - so pushing to subscribed dashboards costs nothing extra. The mobile app deliberately does *not* hold a socket. Battery and background execution constraints make a persistent connection the wrong trade for a client that needs to know about a handful of events a day. It uses push notifications and refreshes on foreground. **Same backend event, different transport per client, chosen by that client's constraints.** ## What the hardware taught me The device could not be updated, could not be trusted about time, and could not stay connected. Designing for it forced idempotent writes, partial-success responses, additive-only evolution and explicit payload control. Every one of those made the dashboard and the mobile app more robust too. The hardest client is the best specification you will get. --- ## Let the Agent Propose, Not Write - URL: https://musfiqdehan.com/blog/let-the-agent-propose-not-write - Published: 2026-01-29 - Topic: AI engineering (10 min read) - Tags: LLM, Agents, Gemini, LangChain, Reliability, Architecture - Summary: The single boundary that separates an AI feature you can safely point at real customer data from a demo you cannot: the model proposes, the application decides. Plus validated output, provider failover and cost tracking treated as engineering problems rather than prompting problems. ## The short version **In everyday terms.** AI assistants are impressive right up until they misunderstand something. If a misunderstanding results in a change that has already been made to real records, you do not have a feature - you have a way of generating incidents with a friendly interface. The fix is a single rule: **the AI is allowed to suggest, never to act.** It prepares a precise, structured request; ordinary software then checks that request against the same permission rules a human would face, and only then carries it out. A misunderstood instruction becomes a rejected suggestion instead of a corrupted customer list. **Why a business should care.** This is what makes an AI feature launchable rather than permanently stuck in a demo. It also makes the behaviour auditable: every action has a record of what was asked, what was proposed, and what was allowed. Three related concerns get the same treatment - output is validated before it is stored, a second provider takes over when the first fails, and spend is tracked per request so the monthly cost is knowable in advance. **What follows** is how that boundary is implemented, and what happens at each point it is tested. --- The demo is always impressive. You say "add everyone who opened the March campaign to a new segment and tag them warm," and it happens. The question that decides whether it ships is different: **what happens when the model misunderstands?** If the answer involves an `UPDATE` that already ran, you do not have a feature. You have an incident generator with a friendly interface. [ECamp](https://musfiqdehan.com/projects/ecamp) has a Gemini-powered agent for contact management, driven by natural language and voice. It runs against production data. It does so because of one architectural rule. ## The rule **The agent never writes. It proposes.** The model's output is not an action. It is a structured proposal, validated against a declared schema, subjected to the same permission checks a human request would face, and executed by ordinary application code that has no idea an LLM was involved. ```python class AddToSegment(BaseModel): action: Literal["add_to_segment"] filter: ContactFilter # a constrained query, not raw SQL segment_name: str tags: list[str] = [] class DeleteContacts(BaseModel): action: Literal["delete_contacts"] filter: ContactFilter confirm_count: int # the model must state how many it expects Proposal = Annotated[ Union[AddToSegment, DeleteContacts, ...], Field(discriminator="action"), ] ``` The model chooses among operations that exist. It cannot invent one, cannot compose raw SQL, and cannot reach anything outside the declared set. An intent that maps to no operation fails validation and becomes a clarifying question - which is exactly the right behaviour. Note `confirm_count` on the destructive operation. The model must state how many records it believes it is affecting. If the count and the filter disagree by more than a small margin, the proposal is rejected before execution. A misunderstanding becomes a rejection, not a deletion. ## The three checks between proposal and execution **Schema validation.** Does this parse into a known operation with well-typed fields? Invalid output triggers a repair retry with the validation error included as context - models are markedly better at fixing a specific complaint than at getting it right on the second blind attempt. After a small number of failed repairs, it is an error, not an infinite loop. **Permission resolution.** The proposal runs through the exact authorization path a human API request would. Same tenant scoping, same role checks, same subscription gate. The agent is not a privileged caller; it is a caller that happens to have been assembled by a model. **Blast radius.** Any operation exceeding a threshold - record count, irreversibility, cost - requires explicit human confirmation with the effect described in plain language: "This will permanently delete 3,412 contacts across 2 segments." Not the model's summary of what it thinks it is doing. The application layer's description of what will actually happen. Together these mean the worst case of a misunderstood instruction is a rejected proposal or a confirmation dialog. Never a silent mutation. ## Schema-validated extraction, everywhere The same discipline applies outside agents. In [NewsCrawl](https://musfiqdehan.com/projects/newscrawl), the LLM extracts structured fields from article text, and the pipeline treats every response as untrusted input: - Parse against the declared schema. Reject on mismatch. - Retry with the validation error attached, bounded. - On repeated failure, record the document as extraction-failed and continue. One bad document must not stop a pipeline. The alternative - parsing hopefully and writing what you got - produces a database where some fields are strings, some are strings containing JSON, and one is an apology. Every downstream consumer inherits that mess forever. ## Providers are dependencies, so treat them like dependencies An LLM API is a third-party network service with rate limits, latency spikes and outages. Nothing about it deserves special trust. **Fallback.** Primary provider rate-limits or times out, the call routes to a secondary. The pipeline degrades in quality rather than stopping. Which requires that prompts and schemas are not tuned so tightly to one provider that they break on another - a constraint worth accepting up front. **Timeouts and circuit breaking.** A hanging call holds a worker. Aggressive timeouts, and a breaker that stops sending to a provider that is failing consistently, so you fail fast instead of queueing into a wall. **Cost accounting per call.** Tokens in, tokens out, model, cost, attributed to the document or campaign that caused it: ```python usage.record( provider=provider.name, model=provider.model, prompt_tokens=r.usage.prompt_tokens, completion_tokens=r.usage.completion_tokens, cost_usd=provider.price(r.usage), attributed_to=doc_id, ) ``` This is the least glamorous part and the one I would keep if I could only keep one. Without it, "is this feature worth its cost?" is a discussion. With it, it's a query. It is also how you find the prompt change that quietly tripled your bill, in the week it happens rather than at the end of the month. ## Idempotency, again Retries mean the same extraction can run twice on the same document. Same rule as everywhere else in the system: key results on the content, upsert rather than insert, and a repeated call is a wasted API call rather than a duplicated row. Cache on that key too. In a pipeline that reprocesses documents after a schema change, the cache is often the majority of the savings. ## Where the intelligence actually goes The pattern underneath all of this: **the model supplies intent; the application supplies authority.** The model is genuinely good at turning "everyone who opened the March campaign, tag them warm" into a structured filter. It is not good at deciding whether the caller may do that, whether three thousand records is a reasonable blast radius, or whether the operation is reversible. Those are questions with correct answers your code already knows. Keep the boundary sharp and you get the useful part of an LLM without handing it the keys. Blur it and you have built a system whose safety properties depend on a probability distribution. --- ## A Release Should Be One Git Tag - URL: https://musfiqdehan.com/blog/tag-driven-python-releases - Published: 2025-12-11 - Topic: Engineering practice (9 min read) - Tags: Python, Poetry, GitHub Actions, CI/CD, Open Source, Versioning - Summary: Every manual step in a release is a step somebody skips at 11pm - and the skipped step is the one customers find. How a forty-minute checklist became one command, with version numbers, release notes, builds and publishing all produced by automation. ## The short version **In everyday terms.** Releasing software used to mean a checklist: update the version number in two places, write the release notes, build it, upload it, tag it. Every item on that list is something a tired person will eventually skip, and the skipped item is discovered by users. Now the whole release is one command. Everything else - tests, version number, release notes, packaging, publishing to three places - is produced automatically from the work that was already done. **Why a business should care.** Releases that are painful get postponed, and postponed releases arrive as large risky batches instead of small safe ones. Making a release cheap is what lets fixes reach customers the same week they are written. Automatic version numbering also tells users whether an update is routine or risky, which is precisely what a customer needs to know before accepting it. **What follows** is the pipeline itself, and why the same machinery is what a production deployment needs. --- The first release of [Data Preprocessors](/projects/Data Preprocessors) took me about forty minutes. Bump the version in `pyproject.toml`. Remember to bump it in `__init__.py` too. Write the changelog. Build. Realise the build included a stale artefact from last time. Clean. Rebuild. Upload. Tag the commit. Push the tag. The second release took forty minutes as well, minus the parts I forgot. Every one of those steps is a step someone will skip when they are shipping a fix at 11pm. So the release became one command: ```bash git tag v1.2.3 && git push --tags ``` Everything else is CI's problem. ## The version lives in exactly one place Two sources of truth for a version number is one too many. The tag is the source; the package derives from it. ```toml [tool.poetry] name = "Data Preprocessors" version = "0.0.0" # placeholder - CI overwrites from the tag ``` ```yaml - name: Derive version from tag run: | VERSION="${GITHUB_REF_NAME#v}" poetry version "$VERSION" echo "VERSION=$VERSION" >> "$GITHUB_ENV" ``` And in the package itself, read what was installed rather than hard-coding a second copy: ```python from importlib.metadata import version, PackageNotFoundError try: __version__ = version("Data Preprocessors") except PackageNotFoundError: # running from a source checkout __version__ = "0.0.0+local" ``` Now `pip show`, `__version__` and the git tag cannot disagree, because there is only one number and two views of it. ## The changelog is a build artefact Hand-written changelogs are accurate for about three releases. Then they lag, then they are wrong, then nobody reads them - which is worse than not having one, because people trusted it. Conventional commit messages make the changelog derivable: ``` feat(tokenizer): add Bangla sentence splitter fix(cleaner): preserve unicode combining marks feat!: drop Python 3.8 support ``` CI groups commits since the previous tag into Features / Fixes / Breaking Changes and attaches the result to the GitHub release. The `!` marker flags a breaking change, which is also the signal that the tag should be a major bump - and CI can *check* that, failing the release if a breaking commit is going out under a patch tag. That check is the part that matters. Semantic versioning that relies on the releaser's judgement at 11pm is semantic versioning in name only. ## Reproducible builds, or you are just hoping `poetry.lock` pins the fully resolved dependency graph, not just the direct constraints. CI installs from the lockfile, which means the wheel built in the pipeline is the wheel built on my laptop. Without a lockfile, a transitive dependency releasing a broken patch version turns your release into a mystery: same code, same command, different result. With one, the resolution is a reviewed, committed artefact and dependency updates are a pull request you can read. ## The pipeline ```yaml on: push: tags: ['v*'] jobs: test: strategy: matrix: python-version: ['3.9', '3.10', '3.11', '3.12'] steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - run: pipx install poetry && poetry install --sync - run: poetry run pytest -q - run: poetry run ruff check . release: needs: test permissions: id-token: write # trusted publishing - no long-lived PyPI token contents: write steps: - uses: actions/checkout@v4 with: { fetch-depth: 0 } # full history - the changelog needs it - run: poetry version "${GITHUB_REF_NAME#v}" - run: poetry build - uses: pypa/gh-action-pypi-publish@release/v1 - run: gh release create "$GITHUB_REF_NAME" --generate-notes dist/* ``` Four details worth stealing: **`needs: test`.** The release job cannot start until the full matrix passes. A broken release is not possible unless the tests are wrong. **`fetch-depth: 0`.** Actions checks out a shallow clone by default, and a shallow clone has no history to build a changelog from. This one costs an afternoon exactly once. **Trusted publishing.** OIDC instead of a long-lived PyPI token in repository secrets. Nothing to rotate, nothing to leak. **The same artefacts everywhere.** Build once; publish that build to PyPI, attach it to the GitHub release, push the Docker image. Never rebuild per target - that is how "it works from PyPI but not from the image" happens. ## Why this belongs on a backend engineer's résumé A library forces the discipline earlier than a service does. Once someone else can `pip install` your code, a breaking change stops being a refactor and becomes an event with consequences for people you will never meet. But the machinery is identical to what a deployment pipeline needs: one source of truth for the version, artefacts built once and promoted, changes derivable from history, and a release procedure with no step a tired human can skip. I now build the same thing for [multi-tenant SaaS deployments](https://musfiqdehan.com/blog/holding-10k-dau-before-you-scale-up). A small open-source package was the cheapest possible place to learn it. ---