Sawit — Instagram reels, turned into notes you can search

The problem was never saving. It’s finding.

Instagram’s Saved folder is a wall of thumbnails. No search, no text, no way to ask “what was that budget split?” six months later.

So people do the next best thing — they share reels to themselves in DMs — and end up with a second pile they also can’t query. The useful part of a reel is about forty seconds long. Sawit extracts it and makes it searchable by meaning, not just by keyword.

You ask	It finds	Shared words
how should i split my salary	Allocate monthly net income using a 55/5/10/15/15 split	0
something sweet to cook	Mango Sago Dessert Recipe	0
finding candidates to hire	Juicebox: build candidate shortlists and outreach	0

Real queries against a real library. Each note shares not one word with the question that found it.

What it produces

A finance reel with no spoken audio at all — a caption over music. Here is the note that came back.

{
  "title":     "Allocate monthly net income using a 55/5/10/15/15 split",
  "category":  "finance",
  "one_liner": "Divide net income into 55% essentials, 5% guilt-free spending,
                 10% debt or investing, 15% short-term, 15% long-term.",
  "steps": [
    "Multiply monthly net income by 0.55 for essential expenses",
    "Multiply by 0.05 for guilt-free spending money",
    "Multiply by 0.1 and apply to debt, or invest it",
    "Multiply by 0.15 for short-term goals",
    "Multiply by 0.15 and invest for long-term wealth"
  ],
  "key_facts": [
    {"label": "Essential expenses share", "value": "55%"},
    {"label": "Total allocation",        "value": "100%"}
  ],
  "caveats": ["Based on monthly net income after taxes, not gross"]
}

The transcript for that reel was empty. Every field above was read off the video frames.

The decision the whole project turned on

Reels put their numbers on screen and never say them out loud. So the frames aren’t a supplement to the transcript — they’re usually the entire source. Which makes one unglamorous number load-bearing: how many images the model accepts per request.

1 frame · llama-3.2-90b-vision

title: "Gamble King"
summary: "Gamble King"
tags: ["Gamble King"]

It read one card and repeated it into every field.

8 frames · nemotron-3-nano-omni

title: "Slang wordplay — Stu(dies / died / dying)
against Smo(king), Drin(king), Gamble(king)"
takeaways: [] category: other

It read the whole visual pun — and correctly returned nothing to take away, because it’s a joke.

Same reel. Same 14-character transcript. The only differences were how much of the screen the model could see, and whether the prompt allowed it to say there is nothing here. The default model takes 12 images; both llama-vision models take exactly one.

How it works

Eight stages. The share sheet never waits on any of them.

The stack, and why each piece

Layer	Choice	Why this one
API	FastAPI	Returns `202` in ~0s and works in a background task, so the share sheet never blocks
Fetch	yt-dlp	Falls back to the post caption when a share is an image carousel with no video stream
Render	ffmpeg / ffprobe	16 kHz mono wav, plus 8 stills sampled inside the clip — the first and last frames are a title card and “follow for more”
ASR	faster-whisper, int8	Free and local. Serialised behind a lock, because the model is the memory budget
Extraction	nemotron-3-nano-omni-30b-a3b	30B MoE, ~3B active, multimodal, free tier — and it takes 12 images per request
Embeddings	nemotron-3-embed-1b	2048-dim, same endpoint and key as extraction, so semantic search adds no second service
Storage	SQLite + FTS5 + vectors	One file. No ORM, no vector database, no Redis
Auth	hashlib.scrypt + hmac	Standard library only. Signed, expiring sessions; per-account API keys

Three things that took real work

Typed output, three ways

One Pydantic schema, enforced via Anthropic’s parser, strict json_schema with $refs inlined, or plain JSON mode — because that’s where open models’ constrained decoding falls over.

A salvage parser

Reasoning models narrate before they answer. Rather than losing the reel to a preamble, it scans brace-matched candidates and takes the first that validates.

Isolation in the constructor

SQLite has no row-level security, so a Store is bound to an account and every query filters inside. Endpoints can’t forget a filter they never write.

Eight bugs, six of which couldn’t reproduce locally

The most useful thing in the repository.

Output file does not contain any stream

Instagram serves a video-only stream to datacenter IPs where it gives a home connection both. Failed in production, passed locally, on the same URL.

Session cookie shipped without Secure

Railway terminates TLS at its proxy, so the app saw plain http inside the container and skipped the flag on the cookie holding the API key.

Out of Memory × 3

Two reels at once each built their own Whisper model — lru_cache doesn’t serialise a concurrent miss, so the cache was no protection.

12 notes for 8 reels

No URL dedup. One reel had been downloaded, transcribed and sent to the model four separate times.

Clearing the search did nothing

The form carried the active category in a hidden field, so emptying the filter you could see re-submitted the one you couldn’t.

did not return a usable note

A reasoning model put its narration before the JSON. The parser now recovers the object instead of failing the reel.

The pattern worth extracting: five of these are environment differences, not logic errors. Datacenter IP versus home IP. TLS at a proxy versus served directly. Concurrent versus sequential. None are findable by writing more unit tests — only by deploying and then driving the real thing.

Run it yourself

Python 3.12+, ffmpeg, and a free key from build.nvidia.com. About five minutes.

git clone https://github.com/prajwal2308/SAWIT.git
cd SAWIT
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
python -c "import secrets; print('SAWIT_API_KEY=' + secrets.token_urlsafe(32))"
# put that key and your NVIDIA_API_KEY in .env, set SAWIT_LLM=nvidia

uvicorn app.main:app --reload

Open localhost:8000/?k=<your key>. Everything else has a working default — the model, the embedding model, frame count and transcription are already set to values that were tested together.

Full README on GitHub Architecture

What it actually taught

Five of eight bugs were environment differences. Not logic errors. No amount of unit testing finds them; a deploy does.
The prompt was as load-bearing as the model. The same model went from emitting Gamble(king): Gamble(king) as a “key fact” to correctly returning empty fields — because the instructions changed to say an empty field is a correct answer. Capability was never the bottleneck; permission to say nothing was.
Put invariants where they can’t be forgotten. Account isolation lives in a constructor, not in a check each endpoint remembers. Adding routes stopped adding holes.
The hardest problem wasn’t technical. The pipeline took a day. Getting a person to press one button repeatedly is what didn’t work — and no architecture fixes that.