Local AI Image Recognition
A photo server my family uploads to from their phones. Ask for a photo in plain English and it answers, without anything leaving hardware I own.
Content search over your own photos usually means giving them to someone else.
Cloud photo services are genuinely good at finding a picture by what's in it. The price is that your entire library lives on someone else's hardware behind a subscription, and the model doing the looking isn't one you run. For a family album full of kids, birthdays, holidays, and thousands of photos of the same dozen faces, that's a lot to hand over for a search box.
The build started as a question about whether the good version of that feature is reproducible from open-source parts on a machine in my own house. Not a toy: something the people I live with would actually use, from their phones, without knowing or caring what's running underneath.
What it has to get right.
No cloud SDKs, no external embedding service, no telemetry, ever. The language model parses text into JSON and nothing else. It never sees a photo, never writes to disk, and has no route to the internet.
A household is the unit, not a person. Everyone uploads into one pool from their own phone, and everyone can search all of it, including photos somebody else took.
Local hosting shrinks the attack surface; it doesn't make anything unhackable. Network isolation, container hardening, originals mounted read-only, and a scoped read-only key mean a breach of the agent still doesn't reach the photos.
The library is mine. I uploaded these photos and named every person in them by hand. What makes that tractable is that Immich clusters similar faces before you name anything, so labelling a person is one action against a cluster of hundreds of photos rather than a tag applied per photo. Faces are pixelated throughout this page because the library is real people; the names are not.
Let Immich do the seeing. Let a small local model do the reading.
Immich already solves the hard vision problems. It clusters faces, computes CLIP embeddings for semantic search, reads EXIF for dates and places, and exposes all of it through a REST API. Rebuilding any of that would have been the wrong project.
So the piece I wrote is deliberately thin: a FastAPI agent whose only job is turning an English question into structured, read-only Immich API calls. A 7B model running under Ollama parses “photos of ben at the beach last summer” into people, a date phrase, and a semantic query, and the agent resolves those into the same filters the Immich UI exposes. The model is a parser, not an oracle, which is what keeps it cheap enough to run on CPU and safe enough to hand a read-only key.
Day to day nobody touches a terminal. Everyone installs the Immich app, logs into the shared account, turns on backup, and their camera roll flows into the pool. They search from the app, or in plain English from the browser page.
Three parts, none of which phone home.
Immich, on my GPU
Phones back up to the server over the LAN. Immich clusters faces, computes CLIP embeddings, and reads EXIF, with ML accelerated on an RTX 5070 Ti.
Ollama, on CPU
A 7B model turns the sentence into a JSON intent: people, date phrase, semantic query, location, confidence. Light enough for CPU because it only ever handles text.
A read-only agent
FastAPI resolves names to person IDs and phrases to date ranges, then calls Immich with a key scoped to four read permissions and returns a grid.
From a sentence to a set of photos.
The first two steps happen once, when photos arrive. The last three run on every question.
Photos arrive from phones and get clustered
Each person enables backup in the Immich app against the shared account, so uploading is the camera roll syncing rather than a task anyone performs. As assets land, Immich runs face detection and groups the crops it believes are the same person, builds CLIP embeddings for the image content, and extracts capture date and GPS from EXIF.
Name each cluster once
This is the only manual step, and it's the one that makes everything after it work. A cluster arrives unnamed with hundreds of photos already in it; typing a name attaches that identity to all of them at once, and to anything matching that arrives later. It is the difference between labelling a library and labelling a dozen faces.
Parse the question locally
The agent sends the user's sentence, and only the sentence, to a 7B model running under Ollama, constrained to return an intent object: which people are mentioned, any date phrase, the leftover semantic description, a location, and a confidence. Because the model's job is narrow, a small quantized model on CPU is enough, and being small is what lets it run on a machine that is also serving photos.
Resolve the intent into Immich's own filters
Names are matched against the named clusters to get person IDs, the date phrase becomes a concrete range, and the remainder becomes the CLIP query. Each maps onto a filter Immich already supports, so the agent is composing an existing search rather than implementing one.
Query read-only and return the grid
The call goes out under an API key holding four permissions (read and view assets, read people, read albums) and the response comes back as thumbnails and deep links into Immich. The agent has no write path to revoke, because it was never granted one.
Choices that shaped the build.
What I chose, and what it cost.
The shared account over per-person accounts
CostAnyone signed into the vault can delete any photo, including one they didn't upload. Thirty-day trash and the two-drive backup tooling cover the accident case, but it's a trust assumption rather than an enforced boundary. Fine for a family, wrong for anything wider.
A local 7B model instead of an API
CostThe first query after a cold start takes about a minute while 5.5 GB of weights load into RAM, then settles to a few seconds and stays warm for thirty minutes. A hosted model would have no warm-up, but a hosted model is the thing this project exists to avoid.
The obvious multi-user model is one account per person: everybody uploads to their own library, a shared album makes it all visible, and delete stays scoped to whoever uploaded it. I built toward that and then tested it instead of assuming it, which is how I found that Immich's search is owner-scoped. A member of a shared album searching with an album filter got zero of the other members' photos, because the filter only narrows your own assets within that album. Testing the admin account closed the door properly: an admin search over its own 898 photos excluded a second user's asset, and the admin could not fetch that asset or its thumbnail at all. Admin means managing users and settings, not seeing everyone's content. So whole-pool search and per-uploader delete cannot both come out of Immich as it ships: one needs a single account, the other needs separate ones. I took whole-pool search, because a search tool that silently returns a subset of the library is worse than no search tool, and documented the per-account design as the thing to build when delete boundaries matter more than convenience.
Where I'd take it.
Build the per-account model: a read-only key per user, the agent searching each library and merging results, so per-uploader delete becomes enforceable without giving up whole-pool search.
Move onto always-on hardware. The stack currently lives on a desktop that has to be awake for anyone's phone to reach it, which is the single biggest gap between this and an appliance.
Actually test a restore. The two-drive backup scripts are written and scheduled, but a backup nobody has restored from is a hypothesis, not a backup.