Skip to main content
PortfolioBlogAboutContact
Hero image

Qdrant File Mode Held Up Until My Agent Took the Lock

Published: October 20, 2026

On the evening of 22 September 2026 I registered a new search tool with my agent gateway and asked a one-shot session to call it. About 49 seconds after the registration, the gateway hot-reloaded its MCP servers, spawned its own copy of my retrieval server and opened the Qdrant store. My test session tried to connect a minute later and failed with a CancelledError. lsof corpus/qdrant/.lock named the process holding the file: the gateway's child, which would keep it for as long as the gateway ran.

5 hours later the desktop app hit the same wall. Every time its backend started, it picked up the tool from config, tried to open the store and gave up with "failed initial connection after 3 attempts, parking until a reconnect is requested." The index and the data were both intact. A second process wanted the store, and Qdrant's local file mode allows exactly one owner.

If you are building RAG behind an agent and started with QdrantClient(path=...), this post is the decision I made the next day: why file mode was the right start, the moment it stopped fitting, the 11-second migration to a server and the recall number that dropped even though not a single byte of data changed.

The Project and Why I Started in File Mode on Purpose

The system is a persona RAG in Polish: about 1,045 YouTube transcripts cut into 12,886 chunks, with e5-large 1024-dim dense vectors and a Polish-stemmed BM25 sparse arm. Search is hybrid, fused with DBSF and grouped by video_id, so a caller gets one hit per video and reads the whole transcript file it points at. It is exposed as one read-only MCP tool that an agent (Hermes), a web chat and a video call all use.

I started in file mode deliberately, and I wrote the reason into the build plan:

  • $0 and no accounts. No Docker, no cloud cluster, no network at query time.
  • The vendor positions it this way. The qdrant-client README describes local mode as useful for "development, prototyping and testing" and tells you to switch to server mode when you need to scale.
  • One parameter separates the two. QdrantClient(path='corpus/qdrant') and QdrantClient(url=..., api_key=...) take the same calls afterwards.

For the first 2 weeks that was correct. I was the only process touching the store.

What Works in File Mode and the 3 Gaps I Found

Before building anything I checked that local mode supported every query I needed, and it did. Hybrid prefetch with dense plus sparse, RRF and DBSF fusion, query_points_groups(group_by='video_id') and DatetimeRange date filters all ran on day one.

The same Phase 0 run surfaced 3 gaps, all from real output:

  1. Payload indexes are ignored. Creating one prints "Payload indexes have no effect in the local Qdrant. Please use server Qdrant if you need payload indexes." Filters still return correct results, only without acceleration.
  2. There are no native snapshots. create_snapshot raises NotImplementedError. I wrote my own backup instead: a gzipped JSONL export of every point, vectors and payload included. That export later turned out to be the migration path.
  3. One process owns the folder. In qdrant-client 1.19.0 the local store takes an exclusive, non-blocking lock on the folder. A second opener gets "Storage folder ... is already accessed by another instance of Qdrant client. If you require concurrent access, use Qdrant server instead." This applies to readers too, so a read-only search tool counts as an owner.

There is a 4th property I only confirmed later in the client source: local mode "performs exact (brute-force) search", which is why HNSW settings have no effect there. At 12,000 points that is fine for one user. It matters once you measure latency, which comes up again below.

The Day the Lock Became the Architecture

An MCP server is supposed to hold its connection for the life of the session. That is good design for a tool. My gateway kept its retrieval child alive for as long as the gateway itself ran, which meant 24 hours a day. In file mode, that turned the lock into the shape of the whole system:

  • Seeding new transcripts needed a "gateway-off window". I wrote a window.sh that stops the gateway, waits until the lock is free, runs the job and starts the gateway again.
  • The planned daily sync cron would have had to schedule itself around that window.
  • The desktop app could not use retrieval at all while the gateway was up.

One rule stayed fixed through all of it: never delete or break the .lock file. The error is telling you the truth about who owns the store. Removing the lock to get past it would let 2 processes write to the same files.

What I did instead was make the failure readable. The shared Qdrant client turns the raw RuntimeError into a named error with a message a human can act on, and the health check exits with code 3 LOCKED:

LOCK_MESSAGE = 'already accessed by another instance'


def connect(url: str | None, api_key: str | None, path: Path | str):
  try:
    conn = client(url=url, api_key=api_key, path=path)
    conn.get_collections()
    return conn
  except RuntimeError as exc:
    if LOCK_MESSAGE in str(exc):
      raise LockedStoreError(
          f'Qdrant store at {path} is locked by another process '
          '(a seeding run or another server). Stop that process and retry. '
          'Do not delete or break the .lock file.'
      ) from exc
    raise

Readable errors made the problem easy to diagnose, but every second process was still locked out. The signal to move was clear: a second process needed the store, and that arrived on the same day the agent went live.

The Move to a Server Took 11 Seconds of Import

The server came up with the standard image (Qdrant 1.19.1) and its own storage folder, so the file store stayed untouched as the rollback target:

docker run -d --name qdrant-ntc -p 6333:6333 -p 6334:6334 \
  -v "$PWD/corpus/qdrant-server:/qdrant/storage" qdrant/qdrant

Then I needed the data in it. Re-embedding was the slow option: e5-large on this Mac's CPU runs at 0.5 to 4.6 chunks per second depending on load, so a rebuild would have taken about an hour. The JSONL export from the snapshot workaround already held every vector, so I added an --import-snapshot command to the seeding app and streamed it in:

  • 12,539 points written in about 11 seconds, about 1,125 points per second;
  • zero re-embedding, with the point count checked against the export afterwards.

The switch between modes is one factory function. When QDRANT_URL is set it returns a server client, and otherwise it opens the file store. The MCP registration carries the env var, so every process that spawns the tool (the gateway, the desktop backend, the CLI, the cron) ends up in the same mode without anyone editing code.

The proof came the same night: the Telegram bot answered an archive question through the tool at 23:49 and the desktop app did the same at 23:57, while the gateway was running. Then I ran the rollback drill: unset QDRANT_URL, and file mode worked against the untouched store. Keeping that path alive cost nothing.

Same Bytes, a Different Recall Number

My quality gate is a set of 49 probes built from the corpus, scored by recall@8. In file mode it read 0.939 overall and 0.960 on narrow queries. In server mode the same set read 0.918 and 0.920.

My first assumption was that the import had lost or changed something, so I compared the 2 stores and found the same points, the same dense and sparse vectors, the same prefetch lists and the same fused scores. Exactly 1 probe (hot-pot) had changed outcome.

The cause was tie-breaking. Local mode is Python code inside qdrant-client and the server is Rust, and when several chunks have equal fused scores they order them differently. That reshuffles the deeper fused ranks, which changes which videos survive grouping. Forcing exact search on the server did not change it, and there is no setting for it.

Both readings cleared the gate (overall at least 0.80, narrow at least 0.70), so I accepted the server number and relabelled every figure in the docs with its mode. If you prototype in file mode and gate on recall, re-baseline after the move and expect small movement on ties. Treating that drop as a regression would have sent me hunting for a bug that did not exist.

Latency: The Stale Number I Almost Kept Quoting

The docs carried a file-mode latency figure from an earlier run: total p50 497 ms, with grouped search at 474 ms of it. It kept getting quoted as current long after the store had moved.

When I measured server mode properly, a warm run over 12 queries gave:

  • total p50 49.6 ms;
  • embedding p50 23.2 ms;
  • grouped search p50 9.0 ms;
  • cache hit 0.18 ms from the sqlite response cache.

The 2 runs come from different days and slightly different corpus sizes, so I don't present it as a clean 10x benchmark. The direction matches what the client source says, though: an exact scan in Python against HNSW in a Rust server. Even on a heavily loaded day (OBS, a camera and several agents running), grouped search on the server took 58 ms and the embedding step was about 59 percent of the total. The store stopped being the part worth optimizing.

What went wrong there was a figure without a mode label. Every eval and latency number in that project now states which mode produced it.

What Server Mode Costs You

A server fixes concurrency and adds new ways for things to break. These are the ones I hit:

  • The store can be down. The health check now exits 1 UNREACHABLE when Docker or the container is stopped. The client is written to fail loudly in that case. Falling back to the file store would serve whatever it held the last time someone synced it, which is worse than an error.
  • Disk. The server store is about 290 MB, but one native snapshot is about 470 MB, so keeping 3 costs about 1.4 GB. On a Mac with around 18 GB free, that is a number to watch.
  • Snapshots live in the container. Native snapshots land at /qdrant/snapshots inside the container, outside the storage folder I mounted. I found that out when I went looking for them on the host.
  • Payload indexes start to matter, so I created the date and video_id indexes as part of the move.

The move also opened the next step. The same factory now points a small VPS at a Qdrant Cloud free-tier cluster, refreshed nightly from the local server. That setup deserves its own post.

My Rule: When to Stay in File Mode and When to Move

Stay in file mode when exactly one process owns the store for its whole life:

  • a notebook or a CLI you run by hand;
  • unit tests and CI, where a fresh folder per run is a feature;
  • a single-user prototype where you are still deciding what to index.

I still use it for tests and as the rollback target.

Move to a server as soon as any of these shows up:

  • a second reader or writer: an agent gateway, MCP children, a cron, a desktop app;
  • a need for native snapshots;
  • filtered search on a growing collection, where payload indexes would help;
  • more than 1 machine.

If you are starting today, this is the checklist I would give you:

  1. Put the client behind one factory on day one, selected by an env var, so the move is configuration.
  2. Back up with a portable export you can import anywhere. Mine paid for itself the first time I used it.
  3. Label every eval and latency number with its mode.
  4. Never touch the lock file. Make the error readable and fix the ownership.

For an agent in particular, plan for the server from the start. An MCP tool that the agent runtime keeps alive is a long-running owner, and with file mode that is enough to lock every other process out. The rules for this project, including "never break the lock", live in an ops skill the agent reads before it touches the store. I wrote about that approach in Stop Writing Prompts. Start Writing Skills., and the way agents keep connections open is one of the patterns from Your API Isn't Ready for Agent Customers.

Where to Get the Files Behind This Build

File mode was the right call for the first 2 weeks of this build, and the server was the right call from the evening the agent went live. What made the switch cheap was the factory and the export, both written before I needed them.

If you are wiring RAG into agents and want to see the real specs, runbooks and skills behind builds like this one, join the Agentic Architect AI Lab on Skool. It is free to join, and I post the working files there along with the mistakes that shaped them.

Sources

Building with AI beyond this article?

I run The Agentic Architect Lab, live builds, agent workflows, and a playbook for technical founders shipping solo. No toy demos.

Join the Lab
Next Post →