Quartermaster

v1.7.2 · Windows, Linux, macOS, Docker

Run any model without tuning a single flag.

Quartermaster runs AI on your own computer: chat, images and voice, for you or for everyone at home. Point it at your models folder and it works out what fits in your VRAM, launches each model with computed flags, and hot-swaps between them behind one OpenAI- and Anthropic-compatible API.

Download for Windows Read the guide

v1.7.2 · all releases · Docker and other platforms

Load planqwen3.8-27b-ud3-iq4_xs
22.3/ 24.0 GB24 GB card
  • Weights15.7
  • KV cache3.5
  • Compute1.4
  • Draft + checkpoints1.2
  • Headroom0.5
  • System1.2
Context
100k
GPU layers
65 / 65
Offload
none
Generated
llama-server -m Qwen3.8-27B-UD3-IQ4_XS.gguf \
  -c 102400 -ngl 99 --port ${PORT}

NVIDIA (CUDA), AMD (ROCm or Vulkan), Intel Arc (Vulkan) and Apple Silicon (Metal). No GPU works too, just slower.

Private AI at home

I want my own ChatGPT

Chat, web search, pictures and voice, running on your computer. Your conversations stay with you, and everyone at home gets their own login.

See the chat app ↓

Models and tooling

I run models and build on them

One OpenAI and Anthropic endpoint for every model you own, with context, offload and KV cache sized to your VRAM. Works with Claude Code, Open WebUI and the SDKs.

Compare with Ollama and LM Studio ↓

The chat app

Your own ChatGPT, on your own computer

Quartermaster comes with a chat app that works like the ones you already know: type a question, get an answer. The difference is where it runs. The AI is on your computer, your conversations are saved there, and nobody is reading them, training on them or charging you for them by the month.

Chat: An ordinary conversation. The model's thinking is tucked away, the answer is not. Chat: An ordinary conversation. The model's thinking is tucked away, the answer is not.

Chat An ordinary conversation. The model's thinking is tucked away, the answer is not.

What you need: a PC with a graphics card that has 8 GB of memory or more, or a Mac with Apple Silicon. The built-in model browser shows which models fit your machine before you download one.

The Quartermaster dashboard The Quartermaster dashboard

Compared

Where it differs from Ollama and LM Studio

Both are good at getting one model running. Quartermaster is built for the box that runs several, of different kinds, on a fixed amount of VRAM. Checked against their docs as of September 2026; tell us if something here has gone stale.

QuartermasterOllamaLM Studio
Context length fitted to your VRAM, per modelComputed from the GGUF and what is freeOne default by total VRAM: 4k under 24 GBFixed default, set by hand
KV cache and MoE experts placed for youKV type is an env var, no expert offloadBoth are manual toggles
Several models, swapped in by the requestEvicts only what no longer fitsOne on-demand model at a time
A long chat survives its model being swapped outKV cache saved to disk and restoredNot in their docsNot in their docs
Image generation, speech and transcriptionSame catalog, same APIImage generation was removed in 0.32.6
Uses your GGUF files where they already areImported and copied into its own storeExpects its own folder layout
Chat UI for the rest of the house, with loginsWeb search, images and voice includedAPI only over the networkAPI only over the network
Run your own llama.cpp buildOr follow any GitHub repo's releasesEngine is bundledChoose among its runtime versions
OpenAI and Anthropic APIs
Speculative decodingMTP picked automaticallyApple Silicon only
Curated model library, one click to pullHugging Face search, no curated list
MLX engine on Apple Siliconllama.cpp with Metal
Open sourceMITMITApp is closed, SDKs are MIT

Your tools

Point what you already use at it

One port answers both the OpenAI and the Anthropic API, so anything with a base URL setting works unchanged: Claude Code, Open WebUI, Continue, Cline, SillyTavern, the OpenAI SDKs. The model field picks the model, and if it isn't loaded, that request loads it.

Claude Code on your own GPU

Claude Code talks to the Anthropic API, and so does Quartermaster. Set the base URL and name a local model for both of its slots, so its background calls do not ask for a Claude model you do not have. The token is only checked if you have created API keys.

shell 
export ANTHROPIC_BASE_URL=http://127.0.0.1:1250export ANTHROPIC_AUTH_TOKEN=qm-...  # any value if you have no keysexport ANTHROPIC_MODEL=qwen3.8-27b-ud3-iq4_xsexport ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3.8-27b-ud3-iq4_xsclaude

Features

It works the machine out for you

One endpoint in front of every model you own: llama-server, stable-diffusion.cpp, speech, transcription, rerank, embeddings, upscaling. The sizing is decided for you instead of by you, and the chat app sits on top of all of it.

01Automatic configuration

Config that writes itself, then hands you the pen

Point it at a folder. Every GGUF is identified from its own header, and context length, GPU offload, CPU-MoE split and KV-cache sizing are computed per model and per architecture. No hand-written config block per model, and no second block when a new quant lands.

Per-model parameters: Context, KV cache, offload and speculative decoding, with the computed default one click away. Per-model parameters: Context, KV cache, offload and speculative decoding, with the computed default one click away.

Per-model parameters Context, KV cache, offload and speculative decoding, with the computed default one click away.

02Load planning

It knows what will fit before it loads it

Free VRAM is sampled at startup and every model is sized against what is actually left, not against the number on the box. The gauge breaks a load down the way the card sees it, so an estimate that is about to go wrong is visible before you press load rather than after the driver kills the process.

VRAM breakdown: One bar per segment: weights, KV cache, compute buffer, and what the rest of the system already holds. VRAM breakdown: One bar per segment: weights, KV cache, compute buffer, and what the rest of the system already holds.

VRAM breakdown One bar per segment: weights, KV cache, compute buffer, and what the rest of the system already holds.

03Getting and keeping models

Find a model, download it, run it

Search Hugging Face from inside the app, compare quants against the VRAM you actually have, and download into your models folder with the transfer resumable if it breaks. What lands is picked up and configured without a restart.

Model hub: Search, pick a quant, download into your models folder. Model hub: Search, pick a quant, download into your models folder.

Model hub Search, pick a quant, download into your models folder.

04Bring your own backend

Any inference server you have, and any repo you follow

The backends it ships with install themselves from their upstream GitHub releases, but nothing is hard-wired to them. Point a row at a binary you built yourself and it joins the same registry: picked per model class, launched with the flags the config generates, no different from a managed install. Or hand it a GitHub repo and it follows that project's releases for you.

Settings, Backends: Managed installs on top, the registry they write into underneath: every row is a binary Quartermaster can spawn, whether it installed it or you did. Settings, Backends: Managed installs on top, the registry they write into underneath: every row is a binary Quartermaster can spawn, whether it installed it or you did.

Settings, Backends Managed installs on top, the registry they write into underneath: every row is a binary Quartermaster can spawn, whether it installed it or you did.

Also in the box

The parts you notice on day thirty

Less visible than a gauge, and the reason it keeps working once it is part of your setup.

On-demand model swapping

One endpoint, every model. A request naming a model that isn't loaded swaps it in, evicting whatever no longer fits, and holds the group together when several have to coexist.

KV-cache that survives eviction

Snapshots a slot's KV-cache to disk before the model is evicted and restores it when the conversation comes back, so a long chat isn't re-prefilled because a throwaway request borrowed the GPU.

Multi-port catalogs

Bind several listeners on one shared scheduler, each with its own /v1/models view. Loading on one port can evict on another: one process, one GPU accounting.

Observe what it is doing

Activity, streaming logs, per-model performance and context use on one page, so a slow request is something you can look at rather than guess about.

Safe to put on your LAN

API keys can be scoped to individual models. Bind the API to your tailnet and the dashboard and config endpoints answer to localhost only unless you widen them yourself.

Drivable from the outside

Prometheus metrics, a log stream you can pipe, ops endpoints to load and unload a model on demand, and a config file that hot-reloads when you edit it. No plugin system to learn: the surface is HTTP and YAML.

How it started

A config file I got tired of writing

Every model meant another hand-written block of config: how much context, how many layers on the GPU, how big the KV cache, whether the experts go on the CPU. Then a new quant lands and you do it again. Sometimes I wanted a configuration with high context, other times I wanted a very lean model load to allow something else on my GPU at the same time. The solution I came up with is the variants system.

It started as a fork of llama-swap, which had the swapping right, and grew in the obvious direction from there: read the GGUF header, measure the VRAM that is actually free, and compute the numbers instead of typing them. Once that worked the rest followed. Image and audio backends in the same catalog, several ports sharing one scheduler, a KV-cache that survives being evicted, and a UI that shows what the box is doing rather than a log you have to tail.

It is its own thing now.

Questions

Before you download

Is it free?

Yes. It is open source under the MIT license, with no account, no subscription and no paid tier. The models are free to download too.

Does it send my data anywhere?

Your chats, pictures and recordings stay on your computer, and there is no telemetry. It goes online to check for a newer version of itself and to download the models and components you pick. In a chat, the web tools make outside requests when the model uses them: a web search, a page it opens, a news feed, the weather or an exchange rate. Only the lookup goes out, never the conversation.

Will it be as good as ChatGPT?

Not quite, and it is better to know that up front. The models that fit on a home graphics card are smaller than the ones behind ChatGPT or Claude. On a 24 GB card they handle everyday questions, writing, summaries and coding help well; on an 8 GB card expect something closer to a capable assistant than an expert. What you get in return is privacy and no monthly bill.

I already use Ollama or LM Studio. Do I start over?

Not with LM Studio: its models folder is plain GGUF files, so point Quartermaster at it and they show up. Ollama keeps its models renamed inside its own store, so those need downloading again, which the built-in browser does in a few clicks. Both can stay installed; just don't run them at the same time, since they would be sharing one graphics card.

What happens if a model is too big for my graphics card?

It still runs: the part that does not fit goes to your regular memory and processor, which is slower. The model browser marks what fits your card before you download, so this is usually a choice rather than a surprise.

Can other people in my house use it?

Yes. The chat app has its own address on your home network, and each person signs up with their own login and gets their own history. It runs on the one computer with the graphics card; everyone else just opens it in a browser.

Install

One binary, every platform

Windows, Linux, macOS and Docker, from the same single binary. The installer and the Docker image bring the inference backends with them; everywhere else you install them from Settings on first run, or point at ones you already have.

Windows installer

A per-user install, no admin rights needed. The first-run wizard fetches the inference backends, asks for your models folder, and generates a config before the window opens.

Download v1.7.2 33 MB · per-user · no admin rights

Prefer to build it yourself? Building from source takes Go 1.26+ and Node 24 for the UI, one binary out the other end.

Documentation

The manual ships inside the app

These are the same help articles behind the Help button in the sidebar, and the assistant in the playground searches them as a tool, so the quickest way to learn Quartermaster is to open the chat and ask. Why did my model get evicted, what does this flag do, how do I cap the VRAM budget: it reads the manual and answers with your setup in front of it.

Getting started

Models & config

Playground

Monitoring & VRAM

API & access

Troubleshooting