Quartermaster

Run any model without tuning a single flag

Quartermaster is an all-in-one local inference platform. Point it at your models folder: it works out what fits in your VRAM, launches each model with computed flags, and hot-swaps between them on demand behind one OpenAI- and Anthropic-compatible API.

Download for Windows Other install options Read the guide

v1.0.2 · all releases · Docker and source below

Dashboard
The Quartermaster dashboard

It works the machine out for you

One endpoint in front of every model you own, the sizing decisions made for you instead of by you, and a front end for all of it.

Automatic configuration

Config that writes itself, then hands you the pen

Point it at a folder. Every GGUF is identified from its own header, and context length, GPU offload, CPU-MoE split and KV-cache sizing are computed per model and per architecture. No hand-written config block per model, and no second block when a new quant lands.

  • Every computed number is an editable field, not a wall you have to work around.
  • Save a tuned set as a named variant and run it alongside the default: a long-context one for documents, a lean one for quick calls.
  • Changed your mind? Reset one field, or the whole model, back to what it computed.
Per-model parameters
Per-model parameters: Context, KV cache, offload and speculative decoding, with the computed default one click away.

Per-model parametersContext, KV cache, offload and speculative decoding, with the computed default one click away.

Load planning

It knows what will fit before it loads it

Free VRAM is sampled at startup and every model is sized against what is actually left, not against the number on the box. The gauge breaks a load down the way the card sees it, so an estimate that is about to go wrong is visible before you press load rather than after the driver kills the process.

  • Weights, KV cache and compute buffer are accounted separately, per architecture.
  • The compute buffer is the one large-vocab models silently spill on. It is priced in.
  • System usage is part of the budget, so the number is what is free for you, not what is free in theory.
VRAM breakdown
VRAM breakdown: One bar per segment: weights, KV cache, compute buffer, and what the rest of the system already holds.

VRAM breakdownOne bar per segment: weights, KV cache, compute buffer, and what the rest of the system already holds.

Text, image and audio

A playground, not just a proxy

It orchestrates llama-server, stable-diffusion.cpp, TTS and transcription servers, rerank and embedding models, upscaling and segmentation, all behind one OpenAI-compatible surface. Then it gives you a front end for them, on its own port with per-user login and server-side history, so a model is not just reachable the moment it is discovered, it is useful, with nothing else installed in front of it.

  • An everyday helper: a shopping assistant mode to help you browse the internet and compare prices, then lists you the options, tell you what the weather does tomorrow, rewrite a piece of text according to your instructions, and inspect the diff, or even tell you to help you with your model config
  • Web search and tool calling are wired in, so the answer isn't limited to what the weights happen to remember, and the reasoning stream can be collapsed or hidden.
  • It can explain Quartermaster itself. The help articles are one of its tools, so "why did my model get evicted?" is a question you can ask in the chat.
  • Generate and edit images against the same catalog, LoRAs and reference images included.
  • Speak and transcribe without leaving the tab.
Chat
Chat: An ordinary conversation, with the thinking stream kept out of the answer.

ChatAn ordinary conversation, with the thinking stream kept out of the answer.

Getting and keeping models

Find a model, download it, run it

Search Hugging Face from inside the app, compare quants against the VRAM you actually have, and download into your models folder with the transfer resumable if it breaks. What lands is picked up and configured without a restart.

  • Quants are listed with the fit already worked out, so the pick is not a guess.
  • Variants of one GGUF group together instead of flooding the list.
  • Text and image models share the catalog and the same management surface.
Model hub
Model hub: Search, pick a quant, download into your models folder.

Model hubSearch, pick a quant, download into your models folder.

Bring your own backend

Any inference server you have, and any repo you follow

The backends it ships with install themselves from their upstream GitHub releases, but nothing is hard-wired to them. Point a row at a binary you built yourself and it joins the same registry: picked per model class, launched with the flags the config generates, no different from a managed install. Or hand it a GitHub repo and it follows that project's releases for you.

  • Track any repo: pick one real asset out of one real release and the match pattern is derived from it, build numbers and dates become wildcards, so next week's build of the same flavour still resolves. There is no regex to write.
  • Builds are installed side by side and versioned. Switch which one a backend runs, or roll back to the last one that worked, without reinstalling anything.
  • Locally compiled binaries coexist with managed ones, and an install never quietly steals the default from a backend you set up yourself.
Settings, Backends
Settings, Backends: Managed installs on top, the registry they write into underneath: every row is a binary Quartermaster can spawn, whether it installed it or you did.

Settings, BackendsManaged installs on top, the registry they write into underneath: every row is a binary Quartermaster can spawn, whether it installed it or you did.

And much more

A plethora of features to make it a breeze to run your models the way you want to

On-demand model swapping

One endpoint, every model. A request naming a model that isn't loaded swaps it in, evicting whatever no longer fits, and holds the group together when several have to coexist.

KV-cache that survives eviction

Snapshots a slot's KV-cache to disk before the model is evicted and restores it when the conversation comes back, so a long chat isn't re-prefilled because a throwaway request borrowed the GPU.

Multi-port catalogs

Bind several listeners on one shared scheduler, each with its own /v1/models view. Loading on one port can evict on another: one process, one GPU accounting.

Observe what it is doing

Activity, streaming logs, per-model performance and context use on one page, so a slow request is something you can look at rather than guess about.

Safe to put on your LAN

API keys can be scoped to individual models. Bind the API to your tailnet and the dashboard and config endpoints answer to localhost only unless you widen them yourself.

Drivable from the outside

Prometheus metrics, a log stream you can pipe, ops endpoints to load and unload a model on demand, and a config file that hot-reloads when you edit it. No plugin system to learn: the surface is HTTP and YAML.

How it started

A config file I got tired of writing

Every model meant another hand-written block of config: how much context, how many layers on the GPU, how big the KV cache, whether the experts go on the CPU. Then a new quant lands and you do it again. Sometimes I wanted a configuration with high context, other times I wanted a very lean model load to allow something else on my GPU at the same time. The solution I came up with is the variants system.

It started as a fork of llama-swap, which had the swapping right, and grew in the obvious direction from there: read the GGUF header, measure the VRAM that is actually free, and compute the numbers instead of typing them. Once that worked the rest followed. Image and audio backends in the same catalog, several ports sharing one scheduler, a KV-cache that survives being evicted, and a UI that shows what the box is doing rather than a log you have to tail.

It is its own thing now.

Installations

Windows, Linux, macOS and Docker, from the same single binary. The installer and the Docker image bring the inference backends with them; everywhere else you install them from Settings on first run, or point at ones you already have.

Windows installer

A per-user install, no admin rights needed. The first-run wizard fetches the inference backends, asks for your models folder, and generates a config before the window opens.

Download v1.0.2 · 32 MB

Linux and macOS

The same wizard, in your browser: it fetches a verified binary, the backends for your GPU, and a config built from your models folder. It prints the address it serves on and holds the terminal until you finish, so open that address if no browser appears. On macOS clear the quarantine flag first. Or take the bare static binary instead, amd64 and arm64 for Linux and Apple silicon for macOS, and install the backends later from Settings.

linux and macos 
chmod +x quartermaster-setup-darwin-arm64-*xattr -d com.apple.quarantine quartermaster-setup-darwin-arm64-*./quartermaster-setup-darwin-arm64-*

Docker

One image, for linux/amd64 and linux/arm64, and it serves the moment it starts: llama-server and sd-server ship inside it, the same upstream Vulkan builds a desktop install downloads. Point it at your models folder and open the dashboard.

docker 
docker run -p 127.0.0.1:1250:8080 -v quartermaster-data:/data \  -v ./models:/data/models ghcr.io/quartermaster-labs/quartermaster:latest

Prefer to build it yourself? Building from source takes Go 1.26+ and Node 24 for the UI, one binary out the other end.

Documentation

The manual ships inside the app

These are the same help articles behind the Help button in the sidebar, and the assistant in the playground searches them as a tool, so the quickest way to learn Quartermaster is to open the chat and ask. Why did my model get evicted, what does this flag do, how do I cap the VRAM budget: it reads the manual and answers with your setup in front of it.