What Happened
A post on Reddit's r/LocalLLaMA community is driving attention to llama-swap, a Go-based proxy that handles dynamic model switching for local AI inference. The tool, which now has 2.6k GitHub stars and 195 forks, is positioning itself as a more flexible alternative to Ollama and LM Studio for people running multiple local models.
The core pitch: llama-swap sits between your client apps and any OpenAI or Anthropic-compatible inference server, automatically loading and unloading models based on incoming API requests. Unlike Ollama or LM Studio, which are self-contained inference platforms, llama-swap works with whatever backend you prefer - llama.cpp, vLLM, TabbyAPI, stable-diffusion.cpp, or any OpenAI-compatible server.
Key features include TTL-based automatic unloading (so idle models don't eat your VRAM), model groups for running multiple models concurrently, dynamic port assignment, a real-time web dashboard for monitoring, and configuration hot-reload. It ships as a single Go binary with zero dependencies, plus Docker images for CUDA, Vulkan, Intel, and MUSA platforms.
The Reddit post's author specifically called out the "load models on demand" feature that kept them locked into Ollama and LM Studio, noting that llama-swap supports the same capability while working with any underlying provider.
Why It Matters
If you run local models, your choice of serving infrastructure directly impacts what you can do. Ollama and LM Studio are popular because they're simple - download a model, click run. But that simplicity comes with lock-in. You use their inference engine, their quantization formats, their API layer.
llama-swap takes the opposite approach: it's a thin routing layer that lets you use the best backend for each model. Running a coding model on llama.cpp with specific sampling parameters while your chat model runs on vLLM with different settings? That's the use case.
For people with limited VRAM (most people), the automatic hot-swapping is the killer feature. Instead of manually stopping one model to free memory for another, llama-swap handles it transparently based on which model your application requests. TTL-based unloading means unused models get evicted automatically.
Our Take
The local LLM ecosystem has a maturity problem. Tools like Ollama made local models accessible, but they also created walled gardens. You're running models through Ollama's server, with Ollama's quantization support, and Ollama's API quirks.
llama-swap represents the next phase: composable infrastructure. It doesn't care about model management or downloads - it cares about routing requests to the right backend and managing VRAM. That's a smaller, more focused problem, and it solves it well.
The 2.6k stars suggest this isn't just a niche project anymore. If you're running multiple local models and feeling the friction of switching between them, llama-swap is worth testing. The zero-dependency single binary makes it low-risk to try. Just keep in mind it doesn't replace Ollama for model downloading and management - it replaces how you serve and switch between models at runtime.