Running Gemma 4 locally on a 16GB GPU card? The right quantization makes or breaks the experience.
Google's Gemma 4 26B A4B is a Mixture of Experts model - meaning it has 26 billion total parameters but only activates around 4 billion for any given request. That architecture keeps memory usage far lower than a traditional 26B model, which is exactly why it fits on a 16GB card at all.
Quantization is the process of compressing a model's numerical weights to take up less space, at some cost to quality. Too aggressive, and the model loses reasoning ability. Too conservative, and it won't fit in VRAM. The sweet spot for 16GB cards right now is the Unsloth IQ4_XS variant, available on Hugging Face as gemma-4-26B-A4B-it-UD-IQ4_XS.gguf. The IQ4 format stores each parameter using roughly 4 bits - about half the memory of a full-precision model - and the XS compression keeps the file manageable for the memory budget.
Bartowski also published quantizations of the same model, and they run fine, but head-to-head testing shows the Unsloth version preserves reasoning quality better while keeping vision support (the ability to analyze images) intact. If you don't need image understanding, you can push to a more compressed format and recover some quality on text tasks.
16GB VRAM cards include the RTX 4080, RTX 3090, RTX 4060 Ti 16GB, and the Mac M-series chips (which share system memory). The model loads via llama.cpp or Ollama using the GGUF file format.
For anyone on a tighter 8GB card, the smaller Gemma 4 variants are the practical option - the 26B A4B, even compressed, needs the full 16GB to breathe.