Running large language models locally has a well-known pain point: prompt processing on bigger models (think 27 billion parameters and up) can be painfully slow, making the tool feel unusable even when token generation speed is fine.
A fix that's gaining traction among local LLM users turns out to be surprisingly simple. The --ubatch-size flag in llama.cpp (the popular open-source tool for running models on consumer hardware) controls how many tokens get processed at once during the initial prompt evaluation. Setting this value to match your GPU's L3 cache size in megabytes appears to produce significant speed improvements.
For example, on an AMD RX 9070 XT with 64MB of L3 cache, setting --ubatch-size 64 made prompt processing on Qwen 27B jump from sluggish to actually usable. The logic makes sense: you're aligning the batch size with the fast memory your GPU can access without hitting slower VRAM, reducing bottlenecks during the compute-heavy prompt evaluation phase.
This is a narrow tip for a narrow audience - people running quantized models locally through llama.cpp rather than using cloud APIs. But for that group, the difference between a 5-second and a 30-second prompt evaluation is the difference between a usable tool and a frustrating one. If you're experimenting with local models on consumer GPUs, it's worth trying different --ubatch-size values matched to your specific hardware's cache specs.