LLM Inference Optimization: llama.cpp, GGUF Quantization, Hybrid

You launch Llama 3 70B on a server with 64 GB RAM and no GPU? Out-of-memory errors, latency 50+ seconds per token — familiar pain. Our team solved this for a dozen projects: we use llama.cpp with GGUF quantization. Result: 5–12 token/sec on a dual-socket Xeon, the model fully on-premise, no cloud costs. Additionally, tuning a CPU+GPU hybrid can squeeze up to 20–30 token/sec on a 70B model with even a single RTX 4090.

Why llama.cpp is the best choice for CPU inference

llama.cpp is a C++ implementation of LLM inference with aggressive optimizations for CPU and hybrid CPU+GPU output. The main feature is Quantization GGUF, which compresses models without critical quality loss. Quantization reduces the precision of model weights, enabling efficient deployment on resource-constrained hardware. Q4_K_M is 4x more memory-efficient than F16 while preserving 96% quality — the best balance for production loads.

Quantization type comparison for a 7B model:

Q4_K_M preserves 95–98% of F16 quality at 4x smaller size — the best balance for CPU inference. For production, we almost always choose it because it gives acceptable accuracy with minimal memory consumption. This makes llama.cpp ideal for edge devices with limited memory.

How to set up a CPU+GPU hybrid in 1 day?

If GPU memory isn't enough for the entire model, llama.cpp allows offloading some layers to the GPU. A typical command for a 70B model with 24 GB GPU:

./server -m llama-3-70b-q4km.gguf -ngl 32 --host 0.0.0.0 --port 8080

The -ngl 32 parameter offloads 32 layers to the GPU (usually ~60% of the model). The rest goes to the CPU. We automatically select the number of layers for your hardware to maximize throughput.

Advanced optimizations such as KV cache reuse and flash attention further reduce latency. Model parallelism can also be employed to distribute layers across multiple devices.

Step-by-step hybrid setup:

1. Compile llama.cpp with CUDA or Metal support: make LLAMA_CUDA=1.
2. Download the model in GGUF format (we recommend Q4_K_M).
3. Run with the -ngl N parameter, where N is the number of layers that fit in VRAM.
4. Check GPU occupancy via nvidia-smi. Aim for >90% utilization without swapping.
5. Tune batch size (--batch-size) and number of threads (--threads) for the CPU.
6. Conduct load testing: measure p99 latency and throughput.

What problems do we solve?

Case: fintech company, 70B model for transaction analysis. The client wanted full on-premise without sending data to the cloud. Hardware: 2×Xeon Gold 6248, 128 GB RAM, one RTX A6000 (48 GB). We set up Q4_K_M + offloading 32 layers to the GPU. Result: 15 token/sec, p99 latency <200ms, cloud GPU savings up to $3000 per month. The project was deployed in 2 weeks. Compare: cloud API would give 30–40 token/sec, but would cost $5000/month and require data transfer to a third party. Our certified engineers guarantee seamless deployment with 5+ years of MLOps experience and 40+ LLM deployment projects.

What is included in the work?

• Analysis of target hardware and latency requirements
• Compilation of llama.cpp with optimizations for the specific CPU (AVX2, AVX-512)
• Quantization selection and quality testing
• Configuration of llama-server (OpenAI-compatible API, batching, limits)
• Load testing with p99 latency measurements
• Operational documentation and 3-hour training session for your team
• Access to monitoring dashboard and ongoing support

Typical mistakes when setting up llama.cpp

• Using a build without AVX on an old CPU — inference drops by 2–3x
• Unnecessarily large context length (8192) — latency increases
• Wrong quantization choice (e.g., Q4_0 for accuracy-critical tasks)

How to increase inference speed on CPU?

Use a llama.cpp build tailored to your CPU with flags -mavx2 or -mavx512. Choose Q4_K_M quantization. Reduce context length (e.g., 4096 instead of 8192). Disable unnecessary attention layers if the model allows. Increase the number of threads (--threads) to the number of physical cores. On dual-socket configurations, tune NUMA balancing. These steps yield a 2–4x speedup on CPU.

Performance on real hardware

Timelines and cost

Inference setup takes from 1 to 3 weeks depending on complexity. Cost is calculated individually after analyzing your hardware and requirements. Get a consultation — we will assess your project free of charge. Our experience: 5+ years in MLOps, 40+ LLM deployment projects. Contact us to discuss optimizing your inference.