Run cutting-edge local AI on everyday hardware.
Zero monthly fees. Complete privacy. Your data stays local.
Create and process documents, images, and files entirely on your device.
Mostlysane is a curated stack of three research forks merged into a single llama.cpp build. The goal: make the latest model architectures and context sizes accessible on the hardware people actually own.
Ported from a 1.58-bit ternary research fork. Weights become {-1, 0, +1} — 2 bits per parameter. Enables a new branch of models such as Bonsai 8B to run with minimal memory overhead.
K-side compression via TurboQuant: turbo4 (4-bit), turbo6 (6-bit), mixed precision. V-side via turbo3 (Walsh-Hadamard + PolarQuant, ~3.125 bit). Reduces KV cache to 3Gb even at 512K context.
Per-layer entropy profiling identifies which layers need full 8-bit K and which can use compressed types. Mixed precision keeps quality where it counts, memory savings everywhere else. Profiles for 5 models included, and a tool to create your own.
New MoE architectures (Qwen3.6 35B A3B) unlock massive capability. But without custom load configurations and optimizations to caching, the KV cache consumes gigabytes at longer contexts — pushing them beyond reach of 8–12 GB GPUs. With our stack, MoE models put the important layers into GPU, reduce KV Cache without loss of quality and provide great performance, even with a massive 512k context window.
Full-precision K/V cache at 512K context: ~11 GB
Qwen3.6 35B requires 24 GB+ GPU or reduced context
Calibrated entopy with TurboQuant K+V at 512K context: ~3 MB
Qwen3.6 35B fits on 8 GB GPU with full 512K context
Some things are more important that others. Mixed precision preserves quality on attention-critical layers.
Calibrated per model — set and forget.
We test configs against real-world multi-length recall & coding tasks and visual demos. Compression doesn't matter if quality suffers.
| Config | K Bits | V Bits | Quality | Prefill | Gen | 512K Cache |
|---|---|---|---|---|---|---|
| Default F16 K & V, No optimisation | 16 | 16 | Baseline | 3600 t/s | 12 t/s | ~10,500 MB |
| Q8_0 + Turbo3_0 with Entropy | 8 | 3.125 | Production | 3372 t/s | 107 t/s | ~3,000 MB |
| TurboQuant 6_0 + TurboQuant 3_0 | 6 | 3.125 | Lossy | 3168 t/s | 99 t/s | ~3,200 MB |
| TurboQuant 4_0 + TurboQuant 3_0 | 4 | 3.125 | Fragile | 3268 t/s | 104 t/s | ~2,700 MB |
Benchmarked on AMD Rzyen 3800x, RTX 3070 Ti 8GB • Qwen3.6 35B A3B See visual comparisons →
Generate a complete config tailored to your hardware in under 60 seconds.
Get My Config
Or dive straight in: curl -sSL mostlysane.ai/install.sh | bash