Local LLM Hardware Evaluation - 4 Scenarios

Executive Summary

For cost-effective local LLM inference, **CPU-only with 64GB RAM (Scenario 1)** can run 13B models at usable speeds. **RTX 3060 8GB (Scenario 2)** is the sweet spot for 7B-8B models. **RTX 4090 24GB (Scenario 3)** enables 30B+ models at high speed. **Mac/Unified Memory (Scenario 4)** offers unique advantages for large models but at higher cost.

---

Scenario 1: Mini-PC (Intel 9500T, 64GB DDR4, No GPU)

**Status: You already own this**

What You Can Run

|-----------|-------------|----------|-------|-------------|

| **7B** | Q8_0 | ~8-9 GB | 3-5 tok/s | ⚠️ Slow but usable |

| **13B** | Q8_0 | ~13-15 GB | 1-2 tok/s | ❌ Too slow |

| **30B+** | Any | 18GB+ | <1 tok/s | ❌ Not feasible |

Key Limitations

• **CPU-bound inference**: The 9500T is a low-power dual-core CPU. Expect ~5-10 tok/s on 7B models, which is usable but not fast.

• **No AVX-512**: The 9500T only has AVX2, missing the vector instructions that make modern CPU inference fast.

• **Memory bandwidth**: DDR4 at likely 2400-2666 MT/s gives ~20-30 GB/s. For comparison, an RTX 3060 has 224-360 GB/s.

• **Power efficiency**: Good for always-on (35W TDP), but slow.

Best Models for This Setup

1. **Phi-4 mini 3.8B Q8_0** — Fastest option, good quality for size

2. **Llama 3.2 3B Q8_0** — Meta's efficient small model

3. **Qwen 2.5 3B Q8_0** — Strong multilingual support

4. **Mistral 7B Q4_K_M** — If you can tolerate 5-8 tok/s

Verdict

**Usable for small models only.** Good for a quiet, always-on assistant running 3B models. 7B models work but feel sluggish. Forget about 13B+.

**Cost to upgrade to Scenario 2**: ~$200-250 for a used RTX 3060 12GB (which fits in most mini-PCs with a PCIe riser and external PSU, or a new case).

---

Scenario 2: Desktop (Intel i5-12500, 96GB DDR4, RTX 3060 8GB)

**Status: You already own this**

What You Can Run

|-----------|-------------|-----------|-------|-------------|

| **7B** | Q4_K_M | ~5 GB | 40-50 tok/s | ✅ Fast |

| **13B** | Q8_0 | ~11 GB | ❌ | ❌ Won't fit |

| **14B+** | Q4_K_M | 9GB+ | ❌ | ❌ Won't fit |

Key Characteristics

• **GPU offloading**: The RTX 3060 8GB can offload most/all layers of 7B models, giving GPU-speed inference.

• **CPU fallback**: With 96GB system RAM, you can run models entirely on CPU if they don't fit in VRAM, but at 1/10th the speed.

• **Hybrid mode**: For 13B models at Q4, you can offload ~20-25 layers to GPU and run the rest on CPU. This gives ~10-15 tok/s.

• **i5-12500**: 6 cores/12 threads with AVX2. Decent for CPU fallback, but not as fast as modern CPUs.

Best Models for This Setup

1. **Qwen 3.5 9B Q4_K_M** — Best quality that fits comfortably (~6GB VRAM)

2. **Llama 3.1 8B Q8_0** — Maximum quality at 7-8B size (~8GB VRAM, tight)

3. **Mistral 7B Q8_0** — Fast and high quality (~7GB VRAM)

4. **Phi-4 14B Q4_K_M** — Largest model that fits, but tight (~8.7GB VRAM)

Verdict

**The sweet spot for cost-effective local inference.** You can run 7B-9B models at excellent speed (25-50 tok/s) and 13B models at acceptable speed (15-20 tok/s). The 96GB RAM means you can also experiment with CPU inference on larger models.

**Cost to upgrade to Scenario 3**: ~$1,200-1,600 for a used RTX 4090 24GB.

---

Scenario 3: Desktop Upgraded (i5-12500, 96GB DDR4, RTX 4090 24GB)

**Status: Expensive but powerful upgrade**

What You Can Run

|-----------|-------------|-----------|-------|-------------|

| **7B** | FP16 | ~14 GB | 50-60 tok/s | ✅ Overkill |

| **7B** | Q8_0 | ~8 GB | 90-110 tok/s | ✅ Extremely fast |

| **13B** | Q8_0 | ~13 GB | 50-60 tok/s | ✅ Fast |

| **14B** | Q8_0 | ~14 GB | 45-55 tok/s | ✅ Fast |

| **32B** | Q4_K_M | ~18-20 GB | 30-40 tok/s | ✅ Good |

| **34B** | Q4_K_M | ~19-20 GB | 30-40 tok/s | ⚠️ Near limit |

| **70B** | Q4_K_M | ~38-40 GB | 5-10 tok/s | ❌ Won't fit (needs 40GB) |

| **70B** | Q2_K | ~25 GB | 3-5 tok/s | ⚠️ Degraded quality |

Key Characteristics

• **1008 GB/s memory bandwidth**: 3-4x faster than RTX 3060. This directly translates to tokens/sec.

• **24GB VRAM ceiling**: The defining limit. You can run 30B models comfortably, 34B models at the edge, but 70B models require aggressive quantization or don't fit.

• **Power draw**: 450W TDP. You'll need a 850W+ PSU.

• **Bottleneck**: The i5-12500 might bottleneck prompt processing (not token generation) for very large contexts, but it's fine for most use.

Best Models for This Setup

1. **Qwen 3.5 32B Q4_K_M** — Sweet spot: excellent quality, fits comfortably (~18GB)

2. **Llama 3.3 70B Q2_K** — If you absolutely need 70B (degraded quality, ~25GB)

3. **Mixtral 8x7B Q4_K_M** — MoE architecture, ~16GB, very fast (can hit 195 tok/s!)

4. **Qwen 3.5 14B Q8_0** — Maximum quality for smaller model (~14GB)

Verdict

**Professional-grade local inference.** You can run 30B models at 30-40 tok/s, which is genuinely useful. 70B models require compromises. The 4090 is the best single-GPU option for models under 34B parameters.

**Cost**: ~$1,200-1,600 used (or $1,599 new if you can find MSRP).

---

Scenario 4: Mac or Specialized Computer (Unified Memory)

**Status: Most expensive, unique advantages**

Option A: Mac mini M4 Pro 48GB (~$1,799)

|-----------|-------------|----------|-------|-------------|

| **7B** | Q8_0 | ~8 GB | 20-25 tok/s | ✅ Good |

| **13B** | Q8_0 | ~14 GB | 15-18 tok/s | ✅ Good |

| **32B** | Q4_K_M | ~20 GB | 12-18 tok/s | ✅ Good |

| **70B** | Q4_K_M | ~40 GB | 8-12 tok/s | ⚠️ Fits but slow |

| **70B** | Q8_0 | ~70 GB | ❌ | ❌ Won't fit |

| **100B+** | Q4_K_M | ~50-60GB | 3-6 tok/s | ⚠️ Very slow |

Option B: Mac Studio M4 Max 64GB (~$3,999)

|-----------|-------------|----------|-------|-------------|

| **32B** | Q8_0 | ~30 GB | 15-20 tok/s | ✅ Good |

| **70B** | Q4_K_M | ~40 GB | 8-15 tok/s | ✅ Good |

| **70B** | Q8_0 | ~70 GB | ❌ | ❌ Won't fit |

| **100B+** | Q4_K_M | ~50-60GB | 6-12 tok/s | ⚠️ Slow |

Option C: Mac Studio M4 Max 128GB (~$7,999)

|-----------|-------------|----------|-------|-------------|

| **70B** | Q8_0 | ~70 GB | 6-12 tok/s | ✅ Fits! |

| **100B+** | Q4_K_M | ~50-60GB | 6-12 tok/s | ✅ Fits |

| **405B** | Q2_K | ~100GB | 2-3 tok/s | ⚠️ Very slow |

Key Characteristics

• **Unified memory**: All RAM is VRAM. No PCIe bottleneck. A 64GB Mac has 64GB of effective VRAM — more than an RTX 4090.

• **Lower bandwidth**: M4 Pro has 273 GB/s (vs 4090's 1008 GB/s). M4 Max has 400-546 GB/s. This means slower tokens/sec than the 4090 for models that fit on both.

• **Silent operation**: ~10-45W power draw vs 450W for the 4090.

• **No CUDA**: Uses Metal backend. Some models/features may not be supported as quickly as CUDA.

• **Not upgradeable**: You cannot add RAM after purchase.

Best Models for This Setup

1. **M4 Pro 48GB**: Qwen 3.5 32B Q4_K_M — the reason to buy this tier

2. **M4 Max 64GB**: Llama 3.3 70B Q4_K_M — only consumer device that runs this well

3. **M4 Max 128GB**: Llama 3.1 405B Q2_K — if you want the biggest model possible

Verdict

**Best for large models (70B+) and silent operation.** The 4090 is faster for models under 24GB, but Mac wins for models 32B-70B because they actually fit. The 128GB Mac Studio is the only consumer device that can run 70B at Q8 or 100B+ models.

**Cost**: $1,799 (M4 Pro 48GB) to $7,999 (M4 Max 128GB).

---

Cost-Effectiveness Comparison

|---------|------|-----------|-------|---------|

---

Recommendations by Use Case

"I'm poor and need the cheapest solution"

**Stick with Scenario 2 (your desktop).** Run Qwen 3.5 9B Q4_K_M at 30-40 tok/s. It's genuinely useful and costs $0.

"I want to run uncensored models locally"

**Scenario 2 or 3.** Download GGUF models from Hugging Face (TheBloke, bartowski) and run via Ollama or llama.cpp. Popular uncensored models:

• **Dolphin-Llama 3.1 8B** (Q4_K_M, ~5GB)

• **Mistral 7B OpenOrca** (Q4_K_M, ~4.5GB)

• **Llama 3.1 70B Uncensored** (requires Scenario 3 at Q2 or Scenario 4)

"I want the best quality possible"

**Scenario 3 (RTX 4090)** for 30B models at Q4, or **Scenario 4b (Mac Studio 64GB)** for 70B models at Q4.

"I want thinking/reasoning models"

**Scenario 3 or 4a.** Models like DeepSeek-R1-Distill-Qwen-32B or Qwen3-32B with thinking mode require 20-30GB RAM/VRAM.

"I want to run multiple models simultaneously"

**Scenario 4 (Mac with 48GB+)**. Unified memory makes it easier to load multiple models without VRAM fragmentation.

---

Software Setup

For all scenarios, use **Ollama** (easiest) or **llama.cpp** (most flexible):

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Run a model
ollama run qwen2.5:14b
ollama run llama3.1:8b
ollama run phi4:14b

# Or download GGUF directly and run with llama.cpp
./llama-server -m model.gguf --port 8080

For CPU-only (Scenario 1), use **ik_llama.cpp** fork for better CPU performance.

---

Bottom Line

| Your Goal | Recommendation |

|-----------|---------------|

| **Spend nothing** | Scenario 2 (desktop) — already excellent |

| **Best bang for buck** | Scenario 3 (RTX 4090) — $1,400 for 30B models |

| **Run 70B models** | Scenario 4b (Mac Studio 64GB) — $4,000 |

| **Silent/efficient** | Scenario 4a (Mac M4 Pro 48GB) — $1,800 |

| **Maximum everything** | Scenario 4c (Mac Studio 128GB) — $8,000 |

**My recommendation**: Start with Scenario 2 (free). If you need more, save for an RTX 4090 (used, ~$1,200-1,400). Only go Mac if you specifically need 70B+ models or silent operation.

*Document generated 2026-05-14 based on current market data and benchmarks.*