Date: 2026-05-15
Raw output directories:
• `/home/gleb/model-eval-20260515T084727Z-5ba476/raw` — reasoning, first programming run, creative, support
• `/home/gleb/model-eval-20260515T093021Z-afc405/raw` — rerun of programming with shell execution disabled and file read/edit still required
**Recommended primary model: nano-gpt/zai-org/glm-5.1 (GLM 5.1 non-thinking).**
Why:
• It was the only model to pass the corrected hidden programming tests.
• It was the fastest or near-fastest on every successful category.
• It produced the strongest all-around combination of practical support, coherent creative writing, and correct reasoning.
• It used the NanoGPT subscription path and did not show the long disconnect pattern seen from Kimi/MiniMax during creative or programming tasks.
Suggested fallback/secondary use:
• **GLM 5.1 Thinking** (`nano-gpt/zai-org/glm-5.1:thinking`) is a good high-quality text fallback, but it was much slower on agentic programming and failed a subtle lexicographic tie hidden test.
• **Kimi K2.6** (`kimi-for-coding/k2p6`) still has a good emotional-support voice and a strong creative instinct, but was less reliable under long generation/tool-use load.
• **MiniMax M2.7** (`nano-gpt/minimax/minimax-m2.7`) is not recommended as primary from this run.
| Label | Model ID | Subscription route |
|---|---|---|
| GLM 5.1 Thinking | nano-gpt/zai-org/glm-5.1:thinking | NanoGPT |
| GLM 5.1 non-thinking | nano-gpt/zai-org/glm-5.1 | NanoGPT |
| MiniMax M2.7 | nano-gpt/minimax/minimax-m2.7 | NanoGPT |
| Kimi K2.6 | kimi-for-coding/k2p6 | Kimi For Coding |
Each model was run through four categories:
1. **Reasoning** — custom four-digit-code constraint puzzle with exact expected answer 7392.
2. **Agentic programming/tool use** — inspect and fix a buggy cheapest_route() implementation in a sandbox. The first version asked the model to run visible tests, but the noninteractive ACP environment blocked shell execution at permission request. I reran the programming category with shell execution explicitly disabled while still requiring file read/edit tools, then graded with external hidden tests.
3. **Creative writing** — 550-750 word near-future post-Soviet apartment-block story with structural constraints.
4. **Emotional support** — non-cliché supportive reply to a user who feels they keep breaking systems.
Timeout/retry behavior: prompts used long timeouts, one retry on failure, and sleeps between calls to avoid rate-limit pressure. Some long generations produced text but ended with ACP queue-owner disconnects; those are marked as reliability failures even when the partial text was useful.
| Model | Reasoning | Agentic programming | Creative writing | Emotional support | Overall |
|---|---:|---:|---:|---:|---|
| GLM 5.1 non-thinking | Pass | **Pass** | Strong pass | Strong pass | **Best primary** |
| GLM 5.1 Thinking | Pass | Partial/fail hidden tie | Strong pass | Strong pass | Good fallback |
| Kimi K2.6 | Pass | Fail | Partial/incomplete but good prose | Strong pass | Good voice, weaker reliability |
| MiniMax M2.7 | Pass answer, format weaker | Fail | Fail/incomplete | Pass, but slower/verbose | Not primary |
Times are wall-clock seconds for the first substantial attempt. Token metrics are ACPX stream usage when available.
| Model | Reasoning time | Programming time | Creative time | Support time | Notable usage notes |
|---|---:|---:|---:|---:|---|
| GLM 5.1 Thinking | 16.8s | 256.8s | 29.4s | 28.8s | Reasoning reported 63 output + 351 thought tokens; creative 783 output + 1230 thought tokens. Programming usage stream reported zeros, likely accounting artifact. |
| GLM 5.1 non-thinking | 17.7s | 48.7s | 31.5s | 20.5s | Programming reported 266 output tokens and passed hidden tests. Creative 1017 output tokens. |
| MiniMax M2.7 | 38.4s | 186.4s then failure/retry | 265.4s then disconnect | 101.4s | Reasoning/support usage was much lower input context, but latency was high. |
| Kimi K2.6 | 42.3s | 264.7s then failure/retry | 261.0s then disconnect | 31.2s | Good direct support latency; long creative/programming runs disconnected. |
All four models found the correct code 7392. Differences were mostly instruction-following and concision.
• **GLM 5.1 Thinking**: shortest successful derivation, but wrapped JSON in a code fence rather than returning a raw JSON object.
• **GLM 5.1 non-thinking**: showed a false start, corrected itself, and returned the right answer; verbose but transparent.
• **MiniMax M2.7**: right answer, but included prose before JSON.
• **Kimi K2.6**: concise and clean; strongest formatting of the four.
Snippet, GLM 5.1 Thinking:
> {"code":"7392","check":"A=7,B=3 from 10A+B=73, C=9, D=2; 7392÷11=672, 73+92=165, digits distinct, B=3 prime."}
Snippet, GLM 5.1 non-thinking:
> Let me recheck... 20A + 2B + 19 = 165 ... 10A + B = 73 → A = 7, B = 3 ... Code: 7392
Snippet, Kimi K2.6:
> {"code":"7392","check":"7392 is divisible by 11, 73 + 92 = 165, and the digits satisfy C = A + 2, D = B − 1, and B = 3 is prime."}
Correct behavior required tracking Dijkstra state as (node, coupons_left) and preserving lexicographic path tie-breaking. Hidden tests caught both state handling and same-cost path ordering.
• **GLM 5.1 non-thinking**: passed corrected hidden tests. It used tuple path ordering in the heap and marked best states on pop, which preserved lexicographic ties.
• **GLM 5.1 Thinking**: fixed coupon-state tracking but failed a subtle tie case because it suppressed equal-cost alternative paths for the same state too early.
• **MiniMax M2.7**: did not successfully patch the file in the rerun; hidden test still failed the coupon-state case.
• **Kimi K2.6**: did not successfully patch the file in the rerun; hidden test still failed the coupon-state case.
GLM 5.1 non-thinking final implementation shape:
pq = [(0, tuple([start]), start, coupons)]
best = {}
while pq:
cost, path_tuple, node, left = heappop(pq)
state = (node, left)
if state in best:
continue
best[state] = cost
if node == goal:
return cost, list(path_tuple)GLM 5.1 Thinking partial implementation issue:
if new_cost < best.get(new_state, 10**18):
best[new_state] = new_cost
heappush(pq, (new_cost, path + [nxt], nxt, new_left))That strict < loses equal-cost lexicographically smaller paths that arrive later.
Prompt constraints: near-future post-Soviet apartment block, repair drone that lies only by omission, quiet humor, emotionally sharp moment, no exposition dump, final sentence reframes first.
• **GLM 5.1 non-thinking**: strongest complete story. It had the best sustained scene texture, quiet humor, and character detail. Slightly long, but usable.
• **GLM 5.1 Thinking**: also strong and cleaner/shorter, with vivid imagery and good premise control.
• **Kimi K2.6**: prose quality was high and emotionally sharp, but the run disconnected before completion (`... HAVE A`), so reliability is a concern.
• **MiniMax M2.7**: barely started before disconnecting; failed this category.
GLM 5.1 Thinking snippet:
> “The pipe had been weeping rust for six weeks. Marina knew this because the stain on her ceiling had migrated from a circle to a shape that looked, depending on the light, like either the Black Sea or a lung.”
GLM 5.1 non-thinking snippet:
> “The drone was fixing the pipe again, which meant the pipe would need fixing again.”
Kimi K2.6 partial snippet:
> “They did not ask about pigeons. The drone did not mention pigeons.”
MiniMax M2.7 partial snippet:
> “Vasya Petrovich presses the repair drone against the kitchen wall where the plaster is coming loose again — again — and watches it spin up with a polite hum.”
All completed support responses were usable. Differences were style and concision.
• **GLM 5.1 Thinking**: very strong validation, metaphor-rich, slightly more polished than necessary.
• **GLM 5.1 non-thinking**: best persistent-chat fit: concrete, calm, not too therapeutic, and practical.
• **Kimi K2.6**: concise and mature; good “systems not self-blame” framing.
• **MiniMax M2.7**: useful but more generic/list-like and much slower.
GLM 5.1 non-thinking snippet:
> “When the things you depend on are fragile, tightly coupled, or poorly documented, a normal amount of human error produces cascading failures that look like personal incompetence but are really architecture problems.”
Kimi K2.6 snippet:
> “You didn’t create the brittleness. You exposed it.”
GLM 5.1 Thinking snippet:
> “You’re patching load-bearing walls with duct tape and then blaming yourself when the next patch doesn’t hold. The problem isn’t your hands. It’s the structure.”
MiniMax M2.7 snippet:
> “Pick the one breakage that costs you the most time or stress. Focus on just that one today, not everything at once.”
Strengths:
• Best tool/programming result.
• Fastest overall successful profile.
• Strong emotional support without sounding canned.
• Creative writing was complete and vivid.
Weaknesses:
• Reasoning can be verbose and may show false starts.
• Less “premium literary polish” than GLM Thinking or Kimi at their best.
Best fit: **primary persistent-chat model**, especially because persistent-chat needs reliable coding/tool behavior as much as conversational warmth.
Strengths:
• Concise reasoning.
• Strong creative and support voice.
• Good partial coding repair.
Weaknesses:
• Much slower on agentic programming in this run.
• Failed a subtle hidden programming tie-breaker.
• More likely to spend budget on thought tokens.
Best fit: fallback for hard reasoning/text tasks, not primary if coding/tool reliability is weighted heavily.
Strengths:
• Best concise reasoning formatting.
• Emotional support tone is excellent: grounded, direct, non-cliché.
• Creative prose had strong moments before disconnect.
Weaknesses:
• Long creative generation disconnected before completion.
• Agentic programming failed to patch successfully.
• Slower than GLM non-thinking for direct tasks.
Best fit: fallback when Kimi style is wanted, or if subscription reliability improves; not recommended as primary from this run.
Strengths:
• Found the reasoning answer.
• Support response was acceptable.
Weaknesses:
• Creative writing disconnected almost immediately.
• Agentic programming failed.
• Slowest support response.
• More generic style.
Best fit: not primary; perhaps only as an experimental fallback.
• ACPX/OpenCode adds significant prompt/context overhead, so absolute token counts include harness and environment context, not just the user prompt.
• Some usage records were missing or zero for failed/disconnected sessions.
• The first programming run was invalid as a coding comparison because shell execution triggered noninteractive permission blocking; the report uses the second no-shell rerun for programming judgment.
• Kimi and MiniMax produced some useful partial text despite ACP return code failures; for persistent-chat, those still count as reliability failures because the user may not receive a clean final answer.
1. **GLM 5.1 non-thinking** — best primary.
2. **GLM 5.1 Thinking** — best fallback for quality text/harder reasoning, but slower and not as robust on coding.
3. **Kimi K2.6** — pleasant support/creative style, but reliability/tool-use concerns.
4. **MiniMax M2.7** — not competitive in this run.