uh so like i gave a model like 20 senses so like yeah https://huggingface.co/heterodoxin/qwen3-8b-supermultimodal

This is the harness companion to my Qwen3.8-27B benchmark post. That post made the model fast; this one makes it useful: pi (the coding agent) against a local llama-server, tool calling, thinking control — and response times that don't hurt.

This is a setup guide, not a benchmark post. Every trap, config, and extension here is what I actually run daily. The benchmark side (the game-build harness, scorer, runtime gates, playtest protocol) lives in neon-ladder — this post links to it where relevant but doesn't duplicate it.

Everything below was verified live on my Flow Z13 (Ryzen AI Max+ 395, 8060S, 128GB): a 10-module game built in one session, 1,587 lines of working code, all from server logs and session files — not estimates.

Note: writing is AI-assisted; every number and config here comes from my own runs.

TL;DR

• Working recipe: pi + llama-server (Nathan's strix-halo Vulkan fork) + the Sharp chat template, 256k ctx (the model's training cap), maxTokens 32768, thinking wired via compat.chatTemplateKwargs. Verified end to end with request dumps and session logs.
• pi's defaults will silently sabotage a reasoning model: maxTokens 16384 can be eaten entirely by thinking, and thinking flags don't reach llama.cpp's template without compat-level wiring.
• Session economics are great: ~94% KV cache hit rate, stable across session types (a 16-turn game build and a 44-turn tool-heavy refactor both landed at 94.1-94.2%); only the first turn pays full prefill.
• Effort control works after wiring: off produced literally zero thinking tokens, and the level you pick changes code quality, not just speed (details in the build test).
• At pi's default temperature 0.8, planning-heavy prompts occasionally sample an instant-EOS first turn (one token, done). Retrying the identical prompt inherits the failure from cache; retry with a perturbed prompt or run temperature 0.
• Don't chase deep context; compact before it gets expensive. Auto-compaction set to fire around 95k keeps every turn in the fast band (decode 26+ t/s, prefill ~200 t/s) while sessions past ~140k pay 17-19 t/s decode and ~140-175 t/s prefill. One settings line does it.

The server side (brief)

Two server profiles, same machine (one resident at a time):

27B (daily driver): UD-Q4_K_XL (v3) + DFlash2 Q4_K_M drafter n4, f16 KV, drafter KV q8_0, -c 262144, power pinned with my z13ctl+ profile.

Flash-Next (speed lane): UD-IQ4_XS + native MTP Q8_0 sidecar, fixed n4, q8_0 KV, -c 131072, and --reasoning-effort medium --reasoning-budget 2048 — those flags are mandatory (without them, Flash-Next burns its entire output budget on reasoning and emits nothing). Full configs in the neon-ladder repo.

Ubatch 4096 for normal work; for deliberate deep fills use 2048 (probed clean through 139k) or 1024 (proven at 145k). The model's training cap is 262144, and the full config runs healthy there at ~55GB RAM. Three harness-relevant facts worth knowing:

• -ub 4096 has a hard ceiling: past ~140k tokens of fill it hits a deterministic Vulkan device-lost (twice, at nearly the same depth). -ub 2048 passed the same style of probe at 138.8k and -ub 1024 completed a real 144k session; the ceiling moves with ubatch, so smaller ubatch buys depth.
• Allocating big context costs nothing until filled: decode at 8k depth was identical with -c 65536 and -c 98304. Allocate the max.
• Deep sessions work but get slow linearly: a 144k-token agent session (resumed after a crash) decoded at 17-19 t/s throughout, with draft acceptance 0.62-0.92 the whole way. That's why the compaction setting below matters more than any ubatch choice.

Installing and wiring pi

pi is a terminal coding agent with unusually good local-model support. Install it, then point it at llama-server via ~/.pi/agent/models.json (not settings.json, that file ignores provider blocks):

json { "providers": { "llamacpp": { "baseUrl": "http://127.0.0.1:8080/v1", "api": "openai-completions", "apiKey": "dummy", "models": [ { "id": "qwen3.8-27b", "reasoning": true, "contextWindow": 262144, "maxTokens": 32768, "compat": { "thinkingFormat": "chat-template", "chatTemplateKwargs": { "reasoning_effort": {"$var": "thinking.effort"}, "enable_thinking": {"$var": "thinking.enabled"} } }, "thinkingLevelMap": { "minimal": null, "low": "low", "medium": "medium", "high": "high", "xhigh": null, "max": null } } ] } } }

Then pi --provider llamacpp/qwen3.8-27b, or set defaultProvider/defaultModel in settings.json.

Every field in that entry is load-bearing, and several of them exist because of a trap:

Trap 1: the silent cloud fallback

If pi can't resolve your provider config, it does not error. It uses whatever else is configured: your run can look successful while the session log shows a nonzero dollar cost and your server has processed zero requests, because pi has been talking to a cloud provider the whole time.

Always verify a local run server-side. Watch curl localhost:8080/metrics while the agent works: if prompt_tokens_total isn't climbing, you're not local.

Trap 2: maxTokens 16384 is a thinking bomb

pi's default maxTokens is 16384. For a reasoning model on a planning-heavy prompt, that's not an output budget, it's a thinking budget: a "build a game" prompt can spend all 16,384 tokens on reasoning and hit the length cap with zero code emitted, with stopReason: length in the session log and a model that looks "stuck."

Set maxTokens explicitly. 32768 covers everything in a normal tool-using session, including a turn that writes two files back-to-back. A length-capped turn is also not fatal: the next turn continues without corruption.

Trap 3: thinking flags don't reach the template by default

This is the subtle one, and the failure is silent.

llama-server's chat template (the Sharp template from the benchmark post) accepts chat_template_kwargs: enable_thinking and reasoning_effort. pi has flags for thinking levels (--thinking off/low/..., shift+tab to cycle). But between the two sits a mapping layer:

• The mapping config (thinkingFormat, chatTemplateKwargs) must live under compat on the model entry. At the top level of the model object it is silently ignored.
• With the wiring correct, --thinking low sends {reasoning_effort: "low", enable_thinking: true} and --thinking off sends {enable_thinking: false}.
• Without it, pi's flags go nowhere and the template defaults to thinking on, medium effort. The model thinks when you told it not to, and everything is slower.

Verify your own wiring before trusting it: point baseUrl at a logging proxy for one run and read the request body. It's ten minutes and it converts "I think it works" into "it works."

The thinkingLevelMap entry hides levels the template doesn't distinguish. The Sharp template has four real states (off, low, medium, high); pi cycles seven by default, three of which are aliases. The map collapses the cycle to the four that exist.

Trap 4: the instant-EOS prompt basin

At pi's default temperature 0.8, a planning-heavy tool prompt occasionally samples a degenerate first turn: the model emits a thinking tag, immediately stops, and the run ends with an empty response and a one-token generation in the server log.

On my game-build prompt this hits roughly one request in three to five. It is sampling behavior, not a server or client bug: replaying the identical request body at temperature 0 never fired it in six runs.

The compounding part is the retry. Resending the same prompt hits the KV cache, inherits the degenerate turn from history, and fails again, which makes the failure look deterministic and hardware-flavored. Retry with a slightly perturbed prompt (any unique marker appended) and it rolls fresh.

For reproducible benches I set "samplingParams": {"temperature": 0.0, "top_p": 0.95, "min_p": 0.05} on the model entry; for everyday sampling, perturbed retries are the fix.

What thinking control buys you

Same planning-heavy prompt, session-verified thinking token counts:

For quick edits use off, for generation-heavy work low or medium, for debugging and architecture high. shift+tab cycles levels live in a session.

One honest note on the Sharp template: it tames runaway reasoning on normal turns (that's in the benchmark post), but it does not bound reasoning on genuinely planning-heavy prompts. The bound comes from your effort setting plus the maxTokens headroom. Template + harness flags together are the complete answer.

Effort level also buys code coordination, not just volume. Two verified game builds, same prompt: the low-effort build passed every static check yet played worse in three measurable ways (ball not glued to the paddle before launch, ball speed tied to the monitor's refresh rate instead of a fixed timestep, flatter difficulty curve).

The medium-effort build got all three right. Syntax is free; the seams between modules are what thinking pays for.

More effort past medium, though, buys breadth instead of correctness.

A high-effort run of the same prompt produced 1,995 lines with three extra self-directed modules (audio, UI, paddle) and 93 tool calls, yet scored 13/15 against medium's perfect 15/15, dropped the same localStorage persistence the low-effort builds drop, shipped a latched input flag that left the keyboard dead at runtime, and took over twice the wall time.

The sweet spot for build-shaped tasks on this model is medium: perfect score, 16 tool calls, about 25 minutes.

That medium result is robust, not a lucky roll: two more independent medium builds (different ubatch, one with five auxiliary-model extensions loaded) scored 14-15/15 in 20-23 minutes each.

A fourth medium build added a per-module test suite to the same prompt: 51 tests written alongside the code, all green on arrival, 14/15 on the same checks, 36 minutes.

That's the tier I spec for real work now: for roughly 15 extra minutes the agent ships its own regression suite with the feature.

The multi-file build test (this became neon-ladder)

To validate the whole stack I had it build "Neon Overdrive", an arcade Breakout game, as a 10-file project: 8 JS modules, CSS, index.html, strict no-placeholder rules, syntax checks required. The full prompt is below so you can run the identical test on your own stack.

Result: 15 turns, 16 tool calls (12 writes, 3 bash checks, 1 read), 1,587 lines, all syntax checks pass, all seven feature requirements present in the code, ~25 minutes wall time.

One turn hit the 32k cap mid-double-file-write and the next turn picked up cleanly. And the game actually plays: paddle reflection angles, armored bricks shifting red to orange to yellow, volatile-chain explosions, tri-ball chaos, the upgrade shop between levels.

There's a built-in bonus to this benchmark: while your agent grinds through someone's 3,000-line refactor, you get a neon Breakout to play. Post your build quality and wall time in the comments; it will be interesting to see how other engines and models handle the identical prompt.

The prompt (paste as-is; it assumes a js/ and css/ dir will be created by the agent):

``` Build "Neon Overdrive", an arcade Breakout game, as a multi-file project you create with tools, file by file. NO external dependencies or CDNs; HTML5 canvas + CSS3 + raw JS only.

Required file structure (use the write tool once per file, complete code every time, zero placeholders): 1. index.html - loads css/styles.css and all js/ files via script tags in dependency order 2. css/styles.css - neon/cyberpunk UI, overlays for menu/pause/shop/game-over 3. js/config.js - constants: canvas size, brick grid, speeds, powerup drop rate (15%), colors 4. js/particles.js - particle engine: spawn(x,y,color), gravity + fade update, dead-particle cleanup 5. js/bricks.js - 5-row grid from an array matrix; standard (1 hit, neon blue), armored (3 hits, red->orange->yellow as damaged), volatile (1 hit, neon green, explodes destroying direct array neighbors) 6. js/balls.js - ball entities in an active balls array; paddle reflection angle from strike position vs paddle center; no game over until the LAST ball is lost; dead-ball cleanup 7. js/powerups.js - falling capsule entities; catching Tri-Ball injects two new balls into the array 8. js/states.js - rigid state machine: menu -> gameplay -> paused -> level clear / game over 9. js/shop.js - between-levels upgrade shop: spend credits on paddle speed or paddle width (persistent) 10. js/main.js - game loop, collision wiring, score/credits, keyboard input, level generation (procedurally harder)

Workflow, in order: A. Write all 10 files (write tool, one call each). B. Run: node --check js/config.js js/particles.js js/bricks.js js/balls.js js/powerups.js js/states.js js/shop.js js/main.js C. If any check fails, fix with the edit tool and re-run until all pass. D. Read index.html to verify every script tag path matches a real file. E. Report per-file line counts, then reply COMPLETE. ```

The prompt, scorer, and a retry wrapper that handles the Trap 4 basin are packaged in neon-ladder.

Scoring it is easy: all 10 files present, node --check passes clean, the seven mechanics are actually implemented (grep for the reflection math, the armored color shifts, the neighbor explosion), and the game runs when you open index.html.

Then play it for two minutes: the ball rides the paddle before launch, speed is framerate-independent, and upgrades survive a page refresh.

Reference numbers for this box: 1,587 lines, 16 tool calls, ~25 minutes at medium effort, zero placeholders.

Session economics over those 16 turns: 94.1% of prompt tokens served from KV cache (pi resends the full conversation every turn; llama-server absorbs it), ~237 t/s on the uncached remainder, decode in the low-to-mid 20s t/s with tool traffic mixed in, acceptance around 64-68%.

That's the whole reason local agentic coding works at all on this hardware: the harness's chat-pattern traffic is almost entirely cache hits, and the GPU only pays for new tokens.

Ling-3.0-tiny as the compaction service

Long sessions eventually need compaction, and there's no law saying the model that summarizes the session has to be the model doing the work.

Ling-3.0-tiny (8B total, 1.3B active, 4.8GB in Q4_K_M) is built for exactly this slot: prefill is its superpower, thinking can be disabled per request, and its hybrid attention keeps KV costs near zero.

The compaction test used a real session transcript: the full game-build session (16 turns of tool calls and results) plus all workspace files, 25,890 tokens in, asked for a structured handoff document (file inventory, verification status, bugs, next steps, constraints).

Result: a 799-token handoff in 19 seconds, and the quality holds up.

Every file and line count matched ground truth (all 10 files, 1,587 total), verification status was correct, and it refused to invent bugs that didn't exist; the constraints section surfaced exactly the architecture details a continuation session needs, from the CONFIG object and the rigid state machine to the last-ball rule and localStorage persistence.

One duplicated bullet was the only flaw, and the same job on the 27B would run roughly 5x slower.

One wiring rule, same family as Trap 3: call it through the chat endpoint (/v1/chat/completions) with chat_template_kwargs: {enable_thinking: false}. On the raw completion endpoint with a bare prompt the model degenerates into echoing workspace state in a repetition loop.

Through the chat endpoint with the template it is clean, fast, correct, and the rule is the same as pi's: the template is not optional.

Honest caveat: both tests sit at 26k and 49k input, not near the 256k ceiling; tiny's window is 256k, so there's room, but treat very deep compaction quality as untested until a session grows that large.

Set the auto-compaction threshold to ~95k and stop thinking about deep context. pi compacts when contextTokens > contextWindow - reserveTokens; the default fires only near the window's end, deep in the slow band. One line in ~/.pi/agent/settings.json moves it:

json { "compaction": { "enabled": true, "reserveTokens": 167144 } }

With contextWindow 262144 that compacts at ~95k: sessions cycle between roughly 95k and 25k (summary plus a 20k verbatim tail), every turn stays in the fast band, and the deep-context tax (device-lost ceilings, mid-teen decode, 140-175 t/s prefill) becomes somebody else's problem.

Compaction fires between agent runs, not mid-run, and each pass costs seconds on tiny.

Validated live: a 142k session crossed the threshold, compacted, and its continuation answered correctly about files read an hour earlier.

It also graduated to daily use: a 44-turn, 49k-token agentic session (TypeScript monorepo work, 44 tool calls) on the 27B, compacted with the extension live. Tiny summarized 8.1k tokens into a 2,282-token handoff in ~19 seconds: prefill at 2,904 t/s, generation at 137 t/s, server-side timings.

The same call on the 27B would have taken roughly two minutes, so ~6x end to end. The summary got every checkable fact right (file list, test counts, verification status) and the session continued cleanly after compaction, which is the real acceptance test.

The auxiliary model playbook

Compaction is just the highest-value slot for a second small model. The same pattern extends across the agent loop, and pi's extension events cover all of it. The full suite, with validation status:

The triage row deserves its caution label: every huge bash dump costs the main model context for the rest of the session, and compressing it with tiny first keeps sessions small enough that compaction fires later or never.

But triage changes what the main model sees, and if tiny drops the one error line that mattered, the 27B makes worse decisions and you won't know why.

Before relying on it, feed it real outputs from your own sessions and verify nothing load-bearing was dropped.

One honest A/B from the game-build workload, all five extensions loaded: zero tiny calls, identical wall time and score, because clean test suites and one-line write confirmations never cross the 6KB triage threshold.

Dormant extensions cost nothing; they earn their keep on fat tool outputs (failing test runs, build logs, repo-wide greps) and long sessions, which is exactly the traffic my daily driving produces.

The division of labor in one line: the 27B reads and writes the code, tiny reads and summarizes everything else.

All of these knobs (compaction threshold, maxTokens, temperature, triage size) are three files deep by default, so I keep a /tune extension next to the suite: /tune prints the live values, /tune compactAt 95 or /tune temperature 0 writes through to the right file with bounds checking, and /tune reset restores the documented defaults.

Readers running this stack on other boxes should adjust compactAt to their own fast-band edge rather than trust mine.

Response time cheatsheet

Biggest levers first, all measured:

1. Thinking level dominates. Reasoning streams at decode speed before you see a word.
2. Session warmth: first turn pays full prefill (~10s), subsequent turns are cache hits and start generating in under a second. Don't restart the server between questions; use -c continuation.
3. Lean context: extensions/skills/AGENTS.md all add to the first-turn bill.
4. Already optimal from the benchmark post: DFlash2 n4, ubatch 4096, f16 KV. Don't shrink -c for speed; allocation is free until used.

Attacking the prefill bill (the APU's real tax)

Dense-27B prefill (~250-300 t/s) is the slowest number in this stack, and a coding agent's traffic is mostly prefill. Everything above already helps (cache hits, tiny offloading), but three more angles are worth knowing:

Keep the cache alive across turns. The 94% hit rate is the single biggest prefill saver, and its enemy is cache invalidation. Two habits preserve it: don't edit early messages mid-session (everything after the edit re-prefills), and let pi's cacheRetention default do its job. The compaction extension already sets cacheRetention: "none" for one-off summaries, which avoids polluting the main prefix.

Shrink what gets re-sent. pi resends the full conversation every turn; that's the protocol. The levers are content levers: tool-result triage from the playbook above (smaller history, smaller resend), and keeping generated outputs from ballooning (thinking low on generation-heavy turns does this too).

Route around the 27B when the job is prefill-shaped. Compaction, branch summaries, commit messages and repo maps are all "read a lot, write a little" jobs, which is exactly the profile where a 1.3B-active MoE crushes a dense 27B. Anything in your workflow that looks like "summarize/index/triage" should default to the aux model; reserve the 27B's prefill for context it genuinely needs to see.

What doesn't work: quantizing the main model below Q5 to speed prefill (prefill is compute-bound, the quant barely moves it, and decode pays the quality), and shrinking -c (allocation is free until filled, as measured above).

Which model when

The control experiment settled it: at matched effort and environment, Flash-Next and the 27B produced identical scores, identical failure sets, and identical line counts on the same contract. Flash-Next got there in 7.8× less wall time with 4.3× less reasoning. The 27B compensated by writing and running its own smoke test mid-build.

Pick the 27B when: - You have under ~91GB of GPU memory - You want 256k context - The job is quality-critical and you want the model that spontaneously verifies its own work

Pick Flash-Next when: - You have 91GB+ available and speed is the product - Your workload is emission-heavy (tool calls, scaffolding) — Flash-Next hits 40 t/s there - Build time matters more than build depth

The simplest rule: if you have the memory, Flash-Next for speed, the 27B for depth. If you don't, the 27B at Q4-v3 is never wrong.

What changed since posting

The Sharp template moved to v22.4.0 (reasoning-effort aliases, inline control tags, thinking-off fast-mode fixes). I A/B'd it on the game bench: score and speed in-family with every number above; the harness repo ships it as default with the earlier version vendored for exact reproduction.

The game-build bench grew runtime gates. A 120-second headless gameplay soak is now the default (frame advancement, reload detection, synthetic play throughout), added after a real freeze past the 60-second mark that shorter gates structurally cannot see. An uncaught page error is fatal; caught per-frame errors warn. The static scorer learned two velocity-multiplication bug patterns that shipped in builds passing every static check.

Static score anti-correlates with playability. The two highest-scoring builds of the richer-contract era were the two broken games. The final grade is and remains the human playtest; the gates are necessary, not sufficient.

Effort plumbing got real. --reasoning-budget was flat 8192 across all thinking levels in every number above; it's now mapped per level (the per-request thinking_budget_tokens field wires it cleanly), and the server accepts a top-level reasoning_effort natively ("none" is a validated zero-reasoning switch). One Sharp caveat: changing reasoning_effort mid-session re-renders the system block and invalidates the whole KV prefix on v22.3.2/v22.4.0 — the inline control tags are the safe per-turn mechanism.

Tool-choice behavior, from the session traces: models edit surgically for small fixes and rewrite whole files for cross-file structure — anchor strings past the context window are the reason. The contract now says so explicitly.

Manual beats headless on speed, loses on reliability: interactive runs (clean context, no extensions) hit 92% GPU utilization and halved walls, but shipped 1-of-3 playable vs the headless runner's 6-for-6. Deliberation is where the self-correction lives.

The bench crossed model families and engines — Flash-Next (125B MoE, native MTP) built the contract first-try at 17/19 on a stack that didn't exist when any of this was written. And a control cell closed the size question: Flash-Next vs 27B at low effort, identical environment — identical scores, identical failure sets, identical line counts. Flash-Next in 7.8× less wall time with 4.3× less reasoning; the 27B wrote and ran its own smoke test mid-build. On explicit contracts, model size buys speed, not quality.

Twelve gameplay-failure classes now, every one found by a human playtest, zero by static score — the last was a ball that vanishes mid-game, shipped in the release-gate build that scored 17/19 and passed its soak. The gates are necessary; the clicking is the grade.

Everything is reproducible from neon-ladder — contract, scorer, runtime gate, runner, one-comment recipe.

Sources

• Benchmark harness (game-build bench, scorer, gates, runner): neon-ladder
• This post's pi extensions (compaction, triage, repomap): not in the repo — they're personal tooling; ask in comments if you want them
• pi (coding agent): https://github.com/earendil-works/pi
• llama-server engine: Nathanw1014/strix-halo-llamacpp (v0.6.5)
• Sharp chat template: https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates
• Ling-3.0-tiny (compaction service): https://huggingface.co/inclusionAI/Ling-3.0-tiny
• Model + drafter + quant revision pinning: see the benchmark post (https://www.reddit.com/r/LocalLLaMA/comments/1vlmh0b)

Happy to answer setup questions. The playbook rows still marked as needing quality testing are exactly that: promising, wired, but not yet proven on real workloads. Treat them as experiments and validate on your own sessions before making them load-bearing.

Defined as AI exceeding human cognitive abilities. 20 years in prison. Plenty of local models already fall under that big of an umbrella in some capacities.

This is why it's not enough to say that you could torrent open models so who cares what the politicians do. They want you to not have access to anything good and will put you in prison for it.

I’ve been running qwen on my personal Mac but I’m getting to the point where’d I’d like to have something always on, running various jobs, and some more ability to experiment and earn about fine tuning.

I’d like to keep things <$5k if possible.

To anyone with any of these 3 platforms, what’s your experience been like? I’m drawn towards the DGX spark for concurrency and CUDA (which I have very little experience with) but I’m a little turned off by it’s memory bandwidth.

I have the most experience with Mac but those prices are eye watering and it feels like a lateral from my personal MacBook Pro.

Is using their q8 version fine or will i get better results on q16?

I mean, this is on a Mac, why is a 8 years old Ubuntu AppImage being halu-installed...?

And this message is in the middle of pulling some tensor metadata from HF. Never even heard of OpenD before this ... totally hallucinated stuff. And this is not a low quant - it's a 5bpw quant, with Q4 the lowest of any tensors.

EDIT: I'm not looking for a solution -> I'm genuinely asking if other people have noticed hallucinations and weird reasoning.

Nice one

DeepSeek finally has eyes. For people who kept a separate vision model beside DeepSeek, that fills an obvious gap.

The awkward part is the box. The checkpoint is 305B, and DeepSeek's vLLM recipe uses one node with four GB300 GPUs. That is far beyond a casual desktop build, especially while accelerators and memory remain expensive.

That ugly hardware bill could still make sense for a team feeding it images all day. You pay once for the machine and stop paying per request, but the machine keeps charging you in power, cooling, and maintenance. If the GPUs stay busy and the product needs predictable latency, I can see the math working. For a weekend project, those cards would spend most of their life idle.

I tried its vision capability through the ZenMux API and came away impressed. Recognition was strong on the images I used, though this was an informal test rather than a benchmark. I have no idea how much changes after quantization and local serving. If you have it running locally, what hardware and quantization are you using? How does the visual quality compare, and what TPS do you get?

Sources
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

Goal Run Qwen 3.x 27b locally for agentic coding - I'd also run other models of similar or smaller size for other uses

Would this hardware be appropriate (for starters) or would I hit a point of frustration pretty quickly?

Specs - MSI B650 tomahawk motherboard (included the info b/c I know you can't really run 2 GPUs in here, but I could swap this for another AM5 that can handle x8/x8, something like the X870E?) - Gskill 64gb of memory at 6000mhz and 36cl - I've read offloading some context to system RAM can help but performance takes a hit - 7800x3d cpu - (what the user is selling; might matter if I want to run 2 GPUs with an upgraded MoBo) - 7900xtx 24gb vram - I could add a second one eventually

I can get this for around $2,500 used. Or for the money, would it be worth it to take slower bandwidth but get a 64gb AI 395+ system?

I've only had the chance to try small models on a laptop, AMD 7640U with 64gb system ram (helps me load models but using CPU is a terrible experience), and other small models on a 24gb ram M5 Mac.

When the budget is limited, this all feels like a dance between: - A spacious but slow camper van, bigger job, but slow - A fast hatchback, smaller jobs, but significantly faster.

There doesn't seem to be a way to get into the 96gb + (vram or unified) territory under $4k, right?

Got tired of WebGPU browser tabs crashing when models exceed maxStorageBufferBindingSize or lack shader-f16 support.

Built browser-llm-fit to probe client hardware limits and rank browser-executable models before downloading weights.

import fit from 'browser-llm-fit';

const res = await fit('SmolLM2-135M');
console.log(res.fits, res.speed); // true, '45-65 tokens/sec'

fit('model') tests a model. fit() returns all models sorted by hardware fit.

• Live: https://h3manth.com/ai/browser-llm-fit/
• Repo: https://github.com/hemanth/browser-llm-fit
• npm: npm install browser-llm-fit

Feedback on odd GPU setups and mobile WebGPU is appreciated!

Continual learning isn't some fake term but a real goal and arguably one of the most important open ones in 2026. It roughly means: ‘building systems that keep acquiring useful knowledge or skills after deployment without a full retrain’.

IMO, the problem is that it's a problem setting being used as if it named a mechanism. When I talk to someone who says "we do continual learning," I now have to guess whether they mean gradients running in prod, a markdown file the agent appends to, a 5M-token context or a nightly distillation job. 

Here's how I'd break it down: long-context ICL, text optimization, recurrent latent memory, per-task test-time training, and online parametric continual fine-tuning

1. long-context ICL

Weights stay frozen and the model learns the task from context optimization, i.e., by conditioning on demos, feedback, and history in the current context. The idea is that a sufficiently large context window could provide enough ICL for an agent to learn a task without fine-tuning.
Key limitation: a very large working memory and no transfer from current context to long-term memory.

2. text optimization over the mutable text layer

The system rewrites the mutable text layer around a frozen model: system prompts, skill files, playbooks, memory stores, retrieval indices, harness code.
Important tradeoff: forgetting doesn't vanish here, it moves from weight interference to memory construction and retrieval, where old and new experiences still compete for a bounded context.

3. recurrent/architectural latent memory with frozen weights

Task info gets written into an evolving internal state not a growing KV cache or some text file and parameters stay fixed.
Multi-timescale self-modifying architectures sit here. BDH-CQ is another clean illustration of the pattern: each demonstration from the train-test set is integrated into a recurrent memory that gradually builds an internal representation of the task. This is the starting point for reasoning about test-test inputs in a separate reasoning loop. The model thus adapts at inference time through state updates alone, without modifying its parameters 

4. per-task test-time training with gradients

The system turns demonstrations into a small training set, performs gradient updates for the current task, generates an answer, and may then discard those updates.
Some ARC pipelines (including the evaluated HRM/TRM setups) use this kind of task-specific optimization. I’m not sure it is truly “continual” if nothing persists across tasks, but it frequently gets grouped under that label.

5. online parametric continual fine-tuning

Gradients are applied persistently after deployment. The main challenges are finding good labels/rewards at test time and learning new information without destroying existing capabilities, using techniques such as replay, regularization, parameter isolation, or sparse and targeted updates.
Some self-editing systems are hybrids: the model produces its own fine-tuning data in text, but the resulting update is stored in weights. Imp tradeoff: catastrophic forgetting and accumulated weight updates may not survive a base-model upgrade. It’s also hard to find proper signals from which we can back-propagate at inference time.

That’s how I have split the term and I’m curious where people here disagree and if I missed any.

Here is the news you may have missed:

Nvidia already demonstrated 100% on ARC AGI-3, using their novel harness AVO:

https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-arc-agi-3-demonstrating-a-frontier-level-general-purpose-architecture-for-long-horizon-autonomous-agents/#

OpenAI didn't use the standard harness in the ARC AGI-3, but their own.

I built a benchmark that measures whether your local LLM stack can actually build something, not just generate tokens. It caught failure modes that llama-bench, PPL, and every static check I ran were structurally blind to. It's public, it runs in ~25 minutes per cell, and I want your numbers in the comments.

What it does

A coding agent builds a 10-file HTML5 canvas game from a fixed contract. Then three gates grade it: static checks score the code, a headless browser soaks the running game for two minutes, and you playtest it. The benchmark subject is the whole stack — engine, quant, drafter, speculative decoding, chat template, contract, effort level — not just the model.

What it caught that nothing else did

Twelve gameplay-failure classes so far. Every single one found by a human playtest, zero by static score:

A ball that fires at 7× speed because the velocity math multiplied by its own magnitude twice. A ball that vanishes mid-game in a build that scored 17/19 static and passed its runtime soak. A menu that ignores Enter because the keydown handler was wired but the model never re-read its own state machine. A cascade that clears every brick on the first hit because the "explosion" function recursed without a visited set.

These are not hypothetical — each one shipped in a build that passed node --check, matched every static pattern, and rendered at 60fps.

Two "architectural blind spots" cured by one sentence each. A quant tier that auto-launched the ball on every roll (3-for-3) until an explicit SERVE RULE in the contract fixed it. A model that failed the same physics subsystem on every roll (3-for-3) until an explicit PAD PHYSICS clause fixed it. Both times I thought I'd found a model limitation. Both times it was a spec gap.

A stack that benchmarks beautifully and still can't build. I ran two community stacks on the same model, same contract, same effort. One produced try-1 successes at 17/19. The other went 0-for-15. The difference was invisible to every standard benchmark — the failing stack decoded at the same speed, passed the same checks. Only the build workload saw it.

Model size buys speed, not quality. At matched effort and environment, a 125B MoE and a 27B produced identical scores, identical failure sets, and identical line counts (1,412 each). The MoE got there in 7.8× less wall time. The 27B spontaneously wrote and ran its own smoke test mid-build. On explicit contracts, pick by token budget, not quality assumption.

Run one cell (~25 min)

```bash git clone https://github.com/aic0d3r/neon-ladder && cd neon-ladder

start your llama-server (reference configs in the README)

then:

bash run.sh build-run1 game-run1 medium "$(cat contract.txt)"

... the runner gates it, and on success prints your result line

playtest:

open build-run1/index.html, press Enter, play two minutes

QUANT="your-quant" DRAFTER="your-drafter" PLAYTEST="Y or N + what you saw" \ RIG="your hardware + engine" bash report.sh build-run1 game-run1 ```

Post one comment

quant / drafter / effort / wall / static (x/19) / soak / playtest Y-N — rig + engine

Three real examples from my runs:

UD-Q4_K_XL-v3 / DFlash2-Q4_M n4 / medium / 16min / static 15/19 / SMOKE-OK / Y — plays great — Strix Halo, Nathan v0.7.3 UD-IQ4_XS / MTP Q8_0 n4 / low / 5min / static 17/19 / SMOKE-OK / Y — most interesting build — Strix Halo, Nathan v0.7.3 UD-Q4_K_XL-v3 / DFlash2-Q4_M adaptive n3-7 / medium / 21min / static 16/19 / SMOKE-OK / N — ball disappears mid-game — Strix Halo, Nathan tip

That last one is the release-gate build — 17/19 static on a later rescore, passed its soak, and the ball still vanishes when you play it. That's why the playtest is the grade.

The repo

github.com/aic0d3r/neon-ladder — contract, scorer, runtime gate, runner, result-line generator, one-comment recipe. Everything versioned, everything reproducible.

Full numbers and methodology: my 27B stack guide, my [Flash-Next post](link-to-come), and the [pi agent-setup guide](link-to-come).

Your runs are the next cells.

It started as a fork of llama-swap, but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing.

The main idea is that you point it at your models folder and it configures things for you. It reads the GGUF headers, measures how much VRAM you actually have free, and works out context length, GPU offload, CPU/MoE split and KV cache size per model. All of it stays editable per model if you disagree with what it picked.

It is not only text. llama.cpp for LLMs, with the Vulkan, CUDA, ROCm or CPU build downloaded and kept updated for you, stable-diffusion.cpp for images (SD, SDXL, Flux, Qwen-Image, LoRAs, upscaling), and vLLM if you already have it set up. You can register any other backend yourself by pointing at an executable, which is how I run TTS, and how you would run a llama.cpp fork like ik_llama. Everything sits behind one OpenAI-compatible API on one port, with a single scheduler, so models swap in and out without fighting each other for VRAM.

There is also a chat playground built in with web search, and a Hugging Face browser to search for a model, pick a quant and download it straight into the models folder and much more!

If you are interested, you can read more about it here. MIT licensed.

I start with some informations gathered thorough endless posts reading on this sub and online:

Inference and hardware optimization projects

• https://dwarfstar.sh/ - inference engine optimization proposal
• https://github.com/JustVugg/colibri - Treats VRAM + RAM + storage as one managed inference memory hierarchy, with expert streaming, caches and a strong focus on MoE.
• https://openfreedom.it/ - agentic harness proposal
• https://github.com/exo-explore/exo - Core source for heterogeneous/topology-aware Mac clustering
• https://github.com/ml-explore/mlx/discussions/3481 - JACCL real-world TB5 transfer experiment
• https://github.com/georgiedekker/mlx_distributed_ring_inference - distributed inference through TCP/Ring over TB3/TB4, without requiring RDMA
• https://github.com/sqliteai/warp - WARP — Weight-Aware Runtime and Paging
• https://github.com/kqb/mlx-od-moe - on-demand experts on Apple Silicon: memory-mapped expert storage, shadow predictor, prefetcher and LRU
• https://www.houmo.cn/1/35/NewsDetails.html - Houmo is developing DRAM-PIM with compute embedded in DRAM arrays, targeting >1 TB/s internal bandwidth and ~3× energy-efficiency improvement over its current generation.
• https://www.d-matrix.ai/wp-content/uploads/2024/11/d-Matrix-WhitePaper-Technical-FINAL.pdf and d-Matrix 3DIMC announcement - 3D DRAM + digital in-memory compute, highly aligned with the “move compute toward weights rather than weights toward compute” idea
• https://www.lucebox.com/ - workstation optimized for local AI use for approx. 7000$

Inference Research papers

• MDI-LLM - Model-Distributed Inference for LLMs at the Edge Model partitioning across low-power nodes and recurrent pipeline parallelism to reduce device idle time. MDI-LLM paper
• WDMoE - Wireless Distributed Mixture of Experts Distributes experts across edge/mobile devices and jointly optimizes expert selection and communication latency. Includes a physical NVIDIA Jetson testbed. WDMoE paper
• OD-MoE - On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference Very relevant to our expert-prediction idea. Uses a predictor to forecast experts several layers ahead and loads them just in time across distributed nodes. Reports 99.94% expert-prediction accuracy and about 75% of fully cached decoding speed while using one-third the GPU memory in its tested setup. OD-MoE paper
• MoE-SpeQ - speculative decoding + proactive expert prefetching Almost directly relevant to the question we uncovered around streamed MoEs. A draft model predicts future experts so their transfer can overlap computation. Reports up to 2.34× over its offloading baseline. MoE-SpeQ paper
• SP-MoE - speculative decoding and prefetching for MoEs Speculation-aware expert offloading, speculative expert prefetch, asynchronous batched I/O and compute/I/O pipelining. SP-MoE paper
• MoE-Spec - Expert Budgeting for Efficient Speculative Decoding Important counterargument to “speculation automatically fixes MoE.” Shows that verifying deeper speculative trees can activate too many unique experts, increasing memory pressure; proposes explicit expert budgeting. MoE-Spec paper

WTF

They really cooked.

https://simple-bench.com/

I’m a primary school teacher currently building out a local AI setup to help with my workload, and I’ve hit a bit of a wall. I’m looking for some advice on whether I’m approaching my workflow correctly or if there’s a better way to structure my models and tools.

My Setup:

• Main Workstation: AMD Ryzen 5 5600X, 64GB RAM, dual GPU setup (an 8GB card for display/dictation and a 16GB card dedicated to running my LLMs).
• Infrastructure: I run a small lab with a few NAS servers and two ThinkCentres. One acts as my main orchestrator and the other is my "school" node.
• Workflow: I use Tailscale to sync everything to my work computer. I have a massive personal knowledge base (vaults) containing scanned books, lesson plans, and student notes.
• Creative/UI: I use ComfyUI for image generation to build classroom resources.

What I’m doing with AI:
I use a combination of dictation and LLMs to streamline my admin. For example, I’ll dictate notes about a student's progress; the AI processes this, writes it into a specific Markdown file, and I have a system that then routes that note to the correct student file. I also use AI to help build web pages for interactive classroom activities on my smartboard.

The Problem (The "A/B" Wall):
I am trying to move more towards open-source/local models to handle these tasks. To troubleshoot, I have set up an A/B testing system: I’ll give the exact same prompt and context to both a closed-source model (using Luna/ChatGPT) and my local model to see how they differ.

While Luna handles the task perfectly, my local models (even when I've squeezed a 27B model into Q4) frequently fail. Specifically:

• Tool/Skill Failure: Even though I believe I’ve defined my "skills" (MCP servers/tools) correctly, the local models frequently pull the wrong tools or fail to trigger them at all.
• Logic Drift: The models often "break" the logic of my lesson plans or deviate from the structure I've provided. They often go in directions that even Luna wouldn't take, even when the context is identical.
• Instruction Following: It feels like the models aren't respecting the boundaries of the skills I've built, making them unpredictable and unreliable for my daily classroom workflow.

My questions for the community:

1. Function Calling & Tool Use: If my MCP servers/tools work for one model but fail for others, is this a known limitation of smaller/quantized models? How can I refine my tool definitions to be more "robust" for local LLMs?
2. Instruction Following: Am I missing a specific way to prompt or structure my system instructions to prevent the model from "wandering" away from the lesson plan or the intended tool?
3. Model Recommendations: For a workflow that relies heavily on precise tool use and following complex, multi-step instructions, which local models are currently the "gold standard" for reliability?

I'd love to hear from anyone else using local LLMs for professional organisation or education. Am I overcomplicating it, or am I just missing a key piece of the puzzle?

Would it be feasible to buy a refurbished 8xa100 server, either rent it out on vast.ai, or serve a model via a similar service per token in that exists.

I’d likely have to either put 1600-200”0w of second hand solar on my shed, add fire suppression and a rack, or colocate with a reputable data centre.

The idea would be to recoup the initial purchase, and then transition to serving myself. Ideally I’d sell tokens not gpu time because that would allow be to use the machine while it’s being monitored, but that market seems way harder?

Managed to get this small model to run on the $250 MSRP SoC board level computer.

The inference speed is kind usable. Used 4-bit quant, q8 kv cache, 7.4 GiB memory supports 128K context length. Device tops at 25W power, and idle less than 10W. Quite suitable for a simple agent running 24/7.

Needle in a haystack test pass at 128K context length. 2046 needles passed out of 2048 needles.

- **2048-needle (fully random unique word+number pairs, seed 20260902): 2044/2048 (99.8%) @ 90K prompt**, finish=stop (no truncation), 4 misses (2 partial word-only). u/120K prompt: 942/2048 but truncated by the 128K KV ceiling (120,287 + 10,785 = 131,072, finish=length) — misses 99% in the 50–100% depth bands, i.e. unanswered tail, not retrieval failures. 90K is the effective ceiling where the full 2048-pair answer (~23K completion tokens) fits.

- **llama-benchy (pp2048/tg512, 3 runs, in-bench coherence check passed):**

| conc | pp tok/s | tg agg tok/s | tg per-req tok/s | ttfr ms |
|---:|---:|---:|---:|---:|
| 1 | 571.7 | 13.7 | 13.7 | 3,857 |
| 2 | 384.9 | 22.6 | 11.6 | 9,653 |
| 4 | 373.5 | 26.4 | 7.1 | 17,436 |

I liked the Nvidia that focused on just GPUs for gaming, not on the Nvidia of today which seem want power consolidation.

Modelscope is another platform for those that simply want to know an alternative if things go south. However, time will tell what happens to huggingface after the deal is finalized

Link: https://modelscope.cn/home, and https://modelscope.ai/home

It's been 3 weeks since Qwen3.8-27B release. This model got 0-day support & in last 3 weeks, some optimizations & fixes happened on llama.cpp side.

Meanwhile

• DFlash2 support landed on llama.cpp last week.
• Ubuntu 26.04.1 got released.
• ROCm 10.0 got released & llama.cpp also up with Version 10.0 now (Applicable to only AMD cards)
• Other optimizations & fixes on llama.cpp side

So how much t/s are you getting now with all optimizations & stuff?

Please share your extreme optimized full llama.cpp command & your t/s stats .... both pp & tg(Good to have multiple combinations like MTP/MTP+ngram/DFlash2/etc.,, Vision/mmproj, multiple context size 128-256K, etc.,).

Also share your optimized build config(CMAKE command) if you're compiling manually. I remember that compiled version gives additional boost.

Note : Expecting to see optimizations like this(weeks old thread) which contains all stuff. That kind of stats want to see here.

The job of doing inference is not the same for every model ou there, as it depends on the architecture and configurations and the available machines. Many times we would not have the right machines to test out a new model, or learn about new algorithms , like speculative decoding etc. What's the best way for learning?

TLDR; Is there a hackernews for LLM inference?

Which KV cache do you use F16 or BF16 for Q6 quant? What is the difference between them?

I just think it’s funny to observe myself, scrolling through eBay looking for GPU deals to feed my AI addiction. Some things are obviously scams, but it’s just so fun to actually look and compare prices and performance and then you forget that you actually only have $6 in your bank account because you just spent money you could have saved on a different GPU last week. (I’m joking of course) but this is what the experience feels like to me. It feels almost like gambling, at least the thrill is there. Will the value of my GPU keep going up? (Also of note, I’m not actually that irresponsible, I own a B65 gpu that I just got a few weeks ago, other than that just a few v100’s that I’m not using yet). Do you guys feel the GPU/local ai thrill as well? What’s the experience like for you?

We have been working on an open-source, model-neutral agent harness for general purpose agents called TrueForge, and wanted to understand how much the harness itself actually matters.

So we ran 14 tasks from DevRev Enterprise-Bench through multiple harness/model combinations, three times each with a blind judge.

The result that surprised us most:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But TrueForge used about 63% fewer tokens and cost about 30% less per run.

We saw a similar difference in tool usage: TrueForge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

The difference comes from the agent loop itself: less context carried between turns, compaction, fewer tool calls, and large outputs being kept out of the model context where possible.

Then we tried changing the model.

TrueForge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

For me, this is the more interesting consequence of keeping the harness model-neutral.

You get two independent levers:

1. Make the runtime more token-efficient.
2. Use whichever model gives you the right price/performance for the workload.

TrueForge itself is fairly simple: it handles the agent loop, context management, tools/MCP, subagents, approvals, persistent sessions, and sandbox integration. It is MIT licensed and works with OpenAI-compatible endpoints, so you can point it at hosted models or models you are running yourself.

This is still early.

The OSS runtime does not yet have first-class tracing/eval tooling. We don't ship our own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So I wouldn't claim that TrueForge replaces a mature managed agent platform feature-for-feature today.

What I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

We put the benchmark harness and methodology in the repo specifically so people can reproduce it, change the models, or tell us where the comparison is unfair.

Repo: https://github.com/truefoundry/trueforge

Benchmark methodology: https://www.truefoundry.com/blog/engineering/trueforge-vs-claude-managed-agents-benchmark/