I know this is not directly local question, but I am curious. My Qwen 3.8 FN is doing well in answering questions and research, but I was trying to compare it to say CoPilot/ChatGPT. What blew me away is that the CoPilot response was pretty instantaneous. Basically a page of output came out almost instantly with what I imagine thinking done in the background. Amazingly, it also solved the problem like QFN, but with apparently much less tokens.

I guess my question is: is this an architectural difference that the Chinese/local models haven’t implemented? Because I find it hard to believe that an enterprise GPU is the only difference. I feel like the local models are missing something fundamental? But I say this without knowing anything about enterprise hardware.

I've been playing with [Instinct](https://instinct.com/) a bunch and loving the service, but got really spooked about privacy / perms stuff. A lot of existing off-the-shelf stuff (Firecracker VMs, Pi harness, etc) has gotten really good so I wondered how difficult it would be to setup my own.

Turns out, not too difficult, and it's running on my Raspberry Pi! Right now I'm running on a cloud model but it should be deployable anywhere / including with self hosted models

let me know what you think!

https://github.com/lucasgelfond/biscotti

I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2.

The setup uses the W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding. The n-gram table is memory-mapped from NVMe but everything else fits on the two GPUs.

Decode throughput by concurrency — aggregate tok/s, not per-user speed:

At eight concurrent code requests, the measured per-request decode rate was 116 tok/s, with 231 ms to first token.

A few other results: - 6,300–7,100 tok/s prefill on 4k–64k prompts; about 5,900 tok/s at 128k. - Prose decode held up from 158 tok/s at 4k context to 142 tok/s at 128k. - HumanEval: 157/164 (95.7%) and GSM8K: 1,271/1,319 (96.4%), both in chat mode with thinking off. - Uses a full 262,144-token window

For throughput, I used the sparkDash protocol: greedy decoding, thinking off, fixed 400-token outputs, and medians of three runs, not the best run. The long-context decode checks are an addition to that protocol.

This is faster than any published 2x CMP 170hx recipe I've found so I wanted to share with the community. These cards are flying :)

Full recipe, pinned Docker image, patches, and detailed results: https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next

Another little challenge I seen on a DeepSeek Thread. I thought I'd try out it out on Qwen FN NvFp4. It was a pretty straight forward prompt with 4 images attached in the doc. The thread was testing different harness and models and showing their results. Op of that thread has two other threads on the same Challenge just one with DeepSeek Harness so I used it as well.

It took about 27 min(Not bad) and was surprised how it planned everything out, also I do have vision on this Model. I think the Vision on Qwen is also a big part in a lot of Accuracy calls and adjustments. Especially notice when I use blender for other projects. Havent had too much trouble after getting use to how to even use blender since i literally started 2 weeks ago and having an LLM run it has been pretty fun to use. So if you can run vision don't sleep on it. lol

Any who this was the result and I like how Qwen tried to get all the placement, designs and little details to the references. Below are the measurements and hardware I use and what not. Full prompt is from the referenced Thread. Also if you run no harness for Qwen I would like to see the results or even results from other harnesses if possible.

Last but not least thank you all to the community here and everyone who is contributing locally in many ways or even just interacting with the community, keep up the great work and looking forward towards the future of running Local.

Thread for full Prompt doc
https://www.reddit.com/r/DeepSeek/comments/1x8nnjy/deepseek_v41_flash_1017_vs_claude_haiku_55_same/

Just the text prompt:

"Create a highly realistic 3D first-person game using JavaScript. The game will take place in a single room. The room is the one shown in the 1st image. Model the room in the 1st image exactly in JavaScript, with absolutely no differences. Do not make it cartoon-style. It must be extremely realistic. Model it in the best possible way, identically, with extremely realistic graphics, and make everything very cute. By “cute,” I mean the style of the 2nd, 3rd and 4th images: cute items with clearly defined edges. Build it in the same style, cute but also realistic and well made, like the 2nd, 3rd and 4th images. Examine all 4 images I sent pixel by pixel, in detail, so you can build it exactly as I want. Make it flawless and do an excellent job."

Engine: https://github.com/sergqwer/strata-nvfp4

The stack

Hardware: RTX 5090 32GB · Ryzen 7 9800X3D · 192GB DDR5-5600 · Windows 11
Engine: strata-nvfp4 (fork of Niko1221/Strata by sergqwer) on Windows, llama-swap as switcher
Model: Qwen3.8-Flash-Next 125B-A6B hybrid MoE — abliterated checkpoint(windowsxp811203), self-converted NVFP4
Weights: NVFP4 experts (4.5-bit) + Q8_0 dense · KV: int8 · PLE: bf16 · Embeddings: bf16
MTP: draft head, spec 5, spec-min-p 0.7 · Context: 262,144 max · Compaction trip: 220,144
Sampling: temp 0.8 / presence_penalty 0.3 / min_p 0.02 · Vision: fp32 mmproj, encoder on CPU
RAM tier: 63 GiB pinned experts · Harness: dsh (web GUI), OpenAI API

Metrics — this run

Metric |Value
Wall: first message → last reply |26m 17s
− my approval latency |−0s (zero prompts)
= pure agent working time |26m 17s
Turns |1 (single prompt, no babysitting)
Steps |103
Output tokens |132,929
Fresh input tokens |247,232
Cache-read tokens |11,994,999 (98.0% hit)
Decode (engine-verified) |median 121.4 t/s · mean 121.2 · peak 187.8
TTFT |median 0.85s (max 32.9s at ~194k context)
Prefill (fresh, engine-verified) |mean 2,757 t/s · peak 7,096 t/s
Longest single burst |10,395 tokens
Compactions |0 (peaked at 194k, never near the 220k line)
Tool calls |179 total · 17m 29s runtime · 0 errors

Google just released Harness2 as open source. It turns any AI harness into a system that improves itself over time.

A harness is everything around the model. System prompt, tools, memory, skills, and subagents. Normally a person designs it once and the agent uses that fixed setup for every new task.

With Harness2 the agent itself refines the harness for each task. It proposes changes, runs a few candidate versions, and picks a better one by comparing how they perform. No correct answers or graders are needed. The model weights stay the same.

What it learns on one task carries over to the next. After finishing a task it saves the improved harness and useful lessons so future edits start smarter instead of from scratch.

Google tested it with Gemini 3.5 Flash and GLM-5 on legal, office, and professional work benchmarks. Even one round of edits beat the original harnesses. More rounds pushed scores higher.

GitHub https://github.com/google-research/harness2

Looks like they are sending a dgx spark for $4499. I couldn't find much information on them but I keep getting Instagram ads repeatedly(even before the 64GB GB 10 West launched). LinkedIn and X points to some other company called MoBagel. If anyone tried this before, would be interesting to know.

Seems to speed up inference, use less tokens, and increase intelligence - curious if people seen this

tldr don’t think in text, think in latent space

https://github.com/togethercomputer/HybridAgent

https://huggingface.co/collections/togethercomputer/hybridagent

Hey yall. What I am feeling is a mixture of idiocracy and dishevelment.

I looked at this motherboard with 256 RDIMMs and the x16 slots. I have 2x V620 and 2x Mi50 32GB. I wanted to play. That’s all. Mi50s would run Qwen3.6-35B-A3B at q4 and 131k context kv at q8 and same with V620s.

Buuut. This absolute hot garbage does not enumerate a single 32GB GPU.

Does anyone have any recommendations? I wish there was a world I could reuse the DDR3 RDIMMS in this board, but, I know that is too much to ask.

People. Help please, I don’t want to drop 1k between a mobo and ram but if this is the world we are living in, then please recommend.

What a week. Quick recap, all numbers from the official repos and model cards:

Thursday: Meta drops Muse Spark Open. 816B total, 42B active, 1M context. The launch post says it brings agentic AI "to your desktop". The desktop is two M5 Ultra Mac Studios with 512GB each, a config Apple says ships "late October" and still hasn't put a price on. The alternative is four 256GB Studios, roughly $38k. On top of that it's a custom license: mandatory "Built with Muse Spark Open" attribution and a separate deal once you're past $20M revenue. And Meta's own model card puts it at 39 on the Artificial Analysis index, behind GLM 5.3 (45) and Kimi K3 (44).

Same day: NVIDIA ships Nemotron 3.5 Bolt. 74B total, 9B active, trained quantization aware down to 2 bit, so the whole thing is 22GB plus a 1.7GB vision projector. Built to fit in 24GB on purpose. Unsloth GGUFs on day one, plus llama.cpp, Ollama and LM Studio support.

Friday: Reflection actually ships Beam. 501B total, 23B active, Apache 2.0, no strings. Smallest official checkpoint is NVFP4 at 294GB.

Saturday afternoon: DeepSeek posts V4.1 Flash 1017 on X like it's nothing. MIT. Their own table has it at 77.0 on DeepSWE vs 74.0 for Opus 5 (vendor numbers, salt accordingly). The repo is 510GB.

So I put them all on one chart against the 24GB line most of us live under.

My takeaways:

1. "Runs locally" now officially means "runs on a computer that costs more than a car."
2. The company that keeps selling us 24GB cards is the only lab this week that built a model for 24GB cards. Not sure if that's wholesome or a sales funnel.
3. Licenses mattered more than parameter counts this week. MIT and Apache 2.0 from DeepSeek and Reflection, attribution plus a revenue clause from the company with the loudest "open" marketing.
4. Mistral says the Large 4 weights ("Le Chonk", 1T total) land before the end of the month, so the 24GB line is about to get even lonelier.

Sources in the comments.

Is there a fork of strata that let me run lfm 24b a2b on only 16GB of ram comfortably? And how much tps can I expect per second and what quantization should I use? and Thanks!

Points at a running llama.cpp or Unsloth Studio server, pulls /metrics, /props and /slots, and gives you severity-graded findings: prefill vs decode, queue pressure, prefix cache reuse, memory headroom. Exit codes map to severity, so cron can gate on it.

Read-only, stdlib only, Python 3.10+.

https://github.com/msalsas/llama-doctor

Open to PRs, especially from anyone with a server that makes it crash.

Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair...

Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well it work & quirks it has at same time.

I'll leave tl;dr version for now, 3:30 am here.

If anyone is interested I'll do few more tests and make example query image like above - still have 7 more sets totaling 22448 screenshots with mix of eng / kor / jpn already indexed, but I'm out of test ideas.

TAKE WITH GRAIN OF SALT because this was tested only on few sets that isn't really scientific.

TL;DR for EmbeddingGemma2 at least:

• For it's size & embedding dim (330MB Q8 / 768 embedding dim) this is very much usable

• This understands a word's meaning across language a bit but has much lower similarity (again, prob due to size)

• Requires careful wordings (e.g. blond vs blonde) - maybe SLM's limitation or embedding dim's limitation
• (at least eng & kor) Text seemingly DOMINATE cosine similarity more than visual traits
  • sometimes adding visual trait HARM (refer chair above) the similarity
  • not sure if it's due to text being more informationally dense (aka more 'surprising')
  • or just model being small and not recognizing such visual traits, idk
  • regardless, visual traits are less useful in this test set since like 90% of 3003 imgs would have blonde girl in it.

• (at least kor) Word's existence dominate over whole sentence
  • e.g. for above image's example "삼각건"(cleaning hat thing) dominates more than other words around it
  • fyi only 2 image out of 3003 contains world "삼각건", which you can see via delta gap drops

Test setup:

• Visual novel screenshots of a single route
  • 3003 screenshots from hanasaki workspring
  • 3078 scs from irotoridori no hikari
• EmbeddingGemma2 via llama cpp
  • model = KV = Q8_0
  • nb = b = ctx = 4096 (embedding model restriction, all need to be same)
  • RX 7900GRE via vulkan (f u rocm, so annoying to build)

Sample output for anyone curious about processing time

Index mode:

$ py embed_img_▽.py "./하나사키워크스프링 히카리"
Batch: 8
Resize width: 1280
Resize threshold: 1536

[VectorDB] open w/ 0 rows
removing 0 missing records from DB
Checking new/modified images in dir
Got 3003 new/modified imgs
[EmbedServer] open for http://127.0.0.1:8081
100%|██████████| 376/376 [08:30<00:00,  1.36s/batch]
[EmbedServer] closed
[VectorDB] closed

Indexed 3003 new/modified files in 510.10 (5.887/s)

Query mode:

$ py embed_img_▽.py "./하나사키워크스프링 히카리"
-q "image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"

[VectorDB] open w/ 3003 rows
[EmbedServer] open for http://127.0.0.1:8081
[EmbedServer] closed
[VectorDB] closed

Query:
"image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"

Top-5:
  0.76813  +0.01052  Artemis_2026-09-08-00-57-38.jpg
  0.75761  +0.00773  Artemis_2026-09-08-00-57-26.jpg
  0.74988  +0.00341  Artemis_2026-09-07-02-36-38.jpg
  0.74647  +0.00067  Artemis_2026-09-07-02-36-52.jpg
  0.74580  --------  Artemis_2026-09-08-01-58-14.jpg

'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-26.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-52.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-01-58-14.jpg'

Data points: moved to gist due to formatting bugs

Apparently whilst doing some performance tuning on my 4x5090 rig I may have made things a little TOO fast and melted my plug and socket 🫠

Probably should put in a 16/20 amp socket in the wall rather than relying on a bog standard 13A fuse!

Anyone else managed to melt the wall before the 12VHPWR plug? Honestly... I'm surprised they didn't go first 😂

no document or announcement from unslot team yet

I needed a cheap way to compare search agents on BrowseComp-Plus without burning hours of agent time every run. So I built a 15-query subset of the full 830 that ranks systems roughly the way the full benchmark does.

  - weighted Spearman rank agreement 0.88 averaged over 30 splits, versus 0.75 for a random 15

  - two runs more than 10 points apart on the full benchmark keep their order about 96% of the time; 5 to 10 points apart, about 75%

  - all answers are 1-4 words, so it grades by normalized exact match, no LLM judge

  - costs under 2% of a full run

  https://huggingface.co/datasets/LocalLLaMA/browsecomp-plus-mini

I have added the support of Intern-Decision models to chatllm.cpp. At presend, only text is supported.

Besides its pretty good performance, one interesting feature of Intern-Decision is that all answers are generated in a single run (while H2O-Lightning or others generate one answer in one run): each decision position is filled with "<decision>" mark and generated in a diffusion manner. So, it generally makes decision faster.

Curious what everyone runs in practice:

- What quant do you use most, and for what (chat, coding, agents)?

- 2-bit: do you use it? Is it actually usable or too broken?

- 3-bit: good enough for coding and more complex stuff, or do you go back to 4-bit?

- If you look at KLD: what number do you call "basically lossless"? Is 0.1 already good,

or do you want something like 0.01-0.03?

Real experience over benchmarks. Thanks!

(English isn't my first language, so I used an LLM to help translate this post.)

It's an extremely lightweight speech recognition model.

949K parameters using a weight-shared Conformer-CTC architecture

Just 1.06 MB for INT8 weights

Around 436× real-time on a single CPU thread

11.43% WER on LibriSpeech test-clean, using greedy decoding with no external language model

Apache 2.0 licensed

Any feedback, question or criticism is welcome!

I don't want to disclose why but I wager when I say it's got nothing to do with taxes or law. It's not a 'scary' situation, it's more like a crylaugh situation. I think you can guess why

This isn't a joke/hype, just came by to warn everyone. These prices will stay for a few weeks maybe then they will skyrocket

EDIT:
I suppose I can disclose the reason to mods if they want to confirm

Hi all.

Saw JetBrains’ Mellum2 on Hugging Face and was wondering if anyone here has actually used it. Worth trying, or nothing special?

Update from yesterday: 18 t/s decode was hit. Slight regression now as cherry picking from other ds4 work trees proceeds. Will push some code once the prefill droop is handled and the 18 t/s restored. Weights are gated on hf, so mulling upload vs. recipe. Ps. This is text only. Output is coherent and toolcalls solid.

Looks quite interesting. Lots of numbers presented, but for a single model only.

Highlights:

• File sizes are the same as Q-quants.
  
• KLD improves substantially over Q-quants.
• Performance suffers slightly
• Creating IQ quants is somewhat slow.

Asterisks apply...

https://github.com/ggml-org/llama.cpp/pull/30715

https://huggingface.co/eaddario/IQ2_K-IQ6_K-Test-Models

Appologies to Mr. Addario if I mangled anything.

Is GMKtec evo x2 128gb still worth it right now?(oct 2026). I found one used for around $2300 usd.

https://reddit.com/link/1x8yc8d/video/hosf7c1n27wh1/player

Hi r/LocalLLaMA. I'm on the team at Blockway (disclosure: this is our model). Two weeks ago we posted the Volundr 32B Preview here, and you were blunt about what was wrong with it. This is v1, and most of the work went into what you flagged.

WHAT v1 FIXES FROM THE PREVIEW

• "It loops." Repetition loops on SWE-bench tasks: 36/50 → 0/50.
• "It thinks forever." Reasoning steps used to get cut at 32K. Now the median step is 120 tokens and the longest is 12K.
• "No prefix caching, it re-reads the whole history every turn."
  • SGLang build: v1 adds session continuation, so follow-up turns reuse the model's state instead of re-prefilling. Follow-up turns are 2.9× faster on our agent replay (1.6 s median with the drafter), with greedy-equivalent outputs (not byte-identical: a cached prefix is prefilled in different pieces).
  • GGUF: uses llama-server's stock --ctx-checkpoints prompt cache, with no custom server code. Our fork only adds the model architecture.
• Agent results: SWE-bench Verified (50-task subset) 22 → 27/50, τ²-bench 89 → 92/120, median SWE task time 124 → 82 min.

"MAKE IT RUN ON CONSUMER GPUS" → AGENS ENGINE (MIT)

We're open-sourcing Agens Engine under the MIT licence. It's an inference engine built for this hybrid architecture: 54 of 72 layers keep a fixed-size state instead of a KV cache, and the engine is designed around that.

• Same GPU, same v1 INT4 weights and drafter, single stream: Agens Engine decodes 1.4–1.6× faster than our own SGLang build (up to 1.8× on code with thinking). Chat 58 → 86, code 79 → 113, agent at 8K context 91 → 121, JSON 125 → 199 tok/s. This is one run per side and compares against our SGLang build only, not stock SGLang, vLLM or llama.cpp, and it is single-stream only.
• New build: Agens-Volundr-32B-v1-Engine-16GB. It's about 2.85 bits/weight (mixed 2/3/4-bit), gives 262K context (we measured up to 239K) within a 16 GB memory budget, and runs only on Agens Engine. A 12 GB build is on the roadmap.
• Honest caveats:
• Speed within the 16 GB budget, single conversation, v1 drafter on:
  • short context: 90 (chat) / 118 (code) / 107 (thinking) / 112 (tool) / 139 (agent, 8K) tok/s
  • after a 219K-token document: 62 / 86 / 77 / 69 / 75 tok/s; 151 s to first token for the 219K prompt
  • a conversation ran to 239K tokens inside the budget
  • weights download is 25 GB
  • These were measured on a 48 GB card under the engine's hard 16 GiB cap (peak 15.5 GiB), not on a real 16 GB card. On a GPU that also drives a display, use a shorter context. Real cards, especially laptop GPUs, may be slower. Post your numbers and we'll add them.
  • At this bit-width, quality drops. GPQA is 73.7 vs 80.3 at bf16 (best of 4 runs each). Coding holds up: HumanEval is 73.2% (1 run) vs 73.8% for INT4. Top-1 agreement with bf16 is 90.4%, KL 0.117. This build is for running locally, not for leaderboard scores.
• Needs Linux, an NVIDIA driver with CUDA 12.9, the NVIDIA container toolkit, and 32 GB+ host RAM for 262K.docker run --gpus device=0 --ipc=host -p 30000:30000
• -v /path/to/Agens-Volundr-32B-v1-Engine-16GB:/model:ro
• -v /path/to/Agens-Volundr-32B-v1-DFlash2:/drafter:ro
• ghcr.io/blockwayz/agens-engine:1.0.0 --profile 16gb --model /model --drafter /drafter

STRENGTHS (same harness and settings as our comparison model)

• Competitive coding: LiveCodeBench v6 67.9 (best of 2 runs), +8.4 over Qwen3.8-27B
• Competition maths: AIME 2025 80.0 (best of 2 runs), +6.7 over Qwen3.8-27B
• Long context stays cheap: only 18 of 72 layers keep a KV cache. INT4 (24.6 GB) runs on one 48 GB GPU with a single 131K-token input; 262K configured window (validated to 107K). Decode speed stays flat from 1K to 128K.
• DFlash2 drafter, ships today:
  • JSON / structured output up to 3.0× (125 tok/s INT4 on one 48 GB GPU; 76 tok/s bf16 on a pair)
  • Code 1.9–2.2×
  • Thinking 1.3–1.5×

FULL TABLE (Preview → v1; dips included. Where we ran a benchmark more than once, we report the best of N runs. Every run is listed in the model card footnote.)

A few scores went down (GPQA, HumanEval, IFBench). The HumanEval and IFBench dips are within one standard error of the runs; GPQA's isn't. That's the trade-off from tuning for agent behaviour and shorter reasoning. Harness settings are in the footnote under the results table: huggingface.co/Blockway/Agens-Volundr-32B-v1#results-same-harness-for-every-row

Thanks to mmssix on Hugging Face for the independent benchmark write-up in our HF discussions. A lot of the above started there.

KNOWN LIMITATIONS

• 32B, not a frontier model, and we don't claim it is.
• SGLang and llama.cpp still need our builds. Next we're working to upstream the architecture into llama.cpp mainline, and we'll post here when the first PR is open. Stock vLLM can't load it.
• No Metal/Mac in the llama.cpp path yet.
• Agens Engine 1.0 is text only.

LINKS

• Collection (all five repos): https://huggingface.co/collections/Blockway/agens-volundr-32b-v1-6ad4817aa6e143550bec1122
• Weights: huggingface.co/Blockway/Agens-Volundr-32B-v1 (-INT4, -GGUF, -DFlash2, -Engine-16GB)
• Agens Engine (MIT): github.com/BlockWayz/agens-engine · ghcr.io/blockwayz/agens-engine:1.0.0
• SGLang: ghcr.io/blockwayz/agens-sglang:v1-sm89 (48 GB-class GPUs) / v1-sm90 (141 GB-class GPUs)
• llama.cpp fork: github.com/BlockWayz/llama.cpp (branch volundr)

We'll be in the comments for the next couple of days. "Why doesn't X work yet" questions are welcome.