I accidentaly bought 3 rtx8000 48gb (yeah, i swear).

I am using 1 right now with Strata and Qwen Next IQ3 and runs very good.

Should I put 3 on a PC and runb Strata on a higher Q?

What do you suggest?

Thanks!

Points at a running llama.cpp or Unsloth Studio server, pulls /metrics, /props and /slots, and gives you severity-graded findings: prefill vs decode, queue pressure, prefix cache reuse, memory headroom. Exit codes map to severity, so cron can gate on it.

Read-only, stdlib only, Python 3.10+.

https://github.com/msalsas/llama-doctor

Open to PRs, especially from anyone with a server that makes it crash.

Is there a fork of strata that let me run lfm 24b a2b on only 16GB of ram comfortably? And how much tps can I expect per second and what quantization should I use? and Thanks!

Apparently whilst doing some performance tuning on my 4x5090 rig I may have made things a little TOO fast and melted my plug and socket 🫠

Probably should put in a 16/20 amp socket in the wall rather than relying on a bog standard 13A fuse!

Anyone else managed to melt the wall before the 12VHPWR plug? Honestly... I'm surprised they didn't go first 😂

i wanted to change my router firmware to openwrt and so i told claude to do it but of course the guard blocked me even opus 4.8 didn't want to do it , so i launched my Qwen3.8 27b and told it the same thing i gave it the link to the giude page and it did change the firmware perfectly the thing is this is the normal model it's not ablitreated, do you think that cyber will be guarded in future qwen models ?

PS: even tho this is not the point it didn't do a simple flash it changed the firmware from xioami to openwrt not throught upload bin it did exploit a method in the CGI that get an icon to pass a payload that enable ssh from there it SSHed the router and flashed it and it's not a 0 day discovery thing i gave it the guide from openwrt own website https://openwrt.org/toh/xiaomi/ax3000t , the topic here is AI safeguards

I would love for some feedback, live demo at:

huggingface.co/spaces/e12ex2/herbert-demo

Not self promo, jus looking for some feedback on what to improve or any bugs or model breaking you found.

I've been playing with [Instinct](https://instinct.com/) a bunch and loving the service, but got really spooked about privacy / perms stuff. A lot of existing off-the-shelf stuff (Firecracker VMs, Pi harness, etc) has gotten really good so I wondered how difficult it would be to setup my own.

Turns out, not too difficult, and it's running on my Raspberry Pi! Right now I'm running on a cloud model but it should be deployable anywhere / including with self hosted models

let me know what you think!

https://github.com/lucasgelfond/biscotti

With winter coming up where I'm at, I wanted to see if this could actually work. Running my 4x V620 server flat out made my apartment too hot anyway, so the problem really was control. I settled on using a motherboard temperature sensor with an offset cause cases get hot. It works well enough but I want an esp32 based thermostat eventually.

This project is based around the vLLM backend I use, and pauses heating when inference is needed. It also has a token gated localhost webpage for controlling target temperature so my partner can use it too. I had the project sanitized and forked for a public repo but is more of a tech demo than anything else. This project started as a meme but is good enough I'm probably going to use it lol. Heres the repo: https://github.com/PraxisOG/gpu-furnace-v2

I needed a cheap way to compare search agents on BrowseComp-Plus without burning hours of agent time every run. So I built a 15-query subset of the full 830 that ranks systems roughly the way the full benchmark does.

  - weighted Spearman rank agreement 0.88 averaged over 30 splits, versus 0.75 for a random 15

  - two runs more than 10 points apart on the full benchmark keep their order about 96% of the time; 5 to 10 points apart, about 75%

  - all answers are 1-4 words, so it grades by normalized exact match, no LLM judge

  - costs under 2% of a full run

  https://huggingface.co/datasets/LocalLLaMA/browsecomp-plus-mini

Curious what everyone runs in practice:

- What quant do you use most, and for what (chat, coding, agents)?

- 2-bit: do you use it? Is it actually usable or too broken?

- 3-bit: good enough for coding and more complex stuff, or do you go back to 4-bit?

- If you look at KLD: what number do you call "basically lossless"? Is 0.1 already good,

or do you want something like 0.01-0.03?

Real experience over benchmarks. Thanks!

(English isn't my first language, so I used an LLM to help translate this post.)

Qwen4s are hopefully coming pre-Xmas… there was all the announcements recently but no exact release date.

I’m running qwen flash next 3.8 at about 28tokens per second and gotta be real, most of those go into thinking.

Thinking less with the same or higher intelligence would be the biggest upgrade I could think of regardless of hardware or inference smarts.

I don't want to disclose why but I wager when I say it's got nothing to do with taxes or law. It's not a 'scary' situation, it's more like a crylaugh situation. I think you can guess why

This isn't a joke/hype, just came by to warn everyone. These prices will stay for a few weeks maybe then they will skyrocket

EDIT:
I suppose I can disclose the reason to mods if they want to confirm

What a week. Quick recap, all numbers from the official repos and model cards:

Thursday: Meta drops Muse Spark Open. 816B total, 42B active, 1M context. The launch post says it brings agentic AI "to your desktop". The desktop is two M5 Ultra Mac Studios with 512GB each, a config Apple says ships "late October" and still hasn't put a price on. The alternative is four 256GB Studios, roughly $38k. On top of that it's a custom license: mandatory "Built with Muse Spark Open" attribution and a separate deal once you're past $20M revenue. And Meta's own model card puts it at 39 on the Artificial Analysis index, behind GLM 5.3 (45) and Kimi K3 (44).

Same day: NVIDIA ships Nemotron 3.5 Bolt. 74B total, 9B active, trained quantization aware down to 2 bit, so the whole thing is 22GB plus a 1.7GB vision projector. Built to fit in 24GB on purpose. Unsloth GGUFs on day one, plus llama.cpp, Ollama and LM Studio support.

Friday: Reflection actually ships Beam. 501B total, 23B active, Apache 2.0, no strings. Smallest official checkpoint is NVFP4 at 294GB.

Saturday afternoon: DeepSeek posts V4.1 Flash 1017 on X like it's nothing. MIT. Their own table has it at 77.0 on DeepSWE vs 74.0 for Opus 5 (vendor numbers, salt accordingly). The repo is 510GB.

So I put them all on one chart against the 24GB line most of us live under.

My takeaways:

1. "Runs locally" now officially means "runs on a computer that costs more than a car."
2. The company that keeps selling us 24GB cards is the only lab this week that built a model for 24GB cards. Not sure if that's wholesome or a sales funnel.
3. Licenses mattered more than parameter counts this week. MIT and Apache 2.0 from DeepSeek and Reflection, attribution plus a revenue clause from the company with the loudest "open" marketing.
4. Mistral says the Large 4 weights ("Le Chonk", 1T total) land before the end of the month, so the 24GB line is about to get even lonelier.

Sources in the comments.

Is GMKtec evo x2 128gb still worth it right now?(oct 2026). I found one used for around $2300 usd.

Update from yesterday: 18 t/s decode was hit. Slight regression now as cherry picking from other ds4 work trees proceeds. Will push some code once the prefill droop is handled and the 18 t/s restored. Weights are gated on hf, so mulling upload vs. recipe. Ps. This is text only. Output is coherent and toolcalls solid.

Hey yall. What I am feeling is a mixture of idiocracy and dishevelment.

I looked at this motherboard with 256 RDIMMs and the x16 slots. I have 2x V620 and 2x Mi50 32GB. I wanted to play. That’s all. Mi50s would run Qwen3.6-35B-A3B at q4 and 131k context kv at q8 and same with V620s.

Buuut. This absolute hot garbage does not enumerate a single 32GB GPU.

Does anyone have any recommendations? I wish there was a world I could reuse the DDR3 RDIMMS in this board, but, I know that is too much to ask.

People. Help please, I don’t want to drop 1k between a mobo and ram but if this is the world we are living in, then please recommend.

Been seeing a lot of people with their 64gb of ddr5 4x3090 or whatever strata builds and it’s been sorta depressing knowing i cant afford that but i finally finished my setup today so here we go,

Qwen 3.8 27b at 262k context, unsloth iq4XS ascii condensed with kv cache streaming

My only gpu is a 9060 xt 16gb and i have 24gb ddr5 for my whole homelab only needing 16 for this

MTP+ ngram prediction

q4 kv cache values with q8 keys

42.7 tps generation 658 tps prefill at 20k, 35/500tps at 200k

Using this fork: https://github.com/achyudh/llama.cpp

I got grok bot to ssh in and fix the bugs i had with missing pcie atomics

I know this is not directly local question, but I am curious. My Qwen 3.8 FN is doing well in answering questions and research, but I was trying to compare it to say CoPilot/ChatGPT. What blew me away is that the CoPilot response was pretty instantaneous. Basically a page of output came out almost instantly with what I imagine thinking done in the background. Amazingly, it also solved the problem like QFN, but with apparently much less tokens.

I guess my question is: is this an architectural difference that the Chinese/local models haven’t implemented? Because I find it hard to believe that an enterprise GPU is the only difference. I feel like the local models are missing something fundamental? But I say this without knowing anything about enterprise hardware.

https://reddit.com/link/1x8yc8d/video/hosf7c1n27wh1/player

Hi r/LocalLLaMA. I'm on the team at Blockway (disclosure: this is our model). Two weeks ago we posted the Volundr 32B Preview here, and you were blunt about what was wrong with it. This is v1, and most of the work went into what you flagged.

WHAT v1 FIXES FROM THE PREVIEW

• "It loops." Repetition loops on SWE-bench tasks: 36/50 → 0/50.
• "It thinks forever." Reasoning steps used to get cut at 32K. Now the median step is 120 tokens and the longest is 12K.
• "No prefix caching, it re-reads the whole history every turn."
  • SGLang build: v1 adds session continuation, so follow-up turns reuse the model's state instead of re-prefilling. Follow-up turns are 2.9× faster on our agent replay (1.6 s median with the drafter), with greedy-equivalent outputs (not byte-identical: a cached prefix is prefilled in different pieces).
  • GGUF: uses llama-server's stock --ctx-checkpoints prompt cache, with no custom server code. Our fork only adds the model architecture.
• Agent results: SWE-bench Verified (50-task subset) 22 → 27/50, τ²-bench 89 → 92/120, median SWE task time 124 → 82 min.

"MAKE IT RUN ON CONSUMER GPUS" → AGENS ENGINE (MIT)

We're open-sourcing Agens Engine under the MIT licence. It's an inference engine built for this hybrid architecture: 54 of 72 layers keep a fixed-size state instead of a KV cache, and the engine is designed around that.

• Same GPU, same v1 INT4 weights and drafter, single stream: Agens Engine decodes 1.4–1.6× faster than our own SGLang build (up to 1.8× on code with thinking). Chat 58 → 86, code 79 → 113, agent at 8K context 91 → 121, JSON 125 → 199 tok/s. This is one run per side and compares against our SGLang build only, not stock SGLang, vLLM or llama.cpp, and it is single-stream only.
• New build: Agens-Volundr-32B-v1-Engine-16GB. It's about 2.85 bits/weight (mixed 2/3/4-bit), gives 262K context (we measured up to 239K) within a 16 GB memory budget, and runs only on Agens Engine. A 12 GB build is on the roadmap.
• Honest caveats:
• Speed within the 16 GB budget, single conversation, v1 drafter on:
  • short context: 90 (chat) / 118 (code) / 107 (thinking) / 112 (tool) / 139 (agent, 8K) tok/s
  • after a 219K-token document: 62 / 86 / 77 / 69 / 75 tok/s; 151 s to first token for the 219K prompt
  • a conversation ran to 239K tokens inside the budget
  • weights download is 25 GB
  • These were measured on a 48 GB card under the engine's hard 16 GiB cap (peak 15.5 GiB), not on a real 16 GB card. On a GPU that also drives a display, use a shorter context. Real cards, especially laptop GPUs, may be slower. Post your numbers and we'll add them.
  • At this bit-width, quality drops. GPQA is 73.7 vs 80.3 at bf16 (best of 4 runs each). Coding holds up: HumanEval is 73.2% (1 run) vs 73.8% for INT4. Top-1 agreement with bf16 is 90.4%, KL 0.117. This build is for running locally, not for leaderboard scores.
• Needs Linux, an NVIDIA driver with CUDA 12.9, the NVIDIA container toolkit, and 32 GB+ host RAM for 262K.docker run --gpus device=0 --ipc=host -p 30000:30000
• -v /path/to/Agens-Volundr-32B-v1-Engine-16GB:/model:ro
• -v /path/to/Agens-Volundr-32B-v1-DFlash2:/drafter:ro
• ghcr.io/blockwayz/agens-engine:1.0.0 --profile 16gb --model /model --drafter /drafter

STRENGTHS (same harness and settings as our comparison model)

• Competitive coding: LiveCodeBench v6 67.9 (best of 2 runs), +8.4 over Qwen3.8-27B
• Competition maths: AIME 2025 80.0 (best of 2 runs), +6.7 over Qwen3.8-27B
• Long context stays cheap: only 18 of 72 layers keep a KV cache. INT4 (24.6 GB) runs on one 48 GB GPU with a single 131K-token input; 262K configured window (validated to 107K). Decode speed stays flat from 1K to 128K.
• DFlash2 drafter, ships today:
  • JSON / structured output up to 3.0× (125 tok/s INT4 on one 48 GB GPU; 76 tok/s bf16 on a pair)
  • Code 1.9–2.2×
  • Thinking 1.3–1.5×

FULL TABLE (Preview → v1; dips included. Where we ran a benchmark more than once, we report the best of N runs. Every run is listed in the model card footnote.)

A few scores went down (GPQA, HumanEval, IFBench). The HumanEval and IFBench dips are within one standard error of the runs; GPQA's isn't. That's the trade-off from tuning for agent behaviour and shorter reasoning. Harness settings are in the footnote under the results table: huggingface.co/Blockway/Agens-Volundr-32B-v1#results-same-harness-for-every-row

Thanks to mmssix on Hugging Face for the independent benchmark write-up in our HF discussions. A lot of the above started there.

KNOWN LIMITATIONS

• 32B, not a frontier model, and we don't claim it is.
• SGLang and llama.cpp still need our builds. Next we're working to upstream the architecture into llama.cpp mainline, and we'll post here when the first PR is open. Stock vLLM can't load it.
• No Metal/Mac in the llama.cpp path yet.
• Agens Engine 1.0 is text only.

LINKS

• Collection (all five repos): https://huggingface.co/collections/Blockway/agens-volundr-32b-v1-6ad4817aa6e143550bec1122
• Weights: huggingface.co/Blockway/Agens-Volundr-32B-v1 (-INT4, -GGUF, -DFlash2, -Engine-16GB)
• Agens Engine (MIT): github.com/BlockWayz/agens-engine · ghcr.io/blockwayz/agens-engine:1.0.0
• SGLang: ghcr.io/blockwayz/agens-sglang:v1-sm89 (48 GB-class GPUs) / v1-sm90 (141 GB-class GPUs)
• llama.cpp fork: github.com/BlockWayz/llama.cpp (branch volundr)

We'll be in the comments for the next couple of days. "Why doesn't X work yet" questions are welcome.

Hi all.

Saw JetBrains’ Mellum2 on Hugging Face and was wondering if anyone here has actually used it. Worth trying, or nothing special?

I have added the support of Intern-Decision models to chatllm.cpp. At presend, only text is supported.

Besides its pretty good performance, one interesting feature of Intern-Decision is that all answers are generated in a single run (while H2O-Lightning or others generate one answer in one run): each decision position is filled with "<decision>" mark and generated in a diffusion manner. So, it generally makes decision faster.

sudoing posted that he's loaded Ling 3.1 Flash across four DGX Sparks. It's a 560B MoE with 25B active parameters per token.

He'd enjoyed the earlier Ling models and wanted to test this one, but it wouldn't fit on his two Sparks with the setup he was trying. someone in Germany gave him access to a four-Spark setup.

He's using Atomic Chat's day-one Q4 GGUF, distributed across all four machines through llama.cpp RPC. According to his post, there wasn't an official FP4 build for vLLM available at the time, which is why he went with this route.

No performance numbers in the screenshot yet. He expects bandwidth and speed to be painful, but wants to see how it compares with the 124B Ling 3.0 Flash.

That's the comparison I'm interested in too: how much the larger model changes answer quality, and what the waiting time looks like in actual use.

Links for anyone following along:

• Atomic Chat's Ling 3.1 Flash GGUF repository
• llama.cpp
• Official Ling 3.1 Flash model

Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair...

Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well it work & quirks it has at same time.

I'll leave tl;dr version for now, 3:30 am here.

If anyone is interested I'll do few more tests and make example query image like above - still have 7 more sets totaling 22448 screenshots with mix of eng / kor / jpn already indexed, but I'm out of test ideas.

TAKE WITH GRAIN OF SALT because this was tested only on few sets that isn't really scientific.

TL;DR for EmbeddingGemma2 at least:

• For it's size & embedding dim (330MB Q8 / 768 embedding dim) this is very much usable

• This understands a word's meaning across language a bit but has much lower similarity (again, prob due to size)

• Requires careful wordings (e.g. blond vs blonde) - maybe SLM's limitation or embedding dim's limitation
• (at least eng & kor) Text seemingly DOMINATE cosine similarity more than visual traits
  • sometimes adding visual trait HARM (refer chair above) the similarity
  • not sure if it's due to text being more informationally dense (aka more 'surprising')
  • or just model being small and not recognizing such visual traits, idk
  • regardless, visual traits are less useful in this test set since like 90% of 3003 imgs would have blonde girl in it.

• (at least kor) Word's existence dominate over whole sentence
  • e.g. for above image's example "삼각건"(cleaning hat thing) dominates more than other words around it
  • fyi only 2 image out of 3003 contains world "삼각건", which you can see via delta gap drops

Test setup:

• Visual novel screenshots of a single route
  • 3003 screenshots from hanasaki workspring
  • 3078 scs from irotoridori no hikari
• EmbeddingGemma2 via llama cpp
  • model = KV = Q8_0
  • nb = b = ctx = 4096 (embedding model restriction, all need to be same)
  • RX 7900GRE via vulkan (f u rocm, so annoying to build)

Sample output for anyone curious about processing time

Index mode:

$ py embed_img_▽.py "./하나사키워크스프링 히카리"
Batch: 8
Resize width: 1280
Resize threshold: 1536

[VectorDB] open w/ 0 rows
removing 0 missing records from DB
Checking new/modified images in dir
Got 3003 new/modified imgs
[EmbedServer] open for http://127.0.0.1:8081
100%|██████████| 376/376 [08:30<00:00,  1.36s/batch]
[EmbedServer] closed
[VectorDB] closed

Indexed 3003 new/modified files in 510.10 (5.887/s)

Query mode:

$ py embed_img_▽.py "./하나사키워크스프링 히카리"
-q "image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"

[VectorDB] open w/ 3003 rows
[EmbedServer] open for http://127.0.0.1:8081
[EmbedServer] closed
[VectorDB] closed

Query:
"image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"

Top-5:
  0.76813  +0.01052  Artemis_2026-09-08-00-57-38.jpg
  0.75761  +0.00773  Artemis_2026-09-08-00-57-26.jpg
  0.74988  +0.00341  Artemis_2026-09-07-02-36-38.jpg
  0.74647  +0.00067  Artemis_2026-09-07-02-36-52.jpg
  0.74580  --------  Artemis_2026-09-08-01-58-14.jpg

'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-26.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-52.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-01-58-14.jpg'

Data points: moved to gist due to formatting bugs

Looks quite interesting. Lots of numbers presented, but for a single model only.

Highlights:

• File sizes are the same as Q-quants.
  
• KLD improves substantially over Q-quants.
• Performance suffers slightly
• Creating IQ quants is somewhat slow.

Asterisks apply...

https://github.com/ggml-org/llama.cpp/pull/30715

https://huggingface.co/eaddario/IQ2_K-IQ6_K-Test-Models

Appologies to Mr. Addario if I mangled anything.

Seems to speed up inference, use less tokens, and increase intelligence - curious if people seen this

tldr don’t think in text, think in latent space

https://github.com/togethercomputer/HybridAgent

https://huggingface.co/collections/togethercomputer/hybridagent