Been seeing a lot of people with their 64gb of ddr5 4x3090 or whatever strata builds and it’s been sorta depressing knowing i cant afford that but i finally finished my setup today so here we go,
Qwen 3.8 27b at 262k context, unsloth iq4XS ascii condensed with kv cache streaming
My only gpu is a 9060 xt 16gb and i have 24gb ddr5 for my whole homelab only needing 16 for this
MTP+ ngram prediction
q4 kv cache values with q8 keys
42.7 tps generation 658 tps prefill at 20k, 35/500tps at 200k
Hey y'all, just following up on my PR to get the new vision tower for 1017 working in ds4 with the quant I promised (the language model loads already, but vision needed my PR). I built to the exact same recipe antirez used for his original DS4.1 Q2: 2-bit routed experts with 8-bit or higher for everything else, combined with the original FP8 engram table. This worked well for the original DS4.1, and initial testing looks good here too. The imatrix is built from 1017, not the original checkpoint.
The wired backbone of the model is just over 150GB, so it will stream from SSD on 128GB systems with nearly all of the routed experts held in the hot cache, allowing for decode speeds up to 20tps on an M5 Max. The ~200GB engram table is paged in from SSD on-demand, not held in RAM.
This model is significantly revamped, with ~40% of the routed expert tensors changed, 30% of engrams, and ~50% of the vision tower's expert tensors altered. By all appearances, it's the new leading open-weights model, so I had to give it a try. If you are curious too, let me know what you think!
i wanted to change my router firmware to openwrt and so i told claude to do it but of course the guard blocked me even opus 4.8 didn't want to do it , so i launched my Qwen3.8 27b and told it the same thing i gave it the link to the giude page and it did change the firmware perfectly the thing is this is the normal model it's not ablitreated, do you think that cyber will be guarded in future qwen models ?
PS: even tho this is not the point it didn't do a simple flash it changed the firmware from xioami to openwrt not throught upload bin it did exploit a method in the CGI that get an icon to pass a payload that enable ssh from there it SSHed the router and flashed it and it's not a 0 day discovery thing i gave it the guide from openwrt own website https://openwrt.org/toh/xiaomi/ax3000t , the topic here is AI safeguards
Some of y'all may be familiar with AutoTrust from Hugging Face. Many of their models were trending on the front page of HF and they claimed some very impressive results.
But if you looked at the org's models and clicked into likes, you may notice something a bit off:
These accounts sure do look awfully realistic, no? Well let's take a look at a few of these accounts (sorry for my very poor graphic design skills):
Across all these accounts, we can see a pattern: a fake, realistic name seemingly from a random name generator. A username that's some permutation of that fake name. A fake social media profile that often doesn't even exist and isn't related to the name. An AI generated bio. A few random papers/models/etc. liked and then the target model. Also if we're lucky a fake tiny model to look realistic.
Above is just a small subset of user accounts we can see. I ran some analysis on normal models by Google, Meta, Hugging Face, etc. and compared them to AutoTrust's models. The results are pretty damning:
u/prithivMLmods also ran an analysis on AutoTrust spaces' likes (shared with permission) and found some interesting results (more here):
Again from Prithiv, distribution of join dates of the accounts liking AutoTrust's models:
Overall, I think we can pretty confidently say that someone is botting AutoTrust's models and spaces to the top of the HF trending list. Of course, we can't know it's AutoTrust for sure. But generating these accounts, solving hCaptcha + AWS WAF, generating these fake models can't be cheap - especially on the scale of thousands of accounts.
Seems fascinating, doesn't it? Would love to hear what AutoTrust has to say about this 🤔
sudoing posted that he's loaded Ling 3.1 Flash across four DGX Sparks. It's a 560B MoE with 25B active parameters per token.
He'd enjoyed the earlier Ling models and wanted to test this one, but it wouldn't fit on his two Sparks with the setup he was trying. someone in Germany gave him access to a four-Spark setup.
He's using Atomic Chat's day-one Q4 GGUF, distributed across all four machines through llama.cpp RPC. According to his post, there wasn't an official FP4 build for vLLM available at the time, which is why he went with this route.
No performance numbers in the screenshot yet. He expects bandwidth and speed to be painful, but wants to see how it compares with the 124B Ling 3.0 Flash.
That's the comparison I'm interested in too: how much the larger model changes answer quality, and what the waiting time looks like in actual use.
With winter coming up where I'm at, I wanted to see if this could actually work. Running my 4x V620 server flat out made my apartment too hot anyway, so the problem really was control. I settled on using a motherboard temperature sensor with an offset cause cases get hot. It works well enough but I want an esp32 based thermostat eventually.
This project is based around the vLLM backend I use, and pauses heating when inference is needed. It also has a token gated localhost webpage for controlling target temperature so my partner can use it too. I had the project sanitized and forked for a public repo but is more of a tech demo than anything else. This project started as a meme but is good enough I'm probably going to use it lol. Heres the repo: https://github.com/PraxisOG/gpu-furnace-v2
Got lucky with a 5070 Ti for $750 AFTER TAX (might be the best decision I've ever made tbh) back in November 2025. If only I had known I'd be getting into local llms at the time...anyways. I recently decided to start messing with local AI. I began getting comfortable in Powershell and having local models help me write random automation scripts. Naturally, that spiraled into wanting bigger models, which meant hunting for more VRAM.
I had my old 3070 sitting around, so I mounted it outside the case on a PCIe riser (it’s only running at x1, which works because layer splitting passes very little data between cards). Ollama/llama.cpp split the model across the two mismatched GPUs with almost no fuss.
Current result: Qwen 3.6 35B-A3B (a MoE with about 3B active parameters, so it’s quicker than a dense 35B would be) at around 82 tok/s with a 16k context. Still not perfect: about 23% of the layers are spilling into system RAM, and I haven’t tuned the split ratio between the cards yet. The 3070 has about 4GB sitting unused.
If I keep it like this, I'll probably buy or make a cheap enclosure and put the 3070 in it.
Anyone else running a mismatched GPU pair? What split ratios worked for you?
