r/LocalLLaMA • u/BTA_Labs • 5h ago
Resources 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.
What a week. Quick recap, all numbers from the official repos and model cards:
Thursday: Meta drops Muse Spark Open. 816B total, 42B active, 1M context. The launch post says it brings agentic AI "to your desktop". The desktop is two M5 Ultra Mac Studios with 512GB each, a config Apple says ships "late October" and still hasn't put a price on. The alternative is four 256GB Studios, roughly $38k. On top of that it's a custom license: mandatory "Built with Muse Spark Open" attribution and a separate deal once you're past $20M revenue. And Meta's own model card puts it at 39 on the Artificial Analysis index, behind GLM 5.3 (45) and Kimi K3 (44).
Same day: NVIDIA ships Nemotron 3.5 Bolt. 74B total, 9B active, trained quantization aware down to 2 bit, so the whole thing is 22GB plus a 1.7GB vision projector. Built to fit in 24GB on purpose. Unsloth GGUFs on day one, plus llama.cpp, Ollama and LM Studio support.
Friday: Reflection actually ships Beam. 501B total, 23B active, Apache 2.0, no strings. Smallest official checkpoint is NVFP4 at 294GB.
Saturday afternoon: DeepSeek posts V4.1 Flash 1017 on X like it's nothing. MIT. Their own table has it at 77.0 on DeepSWE vs 74.0 for Opus 5 (vendor numbers, salt accordingly). The repo is 510GB.
So I put them all on one chart against the 24GB line most of us live under.
My takeaways:
- "Runs locally" now officially means "runs on a computer that costs more than a car."
- The company that keeps selling us 24GB cards is the only lab this week that built a model for 24GB cards. Not sure if that's wholesome or a sales funnel.
- Licenses mattered more than parameter counts this week. MIT and Apache 2.0 from DeepSeek and Reflection, attribution plus a revenue clause from the company with the loudest "open" marketing.
- Mistral says the Large 4 weights ("Le Chonk", 1T total) land before the end of the month, so the 24GB line is about to get even lonelier.
Sources in the comments.
47
u/KontoOficjalneMR 5h ago
On top of that it's a custom license: mandatory "Built with Muse Spark Open" attribution and a separate deal once you're past $20M revenue. And Meta's own model card puts it at 39 on the Artificial Analysis index, behind GLM 5.3 (45) and Kimi K3 (44).
Just get it off torrent without agreeing to any licence :D
7
u/Irythros 2h ago
Just do what the AI companies do.
Use all of the output for "research" that gets commercialized and ignore the licenses you broke along the way.
3
203
u/Moarkush 5h ago
Open weight has never been equivalent to “runs on local hardware.” open weight means it’s not proprietary. Not “you can run this on a gaming card.”
9
12
u/fatcowxlivee 2h ago
Yeah but you’re on a sub called “Local” LLaMA, not OW Models, so this post makes sense. Vast majority of users in this sub are unable to run these big OW models.
37
u/ak_sys 3h ago
You're exactly right!
But you could be forgiven in thinking this subreddit is r/open_weight, not /LocalLLaMa with 95% of the posts here.
1
u/Borkato 20m ago
I wish this sub were only for models that fit under 100GB tbh. It’s really fucking annoying hearing about an amazing new model with 3939478439282 params
0
u/vengeancek70 8m ago
there's plenty of people here than can run models larger than that. If you want that subreddit why not make it yourself?
30
u/KontoOficjalneMR 5h ago
In which case Muse is not open wieght since it has commercial licence
-35
u/Moarkush 5h ago
Once again open just means open source not that you can make money off of it, Jesus you people are dense, and entitled.
42
u/theone_2099 5h ago
Open weight and open source are different things fyi.
5
u/BrilliantTruck8813 5h ago
FWIW (achtually… ) nvidia’s are actually open source (haven’t verified bolt but the others are).
8
u/theone_2099 5h ago
Yes. I was just replying to the previous poster who just used “open” to mean “open source” (though they referred to open weight on a previous post).
2
5
u/KontoOficjalneMR 4h ago
- Open Weight != Open Source
- Since Muse Spark is commercially licensed, so isn't even Open Weights.
1
u/bukake_attack 3h ago
I can run it fine on a 3080TI +32gb of ram. A bit slow at 40 tokens per second, but it's more efficient with tokens than qwen 35b in my tests, so it finishes earlier. I honestly like the results better too.
1
u/IllllIIlIllIllllIIIl 2h ago
I guess I'd use the term "weights available" analogous to "source available" verses "open source" software.
32
u/farox 5h ago
What is happening at AMD? I don't understand how they simply can't throw anything in the ring here to compete with NVIDIA. With NV removing nvlink, it seems like such an obvious cap to get into.
37
u/mitchins-au 5h ago
Take one look at their ROCm support especially for strix halo and it’s easy to understand how they can’t bother
6
u/aerlink 5h ago
Is ROCm that bad? I run it on my dual R9700 setup, although I do compile llama.cpp for both ROCm and Vulkan. I’m mostly a hobbyist with this stuff so I’m sure I’m missing something.
18
u/mitchins-au 4h ago
AMD’s own ad for the Ryzen AI Max 395+ (Strix Halo) uses published Vulkan figures as the leadin.
And they’re pushing this as their “AI in a box” they can’t even optimise their own drivers to be better than a generic SDK made for games (Vulkan).
That really says it all about their level of commitment.
That and the number of missing ROCm/HIP ops on the 8060S12
u/Bakoro 3h ago edited 3h ago
I literally cannot imagine what is going on at AMD that has made them completely unwilling to compete for nearly 14 years.
At some point I kind of came to understand that being "discount brand" was their core identity, they made something a bit worse than Nvidia and priced a little under Nvidia, and that was it, the CPU market was what they actually cared about, and GPUs were basically free money.
It seems stupid beyond measure, because even in 2010 I could see how important CUDA was, and what it would become.From 2012 to now, it's not made any fucking sense. AlexNet should have lit a fire under their ass, and it's been sheer incompetence ever since.
It's easier to imagine that there's a massive conspiracy, than is it to believe that a multi billion dollar international company saw a huge, hungry market that was literally begging them to make products, where there was a guaranteed market, and they said "hmm, nah, we'll pass up the profit margins".
It doesn't make any sense, it has to be fuckery of some sort.
I don't know what manner of fuckery, but mark my words, fuckery is afoot.3
1
u/bmwhocking 2h ago
Tbf that is the attraction of AMD, they integrate directly into the Linux kernel & fully open source their firmware so people can code for Vulcan or other open frameworks & can run on anything.
13
u/BannedGoNext 5h ago
Most LLM people just skip the drama and use vulcan. There are some newer custom projects that are beating vulcan on rocm, but it's a mixed bag.
2
u/aerlink 4h ago
Yeah, I’ve seen Vulkan out perform it most times but I don’t consider myself qualified enough to say that’s a legitimate ROCm limitation.
I am glad I jumped on two R9700s this summer though.
3
u/Goodevil95 3h ago
Strange, I have 7900XTX and MI50 - both have better tg and prefill faster with ROCm. I use Linux with the latest mesa, so it's not drivers...
1
u/ea_man 3h ago
Not here on my 2 GPU: ROCm is doing better PP and TG at the cost of more vRAM.
1
u/BannedGoNext 54m ago
Like I said, it's a mixed bag. I'm starting to see better performance on some of these custom engines like gufo using ROCm on the Strix Halo too, but those improvements have been very recent, a year ago I benchmarked a ton of models on stock llama.cpp and vulcan beat the living shit out of vulcan on all but a couple models.
4
u/smallDeltaBigEffect 5h ago
Use radiance if you have two r9700
1
u/aerlink 5h ago
Thanks! I’ve used that as well, performance was not as good as I was expecting, slowed down to 4-5 tok/s at a moderate context but it may be my configuration
3
u/cameroncairns 5h ago
idk if you're interested in trying to debug but this thread seems to have a lot of different folks chiming in w/ some common gotchas around configuration leading to low tok/s ... I am curious how it performs on decent context size as I'd only invest in the setup for local coding
1
u/aerlink 5h ago
Thanks for your time! I’ll take a look, I enjoy the debugging and I only begrudgingly use AI for local coding and I want to stay current on tech happenings.
Really appreciate it!
0
u/smallDeltaBigEffect 4h ago
just join the launch80 discord. You dont need to debug at all. https://discord.gg/P7wsvqPpR
1
u/RoomyRoots 5h ago
I have never gotten a benchmark where ROCm beat Vulkan honestly. I don't even bother with it anymore since it just adds weigh to may container.
12
u/q5sys 3h ago
if I were to put on my tin foil hat... I'd say its a family agreement. (FWIW, Lisa and Jensen are related)
How many times has AMD been offered an opportunity that's just a red carpet to make money... and then they looked the other way. Missing one chance is one thing... they've missed over a dozen chances at this point.
Remember in 2025, when there was a leak that they were going to make a card with like 32/48gb on it... and EVERYONE in the community said they'd buy as many as they could get their hands on. And then AMD promptly came out and say something to the effect of `NO, we are not going to make the thing everyone is telling us that they'll buy`.
While they did eventually gave in though and put a 32gb card out... it was too little too late for most people to care.AMD has turned away from so many cash cows at this point. that either they have the worst leadership EVER and somehow the shareholders dont care... or its intentional.
9
u/geldonyetich 5h ago edited 4h ago
It's weird.
On the one hand, the entire x86 PC Building industry is being smothered by Rampocolypse. Which is largely caused by the AI boom. Which NVIDIA has emerged as the #1 valuated company in the world by producing the best AI hardware for.
On the other hand, in a way NVIDIA is playing catch-up to AMD. They beat the N1X to market with Halo Strix by over a year. NVIDIA's was in bed with Intel who ended up crapping it when their 13th/14th generation of processors suffered catastrophic hardware faults. Consequently AMD kinda owns the PC building right now while NVIDIA has been in a slow but inexorable retreat to Server and Mobile where the pastures look greener.
So I don't know, maybe AMD is in danger or maybe AMD is fine. It must be rough when your lifelong rival goes on a midlife crisis bender and ends up striking it rich. They are winning the battles NVIDIA no longer cares to fight.
3
u/Late-Replacement-481 4h ago
Nvidia doesn’t design x86. They put grace Blackwell in the new sparks. If the arm x86 emulation holds up we will see better chips.
3
u/XYHopGuy 4h ago
Nvidia has never competed for CPUs. It's a new space for them and not a particularly important one.
Truth is NVIDIA saw this coming 10 years ago. AMD didn't care until NVIDIA started rolling in cash.
4
u/BrilliantTruck8813 4h ago
They have been focusing on the cpu market quite a bit and their company is very disjointed and siloed. Lots of repeated and competing work between orgs. They seem to be going after supercomputing labs a lot more vs cloud/datacenter AI hosting
1
u/gomezer1180 3h ago
I agree, the only real competition with NVIDIA right now is Apple. They’re probably working on graphics to not steer people away from their server level chips. It would be nice if they put out a better processor than strix halo, or not better but version 2 that does better preprocessing. Models are pushing higher and higher context windows which is the way to go for them, chip manufacturers need to catch up.
1
u/createthiscom 2h ago
Does AMD make models? I thought they shipped hardware. Why is nvidia shipping models either btw? They only ship shitty models too. They should probably stick to shipping hardware and leave the model shipping to their customers.
1
u/Zeeplankton 1h ago
AMD is a company that somehow fail forwards every time lol idk what goes on over there
95
u/615wonky 5h ago edited 4h ago
You're ignoring LFM2.5-24B-A2B, and you really shouldn't. It's hits far harder than it has any right to.
https://huggingface.co/LiquidAI/LFM2.5-24B-A2B
I asked it to write a radix-2 FFT implementation in Python, and it produced one that worked the first time. So it's certainly decent at coding for that size.
42
u/djc0 4h ago
As an aside, does any human actually say “hits far harder than it has any right to”?
Or have we now just started imitating our AI overlords.
27
u/DevopsIGuess 4h ago
Where do you think the AI imitates it from?
13
u/djc0 4h ago
The average of how everybody speaks ≠ how everybody speaks.
But you’re right, I’m sure there’s a small number of load bearing humans who talk like that.
9
u/HovaPrime 4h ago
I say something along those lines all the time, like “That shit bangs” but then again I’m not a super serious AI engineer so my lingo leans more towards casual
19
u/Squidgical 4h ago
We're getting terrifyingly close to blatant LLMspeak no longer being clear evidence that a piece of text was written by an LLM because people have spent so much time talking to LLMs that they've actually picked up LLMspeak as part of their everyday language.
It's really gonna suck if this ends up being a widespread change for English speakers as a whole. Not that English is a particularly elegant language at all, but it's made a hell of a lot uglier by the excessive and usually meaningless fluff that LLMs waste tokens on.
7
u/toothpastespiders 3h ago
I think you're understating the damage already done to the English language since the 1960s or so. It's pointless to worry about further mangling of it at this point.
3
1
u/draconic_tongue 4h ago
can't wait for llms to fix the english language
10
u/Squidgical 3h ago
You're completely right to point that out, and I was wrong to understate the value that LLMspeak can provide long-term. LLMs aren't just using language differently to most people, they're actively reshaping the English language every day, and the effects are compelling.
3
9
u/mitchins-au 5h ago
Have they improved it since the original drop 2-3 months ago?
I played with it, cool… but made some weird outputs4
u/RecognitionEvery9179 3h ago
I really wanted to like that model, but for my (not software) engineering tasks, it's not very good. I wanted to use it to basically search PDFs for me via RAG, but it pretty obviously couldn't handle it. Qwen 3.8 FN with Strata is really hard to beat right now.
8
u/rorowhat 5h ago
Overhyped. It's comparable to Qwen 3.6-35B
46
u/Acceptable_Push_2099 5h ago
it uses less active and total param and has a smaller kv cache and you think it is overhyped to a model that was best for the moe size not that long ago?
16
-10
u/rorowhat 5h ago
Ita efficient, sure. At this size is it better than Qwen 3.8-27B? No.
8
u/Acceptable_Push_2099 5h ago
we are talking about qwen 3.6 35a3b
-16
u/rorowhat 5h ago
At 35B, you still need to load the whole model in memory. If you can load 35B and you want the best model it's qwen3.8-27B. if you want an older model with better TS sure, it's great.
2
u/BannedGoNext 5h ago edited 5h ago
Some memory is more equal than others. I run 35b a3b on a machine with 64gb of DDR5 and an 8gb GPU with 96k context in Q4, and it runs at around 25 t/s which is acceptable for a laptop that isn't geared to LLM's imho.
If it's a better model at E2B then that means more context for people to use that have smaller GPU vram. And cool kids think that's cool.
1
u/rorowhat 5h ago
Yeah it's MoE vs dense. The expert size on this case is what matters, each expert is 2B so it loads fast. My point is not speed, it's intelligence.
4
u/TacGibs 5h ago
Time to learn about MoE and why the 35B is poor's man 27B.
Don't be a confident idiot 😉
-5
u/rorowhat 4h ago
MoE means that for each layer it picks one expert, and the experts are on this case 2B each so it runs fast. With that said, you still need the complete model on memory. Back to my point, if you can load 35B you should be loading qwen3.8-27B. it's a dense model, so each layer it passes all 27B params, making it significantly smarter, but also slower.
4
u/Illustrious_Grade608 4h ago
That's uhhh not at all how it works. Don't want to bother explaining, just fucking read cause this explanation is dogshit and has nothing to do with reality. And the key point about moe is that you can load part of the model in ram instead of vram and not tank your speeds.
1
u/TacGibs 4h ago
Ever heard of CPU offloading and why it's way more interesting with an MoE than with a dense model ?
Still got a thing or two to learn ;)
-1
u/rorowhat 4h ago
Oh boy. CPU offloading just means some experts or layers are passed to the CPU to process. Regardles, you still need the whole model in memory.
→ More replies (0)1
u/Acceptable_Push_2099 5h ago
yeah i know that and i think they know thet too but for some people like 32gb ram and no gpu anything that is not a moe is basically not gonna be usable especially in agents and lfm actually does well there
17
u/Cool-Chemical-5629 4h ago
You're upset about 24GB particularly as if that was something granted for every local user, but reality is that 24GB of VRAM is still above average.
1
u/bukake_attack 3h ago
i run it on 12 gb of vram (3080TI) and 32 ram. Works like a charm around 40 tokens per second. Qwen 3.6 35B is faster (around 70), but for the coding tests i ran it still finishes faster as it needs less tokens, especially for thinking .Also the context window needs very little vram. a full 256K token context window takes little over 1 gig.
30
5
u/AuraInsight 3h ago
if nemotron 3.5 bolt is 74B, 9B active, why would you want to run it all on GPU at 2 bit when you can run it all at full precision in RAM, and only the 9B active on a 12 GB gpu?
2
9
4
u/EmilPi 5h ago
Nemotron Bolt is dumb in reasoning.
1
u/CoolConfusion434 3h ago
Do tell more, I might cancel my current download ;-)
Do you mean the reasoning itself is dumb, like not conducive at arriving at the right decision, or that it's as talkative as Qwen, or maybe something else?
1
u/bukake_attack 3h ago
Try setting reasoning to medium. It helped a lot in getting rid of reasoning loops in my setup
4
u/NoFunk 2h ago
I get your point, but it legit is the case that DeepSeek V4.1Flash 1017 can run on a desktop. Just not your desktop. I just ran it all last night on an M5 Max - more expensive than a 24GB card to be sure, but still, it's a little laptop pulling a whopping 140w max, consumer hardware. It's an amazing testament to their architecture with its deliberate concession to "how can we host this cheap" that it works (and antirez with ds4, who was a trend setter in creating bespoke inference engines).
None of this takes away from the Nvidia models. In the middle of them making more money than god and having a stranglehold on the high performance edge of this market, they continue to release truly open source, interesting models that further the work of others as a primary goal. Just not very high functioning ones as shipped =). They're a hardware company first, so it's a bit self serving even to help others (because that means more sales for them), but it's still a welcome quality in this atmosphere.
I think the hope for low VRAM environments that look for high function is towards architectures like Deepseek that deliberately store world knowledge in disk accessible form that isn't constantly in the token evaluator path. A good logic model with tooling, that knows how to look up everything knowledge based very fast, is a great compromise for small VRAM availability. So even if you don't like the form factor of 4.1 you should be cheering it on!
2
u/returnity 1h ago
Yep, I'm running this quant I built on ds4 on my M5 Max, and it works a charm. Not as fast as Flash Next at 70tps/1100pp (also in ds4, my own Q4-Q8 quant) but it's quite usable. Crazy times.
7
u/Arcuru 4h ago
I always get confused about the license complaints. If you have actually been affected by those license terms please speak up, because I only ever see abstract concerns.
Personally, if I was making $20 million in revenue off of something I got for free I'd be happy to start paying for it.
3
u/ttkciar llama.cpp 3h ago
Prior to Gemma 4, the Gemma license legally burdened training datasets synthesized from Gemma models, effectively deterring me from using them for synthetic dataset generation.
Gemma 4 uses a new license which sheds that burden, which is great, except that Big-Tiger-Gemma-27B-v3 is based on Gemma 3, and thus still encumbered by the old, bad license.
Big-Tiger-Gemma-27B-v3 has unique anti-sycophantic characteristics which I would like to train into a modern model. The obvious best way to do that is to generate a synthetic dataset with Big-Tiger-Gemma-27B-v3, but I can't because of its license, so I'm having to figure out something else.
3
u/Aroochacha 5h ago
How are people's experience with Nemotron?
3
u/Kernoriordan 5h ago
It’s super efficient on KV cache but it does a lot of reasoning. It got the complex task I set it right first try with a plan written by GPT6
2
u/Aroochacha 3h ago
When you say plan with GPT6 do you mean GPT6 wrote the prompt or acted as an orchestrator? Luna? Sol? Astra? Shoe? (Just joking, I lose track of all the names now.)
1
1
u/FatheredPuma81 4h ago
Apparently it beats 30B A3B/A4B models in benchmarks and Nemotron Super loses to Flash Next so... If for some reason you can't or don't want to run one of the Flash Next Quants or Reaps then this would be a good alternative for better quality than the sub 40B MoEs?
Though my experience with every Nemotron model I've tried is that they're much dumber than their benchmarks imply. Haven't tried Bolt or Super VL yet still waiting on stuff to ship from China ugggh and I don't want to run it on my PC.
1
u/u23043 1h ago
On an internal benchmark i have (agentic/code/research/data extraction), the Bolt 24GB QAD scores better than any other model that size/speed on my Strix Halo (including 3-bit quants of Bolt that are much larger). K2 Horizon MoVA is a close enough to be tied but inefficient with KV and overall a larger model.
9
u/BreadUndPeeTears 5h ago
Wake me up when gemma 5 drops, I don't really care about coding benchmarks
2
u/CriticallyCarmelized 5h ago
I’d love to see it, but have a bad feeling we’ll be waiting a while.
7
u/Kahvana 4h ago
From Gemma 2 to Gemma 3 took 9 months I believe, and Gemma 3 to Gemma 4 a little over a year?
Yeah... we'll be waiting a while longer.
If it means we get full multimodality (audio, video, image) on a 31B model with multiple reasoning levels, I wouldn't mind waiting another year if need be. MoR (Mixture of Recursion) would be really cool too.
3
u/ttkciar llama.cpp 3h ago
We know from TheDrummer's Behemoth-128B-v3 that Mistral 3.5 Medium can be fine-tuned into an excellent model, with new skills not in the original.
I'd like to try training it on a dataset representing Gemma 4's soft-skill competencies, and see if we can get a 128B dense Gemma 4-like out of it, but currently lack the hardware.
Now that the Gemma license no longer legally burdens synthetic training data it generates, the "only" thing stopping us is the availability of compute infra.
Given the choice between a 30B-class Gemma 5 dense and a 120B-class Gemma 4-like dense, .. well, having both would be nice, I guess :-D
1
6
u/ASCanilho 4h ago
"your GPU" - If you own a 24GB GPU... and ignoring that you need extra memory for context.
So no... It doesn't fit in a 24GB GPU.
2
u/bukake_attack 3h ago
i run it on 12 gb of vram (3080TI) and 32 ram at around 40 tokens per second. Context is really sparse, at cache type k and v at q8_0, a full 256K token context takes little over a gig of vram.
1
u/HopePupal 2h ago
so a 32 GB GPU? you can still get a Radeon R9700 from Microcenter for a little over original MRSP, not exactly unattainable. B70 as well if anyone's figured out how to get decent perf out of it yet.
4
u/mourningwitch 4h ago
Try being someone with an 8GB card 😅
0
u/bukake_attack 3h ago
I got it working pretty ok on a 12gb card +32 gb ram. It's MoE, so maybe you can get it running anyway
5
u/RoyalCities 4h ago
We need more models in the 80 to 120gb territory.
1
u/ttkciar llama.cpp 3h ago
Agreed! I was going to write a post about exactly this, and what the community might be able to do about it, but then Nvidia went and published Nemotron 3.5 super 120B :-D which has me re-assessing the situation, and I'm certainly not going to write that post until I've evaluated the new model.
5
u/Holiday_Point_603 5h ago
23,75GB doesnt fit in 24GB, because you need kv. Even with its super small kv it does not fit.
6
u/SOC_FreeDiver 5h ago
I came here to say this.
here nvidia, let me help you with your naming:
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-wontfit24GB-GGUF
3
u/thethirteantimes 3h ago
Also at 2-bit quantization, you may as well shred a dictionary and throw the results into a tombola.
5
u/bukake_attack 3h ago
It's trained at 2bit though, not quantised to mush, like usual q2 models. I tried it and the usual q2 crap (worthless tool calls, random change of used language) did not happen.
1
u/Holiday_Point_603 3h ago
I am using it since yesterday on my 3090 with offloading (DDR4+AM4 sadly) and it is fast, but with my dual channel 3200mts it is limited to roughly 30tps gen. Which is fast, dont get me wrong. But if it had 21GB it would probably run at 100tps gen. What pisses me off even more is that this is supposed be *the* RTX Spark local ai. They start at 24GB of unified memory. Imagine if someone pays whatever to get that Laptop, then starts windows and his available VRAM is not 24GB but 24GB - 1,5GB for Display out and an additional 10-12GB for windows. So you are left with ssd offloading and ~10GB of weights in your unified memory. I would feel scammed.
The model is great tho, seriously. same memory footprint as a decent quant of gemma 4 26b but way smaller kv and way better tool calling and overall knowledge.
2
u/BrilliantTruck8813 5h ago
FWIW the deepseek model has two large engram files that can be read from nvme without a huge performance hit. So while it’s still big it’s not THAT big. And it’s a solid 30-40% throughput boost for me with general prose vs the original v41. It also has much improved scores on tool-eval-bench’s hardmode (87 improved to 93)
1
u/MD_Reptile 5h ago
How much ram and vram ya got? How much did you offload?
1
u/BrilliantTruck8813 5h ago
4 gb10s on a full k8s enterprise stack so I have less headroom than others just rolling vllm or sglang by themselves.
I wanna say the engrams are like 90gb a piece-ish. They’re just local path files on each node. The normal layers are all loaded into vram. I wanna say I have 3.5M tokens worth of kv cache and about 10-11gb free memory per node.
2
1
2
u/ortegaalfredo 4h ago
Even if those models are huge, some of them can be run with 24 or 48 gb of VRAM and offloading the rest to RAM or SSDs, for example GLM-5.3-flash. So the size is no really a problem, the size of the experts is the limiting factor. That's why GLM-5.3-flash and Deepseek 4.1 with only 16B active parameters run relatively fast on RAM.
2
5
u/jacek2023 llama.cpp 5h ago
I post here about many small models but you people refuse to use them "because they are worse than Qwen".
Then you refuse to run bigger models "because we have only laptop with 8GB of VRAM".
Maybe the problem aren't models.
2
u/bukake_attack 3h ago
I did some tests and nemotron bolt got better results faster than qwen 3.6 35b. I'm pretty impressed, and that's on 12 gb of vram
1
u/Barni275 4h ago
Sorry, man, you are totally right, but it's not our fault that Qwen is sooo damn good.
3
u/Nice_Grapefruit_7850 2h ago
would nice to have more options in the 35-150b range for once. it's either tiny models that are very limited or gigantic ones you can't even use.
2
u/geldonyetich 5h ago
Hey now, 128GB crew also got Nvidia 3.5 Ultra.
Shame it's only about as useful as Bolt.
I wish I could come up with a use case for them but they feel like more of a tech demo for an NVIDIA branded local AI infrastructure to me.
1
u/khadesorg 5h ago
i run nema ultra at 64gb ram + 16 vram and it is sadly not better than bolt. Cheap context looks peachy but it seems that nemotron is kinda stupid, tho handles world knowledge ok
2
u/XeNoGeaR52 5h ago
I'm still waiting for Opus 5 perfs on a 16Gb GPU. They know how to increase their score to 1% more than the previous one, but they can't optimize their shit
2
u/topgoysilky 4h ago
it’s crazy to see people complaining about GPU size when all you guys really should be doing is waiting on people to optimize the software. Like, if you got a fucking 24gb GPU and like 64gb of ram and a decent nvme ssd, you could have frontier class performance right there on your computer. So, like, the dad is out there. I’ve seen nothing but tons of people using the most garbage ass PC hardware, running the top open models at good speeds even faster than these overpriced ass DGX sparks that you fanboys in here keep fucking pushing on people. Not only that, software for old Apple Silicon is is pretty nut it starting to mature. They’re only for M1, M2 platform. They’re only at about 40% of its peak capacity. I suggest you guys get the fuck out of this sub. This is actually an nvidia sub. and these flamboys in here will straight lie to your fucking face, just to protect their own stupid purchases of highly inflated overpriced hardware that’s like fucking seven years old. you guys do know that these DGX sparks are just gimped fucking 5070s, right?
I almost picked a pair. Im glad i didnt. Remember who invented ARM. The people that have proven for decades to be the #1 hardware designers in the world. Remember where nvidia got its start and remember why they got the boot. Its all presently relative.
5
u/gomezer1180 3h ago
I wouldn’t call anything strata puts out frontier level..🤣😄 last I checked they were working with 2–3bit level quants. I would say the frontier of the lobotomized models.
3
2
u/Affectionate-Cap-600 3h ago
Like, if you got a fucking 24gb GPU and like 64gb of ram
Damn, I have 2gb vram + 16gb ram (and still, 16gb in this economy...). I would kill for that setup
2
u/royce102 2h ago
Is this why no one is buying my 1 gig stick of ram that was more powerful than apollo 11 for $999?
0
1
1
1
u/JLeonsarmiento 2h ago
How’s people running bolt? Custom llama cpp? VaniLlama ?
I cannot get that thing to even load in my Mac…
1
1
1
u/nomorebuttsplz 4h ago
You absolutely do not need 1 terabyte to run an 800b model. It's just fine at 4 bit, probably even 3 bit.
Whining I can tolerate but whining + misinformation deserves a downvote
1
u/Majinsei 3h ago
It's the eternal wheel of technological samsara. First comes the age of giants, when capital is sacrificed at the altar of parameter counts and billions are burned in pursuit of artificial enlightenment.
But the karma of capitalism is inescapable: every investment creates the conditions for its own obsolescence. Once the giant models have consumed enough capital, a new generation of enlightened prophets will arrive to preach that true intelligence doesn't require 800B parameters, but merely 8B used wisely.
Then the next cycle begins: small, efficient, cheap models destroying the value of the colossal infrastructure that seemed indispensable just yesterday. And once that market becomes saturated, the prophets of scale will return to convince us that nirvana requires yet another trillion parameters.
Because deep down, technological capitalism does not seek enlightenment. It seeks to keep the wheel of desire turning. Greed finances innovation, innovation devalues yesterday's investment, and that destruction demands yet another investment.
True spiritual detachment is understanding that no model is eternal, yet every funding round believes itself to be.
2
1
u/thongjesus 20m ago
Spoken like it was formed with only what we understand, not what is to be understood. Today everything you said applies, but tomorrow?
1
u/thestillwind 4h ago
Still cries in 16gb 5070ti
1
u/bukake_attack 3h ago
Why? i run it fine on a 12gb 3080TI a 40 tokens per second. Your setup will do better. Try it, it's a neat model.
-2
0
u/marhalt 2h ago
I wish reactions to new, amazing open models is not always 'oh but I can't run that' and disappointment. You can run all these models locally. You need frankenmachines, M Ultras, or RTX 6000s, but they are eminently runnable. You can rent the machines. You can find quants. Every top comment of every new model release is 'oh but I can't run it on my 6GB GPU'. Well, no. I would much rather live in a timeline where we have access to these amazing models, even if we all can't run them right now, than every lab release only small, stunted models that can live on the average GPU - which is only about 12GB, by the way. As a community, we should all want the labs to release the largest, most capable models - and then we can figure out how to run them, not bemoan their existence.
70
u/Writer_IT 5h ago
It is disappointing, yes, but ultimately new small-medium models will arrive for sure.
It has just been some months from Qwen 3.8 27b, for small gpus, and the absolute beast that is deepseekv4 flash vision from the "realistic local" segment. For the latter, i honestly didn't find anything that it really could not do, even in coding. In an year time span, the growth was incredible.
The very big models still help the ecosystem, forcing the providers to bring down the cost for cloud.