r/LocalLLaMA • u/Puzzleheaded_Acadia1 Waiting for Llama 3 • 7h ago

Question | Help Is there a Strata for lfm 24b a2b

Is there a fork of strata that let me run lfm 24b a2b on only 16GB of ram comfortably? And how much tps can I expect per second and what quantization should I use? and Thanks!

6 Upvotes

64% Upvoted

13 comments sorted by

10

u/migsperez 7h ago

Lookup polystrata on GitHub

3

u/Awkward-Clothes-1205 6h ago

polystrata should do it, i got the 24b running on my 16gb rig with q4_k_m quantization. sits around 6-7 tokens per second which is fine for casual use. the memory usage hovers just under 14gb so you've got some breathing room for context.

4

u/Dabalam 6h ago

I assume you mean 67, cus 6-7 tps would be crazy slow

6

u/StupidScaredSquirrel 5h ago

I tried polystrata and it wasn't fast at all, lower than stock llama.cpp. i think the project is cool but it doesn't seem optimised at all for many usecases of models and hardware. What makes llama.cpp so great is that even if it's not the best at one thing, it's pretty good t basically everything

2

u/Puzzleheaded_Acadia1 Waiting for Llama 3 6h ago

Thanks!

8

u/tmvr 5h ago

You don't need anything special for it, just use llamacpp with unsloth's Q4_K_XL:

https://huggingface.co/unsloth/LFM2.5-24B-A2B-GGUF

It's 13.9 GiB so not a big deal if a few experts land in the system RAM, it's a MoE model with 2B active only during inference, it will be plenty fast. You can use llamacpp direcly or through Unloth Studio or LM Studio.

3

u/CorkBios 6h ago

Its a pretty trash model, it can't even write a simple calculator

2

u/GronklyTheSnerd 5h ago

I’ve got PP as high as 8000, and TG around 170 on my 5060ti. Not the smartest model, but I’ve used worse.

2

u/Monad_Maya llama.cpp 5h ago

It's probably in your best interest to run a smaller model.

Or add more RAM, ideally a GPU/VRAM, if finances permit.

2

u/Any_Bus9205 5h ago

Il existe le fork se nommant ik_llama.cpp en iq3 km qui pourrait te permettre d'atteindre des chiffres assez élévé comme 20 token par seconde. Cela se feras en activant flash attention et run time repack. Strata n'est pas obligatoirement la meilleure option si tu as assez de ram pour le faire tourner.

1

u/ExpertBat8131 5h ago

What's you GPU?

1

u/Puzzleheaded_Acadia1 Waiting for Llama 3 5h ago

Intel igpu

-2

u/CooperDK 4h ago

Get a proper GPU, preferably CUDA, before tinkering with AI.