GLM-4.7-Flash on 2x RTX 3090: My Hands-On Experience

Last time I wrote about GLM-4.7-Flash, I was annoyed. It ran at a third of the speed I expected on my mini PC, fell off a cliff as context grew, and the fix turned out to be a runtime that actually implements Multi-head Latent Attention. If you missed it, the whole diagnosis is in the mini-PC writeup. I ended that one with a promise: fixing a model is not the same as it being worth running. The real question is whether GLM-4.7-Flash earns its place against a boring, plain grouped-query model of the same size that is already fast. So I moved it onto a proper pair of NVIDIA cards and measured everything. This is the hands-on writeup, on 2x RTX 3090.

A quick note on why I care beyond curiosity. I build ModelDirectory, and a real chunk of its backend runs on local hardware rather than rented GPUs. Every "should I run this model" decision is a cost decision for me, not a hobby. So I do not want the model that benchmarks well in a headline. I want the one that is cheaper per useful token on hardware I own, and right now that hardware is two RTX 3090s.

The rig: two RTX 3090s, 48 GB, and one slow link

The box is an ASUS board with a Ryzen 5 3600X and two NVIDIA RTX 3090s, 24 GB of GDDR6X each, 48 GB total. These are consumer cards, not data-center parts, and that matters in ways I will get to. There is no NVLink bridge, peer-to-peer is disabled in the driver the way it is on every consumer RTX 3090, and one of the two cards sits on a slower PCIe link than the other. So this is a realistic two-card desktop, not an idealized server. When I measured the card-to-card bandwidth directly it came back at about 1.4 GB/s, which is slow, and that number turns out to decide a lot.

For the model I pulled a Q4-class GLM-4.7-Flash (30B total, roughly 3B active, mixture-of-experts with MLA). As the control I used qwen3.5:27b, a plain grouped-query model in the same weight class that I already trust on the RTX 3090. Same runtime for both, ollama, so nothing about the tooling favors one side. The full setup, the exact ollama and vLLM configs, every number, and the charts are in a benchmarks writeup I put up, with the scripts and raw CSVs linked from there, if you want to reproduce any of it. The single question going in: on an RTX 3090, does GLM-4.7-Flash finally beat the boring fast model, or does the fancy attention design still cost you?

Test 1: decode speed against context

Here is decode speed as the context grows, GLM versus Qwen, both on a single RTX 3090.

GLM opens huge: 88.5 tokens a second at 4K on the RTX 3090, more than twice the plain model. Then it decays, hard. 62, then 44, then 28, and by 128K it is down to 16.8. Qwen barely moves the whole way: 34, 32, 31, 29, and still 24.4 at 128K. They cross right around 64K. Below that, GLM is the clear winner. Above it, the unglamorous grouped-query model is both faster and simpler.

That is the whole story in one picture, and it is not the story I expected. I assumed a single RTX 3090 would just make GLM win everywhere. It does not. It makes GLM win the short-context race by a mile and lose the long-context one.

Test 2: prefill, the part people forget

Decode speed is what you feel while the model is typing. Prefill is what you wait through before the first token shows up, and on long prompts it dominates. On the RTX 3090 the two models behave very differently here too.

| context | GLM-4.7-Flash prefill | qwen3.5:27b prefill |
|---|---|---|
| 4K | 2626 tok/s | 979 tok/s |
| 16K | 1849 tok/s | 947 tok/s |
| 32K | 1308 tok/s | 899 tok/s |
| 64K | 805 tok/s | 780 tok/s |
| 128K | 450 tok/s | 631 tok/s |

GLM ingests a short prompt almost three times faster than Qwen, which is great for interactive, short-turn work. But look at the trend: GLM's prefill falls off with context while Qwen's holds, and by 128K the boring model actually prefills faster. It is the same shape as the decode chart. GLM is a sprinter, Qwen is a diesel. On an RTX 3090 that tradeoff is consistent across both halves of inference, not just decode.

The RTX 3090 versus the mini PC

Before the head to head, it is worth showing what moving off the mini PC actually bought. Same model, same weights, two very different machines.

On the mini PC this model did 7.3 tok/s at 4K and ran out of memory by 128K. On the RTX 3090 it does 88.5 at 4K, a twelve times jump, and still holds 16.8 tok/s at 128K where the mini PC could not even load the context. Worth saying plainly: on the mini PC I had to switch to a special llama.cpp fork to get MLA working at all, while on the RTX 3090 mainline ollama already implements the MLA path, so the fork was not needed here. The cliff from the last post was a missing-kernel and cache-size problem, and a real RTX 3090 with a real MLA path fixes it out of the box.

Where MLA actually pays: VRAM

MLA, the memory trick DeepSeek introduced with V2, compresses the KV cache into a small latent. When the runtime implements it, long-context memory stays tiny. Watch what that does to VRAM on a 24 GB RTX 3090.

The dashed line is 24 GB, the ceiling of a single RTX 3090. GLM stays under it almost the whole way and uses 27.7 GB at 128K, while Qwen needs 31.5 GB at the same depth and crosses the single-card ceiling much earlier. So GLM packs more context onto one RTX 3090 before it has to spill to the second card. That is the MLA payoff, and it is real.

But notice the trade: at that same 128K, GLM is the slower model. MLA buys you long-context memory headroom and charges you compute per token to do it. On the RTX 3090 it is a memory-for-speed swap, not a free win. I also tried a q8 KV cache to squeeze more context onto one card, and it was not worth it: slower at every depth from the dequant overhead, and MLA's cache is already small enough that it did not even earn its keep. Leave the KV cache at the default on this hardware.

Test 3: 200K context on the RTX 3090

GLM-4.7-Flash's ceiling is about 202,752 tokens. I packed it near full on the RTX 3090 pair, planted a passphrase near the top, and asked for it back at the very end, the classic needle in a haystack. It stayed alive and answered correctly. So the capacity is genuinely there on 48 GB of RTX 3090, and the model does not fall apart at the top of its window. The open question is never whether it fits, it is whether you want to wait at the speed it runs that deep, which is back to the decode chart.

One RTX 3090 or two?

Two cards, so surely you just throw both RTX 3090s at it. It is not that simple, and this was the most interesting part to measure. Remember that 1.4 GB/s link between the cards.

For a single stream, one RTX 3090 wins short context outright: 88.5 against 55.9 at 4K, because splitting the model across both cards (tensor parallel in vLLM) means every token pays an all-reduce toll over that slow link, and at short context that toll costs more than the second card saves. They cross near 32K. By long context the two-card setup wins about two times, 36 tok/s against 17, because now the work per token is large enough that the link tax is a small fraction of it, and splitting the KV cache across both RTX 3090s doubles the memory bandwidth that the decode is actually bound by.

And if you are serving load rather than chatting, it is not close. Batched across concurrent streams, the two RTX 3090s hit around 498 tokens a second aggregate, roughly five to six times the single-card single-stream number. So the honest rule for a dual RTX 3090 box is: one card for short interactive work, both cards for long context or for serving many users at once. An NVLink bridge would push the crossover earlier, but you do not need it to win at long context, the split KV cache already does that.

So, should you run it?

Here is the verdict I actually landed on, and it is a boring, useful one: on 2x RTX 3090 it depends on your context length and your workload.

Short prompts, single user, interactive: GLM-4.7-Flash is clearly faster, run it on one RTX 3090.
Long context, single user: the plain grouped-query model is faster and simpler. Run the boring one, still on one card.
Serving load: split it across both RTX 3090s with tensor parallel and take the roughly 5 to 6 times throughput.

For my own use on ModelDirectory, most of the backend work is short-to-medium context, so GLM earns its slot on a single RTX 3090. The moment I need long-context passes I switch to the boring model without a second thought. That is the thing benchmarks rarely say out loud: the best model is a function of the job, not a leaderboard row, and even the right pair of RTX 3090s does not change that.

None of this came out of nowhere, and most of what helped was other people writing up their own runs. There is good HackerNoon writing on optimizing local inference for 8 GB GPUs and a clean guide to quantizing with llama.cpp, both of which I leaned on. And the first chapter on this exact box, before I moved off the mini PC, is the two-model benchmark I started with.

If you want to see what I actually build with this hardware, the model side is ModelDirectory, and the 3D side is GeometryViewer, a browser 3D viewer that also leans on cheap local compute. Both exist because running things yourself on hardware like a pair of RTX 3090s, carefully, is still cheaper than renting when you measure it honestly.

Next I want to push the same question onto much longer real workloads, not synthetic needles, and see whether the 64K crossover holds when the tokens are doing actual work. That one needs a better harness, and it is the next post.