GLM-5.3-Flash is Z.ai's large mixture-of-experts model, and its NVFP4 checkpoint can serve the model's full context window of 1,048,576 tokens on a single GPU. The problem most teams hit is mechanical, not intellectual: the checkpoint is 191 GB on disk, its KV cache at full 1,048,576-token context needs millions of tokens of cache memory, and on a workstation-class box the serve process dies with an out-of-memory error even when the GPU looks empty. We hit that failure on our own NVIDIA DGX Station GB300, traced it to a NUMA page-cache trap that is specific to this class of machine, and fixed it in a way you can copy.
This post is the recipe. Petronella Technology Group, Inc. has published it as an MIT-licensed repository at github.com/capetron/glm-5.3-flash-dgx-station-gb300, and we have run this exact configuration on our own GB300 since 2026-09-25. Below you get the files the repo contains, the reason behind each vLLM flag, the memory trap and its three-part fix, the speeds we measured at go-live, the honest limits of the checkpoint, and the points where you should stop tuning and call us.
Key takeaways
- The recipe serves the checkpoint
nvidia/GLM-5.3-Flash-NVFP4with vLLM at tensor parallel 1 on a single NVIDIA DGX Station GB300, at the model's full 1,048,576-token context. - The KV cache is fp8 and holds 5,933,986 tokens, which is room for 5.66 full 1M-token requests at once.
- The main failure is a NUMA trap: file I/O on the host fills page cache, the kernel places those pages in the GPU's HBM, and vLLM then dies allocating the KV cache with a self-contradicting out-of-memory message.
- The fix is three parts: drop the page cache before every start, run heavy I/O with
numactl --membind=0, and never pin the vLLM container to node 0. - Measured at go-live: 145.9 tok/s for one user, 1,464 tok/s aggregate at 32 users, and 3 of 3 needle retrievals passed at 968,377 prompt tokens.
What the repository gives you
The repository is deliberately small. It is not a framework, it is the set of files that turned a fragile manual launch into a service that comes back the same way every time. Everything in it has been running on our machine since 2026-09-25, and the numbers in this post came from that machine.
The four files that matter
The repo contains a systemd unit at systemd/glm53-flash.service, a pre-start script at systemd/glm53-flash-prestart.sh, a verification script at scripts/smoke-test.sh, and the go-live record at results/go-live-2026-09-25.md. The systemd unit is the core: it starts the vLLM container, applies the restart policy, and runs the pre-start script before the container ever launches. The results file records the measurements we quote below, so you can compare your own machine against ours without trusting anyone's memory.
Why systemd owns the container instead of Docker's restart flag
A common shortcut is to give the container a Docker restart policy and walk away. We do not do that, and the repo does not either. Docker has no --restart flag in this setup: systemd owns restarts. That choice buys two things. First, the restart policy lives in one readable unit file, with a cap on how often a crash can retry. Second, the pre-start script runs on every attempt, which matters because the memory trap in the next section has to be cleared before each start, not just the first one.
The smoke test
After the service starts, the smoke test at scripts/smoke-test.sh checks three things: the model is listed, one chat answer comes back, and one get_weather tool request comes back as a structured tool_calls entry. That last check is the one teams skip and regret. A model that answers chat but mangles tool calls will pass a casual curl and then break every agent you build on top of it, so the battery treats tool calling as a launch blocker, not a nice-to-have.
The service configuration, flag by flag
The unit starts vLLM with a short list of flags, and each one exists for a reason. Here is the list and the thinking behind it, so you can adapt it to your own machine knowing what you are changing.
Context length and concurrency
The flags --max-model-len 1048576 and --max-num-seqs 256 set the ceiling. The first is the model's full context, and the whole point of this recipe is to run it at that ceiling rather than trimming it to make memory math easier. The second allows up to 256 sequences in flight, which is also the concurrency of the peak we measured. If you know your workload never exceeds a few long documents at a time, lowering --max-num-seqs is a safe change; lowering --max-model-len changes what the service is for.
Memory utilization and caching
GPU memory utilization is set to 0.94. That is the share of GPU memory vLLM may claim for weights and KV cache; the rest stays unallocated as headroom. Chunked prefill and prefix caching are both on. Chunked prefill lets a very long prompt be processed in pieces so a single giant request does not monopolize the engine, and prefix caching means repeated shared prefixes, such as the same system prompt across requests, do not pay their cost twice.
Tool and reasoning parsers
The flags --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 tell vLLM how to parse this model's tool-call and reasoning output formats. GLM models emit tool calls in their own structured dialect, and without the matching parser the JSON arrives wrapped in text your client cannot use. These parser names are pinned to this vLLM build; if you move to a newer image, re-test, because parser names and behavior can change between builds.
The API surface
The service exposes an OpenAI-compatible API on port 8888, and the served model name is glm-5.3-flash. Because the surface is OpenAI-compatible, existing client code that points at a hosted endpoint can usually be redirected to this server by changing the base URL and model name. That is a practical advantage when you are migrating an internal tool from a cloud API to a private one: the code barely changes, the data stops leaving the building. Our broader work on on-premise AI is built around exactly that migration.
How much memory full context really takes
The reason this recipe exists on this hardware is arithmetic. The checkpoint is 191 GB on disk, and at full 1,048,576-token context the KV cache is the second consumer, and it is the one that moves.
The KV cache budget
With the KV cache in fp8, taken from the checkpoint's own configuration, this configuration holds 5,933,986 tokens of cache. Divided by a 1M-token request, that is room for 5.66 full 1M-token requests at once. Those two numbers are the capacity planning table for the whole service: if your workload consists of four concurrent million-token sessions, they fit; if you expect twenty, they do not, and you will see requests queue behind each other rather than fail. The fp8 cache is what makes this fit on one GPU at all, and it comes from the checkpoint config rather than being a flag we chose, which is why a different checkpoint changes the math.
Why we pinned the container image
The vLLM container image is pinned by digest, specifically the nightly-aarch64 build of 2026-09-24. Pinning by digest means a tag moving upstream cannot silently change what runs on your machine. This matters more than it sounds: a newer build may change memory use or parser names, and either change can take a working service down days after you stopped looking at it. When you deliberately want to upgrade, move the pin consciously and re-test before you rely on it.
The GB300 memory trap
This is the section that will save you a weekend. On the GB300, the GPU's HBM is NUMA node 1, and the system's LPDDR memory is node 0. That topology creates a failure mode that looks like a vLLM bug but is actually the kernel doing its job with no guidance.
What actually happens
Heavy file I/O on the host, such as downloading or copying weights or running rsync, fills node 0's page cache. The kernel then places file pages in HBM, because from the kernel's perspective that is available memory too. When vLLM starts, it sees less free GPU memory than it expects, and it dies while allocating the KV cache with a self-contradicting out-of-memory error such as Tried to allocate 151 GiB ... 159.43 GiB free. Read that message twice: it claims 159.43 GiB is free and that a 151 GiB allocation failed. That contradiction is the signature of the trap, and once you have seen it you will recognize it instantly.
Fix part 1: drop the page cache before every start
The first fix is to drop the page cache before every start with sync; echo 1 > /proc/sys/vm/drop_caches, which is exactly what the pre-start script does. The emphasis on every start is the part people miss: clearing the cache once at install time does nothing for the restart next Tuesday, after a backup job or a weight download has refilled the cache. Never start the container by hand without it, because a hand-launched container skips the pre-start script and re-exposes you to the trap.
Fix part 2: keep heavy I/O off the HBM node
The second fix is to run downloads and copies with numactl --membind=0, or inside a container started with docker run --cpuset-mems 0, so their cache stays in the LPDDR system memory. This applies even while the service is running: a download for the next model version would otherwise leave file pages in HBM, ready to break the next restart. The install instructions in the repo wrap the weight download in numactl --membind=0 huggingface-cli download ... for exactly this reason.
Fix part 3: do not pin the vLLM container to node 0
The third fix is a warning against overcorrecting. After learning that I/O belongs on node 0, the tempting next step is to pin everything to node 0. Do not. Pinning the vLLM container itself to node 0 with --cpuset-mems 0 makes CUDA initialization fail with Error 802. The vLLM process needs to reach the GPU's own memory, so it must not be confined to the system memory node. The rule is simple: I/O jobs get node 0, the serving container gets the whole machine.
The pre-flight check
Before any start, check grep FilePages /sys/devices/system/node/node1/meminfo and confirm the value is near 0. If it is not, something has been writing files into the HBM node's cache, and starting now will reproduce the crash. This one grep, run before a start, turns a mysterious intermittent failure into a visible, preventable condition.
If you are staring at this failure on your own GB300 right now and would rather have someone who has fixed it walk you through it, call Penny at 919-348-4912 or use the contact form, and we will help you get the service stable.
Installation and startup, end to end
With the trap understood, installation is short. The steps below mirror the repository's instructions.
- Download the weights with the NUMA guard:
numactl --membind=0 huggingface-cli download nvidia/GLM-5.3-Flash-NVFP4. The page cache from those 191 GB stays in system memory, not in the GPU's HBM. - Install the two service files from the repo, the unit and the pre-start script, into systemd.
- Enable and start with
systemctl enable --now glm53-flash. - Wait about 7 minutes from
systemctl startto serving, then run the smoke test.
The 7-minute startup deserves a sentence of explanation, because a 7-minute wait looks like a hang if you do not know the budget. Loading weights takes about 77 seconds. The rest is compile time and CUDA graph capture, which vLLM does once at startup so that individual requests afterwards run against pre-captured graphs. If your service takes noticeably longer than that on the same hardware, check the pre-flight grep from the previous section before you assume the model is the problem.
The restart policy
The unit restarts the service on failure 60 seconds apart, at most 3 starts per hour. After that, the unit stays down. That cap is deliberate: if the service is crashing in a loop, for example on an out-of-memory condition, unbounded restarts hammer the machine and bury the first failure's logs under the ninth attempt's. Three starts per hour is enough to survive a transient fault and still stop a loop. To recover after a capped-out unit, run systemctl reset-failed glm53-flash and start again, ideally after checking the pre-flight condition that the pre-start script enforces anyway.
Preserving the evidence
The pre-start script saves the previous container's logs before removing it. The reason is blunt: docker rm erases the evidence you need after a crash. When a container that crashed is removed, its logs go with it, and you are left reconstructing the failure from systemd's memory of exit codes. Saving the logs first costs nothing and preserves the one artifact that tells you why the last attempt died.
What speed to expect
All numbers in this section were measured on our GB300 at go-live on 2026-09-25, on this exact service. The harness ran a burst per concurrency level, generated 256-token answers, and took 3 repeats after a warm-up. One honest caveat that shapes how you read everything below: the prompts were short, with a median of 39 to 40 prompt tokens, so these are decode numbers, not long-context numbers.
Single user
With one user, the model decodes at 145.9 tok/s, and time to first token at the median is 0.061 s. For interactive work, that is a fast, fluid experience: answers start appearing essentially immediately, and the stream reads quickly.
32 concurrent users
With 32 users at once, the service sustains 1,464 tok/s aggregate, which is a median of 55.6 tok/s per user, with time to first token p50 at 0.985 s, and 0 failures across the run. Read that as the shape of shared deployment: each user gets 55.6 tok/s instead of 145.9, the median first token arrives in 0.985 s, and the machine as a whole does far more work. For a team hitting one private endpoint, that is the difference between an experiment and a service.
Peak
In the model evaluation, the peak aggregate we measured was 2,517 tok/s at 256 users, which is 29 tok/s per user. That is the ceiling of the configuration: 256 sequences in flight, the engine's --max-num-seqs limit, each one slower on its own. For capacity planning, it tells you how much total work one GB300 does when you fill it.
Long context, the actual reason you are here
Decode speed with short prompts is the easy part. The long-context check is the one that validates the whole configuration. It passed 3 of 3 needle retrievals at 968,377 prompt tokens, which is 90% of the max context, at three depths, with each request taking 37.0, 35.8 and 33.5 seconds. A needle retrieval buries a specific fact deep in a huge prompt and asks the model to find it, at shallow, middle, and deep positions. Passing at all three depths at nearly a million tokens is the evidence that the full context is usable, not just accepted.
Quality gates before go-live
Speed without quality is just fast nonsense, so the go-live checklist included batteries. The tool-call battery passed 38 of 38, and the agent battery passed 15 of 15, both with vendor sampling. On our closed-book CMMC battery, this checkpoint scored 26 of 35 with vendor sampling, the best result of any model we measured on that battery, and the total across all eval items was 117 of 127 on this checkpoint. We publish the full comparison on our GLM-5.3-Flash benchmark page and across the leaderboards.
The honest limits
Every configuration has edges, and a recipe that hides them is marketing. Here is where this one stops being comfortable.
No speculative decoding
This recipe runs without speculative decoding. Multi-token prediction fails to load with this checkpoint in this vLLM build, with a tensor shape mismatch, size of tensor a (1024) must match ... (2048). The practical cost is single-user decode speed, as the next section shows. If a future vLLM build fixes the shape handling, re-test, because the upside is real.
The faster checkpoint that we do not run for accuracy-sensitive work
A different NVFP4 checkpoint, from RedHatAI, does support multi-token prediction, and on the same machine it reached 231 tok/s for one user and 1,760 at 32 users. But it lost measurable closed-book compliance recall, so we do not run it for accuracy-sensitive work. The same RedHatAI checkpoint without multi-token prediction measured 144 tok/s for one user and 1,633 at 32 users, with prefill of 28,610 / 33,076 / 32,511 tok/s at 8K / 32K / 128K prompts. Note carefully that those prefill numbers belong to the RedHatAI checkpoint, not to the recipe's nvidia checkpoint. Our rule is that throughput you cannot trust is not throughput, and on the compliance battery the accuracy difference decided the question. If your workload leans on recalled facts, run the recipe as published.
No authentication on the endpoint
As configured, vLLM binds 0.0.0.0 with no authentication. There is no API key. On a private machine on a private network, that is a deliberate simplification. Anywhere else, it is an exposed model server, and anyone who can reach the port can use your GPU and read whatever the model will say. Put it behind a firewall, a reverse proxy with auth, or a private network before it touches anything shared. This is the same discipline we apply across every private AI deployment: the model is inside your perimeter, so the perimeter has to be real.
Tested on one machine
This configuration has been tested on a DGX Station GB300 only. Other GB300 systems should work, but we have not measured them, and we will not claim numbers we did not collect. Our methodology page explains how we measure and what we have retracted when a measurement did not hold up, and we extend the same standard to this post.
How this compares to the rest of our fleet
A single number means little without neighbors. Across our measured configurations, all published on the LLM benchmarks hub, the same model lands very differently by hardware.
Against a four-GPU H200 server
On a four-GPU H200 NVL server with FP8 weights, tensor parallel 4 and multi-token prediction, GLM-5.3-Flash decodes at 238 tok/s for one user and 2,254 tok/s aggregate at 32 users. That is faster than the GB300 on both counts, which is what four data-center GPUs with MTP should buy you. The GB300's case is different: it is one box with no multi-GPU interconnect to configure, and it holds the full 1M context. If you already run an H200 cluster, this recipe is not a replacement; it is the single-node answer for a desk-side or branch deployment.
Against GB10 pods
On a four-unit GB10 (DGX Spark) pod, the model serves one user at 44.9 tok/s. On two GB10 units, it fits only six sequences at 24.1 tok/s per user, and our page on GB10 recommends long-context GLM on four GB10 units or more. On eight GB10 units, it failed to start for us, which we report as measured rather than smoothed over. On these numbers the GB300 is well ahead of every GB10 layout we measured for this model; pricing and power figures for the Station are on our GB300 power and efficiency page.
Why we run vLLM here
The same repo, glm-5.3-flash-dgx-station-gb300, includes the go-live record so you can diff your startup logs against a known-good run. We are often asked when to choose vLLM over simpler servers. The short version of our live comparison is on our page about Ollama versus vLLM on Blackwell: for a checkpoint at this size, at this context length, with structured tool calling, under concurrent load, vLLM's scheduler and cache management are the tools that make the numbers above reachable. If your workload is a single user with short prompts, the calculus changes, and it is worth reading that comparison before choosing. And for smaller footprints, our page on the RTX 6000 Pro Blackwell multi-GPU vLLM setup covers the workstation-class middle ground.
When to call us
This recipe is complete enough to run on your own: the repository, the service files, the smoke test, and the go-live record are all public, and everything in this post traces to measurements on our own machine. That said, there is a difference between a configuration that works and one that is operated well. If you are evaluating a GB300 or similar hardware for private AI, if your team has hit the memory trap on other GB300-class systems and wants a second pair of eyes, or if you want this model serving your own data behind your own firewall with proper authentication in front of it, that is the work we do every day. Petronella Technology Group, Inc. designs and runs private AI on hardware our clients own, including for teams working toward CMMC compliance, and we have been doing it from Raleigh, NC since 2002.
Call Penny at 919-348-4912, or use the contact form at petronellatech.com/contact-us. We will talk through your workload, the context lengths you actually need, and whether this configuration is the right fit, before anyone spends money on hardware. If you want the raw numbers first, start with the GLM-5.3-Flash benchmark page and the DGX Station GB300 overview, and bring questions.
Related reading
- Self-Hosted LLM Benchmarks for Private AI
- From Two DGX Sparks to a Switched Fabric: Cluster Topology
- Mac Studio M5 Ultra vs DGX Spark: 1.2 TB/s vs 273 GB/s
- DGX Spark Memory Bandwidth: 273 GB/s vs the 25 GB/s Link
- DGX Spark Shutdown Under Load: Our 8-Unit GB10 Diagnosis
FAQ
How much memory does GLM-5.3-Flash need to run at full context on one GB300?
The checkpoint is 191 GB on disk, and the fp8 KV cache in this configuration holds 5,933,986 tokens, which is room for 5.66 full 1M-token requests at once. GPU memory utilization is set to 0.94 so allocation has headroom.
Why did vLLM die allocating the KV cache when the GPU looked empty?
Heavy file I/O on the host fills node 0's page cache, the kernel then places file pages in the GPU's HBM, and vLLM sees less free GPU memory than it expects. The result is a self-contradicting out-of-memory error such as Tried to allocate 151 GiB ... 159.43 GiB free.
How do I stop the out-of-memory failure from coming back?
Three parts: drop the page cache before every start with sync; echo 1 > /proc/sys/vm/drop_caches, run downloads and copies with numactl --membind=0, and never pin the vLLM container to node 0 because that fails CUDA initialization with Error 802.
How fast is GLM-5.3-Flash on one DGX Station GB300?
At go-live on 2026-09-25 it decoded at 145.9 tok/s for one user with time to first token p50 of 0.061 s, and 1,464 tok/s aggregate at 32 users, a median of 55.6 tok/s per user, with 0 failures. Peak was 2,517 tok/s at 256 users.
Does it really work at nearly the full million-token context?
Yes. The long-context check passed 3 of 3 needle retrievals at 968,377 prompt tokens, which is 90% of the max context at three depths, with each request taking 37.0, 35.8 and 33.5 seconds.
Why is there no API key on the endpoint?
vLLM binds 0.0.0.0 with no authentication as configured. That is acceptable only on a private network, so put the service behind a firewall, a reverse proxy with auth, or a private network before exposing it to anything shared.
Why do you not run the faster RedHatAI checkpoint?
The RedHatAI NVFP4 checkpoint with multi-token prediction reached 231 tok/s for one user and 1,760 at 32 users, but it lost measurable closed-book compliance recall, so we do not run it for accuracy-sensitive work. The recipe's nvidia checkpoint scored 26 of 35 on our closed-book CMMC battery, the best result of any model we measured on that battery.
Does this recipe work on other hardware?
It has been tested on a DGX Station GB300 only. Other GB300 systems should work, but we have not measured them. For comparison, the same model runs at 238 tok/s for one user on a four-GPU H200 NVL server, and at 44.9 tok/s on a four-unit GB10 pod.
Run it the way we do
The repository is public under the MIT license, the service files are ready to install, and the smoke test tells you within minutes whether your deployment matches ours. Start there, and if you want help sizing the hardware, hardening the endpoint, or building the private AI workflows that sit on top of it, call Penny at 919-348-4912 or use the contact form. Petronella Technology Group, Inc. has designed and run private AI for clients since 2002, and we are glad to help you put a million tokens of context to work on hardware you own.
Reference build sheets, a hardening checklist for open-weight model hosting and a data-sovereignty map for running AI on your own hardware.
Get the free guide
