A silent bug can stall training of some gated models on new languages and chat tags

The paper is "Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models". It has six authors and was posted to arXiv as a preprint. In my words, a preprint is a paper shared before peer review. I read the abstract page and the full text of version 1. I did not rerun any experiment. Everything below comes from the paper unless I say it is my reading.

What goes wrong

In my words, a language model turns each token into a list of numbers, called an embedding. Many models keep one table of these for reading text in and a separate one for writing text out. The paper calls those models untied.

The paper says that "When a token never appears in pretraining text", its input row gets zero gradient, so no learning signal. Meanwhile, "AdamW weight decay shrinks untied embeddings". In my words, weight decay is a standard training setting that pulls every weight a little toward zero at each step. Over a long run, such rows fall below a millionth of the size of ordinary ones.

Gated models multiply part of what they read by a gate. In my words, a gate is a learned switch, and in these designs it closes when its input is zero. The paper shows that one near-zero token can then make gradients grow by more than 30, and up to 35, orders of magnitude. Once one gradient value passes about 18.4 quintillion, the abstract says, "FP32 global-norm clipping in PyTorch, Hugging Face Trainer, and FSDP2 can overflow to infinity" and zero the whole batch. In the paper's words, the token "raises neither exceptions nor NaN losses".

Where the 20% comes from

The headline number comes from one experiment. The authors ran Nemotron-3-Nano-4B through further training on Telugu for 1,000 steps, with 3 random seeds. In that run, with batches of 32 sequences, near-zero tokens appeared in 22.5% of batches, and 88.1% of those batches overflowed. So updates were lost on 19.8% of all steps, about 198 steps on average.

My reading: it is a measured result for one model, language and run, not a rate for gated models in general.

Which text sets it off

The authors sampled 8,000 documents from each dataset, across 15 text collections and 14 models. From per-document rates, they estimated how many batches of 128 sequences would contain a near-zero token. English web text, English chat data, code and maths came out at 0.0%. "Multilingual and template workloads hit routinely". The reported ranges are 10.6 to 18.8% for the multilingual Aya data and 70.5 to 97.4% for Thai. They reach 100% for Mongolian with Falcon-H1, and for one chat tag in Kimi-Linear-48B-Base when the tag comes first. These are estimates, not counts from training. In the Telugu run above, the measured share was 22.5%.

"Affected tokens are ordinary strings", the paper says. They include Telugu and Basque word pieces, Mongolian words, some code fragments with CRLF line breaks, and chat tags. "Fine-tuned and distilled derivatives inherit these rows from their base checkpoints".

How common the weak spot is

The audit started from the most-downloaded text-generation repositories on Hugging Face in October 2026, plus targeted lists of gated models. It covers 291 repositories. None of the 101 weight-tied checkpoints had reachable near-zero tokens. Among untied models, 30.9% of Transformers had them, but in the configurations the authors tested, non-initial positions or a start token prevent the failure. Among untied gated models, the share was 52.6%, or 20 of 38.

By my arithmetic, counting the 21 weight-tied gated models as well, it is 20 of 59 gated models, about a third. Both shares describe the authors' sample, not every model in use.

Who is affected, and who is not

• Running a model. Not affected, the paper says.
• Light fine-tuning. In my words, LoRA is a cheap fine-tuning method that keeps the main weights frozen. With the embeddings frozen, the paper found normal gradients across nine checkpoints. My reading: LoRA setups that also train the embeddings, for example to add new chat tokens, fall outside that check.
• Full training of all weights. Affected in vulnerable gated models when the data contains near-zero tokens. That includes reinforcement learning, where "full-parameter RL updates still trigger on prompts with near-zero chat tags".

The proposed fixes

For models already released, the authors propose GroupClip. In my words, it clips the input embedding's gradient separately, with higher precision or rescaling. The paper says it takes "three lines" of code. In the Telugu run, its 64-bit version cut stalled steps to zero.

For training from scratch, they propose GateFloor, which "keeps the gate positive at zero input". Its from-scratch runs "reach only 370M" parameters, with one or two seeds each. The further-training runs covered models of 1.5 to 4 billion parameters.

The two versions of the abstract differ on cost. The arXiv listing says GroupClip's rescaled version has "negligible measured end-to-end overhead". The full text says "two remedies with no step-time or peak-memory overhead". I did not test either claim.

What I could not check

• Replication. This is a preprint. I found no independent rerun of its results.
• The code. The arXiv page lists the authors' ancillary files, with scripts for GroupClip and GateFloor, the 291-repository census and a reproducibility check. I did not run them.
• Large runs. The further-training runs last 1,000 steps on models of 1.5 to 4 billion parameters, and the from-scratch runs stop at 370 million. The paper does not show how often the fault hits longer, larger runs.
• Fixes by model makers. The paper does not say whether any affected checkpoint has been repaired.

Sources

• paper Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models, arXiv 2611.04449 (abstract page) (archived copy)
• paper Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models, full text, version 1 (November 3, 2026) (archived copy)