A sharper attention mechanism, four residual streams, and aggressive API pricing — but a model this large still puts self-hosting firmly in data-center territory.
On August 28, Tencent's Hy team released Hy4 Preview, a 770-billion-parameter mixture-of-experts language model under the Apache 2.0 license. It is more than twice the size of Hy3, expands the advertised context window from 256K to 1 million tokens, and ships as a roughly 1.56TB BF16 checkpoint. Tencent's official Hy4 announcement Hy4 Preview on Hugging Face
The headline is 770 billion parameters. The more interesting story is how Tencent is trying to make those parameters practical: by activating only a fraction of them per token, making long-context attention sparse, expanding the residual pathway, and building speculative decoding directly into the checkpoint.
It is also a useful case study in a distinction that is getting harder to ignore: open weights do not necessarily mean accessible hardware.
The generation-over-generation jump
Specification | Hy3 | Hy4 Preview |
|---|---|---|
Backbone parameters | 295B | 770B |
Active backbone parameters | 21B | 49B |
MTP parameters | 3.8B | 10B |
Context window | 256K | 1M |
Model weights | ~598GB BF16 | ~1.56TB BF16 |
Tencent's Hy3 release lists 295B total parameters, 21B active parameters, a 3.8B MTP layer and a 256K context window. Hy4 moves to a 770B backbone with 49B active parameters, a 10B MTP layer and a 1M-token context. The Hy4 specification table explicitly excludes the MTP layer from the 770B headline figure, so the auxiliary prediction module should not be silently mixed into that backbone comparison. Hy3 official repository Hy4 official repository
Hy4's 770B parameters are spread across 78 layers. The first layer uses a dense feed-forward network; the remaining 77 use MoE blocks with 256 routed experts and one shared expert. Each token activates eight routed experts plus the shared expert, producing the 49B active-parameter figure. Hy4 architecture and model specifications
So Hy4 is not simply "a 770B model." It is a model with an enormous capacity pool whose per-token computation is deliberately much smaller than the headline parameter count suggests.
What changed under the hood
Gated sparse attention for million-token context
The most consequential architectural change is the attention mechanism.
In conventional full attention, the amount of pairwise interaction grows rapidly as the context grows. DeepSeek Sparse Attention, or DSA, reduces that burden by using a lightweight indexer to identify relevant positions and then applying the expensive attention calculation to a selected subset rather than the entire context. DeepSeek introduced DSA in the experimental V3.2-Exp release in September 2025 specifically to improve long-context training and inference efficiency. DeepSeek V3.2-Exp announcement
Hy4 uses a Gated DSA variant and combines it with IndexCache, which Tencent describes as enabling cross-layer sparse-index reuse. Its published configuration sets the sparse index top-k to 2,048. Hy4 technical configuration
The distinction matters because DSA does not simply turn quadratic attention into linear attention. The sparse selection mechanism still has computational cost. What changes is the expensive full-attention stage: instead of computing attention over the entire context, the model operates on a much smaller selected set.
The IndexCache work tackles another part of the problem. Its research paper argues that sparse indices produced by neighboring layers are sufficiently similar that rebuilding them independently is wasteful; reusing those indices can remove a large share of indexer computation while retaining model quality in their experiments. IndexCache research paper
Tencent explicitly says Hy4 uses IndexCache for cross-layer sparse-index reuse. It has not, however, published a full technical report explaining every detail of its Gated DSA implementation. That means any more specific explanation of what the gate learns should be treated as an inference, not a documented fact.
The practical takeaway is simpler: Hy4's 1M-token context is backed by an attention architecture designed to avoid paying the full dense-attention cost across that entire window.
Four residual streams
Hy4 also changes the residual pathway.
Standard transformers maintain a single residual stream through the network. Hyper-Connections expand that into multiple streams and learn how information is routed between them. The original Hyper-Connections work proposed the approach as a way of addressing limitations associated with conventional residual connections, including a trade-off between gradient behavior and representation collapse. Hyper-Connections paper
Hy4 uses four residual streams in an implementation Tencent calls identity Hyper-Connections (iHC). Hy4 model specifications
That gives the network more pathways through which information can move between layers, but it also introduces a new optimization problem: having multiple streams does not guarantee that a trained model will use all of them equally.
Recent work has investigated this kind of stream dominance, making Hy4's identity-constrained implementation interesting without implying that the broader stability question has been settled. In other words, Tencent is deploying a research direction that is still being actively understood rather than dropping in a universally established replacement for the residual stream. Hyper-Connections research
A built-in draft model for speculative decoding
Hy4 includes a native multi-token prediction (MTP) layer specifically intended to support speculative decoding. Tencent says the MTP layer contains 10B parameters in total, with 0.7B activated. Hy4 official architecture details
Speculative decoding works by letting a cheaper predictive component propose several future tokens, after which the main model verifies them. Accepted predictions allow the system to produce more output per expensive main-model step.
The interesting part of Hy4 is that the MTP machinery is part of the released model rather than requiring users to supply an external draft model. Tencent's deployment examples explicitly enable MTP in both vLLM and SGLang, using the Hy4 FP8 checkpoint. Hy4 deployment recipes
That does not make 770B inference inexpensive. It does show that Tencent is treating serving efficiency as part of the model design itself.
The benchmark result is interesting — but it is Tencent's
Tencent's main comparative result comes from a blind side-by-side evaluation, not an independently administered leaderboard.
The company says 163 internal experts rated outputs on 203 engineering tasks. Hy4 averaged 2.99/4, compared with 2.92 for GLM 5.3 and 2.94 for Kimi K3. Tencent reports a 46.8% win rate against GLM 5.3 and a 51.2% win rate against Kimi K3, with the remaining results split between ties and losses. Hy4 benchmark details
That is evidence that Tencent considers Hy4 competitive with those models. It is not, by itself, proof of a definitive ranking.
The difference is important because the score gaps are small and the evaluation is designed by the model's developer. A different task distribution, evaluator pool or scoring methodology could produce a different ordering.
For now, the fairest description is that Hy4 has a strong self-reported showing at the open-model frontier, while the independent benchmark picture is still developing.
Price is a more tangible advantage
Hy4's pricing is unusually aggressive.
Model | Input / 1M tokens | Cached input / 1M | Output / 1M |
|---|---|---|---|
Hy4 Preview | $0.834 | $0.042 | $2.501 |
GLM 5.3 | $1.40 | $0.26 | $4.40 |
Kimi K3 | $3.00 | $0.30 | $15.00 |
Tencent lists Hy4 at $0.834 per million input tokens, $0.042 for cache hits and $2.501 per million output tokens. Tencent Hy4 announcement and pricing
Kimi's official K3 pricing page lists $0.30 cached input, $3.00 uncached input and $15 output, and confirms a 1,048,576-token context window. Kimi K3 official pricing
For GLM 5.3, Z.ai's current listed rates are $1.40 input, $0.26 cached input and $4.40 output. The vendor pricing record does not itself state a context length, so that detail should not be presented as vendor-confirmed here. GLM 5.3 pricing record with Z.ai source
At those list prices, Hy4 is about 3.6× cheaper than Kimi K3 on input and about 1.7× cheaper than GLM 5.3. Its output price is also much lower than either comparison.
The cache price is arguably the more interesting number. Long-running agents and document-heavy applications often resend the same context repeatedly. Dropping the price from $0.834 to $0.042 per million tokens after a cache hit makes repeated reads of large contexts dramatically cheaper.
That turns the million-token window from a headline specification into something closer to a pricing strategy.
Open weights, expensive hardware
Hy4 is released under the Apache 2.0 license, and Tencent has published the weights alongside deployment instructions for vLLM and SGLang. Hy4 license and repository
But the physical size of the model changes what "open" means in practice.
The BF16 checkpoint is about 1.56TB on Hugging Face. Tencent also provides an FP8 version and an eight-way tensor-parallel deployment example for it. Hy4 Hugging Face files Hy4 FP8 deployment recipe
So there are really two different kinds of accessibility here.
The model artifacts are open: you can download them, inspect them, modify them and deploy them under the published license.
The compute is not ordinary: the official serving path assumes a multi-GPU system, and the full BF16 checkpoint alone requires more memory than a conventional workstation can provide.
For the typical developer, that makes a hosted endpoint far more realistic than buying enough hardware to serve the full model locally.
The weights may be open. The infrastructure is still specialized.
What Tencent actually said about Hy4 before launch
There is a useful distinction between what Tencent announced about Hunyuan 4 and what eventually appeared in Hy4 Preview.
During Tencent's August 12 earnings call, management said the company was training a larger Hunyuan 4 and expected to release it later in 2026. In the same broader discussion, executives said Tencent was also upgrading its multimodal capabilities. They did not, in the material available from the call, explicitly say that the forthcoming Hunyuan 4 language model itself would launch as a multimodal checkpoint. August 12, 2026 Tencent earnings-call transcript
That distinction matters because the released Hy4 Preview is text-only. Simon Willison independently described it as a text-input model with no vision support after inspecting the release. Simon Willison's Hy4 analysis
So there is no need to frame this as a broken multimodal promise. The more defensible conclusion is simpler: Tencent discussed Hunyuan's multimodal work and a future Hunyuan 4, while the model that shipped on August 28 is a text model.
The chat template reveals a surprisingly simple reasoning control
Some of the most interesting details are buried not in the model card but in the chat template.
Simon Willison inspected Hy4's chat_template.jinja and found that its reasoning_effort parameter accepts only two values: high and no_think, with high as the default. Hy4 chat-template analysis by Simon Willison
That makes the interface essentially a switch rather than a graduated reasoning dial.
Other current reasoning models expose multiple effort levels or explicit reasoning budgets. Hy4's released template keeps the control much simpler: deep reasoning on, or reasoning off.
Willison also ran his recurring test asking a model to generate an SVG of a pelican riding a bicycle. Hy4 produced a competent vector illustration, but the reasoning trace contained clipped fragments such as:
“Maybe add sunglasses? no.”
Willison's observation was that the internal reasoning was somewhat truncated and ungrammatical. That does not demonstrate a capability failure. It is a small but interesting example of a broader possibility: hidden reasoning tokens do not necessarily need to read like polished human language if their job is simply to help the model solve the task. Simon Willison's pelican test and reasoning trace
The bigger story
Hy4 is easy to describe as "Tencent's 770B model."
That misses the more interesting point.
The architecture is built around a collection of techniques that all attack the same underlying problem: how do you increase model capability without making every token proportionally more expensive to compute?
MoE keeps only part of the 770B parameter pool active for each token. Gated DSA reduces the amount of expensive full attention required over long contexts. IndexCache reuses sparse-selection information across layers. Four residual streams expand the network's information pathways. Native MTP gives the serving stack another route to higher effective decoding throughput. Hy4 architecture overview
The result is not a fundamentally new type of model. It is something more practical: a concentrated bundle of scaling and inference techniques that are increasingly becoming part of the same design problem.
There is a second story, too.
Tencent's own earnings-call commentary makes clear that it is thinking about models, products and compute as one system. Management described Hunyuan as something to be co-designed with products such as WorkBuddy and CodeBuddy, while also emphasizing investment in training larger models and providing inference capacity. Tencent Q2 2026 earnings-call transcript
That helps explain the economics of Hy4. Aggressive API pricing is not separate from the architecture story. It is part of it.
The weights are open. The model is huge. The context is enormous. The serving stack is heavily optimized.
And the most important number may not be 770B at all.
It may be the distance between total model capacity, active computation and the cost of turning that capacity into useful tokens.
Hy4 Preview is Tencent's latest attempt to shrink that distance.
The weights may be open. The compute bill is not.
Sources
Primary
Technical background
Independent analysis and pricing
Comments (0)
Login to post a comment.