Kimi K3 under fire: 2.8T-parameter open model faces distillation claims

A record release meets a political dispute

According to the CoreWeave post, Kimi K3 is the first open model to reach 2.8 trillion parameters. Demand overwhelmed Moonshot's compute capacity, and the company suspended new subscriptions 48 hours after launch.

On July 22, the director of the White House Office of Science and Technology Policy said on X that his office had information that Moonshot distilled Anthropic's Fable model to build K3. He alleged the company used a purpose-built internal platform that rotated access methods to evade detection. Treasury Secretary Scott Bessent floated sanctions. Moonshot denied the claim.

The company then released the weights on schedule and published a full technical report covering architecture, pre-training, post-training, infrastructure and evaluations. CoreWeave stresses that "Reports are not proof of provenance", but calls the document a large, specific and falsifiable artifact. It says the report describes a post-training pipeline that accounts for most of K3's jump without Claude, and also helps explain why the model sometimes feels like Claude.

What the technical report says about the design

K3 is a natively multimodal Mixture of Experts model with 2.8 trillion total parameters, 104.2 billion activated per token, 93 layers, a 160K vocabulary and a context window of up to 1 million tokens. CoreWeave highlights three architectural ideas:

• Hybrid attention: each block stacks three Kimi Delta Attention (KDA) layers followed by one Gated MLA layer, a 3:1 ratio, with a final Gated MLA layer ensuring global attention at the end.
• Attention Residuals: each layer uses a learnable pseudo-query to selectively retrieve from earlier representations. K3 uses Block AttnRes, with 8 blocks of 12 layers, to cut memory and cross-stage communication.
• Stable LatentMoE: routed experts work in a compact latent space while two full-width shared experts handle common transformations. This makes 896 routed experts with 16 active per token affordable.

At that sparsity, the report names two failure modes: exploding activations in the routed branch and load balancing across nearly a thousand experts. The fixes are an RMSNorm before the up-projection plus a new activation called SiTU-GLU, and a method called Quantile Balancing.

K3 applies No Position Encoding to all MLA layers, so it extrapolates to 1M tokens with no RoPE rescaling or YaRN interpolation. For anyone planning to serve the weights, quantization-aware training runs from SFT onward with MXFP4 expert weights and MXFP8 activations. The vision tower, MoonViT-V2, is a 27-layer encoder of roughly 0.4B parameters trained from scratch.

Benchmarks and pricing

K3 ranks first on four of the twelve benchmarks in the report: ProgramBench at 77.8, SWE-Marathon at 42.0, BrowseComp at 91.2 and AutomationBench at 30.8. It is second on Terminal-Bench 2.1 (88.3), FrontierSWE (81.2), Kimi Code Bench 2.0 (72.9), JobBench (54.3) and CharXiv (91.3), and ties GPT-5.5 on Zerobench at 41.0.

Its weaker results are DeepSWE, where its 67.5 trails GPT-5.6 Sol at 73.0 and Fable 5 at 70.0, and GDPval-AA v2, where its 1668 Elo trails Fable 5 at 1747 and GPT-5.6 Sol at 1736. All results use each model's maximum or high reasoning effort.

Cost is where the gap inverts, CoreWeave says, with one caveat. On Moonshot's internal Kimi Code Bench 2.0, K3 sits 4.0 points behind Fable 5 at 38% of the cost. At high effort it matches Claude Opus 4.8's maximum-effort score at roughly a third of the cost. On BrowseComp it takes the top score at $2.03 per task, half the cost of GPT-5.6 Sol. API pricing is $0.30 per million cache-hit input tokens, $3.00 cache-miss and $15.00 output.

A redesign, not an incremental step from K2.7

CoreWeave notes that K2.7 was still fundamentally a K2 model, with a 61-layer MoE backbone, 384 routed experts, MLA attention, SwiGLU activations and a 128K context window. K3, released barely a month later, is a ground-up redesign with 93 layers, 896 routed experts, hybrid KDA and MLA attention, SiTU-GLU and a 1 million token context.

On coding, K2.7 trailed GPT 5.5 on Kimi Code Bench v2, scoring 62.0 versus 69.0. K3 reaches 73.7 under Claude Code and 72.9 under Kimi Code, which CoreWeave describes as moving from chasing GPT 5.5 to trailing only Claude Fable 5. It cautions that the comparison is directional because harnesses and benchmark versions differ.

On reasoning, K3 preserves reasoning history across long agent sessions and trains separate low, high and max reasoning experts before merging them into one model in post-training. On Moonshot's black box system replication benchmark, K2.6 scores 0.560, GPT 5.5 scores 0.893 and Claude Opus 4.8 scores 0.918, while K3 reaches 1.000, fully solving the task.

Why the word distillation matters here

CoreWeave explains that distillation means one model learning from another rather than from raw data alone, and that the K3 dispute hinges on which of at least three methods is being alleged. In classical knowledge distillation, a smaller student is trained on the teacher's full probability distribution over tokens rather than on a single final answer, minimizing the KL divergence at every token position. That dense signal carries information a one-hot label does not, such as which wrong answers are nearly right.

For readers following the story, the practical point is that the allegation and the evidence are separate questions. The weights and report are public, so outside researchers can test Moonshot's account of how K3 was trained. The post's own conclusion is limited: the report offers a pipeline that explains much of the capability jump, but it is not proof either way.