NCP-ArchPreview achieves the same training loss as OLMo-3-7B using only 51.3% of tokens, introducing “Next Concept Prediction” as a scalable alternative to pure autoregressive pretraining.
Researchers from Shanghai AI Laboratory and Shanghai Jiao Tong University’s LUMIA Lab have released NCP-ArchPreview, the first 8.9-billion-parameter language model trained in discrete latent space rather than standard token space . The model debuted at #1 on HuggingFace Daily Papers and drew praise from overseas researchers, with one commentator calling it “possibly the first brick of the future of AI” .
The technical breakthrough is Next Concept Prediction (NCP) — a pretraining objective that predicts discrete concepts spanning multiple tokens, complementing rather than replacing standard Next-Token Prediction . Trained on the fully open 5.73T-token Dolma-3 corpus, NCP-ArchPreview achieved the final pretraining loss of OLMo-3-7B using only 51.3% of the token budget, representing a 1.95x effective convergence speedup .
Downstream performance improved by 2.45 points on macro-average, with GSM8K reasoning jumping nearly 6 points . The learned latent space also enables lightweight domain adaptation — updating just 17 million parameters yields an efficient interface for specialization .
The result challenges the long-held assumption that autoregressive token prediction is the only scalable path to language intelligence, echoing the Latent Diffusion revolution in image generation
