The Kimi K3 model features 2.8 trillion parameters, utilizing a LatentMoE architecture with 16 activated experts out of 896, achieving a 2.5x more efficient scaling compared to previous designs. This architecture enables an extremely high sparsity ratio, surpassing models like Inkling which uses 8 out of 256 experts for its 1 trillion parameters.