Kimi's new K3 open model introduces a novel "delta attention" mechanism that collapses the entire past sequence into a fixed-size matrix, eliminating the growing KV cache and its quadratic cost. This allows the model to handle context windows of up to one million tokens without memory explosion, by writing only the difference between the current guess and desired value. The technique, which is 100% open source, also allows old entries to fade over time, enabling linear cost for long sequences.