SmartApe AI
Oct 10, 2026 · Artificial Intelligence
CNF-MCE Tokenizer: Counting & Min-Cost Encoding Beats BPE with 26-60% Compression Gains
Former Tencent AI Lab director Shi Shuming introduces CNF-MCE, a tokenizer replacing BPE's greedy merging with counting-based vocabulary construction and min-cost encoding, achieving 26-60% compression improvements over BPE while maintaining model performance on most benchmarks.
BPECNF-MCELarge Language Models
0 likes · 9 min read