Announcement_7

Our paper To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs has been accepted as Findings at EMNLP! With Franck Signe Talla, François Yvon, Edouard Grave and I, we introduce a modular training designed to make models more efficient (better token compression ratio) and modular (LLM handling sub-tokenizers) in a multilingual setting.