arXiv - CSCL: "Tokenization Impacts Multilingual Language Modeli…" - CleverLibre Social

arXiv - CSCL @arxiv_cscl@social.cleverlibre.org

Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages. (arXiv:2305.17179v1 [cs.CL])

http://arxiv.org/abs/2305.17179 #arXiv #NLProc

May 30, 2023, 03:06 · · arxiv-cscl · · ·

Sign in to participate in the conversation