arXiv - CSCL: "How do different tokenizers perform on downstream…" - CleverLibre Social

arXiv - CSCL @arxiv_cscl@social.cleverlibre.org

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese. (arXiv:2306.09572v1 [cs.CL])

http://arxiv.org/abs/2306.09572 #arXiv #NLProc

Jun 19, 2023, 03:19 · · arxiv-cscl · · ·

Sign in to participate in the conversation