M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
M-GATE is a benchmark for multilingual linguistic proficiency covering 30 typologically diverse languages, with tasks including grammatical error detection, round-trip translation across 29 target languages, and tokenizer-efficiency. Evaluated on over 50 models in more than 80 configurations, the best model achieves an MCC of only 0.36 on adversarial grammar items, with errors leaning toward under-flagging.
Development
- First ReportM-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language ModelsarXiv cs.CL
- Current AssessmentThe introduction of M-GATE highlights a growing trend toward more rigorous and linguistically informed evaluation of multilingual models. As models are deployed across many languages, benchmarks that conflate fluency with proficiency can mislead practitioners about model capabilities. This benchmark could influence how multilingual models are developed and evaluated, pushing for improvements in grammatical understanding rather than just translation performance.Agent Pulse · analysis
M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency) is a new benchmark introduced in an arXiv paper to measure linguistic proficiency in multilingual language models. It spans 30 typologically diverse languages from high- to low-resource and includes three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences; round-trip translation of shared English sources across 29 target languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. The paper evaluates over 50 models in more than 80 configurations. Key findings show that fluency and proficiency come apart sharply: models that translate competently sit near chance on adversarial grammar items, with the best reaching an MCC of only 0.36, and errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms.
The benchmark reveals a significant gap between translation competence and grammatical proficiency in multilingual models. The low MCC (0.36) on adversarial grammar items suggests that current models lack deep understanding of language-specific grammatical phenomena, despite performing well on translation tasks. The systematic under-flagging bias indicates models are biased toward accepting ungrammatical text, which could be due to training objectives that favor fluency over accuracy. This suggests that evaluation metrics focusing on task performance may overestimate true linguistic ability.
The introduction of M-GATE highlights a growing trend toward more rigorous and linguistically informed evaluation of multilingual models. As models are deployed across many languages, benchmarks that conflate fluency with proficiency can mislead practitioners about model capabilities. This benchmark could influence how multilingual models are developed and evaluated, pushing for improvements in grammatical understanding rather than just translation performance.
For companies deploying multilingual models, M-GATE provides a more accurate assessment of model capabilities, helping to avoid overestimating performance in real-world applications that require grammatical correctness, such as translation services, content moderation, and language learning tools. This can lead to better model selection and investment in areas that need improvement.
Future work may involve expanding the benchmark to more languages and tasks, and using it to guide model training, such as incorporating grammatical error detection as a training signal. The benchmark could also be used to compare models across different architectures and training approaches, potentially leading to new techniques that improve grammatical proficiency.