A new blog post from the Technology Innovation Institute's Alyah, published on Hugging Face, calls for better evaluation of Arabic large language models in the Emirati dialect. The post argues that current Arabic NLP benchmarks are not robust enough to capture regional dialects, which are widely used in daily communication.
The proposed benchmark is designed to test models on Emirati-specific language patterns, including local vocabulary and phrasing. According to the post, such dialect-focused evaluation is necessary because models trained primarily on Modern Standard Arabic may perform poorly in real-world Emirati contexts.
The work highlights a broader issue in Arabic NLP: the gap between formal written Arabic and the spoken dialects used across the region. By focusing on the Emirati dialect, the authors aim to encourage the development of models that serve local users more effectively.