Do AI models answer Arabic speakers in their own dialect?

We sent three open models messages written by native speakers and read what came back. Most of the time the answer came back in formal Arabic. The real replies are below.

Most Arabic speakers text the way they talk: Egyptian, Gulf, Levantine, Moroccan. Formal Arabic is what they read in the news and at school. When a support bot answers a Cairo customer in formal Arabic, it reads stiff and official.

We wanted to know how often that happens, whether it can be fixed, and what fixing it costs. For messages we used MADAR, a set of everyday sentences written natively in each dialect by CAMeL Lab at NYU Abu Dhabi. For models we used three open ones you can run yourself: Gemma-4 31B, Qwen3.8 27B and gpt-oss-20b.

01Out of the box, they answer in formal Arabic

We sent each model 200 messages, 40 per variety, with a system prompt asking it to reply in the user's language and dialect. A grader then read every reply and answered one question: is this in the customer's dialect?

gpt-oss replied in the customer's dialect of the time. Qwen . Gemma did best at . Then we added one instruction naming the dialect, with a few example words. All three went above 90%.

So the models can write the dialects. They default to formal Arabic unless someone tells them which dialect to use. That instruction only works if you know the customer's dialect, and most apps don't.

02Guessing the dialect is where it breaks

If the app doesn't know where the customer is from, the model has to guess. We had Gemma name the dialect first, then answer in it.

Told the dialect, Gemma answered in it of the time. Guessing, . When the guess was wrong, so was the reply. A customer from Doha asking about prescription sunglasses got a friendly answer in Levantine:

Gulf went worst. Gemma guessed it right of the time, which is lower than giving no instruction at all ().

03A small classifier guesses better than the model

We compared Gemma with CAMeLBERT, a small dialect classifier from the same lab. It runs on a laptop in about a tenth of a second. On 1,500 sentences it named the right dialect of the time. Gemma got , and only on Gulf.

One caveat. The classifier was trained on MADAR, the same corpus we tested on, so it has a home advantage. Before we trust the gap we need to test it on Gulf text from somewhere else, like the Gumar corpus or the NADI shared-task data.

04Once it's switched on, how good is it?

With the dialect named, the grader gave Gemma 5 out of 5 in every dialect. Qwen and gpt-oss scored between 4 and 5. Levantine came lowest for both, usually because a Gulf or Egyptian word slipped into a Levantine reply. This one from Qwen scored 3:

Helpfulness scores barely moved with Arabic mode on (gpt-oss and out of 5).

05Answering in English first didn't help

A common trick is to let the model think in English, then translate. We tried it. Gemma wrote each answer in English, then a second call rewrote it in the customer's dialect.

It matched the dialect exactly as often as answering directly (), helpfulness was the same, and it used the tokens plus a second call. Here are both on the same Egyptian message:

06Dialect doesn't cost more to serve

Models charge by the token, and Arabic splits into more tokens than English. We assumed dialect would cost even more than formal Arabic, and said so earlier: 10 to 20% more. That estimate was wrong. It counted tokens per word on 18 prompts we wrote ourselves.

Counted on the same meaning written natively in each variety, Gulf, Egyptian and Levantine use about as many tokens as formal Arabic, often fewer. Moroccan uses up to more. The bigger gap is English. Formal Arabic takes more tokens than English on the models we serve. Jais-3, a model built for Arabic, closes most of that gap. Try a few sentences:

07How we graded, and how far to trust it

Claude Opus 5.5 graded every reply against a fixed rubric, without knowing which setup produced it. To check it, Grok 4.7 graded 360 of the same replies separately. They agreed on of them, and their 1 to 5 dialect scores were within a point of the time.

Where they split, it was mostly replies that mix a dialect with a few formal words. People would argue about those too. Click through the splits:

Both graders are AI models and may share blind spots. A small check by native speakers comes next, starting with Levantine, our weakest dialect.

08What we haven't tested

Phone calls. Voice agents turn speech into text before a model sees it. Speech-to-text may spell dialect words the formal way, or drop them. If it does, detecting the dialect per caller fails on calls. We'll run dialect speech (MGB-3 for Egyptian, MGB-5 for Moroccan) through common speech-to-text, then through the classifier, and compare with the written text. Until then we make no claims about calls.

Next, in order:

• Let the classifier do the guessing and rerun section 02.
• Test the classifier on Gulf text from outside MADAR.
• Native speakers grade a sample of replies.
• The phone test.