Why Arabic is hard, and what we do about it
Arabic words are built from roots with prefixes, suffixes and attached pronouns, so one written token can carry the meaning of a whole English phrase. Short vowels are usually left out, which makes many words ambiguous until context resolves them. Letters change shape by position, and writers vary alef forms and the ending letters of words. A pipeline that ignores this will count the same word as five different words.
Dialect adds another layer. Modern Standard Arabic is used in formal documents, but customers write in Gulf, Najdi, Hijazi and other varieties, often mixed with English and sometimes typed in Latin letters. We do not pick one model and hope. We label a sample of your own text, run several candidate approaches against it, and choose on measured results: precision, recall and error types, reported separately for formal text, dialect and mixed text.
