CaLMQA: Exploring culturally specific long-form question answering across 23 languages

The CaLMQA dataset, a multilingual long-form QA resource in 23 languages, addresses the scarcity of non-English research. It reveals that large language models struggle with low-resource languages and culturally nuanced questions, particularly affecting Tswana, Tongan, and Afar. The study introduces the CALM-Score metric to evaluate performance. Human assessments show that models often lack factual accuracy, omit important details, and exhibit language inconsistencies, emphasizing the need for more research on multilingual LLMs and culturally diverse QA. The table comparing three AI models (CLAUDE-3-OPUS, GPT-4-TURBO, and MIXTRAL-8X22B) reveals factuality issues, with GPT-4-TURBO having more illogical and irrelevant responses, MIXTRAL-8X22B showing hallucinations and cultural missteps, and CLAUDE-3-OPUS excelling in culturally specific questions. Overall, the paper underscores the importance of cultural awareness in AI systems and the need for improved performance across languages.