MDCR: A Dataset for Multi-Document Conditional Reasoning

The paper presents the MDCR dataset, a new benchmark for evaluating AI models' ability to handle multi-document conditional reasoning in real-life scenarios. MDCR addresses the need for models to comprehend complex questions that require cross-document understanding and optimization, focusing on tasks like eligibility determination and maximizing benefits. The dataset tests models on three types of questions, assessing their accuracy in handling condition relationships, unmentioned conditions, and optimization. Current large language models, such as GPT-4 and Llama3-70B, struggle with this task, scoring low on short answer accuracy and conditional reasoning. The study highlights the limitations of existing models and encourages future research in developing more capable systems for multi-document conditional reasoning across various domains.