MTRAG Dataset Guide
Tasks and Conversations¶
| Term | What it means | Example |
|---|---|---|
| Conversation | The whole exchange between a user and an assistant | A conversation about the Arizona Cardinals |
| Turn | Which question we have reached in that conversation | Turn 2 means the second user question |
| Task | Answering the question at that turn | Answer the second question, using the earlier exchange for context |
| Evidence / passages | Document excerpts provided to help answer the question | An excerpt about where the team has played |
| Reference answer | The dataset’s prepared answer, used for comparison | A sample answer or an explanation that the information is unavailable |
| Candidate response | An answer produced by a model being evaluated | The answer shown under GPT-4o or Llama in the viewer |
For example, the first question in one conversation is “Where do the Arizona Cardinals play this week?” This is task 1. The next question asks whether the Cardinals play outside the US. That is task 2 in the same conversation.
When viewing task 2, the earlier question and answer appear together in the complete conversation, with the selected turn highlighted so you can understand what the new question refers to. The selected task is still just the current question.
Answerability Labels¶
These labels describe whether the question can be answered from the benchmark’s document collection. They are not scores for how well a particular AI answered.
| Label in the viewer | Plain meaning | What a suitable answer does | Tasks |
|---|---|---|---|
| ANSWERABLE | The documents contain enough information to answer the question. | Gives an answer supported by the documents. | 709 |
| PARTIAL | The documents support only part of the requested answer. | Answers the supported part and explains what is missing. | 68 |
| UNANSWERABLE | The documents do not contain the information needed to answer. | Explains that there is not enough information, rather than inventing an answer. | 55 |
| CONVERSATIONAL | The message is conversational, such as an acknowledgment, rather than a request for a document-based fact. | Responds naturally to the conversation. | 10 |
The 55 unanswerable tasks and 10 conversational tasks have no reference passages. A search may still return passages for them, but a returned passage does not necessarily answer the question.
Question Type Labels¶
| Label | Description |
|---|---|
| Factoid | Requests a specific fact, such as a name, date or amount. |
| Explanation | Requests an explanation of how or why something happens. |
| How-To | Requests instructions for completing an action. |
| Troubleshooting | Requests help diagnosing or resolving a problem. |
| Comparative | Asks how two or more things differ. |
| Summarization | Requests a shorter account of information. |
| Opinion | Requests a viewpoint or recommendation. |
| Composite | Combines multiple requests in one question. |
| Keyword | Uses keywords or a short search phrase rather than a full question. |
| Non-Question | A conversational statement rather than a question. |
Question type describes the request; Answerability describes whether the documents support an answer. A task can have more than one question-type label.
Files and Evidence¶
The viewer offers different ways of supplying documents to a model:
Reference: passages selected as evidence for the question.
RAG: passages returned by a retrieval system, which may include irrelevant information.
Reference + RAG: selected evidence mixed with retrieved passages. This option is available for only part of the dataset.
These are different document settings, not different answerability labels.
| Dataset file or folder | What you can find there |
|---|---|
| conversations/conversations.json | The 110 complete conversations |
| evaluations/reference_subset_with_human_evaluations.json | Model answers and human ratings for a smaller set of tasks |
| retrieval_tasks/ | Questions and judgments about which documents are relevant |
Fields in a Task¶
| Field | Meaning |
|---|---|
| task_id | Identifies one question at one turn. Use it to locate that task. |
| conversation_id | Groups tasks that belong to the same conversation. |
| turn | The current question’s position in the conversation: first, second, third, and so on. |
| input | The earlier conversation followed by the current user question. |
| contexts | The document excerpts supplied for this task. |
| targets | The prepared reference answer or answers. |
| Answerability | Whether the documents can answer the question; see the label table above. |
| Question Type | A label describing the kind of question. |
| Multi-Turn | A label describing how the question relates to the conversation. |
| Collection | The document collection used for the task. |
A task at turn 1 has no earlier conversation. A task at turn 2 includes the first question and answer before showing the second question. Later tasks include more of that earlier exchange.
Collections and Coverage¶
There are 842 tasks across 110 conversations. Each conversation contains 3–12 tasks, averaging about 7.7. Of the tasks, 110 are first questions and 732 are later questions.
| Collection | Topic | Tasks |
|---|---|---|
| ClapNQ | Questions answered using Wikipedia articles; topics include people, places, events and general knowledge. | 224 |
| Cloud | IBM Cloud documentation: services, setup, configuration and troubleshooting. | 205 |
| FiQA | Finance questions and answers, including investing, markets and personal finance. | 199 |
| Govt | Government documents explaining public services, policies and procedures. | 214 |
| Multi-Turn label | How to read it | Tasks |
|---|---|---|
| Follow-up | Continues or builds on an earlier part of the conversation. | 622 |
| Clarification | Clarifies what was meant in the conversation. | 110 |
| N/A | No earlier turn to relate to; these are the first questions. | 110 |
AI Answers and Human Ratings¶
The human evaluation subset contains 159 tasks from 20 conversations. For each task, it includes a reference answer, a Llama 3.1 405B answer and a GPT-4o answer: 477 responses in total.
In the viewer, reference means the prepared comparison answer, not a model name. The other response headings identify the model that produced the answer. Reference answers were created with model assistance and human review.
| Human rating | What the evaluator checks | Scale |
|---|---|---|
| Naturalness | Does the response read naturally? | 1–4 |
| Appropriateness | Is the response suitable for the question and conversation? | 1–4 |
| Completeness | Does it cover what the question asks? | 1–4 |
| Faithfulness | Does it stay supported by the supplied evidence? | 1–4 |
| Win rate | How does it fare in comparisons with other responses? | 0–100 |
Read each response’s rating table by metric (row) and annotator (column). The released evaluation subset contains 7 distinct evaluator IDs, with 3 evaluators per response. Annotator numbers (for example, 2, 5 and 6) identify different evaluators; they are not scores or the number of people. These are consistent aliases for the evaluator IDs in the dataset, whose real names are not supplied. A column follows the same evaluator across metrics. “Missing” indicates that no score was recorded.
Ratings 1–4 mean No, Mostly No, Mostly Yes and Yes, respectively. Pairwise preference uses a separate 0–100 scale. “Head-to-head” means comparing two responses to the same question and judging which is better. The released win-rate value describes how often a response was preferred over the other responses; it does not measure factual accuracy. The annotations were provided by human evaluators for the MTRAG benchmark. The viewer displays these released judgments; it does not generate new scores.
| Field in the evaluation data | Meaning |
|---|---|
| tasks | The tasks included in the human evaluation subset. |
| documents | The document texts used as evidence. |
| evaluations | The responses and their evaluations. |
| task_id | Connects an evaluated response to its task. |
| model_id | Identifies the model, or marks the prepared reference answer. |
| model_response | The answer being evaluated. |
| annotations | Individual evaluation results, including human ratings. |
| metrics | Descriptions of the evaluation measures. |
References¶
The counts above describe the original MTRAG human dataset. The separate MTRAG-UN extension is not included.
[1] Y. Katsis et al. MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems. 2025. Paper.
[2] IBM. MTRAG human dataset. Dataset documentation.
[3] IBM. MTRAG generation tasks and evidence settings. Generation documentation.
[4] IBM. MTRAG reference subset with human evaluations. Evaluation data.