Our research is in the relatively unexplored area of question answering technologies for patient-specific questions over their electronic health records. A large dataset of human expert curated question and answer pairs is an important pre-requisite for developing, training and evaluating any question answering system that is powered by machine learning. In this paper, we describe a process for creating such a dataset of questions and answers. Our methodology is replicable, can be conducted by medical students as annotators, and results in high inter-annotator agreement (0.71 Cohen's kappa). Over the course of 11 months, 11 medical students followed our annotation methodology, resulting in a question answering dataset of 5696 questions over 71 patient records, of which 1747 questions have corresponding answers generated by the medical students.
In recent years, automatic question answering (QA) systems have made big strides in addressing many different types of question answering problems. IBM's Watson 1 successfully challenged human grand champions on a televised quiz show of Jeopardy! Similarly, systems have also been built for answering factoid questions from structured resources 2,3,4,5 , for answering questions about news stories 6 and even for handling 4 th grade science questions 7 . In the medical domain, similar systems 8,9 have been designed to answer questions on medical knowledge using resources such as MEDLINE abstracts, PubMed Central full-text articles, eMedicine documents, etc.
Patient-specific QA from electronic medical records (EMRs), however, is a new and relatively unexplored problem in clinical natural language processing. The idea is for such a system to be available to physicians during patient visits, for instance, for easy access to information about the patient. A system that would automatically answer patient-specific questions by analyzing the patient’s longitudinal medical records, including clinical narratives such as discharge summaries, progress notes, radiology reports as well as structured procedure, allergy and diagnoses lists, would be of immense value to medical professionals. Using such a system, a physician could get quick answers to questions such as
What medications is the patient taking to control his hypertension? Does she smoke?
Answers to these questions are facts about the patient that would be retrieved or inferred by the system from the patient’s EMR.
We refer to this kind of QA as patient-specific QA, and, although this type of QA is also in the medical domain, it differs from previous QA research initiatives in at least two ways. First, the answers to these questions are not well known facts about the world, but are very specific facts about a specific patient. This typically requires searching for a particular word, phrase or passage in the patient EMR, with the evidence for that fact being localized to one small part of the EMR. Compare this with the factoid/trivia QA, where the fact, as well as the evidence for that fact can be found in many different parts of the underlying knowledge resources. The second major difference for patient specific QA is the lack of question-answer datasets required to train, develop and evaluate such systems.
Most QA systems rely on machine learning technology that learn from training examples of question-answer pairs. To successfully train a machine-learning model for QA, we require the training questions to be realistic, and representative of questions that would be asked by a user of the system. For patient-specific QA, this requirement makes it especially challenging and expensive to obtain a set of such question-answer pairs. It is no easy task to automatically gather questions that a physician would ask about her patients during the course of her day. Furthermore, the correct answers to these questions are words, phrases or passages in the patient’s EMR, and must be manually identified by a medical expert -again, no easy feat.
In this paper, we describe the creation of a dataset of manually curated question-answer pairs for patient-specific QA. We developed a process for generating questions that are likely to be asked by a physician in a real clinical setting, and we also defined a process for annotating corresponding answers from both structured and unstructured parts of the patient’s EMR. The main contributions of this paper are as follows:
(1) We outline a replicable annotation process, with minimal annotator bias, for generating questions similar to what a physician may want to ask about a patient.
(2) We also outline a replicable annotation process for marking answers to the generated questions in the EMR, and then discuss the inter-annotator reliability and factors leading to annotator disagreement.
(3) We present an analysis of the dataset created, as well as some limitations of our methodology.
(4) Our methodology is replicable, can be conducted by medical students as annotators, and results in high inter-annotator agreement (0.71 Cohen’s κ).
(5) Eleven medical students created a question answering dataset of 5696 questions over 71 patient records following our annotation methodology. Of these questions, 1747 have corresponding answers generated by the same medical students. This data was created over a period of 11 months, but the work was conducted intermittently over this period.
In the following sections, we cover the related work from literature, a description of our methods, and our observations on the generated data set.
There have been multiple studies that generate resources and build systems for biomedical question answering. 8,9,10,11 The common question types, semantic models of questions, as well as resources and systems available for biomedical question answering have been well-documented. 8,9 Current state-of-the-art biomed
This content is AI-processed based on open access ArXiv data.
