Leveraging pretrained models for automatic summarization of doctor-patient conversations

In this paper, published in EMNLP Findings, we explore the feasibility of using pretrained transformer models for automatically summarizing doctor patient conversations directly from transcripts.

In recent years, pretrained transformer models (Lewis et. al., 2019; Devlin et al., 2018; Zaheer et al., 2020; Brown et al., 2020) have been responsible for many breakthroughs in natural language processing (NLP), such as improved state-of-the-art performances for a broad range of tasks and the ability to train effective models for low resource tasks.

The demonstrated capability of transfer learning using large pretrained transformer models has led to widespread interest in leveraging these models in less standard NLP domains. Automatic generation of medical summaries from doctor-patient conversation transcripts presents several challenges such as the limited availability of supervised data, the substantial domain shift from the text typically used in pretraining, and potentially the long dialogues that exceed the length limitation of conventional transformers.

We show that pretrained transformer model BART (Lewis et al., 2019) can be fine-tuned to generate highly fluent summaries of surprisingly good quality even with a small dataset.

The dataset used in this paper is based on a collection of 1342 de-identified doctor patient conversations of two major specialties: internal medicine and primary care are annotated by medical scribes using our annotation environment specifically designed for the task. The scribes listen to the conversation audio and fill in necessary information in a simulated electronic health record (EHR) system. The EHR simulator consists of 14 distinct sections such as history of present illness (HPI) and review of system (ROS). We collect multiple references for each conversation, for a total of 21588 annotations. We focus on generating the HPI section here.

We explore multiple ways to generate summaries. All models rely on fine-tuning of pretrained BART models. But in this post, we discuss a specific multi-stage model that provided an effective way to overcome length limitations and gave the best results. The paper provides detailed discussions of all the different single and multi-stage models that were tried.

Task of dialogue summarization in the multi-stage model is achieved in two steps: summarizing portions of input conversation and rewriting aggregated summaries of each portion. The figure below shows the multi-stage chunking approach.