Revisiting text decomposition methods for NLI-based factuality scoring of summaries

One promising approach to measuring factuality is based around Natural Language Inference or NLI. In the typical NLI setup, a model is presented with a pair of sentences, and outputs a distribution over the classes of {entailment, neutral, contradiction}. As NLI seems conceptually similar to factuality scoring (and is now quite a well-studied problem), several prior studies have asked “can we reuse NLI models for factuality scoring? And if so, how?”

One way to use NLI models for factuality scoring is to set the NLI context or "premise" to be the full input text, with the summary forming the NLI hypothesis, and then take the factuality score to be some function ƒ_z of the model output distribution. There are some potential problems with this approach:
 

• NLI models are usually trained with sentence pairs as input and can suffer performance degradation with the longer contexts that arise in summarization.
• The majority of modern NLI models are based on architectures such as the Transformer that use fixed-length input sizes, and it may not be possible for a full document and summary pair to fit into this context.

Another way to use NLI for factuality scoring is what we refer to here as “decomposition-based” scoring, and was introduced in 2019 by Falke et al. It can be explained as follows:

First, we decompose the document and summary into sentences, here labelled D_1...M and S_1...N respectively.

In our recent paper, we revisited this decomposition-based scoring idea, now using a newer set of NLI models, and inspired by other recent studies by Laban et al. and Schuster et al. We found that recent NLI models can indeed perform significantly better at this task than models created even just a couple of years ago. Given these improvements, we then asked whether this sentence level decomposition is still the best way make use of NLI models for factuality scoring. Concretely, we proposed methods for decomposing the input at units in between a sentence and the full document, and at units shorter than a sentence. We describe both ideas below.

Longer contexts: Top-K
As we are working with summaries, which are compressing and aggregating the input in some way, we can expect there to be instances where more than one document sentence is needed to correctly measure the factuality of a single summary statement. We propose a middle ground between using the full document as the premise and sentence level, that we call Top-K. It is computed as follows:

First, we decompose the document and summary into sentences as before and score all sentence pairs using the NLI model.

Shorter contexts: SCUs
To assess levels of granularity that are smaller than sentences, we first considered how people do content-based evaluation of summaries. The Pyramid Method (2004) is still thought of as one of the gold standards for human evaluation of summary content, and so this was our starting point. It’s also based on the idea of granular decomposition, although this time into singular statements of fact referred to as Summary Content Units or SCUs, that can then be extracted and compared between summaries and references.

Here is an example of SCU transformation:

“His weight went up six pounds and he report his diet is not good.”

could be mapped to SCUs:
 

1. “His weight went up 6 lbs.”
2. “He reports his diet is not good.”

As we have no gold SCU data for our evaluation datasets, we approximate SCU decomposition using an automatic method that was proposed by Zhang and Bansal last year. Their approach is based on semantic role labelling with some additional rules/heuristics. Their code is open source, and we used it without modification.

Here is how our example looks when we run it through the SCU approximation:

“His weight went up six pounds and he report his diet is not good.”

using the automatic method, is mapped to SCUs:
 

1. “His weight went up.”
2. “His weight went 6 lbs.”
3. “He reports his diet is not good”
4. “His diet is not good”

The output in this case is more verbose, and probably less correct, although there is still some subjectivity in judging this. For example, in some cases it may be desirable to distinguish between capturing the fact that weight went up, and that it went up by a specific amount.

We evaluated our methods on the SummaC benchmark, which is comprised of the following six datasets: CoGenSumm, XSumFaith, Polytope, FactCC, SummEval, and FRANK. All are broadly from the same domain, namely English news articles, and SummaC standardizes evaluation by casting each task as binary classification, and then measuring performance using balanced accuracy.

We note that one of the stated aims of the FRANK dataset was to go beyond binary factuality metrics, as the authors mention that binary scores can be difficult for people to agree on. And so, they included more granular factuality scores, where the factuality for a summary is a value between 0 and 1. We also evaluated our methods on the FRANK dataset using their original metrics (two correlation coefficients) to see if this leads to different conclusions.

For a granular breakdown of our results, we refer people to the paper, but at a high-level our findings are:
 

• Using newer NLI models (such as the VitaminC-MNLI model) leads to strong and robust performance gains in all evaluation settings.
• It's generally better to use sentence-level decomposition for the summary side of the NLI pair.
• On SummaC, we do better with longer contexts on the premise side. Using as much of the document as we can does best, followed by our Top-K method.
• On FRANK using the original metrics, we see a noticeable drop in performance when using the full document as the premise/context. Our Top-K method does best in terms of Pearson correlation, with sentence-level being the strongest in terms of Spearman.
• On the benchmark tasks, we see no performance improvement when going below the sentence level and using approximate SCUs.

In terms of our earlier example, we show below that when using our NLI-based scoring methods, we can now produce results that are more closely aligned with our intuitions about the quality of the three hypotheses. On the right of the table we see the results of using our Top-K NLI premise and sentence-level NLI hypothesis. We see that hypothesis two is correctly ranked the lowest (by far) of the three hypotheses in terms of factuality, and that hypothesis one and three are scored as being approximately the same.

We revisited recent work on using NLI models to do factuality scoring of summaries and found that techniques based on decomposing documents and summaries into finer levels of granularity work well (although there is still room for improvement). We proposed a new way to select context for scoring with dealing with longer input documents, that holds up well across evaluation on six different datasets. In general however, we find that there is no "correct" level of granularity for all tasks, and still see considerable variation in the performance across different datasets. And so, we note that care must be taken when assessing what to use for your downstream task of interest. So far we see no additional performance benefit in going below the sentence level and using SCUs on these benchmarks, but the SCU decomposition does perform competitively across both benchmarks (and as they are more granular, SCUs may be more interpretable in some cases).