On evaluating legal summaries with ROUGE
Bianca Steffes, Piotr Rataj, Luise Burger, Lukas Roth
Abstract
ROUGE is the most commonly used measure for evaluating summarization algorithms in practice. However, it is questionable whether ROUGE scores adequately reflect the quality of summaries in terms of content. We introduce a metric to measure (legal) content of summaries based on exhaustiveness and concreteness and compare ROUGE scores with values generated by legal experts according to our metric on two tasks. Our results show that ROUGE does not reliably gauge legal content and thus should not be used as a single indicator for the quality of summarization algorithms. For our particular use case we furthermore show one way to increase the reliability of ROUGE by pre-selecting sentences.
Topics & Concepts
ROUGEBaton rougeComputer sciencePolitical scienceNatural language processingLawPoliticsNatural Language Processing TechniquesArtificial Intelligence in LawMulti-Agent Systems and Negotiation