Conference Proceedings
Evaluating the Efficacy of Summarization Evaluation across Languages
F Koto, JH Lau, T Baldwin
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 | ACL | Published : 2021
Abstract
While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.
Grants
Awarded by Department of Foreign Affairs and Trade, Australian Government
Funding Acknowledgements
We are grateful to the anonymous reviewers for their helpful feedback and suggestions. The first author is supported by the Australia Awards Scholarship (AAS), funded by the Department of Foreign Affairs and Trade (DFAT), Australia. This research was undertaken using the LIEF HPC-GPGPU Facility hosted at The University of Melbourne. This facility was established with the assistance of LIEF Grant LE170100200.