Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI-22)
Se3
Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes
Gianluca Moro, Luca Ragazzi
Description
The quadratic memory complexity of transformers prevents long document summarization in low computational resource scenarios. State-of-the-art models need to apply input truncation, thus discarding and ignoring potential summary-relevant contents, leading to a performance drop. Furthermore, this loss is generally destructive for semantic text analytics in high-impact domains such as the legal one. In this paper, we propose a novel semantic self-segmentation (Se3) approach for long document summarization to address the critical problems of low-resource regimes, namely to process longer inputs than the GPU memory capacity and produce accurate summaries despite the availability of only a few dozens of training instances. Se3 segments a long input into semantically coherent chunks, allowing transformers to summarize very long documents without truncation by summarizing each chunk and concatenating the results. Experimental outcomes show that our approach significantly improves the performance of abstractive summarization transformers, even with just a dozen of labeled data, achieving new state-of-the-art results on two legal datasets of different domains and content. Finally, we perform ablations to study how the various components of our method contribute to the performance gain.
Keywords: long document summarization, abstractive summarization, low-resource, transformers, NLP, legal analytics.
Code: https://github.com/disi-unibo-nlp/se3
Citing
If you use Se3 in your research, please cite Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes.
@article{Moro_Ragazzi_2022,
title={Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes}},
volume={36},
url={https://ojs.aaai.org/index.php/AAAI/article/view/21357},
DOI={10.1609/aaai.v36i10.21357},
abstractNote={The quadratic memory complexity of transformers prevents long document summarization in low computational resource scenarios. State-of- the-art models need to apply input truncation, thus discarding and ignoring potential summary-relevant contents, leading to a performance drop. Furthermore, this loss is generally destructive for semantic text analytics in high-impact domains such as the legal one. In this paper, we propose a novel semantic self-segmentation (Se3) approach for long document summarization to address the critical problems of low-resource regimes, namely to process inputs longer than the GPU memory capacity and produce accurate summaries despite the availability of only a few dozens of training instances. Se3 segments a long input into semantically coherent chunks, allowing transformers to summarize very long documents without truncation by summarizing each chunk and concatenating the results. Experimental outcomes show the approach significantly improves the performance of abstractive summarization transformers, even with just a dozen of labeled data, achieving new state-of-the-art results on two legal datasets of different domains and contents. Finally, we report ablation studies to evaluate each contribution of the components of our method to the performance gain.},
number={10},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
author={Moro, Gianluca and Ragazzi, Luca},
year={2022},
month={Jun.},
pages={11085-11093}
}