简介：ParaBLEU: Generative Pretraining for Paraphrase Evaluation
ParaBLEU: Generative Pretraining for Paraphrase Evaluation
With the increasing interest in natural language processing (NLP), the task of evaluating the similarity between two sentences has gained significant attention. Paraphrase evaluation is a crucial subtask in this category, aiming to assess the degree of semantic similarity between two sentences. However, existing paraphrase evaluation datasets are often manually constructed and may suffer from annotation noise, making it difficult to obtain reliable evaluation results. To address this issue, we propose a novel pretraining approach, ParaBLEU, which combines BERT and multilingual language models to perform generative pretraining for paraphrase evaluation.
ParaBLEU focuses on learning language models to generate semantically equivalent sentences from given input sentences. In this way, it can not only reduce the reliance on manually constructed datasets but also improve the robustness of paraphrase evaluation models. Specifically, ParaBLEU first performs masked language model pretraining to force the model to capture the language structure and relationships between words. Then, it adopts denoising pretraining to simulate the annotation noise in real-world datasets and enhance the robustness of the model. Finally, ParaBLEU fine-tunes the pretrained model on downstream paraphrase evaluation tasks, achieving more accurate and reliable evaluation results.
Experimental results on multiple datasets demonstrate the effectiveness of ParaBLEU. Compared with existing methods, ParaBLEU significantly improves the accuracy of paraphrase evaluation and is more robust to annotation noise. In addition, we also show that ParaBLEU can be easily extended to support multiple languages,进一步提高了跨语言paraphrase评价的准确性。整体而言，ParaBLEU为paraphrase评价提供了一种新的有效方法，有望促进自然语言处理领域的发展。
本文提出了一种新型的预训练方法ParaBLEU，用于进行paraphrase评价。ParaBLEU着重于学习语言模型，从给定的输入句子中生成语义上等价的句子。这种方法不仅可以减少对手动构建数据集的依赖，还可以提高paraphrase评价模型的稳健性。具体来说，首先进行遮蔽语言模型预训练，迫使模型捕获单词之间的语言构造和联系；然后采用噪声消除预训练，模拟真实世界数据集中的噪声；最后，在下游的paraphrase评估任务上微调预训练模型，实现更准确可靠的评估结果。
在多个数据集上进行的实验结果表明，与现有方法相比，ParaBLEU显著提高了paraphrase评估的准确性，同时对噪声更具有稳健性。此外，我们还展示了ParaBLEU可以轻松扩展到支持多种语言，这进一步提高了跨语言paraphrase评价的准确性。整体而言，ParaBLEU为paraphrase评价提供了一种新的有效方法，有望促进自然语言处理领域的发展。