关键词:

深度学习, 语义相似性, 对比学习, 蒸馏学习

Abstract:

In text semantic similarity models based on unsupervised contrastive learning, existing approaches often simply divide texts into positive and negative samples, while the model training process only focuses on the overall features of the text. This design has obvious limitations: on one hand, it ignores the fine-grained ranking features between texts, making it difficult to differentiate gradient differences in similarity; on the other hand, the model is insensitive to semantic changes between sentences, resulting in an inability to accurately capture the similarity between texts. To explore fine-grained relationships between samples and enhance the model's ability to perceive semantic changes, this paper proposes a text semantic similarity model based on ranking distillation and difference prediction. First, coarse-grained ranking features are extracted from a pre-trained teacher model and distilled into the student model, enabling it to capture fine-grained ranking features. Second, a difference prediction auxiliary network is designed: the original text is first randomly masked to obtain masked text, then a generator produces reconstructed text, and finally a discriminator predicts the differences between the original text and the reconstructed text, allowing the model to perceive semantic changes between the original and masked texts. Experimental results show that on the text semantic similarity task datasets STS12-STS16, STS-B, and SICK-R, the Spearman correlation coefficient improved on average by 1.16% and 0.82% over the Bert-base and Roberta-base foundations, respectively, compared to advanced models, demonstrating the effectiveness of this model.

Key words: deep learning, semantic similarity, contrastive learning, distillation learning