Abstract: As the mainstream and typical methods in current natural language processing and artificial intelligence, various pre-trained language models perform differently on the downstream tasks, due to their different language modeling, feature representation, model structure, training tasks and pre-training corpus, et al. In order to better ensemble the knowledge in different pre-trained language models and utilize their learning abilities on the downstream tasks, we propose a multi-model ensemble method MME-STS (Multi-Model Ensemble for Semantic Textual Similarity) for semantic textual similarity justification tasks. The model structure and the corresponding feature representations are presented, and three different ensemble strategies based on average values, full-connected layer training and Adaboost algorithm with respect to model ensemble are also proposed. Experimental results show that MME-STS outperforms significantly over single pre-trained language model-based approaches on the two benchmark datasets of SemEval 2014 task 4 SICK and SemEval 2017 STS-B corpus in terms of Pearson correlation coefficient and Spearman coefficient metrics.

Key words: Deep learning, Semantic Textual Similarity, Natural Language Processing, Pre-trained Language Model, Model Ensemble