SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

paper available at https://arxiv.org/pdf/2207.02578
code available at https://github.com/microsoft/unilm/tree/master/simlm


Paper abstract

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval.
It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training.
We use a replaced language modeling objective, which is inspired by ELECTRA,
to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning.
SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries.
We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings.
Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost.


Results on MS-MARCO passage ranking task

Modeldev MRR@10dev R@50dev R@1kTREC DL 2019 nDCG@10TREC DL 2020 nDCG@10
RocketQAv238.886.298.1
coCondenser38.286.598.471.768.4
ColBERTv239.786.898.4
SimLM (this model)41.187.898.771.469.7

数据统计

数据评估

intfloat/simlm-base-msmarco-finetuned浏览人数已经达到29,如你需要查询该站的相关权重信息,可以点击"5118数据""爱站数据""Chinaz数据"进入;以目前的网站数据参考,建议大家请以爱站数据为准,更多网站价值评估因素如:intfloat/simlm-base-msmarco-finetuned的访问速度、搜索引擎收录以及索引量、用户体验等;当然要评估一个站的价值,最主要还是需要根据您自身的需求以及需要,一些确切的数据则需要找intfloat/simlm-base-msmarco-finetuned的站长进行洽谈提供。如该站的IP、PV、跳出率等!

关于intfloat/simlm-base-msmarco-finetuned特别声明

本站OpenI提供的intfloat/simlm-base-msmarco-finetuned都来源于网络,不保证外部链接的准确性和完整性,同时,对于该外部链接的指向,不由OpenI实际控制,在2023年 7月 26日 下午3:56收录时,该网页上的内容,都属于合规合法,后期网页的内容如出现违规,可以直接联系网站管理员进行删除,OpenI不承担任何责任。

相关导航

暂无评论

暂无评论...