|
Doklady Rossijskoj Akademii Nauk. Mathematika, Informatika, Processy Upravlenia, 2024, Volume 520, Number 2, Pages 284–294 DOI: https://doi.org/10.31857/S2686954324700644
(Mi danma607)
|
|
|
|
This article is cited in 2 scientific papers (total in 2 papers)
SPECIAL ISSUE: ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING TECHNOLOGIES
RuSciBench: open benchmark for Russian and English scientific document representations
A. C. Vatolina, N. A. Gerasimenkoabc, A. O. Yaninad, K. V. Vorontsovadc a Federal Research Center "Computer Science and Control" of Russian Academy of Sciences, Moscow, Russia
b SberAI, Moscow, Russia
c Artificial Intelligence Institute M. V. Lomonosov Moscow State University, Moscow, Russia
d Moscow Institute of Physics and Technology, Dolgoprudny, Moscow region, Russia
DOI:
https://doi.org/10.31857/S2686954324700644
Abstract:
Sharing scientific knowledge in the community is an important endeavor. However, most papers are written in English, which makes dissemination of knowledge in countries where English is not spoken by the majority of people harder. Nowadays, machine translation and language models may help to solve this problem, but it is still complicated to train and evaluate models in languages other than English with no or little data in the required language. To address this, we propose the first benchmark for evaluating models on scientific texts in Russian. It consists of papers from Russian electronic library of scientific publications. We also present a set of tasks which can be used to fine-tune various models on our data and provide a detailed comparison between state-of-the-art models on our benchmark.
Keywords:
dataset collection, benchmarking, large language models (LLM), LLM evaluation, representation learning.
Received: 27.09.2024 Accepted: 02.10.2024
Citation:
A. C. Vatolin, N. A. Gerasimenko, A. O. Yanina, K. V. Vorontsov, “RuSciBench: open benchmark for Russian and English scientific document representations”, Dokl. RAN. Math. Inf. Proc. Upr., 520:2 (2024), 284–294; Dokl. Math., 110:suppl. 1 (2024), S251–S260
Linking options:
https://www.mathnet.ru/eng/danma607 https://www.mathnet.ru/eng/danma/v520/i2/p284
|
| Statistics & downloads: |
| Abstract page: | 425 | | References: | 2 |
|