Doklady Rossijskoj Akademii Nauk. Mathematika, Informatika, Processy Upravlenia
RUS  ENG    JOURNALS   PEOPLE   ORGANISATIONS   CONFERENCES   SEMINARS   VIDEO LIBRARY   PACKAGE AMSBIB  
General information
Latest issue
Archive
Impact factor

Search papers
Search references

RSS
Latest issue
Current issues
Archive issues
What is RSS



Dokl. RAN. Math. Inf. Proc. Upr.:
Year:
Volume:
Issue:
Page:
Find






Personal entry:
Login:
Password:
Save password
Enter
Forgotten password?
Register


Doklady Rossijskoj Akademii Nauk. Mathematika, Informatika, Processy Upravlenia, 2024, Volume 520, Number 2, Pages 284–294
DOI: https://doi.org/10.31857/S2686954324700644
(Mi danma607)
 

This article is cited in 2 scientific papers (total in 2 papers)

SPECIAL ISSUE: ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING TECHNOLOGIES

RuSciBench: open benchmark for Russian and English scientific document representations

A. C. Vatolina, N. A. Gerasimenkoabc, A. O. Yaninad, K. V. Vorontsovadc

a Federal Research Center "Computer Science and Control" of Russian Academy of Sciences, Moscow, Russia
b SberAI, Moscow, Russia
c Artificial Intelligence Institute M. V. Lomonosov Moscow State University, Moscow, Russia
d Moscow Institute of Physics and Technology, Dolgoprudny, Moscow region, Russia
Citations (2) English version article
DOI: https://doi.org/10.31857/S2686954324700644
Abstract: Sharing scientific knowledge in the community is an important endeavor. However, most papers are written in English, which makes dissemination of knowledge in countries where English is not spoken by the majority of people harder. Nowadays, machine translation and language models may help to solve this problem, but it is still complicated to train and evaluate models in languages other than English with no or little data in the required language. To address this, we propose the first benchmark for evaluating models on scientific texts in Russian. It consists of papers from Russian electronic library of scientific publications. We also present a set of tasks which can be used to fine-tune various models on our data and provide a detailed comparison between state-of-the-art models on our benchmark.
Keywords: dataset collection, benchmarking, large language models (LLM), LLM evaluation, representation learning.
Received: 27.09.2024
Accepted: 02.10.2024
English version:
Doklady Mathematics, 2024, Volume 110, Issue suppl. 1, Pages S251–S260
DOI: https://doi.org/10.1134/S1064562424602191
Bibliographic databases:
Document Type: Article
UDC: 004.048
Language: Russian
Citation: A. C. Vatolin, N. A. Gerasimenko, A. O. Yanina, K. V. Vorontsov, “RuSciBench: open benchmark for Russian and English scientific document representations”, Dokl. RAN. Math. Inf. Proc. Upr., 520:2 (2024), 284–294; Dokl. Math., 110:suppl. 1 (2024), S251–S260
Citation in format AMSBIB
\Bibitem{VatGerYan24}
\by A.~C.~Vatolin, N.~A.~Gerasimenko, A.~O.~Yanina, K.~V.~Vorontsov
\paper RuSciBench: open benchmark for Russian and English scientific document representations
\jour Dokl. RAN. Math. Inf. Proc. Upr.
\yr 2024
\vol 520
\issue 2
\pages 284--294
\mathnet{http://mi.mathnet.ru/danma607}
\elib{https://elibrary.ru/item.asp?id=80287455}
\transl
\jour Dokl. Math.
\yr 2024
\vol 110
\issue suppl. 1
\pages S251--S260
\crossref{https://doi.org/10.1134/S1064562424602191}
Linking options:
  • https://www.mathnet.ru/eng/danma607
  • https://www.mathnet.ru/eng/danma/v520/i2/p284
  • This publication is cited in the following 2 articles:
    Citing articles in Google Scholar: Russian citations, English citations
    Related articles in Google Scholar: Russian articles, English articles
    Doklady Rossijskoj Akademii Nauk. Mathematika, Informatika, Processy Upravlenia Doklady Rossijskoj Akademii Nauk. Mathematika, Informatika, Processy Upravlenia
    Statistics & downloads:
    Abstract page:425
    References:2
     
      Contact us:
     Terms of Use  Registration to the website  Logotypes © Steklov Mathematical Institute RAS, 2026