Assessing text complexity in Russian as a foreign language: an LLM-based approach and algorithmic toolkit

Full abstract

Read the full abstract

The paper presents a toolkit and an algorithm for assessing the lexical complexity of educational texts used in teaching Russian as a Foreign Language (RFL). In the context of educational digitalization, the relevance of this issue is driven by the need for objective, scalable tools to calibrate the complexity of learning materials. The study employs computational linguistics methods, including a custom Python script (process_word_lists) and large language models such as GLM 4.6, Grok 4 fast, Claude Sonnet, GPT-5, and Gemini 2.5 Pro. The research material comprises two text datasets: (1) a training set, which includes standardized RFL lexical minimums and 268 educational texts spanning levels A1–C1 (according to the CEFR), and (2) a test set consisting of 26 reading texts at levels A2–B1. Expert evaluation and statistical metrics—specifically Cohen’s kappa, Mean Absolute Error (MAE), ordinal accuracy, and nominal accuracy—were used to evaluate classification quality. The proposed algorithm enables highly reliable ranking of texts by CEFR difficulty levels. The revealed variations in the ability of large language models to assess RFL text complexity indicate high accuracy demonstrated by the GLM 4.6 and Grok 4 fast models. The developed algorithm and toolkit allow for the automated calibration of educational materials according to CEFR levels, enhance the reliability of expert evaluation, and expand the potential for developing adaptive educational resources. This functionality is in demand by both textbook authors and RFL test developers when selecting primary and calibrating secondary texts for textbooks and digital educational platforms. Future research prospects involve expanding the test dataset, analyzing texts at B2–C1 levels, and integrating new large language models.