Comparative Evaluation of Transformer-Based Models for Plain Language Classification in Hungarian Legal–Administrative Texts

Full abstract

Read the full abstract

Plain Language seeks to enhance the clarity and comprehensibility of legal and administrative communication; while Natural Language Processing (NLP) offers promising tools for assessing text complexity, most Plain Language classification studies focus exclusively on English, leaving low-resource languages underexplored. This study presents the first systematic evaluation of transformer-based models for sentence-level Plain Language classification in Hungarian tax administrative texts. We benchmarked zero-shot prompting with GPT-4o against fine-tuned open-weight and proprietary models, including huBERT, XLM-RoBERTa, GPT-4o-mini, and Gemini 1.0 Pro, and contextualized these results against previously established lightweight machine learning baselines based on term frequency-inverse document frequency with a support vector machine (TF-IDF + SVM) and fastText. To address data scarcity, we applied translation-based data augmentation using parallel Hungarian–English corpora. The best-performing model achieved a macro-average F1-score of 0.79. Mid-sized models also delivered competitive results, combining accuracy with feasible inference speed and deployment flexibility. Beyond classification performance, we conducted local and aggregated interpretability analysis based on Shapley-values to identify linguistic patterns influencing model decisions. This revealed alignment with known Plain Language features, such as nominalizations and syntactic complexity, as well as biases introduced by frequent domain-specific terms. Our findings demonstrate that Plain Language classifiers can be effectively adapted to low-resource legal–administrative domains. The results support the development of real-time feedback tools that promote linguistic accessibility and contribute to the broader goal of Access to Justice.