Two-Stage Framework for Unsupervised Sentiment Analysis with Model Selection and Semantic Similarity Evaluation

Full abstract

Read the full abstract

Sentiment analysis is widely used to extract user opinions from large-scale textual data. However, in practical settings, sentiment labels are often unavailable, making it difficult not only to perform sentiment classification but also to evaluate whether the predicted labels are reliable. This study proposes a two-stage framework for unsupervised sentiment analysis with model selection and semantic similarity evaluation. The dataset consists of Gemini app reviews, in which a labeled subset was used in the first stage to investigate the behavioral characteristics and predictive patterns of three sentiment analysis approaches: lexicon-based, transformer-based, and large language model (LLM)-based methods. The model with the most suitable performance was then selected and applied to predict sentiment labels for the remaining unlabeled data in the second stage. The predicted labels were further evaluated using embedding-based cosine similarity to assess semantic consistency within sentiment classes and separability between classes. The results show that the LLM-based method using Gemini 2.0 Flash achieved the best performance, with accuracy, balanced accuracy, and F1-score values above 0.91, followed by the transformer-based IndoBERT model, while the InSet lexicon-based method showed the weakest performance. In the second stage, semantic similarity evaluation revealed a high average intra-class similarity of 0.6699 and a low inter-class similarity of 0.1917, resulting in a similarity gap of 0.4783. These findings indicate that the proposed framework can support reliable sentiment prediction in largely unlabeled datasets by combining sample-based model selection with semantic validation.