Impact of Using Synthetic Data Generated by GPT-4o-mini on the Performance of Supervised Machine Learning Models

Abstract

Text classification models rely on large labeled datasets to achieve effective classification. However, collecting this data can be costly and laborious, especially for classification tasks in complex environments with data scarcity. Manual labeling of large data volumes requires significant time and human resources, which can be a barrier for research and various projects by limiting the performance of machine learning models.
The use of synthetic data can improve performance in machine learning models without incurring the costs associated with collection and labeling. This approach is not only more economical but also allows the generation of relevant data that improves model results.
This document presents the results of creating synthetic data with the objective of improving text classification in machine learning models. To carry out this exploration, a subset of Amazon product reviews in Spanish was used, taken from Hugging Face under the MTEB project called Amazon Review Multi. To conduct this study, 10% of the 200,000 available Spanish reviews were randomly extracted, thus recreating a data scarcity scenario. Our objective is clear: to demonstrate that, even with a limited amount of original data, the generation of synthetic data can notably improve the accuracy and effectiveness of machine learning models.
The choice to work with a reduced subset is not coincidental. We want to emphasize that synthetic data can complement and improve results, especially in contexts where data is limited. Despite having a large dataset available, this scarcity simulation allows us to highlight the true potential and benefit of synthetic data. With the extracted dataset, fine-tuning was performed on the GPT-4o-mini model, which resulted in the generation of synthetic data that was subsequently used to evaluate and compare the performance in classification of supervised learning models on two datasets: original and combined (original + synthetic). The results demonstrated significant improvements of up to +5.7% in the combined data group in metrics such as Accuracy, F1 Macro, and Recall. Likewise, improvements of up to +1% were observed in metrics such as ROC-AUC. These findings indicate that the combined use of real and synthetic data through this methodology can considerably improve the performance of learning models, providing a viable alternative in scenarios where data collection is complex and costly.