Random Forest on Frozen IndoBERT Embeddings for Indonesian Political Hoax Detection
Article Sidebar
Main Article Content
Abstract
Political hoaxes shape public opinion, yet Indonesian-language detectors are often reported with a single accuracy figure. This study classifies political news as hoax or non-hoax with frozen IndoBERT embeddings (768 dimensions, mean pooling) and a Random Forest. A public collection of 2,262 articles was split 80:20, leaving 453 test articles (215 hoax, 238 non-hoax). A default forest reached 91.61% accuracy (F1-score 91.40%, AUC 0.9746) and a forest tuned by five-fold grid search reached 92.72% (F1-score 92.52%, AUC 0.9753). The 95% Wilson intervals of the two accuracies (88.7–93.8% and 90.0–94.8%) overlap almost entirely, and a net difference of five articles cannot reach p < 0.05 in an exact McNemar test whatever the overlap of errors (best case p = 0.0625), so tuning is not shown to help. The tuned forest flagged 22 of 238 genuine articles as hoax and missed 11 of 215 hoaxes; 12–13% of test articles received a forest vote share between 0.4 and 0.6. Hoax articles come from a fact-checking site and genuine articles from three news portals, so the classifier may separate sources instead of truth, and no experiment here excludes this. We specify a six-check audit, including keyword ablation, a cross-source test, and a lexical baseline, and conclude that the results show separability within this collection, not hoax detection in general.
Downloads
Article Details
Section
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
How to Cite
References
[1] S. Vosoughi, D. Roy, and S. Aral, “The spread of true and false news online,” Science, vol. 359, no. 6380, pp. 1146–1151, Mar. 2018, doi: 10.1126/science.aap9559.
[2] K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, “Fake news detection on social media: A data mining perspective,” ACM SIGKDD Explor. Newsl., vol. 19, no. 1, pp. 22–36, Sep. 2017, doi: 10.1145/3137597.3137600.
[3] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP,” in Proc. 28th Int. Conf. Comput. Linguist. (COLING), 2020, pp. 757–770, doi: 10.18653/v1/2020.coling-main.66.
[4] B. Wilie et al., “IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding,” in Proc. 1st Conf. Asia-Pacific Chapter Assoc. Comput. Linguist. and 10th Int. Joint Conf. Natural Lang. Process., 2020, pp. 843–857, doi: 10.18653/v1/2020.aacl-main.85.
[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. 2019 Conf. North Amer. Chapter Assoc. Comput. Linguist.: Human Lang. Technol. (NAACL-HLT), vol. 1, Minneapolis, MN, USA, 2019, pp. 4171–4186, doi: 10.18653/v1/N19-1423.
[6] C. Jocelynne, I. G. N. L. Wijayakusuma, and L. P. I. Harini, “Detection of political hoax news using fine-tuning IndoBERT,” J. Appl. Informatics Comput., vol. 9, no. 2, pp. 354–360, Mar. 2025, doi: 10.30871/jaic.v9i2.8989.
[7] D. Y. Yefferson, V. Lawijaya, and A. S. Girsang, “Hybrid model: IndoBERT and long short-term memory for detecting Indonesian hoax news,” IAES Int. J. Artif. Intell., vol. 13, no. 2, pp. 1913–1924, Jun. 2024, doi: 10.11591/ijai.v13.i2.pp1913-1924.
[8] L. Riyandari, S. S. Andiko, S. Delimasari, and S. Briandoko, “Optimization of random forest model via GridSearchCV for hoax news detection,” in Proc. Nat. Conf. Elect. Eng., Informatics, Ind. Technol., Creative Media (CENTIVE), 2025, pp. 208–213, doi: 10.20895/centive.v2025i1.545.
[9] R. Geirhos et al., “Shortcut learning in deep neural networks,” Nat. Mach. Intell., vol. 2, no. 11, pp. 665–673, Nov. 2020, doi: 10.1038/s42256-020-00257-z.
[10] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith, “Annotation artifacts in natural language inference data,” in Proc. 2018 Conf. North Amer. Chapter Assoc. Comput. Linguist.: Human Lang. Technol., vol. 2, 2018, pp. 107–112, doi: 10.18653/v1/N18-2017.
[11] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324.
[12] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. 2019 Conf. Empirical Methods Natural Lang. Process. and 9th Int. Joint Conf. Natural Lang. Process. (EMNLP-IJCNLP), 2019, pp. 3980–3990, doi: 10.18653/v1/D19-1410.
[13] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
[14] G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” J. Mach. Learn. Res., vol. 11, pp. 2079–2107, 2010.
[15] L. D. Brown, T. T. Cai, and A. DasGupta, “Interval estimation for a binomial proportion,” Stat. Sci., vol. 16, no. 2, pp. 101–133, May 2001, doi: 10.1214/ss/1009213286.
[16] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, Jun. 1947, doi: 10.1007/BF02295996.
[17] T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Comput., vol. 10, no. 7, pp. 1895–1923, Oct. 1998, doi: 10.1162/089976698300017197.
[18] A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” in Proc. 22nd Int. Conf. Mach. Learn. (ICML), 2005, pp. 625–632, doi: 10.1145/1102351.1102430.
[19] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), 2017, arXiv:1706.04599.
[20] R. J. Connor, “Sample size for testing differences in proportions for the paired-sample design,” Biometrics, vol. 43, no. 1, p. 207, Mar. 1987, doi: 10.2307/2531961.