Random Forest and XGBoost for Mobile-Number Validity Classification
Article Sidebar
Main Article Content
Abstract
Invalid mobile numbers in registration forms cause failed notifications, and a deterministic validator catches only the cases its rules cover. This study asks whether ensemble classifiers can learn a validity rule from syntactic features, and which does it better. We built a dataset of 46,485 Indonesian numbers and labeled them by rule: prefix 08, 10–13 digits, and an allocation list for the first four digits (8.68% invalid). Random Forest and XGBoost were trained after SMOTE on 37,188 records and tested on 9,297 untouched records (807 invalid). With a binary prefix-validity flag supplied (scenario 1), both models made 53–54 errors (about 99.4% accuracy). With the third and fourth digits as numeric features instead (scenario 2), XGBoost made 9 errors (99.90%) and Random Forest 75 (99.19%); exact bounds show the difference is significant under any overlap of errors (p < 10^−13). XGBoost also trained about 36 times faster, on undocumented hardware. Because the labels come from rules that the features reproduce, the accuracies measure rule recovery, not detection of well-formed but wrong numbers; no independent ground truth or rule-based baseline was evaluated. The results support XGBoost as a learned approximation of the rule and call for validation against verified contact status.
Downloads
Article Details
Section
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
How to Cite
References
[1] L. L. Pipino, Y. W. Lee, and R. Y. Wang, “Data quality assessment,” Commun. ACM, vol. 45, no. 4, pp. 211–218, Apr. 2002, doi: 10.1145/505248.506010.
[2] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM Comput. Surv., vol. 41, no. 3, pp. 1–58, Jul. 2009, doi: 10.1145/1541880.1541882.
[3] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, 2002, doi: 10.1613/jair.953.
[4] A. Fernández, S. García, F. Herrera, and N. V. Chawla, “SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary,” J. Artif. Intell. Res., vol. 61, pp. 863–905, 2018, doi: 10.1613/jair.1.11192.
[5] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324.
[6] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., San Francisco, CA, USA, 2016, pp. 785–794, doi: 10.1145/2939672.2939785.
[7] S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance,” ACM Trans. Knowl. Discov. Data, vol. 6, no. 4, pp. 1–21, Dec. 2012, doi: 10.1145/2382577.2382579.
[8] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
[9] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res., vol. 13, pp. 281–305, 2012.
[10] G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” J. Mach. Learn. Res., vol. 11, pp. 2079–2107, 2010.
[11] L. D. Brown, T. T. Cai, and A. DasGupta, “Interval estimation for a binomial proportion,” Stat. Sci., vol. 16, no. 2, pp. 101–133, May 2001, doi: 10.1214/ss/1009213286.
[12] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLoS ONE, vol. 10, no. 3, Art. no. e0118432, Mar. 2015, doi: 10.1371/journal.pone.0118432.
[13] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, Jun. 1947, doi: 10.1007/BF02295996.
[14] T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Comput., vol. 10, no. 7, pp. 1895–1923, Oct. 1998, doi: 10.1162/089976698300017197.
[15] A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” in Proc. 22nd Int. Conf. Mach. Learn. (ICML), 2005, pp. 625–632, doi: 10.1145/1102351.1102430.
[16] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), 2017, arXiv:1706.04599.