Extreme Data Mining: Inference from Small Datasets

Răzvan Andonie


Neural networks have been applied successfully in many fields. However, satisfactory results can only be found under large sample conditions. When it comes to small training sets, the performance may not be so good, or the learning task can even not be accomplished. This deficiency limits the applications of neural network severely. The main reason why small datasets cannot provide enough information is that there exist gaps between samples, even the domain of samples cannot be ensured. Several computational intelligence techniques have been proposed to overcome the limits of learning from small datasets.
We have the following goals: i. To discuss the meaning of "small" in the context of inferring from small datasets. ii. To overview computational intelligence solutions for this problem. iii. To illustrate the introduced concepts with a real-life application.

Full Text:



R. Andonie, L. Fabry-Asztalos, S. Abdul-Wahid, C. Collar, and N. Salim, "An integrated soft computing approach for predicting biological activity of potential HIV-1 protease inhibitors," in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2006), Vancouver, BC, Canada, July 16-21 2006, pp. 7495–7502.

L. Fabry-Asztalos, R. Andonie, C. Collar, S. Abdul-Wahid, and N. Salim, "A genetic algorithm optimized fuzzy neural network analysis of the affinity of inhibitors for HIV-1 protease," Bioorganic and Medicinal Chemistry, vol. 16, pp. 2903–2911, 2008.

R. Andonie, L. Fabry-Asztalos, C. B. Abdul-Wahid, S. Abdul-Wahid, G. I. Barker, and L. C. Magill, "Fuzzy ARTMAP prediction of biological activities for potential HIV-1 protease inhibitors using a small molecular dataset," IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 99, no. PrePrints, 2009.

R. Andonie and L. Sasu, "Fuzzy ARTMAP with input relevances," IEEE Transactions on Neural Networks, vol. 17, pp. 929–941, 2006.

G. A. Carpenter, S. Grossberg, N. Markuzon, J. H. Reynolds, and D. B. Rosen, "Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps," IEEE Transactions on Neural Networks, vol. 3, no. 5, pp. 698–713, 1992.

S. Verzi, G. Heileman, M. Georgiopoulos, and G. Anagnostopoulos, "Universal approximation with fuzzy art and fuzzy ARTMAP," in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN '03), vol. 3, Portland, Oregon, 20-24 July 2003, pp. 1987–1992.

R. Andonie, L. Fabry-Asztalos, C. Collar, S. Abdul-Wahid, and N. Salim, "Neuro-fuzzy prediction of biological activity and rule extraction for HIV-1 protease inhibitors," in Proceedings of the IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB'05), 2005, pp. 113–120.

R. Andonie, L. Fabry-Asztalos, L. Magill, and S. Abdul-Wahid, "A new Fuzzy ARTMAP approach for predicting biological activity of potential HIV-1 protease inhibitors," in Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM 2007), I. C. S. Press, Ed., San Jose, CA, 2007, pp. 56–61.

R. Andonie, "Inference from small training sets - a computational intelligence perspective," University of Ulster, Jordanstown, Nothern Ireland, United Kingdom, invited talk, June 2008.

R. Andonie, L. Fabry-Asztalos, B. Crivat, S. Abdul-Wahid, and B. Abdul-Wahid, "Fuzzy ARTMAP rule extraction in computational chemistry," in IJCNN'09: Proceedings of the 2009 International Joint Conference on Neural Networks. IEEE, 2009, pp. 2961–2967.

R. Andonie, "Extreme data mining: Inference from small datasets," National University of Ireland, Maynooth, Ireland, invited talk, June 2008.

——, "How to learn from small training sets," Dalle Molle Institute for Artificial Intelligence (IDSIA), Manno-Lugano, Switzerland, invited talk, September 2009.

V. Vapnik, Statistical Learning Theory. New York: Wiley, 2000.

J. L. Balcázar and R. V. Book, "Sets with small generalized Kolmogorov complexity," Acta Inf., vol. 23, no. 6, pp. 679–688, 1986.

A. Ambainis, "Application of Kolmogorov complexity to inductive inference with limited memory," in ALT '95: Proceedings of the 6th International Conference on Algorithmic Learning Theory. London, UK: Springer-Verlag, 1995, pp. 313–318.

A. Ambainis, K. Apsitis, C. Calude, R. Freivalds, M. Karpinski, T. Larfeldt, I. Sala, and J. Smotrovs, "Effects of Kolmogorov complexity present in inductive inference as well," in ALT '97: Proceedings of the 8th International Conference on Algorithmic Learning Theory. London, UK: Springer-Verlag, 1997, pp. 244–259.

J.-L. Yuan and T. Fine, "Neural-network design for small training sets of high dimension," IEEE Tnansactions on Neural Networks, vol. 9, pp. 266–280, 1998.

J.-L. Yuan, "Bootstrapping nonparametric feature selection algorithms for mining small data sets," in Proceedings of the International Joint Conference on Neural Networks (IJCNN), 1999, pp. 2526 – 2529.

C. Huang and C. Moraga, "A diffusion-neural-network for learning from small samples," International Journal of Approximate Reasoning, vol. 35, pp. 137–161, 2004.

R. Mao, H. Zhu, L. Zhang, and A. Chen, "A new method to assist small data set neural network learning," in Proceedings of the Sixth International Conference on Intelligent Systems Design and Applications (ISDA'06), 2006, pp. 17–22.

D.-C. Li, C.-S. Wu, T. T.-I., and L. Y.-S., "Using mega-trend-diffusion and artificial samples in small data set learning for early flexible manufacturing system scheduling knowledge," Computers and Operations Research, vol. 34, pp. 966–982, 2007.

D.-C. Li, C.-W. Yeh, T.-I. Tsai, Y.-H. Fang, and S. Hu, "Acquiring knowledge with limited experience," Expert Systems, vol. 24, pp. 162–170, 2007.

D.-C. Li, C.-S. Wu, T.-I. Tsai, and F. M. Chang, "Using mega-fuzzification and data trend estimation in small data set learning for early FMS scheduling knowledge," Comput. Oper. Res., vol. 33, no. 6, pp. 1857–1869, 2006.

T.-I. Tsai and D.-C. Li, "Approximate modeling for high order non-linear functions using small sample sets," Expert Syst. Appl., vol. 34, no. 1, pp. 564–569, 2008.

D.-C. Li and C.-W. Yeh, "A non-parametric learning algorithm for small manufacturing data sets," Expert Syst. Appl., vol. 34, no. 1, pp. 391–398, 2008.

D.-C. Li and C.-W. Liu, "A neural network weight determination model designed uniquely for small data set learning," Expert Syst. Appl., vol. 36, no. 6, pp. 9853–9858, 2009.

I. V. Tetko, A. I. Luik, and G. I. Poda, "Application of neural networks in structure-activity relationships of a small number of molecules," J. Med. Chem., vol. 36, pp. 811–814, 1993.

D. Hecht and G. Fogel, "High-throughput ligand screening via preclustering and evolved neural networks," IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 4, pp. 476– 484, 2007.

M. Cheung, S. Johnson, D. Hecht, and G. Fogel, "Quantitative structure-property relationships for drug solubility prediction using evolved neural networks," in Proceedings of the IEEE World Congress on Computational Intelligence, 2008, pp. 688–693.

H. Lohr, Sampling: Design and Analysis. Duxbury Press, 1999.

J. Gamez, F. Modave, and O. Kosheleva, "Selecting the most representative sample is NP-hard: Need for expert (fuzzy) knowledge," in Fuzzy Systems, 2008. FUZZ-IEEE 2008. (IEEE World Congress on Computational Intelligence). IEEE International Conference on, June 2008, pp. 1069– 1074.

L. Holmstrom and P. Koistinen, "Using additive noise in backpropagation training," IEEE Transactions on Neural Networks, vol. 3, pp. 24–38, 1992.

C. Wang and J. C. Principe, "Training neural networks with additive noise in the desired signal," IEEE Transactions on Neural Networks, vol. 10, pp. 1511–1517, 1995.

K. Wang, J. Yang, G. Shi, and Q. Wang, "An expanded training set based validation method to avoid overfitting for neural network classifier," International Conference on Natural Computation, vol. 3, pp. 83–87, 2008.

G. N. Karystinos and D. A. Pados, "On overfitting, generalization, and randomly expanded training sets," IEEE Transactions on Neural Networks, vol. 5, pp. 1050–1057, 2000.

Y. Liu, J. A. Starzyk, and Z. Zhu, "Optimized approximation algorithm in neural networks without overfitting," IEEE Transactions on Neural Networks, vol. 19, no. 6, pp. 983–995, 2008.

S. Bos and E. Chug, "Using weight decay to optimize the generalization ability of a perceptron," in Proceedings of the 1996 International Conference on Neural Networks. IEEE, 1996, pp. 241–246.

K. Mahdaviani, H. Mazyar, S. Majidi, and M. H. Saraee, "A method to resolve the overfitting problem in recurrent neural networks for prediction of complex systems' behavior," in IJCNN'08: Proceedings of the 2008 International Joint Conference on Neural Networks, 2008, pp. 3723–3728.

R. Reed, "Pruning algorithms - a survey," IEEE Transactions on Neural Networks, vol. 4, pp. 740– 747, 1993.

T.-Y. Kwok and D.-Y. Yeung, "Constructive algorithms for structure learning in feedforward neural networks for regression problems," IEEE Transactions on Neural Networks, vol. 8, pp. 630–645, 1997.

L. Prechelt, "Automatic early stopping using cross validation: Quantifying the criteria," Neural Networks, vol. 11, pp. 761–767, 1998.

I. Dagher, M. Georgiopoulos, G. Heileman, and G. Bebis, "Ordered Fuzzy ARTMAP: a Fuzzy ARTMAP algorithm with a fixed order of pattern presentation," in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 1998), IEEE World Congress on Computational Intelligence, Anchorage, Alaska, 1998, pp. 1717–1722.

I. Dagher, M. Georgiopoulos, G. L. Heileman, and G. Bebis, "An ordering algorithm for pattern presentation in Fuzzy ARTMAP that tends to improve generalization performance," IEEE Transactions on Neural Networks, vol. 10, pp. 768–778, 1999.

S. Tan, M. Rao, and C. P. Lim, "A hybrid neural network classifier combining ordered Fuzzy ARTMAP and the dynamic decay adjustment algorithm," Soft Computing, vol. 12, pp. 765–775, 2008.

J. Tou and R. Gonzales, Pattern recognition principles. Reading, MA: Addison-Wesley, 1976.

DOI: https://doi.org/10.15837/ijccc.2010.3.2481

Copyright (c) 2017 Răzvan Andonie

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

CC-BY-NC  License for Website User

Articles published in IJCCC user license are protected by copyright.

Users can access, download, copy, translate the IJCCC articles for non-commercial purposes provided that users, but cannot redistribute, display or adapt:

  • Cite the article using an appropriate bibliographic citation: author(s), article title, journal, volume, issue, page numbers, year of publication, DOI, and the link to the definitive published version on IJCCC website;
  • Maintain the integrity of the IJCCC article;
  • Retain the copyright notices and links to these terms and conditions so it is clear to other users what can and what cannot be done with the  article;
  • Ensure that, for any content in the IJCCC article that is identified as belonging to a third party, any re-use complies with the copyright policies of that third party;
  • Any translations must prominently display the statement: "This is an unofficial translation of an article that appeared in IJCCC. Agora University  has not endorsed this translation."

This is a non commercial license where the use of published articles for commercial purposes is forbiden. 

Commercial purposes include: 

  • Copying or downloading IJCCC articles, or linking to such postings, for further redistribution, sale or licensing, for a fee;
  • Copying, downloading or posting by a site or service that incorporates advertising with such content;
  • The inclusion or incorporation of article content in other works or services (other than normal quotations with an appropriate citation) that is then available for sale or licensing, for a fee;
  • Use of IJCCC articles or article content (other than normal quotations with appropriate citation) by for-profit organizations for promotional purposes, whether for a fee or otherwise;
  • Use for the purposes of monetary reward by means of sale, resale, license, loan, transfer or other form of commercial exploitation;

    The licensor cannot revoke these freedoms as long as you follow the license terms.

[End of CC-BY-NC  License for Website User]

INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL (IJCCC), With Emphasis on the Integration of Three Technologies (C & C & C),  ISSN 1841-9836.

IJCCC was founded in 2006,  at Agora University, by  Ioan DZITAC (Editor-in-Chief),  Florin Gheorghe FILIP (Editor-in-Chief), and  Misu-Jan MANOLESCU (Managing Editor).

Ethics: This journal is a member of, and subscribes to the principles of, the Committee on Publication Ethics (COPE).

Ioan  DZITAC (Editor-in-Chief) at COPE European Seminar, Bruxelles, 2015:

IJCCC is covered/indexed/abstracted in Science Citation Index Expanded (since vol.1(S),  2006); JCR2018: IF=1.585..

IJCCC is indexed in Scopus from 2008 (CiteScore2018 = 1.56):

Nomination by Elsevier for Journal Excellence Award Romania 2015 (SNIP2014 = 1.029): Elsevier/ Scopus

IJCCC was nominated by Elsevier for Journal Excellence Award - "Scopus Awards Romania 2015" (SNIP2014 = 1.029).

IJCCC is in Top 3 of 157 Romanian journals indexed by Scopus (in all fields) and No.1 in Computer Science field by Elsevier/ Scopus.


 Impact Factor in JCR2018 (Clarivate Analytics/SCI Expanded/ISI Web of Science): IF=1.585 (Q3). Scopus: CiteScore2018=1.56 (Q2);

SCImago Journal & Country Rank

Editors-in-Chief: Ioan DZITAC & Florin Gheorghe FILIP.