Committee polyhedral separability: complexity and polynomial approximation / Khachay M. // Machine Learning. - 2015. - V. 101, l. 1-3. - P. 231-251.

ISSN:
08856125
Type:
Article
Abstract:
We consider the minimum affine separating committee (MASC) combinatorial optimization problem, which is related to ensemble machine learning techniques on the class of linear weak classifiers combined by the rule of simple majority. Actually, the MASC problem is a mathematical formalization of the famous Vapnik–Chervonenkis principle of structural risk minimization in the mentioned class of classifiers. According to this principle, it is required to construct a best performance ensemble classifier belonging to a family of the least possible VC-dimension. It is known that the MASC problem is NP-hard and remains intractable in spaces of any fixed dimension n > 1 even under an additional constraint on the separated sets to be in general position. This special case of the MASC problem called MASC-GP(n) is the main subject of interest of the present paper. To design polynomial-time approximation algorithms for a class of combinatorial optimization problems containing the MASC problem, we propose a new framework, adjusting the well-known Multiple Weights Update method. Following this approach, we construct polynomial-time approximation algorithms with state-of-the-art approximation guarantee for the MASC-GP(n) problem. The results obtained provide a theoretical framework for learning a high-performance ensembles of affine classifiers. © 2015, The Author(s).
Author keywords:
Affine committees; Approximability; Computational complexity; Polyhedral separability
Index keywords:
Artificial intelligence; Combinatorial optimization; Computational complexity; Learning systems; Optimization; Polynomial approximation; Polynomials; Affine committees; Approximability; Combinatorial
DOI:
10.1007/s10994-015-5505-0
Смотреть в Scopus:
https://www.scopus.com/inward/record.uri?eid=2-s2.0-84942368587&doi=10.1007%2fs10994-015-5505-0&partnerID=40&md5=fe55ca2944284432fda61b5d382fa52a
Соавторы в МНС:
Другие поля
Поле Значение
Link https://www.scopus.com/inward/record.uri?eid=2-s2.0-84942368587&doi=10.1007%2fs10994-015-5505-0&partnerID=40&md5=fe55ca2944284432fda61b5d382fa52a
Affiliations Krasovsky Institute of Mathematics and Mechanics UB RAS, Yekaterinburg, Russian Federation; Ural Federal University, Yekaterinburg, Russian Federation; Omsk State Technical University, Omsk, Russian Federation
Author Keywords Affine committees; Approximability; Computational complexity; Polyhedral separability
Funding Details 14-11-00109, RSF, Russian Science Foundation
References Ablow, C., Kaylor, D., A committee solution of the pattern recognition problem (corresp.) (1965) IEEE Transactions on Information Theory, 11, pp. 453-455; Arora, S., Hazan, E., Kale, S., The multiplicative weights update method: A meta-algorithm and applications (2012) Theory of Computing, 8 (1), pp. 121-164; Arora, S., Safra, S., Probabilistic checking of proofs: A new characterization of NP (1998) Journal of the ACM, 45, pp. 70-122; Blum, A., Rivest, R., Training a 3-node neural network is NP-complete (1992) Neural Networks, 5, pp. 117-127; Boyd, S., Vandenberghe, L., (2009) Convex optimization, , Cambridge University Press, Cambridge; Breiman, L., Bagging predictors (1996) Machine Learning, 24 (2), pp. 123-140; Breiman, L., Using iterated bagging to debias regressions (2001) Machine Learning, 45, pp. 261-277; Brown, G.W., Koopmans, T.C., Iterative solutions of games by fictitious play (1951) Activity Analysis of Production and Allocation (pp. 374–376). Cowles Commission Monograph No, p. 13. , New York, Wiley; Cortes, C., Vapnik, V., Support-vector networks (1995) Machine Learning, 20, pp. 273-297; Dinur, I., Regev, O., & Smyth, C. (2002). The hardness of 3-uniform hypergraph coloring. In Proceedings of the 43rd annual IEEE symposium on founations of computer science; Elster, K., Reinhardt, R., Schäuble, M., Donath, G., (1977). Einführung in die nichtlineare Optimierung Mathematisch-Naturwissenschaftliche Bibliothek, p. 63. , BSB BG Teubner Verlagsgesellschaft Leipzig: Vol; Eremin, I., (2002) Theory of linear optimization, , De Gruyter; Eremin, I., Mazurov, V., Astafiev, N., (1983) Improper linear and convex programming problems, , Nauka, Moscow; Feige, U., Threshold of (Formula Presented.) for approximating set cover (1998) Journal of the ACM, 45, pp. 634-652; Freund, Y., Boosting a weak learning algorithm by majority (1995) Information and Computation, 121, pp. 256-285; Freund, Y., Schapire, R.E., Adaptive game playing using multiplicative weights (1999) Games and Economic Behavior, 29 (12), pp. 79-103; Gale, D., Neighboring vertices on a convex polyhedron (1956) Linear inequalities and related system, 38, pp. 255-263; Guruswami, V., Inapproximability results for set splitting and satisfiability problems with no mixed clauses (2004) Algorithmica, 38, pp. 451-469; Hofmann, T., Schölkopf, B., Smola, A., Kernel methods in machine learning (2008) The Annals of Statistics, 36 (3), pp. 1171-1220; Khachai, M., Computational complexity of the minimum committee problem and related problems (2006) Doklady Mathematics, 73, pp. 138-141; Khachai, M., Computational and approximational complexity of combinatorial problems related to the committee polyhedral separability of finite sets (2008) Pattern Recognition and Image Analysis, 18 (2), pp. 236-242; Khachai, M., Mazurov, V., Rybin, A., Committee constructions for solving problems of selection, diagnostics, and prediction (2002) Proceedings of the Steklov Institute of Mathematics, 1, pp. S67-S101; Khachai, M., Rybin, A., A new estimate of the number of members in a minimum committee of a system of linear inequalities (1998) Pattern Recgnition and Image Analysis, 8, pp. 491-496; Khachay, M., On the computational complexity of the minimum committee problem (2007) Journal of Mathematical Modelling and Algorithms, 6 (4), pp. 547-561; Khachay, M., Poberii, M., Complexity and approximability of committee polyhedral separability of sets in general position (2009) Informatica, 20 (2), pp. 217-234; Kobylkin, K., Constraint elimination method for the committee problem (2012) Automation and Remote Control, 73, pp. 355-368; Lin, J., Vitter, J., Complexity results on learning by neural nets (1991) Machine Learning, 6, pp. 211-230; Littlestone, N., Warmuth, M., The weighted majority algorithm (1994) Information and Computation, 108, pp. 212-261; Mazurov, V., Committees of inequalities systems and the pattern recognition problem (1971) Kibernetika, 3, pp. 140-146; Mazurov, V., (1990) Committee method in problems of optimization and classification, , Nauka, Moscow; Mazurov, V., Khachai, M., Committees of systems of linear inequalities (2004) Automation and Remote Control, 65, pp. 193-203; Mazurov, V., Khachai, M., Parallel computations and committee constructions (2007) Automation and Remote Control, 68, pp. 912-921; Megiddo, N., On the complexity of polyhedral separability (1988) Discrete and Computational Geometry, 3, pp. 325-337; Megiddo, N., Tamir, A., On the complexity of locating linear facilities in the plane (1982) Operations Research Letters, 5, pp. 194-197; Nilsson, N., (1965) Learning machines: Foundations of trainable pattern classifying systems, , McGraw-Hill, New York; Schapire, R., The strength of weak learnability (1990) Machine Learning, 5 (2), pp. 197-227; Schapire, R., Freund, Y., (2012) Boosting: Foundations and algorithms, , MIT Press, Cambridge; Schölkopf, B., Smola, A., (2002) Learning with kernels: Support vector machines, regularization, optimization, and beyond, , MIT Press, Cambridge; Syed, U., Schapire, R.E., (2007). A game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems 20, proceedings of the twenty-first annual conference on neural information processing systems, Vancouver, British Columbia (2007) Canada, 3-6, pp. 1449-1456; Wald, A., Statistical decision functions (1949) The Annals of Mathematical Statistics, 20 (2), pp. 165-205
Correspondence Address Khachay, M.; Krasovsky Institute of Mathematics and Mechanics UB RASRussian Federation; email: mkhachay@imm.uran.ru
Publisher Kluwer Academic Publishers
CODEN MALEE
Language of Original Document English
Abbreviated Source Title Mach Learn
Source Scopus