Two-Layer SVM, Towards Deep Statistical Learning
Alireza Kazemi, Reza Boostani, Mahmoud Odeh, Mohammad Rasmi Al-Mousa
Abstract
Support Vector Machine (SVM) is originally a binary large-margin classifier emerged from the concept of structural risk minimization. Multiple solutions such as one-versus-one and one-versus-all have been proposed for creating multi-class SVM using elementary binary SVMs. Also multiple solutions have been proposed for SVM model selection, adjusting margin-parameter C and the Gaussian kernel variance. Here, an improved classifier named SVM-SVM is proposed for multi-class problems which increases accuracy and decreases dependency to margin-parameter selection. SVM-SVM adopts two K-class one-vs-one SVMs in a cascaded two-layer structure. In the first layer, input features are fed to one-vs-one SVM with non-linear kernels. We introduce this layer as a large-margin non-linear feature transform that maps input feature space to a discriminative K*(K-1)/2 dimensional space. To assess our hierarchical classifier, some datasets from the UCI repository are evaluated. Standard one-vs-one SVM and one-vs-one fuzzy SVM are used as reference classifiers in experiments. Results show significant improvements of our proposed method in terms of test accuracy and robustness to the model (margin and kernel) parameters in comparison with the reference classifiers. Our observations suggest that a multi-layer (deep) SVM structures can gain the same benefits as is seen in the deep neural nets (DNNs).