A Geometric Approach to Support Vector Regression

A Geometric Approach to Support Vector Regression

Jinbo Bi and Kristin Bennett

Department of Mathematical Sciences
Rensselaer Polytechnic Institute

Abstract
We develop an intuitive geometric framework for support vector regression (SVR). By examining when epsilon-tubes exist, we show that SVR can be regarded as a classification problem in the dual space. Hard and soft epsilon-tubes are constructed by separating the convex or reduced convex hulls respectively of the training data with the response variable shifted up and down by epsilon. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted datasets. Maximizing the margin corresponds to shrinking the effective epsilon-tube. In the proposed approach, the effects of the choices of all parameters become clear geometrically. The kernelized model corresponds to separating the convex or reduced convex hulls in feature space. Generalization bounds for classification can be extended to characterize the generalization performance of the proposed approach. We propose a simple iterative nearest-point algorithm that can be directly applied to the reduced convex hull case in order to construct soft epsilon-tubes. Computational comparisons with other SVR formulations are also included.

    • This paper has been accepted by Neurocomputing, special issue on support vector machines.
    • CPLEX programs

In this paper, we compared the three support vector regression algorithms, RH-SVR (the proposed approach), C-SVR (the classic SVR algorithm) and nu-SVR. They were implemented in Unix C++ using the commercial optimization software CPLEX 6.6. The source codes can be available under request (bij2@rpi.edu). An appropriate version of CPLEX is required to run the programs.

 

    • NPA package

The proposed geometric approach can be solved using a faster iterative algorithm called the nearest point algorithm (NPA). A preliminary solver has been written using Visual C++ 6.0 under Windows 2000. This implementation was generated as one of the course projects for Operating Systems. The project report can be found here. The package can be downloaded here for non-commercial uses and the author will not take any responsibility for problems encountered when using the package. An updated and improved version of the NPA program may be available under request.

 

    Contact Jinbo Bi (jinbo@engr.uconn.edu) for information about this page.