svm_introductory_ppt by university of texas

SVM ppt is about the how the SVM works

1. University of Texas at Austin
Machine Learning Group
Machine Learning Group
Department of Computer Sciences
University of Texas at Austin
Support Vector Machines

2. 2University of Texas at Austin
Machine Learning Group
Perceptron Revisited: Linear Separators
•Binary classification can be viewed as the task of
separating classes in feature space:
w
T
x + b = 0
w
T
x + b < 0
w
T
x + b > 0
f(x) = sign(w
T
x + b)

3. 3University of Texas at Austin
Machine Learning Group
Linear Separators
•Which of the linear separators is optimal?

4. 4University of Texas at Austin
Machine Learning Group
Classification Margin
•Distance from example x
i to the separator is
•Examples closest to the hyperplane are support vectors.
•Margin ρ of the separator is the distance between support vectors.
w
xw b
r
i
T


r
ρ

5. 5University of Texas at Austin
Machine Learning Group
Maximum Margin Classification
•Maximizing the margin is good according to intuition and
PAC theory.
•Implies that only support vectors matter; other training
examples are ignorable.

6. 6University of Texas at Austin
Machine Learning Group
Linear SVM Mathematically
•Let training set {(x
i, y
i)}
i=1..n, x
iR
d
, y
i 
{-1, 1} be separated by a
hyperplane with margin ρ. Then for each training example (x
i, y
i):
•For every support vector x
s
the above inequality is an equality.
After rescaling w and b by ρ/2 in the equality, we obtain that
distance between each x
s and the hyperplane is
•Then the margin can be expressed through (rescaled) w and b as:
w
T
x
i
+ b ≤ - ρ/2 if y
i
= -1
w
T
x
i
+ b ≥ ρ/2 if y
i
= 1
w
2
2r
ww
xw 1)(y



b
r
s
T
s
y
i
(w
T
x
i
+ b) ≥ ρ/2

7. 7University of Texas at Austin
Machine Learning Group
Linear SVMs Mathematically (cont.)
•Then we can formulate the quadratic optimization problem:
Which can be reformulated as:
Find w and b such that
is maximized
and for all (x
i, y
i), i=1..n : y
i(w
T
x
i + b) ≥ 1
w
2

Find w and b such that
Φ(w) = ||w||
2
=w
T
w is minimized
and for all (x
i
, y
i
), i=1..n : y
i
(w
T
x
i
+ b) ≥ 1

8. 8University of Texas at Austin
Machine Learning Group
Solving the Optimization Problem
•Need to optimize a quadratic function subject to linear constraints.
•Quadratic optimization problems are a well-known class of mathematical
programming problems for which several (non-trivial) algorithms exist.
•The solution involves constructing a dual problem where a Lagrange
multiplier α
i
is associated with every inequality constraint in the primal
(original) problem:
Find w and b such that
Φ(w) =w
T
w is minimized
and for all (x
i, y
i), i=1..n : y
i (w
T
x
i + b) ≥ 1
Find α
1
…α
n
such that
Q(α) =Σα
i - ½ΣΣα
iα
jy
iy
jx
i
T
x
j is maximized and
(1) Σα
iy
i = 0
(2) α
i ≥ 0 for all α
i

9. 9University of Texas at Austin
Machine Learning Group
The Optimization Problem Solution
•Given a solution α
1
…α
n
to the dual problem, solution to the primal is:
•Each non-zero α
i indicates that corresponding x
i is a support vector.
•Then the classifying function is (note that we don’t need w explicitly):
•Notice that it relies on an inner product between the test point x and the
support vectors x
i – we will return to this later.
•Also keep in mind that solving the optimization problem involved
computing the inner products x
i
T
x
j between all training points.
w =Σα
iy
ix
i b = y
k - Σα
iy
ix
i
T
x
k for any α
k > 0
f(x) = Σα
i
y
i
x
i
T
x + b

10. 10University of Texas at Austin
Machine Learning Group
Soft Margin Classification
•What if the training set is not linearly separable?
•Slack variables ξ
i
can be added to allow misclassification of difficult or
noisy examples, resulting margin called soft.
ξ
i
ξ
i

11. 11University of Texas at Austin
Machine Learning Group
Soft Margin Classification Mathematically
•The old formulation:
•Modified formulation incorporates slack variables:
•Parameter C can be viewed as a way to control overfitting: it “trades off”
the relative importance of maximizing the margin and fitting the training
data.
Find w and b such that
Φ(w) =w
T
w is minimized
and for all (x
i ,y
i), i=1..n : y
i (w
T
x
i + b) ≥ 1
Find w and b such that
Φ(w) =w
T
w + CΣξ
i is minimized
and for all (x
i
,y
i
), i=1..n : y
i
(w
T
x
i
+ b) ≥ 1 – ξ
i,
, ξ
i
≥ 0

12. 12University of Texas at Austin
Machine Learning Group
Soft Margin Classification – Solution
•Dual problem is identical to separable case (would not be identical if the 2-
norm penalty for slack variables CΣξ
i
2
was used in primal objective, we
would need additional Lagrange multipliers for slack variables):
•Again, x
i with non-zero α
i will be support vectors.
•Solution to the dual problem is:
Find α
1…α
N such that
Q(α) =Σα
i
- ½ΣΣα
i
α
j
y
i
y
j
x
i
T
x
j
is maximized and
(1) Σα
i
y
i
= 0
(2) 0 ≤ α
i
≤ C for all α
i
w =Σα
i
y
i
x
i
b= y
k(1- ξ
k) - Σα
iy
ix
i
T
x
k for any k s.t. α
k>0
f(x) = Σα
iy
ix
i
T
x + b
Again, we don’t need to
compute w explicitly for
classification:

13. 13University of Texas at Austin
Machine Learning Group
Theoretical Justification for Maximum Margins
•Vapnik has proved the following:
The class of optimal linear separators has VC dimension h bounded from
above as
where ρ is the margin, D is the diameter of the smallest sphere that can
enclose all of the training examples, and m
0 is the dimensionality.
•Intuitively, this implies that regardless of dimensionality m
0 we can
minimize the VC dimension by maximizing the margin ρ.
•Thus, complexity of the classifier is kept small regardless of
dimensionality.
1,min
02
2













 m
D
h


14. 14University of Texas at Austin
Machine Learning Group
Linear SVMs: Overview
•The classifier is a separating hyperplane.
•Most “important” training points are support vectors; they define the
hyperplane.
•Quadratic optimization algorithms can identify which training points x
i
are
support vectors with non-zero Lagrangian multipliers α
i
.
•Both in the dual formulation of the problem and in the solution training
points appear only inside inner products:
Find α
1
…α
N
such that
Q(α) =Σα
i - ½ΣΣα
iα
jy
iy
jx
i
T
x
j is maximized and
(1) Σα
i
y
i
= 0
(2) 0 ≤ α
i
≤ C for all α
i
f(x) = Σα
iy
ix
i
T
x + b

15. 15University of Texas at Austin
Machine Learning Group
Non-linear SVMs
•Datasets that are linearly separable with some noise work out great:
•But what are we going to do if the dataset is just too hard?
•How about… mapping data to a higher-dimensional space:
0
0
0
x
2
x
x
x

16. 16University of Texas at Austin
Machine Learning Group
Non-linear SVMs: Feature spaces
•General idea: the original feature space can always be mapped to some
higher-dimensional feature space where the training set is separable:
Φ: x
→ φ(x)

17. 17University of Texas at Austin
Machine Learning Group
The “Kernel Trick”
•The linear classifier relies on inner product between vectors K(x
i
,x
j
)=x
i
T
x
j
•If every datapoint is mapped into high-dimensional space via some
transformation Φ: x

→ φ(x), the inner product becomes:
K(x
i
,x
j
)= φ(x
i
)

T
φ(x
j
)
•A kernel function is a function that is eqiuvalent to an inner product in some
feature space.
•Example:
2-dimensional vectors x=[x
1 x
2]; let K(x
i,x
j)=(1 + x
i
T
x
j)
2
,
Need to show that K(x
i
,x
j
)= φ(x
i
)

T
φ(x
j
):
K(x
i
,x
j
)=(1 + x
i
T
x
j
)
2
,
= 1+ x
i1
2
x
j1
2
+ 2 x
i1
x
j1

x
i2
x
j2
+ x
i2
2
x
j2
2
+ 2x
i1
x
j1
+ 2x
i2
x
j2
=
= [1 x
i1
2
√2 x
i1x
i2 x
i2
2
√2x
i1 √2x
i2]
T
[1 x
j1
2
√2 x
j1x
j2 x
j2
2
√2x
j1 √2x
j2] =
= φ(x
i
)

T
φ(x
j
), where φ(x) =

[1 x
1
2
√2 x
1
x
2
x
2
2
√2x
1
√2x
2
]
•Thus, a kernel function implicitly maps data to a high-dimensional space
(without the need to compute each φ(x) explicitly).

18. 18University of Texas at Austin
Machine Learning Group
What Functions are Kernels?
•For some functions K(x
i
,x
j
) checking that K(x
i
,x
j
)= φ(x
i
)

T
φ(x
j
) can be
cumbersome.
•Mercer’s theorem:
Every semi-positive definite symmetric function is a kernel
•Semi-positive definite symmetric functions correspond to a semi-positive
definite symmetric Gram matrix:
K(x
1
,x
1
)K(x
1
,x
2
)K(x
1
,x
3
) … K(x
1
,x
n
)
K(x
2
,x
1
)K(x
2
,x
2
)K(x
2
,x
3
) K(x
2
,x
n
)
… … … … …
K(x
n
,x
1
)K(x
n
,x
2
)K(x
n
,x
3
) … K(x
n
,x
n
)
K=

19. 19University of Texas at Austin
Machine Learning Group
Examples of Kernel Functions
•Linear: K(x
i,x
j)= x
i
T
x
j
–Mapping Φ: x
→ φ(x), where φ(x) is x itself
•Polynomial of power p: K(x
i,x
j)= (1+ x
i
T
x
j)
p
–Mapping Φ: x

→ φ(x), where φ(x) has dimensions
•Gaussian (radial-basis function): K(x
i
,x
j
) =
–Mapping Φ: x

→ φ(x), where φ(x) is infinite-dimensional: every point is mapped
to a function (a Gaussian); combination of functions for support vectors is the
separator.
•Higher-dimensional space still has intrinsic dimensionality d (the mapping is
not onto), but linear separators in it correspond to non-linear separators in
original space.
2
2
2
ji
e
xx









p
pd

20. 20University of Texas at Austin
Machine Learning Group
Non-linear SVMs Mathematically
•Dual problem formulation:
•The solution is:
•Optimization techniques for finding α
i’s remain the same!
Find α
1
…α
n
such that
Q(α) =Σα
i
- ½ΣΣα
i
α
j
y
i
y
j
K(x
i
,

x
j
) is maximized and
(1) Σα
iy
i = 0
(2) α
i ≥ 0 for all α
i
f(x) = Σα
iy
iK(x
i,
x
j)+ b

21. 21University of Texas at Austin
Machine Learning Group
SVM applications
•SVMs were originally proposed by Boser, Guyon and Vapnik in 1992 and
gained increasing popularity in late 1990s.
•SVMs are currently among the best performers for a number of classification
tasks ranging from text to genomic data.
•SVMs can be applied to complex data types beyond feature vectors (e.g.
graphs, sequences, relational data) by designing kernel functions for such data.
•SVM techniques have been extended to a number of tasks such as regression
[Vapnik et al. ’97], principal component analysis [Schölkopf et al. ’99], etc.
•Most popular optimization algorithms for SVMs use decomposition to hill-
climb over a subset of α
i’s at a time, e.g. SMO [Platt ’99] and
[Joachims ’99]
• Tuning SVMs remains a black art: selecting a specific kernel and parameters
is usually done in a try-and-see manner.