• No results found

A Brief Digest on Reproducing Kernel Hilbert Space

N/A
N/A
Protected

Academic year: 2020

Share "A Brief Digest on Reproducing Kernel Hilbert Space"

Copied!
5
0
0

Loading.... (view fulltext now)

Full text

(1)

2016 International Conference on Computer, Mechatronics and Electronic Engineering (CMEE 2016) ISBN: 978-1-60595-406-6

A Brief Digest on Reproducing Kernel Hilbert Space

Shou-yu TONG

*

, Fu-zhong CONG and Zhi-xia WANG

Aviation University of Air Force, 130022, Changchun, China *Corresponding author

Keywords: Kernel, Reproducing Kernel Hilbert Space.

Abstract. Reproducing Kernel Hilbert Space (RKHS) is a common used tool in statistics and machine learning to generalize from linear models to non-linear models. In this paper we will try to understand the basic theoretical results in studying RKHS: to construct a RKHS starting from a given kernel function. This view is highly related to the kernel methods for regression and classification in the area of machine learning.

Introduction

Reproducing Kernel Hilbert Spaces (RKHS) have recently received much attention [1, 2, 3] from the statistics and machine learning researchers, due to the popularity of some machine learning algorithms and techniques such as SVM and smoothing splines. Although the theory of RKHS was established quite a long time ago in the field of statistics, machine learning people are mainly putting efforts in figuring out the way this theory could be used to understand some properties of the learning algorithms.

Historically, there are two approaches in studying RKHS. One group of people are concerning with the kernel function K x y( , ) and treat the Hilbert space as a tool of understanding the properties of the function or they simply ignore the space . Another approach is to study the Hilbert space , which has a lot of nice properties, then treat the reproducing kernel as a function related to the space. But later people realize that the positive definite function in the first approach is equivalent to the reproducing kernel function in the second approach.

Notations and Basic Definitions

Before we talk about RKHS, we need some definitions about the underlying spaces and properties.

Definition 1. Banach Space:

A Banach space is a normed space that is complete, in the sense that it contains the limits of all the Cauchy sequence in that space.

Definition 2. Inner Product:

Let be a vector space over , then a mapping , :    is an inner product on if

1, 2, ,

f f f g

  , a a1, 2 :

 f f,  0 and f f,  0 iff f 0  f g,   g f, 

 a f1 1a f g2 2,   a f g1 1,   a2 f g2, 

Definition 3. Hilbert Space:

Now we can define Hilbert space as follows: Hilbert space is a Banach space with an inner product. To study the notion of RKHS, we will need some definition about linear operators.

Definition 4. Linear Operator:

Let , be normed vector spaces, then a mapping L:  is a linear operator if

( ) ( )

L afaL f and (L fg)L f( )L G( ),   a , f g,  .

(2)

A linear operator L is continuous at f0 if   0,   0 such that ‖ ff0‖  implies

0

( ) ( )

L fL f 

‖ ‖ .

We can also define a norm for an operator to characterize its behavior.

Definition 6. Operator Norm:

The operator norm of L is defined as sup ( )

f

L f L

f

 ‖ ‖

‖ ‖

‖ ‖ .

Definition 7. Bounded Linear Operator:

A linear operator L:  is bounded if ‖ ‖L  .

Theorem 1.

Let L:  be a linear operator, then the following conditions are equivalent:  L is continuous on

L is continuous at f0

L is bounded

Next is an important theorem by Riez allowing us to proof the existence of a reproducing kernel.

Theorem 2.Riez Representation Theorem:

Let be a Hilbert space, if L is a continuous linear operator on , then ( )L f  f g,  for some g .

Now we can give the formal definition of the RKHS, which is a Hilbert space of nicely behaved functions.

Definition 8. Evaluation Functional:

Let be a Hilbert space of functions from to . And let f :  be a function in , for

x , the mapping x:  defined as xff x( ) is the evaluation functional for x.

Definition 9. Reproducing Kernel Hilbert Space:

Let be a Hilbert space of functions from to , then we say that is a reproducing kernel Hilbert space if the evaluation functional x is continuous for all x .

Constructing RKHS from a Kernel

So far we are concerning with the reproducing Hilbert space and showing that if is an RKHS then it has a reproducing kernel. But we can also go in the other direction, from a kernel to the corresponding RKHS, and the goal is that given a kernel K, we want to construct an RKHS such that K is the reproducing kernel of .

Let's first define what we mean by a kernel, and a kernel is positive definite from this definition.

Definition 10. Kernel Hilbert Space:

For K:   is called a kernel if K x y( , )K y x( , ), and such that K is positive definite,

meaning that

1 1

( , ) 0.

n n

i j i j i j

K x x  

 



There are two steps of the construction for an RKHS , first we will construct a space 0 which is not necessarily an RKHS, then we will add the limit of every Cauchy sequence in 0 to the space to form an RKHS .

Formally speaking, the construction is as follows:

 Given a positive definite function K:   , define a space

0

1

:

{

( )

|

( ) ( , ), , ,

}

n

i i i i

i

f f xK x x n x

  

     

 Define :

{

f

|

 Cauchy sequence { }fn0,| f xn( ) f x( ) |  0, x

}

(3)

Validity of 0 as Pre-RKHS

Let 0

1 1

( , ), ( , )

n m

i i j j

i j

fK x gK y

 

 

  , define the inner product on 0 as follows:

0

1 1

, ( , )

n m

i j i j i j

f g   K x y

 

  



.

We need to verify that this inner product is well defined. First, the inner product is independent of the choice of i and j since

0

, i ( )i j ( j)

i j

f gg xf y

  

.

Then it is easy to verify the symmetry and linearity, and using the positive definiteness of $K$ we can prove the positivity, which means the inner product defined above is valid.

The next step is to show that the pre-RKHS 0 satisfies the following two properties:

 The evaluation functional x is continuous on 0,  x , which could be shown using the reproducing property of K and Cauchy-Schwarz inequality.

 If a Cauchy sequence { }fn in 0 converges to 0 pointwise, then it converges to 0 in 0 norm. This could be shown using triangle inequality and the fact that Cauchy sequence is bounded.

Define the Inner Product on

Now we want to extend the definition of inner product on 0 to , assume that { },{ }fn gn are Cauchy sequences in 0 and they converge point-wise to f and g in respectively, then let

,

n f gn n

   , we want to show n converges and does not depend on the specific sequence we choose, so this inner product is well defined on .

Firstly it is convergent. Let ,n m ,

0 0

0 0

0 0 0 0

| | | , , |

| , , |

n m n n m m

n m n m n m

n n m m n m

f g f g

f f g f g g

g f f f g g

       

       

‖ ‖ ‖  ‖ ‖ ‖ ‖  ‖

Using the fact that every Cauchy sequence is bounded and both

0 n m

ff

‖ ‖ and

0 n m

gg

‖ ‖ goes

to 0, we know that

0

,

n n

f g

  is convergent.

Now suppose there is another Cauchy sequence {fn}, {gn} that converges to f and g in 0, let

0

,

n fn gn

     , then by the same step as before,

0 0 0 0

| nn|‖ gn‖ ‖ fnfn‖ ‖ fn‖ ‖gngn

By the second property of the pre-RKHS 0, since fnfn and gngn converge point-wise to 0, they also converge in

0

‖ ‖ to 0, so | nn|0 as n, this proves the convergence does not depend on the sequence we choose.

Validity of the Inner Product on

We need to verify that the inner product we defined is valid, which satisfies the three properties of a proper inner product. Proving the linearity and the symmetry is trivial, and we shall prove that

, 0

f f

   if and only if f 0. Suppose that ( )f x   0, x , let { }fn be a Cauchy sequence in

0 that converges to f point-wise, then by the property of 0 that point-wise convergence implies

convergence in norm,

0

2

, lim n, n lim n 0.

n n

f f f f f

 

(4)

For the other direction, suppose that f f,  0, then by definition

0

2

lim n, n lim n 0

nf f  n‖ f ‖  .

Using the first property of the pre-RKHS 0 that the evaluation functional x is continuous and thus bounded,

0

( ) lim n( ) lim x n lim x n 0,

n n n

f x f xff x

  

   ‖ ‖ ‖ ‖  

where ‖x‖ is the operator norm and is finite since x is bounded. Therefore we have shown that the inner product on is both well defined and valid.

Continuity of Evaluation Functional on

Next we want to prove that the evaluation functional x is continuous on , but we need another lemma first, that is if f  and fn0 is a Cauchy sequence that converges point-wise to f , then

n

f also converges to f in ‖ ‖ . To see this, for  0 , let N such that

0 , , m n

ff   m nN

‖ ‖ , then we fix n and consider the sequence {fmfn}, which is another Cauchy sequence converging point-wise to ffn, now by definition of inner product in ,

0

lim

n n

n

f f f f



   

‖ ‖ ‖ ‖

and which says that fn also converges to f in ‖ ‖ .

Now we will prove the evaluation functional x is continuous in . By the first property of the pre-RKHS 0, the evaluation functional x is continuous in 0, which states   C 0, f f1, 20, there is |xf1xf2|Cf1f2‖. Let f0 and x , take f1f and f2 0 in the above statement, then for  0, there exists  0 such that

0

0

f  

‖ ‖ implies |xf | | ( ) | f x  / 2.

Then for any function  such that ‖ ‖ / 2, by the previous lemma we know that there is a function g0 that | ( )g x ( ) |x  / 2 and ‖ g ‖ / 2.

Since

0

ggg   

‖ ‖ ‖ ‖ ‖ ‖ ‖ ‖ , we know | ( ) |g x / 2 , so | ( ) | x  , which proves that x is continuous in by definition.

Completeness of

The final step is to prove that is a Hilbert space, we have defined an inner product

in and shown the evaluation functional is continuous on , the only thing left to be shown is that is complete.

For any Cauchy sequence fn in , since the evaluation functional is continuous on , for any

x , { ( )}f xn is a Cauchy sequence. Define ( ) lim n( )

n

f x f x



 , since f xn( ) and is

complete. Now we want to prove that f is also in and { }fn converges to f in ‖ ‖ . The way to prove this is to construct a sequence { }gn in 0 such that { }gn is a Cauchy sequence and it converges point-wise to f .

Let n 0 be a sequence of real numbers that n 0 as n. Then from the lemma above that

0 is dense in , there is a sequence { }gn in such that

, .

i i i

fg  i

‖ ‖

Let x , then

|g xn( ) f x( ) | | g xn( ) f xn( ) || f xn( ) f x( ) |‖x‖ ‖ gnfn‖ | f xn( ) f x( ) |

since x is bounded and f xn( ) converges to ( )f x , we have |g xn( ) f x( ) |0 as n. Also,

0

i j i j i i i j j j

i j i j

g g g g g f f f f g

f f

 

        

   

‖ ‖ ‖ ‖ ‖ ‖ ‖ ‖ ‖ ‖

(5)

by n 0 and fn is Cauchy sequence, we know gn is also a Cauchy sequence that converges to f

point-wise, so by definition f  . Finally,

0

n n n n

fffggf

‖ ‖ ‖ ‖ ‖ ‖

And therefore fn converges to f in ‖ ‖ , so is complete and thus a reproducing Hilbert space.

Summary

In this project, we investigated the theoretic basis of reproducing kernel Hilbert space, mainly including constructing a RKHS from a kernel.

References

[1] N. Aronszajn, Theory of reproducing kernels. Transactions of the American mathematical society, 68(3): pp. 337–404, 1950.

References

Related documents

The NPT ensemble is implemented similarly to the canonical ensemble, except an additional cell resizing move is implemented. As in the volume transfer move of the classical

displaying MoCA diagnostic cut points, where a cut point for both high sensitivity and specificity is chosen, which stratifies participants into high-, intermediate- and

Methods may be defined in Java or PL/SQL (Oracle) Functions: independent of types, no SELF attribute * compare: java interface

collect, generate, prepare, disseminate, store, share, publish, distribute, update, digitize and/or create Geospatial Data, including Maps, of any spatial

Associate Pofessor of Radiology Chairman, Department of Radiology University Teaching Hospital LKH Feldkirch Feldkirch, Austria. Edith

This combination of Augmented Reality, Computer Supported Collaborative Work (CSCW) and multiple user interaction brings with it a few advantages over common virtual

Used block hierarchical regression models to estimate the effects of religious affiliation and Carnegie/control on each dependent variable (a) without controls (bivariate) (b) with

Copper, ( As Taiwan Approaches the New Millennium (Lanham: University Press of America, 1999); Chun Chieh Huang, Postwar Taiwan in Historical Perspective (College