Unsupervised Learning Reinforcement Learning

Lecture 08

Machine Learning

Outline

•

_{Unsupervised Learning}

•

_{Reinforcement Learning}

Hi Humans !! Hi Machine

Machine Learning Machine Learning Supervised (Task Driven) Supervised (Task Driven) Classification Classification

Decision Tree, NN, Naïve Bayes, KNN, SVM, Discriminant Analysis, Ensemble Methods, Random

Forest

Decision Tree, NN, Naïve Bayes, KNN, SVM, Discriminant Analysis, Ensemble Methods, Random Forest Regression Regression Ordinary LSR, Linear Regression, Logistic Regression,

MARS, LOESS

Ordinary LSR, Linear Regression, Logistic Regression,

MARS, LOESS Unsupervised (Data Driven) Unsupervised (Data Driven) Clustering Clustering kMean, Kmedoids, Fuzzy C-means, Hierarchical, SOM, Hidden Markov Model,

Gaussian Mixture

kMean, Kmedoids, Fuzzy C-means, Hierarchical, SOM, Hidden Markov Model,

Gaussian Mixture Dimension Reduction Dimension Reduction PCA, LDA PCA, LDA Reinforcement (Algorithms learn to react

an environment)

Reinforcement (Algorithms learn to react

an environment) Decision Process Decision Process Reward System Reward System Recommendation Systems Recommendation Systems

Develop predictive model based on

both input and

output data Develop predictive

model based on both input and

output data

Discover an internal representation

Tasks

Clustering

x c Discrete ID

Dimensionality Reduction

x z Continuous

Unsupervised Learning

•

_{Given a training corpus of data points}

– _{Observed value of random variables in Bayesian}

network

– _{Series of data points} – _{Orbits of planets}

•

_{Learn underlying pattern in the data}

– _{Existence and conditional probability of hidden}

variables

Unsupervised Learning Example

• _{2D state space with}

unclassified observations

• _{Learn number and form of}

clusters

• _{Problem of unsupervised}

clustering

– _{Many algorithms proposed}

for it

– _{More research still being}

done for better algorithms, different kind of data, …

*

*

* *

*

*

*

*

*

*

*

*

*

*

*

* *

*

*

_*

*

*

*

*

* *

*

*

*

*

*

*

Unsupervised Learning Algorithm

• _{Define a similarity}

measure, to compare pairs of elements

• _{Starting with no clusters}

– _{Pick seed element}

– _{Group similar elements until}

threshold

– _{Pick new seed from free}

elements and start again

*

*

*

*

*

_*

*

*

*

*

*

* *

*

*

*

*

*

*

*

*

*

*

*

_*

*

* *

*

*

*

*

*

*

Unsupervised Learning Algorithm

• _{Starting with one}

all-encompassing cluster

– _{Find cluster with highest}

internal dissimilarity

– _{Find most dissimilar pair of}

elements inside cluster

– _{Split into two clusters}

– _{Repeat until all clusters have}

internal homogeneity

– _{Merge homogeneous clusters}

Unsupervised Learning Evaluation

•

_{Need to evaluate fitness of relationship}

learned

– _{Number of clusters vs. their internal properties} – _{Difference between clusters vs. internal}

homogeneity

– _{Number of parameters vs. number of hidden}

variables in Bayesian network

•

_{No way of knowing what is the optimal}

K-Means

•

_{Popular unsupervised clustering algorithm}

•

_{Data represented as cloud of points in state}

space

•

_Target

– _{Group points in k clusters}

K-Means

•

_{Start with k random cluster centers}

•

_{For each iteration}

– _{For each data point}

• _{Associate the point to the nearest cluster center} • _{Add to variance}

– _{Move each cluster center to the center of mass of}

associated data point cloud

– _{End when}

K-Means

•

_{We have:}

– Data points: x₁, …, x_i, …, x_n

– Clusters: C₁, …, C_j, … C_k

– Cluster centers: ₁, …, _j, … _k

•

_{Minimize intra-cluster variance}

• _{Collecting and labeling a large training set can be}

very expensive.

• _{Be able to find features which are helpful for}

categorization.

• _{Gain insight into the natural structure of the data.}



_{There are a lot of other Unsupervised}

Learning Methods.



_Examples:

◦ Competitive Learning

◦ Kohonen’s Neural Networks: Self-Organizing Maps

◦ Principal Component Analysis, Autoassociation

Reinforcement Learning

• _{Given a set of possible actions, the resulting state of}

the environment, and rewards or punishment for each state

– _{Taxi driver: tips, car repair costs, tickets} – _{Checkers: advantage in number of pieces}

• _{Learn to maximize the rewards and/or minimize the}

punishments

– _{Maximize tip, minimize damage to car and police tickets:}

drive properly

• _{Learning by trial and error}

• _{Try something, see the result}

– _{Speeding results in tickets, going through a red light}

results in car damage, quick and safe drive results in tips

– _{Checkers pieces in the center of the board are soon lost,}

pieces on the side are kept longer, sacrifice some pieces to take a greater number of enemy pieces

• _{Sacrifice known rewarding actions to explore new,}

potentially more rewarding actions

• _{Develop strategies to maximize rewards while}

minimizing penalties over the long-term

Q-Learning

•

_{Each state has}

– _{A reward or punishment}

– _{A list of possible actions, which lead to other}

states

•

_{Learn value of state-action pairs}

Q-Learning

• _{Update value of previous (t-1) state-action pair based}

on current (t) state-action value

• Q(s_t-1,a_t-1) = [R_t-1 + max_a(Q(s_t,a_t)) – Q(s_t-1,a_t-1)]

– _{Q(s,a): estimated value of state-action pair (s,a)} – R_t: reward of state s_t

– _{: learning rate}

– _{: discount factor of future rewards}

• 0 (future rewards are irrelevant), 1 (future rewards are the same as current rewards)

Exploration Function

•

_{If agent always does action with max Q(s,a), it}

always evaluates the same state-action pairs

•

_{Need exploration function}

– _{Trade-off greed vs. curiosity}

– _{Try rarely-explored low-payoff actions instead of}

well-known high-payoff actions

Exploration Function

• _Define:

– _{Q(s,a): estimated value of (s,a)}

– _{N(s,a): number of times (s,a) has been tried} – _{Rmax: maximum possible value of Q(s,a)}

– _{Nmin: minimum number of times we want the agent to try}

(s,a)

• _{f( Q(s,a), N(s,a) ) =}

– _{Rmax if N(s,a) < Nmin} – _{Q(s,a) otherwise}

• _{Agent picks action with maximum f(.) value}

Limits of RL

•

_Search

– _{Number of state-action pairs can be very large} – _{Intermediate rewards can be noisy}

•

_{Real-world search}

– _{Initial policy can have very poor reward}

– _{Necessary exploration of suboptimal actions can}

Policy

•

_{Learn the optimal policy in decision network}

•



_{: S}



A

•

EU(



) =





_R

t

•

_{Greedy search}

Helicopter Flight Control

•

_{Sustained stable}

inverted flight

– _{Very difficult for}

humans

– _{First AI able to}

Helicopter Flight Control

•

_{Collect flight data with human pilot}

•

_{Learn model of helicopter dynamics}

– _{Stochastic and nonlinear} – _{Supervised learning}

•

_{Learn policy for helicopter controller}

Helicopter Dynamics

•

_States

– _{Position, orientation,}

velocity, angular velocity

– _{12 variables}

•

_{391 seconds of flight data}

– _{Time step 0.1s}

– _{3910 triplets (st, at, st+1)}

•

_{Learn probability distribution P(s}

_|s

_,a

₎

Helicopter Controller

• _{Problem definition}

– _{S: set of possible states} – s₀: initial state (s₀  S)

– _{A: set of possible actions}

– _{P(S|S,A): state transition probabilities} – : discount factor

– _{R: reward function mapping states to values}

• At state s_t, controller picks action a_t, system

Helicopter Controller

•

_{Reward function}

– _{Punish deviation from desired helicopter position}

and velocity

– _R  [-, 0]

•

_{Policy learning}

– _{Reinforcement learning} – _{EU() = t=0} t _R

t

•

_Problem

– _{Stochastic state transitions}

PEGASUS algorithm

• _{Predefined series of random numbers}

– _{Length of series function of complexity of policy}

• _{Use the same series to test all policies}

– _{At time t, each policy encounters the same random event}

• _{Simulate stochastic environment}



_{Robot control}



_{Elevator scheduling (search for patterns)}



_{Telecommunications (finding networks)}



_{Games (Chess, Backgammon)}



_{Financial trading}

Summary of Learning

Supervised Unsupervised Reinforcement

Training

data Data and correct

output Data

States, actions, and

rewards

Learning

target Data-output relationship Patterns in data Policy

Evaluation Statistics Fitness Reward value

Typical

1. What is unsupervised learning? Explain with an example? Why it is important?

2. What is k-means clustering? How it works? 3. What is reinforcement learning?

4. Explain Q-learning in reinforcement learning. 5. Describe exploration function and policy in

reinforcement learning.