IST 4 Information and Logic

IST 4

Information and Logic

HW3 will be returned today average is A

Great job!

mon tue wed thr fri

2 _M1

1

9 _M1

16

1 2

23 _PCP

30

2

7

3

14

21

3 4

28

4 5

4

5

x= hw#x out x= hw#x due

Mx= MQx out Mx= MQx due

classno

oh oh

oh oh

oh oh

oh

oh

oh oh oh oh oh oh

oh oh

oh

= office hours

T

= today

T oh

oh

oh

sun

oh

PCP = Programing Challenge

midterms

oh

oh oh

- Lecture 1: Life: DNA sequences and evolution - Lecture 2: Human brain: natural languages

- Lecture 3: Artificial languages: numbers and writing

(limited) memory and innovation process (artificial languages)

information systems

- Lecture 4: Languages for quantities: Babylonian mathematics

- Lecture 5: Contrast: Babylonian mathematics vs Greek mathematics - Lecture 6: Flow: Euclid to Algorizmi to Fibonacci to Leibniz

- Lecture 8:- Lecture 7: Beyond arithmetic – a language of syntax boxesLeibniz – arithmetic, reasoning – it is all about syntax!

- Lecture 9+10: Boolean Algebra - Proofs are fun!

- Lecture 13: Shannon: design, synthesis, simplicity

- Lecture 11: Boolean is a useful language: functions and syntax boxes

- Lecture 12: Shannon: from symbols to physics

- Lecture 15: Human brain: neural circuits and learning

- Lecture 14: Life: searching for a language – stochastic circuits

A relay circuit is

a physical system for syntax manipulation

Relay circuits are not the only option!

OR gate AND gate

a t b

Linear Threshold (LT) gate (deep) learning:

adjusting the weights

Circuits with Gates

functional gates – syntax boxes AON: AND, OR, Not

LT: Linear Threshold

Efficiency and complexity Feasibility

If feasible, how many

blocks are needed? Algorizm?

Questions about building blocks?

Given a set of building blocks:

What can/cannot be constructed?

AND, OR and NOT (AON)

a b

a b a

b

What is the function computed by this circuit?

longest path from input to output – counting the number of gates

total number of gates in the circuit

2

3

Every 0-1 Boolean Function Can be

Implemented Using A Depth Two AON Circuit

Implement the DNF representation: OR of many ANDs

abc

001010 011

0 11 100 0

101110 111

10 01

XOR of 3 Variables

> > >

> >

a b c a b c a b c a b c

Depth = 2 Size = 5

is the complement

XOR of 3 Variables

How many gates in a depth-2 circuit for XOR of n variables with AON?

XOR of More Variables?

Surprisingly, this is the optimal size for depth-2

Depth-2 AON Circuit for XOR

Theorem: An optimal size depth-2 AON circuit for has gates Proof:

The construction follows from the DNF representation:

normal terms + one OR gate

The lower bound: WLOG

> >

>>

ab c

ab c ab c ab c

>>>> >>

>>>>

ab c

ab c ab c ab c

(i) Every AND gate must have all n inputs

(ii) Every AND gate computes a normal term DNF is a representation, hence,

there are AND gates

???

Without Loss Of

Generality??

Depth-2 AON Circuit for XOR

Proof (cont):

Need to prove: (i) Every AND gate must have all n inputs By contradiction: Assume that there is a gate G ^{with n-1}

inputs . Say x

is missing from G

Assume that:

Hence, the output of the circuit is 1

OR gate has input of 1 Making G=1 ?

>> >

>>

ab c

ab c ab c ab c

>>>> >>

>>>>

ab c

ab c ab c ab c

set a variable to 1

and a complement to 0 Theorem: An optimal size depth-2 AON circuit for

has gates

>

bc

0 1 0

1

Note that the following two assignments force the output of the circuit to be 1:

Depth-2 AON Circuit for XOR

Proof (cont):

Assume that:

Hence, the output of the circuit is 1 (OR gate has input of 1)

Contradiction!!

Q Those assignments have different parities

So what?

Theorem: An optimal size depth-2 AON circuit for

has gates

How many gates in a depth 2 circuit for XOR of n variables with AON?

It is optimal size for depth-2

n=4 , depth 2, size 9

Q: for n=4, arbitrary depth,

suggest a circuit for XOR

with size less than 9?

Size 8 AON Circuit for XOR of Four Variables

XOR(x,y,z) b

c

a XOR(x,y)

d

size 5 size 3

XOR(a,b,c,d)

Idea:

Compute a large XOR by using a circuit of small XOR gates

Arbitrary depth circuit for XOR of n variables with AON?

AON Circuit for XOR

Idea:

Compute a large XOR by using a circuit of small XOR gates

XOR 8 variables

Tree edge = wire

in-degree = 2 leaf =

input edge

node = XOR gate

Idea:

Compute a large XOR by using a circuit of small XOR gates

XOR 8 variables

Circuit size

in AON gates?

Size =

Node size X number of nodes 3 X 7 = 21

Q: Can we do better for 8 variables?

Note that we need size 129 in depth-2…

Idea: Use a larger in-degree?

9 variables

Size =

Node size X number of nodes 5 X 4 = 20

Note that we need size 21 with in-degree 2 XOR

Size 18 for 8 variables

Q: Can we do better for 8 variables?

Idea: Use a larger in-degree?

9 variables

Size =

Node size X number of nodes 5 X 4 = 20

Note that we need size 21 with in-degree 2 XOR

Size 18 for 8 variables

In general, we can prove that degree-3

XOR trees are the best! Size is

AON Constructions for XOR

circuit kind size AON, d-2

AON

optimal

not optimal n=4

9 8

lower bound:

Circuits with Gates

LT: Linear Threshold

Neuron – Neural Gate

LT: Linear Threshold

-2 1

1 ^{0 0} _{0 1}

1 0 1 1

-2 -1 -1 0

0 0 0 1

LT: Linear Threshold

What is the function computed by this gate?

Neural Circuits

feasibility

2 input Linear Threshold (LT) gate

Q: Are LT gates magical?

LT: Linear Threshold

Q: Are LT gates magical?

LT: Linear Threshold

Idea: A Linear Threshold is Magical

Can compute AND, OR and NOT

We showed that we can compute the AND function with an LT gate

-2 1

1 ^{0 0} _{0 1}

1 0 1 1

-2 -1 -1 0

0

0

0

1

Can We Compute an OR Function with an LT Gate?

-1 1

1 ^{0 0} _{0 1}

1 0 1 1

-1 0 0 1

0

1

1

1

Can We Compute a NOT with an LT Gate?

-2 1

Can we compute NOT

without sgn?

More Variables for AND?

Hence is an AND

More Variables for OR?

Hence is an OR

Circuits

Efficiency and complexity

The Functions of the Adder

carry

2 symbol adder c

s

d1 d2

c

sum

XOR with a Single LT Gate

Is it possible to compute with a single LT gate?

Idea: Find weights w

, w

and w

such that:

2 symbol adder c

s

d1 d2

c

Is it possible to compute with a single LT gate?

Answer : NO

Proof: By contradiction

assume it is possible and reach a contradiction co ntr

adict ion !!

Q

2 symbol adder c

s

d1 d2

c

XOR with a Single LT Gate

XOR with More Variables?

Is it possible to compute with a single LT gate?

Idea: suppose that it is possible, and reach a contradiction

However, And, Contradiction

2 symbol adder c

s

d1 d2

c

Need LT circuits

for XOR!

MAJ with a Single LT Gate

Is it possible to compute with a single LT gate?

|X| MAJ

0 0

1 0

2 1

3 1

2 symbol adder c

s

d1 d2

c

AND, OR, XOR and MAJ are symmetric functions

|X| AND OR XOR MAJ

0 0 0 0 0

1 0 1 1 0

2 0 1 0 1

3 1 1 1 1

LT

= the class of Boolean functions that can be realized by a single LT gate.

LT

LT

LT

LT

Q: Which symmetric functions are in LT

?

|X| AND OR XOR MAJ

0 0 0 0 0

1 0 1 1 0

2 0 1 0 1

3 1 1 1 1

Definition:

A symmetric Boolean function is in TH if it has at most a single transition in the symmetric function table

= a transition

Not in TH

In TH

The Class TH

The Class TH - Single Transition

|X| TH

TH

TH

TH

TH

TH

TH

TH

0 1 0 0 0 0 1 1 1

1 1 1 0 0 0 0 1 1

2 1 1 1 0 0 0 0 1

3 1 1 1 1 0 0 0 0

Q: what is |TH| ?

the number TH functions...

A: 2n+2

= a transition

Claim:

Q

0 1

Proof:

|X| TH

TH

TH

TH

TH

TH

TH

TH

0 1 0 0 0 0 1 1 1

1 1 1 0 0 0 0 1 1

2 1 1 1 0 0 0 0 1

3 1 1 1 1 0 0 0 0

The Class TH is in LT ₁

Also true (wo proof): TH functions are the only

symmetric functions that can be realized by a single LT gate

AON and Linear Threshold Circuits XOR example

Need LT circuits

for XOR!

XOR of Three Variables

> > >

> >

a b c a b c a b c a b c

Depth = 2 Size = 5

Size 5 is optimal for AON depth 2

is the complement

Size 4 LT depth 2

LT gates are MORE Powerful

1 1 1

-1 -1 -1

1 1 1

1 -1

-3

1 1 1

-2

FOR XOR: Size 5 is optimal for AON depth 2

1 1 1

-1 -1 -1

1 1 1

1 -1

-3

1 1 1

-2

A

B

C

0 1 2 3

0 1 1 1

0 0 0 1

A B C

1

1 0 0

Can take the sgn or add 1

1 2 1 2

-1 0 -1 0

LT gates are MORE Powerful

LT-l = LT layered

inputs go to first layer only

TH functions

XOR Function: Size of LT vs AON in Depth 2

5 4

AON LT-l

*

*

* = it is optimal Exponential gap in size

5 4

AON LT-l

General construction for symmetric functions

Linear Threshold Circuits

symmetric functions

LT Depth-2 Circuits

-1 +

TH ¹

TH ² _{|X| TH}

1

TH

^TH

^+TH

^-1

0 0 1 0

1 1 1 1

2 1 0 0

???

|X| f(x)

0 0

1 1

2 1

3 0

4 0

Generalization

|X| f(x)

0 0

1 1

2 1

3 0

4 0

Generalization

|X| f(x) TH

0 0 0

1 1 1

2 1 1

3 0 1

4 0 1

Generalization

|X| f(x) TH

TH

0 0 0 1

1 1 1 1

2 1 1 1

3 0 1 0

4 0 1 0

Generalization

|X| f(x) TH

TH

Σ -1

0 0 0 1 0

1 1 1 1 1

2 1 1 1 1

3 0 1 0 0

4 0 1 0 0

Generalization

|X| f(x) TH

TH

Σ -1

0 0 0 1 0

1 1 1 1 1

2 1 1 1 1

3 0 1 0 0

4 0 1 0 0

-1 +

Generalization to SYM

Q: What is the generalization to arbitrary symmetric functions?

Generalization to SYM

Q: What is the generalization to arbitrary symmetric functions?

A: Consider the symmetric function table, it is a sum of non-overlapping 1-intervals

0

0 1

1

Sum of two TH functions

Back to XOR

0 1 2 3 4 5

0 1 0 1 0 1

n TH gates for XOR of n variables

LT-l Circuit Design Algorithm for SYM

0 1 2 3 4 5

1 1 0 1 1 0

f(X)

6 7

1 1

Subtract 1 for every

isolated 1-block

The Layered Construction for SYM Some History

Saburo Muroga 1925- 2009

1959

Was born in Japan

PhD in 1958 from Tokyo U, Japan

1960-1964: Researcher at IBM Research, NY

1964-2002: professor at the University of Illinois, Urbana-Champaign

Majority Decision

The brain and computation

some history...

Aristotle (2300YA) Cajal ^{(~125 YA)}

Santiago Ramón y Cajal 1852 -1934, Spain

- a neuron is of the nervous system

- neurons with each other via specialized junctions, or spaces, between cells –

The Brain is a

Neural Circuit

Nobel Prize in Physiology or Medicine in 1906

Joint with Golgi

neural circuits and logic

some more history... ~75YA

Being Homeless and

Interdisciplinary Research

Warren McCulloch

1899 - 1969 Walter Pitts 1923 - 1969 Neurophysiologist, MD

Warren McCulloch arrived in early 1942 to the University of Chicago, invited Pitts, who was homeless, to live with his family

In the evenings McCulloch and Pitts collaborated.

Pitts was familiar with the work of Leibniz on computing.

They considered the question of whether the nervous system is a kind of universal computing device as described by Leibniz

This led to their 1943 seminal neural networks paper:

A Logical Calculus of Ideas Immanent in Nervous Activity

Logician, Autodidact

Warren McCulloch

1899 - 1969 Walter Pitts 1923 - 1969 Neurophysiologist, MD

This led to their 1943 seminal neural networks paper:

A Logical Calculus of Ideas Immanent in Nervous Activity

Logician, Autodidact

Neural networks ^and Logic Time Memory Threshold Logic

and Learning State Machines

Impact

Neural Networks - Three Waves

194X - McCulloch and Pitts

195X - Rosenblatt (Perceptron, Learning)…

195X - Muroga (LT Circuits)…

196X - Minsky, Papert (analysis and limitations)…

198X - Hopfield (associative memories)

Rumelhart, McClelland, Hinton (learning in networks)

…

197X - Werbos (learning through backpropagation - networks)…

201x - ‘deep learning’

WWW, access to data, fast computation, applications…

Neural Networks - Three Waves

194X - McCulloch and Pitts

195X - Rosenblatt (Perceptron, Learning)…

195X - Muroga (LT Circuits)…

196X - Minsky, Papert (analysis and limitations)…

198X - Hopfield (associative memories)

Rumelhart, McClelland, Hinton (learning)

…

197X - Werbos (learning through backpropagation - networks)…

201x - ‘deep learning’

WWW, access to data, fast computation, applications…

195X - Rosenblatt (Perceptron, Learning)…

Algorithms vs Learning

What is the difference:

Learning vs Algorithms

Awareness of the solution process

No Yes

1234

1394

2628

+

Awareness of the solution process

No Yes

1234 1394 2628 +

Computers outperform humans!!!

Computers outperform humans???

Can computers

outperform the human brain?

Can computers

outperform the human brain?

By learning? algorithmically?

Awareness of the solution process

No Yes

Recent progress:

Tasks where there is

no awareness of the solution process

learning outperforms humans!

HW#5

(6,2,0,2) (4,2,2,3)

LT

= Can be computed by a single LT gate In LT

– show a construction

Not in LT

– show a proof

MQs

2. Entrepreneurship (Thursday 6/7)

1. Everyone has a gift! (Tuesday 6/5)

• Alyica Lee – An Insatiable Curiosity

• Joe Como - The Gift of Laughter

• Karen Guo – The Gift of a New Perspective

• Sara Fish – The Gift of Disability

• Adam Kogan – The Gift of Speech

• Emma Qian – The Gift of Alternative Modes of Mobility

Tuesday, 6/5, 2:30pm –

• LC Chen – Madam CJ Walker

• Andrew Chan – The Caltech Bubble Tea Revolution

• Noah Sock - Ralphie

• Dylan Lu - The Aquarium PC Build

• Jacob Ressler-Craig – The Iraqi Boy

• James McLaughlin – The Bitcoin Era

• George Daghlian - The Struggles of an Entrepreneur

Thursday, 6/7, 2:30pm –

Summary

How?

Teach about the historical ^innovation process

Process ^vs Snapshot

The purpose of IST4 is to impart the sensation of ideas as they are

conceived and not as they are known

Progress happens with

the introduction of new languages

DNA

spoken language written language number systems

mathematics proofs syllogism

Boolean algebra syntax boxes

relays molecular switches

AON gates neural gates

probability stochastic relays

algebra

complexity learning

evolution

synthesis analysis

abstractions algorizms

memory

abacus

brain

axioms

The appearance of life is

the first Information Megamorphosis The appearance of the human brain is the second Information Megamorphosis

What will be

the third Information Megamorphosis?

The Final Question!