IST 4 Information and Logic

IST 4

Information and Logic

mon tue wed thr fri

1 _M1

1

8 _M1

15

1 2

22 PCP

2

29 _M2

6

3

13

3 4

20 PCP

4 5

27

3

5

x= hw#x out x= hw#x due

Mx= MQx out Mx= MQx due

oh oh

oh oh

oh oh

oh

oh

oh oh

oh

oh oh oh

oh

oh oh

oh

= office hours

T

= today

T oh

oh

oh

oh

sun

PCP= Programming Challenge

Midterms

Last Lecture: languages for biology

- Stochastic chemical networks

Stochastic logic design The B- algorithm

Duality

- Molecular switches

DNA strand displacement

- Stochastic flow networks

Feedback helps!

2 1

2 1

2 1

2 3 1

2

3 1

- Lecture 1: Life: DNA sequences and evolution - Lecture 2: Human brain: natural languages

- Lecture 3: Artificial languages: numbers and writing

Information and Logic

- Lecture 4: Languages for quantities: Babylonian mathematics

- Lecture 5: Contrast: Babylonian mathematics vs Greek mathematics - Lecture 6: Flow: Euclid to Algorizmi to Fibonacci to Leibniz

- Lecture 9: Boolean Algebra – a new language for logic - Lecture 7: Beyond arithmetic: a language of syntax boxes - Lecture 8: Leibniz: syntax for reasoning, Aristotle and Boole

- Lecture 11: George Boole and nonbinary algebras

- Lecture 10: Boolean Algebra – from axioms to theorems

- Lecture 13: Shannon: from symbols to physics

- Lecture 12: Boolean is a useful language: functions and syntax boxes - Lecture 14: Shannon: synthesis, simplicity, creativity

- Lecture 15: searching for a language for Biology

- Lecture 16: Neural circuits: feasibility and constructions

A relay circuit is

a physical system for syntax manipulation Relay circuits are not the only option!

OR gate AND gate

a t b

Linear Threshold (LT) gate (deep) learning:

adjusting the weights

Circuits with Gates

functional gates – syntax boxes AON: AND, OR, Not

LT: Linear Threshold

Efficiency and complexity Feasibility

If feasible, how many

blocks are needed? Algorizm?

Questions about building blocks?

Given a set of building blocks:

What can/cannot be constructed?

AON: AND, OR, Not

Depth 2 construction and a lower bound

AND, OR and NOT (AON)

a b

a b a

b

What is the function computed by this circuit?

longest path from input to output – counting the number of gates

total number of gates in the circuit

2

3

Every 0-1 Boolean Function Can be

Implemented Using A Depth Two AON Circuit

Implement the DNF representation: OR of many ANDs

abc

001010 011

0 11 100 0

101110 111

10 01

XOR of 3 Variables

> > >

> >

a b c a b c a b c a b c

Depth = 2 Size = 5

is the complement

XOR of 3 Variables

How many gates in a depth-2 circuit for XOR of n variables with AON?

XOR of More Variables?

Surprisingly, this is the optimal size for depth-2

Depth-2 AON Circuit for XOR

Theorem: An optimal size depth-2 AON circuit for has gates Proof:

The construction follows from the DNF representation:

normal terms + one OR gate

The lower bound: WLOG

> >

>>

ab c

ab c ab c ab c

>>>> >>

>>>>

ab c

ab c ab c ab c

Without Loss Of

Generality??

Depth-2 AON Circuit for XOR

Theorem: An optimal size depth-2 AON circuit for has gates Proof:

The lower bound: WLOG

> >

>>

ab c

ab c ab c ab c

>>>> >>

>>>>

ab c

ab c ab c ab c

x If x=1 the output of the circuit is 1

(OR gate) independent of the other inputs

Not computing XOR!

Depth-2 AON Circuit for XOR

Theorem: An optimal size depth-2 AON circuit for has gates Proof (cont.):

(i) Every AND gate must have all n inputs (ii) Every AND gate computes a normal term

DNF is a representation, hence, there are AND gates

???

>> >

>>

ab c

ab c ab c ab c

>>>> >>

>>>>

ab c

ab c ab c ab c

Depth-2 AON Circuit for XOR

Proof (cont):

Need to prove: (i) Every AND gate must have all n inputs By contradiction: Assume that there is a gate G ^{with n-1}

inputs . Say x

is missing from G

Assume that:

Hence, the output of the circuit is 1

OR gate has input of 1 Making G=1 ^?

>> >

>>

ab c

ab c ab c ab c

>>>> >>

>>>>

ab c

ab c ab c ab c

set a variable to 1

and a complement to 0 Theorem: An optimal size depth-2 AON circuit for

has gates

>

bc

0 1 0

1

Note that the following two assignments force the output of the circuit to be 1:

Depth-2 AON Circuit for XOR

Proof (cont):

Assume that:

Hence, the output of the circuit is 1 (OR gate has input of 1)

Contradiction!!

Q Those assignments have different parities

So what?

Theorem: An optimal size depth-2 AON circuit for

has gates

How many gates in a depth 2 circuit for XOR of n variables with AON?

It is optimal size for depth-2

n=4 , depth 2, size 9

Q: for n=4, arbitrary depth,

suggest a circuit for XOR

with size less than 9?

Size 8 AON Circuit for XOR of Four Variables

XOR(x,y,z) b

c

a XOR(x,y)

d

size 5 size 3

XOR(a,b,c,d)

Idea:

Compute a large XOR by using a circuit of small XOR gates

Arbitrary depth circuit for XOR of n variables with AON?

AON Circuit for XOR

Idea:

Compute a large XOR by using a circuit of small XOR gates

XOR 8 variables

Tree edge = wire

in-degree = 2 leaf =

input edge

node = XOR gate

Idea:

Compute a large XOR by using a circuit of small XOR gates

XOR 8 variables

Circuit size

in AON gates?

Size =

Node size X number of nodes 3 X 7 = 21

Q: Can we do better for 8 variables?

Note that we need size 129 in depth-2…

Idea: Use a larger in-degree?

9 variables

Size =

Node size X number of nodes 5 X 4 = 20

Note that we need size 21 with in-degree 2 XOR

Size 18 for 8 variables

Q: Can we do better for 8 variables?

Idea: Use a larger in-degree?

9 variables

Size =

Node size X number of nodes 5 X 4 = 20

Note that we need size 21 with in-degree 2 XOR

Size 18 for 8 variables

In general, we can prove that degree-3

XOR trees are the best! Size is

AON Constructions for XOR

circuit kind size AON, d-2

AON

optimal

not optimal n=4

9 8

lower bound:

LT: Linear Threshold

Definition

Neuron – Neural Gate

LT: Linear Threshold

-2 1

1 ^{0 0} _{0 1}

1 0 1 1

-2 -1 -1 0

0 0 0 1

LT: Linear Threshold

What is the function computed by this gate?

Neural Circuits

feasibility

2 input Linear Threshold (LT) gate

Q: Are LT gates magical?

LT: Linear Threshold

Q: Are LT gates magical?

LT: Linear Threshold

Idea: A Linear Threshold is Magical

Can compute AND, OR and NOT

We showed that we can compute the AND function with an LT gate

-2 1

1 ^{0 0} _{0 1}

1 0 1 1

-2 -1 -1 0

0

0

0

1

Can We Compute an OR Function with an LT Gate?

-1 1

1 ^{0 0} _{0 1}

1 0 1 1

-1 0 0 1

0

1

1

1

Can We Compute a NOT with an LT Gate?

-2 1

Can we compute NOT

without sgn?

More Variables for AND?

Hence is an AND

More Variables for OR?

Hence is an OR

Neural Circuits

Efficiency and complexity

The Functions of the Adder

carry

2 symbol adder c

s

d1 d2

c

sum

XOR with a Single LT Gate

Is it possible to compute with a single LT gate?

Idea: Find weights w

^, w

^and w

^{such that:}

2 symbol adder c

s

d1 d2

c

Is it possible to compute with a single LT gate?

Answer : NO

Proof: By contradiction

assume it is possible and reach a contradiction co ntr

adict ion !!

Q

2 symbol adder c

s

d1 d2

c

XOR with a Single LT Gate

XOR with More Variables?

Is it possible to compute with a single LT gate?

Hint:

2 symbol adder c

s

d1 d2

c

XOR with More Variables?

Is it possible to compute with a single LT gate?

Idea: suppose that it is possible, and reach a contradiction

However, And, Contradiction

2 symbol adder c

s

d1 d2

c

Need LT circuits

for XOR!

MAJ with a Single LT Gate

Is it possible to compute with a single LT gate?

|X| MAJ

0 0

1 0

2 1

3 1

2 symbol adder c

s

d1 d2

c

AND, OR, XOR and MAJ are symmetric functions

|X| AND OR XOR MAJ

0 0 0 0 0

1 0 1 1 0

2 0 1 0 1

3 1 1 1 1

LT

= the class of Boolean functions that can be realized by a single LT gate.

LT

LT

LT

LT

Q: Which symmetric functions are in LT

?

|X| AND OR XOR MAJ

0 0 0 0 0

1 0 1 1 0

2 0 1 0 1

3 1 1 1 1

Definition:

A symmetric Boolean function is in TH if it has at most a single transition in the symmetric function table

= a transition

Not in TH

In TH

|X| AND AND OR OR

0 0 1 0 1

1 0 1 1 0

2 0 1 1 0

3 1 0 1 0

= a transition

Every TH function has a complement, called TH

TH

TH

TH

TH

The index of TH is the index after the transition

The Class TH - Single Transition

|X| TH

TH

TH

TH

TH

TH

TH

TH

0 1 0 0 0 0 1 1 1

1 1 1 0 0 0 0 1 1

2 1 1 1 0 0 0 0 1

3 1 1 1 1 0 0 0 0

Q: what is |TH| ?

the number of TH functions...

A: 2n+2

= a transition

end with 1 end with 0

The Class TH

end with 1

end with 1

Claim:

Q

0 1

Proof:

|X| TH

TH

TH

TH

TH

TH

TH

TH

0 1 0 0 0 0 1 1 1

1 1 1 0 0 0 0 1 1

2 1 1 1 0 0 0 0 1

3 1 1 1 1 0 0 0 0

The Class TH is in LT ₁

Also true (wo proof): TH functions are the only

symmetric functions that can be realized by a single LT gate

AON and Linear Threshold Circuits XOR example

Need LT circuits

for XOR!

XOR of Three Variables

> > >

> >

a b c a b c a b c a b c

Depth = 2 Size = 5

Size 5 is optimal for AON depth 2

is the complement

Size 4 LT depth 2

LT gates are MORE Powerful

1 1 1

-1 -1 -1

1 1 1

1 -1

-3

1 1 1

-2

FOR XOR: Size 5 is optimal for AON depth 2

1 1 1

-1 -1 -1

1 1 1

1 -1

-3

1 1 1

-2

A

B

C

0 1 2 3

0 1 1 1

0 0 0 1

A B C

1

1 0 0

Can take the sgn or add 1

1 2 1 2

-1 0 -1 0

LT gates are MORE Powerful

LT-l = LT layered

inputs go to first layer only

TH functions

XOR Function: Size of LT vs AON in Depth 2

5 4

AON LT-l

*

*

* = it is optimal Exponential gap in size

5 4

AON LT-l

General construction for symmetric functions

Linear Threshold Circuits

symmetric functions

LT Depth-2 Circuits

-1 +

TH ¹

TH ² _{|X| TH}

1

TH

^TH

^+TH

^-1

0 0 1 0

1 1 1 1

2 1 0 0

???

|X| f(x)

0 0

1 1

2 1

3 0

4 0

Generalization

|X| f(x)

0 0

1 1

2 1

3 0

4 0

Generalization

|X| f(x) TH

0 0 0

1 1 1

2 1 1

3 0 1

4 0 1

Generalization

|X| f(x) TH

TH

0 0 0 1

1 1 1 1

2 1 1 1

3 0 1 0

4 0 1 0

Generalization

|X| f(x) TH

TH

Σ -1

0 0 0 1 0

1 1 1 1 1

2 1 1 1 1

3 0 1 0 0

4 0 1 0 0

Generalization

|X| f(x) TH

TH

Σ -1

0 0 0 1 0

1 1 1 1 1

2 1 1 1 1

3 0 1 0 0

4 0 1 0 0

|X| f(x) TH

TH

TH

TH

0 0 0 1 0 1

1 1 1 1 0 1

2 0 1 0 0 1

3 1 1 0 1 1

4 0 1 0 1 0

Q: What is the generalization to arbitrary symmetric functions?

Q: What is the generalization to arbitrary symmetric functions?

A: Consider the symmetric function table, it is a sum of non-overlapping 1-intervals

Sum of two TH functions

Back to XOR

0 1 2 3 4 5

0 1 0 1 0 1

n TH gates for XOR of n variables

LT-l Circuit Design Algorithm for SYM

0 1 2 3 4 5

1 1 0 1 1 0

f(X)

6 7

1 1

Subtract 1 for every

isolated 1-block

The Layered Construction for SYM Some History

Saburo Muroga 1925- 2009

1959

Was born in Japan

PhD in 1958 from Tokyo U, Japan

1960-1964: Researcher at IBM Research, NY

1964-2002: professor at the University of Illinois, Urbana-Champaign

Majority Decision

The brain and computation

some history...

Aristotle (2300YA) Cajal ^{(~125 YA)}

“Learn by heart”

“coolheaded”

Santiago Ramón y Cajal 1852 -1934, Spain

- a neuron is of the nervous system

- neurons with each other via specialized junctions, or spaces, between cells –

The Brain is a

Neural Circuit

Nobel Prize in Physiology or Medicine in 1906

Joint with Golgi

neural circuits and logic

some more history... ~75YA

Being Homeless and

Interdisciplinary Research

Warren McCulloch

1899 - 1969 Walter Pitts 1923 - 1969 Neurophysiologist, MD

Warren McCulloch arrived in early 1942 to the University of Chicago, invited Pitts, who was homeless, to live with his family

In the evenings McCulloch and Pitts collaborated.

Pitts was familiar with the work of Leibniz on computing.

They considered the question of whether the nervous system is a kind of universal computing device as described by Leibniz

This led to their 1943 seminal neural networks paper:

A Logical Calculus of Ideas Immanent in Nervous Activity

Logician, Autodidact

Warren McCulloch

1899 - 1969 Walter Pitts 1923 - 1969 Neurophysiologist, MD

This led to their 1943 seminal neural networks paper:

A Logical Calculus of Ideas Immanent in Nervous Activity

Logician, Autodidact

Neural networks ^and Logic Time Memory Threshold Logic

and Learning State Machines

Impact

Neural Networks - Three Waves

194X - McCulloch and Pitts

195X - Rosenblatt (Perceptron, Learning)…

195X - Muroga (LT Circuits)…

196X - Minsky, Papert (analysis and limitations)…

198X - Hopfield (associative memories)

Rumelhart, McClelland, Hinton (learning in networks)

…

197X - Werbos (learning through backpropagation - networks)…

201x - ‘deep learning’

WWW, access to data, fast computation, applications…

Neural Networks - Three Waves

194X - McCulloch and Pitts

195X - Rosenblatt (Perceptron, Learning)…

195X - Muroga (LT Circuits)…

196X - Minsky, Papert (analysis and limitations)…

198X - Hopfield (associative memories)

Rumelhart, McClelland, Hinton (learning)

…

197X - Werbos (learning through backpropagation - networks)…

201x - ‘deep learning’

WWW, access to data, fast computation, applications…

1958 - Rosenblatt (Perceptron, Learning)…

July 11, 1928 – July 11, 1971

1962

HW#5

(6,2,0,2) (4,2,2,3)

LT

= Can be computed by a single LT gate In LT

– show a construction

Not in LT

– show a proof

Summary

How?

Teach about the historical ^innovation process

Process ^vs Snapshot

The purpose of IST4 is to impart the sensation of ideas as they are

conceived and not as they are known

Progress happens with

the introduction of new languages

DNA

spoken language written language number systems

mathematics proofs syllogism

Boolean algebra syntax boxes

relays molecular switches

AON gates neural gates

probability stochastic relays

algebra

complexity learning

evolution

synthesis analysis

abstractions algorizms

memory

abacus

brain

axioms