Multi-Layer Neural Networks

The Problem That Nearly Killed Neural Networks

At the end of the previous chapter, we discovered a major limitation.

A perceptron can learn:

• AND
• OR

But cannot learn:

• XOR

The reason is simple.

A perceptron creates only one linear decision boundary.

In two dimensions: w1x1 + w2x2 + b = 0 represents a straight line.

If a problem cannot be separated by a straight line, a single perceptron fails.

This became known as the linear separability problem.

Understanding XOR Geometrically

Consider XOR:

Graphically:

(0, 1)			(1, 1)
  +				   -
  
  
  
  -                +
(0, 0)           (1, 0)

No single line can separate positive and negative examples.

Therefore:

Single Layer Perceptron = Failure

The question became:

What kind of network can solve XOR?