ML0045 Multi-Layer Perceptron

A Multi-Layer Perceptron (MLP) is a feedforward neural network with one or more hidden layers between the input and output layers. Its hidden layers use non-linear activation functions (ReLU, sigmoid, or tanh) to model complex relationships, and it is trained with backpropagation, which adjusts the weights to minimize errors. MLPs are used for classification, regression, and function approximation.

(1) Hidden Layers + Non-Linearity: Each hidden unit applies a non-linear activation to its weighted sum, letting the network compose many linear maps into a curved function.
(2) Overcomes The Perceptron: Unlike a single-layer perceptron (restricted to linearly separable problems), an MLP learns non-linear decision boundaries and handles problems such as XOR.
(3) Universal Approximation: With enough neurons and layers, an MLP can approximate any continuous function, making it a powerful general-purpose model.

Figure 1: An MLP for 3-class classification: 4 inputs, two hidden layers (6 and 4 units) with non-linear activations, and 3 softmax-style outputs. Every layer is fully connected to the next.

Mathematical Formulation:

Where:

• is the activation vector of layer ( is the input).
• are the weight matrix and bias of layer , learned by backpropagation.
• is the hidden non-linearity (ReLU/sigmoid/tanh); is the output map (softmax for classification, identity for regression).

Figure 2: The payoff: on XOR, the perceptron’s single line misclassifies half the positives, while the MLP’s hidden units carve a curved region that separates the classes perfectly.