Equations · Machine learning
ReLU activation
The rectified linear unit: pass positive inputs through unchanged and set negative ones to zero.
ReLU(x) = max(0, x)
Aethel says
ReLU of x equals the maximum of 0 and x
What it says
It is the simplest non-linear function in common use. Being cheap to compute and having a gradient of one for positive inputs, it made deep networks much easier to train than earlier S-shaped activations.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| ReLU | ReLU, rhymes with day-loo | the rectified linear unit |
| max(0, x) | the maximum of 0 and x | the larger of 0 and x |
Where you meet it
Neural-network architectures of every kind.
Questions
How do you say ReLU activation out loud?
ReLU(x) = max(0, x) is read: “ReLU of x equals the maximum of 0 and x.” That is exactly what Aethel says when it meets the formula in a book.
What does ReLU mean in ReLU activation?
ReLU (said “ReLU, rhymes with day-loo”) is the rectified linear unit.
What does max(0, x) mean in ReLU activation?
max(0, x) (said “the maximum of 0 and x”) is the larger of 0 and x.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.