"RELU activations"

ReLU-based activations: analysis and experimental study for deep learning

Activation functions are used in neural networks as a tool to introduce non-linear transformations into the model and, thus, enhance its representation capabilities. They also determine the output range of the hidden layers and the final output. …