ReLU is only half what it should be:

If you double the number of weight parameters in a neural network you can make a better ReLU:

You can combine that idea together with using the fast Walsh Hadamard transform as a connectionist device to make this type of neural network: