ReLU is only half what it should be:

If you double the number of weight parameters in a neural network you can make a better ReLU:

You can combine that idea together with using the fast Walsh Hadamard transform as a connectionist device to make this type of neural network:
https://editor.p5js.org/siobhan.491/sketches/83c-4I3Ys