We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties: It is highly nonconvex, except possibly for a few exotic activation functions... (read more)
PDFMETHOD  TYPE  

ReLU

Activation Functions 