With the rise of deep neural networks for quantum chemistry applications,
there is a pressing need for architectures that, beyond delivering accurate
predictions of chemical properties, are readily interpretable by researchers.
Here, we describe interpretation techniques for atomistic neural networks on
the example of Behler-Parrinello networks as well as the end-to-end model
SchNet. Both models obtain predictions of chemical properties by aggregating
atom-wise contributions. These latent variables can serve as local explanations
of a prediction and are obtained during training without additional cost. Due
to their correspondence to well-known chemical concepts such as atomic energies
and partial charges, these atom-wise explanations enable insights not only
about the model but more importantly about the underlying quantum-chemical
regularities. We generalize from atomistic explanations to 3d space, thus
obtaining spatially resolved visualizations which further improve
interpretability. Finally, we analyze learned embeddings of chemical elements
that exhibit a partial ordering that resembles the order of the periodic table.
As the examined neural networks show excellent agreement with chemical
knowledge, the presented techniques open up new venues for data-driven research
in chemistry, physics and materials science.