When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?

no code implementations • 13 Mar 2024 • Gautham Govind Anil, Pascal Esser, Debarghya Ghoshdastidar

We provide the first convergence results of NTK for contrastive losses, and present a nuanced picture: NTK of wide networks remains almost constant for cosine similarity based contrastive losses, but not for losses based on dot product similarity.

Contrastive Learning