Search Results for author:
Found 113 papers, 66 papers with code
Measuring the Importance of Temporal Features in Video Saliency
Where people look when watching videos is believed to be heavily influenced by temporal patterns.
Are We Done with Object-Centric Learning?
However, with recent sample-efficient segmentation models, we can separate objects in the pixel space and encode them independently.
Understanding the Limits of Lifelong Knowledge Editing in LLMs
Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge.
Project Alexandria: Towards Freeing Scientific Knowledge from Copyright Burdens via LLMs
Paywalls, licenses and copyright rules often restrict the broad dissemination and reuse of scientific knowledge.
Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation
There is growing excitement about the potential of Language Models (LMs) to accelerate scientific discovery.
Testing the limits of fine-tuning to improve reasoning in vision language models
In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment.
Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
We show that CLIP's performance on these samples can be accurately predicted from the pretraining frequencies of individual objects.
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute.
Great Models Think Alike and this Undermines AI Oversight
We study how model similarity affects both aspects of AI oversight by proposing a probabilistic metric for LM similarity based on overlap in model mistakes.
How to Merge Your Multimodal Models Over Time?
In reality, new tasks and domains emerge progressively over time, requiring strategies to integrate the knowledge of expert models as they become available: a process we call temporal model merging.
ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities
Overall, we present a technique for open-ended evaluation, which can aggregate over incomplete, heterogeneous sample-level measurements to continually grow a benchmark alongside the rapidly developing foundation models.
Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli
In this work, we seek to better understand the computational basis for this capability by evaluating a broad range of optical flow models and a neuroscience inspired motion energy model for zero-shot figure-ground segmentation of random dot stimuli.
Centaur: a foundation model of human cognition
Establishing a unified theory of cognition has been a major goal of psychology.
In Search of Forgotten Domain Generalization
In the ImageNet era of computer vision, evaluation sets for measuring a model's OOD performance were designed to be strictly OOD with respect to style.
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?
Our further token-level perplexity observations reveals that the perplexity degradation is due to a handful of tokens that are not informative about the domain.
A Practitioner's Guide to Continual Multimodal Pretraining
In this work, we complement current perspectives on continual pretraining through a research test bed as well as provide comprehensive guidance for effective continual model updates in such scenarios.
CiteME: Can Language Models Accurately Cite Scientific Claims?
We pose the following research question: Given a text excerpt referencing a paper, could an LM act as a research assistant to correctly identify the referenced paper?
Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Models
(2) Indeed, we show how most often, P-RFCL techniques can be matched by a simple and lightweight PEFT baseline.
Identifying latent state transition in non-linear dynamical systems
This work aims to improve generalization and interpretability of dynamical systems by recovering the underlying lower-dimensional latent states and their time evolutions.
The Entropy Enigma: Success and Failure of Entropy Minimization
After many steps of optimization, EM makes the model embed test images far away from the embeddings of training images, which results in a degradation of accuracy.
Wu's Method can Boost Symbolic AI to Rival Silver Medalists and AlphaGeometry to Outperform Gold Medalists at IMO Geometry
In this note, we revisit the IMO-AG-30 Challenge introduced with AlphaGeometry, and find that Wu's method is surprisingly strong.
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation.
Efficient Lifelong Model Evaluation in an Era of Rapid Progress
To address this challenge, we introduce an efficient framework for model evaluation, Sort & Search (S&S)}, which reuses previously evaluated models by leveraging dynamic programming algorithms to selectively rank and sub-select test samples.
Investigating Continual Pretraining in Large Language Models: Insights and Implications
This paper studies the evolving domain of Continual Learning (CL) in large language models (LLMs), with a focus on developing strategies for efficient and sustainable training.
Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization
In a simple setting with direct supervision on the generative factors, we show how learning class-agnostic transformations offers a way to circumvent catastrophic forgetting and improve classification accuracy over time.
Most discriminative stimuli for functional cell type clustering
Thus, for unbiased identification of the functional cell types in retina and visual cortex, new approaches are needed.
Visual cognition in multimodal large language models
A chief goal of artificial intelligence is to build machines that think like people.
Continual Learning: Applications and the Road Forward
Continual learning is a subfield of machine learning, which aims to allow machine learning models to continuously learn on new data, by accumulating knowledge without forgetting what was learned in the past.
Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?
Foundation models like CLIP are trained on hundreds of millions of samples and effortlessly generalize to new tasks and inputs.
Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models
This finding points to a blind spot in current frontier VLMs: they excel in recognizing semantic content but fail to acquire an understanding of visual data-types through scaling.
Provable Compositional Generalization for Object-Centric Learning
Learning representations that generalize to novel compositions of known concepts is crucial for bridging the gap between human and machine perception.
RDumb: A simple approach that questions our progress in continual test-time adaptation
Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time.
Playing repeated games with Large Language Models
In a large set of two players-two strategies games, we find that LLMs are particularly good at games where valuing their own self-interest pays off, like the iterated Prisoner's Dilemma family.
Modulated Neural ODEs
Neural ordinary differential equations (NODEs) have been proven useful for learning non-linear dynamics of arbitrary trajectories.
Disentanglement and Generalization Under Correlation Shifts
Exploiting such correlations may increase predictive performance on noisy data; however, often correlations are not robust (e. g., they may change between domains, datasets, or applications) and models that exploit them do not generalize when correlations shift.
Unsupervised Object Learning via Common Fate
Learning generative object models from unlabelled videos is a long standing problem and required for causal scene modeling.
The Geometry of Adversarial Subspaces
We undertake analysis to characterize the geometry of the boundary, which is more curved within the adversarial subspace than within a random subspace of equal dimensionality.
Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
An important component for generalization in machine learning is to uncover underlying latent factors of variation as well as the mechanism through which each factor acts in the world.
How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
A precise understanding of why units in an artificial network respond to certain stimuli would constitute a big step towards explainable artificial intelligence.
Partial success in closing the gap between human and machine vision
The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets.
DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling
Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction; however, with stagnant progress in the last 3-5 years.
If your data distribution shifts, use self-learning
We demonstrate that self-learning techniques like entropy minimization and pseudo-labeling are simple and effective at improving performance of a deployed computer vision model under systematic domain shifts.
Ranked #1 on
Unsupervised Domain Adaptation
on ImageNet-A
(using extra training data)
Out-of-distribution generalization of internal models is correlated with reward
We investigate the behavior of reinforcement learning (RL) agents under morphological distribution shifts.
Out-of-Distribution Generalization
reinforcement-learning
+1
State-of-the-Art in Human Scanpath Prediction
The last years have seen a surge in models predicting the scanpaths of fixations made by humans when viewing images.
Contrastive Learning Inverts the Data Generating Process
Contrastive learning has recently seen tremendous success in self-supervised learning.
Ranked #1 on
Disentanglement
on KITTI-Masks
Exemplary natural images explain CNN activations better than synthetic feature visualizations
Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images \citep{olah2017feature} with a simple baseline visualization, namely exemplary natural images that also strongly activate a specific feature map.
System Identification with Biophysical Constraints: A Circuit Model of the Inner Retina
Here, we present a computational model of temporal processing in the inner retina, including inhibitory feedback circuits and realistic synaptic release mechanisms.
A Broad Dataset is All You Need for One-Shot Object Detection
We here show that this generalization gap can be nearly closed by increasing the number of object categories used during training.
Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
Even if only a single reference image is given, synthetic images provide less information than natural images ($65\pm5\%$ vs. $73\pm4\%$).
On the surprising similarities between supervised and self-supervised models
In the light of this recent breakthrough, we here compare self-supervised networks to supervised models and human behaviour.
Natural Images are More Informative for Interpreting CNN Activations than State-of-the-Art Synthetic Feature Visualizations
Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al. [45] with a simple baseline visualization, namely natural images that also strongly activate a specific feature map.
EagerPy: Writing Code That Works Natively with PyTorch, TensorFlow, JAX, and NumPy
EagerPy is a Python framework that lets you write code that automatically works natively with PyTorch, TensorFlow, JAX, and NumPy.
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos.
Ranked #1 on
Disentanglement
on Natural Sprites
ImageNet performance correlates with pose estimation robustness and generalization on out-of-domain data
Neural networks are highly effective tools for pose estimation.
Fast Differentiable Clipping-Aware Normalization and Rescaling
When the rescaled perturbation $\eta \vec{\delta}$ is added to a starting point $\vec{x} \in D$ (where $D$ is the data domain, e. g. $D = [0, 1]^n$), the resulting vector $\vec{v} = \vec{x} + \eta \vec{\delta}$ will in general not be in $D$.
Improving robustness against common corruptions by covariate shift adaptation
With the more robust DeepAugment+AugMix model, we improve the state of the art achieved by a ResNet50 model up to date from 53. 6% mCE to 45. 4% mCE.
Ranked #4 on
Unsupervised Domain Adaptation
on ImageNet-R
Benchmarking Unsupervised Object Representations for Video Sequences
Perceiving the world in terms of objects and tracking them through time is a crucial prerequisite for reasoning and scene understanding.
Rotation-invariant clustering of neuronal responses in primary visual cortex
Similar to a convolutional neural network (CNN), the mammalian retina encodes visual information into several dozen nonlinear feature maps, each formed by one ganglion cell type that tiles the visual space in an approximately shift-equivariant manner.
Towards causal generative scene models via competition of experts
Learning how to model complex scenes in a modular way with recombinable components is a pre-requisite for higher-order reasoning and acting in the physical world.
Five Points to Check when Comparing Visual Perception in Humans and Machines
In the second case study, we highlight the difference between necessary and sufficient mechanisms in visual reasoning tasks.
Shortcut Learning in Deep Neural Networks
Deep learning has triggered the current rise of artificial intelligence and is the workhorse of today's machine intelligence.
A simple way to make neural networks robust against diverse image corruptions
The human visual system is remarkably robust against a wide range of naturally occurring variations and corruptions like rain or snow.
Learning From Brains How to Regularize Machines
We propose to regularize CNNs using large-scale neuroscience data to learn more robust neural features in terms of representational similarity.
Pretraining boosts out-of-domain robustness for pose estimation
Neural networks are highly effective tools for pose estimation.
Ranked #1 on
Animal Pose Estimation
on Horse-10
How well do deep neural networks trained on object recognition characterize the mouse visual system?
Recent work on modeling neural responses in the primate visual system has benefited from deep neural networks trained on large-scale object recognition, and found a hierarchical correspondence between layers of the artificial neural network and brain areas along the ventral visual stream.
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving.
Ranked #1 on
Robust Object Detection
on MS COCO
Accurate, reliable and fast robustness evaluation
We here develop a new set of gradient-based adversarial attacks which (a) are more reliable in the face of gradient-masking than other gradient-based attacks, (b) perform better and are more query efficient than current state-of-the-art gradient-based attacks, (c) can be flexibly adapted to a wide range of adversarial criteria and (d) require virtually no hyperparameter tuning.
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions.
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
One-Shot Instance Segmentation
We demonstrate empirical results on MS Coco highlighting challenges of the one-shot setting: while transferring knowledge about instance segmentation to novel object categories works very well, targeting the detection network towards the reference category appears to be more difficult.
Ranked #1 on
One-Shot Instance Segmentation
on MS COCO
Excessive Invariance Causes Adversarial Vulnerability
Despite their impressive performance, deep neural networks exhibit striking failures on out-of-distribution inputs.
A rotation-equivariant convolutional neural network model of primary visual cortex
We present a framework to identify common features independent of individual neurons' orientation selectivity by using a rotation-equivariant convolutional neural network, which automatically extracts every feature at multiple different orientations.
Generalisation in humans and deep neural networks
We compare the robustness of humans and current convolutional deep neural networks (DNNs) on object recognition under twelve different types of image degradations.
Adversarial Vision Challenge
The NIPS 2018 Adversarial Vision Challenge is a competition to facilitate measurable progress towards robust machine vision models and more generally applicable adversarial attacks.
Diverse feature visualizations reveal invariances in early layers of deep neural networks
Here we propose a method to discover invariances in the responses of hidden layer units of deep neural networks.
One-shot Texture Segmentation
We introduce one-shot texture segmentation: the task of segmenting an input image containing multiple textures given a patch of a reference texture.
Towards the first adversarially robust neural network model on MNIST
Despite much effort, deep neural networks remain highly susceptible to tiny input perturbations and even for MNIST, one of the most common toy datasets in computer vision, no neural network model exists for which adversarial perturbations are large and make semantic sense to humans.
Markerless tracking of user-defined features with deep learning
Quantifying behavior is crucial for many applications in neuroscience.
One-Shot Segmentation in Clutter
We tackle the problem of one-shot segmentation: finding and segmenting a previously unseen object in a cluttered scene based on a single instruction example.
Ranked #1 on
One-Shot Segmentation
on Cluttered Omniglot
Trace your sources in large-scale data: one ring to find them all
An important preprocessing step in most data analysis pipelines aims to extract a small set of sources that explain most of the data.
Guiding human gaze with convolutional neural networks
Thus, manipulating fixation patterns to guide human attention is an exciting challenge in digital image processing.
Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models
Such decision-based attacks are (1) applicable to real-world black-box models such as autonomous cars, (2) need less knowledge and are easier to apply than transfer-based attacks and (3) are more robust to simple defences than gradient- or score-based attacks.
Neural system identification for large populations separating “what” and “where”
Traditional methods for neural system identification do not capitalize on this separation of “what” and “where”.
Neural system identification for large populations separating "what" and "where"
Our network scales well to thousands of neurons and short recordings and can be trained end-to-end.
Understanding Low- and High-Level Contributions to Fixation Prediction
This model achieves better performance than all models not using features pre-trained on object recognition, making it a strong baseline to assess the utility of high-level features.
Foolbox: A Python toolbox to benchmark the robustness of machine learning models
Foolbox is a new Python package to generate such adversarial perturbations and to quantify and compare the robustness of machine learning models.
Comparing deep neural networks against humans: object recognition when the signal gets weaker
In addition, we find progressively diverging classification error-patterns between humans and DNNs when the signal gets weaker, indicating that there may still be marked differences in the way humans and current DNNs perform visual object recognition.
Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics
Here we show that no single saliency map can perform well under all metrics.
Comment on "Biologically inspired protection of deep networks from adversarial attacks"
A recent paper suggests that Deep Neural Networks can be protected from gradient-based adversarial perturbations by driving the network activations into a highly saturated regime.
Synthesising Dynamic Textures using Convolutional Neural Networks
Here we present a parametric model for dynamic textures.
Controlling Perceptual Factors in Neural Style Transfer
Neural Style Transfer has shown very exciting results enabling new forms of image manipulation.
DeepGaze II: Reading fixations from deep features trained on object recognition
Here we present DeepGaze II, a model that predicts where people look in images.
Preserving Color in Neural Artistic Style Transfer
This note presents an extension to the neural artistic style transfer algorithm (Gatys et al.).
Image Style Transfer Using Convolutional Neural Networks
Rendering the semantic content of an image in different styles is a difficult image processing task.
Texture Synthesis Using Shallow Convolutional Networks with Random Filters
The current state of the art in parametric texture synthesis relies on the multi-layer feature space of deep CNNs that were trained on natural images.
Signatures of criticality arise in simple neural population models with correlations
Support for this notion has come from a series of studies which identified statistical signatures of criticality in the ensemble activity of retinal ganglion cells.
Neurons and Cognition
A note on the evaluation of generative models
In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional.
A Neural Algorithm of Artistic Style
In fine art, especially painting, humans have mastered the skill to create unique visual experiences through composing a complex interplay between the content and style of an image.
Ranked #1 on
Language Modelling
on A1
(using extra training data)
Generative Image Modeling Using Spatial LSTMs
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels.
A Generative Model of Natural Texture Surrogates
In order to model this variability, we first applied the parametric texture algorithm of Portilla and Simoncelli to image patches of 64X64 pixels in a large database of natural images such that each image patch is then described by 655 texture parameters which specify certain statistics, such as variances and covariances of wavelet coefficients or coefficient magnitudes within that patch.
Texture Synthesis Using Convolutional Neural Networks
Here we introduce a new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition.
Supervised learning sets benchmark for robust spike detection from calcium imaging signals
A fundamental challenge in calcium imaging has been to infer the timing of action potentials from the measured noisy calcium fluorescence traces.
Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
Recent results suggest that state-of-the-art saliency models perform far from optimal in predicting fixations.
Inference and Mixture Modeling with the Elliptical Gamma Distribution
We study modeling and inference with the Elliptical Gamma Distribution (EGD).
How close are we to understanding image-based saliency?
Within the set of the many complex factors driving gaze placement, the properities of an image that are associated with fixations under free viewing conditions have been studied extensively.
Training sparse natural image models with a fast Gibbs sampler of an extended state space
We present a new learning strategy based on an efficient blocked Gibbs sampler for sparse overcomplete linear models.
Evaluating neuronal codes for inference using Fisher information
We characterize the response distribution for the binocular energy model in response to random dot stereograms and find it to be very different from the Poisson-like noise usually assumed.
Neurometric function analysis of population codes
In this way, we provide a new rigorous framework for assessing the functional consequences of noise correlation structures for the representational accuracy of neural population codes that is in particular applicable to short-time population coding.
Bayesian estimation of orientation preference maps
Imaging techniques such as optical imaging of intrinsic signals, 2-photon calcium imaging and voltage sensitive dye imaging can be used to measure the functional organization of visual cortex across different spatial scales.
A joint maximum-entropy model for binary neural population patterns and continuous signals
Second-order maximum-entropy models have recently gained much interest for describing the statistics of binary spike trains.
Hierarchical Modeling of Local Image Features through L_p-Nested Symmetric Distributions
We introduce a new family of distributions, called $L_p${\em -nested symmetric distributions}, whose densities access the data exclusively through a hierarchical cascade of $L_p$-norms.
The Conjoint Effect of Divisive Normalization and Orientation Selectivity on Redundancy Reduction
Bandpass filtering, orientation selectivity, and contrast gain control are prominent features of sensory coding at the level of V1 simple cells.
Receptive Fields without Spike-Triggering
Can we find a concise description for the processing of a whole population of neurons analogous to the receptive field for single neurons?
Ranked #106 on
Image Classification
on STL-10
