Search Results for author:
Found 44 papers, 14 papers with code
Learnable Pooling Regions for Image Classification
Biologically inspired, from the early HMAX model to Spatial Pyramid Matching, pooling has played an important role in visual recognition pipelines.
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision.
Towards a Visual Turing Challenge
As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process.
A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
Over the last two decades we have witnessed strong progress on modeling visual object classes, scenes and attributes that have significantly contributed to automated image understanding.
Hard to Cheat: A Turing Test based on Answering Questions about Images
Progress in language and image understanding by machines has sparkled the interest of the research community in more open-ended, holistic tasks, and refueled an old AI dream of building intelligent machines.
Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question).
Contextual Media Retrieval Using Natural Language Queries
We show that our retrieval system can cope with this variability using personalisation through an online learning-based retrieval formulation.
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering
By combining latest advances in image representation and natural language processing, we propose Ask Your Neurons, a scalable, jointly trained, end-to-end formulation to this problem.
Spatio-Temporal Image Boundary Extrapolation
Furthermore, we show long-term prediction of boundaries in situations where the motion is governed by the laws of physics.
Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
We present Mean Box Pooling, a novel visual representation that pools over CNN representations of a large number, highly overlapping object proposals.
Tutorial on Answering Questions about Images with Deep Learning
Together with the development of more accurate methods in Computer Vision and Natural Language Understanding, holistic architectures that answer on questions about the content of real-world images have emerged.
Natural Language Understanding
Visual Question Answering (VQA)
Long-Term Image Boundary Prediction
Boundary estimation in images and videos has been a very active topic of research, and organizing visual information into boundaries and segments is believed to be a corner stone of visual perception.
A simple neural network module for relational reasoning
Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn.
Image Retrieval with Multi-Modal Query
Question Answering
+2
Learning to Navigate in Cities Without a Map
We present an interactive navigation environment that uses Google StreetView for its photographic content and worldwide coverage, and demonstrate that our learning method allows agents to learn to navigate multiple cities and to traverse to target destinations that may be kilometres away.
Hyperbolic Attention Networks
We introduce hyperbolic attention networks to endow neural networks with enough capacity to match the complexity of data with hierarchical and power-law structure.
Relational inductive biases, deep learning, and graph networks
As a companion to this paper, we have released an open-source software library for building graph networks, with demonstrations of how to use them in practice.
Learning Visual Question Answering by Bootstrapping Hard Attention
Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs.
Ranked #7 on
Visual Question Answering (VQA)
on CLEVR
The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
Visual QA is a pivotal challenge for higher-level reasoning, requiring understanding language, vision, and relationships between many objects in a scene.
Playing the Game of Universal Adversarial Perturbations
We study the problem of learning classifiers robust to universal adversarial perturbations.
Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
In this work, we study the setting in which an agent must learn to generate programs for diverse scenes conditioned on a given symbolic instruction.
Learning To Follow Directions in Street View
Navigating and understanding the real world remains a key challenge in machine learning and inspires a great variety of research in areas such as language grounding, planning, navigation and computer vision.
The StreetLearn Environment and Dataset
These datasets cannot be used for decision-making and reinforcement learning, however, and in general the perspective of navigation as an interactive learning task, where the actions and behaviours of a learning agent are learned simultaneously with the perception and planning, is relatively unsupported.
Learning dynamic polynomial proofs
In this work, we introduce a machine learning based method to search for a dynamic proof within these proof systems.
Sideways: Depth-Parallel Training of Video Models
We propose Sideways, an approximate backpropagation scheme for training video models.
Visual Grounding in Video for Unsupervised Word Translation
Given this shared embedding we demonstrate that (i) we can map words between the languages, particularly the 'visual' words; (ii) that the shared embedding provides a good initialization for existing unsupervised text-based word translation techniques, forming the basis for our proposed hybrid visual-text mapping algorithm, MUVE; and (iii) our approach achieves superior performance by addressing the shortcomings of text-based methods -- it is more robust, handles datasets with less commonality, and is applicable to low-resource languages.
IReEn: Reverse-Engineering of Black-Box Functions via Iterative Neural Program Synthesis
In this work, we investigate the problem of revealing the functionality of a black-box agent.
Broaden Your Views for Self-Supervised Video Learning
Most successful self-supervised learning methods are trained to align the representations of two independent views from the data.
Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning
How many fruits do you have in total?
Gradient Forward-Propagation for Large-Scale Temporal Video Modelling
How can neural networks be trained on large-volume temporal data efficiently?
Learning Altruistic Behaviours in Reinforcement Learning without External Rewards
Such an approach assumes that other agents' goals are known so that the altruistic agent can cooperate in achieving those goals.
Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning.
How many fruits do you have in total?
Measuring CLEVRness: Black-box Testing of Visual Reasoning Models
To answer such a question, we extend the visual question answering framework and propose the following behavioral test in the form of a two-player game.
General-purpose, long-context autoregressive modeling with Perceiver AR
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression.
Ranked #35 on
Language Modelling
on WikiText-103
Measuring CLEVRness: Blackbox testing of Visual Reasoning Models
Visual question answering provides a convenient framework for testing the model's abilities by interrogating the model through questions about the scene.
Transframer: Arbitrary Frame Prediction with Generative Models
We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction.
CLIP-CLOP: CLIP-Guided Collage and Photomontage
The unabated mystique of large-scale neural networks, such as the CLIP dual image-and-text encoder, popularized automatically generated art.
Neural Payoff Machines: Predicting Fair and Stable Payoff Allocations Among Team Members
Cooperative game theory offers solution concepts identifying distribution schemes, such as the Shapley value, that fairly reflect the contribution of individuals to the performance of the team or the Core, which reduces the incentive of agents to abandon their team.
Compressed Vision for Efficient Video Understanding
In this work, we propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos.
Perception Test: A Diagnostic Benchmark for Multimodal Models
We propose a novel multimodal benchmark – the Perception Test – that aims to extensively evaluate perception and reasoning skills of multimodal models.
A Simple, Yet Effective Approach to Finding Biases in Code Generation
This work shows that current code generation systems exhibit undesired biases inherited from their large language model backbones, which can reduce the quality of the generated code under specific circumstances.
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e. g. Flamingo, SeViLA, or GPT-4).
SODA: Bottleneck Diffusion Models for Representation Learning
We introduce SODA, a self-supervised diffusion model, designed for representation learning.
Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models
Large Language Models (LLMs) demonstrate ever-increasing abilities in mathematical and algorithmic tasks, yet their geometric reasoning skills are underexplored.
