Search Results for author:
Found 73 papers, 46 papers with code
A Practitioner's Guide to Continual Multimodal Pretraining
In this work, we complement current perspectives on continual pretraining through a research test bed as well as provide comprehensive guidance for effective continual model updates in such scenarios.
GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
As such, there is a pressing need for a new generation of benchmarks challenging enough for the next generation of LMMs.
On scalable oversight with weak LLMs judging strong LLMs
We find that debate outperforms consultancy across all tasks when the consultant is randomly assigned to argue for the correct/incorrect answer.
HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits
Verifying whether an X note is helpful or whether a Wikipedia edit should be accepted are hard tasks that require grounding by querying the web.
Inverse Constitutional AI: Compressing Preferences into Principles
In constitutional AI, a set of principles (or constitution) is used to provide feedback and fine-tune AI models.
A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision
To this end, we introduce a multi-task Transformer model, CSLR2, that is able to ingest a signing sequence and output in a joint embedding space between signed language and spoken language text.
SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation
In this work, we present SciFIBench, a scientific figure interpretation benchmark.
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs).
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation.
Lifelong Benchmarks: Efficient Model Evaluation in an Era of Rapid Progress
However, with repeated testing, the risk of overfitting grows as algorithms over-exploit benchmark idiosyncrasies.
A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval
Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset.
InstructVideo: Instructing Video Diffusion Models with Human Feedback
To tackle this problem, we propose InstructVideo to instruct text-to-video diffusion models with human feedback by reward fine-tuning.
Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs
Multimodal large language models (MLLMs) have shown remarkable capabilities across a broad range of tasks but their knowledge and abilities in the geographic and geospatial domains are yet to be explored, despite potential wide-ranging benefits to navigation, environmental research, urban development, and disaster response.
Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models
This finding points to a blind spot in current frontier VLMs: they excel in recognizing semantic content but fail to acquire an understanding of visual data-types through scaling.
Simple Baselines for Interactive Video Retrieval with Questions and Answers
To date, the majority of video retrieval systems have been optimized for a "single-shot" scenario in which the user submits a query in isolation, ignoring previous interactions with the system.
RLIPv2: Fast Scaling of Relational Language-Image Pre-training
In this paper, we propose RLIPv2, a fast converging model that enables the scaling of relational pre-training to large-scale pseudo-labelled scene graph data.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO-DET
(using extra training data)
arXiVeri: Automatic table verification with GPT
In this paper, we propose to meet this challenge through the novel task of automatic table verification (AutoTV), in which the objective is to verify the accuracy of numerical data in tables by cross-referencing cited sources.
GPT4GEO: How a Language Model Sees the World's Geography
Large language models (LLMs) have shown remarkable capabilities across a broad range of tasks involving question answering and the generation of coherent text and code.
Zero-shot Unsupervised Transfer Instance Segmentation
Segmentation is a core computer vision competency, with applications spanning a broad range of scientifically and economically valuable domains.
SATIN: A Multi-Task Metadataset for Classifying Satellite Imagery using Vision-Language Models
In this work, we introduce SATellite ImageNet (SATIN), a metadataset curated from 27 existing remotely sensed datasets, and comprehensively evaluate the zero-shot transfer classification capabilities of a broad range of vision-language (VL) models on SATIN.
Can GPT-4 Perform Neural Architecture Search?
We investigate the potential of GPT-4~\cite{gpt4} to perform Neural Architecture Search (NAS) -- the task of designing effective neural architectures.
Large Language Models are Few-shot Publication Scoopers
Driven by recent advances AI, we passengers are entering a golden age of scientific discovery.
DeepMIM: Deep Supervision for Masked Image Modeling
Deep supervision, which involves extra supervisions to the intermediate features of a neural network, was widely used in image classification in the early deep learning era since it significantly reduces the training difficulty and eases the optimization like avoiding gradient vanish over the vanilla training.
Moment Detection in Long Tutorial Videos
To study this problem, we propose the first dataset of untrimmed, long-form tutorial videos for the task of Moment Detection called the Behance Moment Detection (BMD) dataset.
SuS-X: Training-Free Name-Only Transfer of Vision-Language Models
Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models.
Weakly-supervised Fingerspelling Recognition in British Sign Language Videos
Through extensive evaluations, we verify our method for automatic annotation and our model architecture.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Crosslingual Generalization through Multitask Finetuning
We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus.
Ranked #1 on
Question Answering
on StoryCloze
NamedMask: Distilling Segmenters from Complementary Foundation Models
Our method, termed NamedMask, begins by using CLIP to construct category-specific archives of images.
RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection
The task of Human-Object Interaction (HOI) detection targets fine-grained visual parsing of humans interacting with their environment, enabling a broad range of applications.
Ranked #17 on
Human-Object Interaction Detection
on HICO-DET
Automatic dense annotation of large-vocabulary sign language videos
Recently, sign language researchers have turned to sign language interpreted TV broadcasts, comprising (i) a video of continuous signing and (ii) subtitles corresponding to the audio content, as a readily available and large-scale source of training data.
ReCo: Retrieve and Co-segment for Zero-shot Transfer
Semantic segmentation has a broad range of applications, but its real-world impact has been significantly limited by the prohibitive annotation costs necessary to enable deployment.
Scaling up sign spotting through sign language dictionaries
The focus of this work is $\textit{sign spotting}$ - given a video of an isolated sign, our task is to identify $\textit{whether}$ and $\textit{where}$ it has been signed in a continuous, co-articulated sign language video.
A 23 MW data centre is all you need
The field of machine learning has achieved striking progress in recent years, witnessing breakthrough results on language modelling, protein folding and nitpickingly fine-grained dog breed classification.
Unsupervised Salient Object Detection with Spectral Cluster Voting
In this paper, we tackle the challenging task of unsupervised salient object detection (SOD) by leveraging spectral clustering on self-supervised features.
Ranked #1 on
Unsupervised Saliency Detection
on ECSSD
Cross Modal Retrieval with Querybank Normalisation
In this work we first show that, despite their effectiveness, state-of-the-art joint embeddings suffer significantly from the longstanding "hubness problem" in which a small number of gallery embeddings form the nearest neighbours of many queries.
Ranked #5 on
Video Retrieval
on QuerYD
Audio Retrieval with Natural Language Queries: A Benchmark Study
Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho.
Ranked #1 on
Audio to Text Retrieval
on SoundDescs
BBC-Oxford British Sign Language Dataset
In this work, we introduce the BBC-Oxford British Sign Language (BOBSL) dataset, a large-scale video collection of British Sign Language (BSL).
Adaptive Cross-Modal Prototypes for Cross-Domain Visual-Language Retrieval
In this paper, we study the task of visual-text retrieval in the highly practical setting in which labelled visual data with paired text descriptions are available in one domain (the "source"), but only unlabelled visual data (without text descriptions) are available in the domain of interest (the "target").
Aligning Subtitles in Sign Language Videos
The goal of this work is to temporally align asynchronous subtitles in sign language videos.
Audio Retrieval with Natural Language Queries
We consider the task of retrieving audio using free-form natural language queries.
Ranked #1 on
Audio/Video to Text Retrieval
on AudioCaps
Sign Segmentation with Changepoint-Modulated Pseudo-Labelling
The objective of this work is to find temporal boundaries between signs in continuous sign language.
TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval
In recent years, considerable progress on the task of text-video retrieval has been achieved by leveraging large-scale pretraining on visual and audio datasets to construct powerful video encoders.
All you need are a few pixels: semantic segmentation with PixelPick
A central challenge for the task of semantic segmentation is the prohibitive cost of obtaining dense pixel-level annotations to supervise model training.
On the Origin of Species of Self-Supervised Learning
In the quiet backwaters of cs. CV, cs. LG and stat. ML, a cornucopia of new learning systems is emerging from a primordial soup of mathematics-learning systems with no need for external supervision.
Read and Attend: Temporal Localisation in Sign Language Videos
Our contributions are as follows: (1) we demonstrate the ability to leverage large quantities of continuous signing videos with weakly-aligned subtitles to localise signs in continuous sign language; (2) we employ the learned attention to automatically generate hundreds of thousands of annotations for a large sign vocabulary; (3) we collect a set of 37K manually verified sign instances across a vocabulary of 950 sign classes to support our study of sign language recognition; (4) by training on the newly annotated data from our method, we outperform the prior state of the art on the BSL-1K sign language recognition benchmark.
Quantum Self-Supervised Learning
The resurgence of self-supervised learning, whereby a deep learning model generates its own supervisory signal from the data, promises a scalable way to tackle the dramatically increasing size of real-world data sets without human annotation.
Sign language segmentation with temporal convolutional networks
The objective of this work is to determine the location of temporal boundaries between signs in continuous sign language videos.
QuerYD: A video dataset with high-quality text and audio narrations
We introduce QuerYD, a new large-scale dataset for retrieval and event localisation in video.
Watch, read and lookup: learning to spot signs from multiple supervisors
The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video.
Seeing wake words: Audio-visual Keyword Spotting
The goal of this work is to automatically determine whether and when a word of interest is spoken by a talking face, with or without the audio.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality.
Ranked #6 on
Sign Language Recognition
on WLASL-2000
State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication
Peer review forms the backbone of modern scientific manuscript evaluation.
Iterative Averaging in the Quest for Best Test Error
We analyse and explain the increased generalisation performance of iterate averaging using a Gaussian process perturbation model between the true and batch risk surface on the high dimensional quadratic.
Disentangled Speech Embeddings using Cross-modal Self-supervision
The objective of this paper is to learn representations of speaker identity without access to manually annotated data.
Unsupervised Learning of Landmarks by Descriptor Vector Exchange
Equivariance to random image transformations is an effective method to learn landmarks of object categories, such as the eyes and the nose in faces, without manual supervision.
Ranked #1 on
Unsupervised Facial Landmark Detection
on 300W
Use What You Have: Video Retrieval Using Representations From Collaborative Experts
The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge.
Ranked #24 on
Video Retrieval
on MSVD
Deep Industrial Espionage
The theory of deep learning is now considered largely solved, and is well understood by researchers and influencers alike.
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
We also propose a parametric gather-excite operator pair which yields further performance gains, relate it to the recently-introduced Squeeze-and-Excitation Networks, and analyse the effects of these changes to the CNN feature activation statistics.
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets.
Ranked #3 on
Facial Expression Recognition (FER)
on FERPlus
Facial Emotion Recognition
Facial Expression Recognition (FER)
+1
Semi-convolutional Operators for Instance Segmentation
Object detection and instance segmentation are dominated by region-based methods such as Mask RCNN.
Small steps and giant leaps: Minimal Newton solvers for Deep Learning
Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration.
PyTorch CurveBall - A second-order optimizer for deep networks
We propose a fast second-order method that can be used as a drop-in replacementfor current deep learning solvers.
Learnable PINs: Cross-Modal Embeddings for Person Identity
We propose and investigate an identity sensitive joint embedding of face and voice.
Self-supervised Learning of Geometrically Stable Features Through Probabilistic Introspection
Self-supervision can dramatically cut back the amount of manually-labelled data required to train deep neural networks.
Seeing Voices and Hearing Faces: Cross-modal biometric matching
We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching, (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available), and (iii) we use human testing as a baseline to calibrate the difficulty of the task.
Substitute Teacher Networks: Learning with Almost No Supervision
Learning through experience is time-consuming, inefficient and often bad for your cortisol levels.
Squeeze-and-Excitation Networks
Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2. 251%, surpassing the winning entry of 2016 by a relative improvement of ~25%.
Ranked #6 on
Object Detection
on DSEC
Stopping GAN Violence: Generative Unadversarial Networks
While the costs of human violence have attracted a great deal of attention from the research community, the effects of the network-on-network (NoN) violence popularised by Generative Adversarial Networks have yet to be addressed.
Unknowable Manipulators: Social Network Curator Algorithms
For a social networking service to acquire and retain users, it must find ways to keep them engaged.
Learning Grimaces by Watching TV
As a starting point, we consider the problem of relating facial expressions to objectively measurable events occurring in videos.
