Search Results for author:
Found 76 papers, 38 papers with code
Evaluating the Generation Capabilities of Large Chinese Language Models
This paper presents CG-Eval, the first comprehensive evaluation of the generation capabilities of large Chinese language models across a wide range of academic disciplines.
AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?
We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and a top-down approach that infers the goal of the actor and plans the needed procedure to accomplish the goal.
Does Visual Pretraining Help End-to-End Reasoning?
A positive result would refute the common belief that explicit visual abstraction (e. g. object detection) is essential for compositional generalization on visual reasoning, and confirm the feasibility of a neural network "generalist" to solve visual recognition and reasoning tasks.
NLOS Dies Twice: Challenges and Solutions of V2X for Cooperative Perception
Multi-agent multi-lidar sensor fusion between connected vehicles for cooperative perception has recently been recognized as the best technique for minimizing the blind zone of individual vehicular perception systems and further enhancing the overall safety of autonomous driving systems.
Federated Learning over a Wireless Network: Distributed User Selection through Random Access
This study proposes a network intrinsic approach of distributed user selection that leverages the radio resource competition mechanism in random access.
Goal-Conditioned Predictive Coding as an Implicit Planner for Offline Reinforcement Learning
Recent work has demonstrated the effectiveness of formulating decision making as a supervised learning problem on offline-collected trajectories.
How can objects help action recognition?
In this paper, we investigate how we can use knowledge of objects to design better video models, namely to process fewer tokens and to improve recognition accuracy.
Dense Video Object Captioning from Disjoint Supervision
We show our task is more general than grounding, and models trained on our task can directly be applied to grounding by finding the bounding box with the maximum likelihood of generating the query sentence.
2nd Place Winning Solution for the CVPR2023 Visual Anomaly and Novelty Detection Challenge: Multimodal Prompting for Data-centric Anomaly Detection
This technical report introduces the winning solution of the team Segment Any Anomaly for the CVPR2023 Visual Anomaly and Novelty Detection (VAND) challenge.
AVIS: Autonomous Visual Information Seeking with Large Language Models
Second, we use examples of user decision-making to provide our LLM-powered planner and reasoner with relevant contextual instances, enhancing their capacity to make informed decisions.
Segment Any Anomaly without Training via Hybrid Prompt Regularization
We present a novel framework, i. e., Segment Any Anomaly + (SAA+), for zero-shot anomaly segmentation with hybrid prompt regularization to improve the adaptability of modern foundation models.
Ranked #1 on
Anomaly Detection
on KSDD2
End-to-End Spatio-Temporal Action Localisation with Video Transformers
The most performant spatio-temporal action localisation models use external person proposals and complex external memory banks.
Ranked #1 on
Action Recognition
on AVA v2.1
(using extra training data)
Comparing Trajectory and Vision Modalities for Verb Representation
Given the importance of 3D space in formal models of verb semantics, we expect that these 2D images would result in impoverished representations that fail to capture nuanced differences in meaning.
Steerable Equivariant Representation Learning
In this paper, we propose a method of learning representations that are instead equivariant to data augmentations.
DEJA VU: Continual Model Generalization For Unseen Domains
To overcome these limitations of DA and DG in handling the Unfamiliar Period during continual domain shift, we propose RaTP, a framework that focuses on improving models' target domain generalization (TDG) capability, while also achieving effective target domain adaptation (TDA) capability right after training on certain domains and forgetting alleviation (FA) capability on past domains.
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
REVEAL consists of four key components: the memory, the encoder, the retriever and the generator.
Ranked #1 on
Visual Question Answering (VQA)
on A-OKVQA
(Accuracy metric)
ConSpec: honing in on critical steps for rapid learning and generalization in RL
These critical steps are challenging to identify with traditional reinforcement learning (RL) methods that rely on the Bellman equation for credit assignment.
A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect Detection
The proposed DKAN method follows a pretraining-finetuning transfer learning paradigm and a knowledge distillation framework is designed for fine-tuning.
TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency
In this work, we focus on summarizing instructional videos, an under-explored area of video summarization.
Beyond Transfer Learning: Co-finetuning for Action Localisation
Transfer learning is the predominant paradigm for training deep networks on small target datasets.
Do Trajectories Encode Verb Meaning?
Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world.
AVATAR: Unconstrained Audiovisual Speech Recognition
Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning Audio-Video Modalities from Image Captions
To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort.
Ranked #19 on
Zero-Shot Video Retrieval
on MSR-VTT
(using extra training data)
Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?
CompMap first asks a VL model to generate primitive concept activations with text prompts, and then learns to construct a composition model that maps the primitive concept activations (e. g. the likelihood of black tail or red wing) to composite concepts (e. g. a red-winged blackbird).
Trajectory balance: Improved credit assignment in GFlowNets
Generative flow networks (GFlowNets) are a method for learning a stochastic policy for generating compositional objects, such as graphs or strings, from a given unnormalized density by sequences of actions, where many possible action sequences may lead to the same object.
Multiview Transformers for Video Recognition
Video understanding requires reasoning at multiple spatiotemporal resolutions -- from short fine-grained motions to events taking place over longer durations.
Ranked #4 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
Masking Modalities for Cross-modal Video Retrieval
Our proposal is to pre-train a video encoder using all the available video modalities as supervision, namely, appearance, sound, and transcribed speech.
Does Vision-and-Language Pretraining Improve Lexical Grounding?
Linguistic representations derived from text alone have been criticized for their lack of grounding, i. e., connecting words to their meanings in the physical world.
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets
In this work, we propose an anchor-free and end-to-end trajectory prediction model, named DenseTNT, that directly outputs a set of trajectories from dense goal candidates.
Discrete-Valued Neural Communication
Deep learning has advanced from fully connected architectures to structured models organized into components, e. g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes.
Attention Bottlenecks for Multimodal Fusion
Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio.
Ranked #1 on
Action Classification
on Kinetics-Sounds
HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps
To address this issue, we introduce a new challenging task to generate HD maps.
Episodic Transformer for Vision-and-Language Navigation
We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance.
Local Metrics for Multi-Object Tracking
This paper introduces temporally local metrics for Multi-Object Tracking.
Composable Augmentation Encoding for Video Representation Learning
We focus on contrastive methods for self-supervised video representation learning.
Unified Graph Structured Models for Video Understanding
Accurate video understanding involves reasoning about the relationships between actors, objects and their environment, often over long temporal intervals.
ViViT: A Video Vision Transformer
We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification.
Ranked #8 on
Action Classification
on Moments in Time
(Top 5 Accuracy metric, using extra
training data)
Learning Temporal Dynamics from Cycles in Narrated Video
Learning to model how the world changes as time elapses has proven a challenging problem for the computer vision community.
TNT: Target-driveN Trajectory Prediction
Our key insight is that for prediction within a moderate time horizon, the future modes can be effectively captured by a set of target states.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Learning Video Representations from Textual Web Supervision
Based on this observation, we propose to use text as a method for learning video representations.
Multi-modal Transformer for Video Retrieval
In this paper, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT
(text-to-video Mean Rank metric, using extra
training data)
Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos
Despite the recent advances in video classification, progress in spatio-temporal action recognition has lagged behind.
What Makes for Good Views for Contrastive Learning?
Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning.
Ranked #2 on
Contrastive Learning
on imagenet-1k
VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e. g. pedestrians and vehicles) and road context information (e. g. lanes, traffic lights).
Speech2Action: Cross-modal Supervision for Action Recognition
We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments.
Automated Pyramid Summarization Evaluation
Pyramid evaluation was developed to assess the content of paragraph length summaries of source texts.
Unsupervised Learning of Object Structure and Dynamics from Videos
Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning.
Ranked #11 on
Video Prediction
on KTH
Learning Video Representations using Contrastive Bidirectional Transformer
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Modeling Parts, Structure, and System Dynamics via Predictive Learning
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Predicting the Present and Future States of Multi-agent Systems from Partially-observed Visual Data
We present a method which learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.
VideoBERT: A Joint Model for Video and Language Representation Learning
Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube.
Ranked #1 on
Action Classification
on YouCook2
Affordance Learning In Direct Perception for Autonomous Driving
Our results indicate that this method could act as a cheaper way for training data collection in autonomous driving.
Unsupervised Discovery of Parts, Structure, and Dynamics
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Stochastic Prediction of Multi-Agent Interactions from Partial Observations
We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.
Machine Learning Allows Calibration Models to Predict Trace Element Concentration in Soil with Generalized LIBS Spectra
Machine learning has been used to develop the model.
D3D: Distilled 3D Networks for Video Action Recognition
State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input.
Ranked #11 on
Action Recognition
on AVA v2.1
Composing Text and Image for Image Retrieval - An Empirical Odyssey
In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image.
Ranked #2 on
Image Retrieval with Multi-Modal Query
on MIT-States
Actor-Centric Relation Network
A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.
Ranked #15 on
Action Recognition
on AVA v2.1
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning
We propose a measure to estimate domain similarity via Earth Mover's Distance and demonstrate that transfer learning benefits from pre-training on a source domain that is similar to the target domain by this measure.
Ranked #27 on
Fine-Grained Image Classification
on CUB-200-2011
Fine-Grained Image Classification
Fine-Grained Visual Categorization
+1
DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networks
We propose an action recognition framework using Gen- erative Adversarial Networks.
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification.
Ranked #24 on
Action Recognition
on UCF101
(using extra training data)
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
Many seemingly distant annotations (e. g., semantic segmentation and visual question answering (VQA)) are inherently connected in that they reveal different levels and perspectives of human understandings about the same visual scenes --- and even the same set of images (e. g., of COCO).
The iNaturalist Species Classification and Detection Dataset
Existing image classification datasets used in computer vision tend to have a uniform distribution of images across object categories.
Ranked #8 on
Image Classification
on iNaturalist
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
What will happen if we increase the dataset size by 10x or 100x?
Ranked #2 on
Semantic Segmentation
on PASCAL VOC 2007
AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1. 58M action labels with multiple labels per person occurring frequently.
Ranked #6 on
Action Detection
on UCF101-24
TALL: Temporal Activity Localization via Language Query
For evaluation, we adopt TaCoS dataset, and build a new dataset for this task on top of Charades by adding sentence temporal annotations, called Charades-STA.
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
Temporal Action Proposal (TAP) generation is an important problem, as fast and accurate extraction of semantically important (e. g. human actions) segments from untrimmed videos is an important step for large-scale video analysis.
Ranked #8 on
Action Recognition
on THUMOS’14
Complex Event Recognition from Images with Few Training Examples
We propose to leverage concept-level representations for complex event recognition in photographs given limited training examples.
Speed/accuracy trade-offs for modern convolutional object detectors
On the opposite end in which accuracy is critical, we present a detector that achieves state-of-the-art performance measured on the COCO detection task.
Ranked #224 on
Object Detection
on COCO test-dev
ACD: Action Concept Discovery from Image-Sentence Corpora
It obtains candidate action concepts by extracting verb-object pairs from sentences and verifies their visualness with the associated images.
ProNet: Learning to Propose Object-specific Boxes for Cascaded Neural Networks
This paper aims to classify and locate objects accurately and efficiently, without using bounding box annotations.
Ranked #5 on
Weakly Supervised Object Detection
on COCO
Automatic Concept Discovery from Parallel Text and Visual Corpora
Humans connect language and vision to perceive the world.
Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
To solve this problem, we propose a simple yet effective method that takes weak video labels and noisy image labels as input, and generates localized action frames as output.
DISCOVER: Discovering Important Segments for Classification of Video Events and Recounting
Our goal is to find the important segments and capture their information for event classification and recounting.
