Search Results for author:
Found 65 papers, 13 papers with code
What You Say Is What You Show: Visual Narration Detection in Instructional Videos
Narrated "how-to" videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies.
HierVL: Learning Hierarchical Video-Language Embeddings
Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text.
Ego-Only: Egocentric Action Detection without Exocentric Pretraining
On EPIC-Kitchens-100, our Ego-Only even outperforms exocentric pretraining (by 2. 1% on verbs and by 1. 8% on nouns), setting a new state-of-the-art.
Egocentric Video Task Translation
Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e. g., classifying sports in one dataset, tracking animals in another).
HistoPerm: A Permutation-Based View Generation Approach for Learning Histopathologic Feature Representations
In this paper, we introduce HistoPerm, a view generation approach designed for improving the performance of representation learning techniques on histology images in weakly supervised settings.
Deformable Video Transformer
Video transformers have recently emerged as an effective alternative to convolutional networks for action classification.
Calibrating Histopathology Image Classifiers using Label Smoothing
Moreover, we find that using model confidence as a proxy for annotator agreement also improves calibration and accuracy, suggesting that datasets without multiple annotators can still benefit from our proposed label smoothing methods via our proposed confidence-aware label smoothing methods.
Learning To Recognize Procedural Activities with Distant Supervision
In this paper we consider the problem of classifying fine-grained, multi-step activities (e. g., cooking different recipes, making disparate home improvements, creating various forms of arts and crafts) from long videos spanning up to several minutes.
Label Hallucination for Few-Shot Classification
At the same time, training a simple linear classifier on top of "frozen" features learned from the large labeled dataset fails to adapt the model to the properties of the novel classes, effectively inducing underfitting.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Long-Short Temporal Contrastive Learning of Video Transformers
Our approach, named Long-Short Temporal Contrastive Learning (LSTCL), enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories
The standard way of training video models entails sampling at each iteration a single clip from a video and optimizing the clip prediction with respect to the video-level label.
A Multi-View Approach To Audio-Visual Speaker Verification
Although speaker verification has conventionally been an audio-only task, some practical applications provide both audio and visual streams of input.
Is Space-Time Attention All You Need for Video Understanding?
We present a convolution-free approach to video classification built exclusively on self-attention over space and time.
Ranked #2 on
Video Question Answering
on Howto100M-QA
A Petri Dish for Histopathology Image Analysis
With the rise of deep learning, there has been increased interest in using neural networks for histopathology image analysis, a field that investigates the properties of biopsy or resected specimens traditionally manually examined under a microscope by pathologists.
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
We present \textsc{Vx2Text}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio.
Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
By selecting a weight among a fixed set of random values for each individual connection, our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity.
Resolution-Based Distillation for Efficient Histology Image Classification
Our results show that a combination of KD and self-supervision allows the student model to approach, and in some cases, surpass the classification accuracy of the teacher, while being much more efficient.
Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification
Based on the nature of histopathology images, a range of difficulty inherently exists among examples, and, since medical datasets are often labeled by multiple annotators, annotator agreement can be used as a natural proxy for the difficulty of a given example.
COBE: Contextualized Object Embeddings from Narrated Instructional Video
A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms.
Generalized Few-Shot Video Classification with Video Retrieval and Feature Generation
Few-shot learning aims to recognize novel classes from a few examples.
Video Understanding as Machine Translation
With the advent of large-scale multimodal video datasets, especially sequences with audio or transcribed speech, there has been a growing interest in self-supervised learning of video representations.
Stein Variational Inference for Discrete Distributions
In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions.
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation
We introduce a method for simultaneously classifying, segmenting and tracking object instances in a video sequence.
Listen to Look: Action Recognition by Previewing Audio
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical.
Ranked #6 on
Action Recognition
on ActivityNet
STAR-Caps: Capsule Networks with Straight-Through Attentive Routing
Capsule networks have been shown to be powerful models for image classification, thanks to their ability to represent and capture viewpoint variations of an object.
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.
Only Time Can Tell: Discovering Temporal Data for Temporal Modeling
However, in current video datasets it has been observed that action classes can often be recognized without any temporal information from a single frame of video.
UniDual: A Unified Model for Image and Video Understanding
It consists of a shared 2D spatial convolution followed by two parallel point-wise convolutional layers, one devoted to images and the other one used for videos.
Video Modeling with Correlation Networks
Motion is a salient cue to recognize actions in video.
Ranked #76 on
Action Classification
on Kinetics-400
Learning Temporal Pose Estimation from Sparsely-Labeled Videos
To reduce the need for dense annotations, we propose a PoseWarper network that leverages training videos with sparse annotations (every k frames) to learn to perform dense temporal pose propagation and estimation.
Ranked #2 on
Multi-Person Pose Estimation
on PoseTrack2018
(using extra training data)
Attentive Action and Context Factorization
This is a challenging task due to the subtlety of human actions in video and the co-occurrence of contextual elements.
SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition
We demonstrate that the computational cost of action recognition on untrimmed videos can be dramatically reduced by invoking recognition only on these most salient clips.
Ranked #1 on
Action Recognition
on miniSports
Video Classification with Channel-Separated Convolutional Networks
It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks.
Ranked #1 on
Action Recognition
on Sports-1M
DistInit: Learning Video Representations Without a Single Labeled Video
In this work, we propose an alternative approach to learning video representations that require no semantically labeled videos and instead leverages the years of effort in collecting and labeling large and clean still-image datasets.
Ranked #69 on
Action Recognition
on HMDB-51
(using extra training data)
Learning Discriminative Motion Features Through Detection
Our network learns to spatially sample features from Frame B in order to maximize pose detection accuracy in Frame A.
Scenes-Objects-Actions: A Multi-Task, Multi-Label Video Dataset
The videos retrieved by the search engines are then veried for correctness by human annotators.
MaskConnect: Connectivity Learning by Gradient Descent
Such simple connectivity rules are unlikely to yield the optimal architecture for the given problem.
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
There is a natural correlation between the visual and auditive elements of a video.
Ranked #7 on
Self-Supervised Audio Classification
on ESC-50
What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets
The ability to capture temporal information has been critical to the development of video understanding models.
Object Detection in Video with Spatiotemporal Sampling Networks
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable convolutions across time for object detection in videos.
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization
This paper presents a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos.
Ranked #3 on
Temporal Action Localization
on HACS
Detect-and-Track: Efficient Pose Estimation in Videos
This paper addresses the problem of estimating and tracking human body keypoints in complex, multi-person video.
Ranked #7 on
Keypoint Detection
on COCO test-challenge
A Closer Look at Spatiotemporal Convolutions for Action Recognition
In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition.
Ranked #4 on
Action Recognition
on Sports-1M
Connectivity Learning in Multi-Branch Networks
While much of the work in the design of convolutional networks over the last five years has revolved around the empirical investigation of the importance of depth, filter sizes, and number of feature channels, recent studies have shown that branching, i. e., splitting the computation along parallel but distinct threads and then aggregating their outputs, represents a new promising dimension for significant improvements in performance.
Learning to Inpaint for Image Compression
We study the design of deep architectures for lossy image compression.
BranchConnect: Large-Scale Visual Recognition with Learned Branch Connections
We introduce an architecture for large-scale image categorization that enables the end-to-end learning of separate visual features for the different classes to distinguish.
Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images
In this work, we built an automatic image-understanding method that can accurately classify different types of colorectal polyps in whole-slide histology images to help pathologists with histopathological characterization and diagnosis of colorectal polyps.
VideoMCC: a New Benchmark for Video Comprehension
Language has been exploited to sidestep the problem of defining video categories, by formulating video understanding as the task of captioning or description.
Multiple Hypothesis Colorization
In the experiments we show that our proposed method outperforms traditional JPEG color coding by a large margin, producing colors that are nearly indistinguishable from the ground truth at the storage cost of just a few hundred bytes for high-resolution pictures!
Convolutional Random Walk Networks for Semantic Image Segmentation
It combines these two objectives via a novel random walk layer that enforces consistent spatial grouping in the deep layers of the network.
Local Perturb-and-MAP for Structured Prediction
In this work, we present a new Local Perturb-and-MAP (locPMAP) framework that replaces the global optimization with a local optimization by exploiting our observed connection between locPMAP and the pseudolikelihood of the original CRF model.
Network of Experts for Large-Scale Image Categorization
The training of our "network of experts" is completely end-to-end: the partition of categories into disjoint subsets is learned simultaneously with the parameters of the network trunk and the experts are trained jointly by minimizing a single learning objective over all classes.
Recurrent Mixture Density Network for Spatiotemporal Visual Attention
In this work, we propose a spatiotemporal attentional model that learns where to look in a video directly from human fixation data.
Deep End2End Voxel2Voxel Prediction
Over the last few years deep learning methods have emerged as one of the most prominent approaches for video analysis.
Semantic Segmentation with Boundary Neural Fields
To overcome these problems, we introduce a Boundary Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues.
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision
We can view this process as a "Low-for-High" scheme, where low-level boundaries aid high-level vision tasks.
Coupled Depth Learning
Crucially, the depth basis and the regression function are {\bf coupled} and jointly optimized by our learning scheme.
DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection
This section of the network is applied to four different scales of the image input.
Learning Spatiotemporal Features with 3D Convolutional Networks
We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset.
Ranked #9 on
Action Recognition
on Sports-1M
Self-taught Object Localization with Deep Networks
This paper introduces self-taught object localization, a novel approach that leverages deep convolutional networks trained for whole-image recognition to localize objects in images without additional human supervision, i. e., without using any ground-truth bounding boxes for training.
EXMOVES: Classifier-based Features for Scalable Action Recognition
We show the generality of our approach by building our mid-level descriptors from two different low-level feature representations.
Leveraging Structure from Motion to Learn Discriminative Codebooks for Scalable Landmark Classification
In this paper we propose a new technique for learning a discriminative codebook for local feature descriptors, specifically designed for scalable landmark classification.
PiCoDes: Learning a Compact Code for Novel-Category Recognition
In contrast to previous approaches to learn compact codes, we optimize explicitly for (an upper bound on) classification performance.
Exploiting weakly-labeled Web images to improve object classification: a domain adaptation approach
In this paper we investigate and compare methods that learn image classifiers by combining very few manually annotated examples (e. g., 1-10 images per class) and a large number of weakly-labeled Web photos retrieved using keyword-based image search.
