Search Results for author:
Found 117 papers, 44 papers with code
Characterizing the temporal dynamics of universal speech representations for generalizable deepfake detection
Existing deepfake speech detection systems lack generalizability to unseen attacks (i. e., samples generated by generative algorithms not seen during training).
Motion-Guided Masking for Spatiotemporal Representation Learning
This suggests that the random masking strategy that is inherited from the image MAE is less effective for video MAE.
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
To address this issue, we propose MixReorg, a novel and straightforward pre-training paradigm for semantic segmentation that enhances a model's ability to reorganize patches mixed across images, exploring both local visual relevance and global semantic coherence.
Multilingual Lexical Simplification via Paraphrase Generation
After feeding the input sentence into the encoder of paraphrase modeling, we generate the substitutes based on a novel decoding strategy that concentrates solely on the lexical variations of the complex word.
PreDiff: Precipitation Nowcasting with Latent Diffusion Models
We conduct empirical studies on two datasets: N-body MNIST, a synthetic dataset with chaotic behavior, and SEVIR, a real-world precipitation nowcasting dataset.
Temporal Difference Learning for High-Dimensional PIDEs with Jumps
In this paper, we propose a deep learning framework for solving high-dimensional partial integro-differential equations (PIDEs) based on the temporal difference learning.
Challenges of Zero-Shot Recognition with Vision-Language Models: Granularity and Correctness
This paper investigates the challenges of applying vision-language models (VLMs) to zero-shot visual recognition tasks in an open-world setting, with a focus on contrastive vision-language models such as CLIP.
Clickbait Detection via Large Language Models
Clickbait, which aims to induce users with some surprising and even thrilling headlines for increasing click-through rates, permeates almost all online content publishers, such as news portals and social media.
Multi-Modal Wireless Flexible Gel-Free Sensors with Edge Deep Learning for Detecting and Alerting Freezing of Gait in Parkinson's Patients
The inference of the DL model is performed on a low-power microcontroller in the central node.
Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation
It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledge transfer.
ParaLS: Lexical Substitution via Pretrained Paraphraser
Lexical substitution (LS) aims at finding appropriate substitutes for a target word in a sentence.
Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining
In MOTOR, we combine two kinds of basic medical knowledge, i. e., general and specific knowledge, in a complementary manner to boost the general pretraining process.
Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
This work proposes POMP, a prompt pre-training method for vision-language models.
Ranked #1 on
Open Vocabulary Semantic Segmentation
on PascalVOC-20
(hIoU metric)
On the Impact of Voice Anonymization on Speech-Based COVID-19 Detection
With advances seen in deep learning, voice-based applications are burgeoning, ranging from personal assistants, affective computing, to remote disease diagnostics.
Sentence Simplification via Large Language Models
Sentence Simplification aims to rephrase complex sentences into simpler sentences while retaining original meaning.
Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation
Vision-Language Navigation (VLN) is a challenging task which requires an agent to align complex visual observations to language instructions to reach the goal position.
Towards Geospatial Foundation Models via Continual Pretraining
Geospatial technologies are becoming increasingly essential in our world for a wide range of applications, including agriculture, urban planning, and disaster response.
SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation
Instead of purely relying on the alignment from the noisy data, this paper proposes a novel loss function termed SimCon, which accounts for intra-modal similarities to determine the appropriate set of positive samples to align.
AIM: Adapting Image Models for Efficient Video Action Recognition
Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks.
Ranked #1 on
Action Recognition
on Diving-48
(using extra training data)
ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency
Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image.
SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction
SuperScaler is a system that facilitates the design and generation of highly flexible parallelization plans.
What Makes for Good Tokenizers in Vision Transformer?
The architecture of transformers, which recently witness booming applications in vision tasks, has pivoted against the widespread convolutional paradigm.
SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning
Pre-trained large language models can efficiently interpolate human-written prompts in a natural way.
Are Multimodal Models Robust to Image and Text Perturbations?
Multimodal image-text models have shown remarkable performance in the past few years.
CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target.
Edge Deep Learning Enabled Freezing of Gait Detection in Parkinson's Patients
In a validation using a public dataset, the prototype developed achieved a FoG detection sensitivity of 88. 8% and an F1 score of 85. 34%, using less than 20 k trainable parameters per sensor node.
Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pre-trained VLMs and generating pseudo labels for unseen classes in a self-training manner.
Visual Prompt Tuning for Test-time Domain Adaptation
First, DePT plugs visual prompts into the vision Transformer and only tunes these source-initialized prompts during adaptation.
Ranked #2 on
Domain Adaptation
on VisDA2017
Weak Collocation Regression method: fast reveal hidden stochastic dynamics from high-dimensional aggregate data
Taking the collocations of Gaussian functions as the test functions in the weak form of the FP equation, we transfer the derivatives to the Gaussian functions and thus approximate the weak form by the expectational sum of the data.
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting
With the explosive growth of the spatiotemporal Earth observation data in the past decade, data-driven models that apply Deep Learning (DL) are demonstrating impressive potential for various Earth system forecasting tasks.
Ranked #1 on
Earth Surface Forecasting
on EarthNet2021 OOD Track
Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations
In this paper, we show that recent advances in self-supervised feature learning enable unsupervised object discovery and semantic segmentation with a performance that matches the state of the field on supervised semantic segmentation 10 years ago.
Pixel-level Correspondence for Self-Supervised Learning from Video
While self-supervised learning has enabled effective representation learning in the absence of labels, for vision, video remains a relatively untapped source of supervision.
Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition
However, in real-world applications, it is common for the training sets to have long-tailed distributions.
MixGen: A New Multi-Modal Data Augmentation
Data augmentation is a necessity to enhance data efficiency in deep learning.
ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts
Vision-Language Navigation (VLN) is a challenging task that requires an embodied agent to perform action-level modality alignment, i. e., make instruction-asked actions sequentially in complex visual environments.
NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results
The aim was to design a network for single image super-resolution that achieved improvement of efficiency measured according to several metrics including runtime, parameters, FLOPs, activations, and memory consumption while at least maintaining the PSNR of 29. 00dB on DIV2K validation set.
Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue
Turn-taking, aiming to decide when the next speaker can start talking, is an essential component in building human-robot spoken dialogue systems.
Chinese Idiom Paraphrasing
Idioms, are a kind of idiomatic expression in Chinese, most of which consist of four Chinese characters.
Harnessing Interpretable Machine Learning for Holistic Inverse Design of Origami
This work harnesses interpretable machine learning methods to address the challenging inverse design problem of origami-inspired systems.
ImpDet: Exploring Implicit Fields for 3D Object Detection
Conventional 3D object detection approaches concentrate on bounding boxes representation learning with several parameters, i. e., localization, dimension, and orientation.
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training
Multiple datasets and open challenges for object detection have been introduced in recent years.
Ranked #1 on
Object Detection
on BigDetection val
Building Robust Spoken Language Understanding by Cross Attention between Phoneme Sequence and ASR Hypothesis
Building Spoken Language Understanding (SLU) robust to Automatic Speech Recognition (ASR) errors is an essential issue for various voice-enabled virtual assistants.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Prompt-Learning for Short Text Classification
In the short text, the extremely short length, feature sparsity, and high ambiguity pose huge challenges to classification tasks.
Learning Canonical F-Correlation Projection for Compact Multiview Representation
With this framework as a tool, we propose a correlative covariation projection (CCP) method by using an explicit nonlinear mapping.
Contrastive Instruction-Trajectory Learning for Vision-Language Navigation
The vision-language navigation (VLN) task requires an agent to reach a target with the guidance of natural language instruction.
Blending Anti-Aliasing into Vision Transformer
In this work, we analyze the uncharted problem of aliasing in vision transformer and explore to incorporate anti-aliasing properties.
CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
Contrastive learning allows us to flexibly define powerful losses by contrasting positive pairs from sets of negative samples.
An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages
The availability of parallel sentence simplification (SS) is scarce for neural SS modelings.
Progressive Coordinate Transforms for Monocular 3D Object Detection
Recognizing and localizing objects in the 3D space is a crucial ability for an AI agent to perceive its surrounding environment.
Video Contrastive Learning with Global Context
Our formulation is able to capture global context in a video, thus robust to temporal content change.
A Unified Efficient Pyramid Transformer for Semantic Segmentation
Semantic segmentation is a challenging problem due to difficulties in modeling context in complex scenes and class confusions along boundaries.
Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
Specifically, we propose a Dynamic Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the navigator to move to the wrong target by destroying the most instructive information in instructions at different timesteps.
Optimal Resource Allocation for Serverless Queries
We introduce a system for optimal resource allocation that can predict performance with aggressive trade-offs, for both new and past observed queries.
Deep Learning for Embodied Vision Navigation: A Survey
A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc.
AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation
Neural network-based semantic segmentation has achieved remarkable results when large amounts of annotated data are available, that is, in the supervised case.
Analyzing Adversarial Robustness of Deep Neural Networks in Pixel Space: a Semantic Perspective
Previous works study the adversarial robustness of image classifiers on image level and use all the pixel information in an image indiscriminately, lacking of exploration of regions with different semantic meanings in the pixel space of an image.
Domain Consensus Clustering for Universal Domain Adaptation
To better exploit the intrinsic structure of the target domain, we propose Domain Consensus Clustering (DCC), which exploits the domain consensus knowledge to discover discriminative clusters on both common samples and private ones.
Ranked #3 on
Partial Domain Adaptation
on Office-31
VidTr: Video Transformer Without Convolutions
We first introduce the vanilla video transformer and show that transformer module is able to perform spatio-temporal modeling from raw pixels, but with heavy memory usage.
Ranked #14 on
Action Classification
on Charades
SOON: Scenario Oriented Object Navigation with Graph-based Exploration
In this task, an agent is required to navigate from an arbitrary position in a 3D embodied environment to localize a target following a scene description.
A phase field model for mass transport with semi-permeable interfaces
Numerical simulations first illustrate the consistency of theoretical results on the sharp interface limit.
Numerical Analysis Numerical Analysis 76Z99, 92B05, 76R50
Rapid Multi-Physics Simulation for Electro-Thermal Origami Systems
Electro-thermally actuated origami provides a novel method for creating 3-D systems with advanced morphing and functional capabilities.
Robotics
Three-fold Weyl points in the Schrödinger operator with periodic potentials
Weyl points are degenerate points on the spectral bands at which energy bands intersect conically.
Mathematical Physics Mathematical Physics Spectral Theory
CrossNorm and SelfNorm for Generalization under Distribution Shifts
Can we develop new normalization methods to improve generalization robustness under distribution shifts?
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification
Semi-supervised learning through deep generative models and multi-lingual pretraining techniques have orchestrated tremendous success across different areas of NLP.
Unity of Opposites: SelfNorm and CrossNorm for Model Robustness
CrossNorm exchanges styles between feature channels to perform style augmentation, diversifying the content and style mixtures.
Self-Motivated Communication Agent for Real-World Vision-Dialog Navigation
Vision-Dialog Navigation (VDN) requires an agent to ask questions and navigate following the human responses to find target objects.
A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT.
Global Regularity to Incompressible Viscoelastic System With a Class of Large Initial Data
The global existence of solutions to incompressible viscoelastic flows has been a longstanding open problem, even for the global weak solution.
Analysis of PDEs 76A10, 76D03, 35B65
NUTA: Non-uniform Temporal Aggregation for Action Recognition
In the world of action recognition research, one primary focus has been on how to construct and train networks to model the spatial-temporal volume of an input video.
A Comprehensive Study of Deep Video Action Recognition
Video action recognition is one of the representative tasks for video understanding.
Scale Aware Adaptation for Land-Cover Classification in Remote Sensing Imagery
Land-cover classification using remote sensing imagery is an important Earth observation task.
Towards Good Practices in Self-supervised Representation Learning
Self-supervised representation learning has seen remarkable progress in the last few years.
Improving adversarial robustness of deep neural networks by using semantic information
The vulnerability of deep neural networks (DNNs) to adversarial attack, which is an attack that can mislead state-of-the-art classifiers into making an incorrect classification with high confidence by deliberately perturbing the original inputs, raises concerns about the robustness of DNNs to such attacks.
LSBert: A Simple Framework for Lexical Simplification
Lexical simplification (LS) aims to replace complex words in a given sentence with their simpler alternatives of equivalent meaning, to simplify the sentence.
Identification of hydrodynamic instability by convolutional neural networks
The onset of hydrodynamic instabilities is of great importance in both industry and daily life, due to the dramatic mechanical and thermodynamic changes for different types of flow motions.
When Machine Learning Meets Multiscale Modeling in Chemical Reactions
Due to the intrinsic complexity and nonlinearity of chemical reactions, direct applications of traditional machine learning algorithms may face with many difficulties.
Generating Semantically Valid Adversarial Questions for TableQA
Finally, we demonstrate that adversarial training with SAGE augmented data can improve performance and robustness of TableQA systems.
Revealing hidden dynamics from time-series data by ODENet
To derive the hidden dynamics from observed data is one of the fundamental but also challenging problems in many different fields.
Improving Semantic Segmentation via Self-Training
In the case of semantic segmentation, this means that large amounts of pixelwise annotations are required to learn accurate models.
ResNeSt: Split-Attention Networks
It is well known that featuremap attention and multi-path representation are important for visual recognition.
Ranked #8 on
Instance Segmentation
on COCO test-dev
(APM metric)
Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior
Most of previous work on adversarial attack mainly focus on image models, while the vulnerability of video models is less explored.
Vision-Dialog Navigation by Exploring Cross-modal Memory
Benefiting from the collaborative learning of the L-mem and the V-mem, our CMN is able to explore the memory about the decision making of historical navigation actions which is for the current step.
Generalizing Deep Models for Overhead Image Segmentation Through Getis-Ord Gi* Pooling
Inspired by this, we investigate methods to inform or guide deep learning models for geospatial image analysis to increase their performance when a limited amount of training data is available or when they are applied to scenarios other than which they were trained on.
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information.
Ranked #13 on
Vision and Language Navigation
on VLN Challenge
On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Means
In this paper, we study a simple algorithm to construct asymptotically valid confidence regions for model parameters using the batch means method.
Uncertainty Quantification and Exploration for Reinforcement Learning
We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy.
On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages
Recent work has validated the importance of subword information for word representation learning.
Semi-supervised representation learning via dual autoencoders for domain adaptation
Domain adaptation aims to exploit the knowledge in source domain to promote the learning tasks in target domain, which plays a critical role in real-world applications.
Motion-Aware Feature for Improved Video Anomaly Detection
Motivated by our observation that motion information is the key to good anomaly detection performance in video, we propose a temporal augmented network to learn a motion-aware feature.
Lexical Simplification with Pretrained Encoders
Lexical simplification (LS) aims to replace complex words in a given sentence with their simpler alternatives of equivalent meaning.
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating).
Learning Instance Activation Maps for Weakly Supervised Instance Segmentation
However, learning the full extent of pixel-level instance response in a weakly supervised manner remains unexplored.
Ranked #10 on
Image-level Supervised Instance Segmentation
on PASCAL VOC 2012 val
(using extra training data)
Image-level Supervised Instance Segmentation
object-detection
+5
Bayesian Learning for Neural Dependency Parsing
While neural dependency parsers provide state-of-the-art accuracy for several languages, they still rely on large amounts of costly labeled training data.
Exploring Temporal Information for Improved Video Understanding
In this dissertation, I present my work towards exploring temporal information for better video understanding.
A Systematic Study of Leveraging Subword Information for Learning Word Representations
The use of subword-level information (e. g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning.
Using Conditional Generative Adversarial Networks to Generate Ground-Level Views From Overhead Imagery
This paper develops a deep-learning framework to synthesize a ground-level view of a location given an overhead image.
Improving Semantic Segmentation via Video Propagation and Label Relaxation
In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks.
Ranked #1 on
Semantic Segmentation
on KITTI Semantic Segmentation
(using extra training data)
Gated Transfer Network for Transfer Learning
Deep neural networks have led to a series of breakthroughs in computer vision given sufficient annotated training datasets.
Random Temporal Skipping for Multirate Video Analysis
However, this does not work well for multirate videos, in which actions or subactions occur at different speeds.
What Is It Like Down There? Generating Dense Ground-Level Views and Image Features From Overhead Imagery Using Conditional Generative Adversarial Networks
More significantly, we show the generated images are representative of the locations and that the representations learned by the cGANs are informative.
Learning Optical Flow via Dilated Networks and Occlusion Reasoning
Despite the significant progress that has been made on estimating optical flow recently, most estimation methods, including classical and deep learning approaches, still have difficulty with multi-scale estimation, real-time computation, and/or occlusion reasoning.
Parsing Tweets into Universal Dependencies
Nonetheless, using the new treebank, we build a pipeline system to parse raw tweets into UD.
Ranked #2 on
Dependency Parsing
on Tweebank
Weakly Supervised Instance Segmentation using Class Peak Response
Motivated by this, we first design a process to stimulate peaks to emerge from a class response map.
Ranked #11 on
Image-level Supervised Instance Segmentation
on PASCAL VOC 2012 val
(using extra training data)
General Classification
Image-level Supervised Instance Segmentation
+2
Towards Universal Representation for Unseen Action Recognition
Unseen Action Recognition (UAR) aims to recognise novel action categories without training examples.
Ranked #13 on
Action Recognition
on ActivityNet
Spatial Morphing Kernel Regression For Feature Interpolation
We also show that the spatial morphing kernel improves the results.
Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images
We perform fine-grained land use mapping at the city scale using ground-level images.
Soft Proposal Networks for Weakly Supervised Object Localization
Weakly supervised object localization remains challenging, where only image labels instead of bounding boxes are available during training.
Ranked #2 on
Weakly Supervised Object Detection
on COCO
Weakly Supervised Object Detection
Weakly-Supervised Object Localization
DenseNet for Dense Flow
Classical approaches for estimating optical flow have achieved rapid progress in the last decade.
Large-Scale Mapping of Human Activity using Geo-Tagged Videos
This paper is the first work to perform spatio-temporal mapping of human activity using the visual content of geo-tagged videos.
UC Merced Submission to the ActivityNet Challenge 2016
This notebook paper describes our system for the untrimmed classification task in the ActivityNet challenge 2016.
Hidden Two-Stream Convolutional Networks for Action Recognition
State-of-the-art action recognition approaches rely on traditional optical flow estimation methods to pre-compute motion information for CNNs.
Ranked #18 on
Action Recognition
on UCF101
Guided Optical Flow Learning
We study the unsupervised learning of CNNs for optical flow estimation using proxy ground truth data.
Deep Local Video Feature for Action Recognition
We investigate the problem of representing an entire video using CNN features for human action recognition.
Efficient Action Detection in Untrimmed Videos via Multi-Task Learning
We employ a multi-task learning framework that performs the three highly related steps of action proposal, action recognition, and action localization refinement in parallel instead of the standard sequential pipeline that performs the steps in order.
Land Use Classification using Convolutional Neural Networks Applied to Ground-Level Images
Land use mapping is a fundamental yet challenging task in geographic science.
Spatio-Temporal Sentiment Hotspot Detection Using Geotagged Photos
We perform spatio-temporal analysis of public sentiment using geotagged photo collections.
Depth2Action: Exploring Embedded Depth for Large-Scale Action Recognition
This paper performs the first investigation into depth for large-scale human action recognition in video where the depth cues are estimated from the videos themselves.
