Search Results for author:
Found 79 papers, 32 papers with code
Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
The scalability of Nystr\"{o}mformer enables application to longer sequences with thousands of tokens.
Ranked #13 on
Semantic Textual Similarity
on MRPC
(F1 metric)
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
Ranked #11 on
Open Vocabulary Object Detection
on MSCOCO
(using extra training data)
ActionFormer: Localizing Moments of Actions with Transformers
Self-attention based Transformer models have demonstrated impressive results for image classification and object detection, and more recently for video understanding.
Ranked #2 on
audio-visual event localization
on UnAV-100
Where a Strong Backbone Meets Strong Features -- ActionFormer for Ego4D Moment Queries Challenge
This report describes our submission to the Ego4D Moment Queries Challenge 2022.
Ranked #1 on
Temporal Action Localization
on Ego4D MQ test
NMS Threshold matters for Ego4D Moment Queries -- 2nd place solution to the Ego4D Moment Queries Challenge 2023
This report describes our submission to the Ego4D Moment Queries Challenge 2023.
SnAG: Scalable and Accurate Video Grounding
In this paper, we study the effect of cross-modal fusion on the scalability of video grounding models.
The CAMELS Multifield Dataset: Learning the Universe's Fundamental Parameters with Artificial Intelligence
We present the Cosmology and Astrophysics with MachinE Learning Simulations (CAMELS) Multifield Dataset, CMD, a collection of hundreds of thousands of 2D maps and 3D grids containing many different properties of cosmic gas, dark matter, and stars from 2, 000 distinct simulated universes at several cosmic times.
Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning
We propose a MIL-based method for WSI classification and tumor detection that does not require localized annotations.
Learning to Predict the Cosmological Structure Formation
We build a deep neural network, the Deep Density Displacement Model (hereafter D$^3$M), to predict the non-linear structure formation of the Universe from simple linear perturbation theory.
Interpretable and Accurate Fine-grained Recognition via Region Grouping
Our results compare favorably to state-of-the-art methods on classification tasks, and our method outperforms previous approaches on the localization of object parts.
nbodykit: an open-source, massively parallel toolkit for large-scale structure
The package is extensively documented at http://nbodykit. readthedocs. io, which also includes an interactive set of example recipes for new users to explore.
Instrumentation and Methods for Astrophysics Cosmology and Nongalactic Astrophysics
Learning to Generate Scene Graph from Natural Language Supervision
To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
Comprehensive Image Captioning via Scene Graph Decomposition
We address the challenging problem of image captioning by revisiting the representation of image scene graph.
The Quijote simulations
The Quijote simulations are a set of 44, 100 full N-body simulations spanning more than 7, 000 cosmological models in the $\{\Omega_{\rm m}, \Omega_{\rm b}, h, n_s, \sigma_8, M_\nu, w \}$ hyperplane.
Cosmology and Nongalactic Astrophysics Instrumentation and Methods for Astrophysics
Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations
In this work, we propose to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations.
Learning Two-Branch Neural Networks for Image-Text Matching Tasks
Image-language matching tasks have recently attracted a lot of attention in the computer vision field.
The CAMELS project: Cosmology and Astrophysics with MachinE Learning Simulations
We present the Cosmology and Astrophysics with MachinE Learning Simulations --CAMELS-- project.
Cosmology and Nongalactic Astrophysics Astrophysics of Galaxies Instrumentation and Methods for Astrophysics
Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers
In this work, we exploit temporal redundancy between subsequent inputs to reduce the cost of Transformers for video processing.
Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video
Motivated by this, we adopt intentional hand movement as a future representation and propose a novel deep network that jointly models and predicts the egocentric hand motion, interaction hotspots and future action.
The Secrets of Salient Object Segmentation
The dataset design bias does not only create the discomforting disconnection between fixations and salient object segmentation, but also misleads the algorithm designing.
Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
Our core innovation is the learning of a region-phrase score function, based on which an image-sentence score function is further constructed.
ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles
In this paper we introduce ApproxDet, an adaptive video object detection framework for mobile devices to meet accuracy-latency requirements in the face of changing content and resource contention scenarios.
A Simple Transformer-Based Model for Ego4D Natural Language Queries Challenge
This report describes Badgers@UW-Madison, our submission to the Ego4D Natural Language Queries (NLQ) Challenge.
Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning
An emerging solution with recent success in vision and NLP involves finetuning a foundation model on a selection of relevant tasks, before its adaptation to a target task with limited labeled samples.
Disconnected Covariance of 2-point Functions in Large-Scale Structure
We verify the analytic covariance against the sample covariance from the galaxy mock simulations in two test cases: (1) the power spectrum multipole covariance, and (2) the joint covariance of the projected correlation function and the correlation function multipoles.
Cosmology and Nongalactic Astrophysics
Rotation method for accelerating multiple-spherical Bessel function integrals against a numerical source function
In analysing these datasets recomputation of these integrals a substantial number of times, for instance to update perturbation theory predictions or covariance matrices as the input linear power spectrum is changed, will be one piece in a Monte Carlo Markov Chain cosmological parameter search: thus the overall savings from our method should be significant.
Cosmology and Nongalactic Astrophysics Instrumentation and Methods for Astrophysics
Field Level Neural Network Emulator for Cosmological N-body Simulations
We build a field level emulator for cosmic structure formation that is accurate in the nonlinear regime.
Super-resolving Dark Matter Halos using Generative Deep Learning
Generative deep learning methods built upon Convolutional Neural Networks (CNNs) provide a great tool for predicting non-linear structure in cosmology.
Simple lessons from complex learning: what a neural network model learns about cosmic structure formation
We find our model generalizes well to these well understood scenarios, demonstrating that the networks have inferred general physical principles and learned the nonlinear mode couplings from the complex, random Gaussian training data.
Sequential Model for Predicting Patient Adherence in Subcutaneous Immunotherapy for Allergic Rhinitis
Conclusion: We creatively apply sequential models in the long-term management of SCIT with promising accuracy in the prediction of SCIT nonadherence in Allergic Rhinitis (AR) patients.
Learning to Grasp Without Seeing
We believe this is the first attempt at learning to grasp with only tactile sensing and without any prior object knowledge.
Deep Crisp Boundaries: From Boundaries to Higher-level Tasks
These ConvNet based edge detectors have approached human level performance on standard benchmarks.
Learning Deep Structure-Preserving Image-Text Embeddings
This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities.
Ranked #15 on
Image Retrieval
on Flickr30K 1K test
Unsupervised Learning of Edges
In this work we present a simple yet effective approach for training edge detectors without human supervision.
Sense-Aware Neural Models for Pun Location in Texts
In this paper, we focus on the task of pun location, which aims to identify the pun word in a given short text.
Beyond Grids: Learning Graph Representations for Visual Recognition
Our method further learns to propagate information across all vertices on the graph, and is able to project the learned graph representation back into 2D grids.
3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare
Our method produces a compact 3D representation of the scene, which can be readily used for applications like autonomous driving.
Ranked #3 on
Vehicle Pose Estimation
on KITTI Cars Hard
(using extra training data)
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera.
Delving Into Egocentric Actions
We propose to utilize these mid-level egocentric cues for egocentric action recognition.
Gaze-Enabled Egocentric Video Summarization via Constrained Submodular Maximization
Motivated by these applications, this paper focuses on the problem of egocentric video summarization.
Attention Distillation for Learning Video Representations
To this end, we make use of attention modules that learn to highlight regions in the video and aggregate features for recognition.
Ranked #39 on
Action Recognition
on UCF101
Semi Supervised Phrase Localization in a Bidirectional Caption-Image Retrieval Framework
To accomplish this task, our architecture makes use of the rich semantic information available in a joint embedding space of multi-modal data.
Gradients as Features for Deep Representation Learning
We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks.
Obscure: Information-Theoretically Secure, Oblivious, and Verifiable Aggregation Queries on Secret-Shared Outsourced Data -- Full Version
Despite exciting progress on cryptography, secure and efficient query processing over outsourced data remains an open challenge.
In the Eye of the Beholder: Gaze and Actions in First Person Video
Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV.
An optimal FFT-based anisotropic power spectrum estimator
Here, we present a faster, optimal means of using FFTs for this measurement.
Cosmology and Nongalactic Astrophysics
AI-assisted super-resolution cosmological simulations
Cosmological simulations of galaxy formation are limited by finite computational resources.
Prism: Private Verifiable Set Computation over Multi-Owner Outsourced Databases
This paper proposes Prism, a secret sharing based approach to compute private set operations (i. e., intersection and union), as well as aggregates over outsourced databases belonging to multiple owners.
AI-assisted super-resolution cosmological simulations II: Halo substructures, velocities and higher order statistics
In this work, we expand and test the capabilities of our recently developed super-resolution (SR) model to generate high-resolution (HR) realizations of the full phase-space matter distribution, including both displacement and velocity, from computationally cheap low-resolution (LR) cosmological N-body simulations.
Learning the Evolution of the Universe in N-body Simulations
Understanding the physics of large cosmological surveys down to small (nonlinear) scales will significantly improve our knowledge of the Universe.
Egocentric Activity Recognition and Localization on a 3D Map
Given a video captured from a first person perspective and the environment context of where the video is recorded, can we recognize what the person is doing and identify where the action occurs in the 3D space?
Inferring Black Hole Properties from Astronomical Multivariate Time Series with Bayesian Attentive Neural Processes
Among the most extreme objects in the Universe, active galactic nuclei (AGN) are luminous centers of galaxies where a black hole feeds on surrounding matter.
Hyperspectral Remote Sensing Image Classification Based on Multi-scale Cross Graphic Convolution
Using a multi-scale convolution algorithm, the input dimensionality reduction features were mined to obtain shallow features, which then served as inputs into a multi-scale graph convolution algorithm to construct the internal relationships between eigenvalues at different scales.
Weakly Supervised Foreground Learning for Weakly Supervised Localization and Detection
Modern deep learning models require large amounts of accurately annotated data, which is often difficult to satisfy.
Towards Non-Line-of-Sight Photography
Non-line-of-sight (NLOS) imaging is based on capturing the multi-bounce indirect reflections from the hidden objects.
Multifield Cosmology with Artificial Intelligence
Although our maps only cover a small area of $(25~h^{-1}{\rm Mpc})^2$, and the different fields are contaminated by astrophysical effects in very different ways, our networks can infer the values of $\Omega_{\rm m}$ and $\sigma_8$ with a few percent level precision for most of the fields.
Robust marginalization of baryonic effects for cosmological inference at the field level
We train neural networks to perform likelihood-free inference from $(25\, h^{-1}{\rm Mpc})^2$ 2D maps containing the total mass surface density from thousands of hydrodynamic simulations of the CAMELS project.
A Simple Baseline for Weakly-Supervised Scene Graph Generation
We investigate the weakly-supervised scene graph generation, which is a challenging task since no correspondence of label and object is provided.
3D Photo Stylization: Learning to Generate Stylized Novel Views from a Single Image
Our key intuition is that style transfer and view synthesis have to be jointly modeled for this task.
Event Neural Networks
Video data is often repetitive; for example, the contents of adjacent frames are usually strongly correlated.
Virtuoso: Video-based Intelligence for real-time tuning on SOCs
First, the system does not consider energy consumption of the models while making a decision on which model to run.
The CAMELS project: public data release
The Cosmology and Astrophysics with MachinE Learning Simulations (CAMELS) project was developed to combine cosmology with astrophysics through thousands of cosmological hydrodynamic simulations and machine learning.
Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging
Computational approach to imaging around the corner, or non-line-of-sight (NLOS) imaging, is becoming a reality thanks to major advances in imaging hardware and reconstruction algorithms.
SmartAdapt: Multi-Branch Object Detection Framework for Videos on Mobiles
In this paper, we ask, and answer, the wide-ranging question across all MBODFs: How to expose the right set of execution branches and then how to schedule the optimal one at inference time?
EnergyMatch: Energy-based Pseudo-Labeling for Semi-Supervised Learning
However, it has been shown that softmax-based confidence scores in deep networks can be arbitrarily high for samples far from the training data, and thus, the pseudo-labels for even high-confidence unlabeled samples may still be unreliable.
Reconstructing the Universe with Variational self-Boosted Sampling
We show that after a short initial warm-up and training phase, VBS generates better quality of samples than simple VI approaches and reduces the correlation length in the sampling phase by a factor of 10-50 over using only HMC to explore the posterior of initial conditions in 64$^3$ and 128$^3$ dimensional problems, with larger gains for high signal-to-noise data observations.
Robust Scene Inference under Noise-Blur Dual Corruptions
This creates a trade-off between these two kinds of image degradations: motion blur (due to long exposure) vs. noise (due to short exposure), also referred as a dual image corruption pair in this paper.
Particle clustering in turbulence: Prediction of spatial and statistical properties with deep learning
The simulation data are used to train a U-Net deep learning model to predict gridded three-dimensional representations of the particle density and velocity fields, given as input the corresponding fluid fields.
mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors
To bridge the gap, we present mRI, a multi-modal 3D human pose estimation dataset with mmWave, RGB-D, and Inertial Sensors.
3D Scene Inference from Transient Histograms
Time-resolved image sensors that capture light at pico-to-nanosecond timescales were once limited to niche applications but are now rapidly becoming mainstream in consumer devices.
InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning
Without relying on model confidence, we propose to measure whether an unlabeled sample is likely to be ``in-distribution''; i. e., close to the current training data.
SimHaze: game engine simulated data for real-world dehazing
Using a modern game engine, our approach renders crisp clean images and their precise depth maps, based on which high-quality hazy images can be synthesized for training dehazing models.
A Review of Adversarial Attacks in Computer Vision
Deep neural networks have been widely used in various downstream tasks, especially those safety-critical scenario such as autonomous driving, but deep networks are often threatened by adversarial samples.
Learned Compressive Representations for Single-Photon 3D Imaging
Single-photon 3D cameras can record the time-of-arrival of billions of photons per second with picosecond accuracy.
FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition
Recent approaches such as ControlNet offer users fine-grained spatial control over text-to-image (T2I) diffusion models.
BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision
These challenges are especially manifested in videos captured by unmanned aerial vehicles (UAV), where the target is usually far away from the camera and often with significant motion relative to the camera.
Towards 3D Vision with Low-Cost Single-Photon Cameras
We present a method for reconstructing 3D shape of arbitrary Lambertian objects based on measurements by miniature, energy-efficient, low-cost single-photon cameras.
A dataset of primary nasopharyngeal carcinoma MRI with multi-modalities segmentation
Multi-modality magnetic resonance imaging data with various sequences facilitate the early diagnosis, tumor segmentation, and disease staging in the management of nasopharyngeal carcinoma (NPC).
