Search Results for author:
Found 209 papers, 89 papers with code
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders.
Ranked #1 on
Semantic Segmentation
on COCO-Stuff full
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models.
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Thus, it is still an open question: shall we pretrain large autoregressive LMs with retrieval?
Voyager: An Open-Ended Embodied Agent with Large Language Models
We introduce Voyager, the first LLM-powered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention.
Eureka: Human-Level Reward Design via Coding Large Language Models
The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating.
Fourier Neural Operator for Parametric Partial Differential Equations
The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces.
Speeding up Fourier Neural Operators via Mixed Precision
In this work, we (i) profile memory and runtime for FNO with full and mixed-precision training, (ii) conduct a study on the numerical stability of mixed-precision training of FNO, and (iii) devise a training routine which substantially decreases training time and memory usage (up to 34%), with little or no reduction in accuracy, on the Navier-Stokes and Darcy flow equations.
Neural Operator: Learning Maps Between Function Spaces
The classical development of neural networks has primarily focused on learning mappings between finite dimensional Euclidean spaces or finite sets.
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Autonomous agents have made great strides in specialist domains like Atari games and Go.
TensorLy: Tensor Learning in Python
In addition, using the deep-learning frameworks as backend allows users to easily design and train deep tensorized neural networks.
Prismer: A Vision-Language Model with Multi-Task Experts
Recent vision-language models have shown impressive multi-modal generation capabilities.
Ranked #1 on
Image Captioning
on nocaps val
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Our approach reduces memory usage by up to 65. 5% in optimizer states while maintaining both efficiency and performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset with up to 19. 7B tokens, and on fine-tuning RoBERTa on GLUE tasks.
VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion
To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images.
3D Semantic Scene Completion from a single RGB image
Depth Estimation
VIMA: General Robot Manipulation with Multimodal Prompts
We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens.
FB-BEV: BEV Representation from Forward-Backward View Transformations
Currently, the two most prominent VTM paradigms are forward projection and backward projection.
Understanding The Robustness in Vision Transformers
Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations.
Ranked #4 on
Domain Generalization
on ImageNet-R
(using extra training data)
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Using this data, we develop ReProver (Retrieval-Augmented Prover): an LLM-based prover augmented with retrieval for selecting premises from a vast math library.
Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
Advances in artificial intelligence (AI) are fueling a new paradigm of discoveries in natural sciences.
Pre-Trained Language Models for Interactive Decision-Making
Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
Long-Short Transformer: Efficient Transformers for Language and Vision
For instance, Transformer-LS achieves 0. 97 test BPC on enwik8 using half the number of parameters than previous method, while being faster and is able to handle 3x as long sequences compared to its full-attention version on the same hardware.
Ranked #1 on
Language Modelling
on enwik8 dev
FreeSOLO: Learning to Segment Objects without Annotations
FreeSOLO further demonstrates superiority as a strong pre-training method, outperforming state-of-the-art self-supervised pre-training methods by +9. 8% AP when fine-tuning instance segmentation with only 5% COCO masks.
Physics-Informed Neural Operator for Learning Partial Differential Equations
Specifically, in PINO, we combine coarse-resolution training data with PDE constraints imposed at a higher resolution.
Neural Operator: Graph Kernel Network for Partial Differential Equations
The classical development of neural networks has been primarily for mappings between a finite-dimensional Euclidean space and a set of classes, or between two finite-dimensional Euclidean spaces.
Multipole Graph Neural Operator for Parametric Partial Differential Equations
One of the main challenges in using deep learning-based methods for simulating physical systems and solving partial differential equations (PDEs) is formulating physics-based data in the desired structure for neural networks.
Fast Training of Diffusion Models with Masked Transformers
For masked training, we introduce an asymmetric encoder-decoder architecture consisting of a transformer encoder that operates only on unmasked patches and a lightweight transformer decoder on full patches.
MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging Occluded VIS dataset by over 10% AP.
Ranked #13 on
Video Instance Segmentation
on YouTube-VIS validation
Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere
Fourier Neural Operators (FNOs) have proven to be an efficient and effective method for resolution-independent operator learning in a broad variety of application areas across scientific machine learning.
Learning Dissipative Dynamics in Chaotic Systems
Chaotic systems are notoriously challenging to predict because of their sensitivity to perturbations and errors due to time stepping.
Diffusion Models for Adversarial Purification
Adversarial purification refers to a class of defense methods that remove adversarial perturbations using a generative model.
InRank: Incremental Low-Rank Learning
To remedy this, we design a new training algorithm Incremental Low-Rank Learning (InRank), which explicitly expresses cumulative weight updates as low-rank matrices while incrementally augmenting their ranks during training.
I$^2$SB: Image-to-Image Schrödinger Bridge
We propose Image-to-Image Schr\"odinger Bridge (I$^2$SB), a new class of conditional diffusion models that directly learn the nonlinear diffusion processes between two given distributions.
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner.
Ranked #4 on
Panoptic Segmentation
on COCO test-dev
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
AFNO is based on a principled foundation of operator learning which allows us to frame token mixing as a continuous global convolution without any dependence on the input resolution.
Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing
Here we present a multi-modal molecule structure-text model, MoleculeSTM, by jointly learning molecules' chemical structures and textual descriptions via a contrastive learning strategy.
Open Vocabulary Learning on Source Code with a Graph-Structured Cache
Machine learning models that take computer program source code as input typically use Natural Language Processing (NLP) techniques.
Probabilistic FastText for Multi-Sense Word Embeddings
We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information.
State-specific protein-ligand complex structure prediction with a multi-scale deep generative model
The binding complexes formed by proteins and small molecule ligands are ubiquitous and critical to life.
Fourier Neural Operator with Learned Deformations for PDEs on General Geometries
The resulting geo-FNO model has both the computation efficiency of FFT and the flexibility of handling arbitrary geometries.
Neural-Fly Enables Rapid Learning for Agile Flight in Strong Winds
Last, our control design extrapolates to unseen wind conditions, is shown to be effective for outdoor flights with only onboard sensors, and can transfer across drones with minimal performance degradation.
FocalFormer3D: Focusing on Hard Instance for 3D Object Detection
For 3D object detection, we instantiate this method as FocalFormer3D, a simple yet effective detector that excels at excavating difficult objects and improving prediction recall.
AugMax: Adversarial Composition of Random Augmentations for Robust Training
Diversity and hardness are two complementary dimensions of data augmentation to achieve robustness.
A Text-guided Protein Design Framework
Current AI-assisted protein design mainly utilizes protein sequential and structural information.
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
Competitive Gradient Descent
We introduce a new algorithm for the numerical computation of Nash equilibria of competitive two-player games.
Implicit competitive regularization in GANs
We show that opponent-aware modelling of generator and discriminator, as present in competitive gradient descent (CGD), can significantly strengthen ICR and thus stabilize GAN training without explicit regularization.
OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation
Operational Space Control (OSC) has been used as an effective task-space controller for manipulation.
MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework
We propose MeshfreeFlowNet, a novel deep learning-based super-resolution framework to generate continuous (grid-free) spatio-temporal solutions from the low-resolution inputs.
Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials
Artificial intelligence for scientific discovery has recently generated significant interest within the machine learning and scientific communities, particularly in the domains of chemistry, biology, and material discovery.
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision.
ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation
The dataset is global in coverage, spans multiple years at high sampling frequency, and is designed such that resulting emulators are compatible with downstream coupling into operational climate simulators.
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching
Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model.
signSGD: Compressed Optimisation for Non-Convex Problems
Using a theorem by Gauss we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD.
signSGD with Majority Vote is Communication Efficient And Fault Tolerant
Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote.
U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow
Here we present U-FNO, a novel neural network architecture for solving multiphase flow problems with superior accuracy, speed, and data efficiency.
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones
Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training.
Controllable and Compositional Generation with Latent-Space Energy-Based Models
In compositional generation, our method excels at zero-shot generation of unseen attribute combinations.
Long-term Forecasting using Higher Order Tensor RNNs
We present Higher-Order Tensor RNN (HOT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics.
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i. e., systematic generalization.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoning of novel concepts.
Ranked #1 on
Few-Shot Image Classification
on Bongard-HOI
Born Again Neural Networks
Knowledge distillation (KD) consists of transferring knowledge from one machine learning model (the teacher}) to another (the student).
Learning compositional functions via multiplicative weight updates
This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions.
StrassenNets: Deep Learning with a Multiplication Budget
A large fraction of the arithmetic operations required to evaluate deep neural networks (DNNs) consists of matrix multiplications, in both convolution and fully connected layers.
SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
A student network then learns to mimic the expert policy by supervised learning with strong augmentations, making its representation more robust against visual variations compared to the expert.
Retrieval-based Controllable Molecule Generation
On various tasks ranging from simple design criteria to a challenging real-world scenario for designing lead compounds that bind to the SARS-CoV-2 main protease, we demonstrate our approach extrapolates well beyond the retrieval database, and achieves better performance and wider applicability than previous methods.
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Noisy Matrix Decomposition
For sparse optimization, we establish that the modified ADMM method has an optimal convergence rate of $\mathcal{O}(s\log d/T)$, where $s$ is the sparsity level, $d$ is the data dimension and $T$ is the number of steps.
Automated Synthetic-to-Real Generalization
Models trained on synthetic images often face degraded generalization to real data.
Contrastive Syn-to-Real Generalization
Training on synthetic data can be beneficial for label or data-scarce scenarios.
Fully Attentional Networks with Self-emerging Token Labeling
With the proposed STL framework, our best model based on FAN-L-Hybrid (77. 3M parameters) achieves 84. 8% Top-1 accuracy and 42. 1% mCE on ImageNet-1K and ImageNet-C, and sets a new state-of-the-art for ImageNet-A (46. 1%) and ImageNet-R (56. 6%) without using extra data, outperforming the original FAN counterpart by significant margins.
Ranked #16 on
Domain Generalization
on ImageNet-C
1st Place Solution of The Robust Vision Challenge 2022 Semantic Segmentation Track
The model is trained on a composite dataset consisting of images from 9 datasets (ADE20K, Cityscapes, Mapillary Vistas, ScanNet, VIPER, WildDash 2, IDD, BDD, and COCO) with a simple dataset balancing strategy.
Generative Adversarial Neural Operators
The inputs to the generator are samples of functions from a user-specified probability measure, e. g., Gaussian random field (GRF), and the generator outputs are synthetic data functions.
Competitive Policy Optimization
A core challenge in policy optimization in competitive Markov decision processes is the design of efficient optimization methods with desirable convergence and stability properties.
Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds
Certified robustness is a desirable property for deep neural networks in safety-critical applications, and popular training algorithms can certify robustness of a neural network by computing a global bound on its Lipschitz constant.
Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control
In this paper, we propose a stability-constrained reinforcement learning (RL) method for real-time voltage control, that guarantees system stability both during policy learning and deployment of the learned policy.
Learning From Noisy Singly-labeled Data
We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data.
Neural Networks with Recurrent Generative Feedback
This mechanism can be interpreted as a form of self-consistency between the maximum a posteriori (MAP) estimation of an internal generative model and the external environment.
Stochastic Activation Pruning for Robust Adversarial Defense
Neural networks are known to be vulnerable to adversarial examples.
Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection
As a result, image resampling alone is not enough to yield a sufficiently balanced distribution at the object level.
Fast Sampling of Diffusion Models via Operator Learning
Diffusion models have found widespread adoption in various areas.
Deep Bayesian Quadrature Policy Optimization
On the other hand, more sample efficient alternatives like Bayesian quadrature methods have received little attention due to their high computational complexity.
DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings.
Active Learning with Partial Feedback
While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes/no binary feedback).
Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs
On complex downstream tasks with limited data, such as fluid flow simulations and fluid-structure interactions, we found CoDA-NO to outperform existing methods on the few-shot learning task by over $36\%$.
Langevin Monte Carlo for Contextual Bandits
Existing Thompson sampling-based algorithms need to construct a Laplace approximation (i. e., a Gaussian distribution) of the posterior distribution, which is inefficient to sample in high dimensional applications for general covariance matrices.
CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learning
It then uses the top trajectories as initialization for gradient descent and applies gradient updates to each of these trajectories to find the optimal action sequence.
Finding Social Media Trolls: Dynamic Keyword Selection Methods for Rapidly-Evolving Online Debates
Online harassment is a significant social problem.
OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
We propose a variational inference framework OCEAN to perform online task inference for compositional tasks.
Augmenting Deep Classifiers with Polynomial Neural Networks
The efficacy of the proposed models is evaluated on standard image and audio classification benchmarks.
Ranked #2 on
Audio Classification
on Speech Commands
Competitive Mirror Descent
Finally, we obtain the next iterate by following this direction according to the dual geometry induced by the Bregman potential.
Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo
One of the key shortcomings of existing Thompson sampling algorithms is the need to perform a Gaussian approximation of the posterior distribution, which is not a good surrogate in most practical settings.
Tensor Regression Networks
First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction.
Compact Tensor Pooling for Visual Question Answering
Performing high level cognitive tasks requires the integration of feature maps with drastically different structure.
Homotopy Analysis for Tensor PCA
For the challenging problem of tensor PCA, we prove global convergence of the homotopy method in the "high noise" regime.
Tensor Contraction Layers for Parsimonious Deep Nets
Specifically, we propose the Tensor Contraction Layer (TCL), the first attempt to incorporate tensor contractions as end-to-end trainable neural network layers.
Training Input-Output Recurrent Neural Networks through Spectral Methods
We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks.
Efficient approaches for escaping higher order saddle points in non-convex optimization
Local search heuristics for non-convex optimizations are popular in applied machine learning.
Provable Tensor Methods for Learning Mixtures of Generalized Linear Models
In contrast, we present a tensor decomposition method which is guaranteed to correctly recover the parameters.
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
We propose a novel algorithm based on tensor decomposition for guaranteed training of two-layer neural networks.
Analyzing Tensor Power Method Dynamics in Overcomplete Regime
We present a novel analysis of the dynamics of tensor power iterations in the overcomplete regime where the tensor CP rank is larger than the input dimension.
A Scale Mixture Perspective of Multiplicative Noise in Neural Networks
Corrupting the input and hidden layers of deep neural networks (DNNs) with multiplicative noise, often drawn from the Bernoulli distribution (or 'dropout'), provides regularization that has significantly contributed to deep learning's success.
Multi-Object Classification and Unsupervised Scene Understanding Using Deep Learning Features and Latent Tree Probabilistic Models
We incorporate contextual information in natural images through a conditional latent tree probabilistic model (CLTM), where the object co-occurrences are conditioned on the extracted fc7 features from pre-trained Imagenet CNN as input.
Provable Methods for Training Neural Networks with Sparse Connectivity
We provide novel guaranteed approaches for training feedforward neural networks with sparse connectivity.
Learning Mixed Membership Community Models in Social Tagging Networks through Tensor Methods
Community detection in graphs has been extensively studied both in theory and in applications.
Score Function Features for Discriminative Learning
In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples.
Guaranteed Scalable Learning of Latent Tree Models
We present an integrated approach for structure and parameter estimation in latent tree graphical models.
Score Function Features for Discriminative Learning: Matrix and Tensor Framework
In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples.
Tensor decompositions for learning latent variable models
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order).
A Tensor Approach to Learning Mixed Membership Community Models
We provide guaranteed recovery of community memberships and model parameters and present a careful finite sample analysis of our learning method.
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition
We first analyze the simple setting, where the optimization problem consists of a loss function and a single regularizer (e. g. sparse optimization), and then extend to the multi-block setting with multiple regularizers and multiple variables (e. g. matrix decomposition into sparse and low rank components).
A Spectral Algorithm for Latent Dirichlet Allocation
This work provides a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of topic models, including Latent Dirichlet Allocation (LDA).
Latent Graphical Model Selection: Efficient Methods for Locally Tree-like Graphs
We consider a challenging instance of this problem when some of the nodes are latent or hidden.
Learning Mixtures of Tree Graphical Models
We consider unsupervised estimation of mixtures of discrete graphical models, where the class variable is hidden and each mixture component can have a potentially different Markov graph structure and parameters over the observed variables.
Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning
The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs–the Rendering Path Normalization (RPN).
Tensor Contraction & Regression Networks
Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer.
Convergence rate of sign stochastic gradient descent for non-convex functions
The sign stochastic gradient descent method (signSGD) utilizes only the sign of the stochastic gradient in its updates.
Long-term Forecasting using Tensor-Train RNNs
We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics.
Stochastic Linear Bandits with Hidden Low Rank Structure
We modify the image classification task into the SLB setting and empirically show that, when a pre-trained DNN provides the high dimensional feature representations, deploying PSLB results in significant reduction of regret and faster convergence to an accurate model compared to state-of-art algorithm.
Tensor Dropout for Robust Learning
CNNs achieve remarkable performance by leveraging deep, over-parametrized architectures, trained on large datasets.
Robust Regression for Safe Exploration in Control
To address this challenge, we present a deep robust regression model that is trained to directly predict the uncertainty bounds for safe exploration.
Learning Causal State Representations of Partially Observable Environments
In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP).
Directivity Modes of Earthquake Populations with Unsupervised Learning
A seismic spectral decomposition technique is used to first produce relative measurements of radiated energy for earthquakes in a spatially-compact cluster.
Out-of-Distribution Detection Using Neural Rendering Generative Models
Our results show that when trained on CIFAR-10, lower likelihood (of latent variables) is assigned to SVHN images.
Multi Sense Embeddings from Topic Models
Distributed word embeddings have yielded state-of-the-art performance in many NLP tasks, mainly due to their success in capturing useful semantic information.
Triply Robust Off-Policy Evaluation
Ours is a general approach that can be used to augment any existing OPE method that utilizes the direct method.
Angular Visual Hardness
We also find that AVH has a statistically significant correlation with human visual hardness.
InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers
Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation.
A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN).
Brain-inspired Robust Vision using Convolutional Neural Networks with Feedback
Primates have a remarkable ability to correctly classify images even in the presence of significant noise and degradation.
Regret Minimization in Partially Observable Linear Quadratic Control
We propose a novel way to decompose the regret and provide an end-to-end sublinear regret upper bound for partially observable linear quadratic control.
Semi-Supervised StyleGAN for Disentanglement Learning
Disentanglement learning is crucial for obtaining disentangled representations and controllable generation.
Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting
We study the problem of adaptive control in partially observable linear quadratic Gaussian control systems, where the model dynamics are unknown a priori.
Logarithmic Regret Bound in Partially Observable Linear Dynamical Systems
We study the problem of system identification and adaptive control in partially observable linear dynamical systems.
Spectral Learning on Matrices and Tensors
PCA and other spectral techniques applied to matrices have several limitations.
Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems
The Info-SNOC algorithm is used to compute a sub-optimal pool of safe motion plans that aid in exploration for learning unknown residual dynamics under safety constraints.
Unsupervised Controllable Generation with Self-Training
Instead, we propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training.
Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems
In this work, we study model-based reinforcement learning (RL) in unknown stabilizable linear dynamical systems.
Model-based Reinforcement Learning
reinforcement-learning
+1
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
We showcase the controllability of our model by replacing the keywords used to generate stories and re-running the generation process.
A Coach-Player Framework for Dynamic Team Composition
The performance of our method is comparable or even better than the setting where all players have a full view of the environment, but no coach.
Transferable Unsupervised Robust Representation Learning
We upend this view and show that URRL improves both the natural accuracy of unsupervised representation learning and its robustness to corruptions and adversarial noise.
Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach
We propose a framework for learning calibrated uncertainties under domain shifts, where the source (training) distribution differs from the target (test) distribution.
Stability and Identification of Random Asynchronous Linear Time-Invariant Systems
In this model, each state variable is updated randomly and asynchronously with some probability according to the underlying system dynamics.
Disentangling Observed Causal Effects from Latent Confounders using Method of Moments
Thus, we provide a principled approach to tackling the joint problem of causal discovery and latent variable inference.
Dynamic Social Media Monitoring for Fast-Evolving Online Discussions
However, collecting social media data using a static set of keywords fails to satisfy the growing need to monitor dynamic conversations and to study fast-changing topics.
Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles
Extracting interesting scenarios from real-world data as well as generating failure cases is important for the development and testing of autonomous systems.
Stable Online Control of Linear Time-Varying Systems
In this work, we propose an efficient online control algorithm, COvariance Constrained Online Linear Quadratic (COCO-LQ) control, that guarantees input-to-state stability for a large class of LTV systems while also minimizing the control cost.
Informing Geometric Deep Learning with Electronic Interactions to Accelerate Quantum Chemistry
Predicting electronic energies, densities, and related chemical properties can facilitate the discovery of novel catalysts, medicines, and battery materials.
Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection
Deep neural networks have reached high accuracy on object detection but their success hinges on large amounts of labeled data.
LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update
Representing deep neural networks (DNNs) in low-precision is a promising approach to enable efficient acceleration and memory reduction.
Tensor Methods in Computer Vision and Deep Learning
Tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions.
Finite-time System Identification and Adaptive Control in Autoregressive Exogenous Systems
Using these guarantees, we design adaptive control algorithms for unknown ARX systems with arbitrary strongly convex or convex quadratic regulating costs.
Auditing AI models for Verified Deployment under Semantic Specifications
We enable such unit tests through variations in a semantically-interpretable latent space of a generative model.
Stability Constrained Reinforcement Learning for Real-Time Voltage Control
Deep reinforcement learning (RL) has been recognized as a promising tool to address the challenges in real-time control of power systems.
Efficient Token Mixing for Transformers via Adaptive Fourier Neural Operators
AFNO is based on a principled foundation of operator learning which allows us to frame token mixing as a continuous global convolution without any dependence on the input resolution.
Cost-Sensitive Hierarchical Classification through Layer-wise Abstentions
For each layer, we also achieve higher accuracy when the overall accuracy is kept fixed across different methods.
Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences.
Polymatrix Competitive Gradient Descent
In this work we propose polymatrix competitive gradient descent (PCGD) as a method for solving general sum competitive optimization involving arbitrary numbers of agents.
Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
It is therefore interesting to study how these two tasks can be coupled to benefit each other.
Adversarially Robust 3D Point Cloud Recognition Using Self-Supervisions
In this paper, we systematically study the impact of various self-supervised learning proxy tasks on different architectures and threat models for 3D point clouds with adversarial training.
InfoCNF: Efficient Conditional Continuous Normalizing Flow Using Adaptive Solvers
Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation.
Disentangled GANs for Controllable Generation of High-Resolution Images
Generative adversarial networks (GANs) have achieved great success at generating realistic samples.
Distributionally Robust Learning for Unsupervised Domain Adaptation
This formulation motivates the use of two neural networks that are jointly trained --- a discriminative network between the source and target domains for density-ratio estimation, in addition to the standard classification network.
Simulation Intelligence: Towards a New Generation of Scientific Methods
We present the "Nine Motifs of Simulation Intelligence", a roadmap for the development and integration of the essential algorithms necessary for a merger of scientific computing, scientific simulation, and artificial intelligence.
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases
We then provide the LM with instruction that consists of this subset of labeled exemplars, the query text to be classified, a definition of bias, and prompt it to make a decision.
ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation
Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability.
Generic Lithography Modeling with Dual-band Optics-Inspired Neural Networks
Lithography simulation is a critical step in VLSI design and optimization for manufacturability.
M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation
In this paper, we propose M$^2$BEV, a unified framework that jointly performs 3D object detection and map segmentation in the Birds Eye View~(BEV) space with multi-camera image inputs.
KCRL: Krasovskii-Constrained Reinforcement Learning with Guaranteed Stability in Nonlinear Dynamical Systems
However, current reinforcement learning (RL) methods lack stabilization guarantees, which limits their applicability for the control of safety-critical systems.
Finite-Time Regret of Thompson Sampling Algorithms for Exponential Family Multi-Armed Bandits
We study the regret of Thompson sampling (TS) algorithms for exponential family bandits, where the reward distribution is from a one-dimensional exponential family, which covers many common reward distributions including Bernoulli, Gaussian, Gamma, Exponential, etc.
Thompson Sampling Achieves $\tilde O(\sqrt{T})$ Regret in Linear Quadratic Control
By carefully prescribing an early exploration strategy and a policy update rule, we show that TS achieves order-optimal regret in adaptive control of multidimensional stabilizable LQRs.
Large Scale Mask Optimization Via Convolutional Fourier Neural Operator and Litho-Guided Self Training
Machine learning techniques have been extensively studied for mask optimization problems, aiming at better mask printability, shorter turnaround time, better mask manufacturability, and so on.
Robust Trajectory Prediction against Adversarial Attacks
We demonstrate that our method is able to improve the performance by 46% on adversarial data and at the cost of only 3% performance degradation on clean data, compared to the model trained with clean data.
AdvDO: Realistic Adversarial Attacks for Trajectory Prediction
Trajectory prediction is essential for autonomous vehicles (AVs) to plan correct and safe driving behaviors.
Context Generation Improves Open Domain Question Answering
Prior work on closed-book QA either directly finetunes or prompts a pretrained language model (LM) to leverage the stored knowledge.
An Adversarial Active Sampling-based Data Augmentation Framework for Manufacturable Chip Design
These synthetic mask images will augment the original limited training dataset used to finetune the lithography model for improved performance.
Real-time high-resolution CO$_2$ geological storage prediction using nested Fourier neural operators
Carbon capture and storage (CCS) plays an essential role in global decarbonization.
DensePure: Understanding Diffusion Models towards Adversarial Robustness
By using the highest density point in the conditional distribution as the reversed sample, we identify the robust region of a given instance under the diffusion model's reverse process.
Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions
Finally, we find that not all transfer scenarios yield a positive gain, which seems related to the PLMs initial performance on the target-domain task.
Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs
Fourier Neural Operators (FNO) offer a principled approach to solving challenging partial differential equations (PDE) such as turbulent flows.
Machine Learning Accelerated PDE Backstepping Observers
State estimation is important for a variety of tasks, from forecasting to substituting for unmeasured states in feedback controllers.
Fourier Continuation for Exact Derivative Computation in Physics-Informed Neural Operators
The physics-informed neural operator (PINO) is a machine learning architecture that has shown promising empirical results for learning partial differential equations.
HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression
Transformers have attained superior performance in natural language processing and computer vision.
Towards Neural Variational Monte Carlo That Scales Linearly with System Size
Quantum many-body problems are some of the most challenging problems in science and are central to demystifying some exotic quantum phenomena, e. g., high-temperature superconductors.
Vision Transformers Are Good Mask Auto-Labelers
We propose Mask Auto-Labeler (MAL), a high-quality Transformer-based mask auto-labeling framework for instance segmentation using only box annotations.
Forecasting subcritical cylinder wakes with Fourier Neural Operators
The learned FNO solution operator can be evaluated in milliseconds, potentially enabling faster-than-real-time modeling for predictive flow control in physical systems.
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Augmenting pretrained language models (LMs) with a vision encoder (e. g., Flamingo) has obtained the state-of-the-art results in image-to-text generation.
PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees
In this paper, we propose PerAda, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts.
BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models
We thus enable seamless open-ended social bias testing of PLMs by domain experts through an automatic large-scale generation of diverse test sentences for any combination of social categories and attributes.
Score-based Diffusion Models in Function Space
They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising.
Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids
To apply this representation to monocular scene reconstruction, we develop a scale calibration algorithm for fast geometric initialization from monocular depth priors.
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs
Deep learning often faces the challenge of efficiently processing dynamic inputs, such as sensor data or user inputs.
Tipping Point Forecasting in Non-Stationary Dynamics on Function Spaces
Tipping points are abrupt, drastic, and often irreversible changes in the evolution of non-stationary and chaotic dynamical systems.
Spacetime Surface Regularization for Neural Dynamic Scene Reconstruction
We propose an algorithm, 4DRegSDF, for the spacetime surface regularization to improve the fidelity of neural rendering and reconstruction in dynamic scenes.
End-to-end 3D Tracking with Decoupled Queries
In this work, we present an end-to-end framework for camera-based 3D multi-object tracking, called DQTrack.
Neural Operators for Accelerating Scientific Simulations and Design
Scientific discovery and engineering design are currently limited by the time and cost of physical experiments, selected mostly through trial-and-error and intuition that require deep domain expertise.
Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs
Our contributions are threefold: i) we enable parallelization over input samples with a novel multi-grid-based domain decomposition, ii) we represent the parameters of the model in a high-order latent subspace of the Fourier domain, through a global tensor factorization, resulting in an extreme reduction in the number of parameters and improved generalization, and iii) we propose architectural improvements to the backbone FNO.
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences.
EKGNet: A 10.96μW Fully Analog Neural Network for Intra-Patient Arrhythmia Classification
We present an integrated approach by combining analog computing and deep learning for electrocardiogram (ECG) arrhythmia classification.
Plasma Surrogate Modelling using Fourier Neural Operators
Predicting plasma evolution within a Tokamak reactor is crucial to realizing the goal of sustainable fusion.
Deep Multimodal Fusion for Surgical Feedback Classification
This work offers an important first look at the feasibility of automated classification of real-world live surgical feedback based on text, audio, and video modalities.
Perspectives on the State and Future of Deep Learning -- 2023
The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time.
