Search Results for author:
Found 251 papers, 136 papers with code
The BrowserGym Ecosystem for Web Agent Research
The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs) for web interaction tasks.
Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion
Blind inverse problems, where both the target data and forward operator are unknown, are crucial to many computer vision applications.
Local Policies Enable Zero-shot Long-horizon Manipulation
We transfer our local policies from simulation to reality and observe they can solve unseen long-horizon manipulation tasks with up to 8 stages with significant pose, object and scene configuration variation.
Neural MP: A Generalist Neural Motion Planner
We perform a thorough evaluation of our method on 64 motion planning tasks across four diverse environments with randomized poses, scenes and obstacles, in the real world, demonstrating an improvement of 23%, 17% and 79% motion planning success rate over state of the art sampling, optimization and learning based planning methods.
HEMM: Holistic Evaluation of Multimodal Foundation Models
Through comprehensive experiments across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e. g., basic skills, information flows, and use cases) that pose challenges to today's models, and (2) distill performance trends regarding how different modeling dimensions (e. g., scale, pre-training data, multimodal alignment, pre-training, and instruction tuning objectives) influence performance.
Tree Search for Language Model Agents
It is the first tree search algorithm for LM agents that shows effectiveness on realistic web tasks.
Dissecting Adversarial Robustness of Multimodal LM Agents
In order to systematically examine the robustness of various multimodal we agents, we propose the Agent Robustness Evaluation (ARE) framework.
Understanding Visual Concepts Across Models
Large multimodal models such as Stable Diffusion can generate, detect, and classify new visual concepts after fine-tuning just a single word embedding.
Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks
Large Language Models (LLMs) have been shown to be capable of performing high-level planning for long-horizon robotics tasks, yet existing methods require access to a pre-defined skill library (e. g. picking, placing, pulling, pushing, navigating).
Leafy Spurge Dataset: Real-world Weed Classification Within Aerial Drone Imagery
We trained image classifiers on these data, and our best performing model, a pre-trained DINOv2 vision transformer, identified leafy spurge with 0. 84 accuracy (test set).
Stylus: Automatic Adapter Selection for Diffusion Models
This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters.
AgentKit: Structured LLM Reasoning with Dynamic Graphs
The chains of nodes can be designed to explicitly enforce a naturally structured "thought process".
Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation
Prompt engineering is effective for controlling the output of text-to-image (T2I) generative models, but it is also laborious due to the need for manually crafted prompts.
Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference
The key idea is to apply a variant of contrastive learning to time series data.
Automatic Question-Answer Generation for Long-Tail Knowledge
Pretrained Large Language Models (LLMs) have gained significant attention for addressing open-domain Question Answering (QA).
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
For decades, human-computer interaction has fundamentally been manual.
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Through extensive quantitative and qualitative analysis, we identify several limitations of text-only LLM agents, and reveal gaps in the capabilities of state-of-the-art multimodal language agents.
Manifold Preserving Guided Diffusion
Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training.
MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts
Advances in multimodal models have greatly improved how interactions relevant to various tasks are modeled.
Contrastive Difference Predictive Coding
Predicting and reasoning about the future lie at the heart of many time-series questions.
Multimodal Graph Learning for Generative Tasks
We study three research questions raised by MMGL: (1) how can we infuse multiple neighbor information into the pretrained LMs, while avoiding scalability issues?
Confronting Reward Model Overoptimization with Constrained RLHF
Large language models are typically aligned with human preferences by optimizing $\textit{reward models}$ (RMs) fitted to human feedback.
Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions
Many open-domain questions are under-specified and thus have multiple possible answers, each of which is correct under a different interpretation of the question.
Contrastive Example-Based Control
In this paper, we propose a method for offline, example-based control that learns an implicit model of multi-step transitions, rather than a reward function.
A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
One-step methods perform regularization by doing just a single step of policy improvement, while critic regularization methods do many steps of policy improvement with a regularized objective.
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data.
Localized Text-to-Image Generation for Free via Cross Attention Control
Despite the tremendous success in text-to-image generative models, localized text-to-image generation (that is, generating objects or features at specific locations in an image while maintaining a consistent overall generation) still requires either explicit training or substantial additional inference time.
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks?
Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present in either alone.
Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
In order to perform multimodal fusion of heterogeneous signals, we need to understand their interactions: how each modality individually provides information useful for a task and how this information changes in the presence of other modalities.
Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data
Robotic systems that rely primarily on self-supervised learning have the potential to decrease the amount of human annotation and engineering effort required to learn control strategies.
Generating Images with Multimodal Language Models
This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs.
Imitating Task and Motion Planning with Visuomotor Transformers
In this work, we show that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for robot manipulation.
SPRING: Studying the Paper and Reasoning to Play Games
Finally, we show the potential of games as a test bed for LLMs.
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
We propose the Plan, Eliminate, and Track (PET) framework.
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities.
Effective Data Augmentation With Diffusion Models
Data augmentation is one of the most prevalent tools in deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning.
Grounding Language Models to Images for Multimodal Inputs and Outputs
We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images.
Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
To tackle this problem, we show that relation alignment can be enforced by encouraging the directed language attention from 'mug' to 'grass' (capturing the semantic relation 'in') to match the directed visual attention from the mug to the grass.
Self-Supervised Object Goal Navigation with In-Situ Finetuning
In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more.
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress.
Paraphrasing Is All You Need for Novel Object Captioning
Since no ground truth captions are available for novel object images during training, our P2C leverages cross-modality (image-text) association modules to ensure the above caption characteristics can be properly preserved.
Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective
In this work, we propose a single objective which jointly optimizes a latent-space model and policy to achieve high returns while remaining self-consistent.
Graph Generative Model for Benchmarking Graph Neural Networks
As the field of Graph Neural Networks (GNN) continues to grow, it experiences a corresponding increase in the need for large, real-world datasets to train and test new GNN models on challenging, realistic problems.
MultiViz: Towards Visualizing and Understanding Multimodal Models
How can we visualize the internal modeling of multimodal interactions in these models?
A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search
Physically rearranging objects is an important capability for embodied agents.
Contrastive Learning as Goal-Conditioned Reinforcement Learning
While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e. g., auxiliary losses, data augmentation).
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Imitating Past Successes can be Very Suboptimal
Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience.
Reasoning over Logically Interacted Conditions for Question Answering
Even more challenging, we only provide evidences for a subset of the conditions, so some questions may not have deterministic answers.
PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
Our paper takes a step towards real-world physical commonsense reasoning by contributing PACS: the first audiovisual benchmark annotated for physical commonsense attributes.
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making.
Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
We propose a zero-shot transfer learning module for HGNNs called a Knowledge Transfer Network (KTN) that transfers knowledge from label-abundant node types to zero-labeled node types through rich relational information given in the HG.
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots.
Learning Weakly-Supervised Contrastive Representations
The first stage is to cluster data according to its auxiliary information.
Conditional Contrastive Learning with Kernel
Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on specific variables.
SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model.
Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
An alternate but important component to consider improving is the interface of the RL algorithm with the robot.
C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field.
ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers
In addition to conditional answers, the dataset also features: (1) long context documents with information that is related in logically complex ways; (2) multi-hop questions that require compositional logical reasoning; (3) a combination of extractive questions, yes/no questions, questions with multiple answers, and not-answerable questions; (4) questions asked without knowing the answers.
FILM: Following Instructions in Language with Modular Methods
In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal.
Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs
However, prior work has found that such recurrent model-free RL methods tend to perform worse than more specialized algorithms that are designed for specific types of POMDPs.
The Information Geometry of Unsupervised Reinforcement Learning
In this work, we show that unsupervised skill discovery algorithms based on mutual information maximization do not learn skills that are optimal for every possible reward function.
Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
Many model-based reinforcement learning (RL) methods follow a similar template: fit a model to previously observed data, and then use data from that model for RL or planning.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Learning Visual-Linguistic Adequacy, Fidelity, and Fluency for Novel Object Captioning
Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i. e., novel) in the training captions.
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding
The few-shot natural language understanding (NLU) task has attracted much recent attention.
Robust Predictable Control
Many of the challenges facing today's reinforcement learning (RL) algorithms, such as robustness, generalization, transfer, and computational efficiency are closely related to compression.
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
Towards Understanding and Mitigating Social Biases in Language Models
As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes.
Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Using computational models, we find that language and multimodal representations of mobile typed text (spanning typed characters, words, keystroke timings, and app usage) are predictive of daily mood.
Online Sub-Sampling for Reinforcement Learning with General Function Approximation
For a value-based method with complexity-bounded function class, we show that the policy only needs to be updated for $\propto\operatorname{poly}\log(K)$ times for running the RL algorithm for $K$ episodes while still achieving a small near-optimal regret bound.
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation.
Ranked #5 on
Speech Recognition
on LibriSpeech test-other
Integrating Auxiliary Information in Self-supervised Learning
Our approach contributes as follows: 1) Comparing to conventional self-supervised representations, the auxiliary-information-infused self-supervised representations bring the performance closer to the supervised representations; 2) The presented Cl-InfoNCE can also work with unsupervised constructed clusters (e. g., k-means clusters) and outperform strong clustering-based self-supervised learning approaches, such as the Prototypical Contrastive Learning (PCL) method; 3) We show that Cl-InfoNCE may be a better approach to leverage the data clustering information, by comparing it to the baseline approach - learning to predict the clustering assignments with cross-entropy loss.
Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
In this paper, we propose a Conditional Contrastive Learning (CCL) approach to improve the fairness of contrastive SSL methods.
Iterative Hierarchical Attention for Answering Complex Questions over Long Documents
We propose a new model, DocHopper, that iteratively attends to different parts of long, hierarchically structured documents to answer complex questions.
Ranked #2 on
Question Answering
on ConditionalQA
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration.
A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
In this report, we relate the algorithmic design of Barlow Twins' method to the Hilbert-Schmidt Independence Criterion (HSIC), thus establishing it as a contrastive learning approach that is free of negative samples.
Focused Attention Improves Document-Grounded Generation
Document grounded generation is the task of using the information provided in a document to improve text generation.
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e. g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence.
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation
Many real-world applications such as robotics provide hard constraints on power and compute that limit the viable model complexity of Reinforcement Learning (RL) agents.
Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes?
Self-supervised Representation Learning with Relative Predictive Coding
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance.
Instabilities of Offline RL with Pre-Trained Neural Representation
In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated.
On Proximal Policy Optimization's Heavy-tailed Gradients
In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function.
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
We present the Open Predicate Query Language (OPQL); a method for constructing a virtual KB (VKB) trained entirely from text.
The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
Although deep reinforcement learning has led to breakthroughs in many difficult domains, these successes have required an ever-increasing number of samples, affording only a shrinking segment of the AI community access to their development.
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private.
Feature-Robust Optimal Transport for High-Dimensional Data
To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence.
Uncertainty Weighted Offline Reinforcement Learning
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration.
Learning To Hallucinate Examples From Extrinsic and Intrinsic Supervision
Learning to hallucinate additional examples has recently been shown as a promising direction to address few-shot learning tasks.
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
In this work, we propose algorithms for cross-modal generalization: a learning paradigm to train a model that can (1) quickly perform new tasks in a target modality (i. e. meta-learning) and (2) doing so while being trained on a different source modality.
C-Learning: Learning to Achieve Goals via Recursive Classification
This problem, which can be viewed as a reframing of goal-conditioned reinforcement learning (RL), is centered around learning a conditional probability density function over future states.
Probing Predictions on OOD Images via Nearest Categories
Overall, adversarially robust networks resemble a nearest neighbor classifier when it comes to OOD data.
Unsupervised Domain Adaptation for Visual Navigation
We further show that our method can be used to transfer the navigation policies learned in simulation to the real world.
Planning with Submodular Objective Functions
We study planning with submodular objective functions, where instead of maximizing the cumulative reward, the goal is to maximize the objective value induced by a submodular function.
Case Study: Deontological Ethics in NLP
Recent work in natural language processing (NLP) has focused on ethical challenges such as understanding and mitigating bias in data and algorithms; identifying objectionable content like hate speech, stereotypes and offensive language; and building frameworks for better system design and data handling practices.
Information Obfuscation of Graph Neural Networks
While the advent of Graph Neural Networks (GNNs) has greatly improved node and graph representation learning in many applications, the neighborhood aggregation scheme exposes additional vulnerabilities to adversaries seeking to extract node-level information about sensitive attributes.
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function
In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semi-supervised approaches.
Ranked #3 on
Text Classification
on AG News
Few-Shot Learning with Intra-Class Knowledge Transfer
Recent works have proposed to solve this task by augmenting the training data of the few-shot classes using generative models with the few-shot training samples as the seeds.
Towards Debiasing Sentence Representations
As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes.
Object Goal Navigation using Goal-Oriented Semantic Exploration
We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category.
Ranked #4 on
Robot Navigation
on Habitat 2020 Object Nav test-std
Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function.
On Reward-Free Reinforcement Learning with Linear Function Approximation
The sample complexity of our algorithm is polynomial in the feature dimension and the planning horizon, and is completely independent of the number of states and actions.
Self-supervised Learning from a Multi-view Perspective
In particular, we propose a composite objective that bridges the gap between prior contrastive and predictive learning objectives, and introduce an additional objective term to discard task-irrelevant information.
Neural Methods for Point-wise Dependency Estimation
Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables.
Neural Topological SLAM for Visual Navigation
This paper studies the problem of image-goal navigation which involves navigating to the location indicated by a goal image in a novel previously unseen environment.
Feature Robust Optimal Transport for High-dimensional Data
To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence.
Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
Value function approximation has demonstrated phenomenal empirical success in reinforcement learning (RL).
Guaranteeing Reproducibility in Deep Learning Competitions
To encourage the development of methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents.
Exploring Controllable Text Generation Techniques
In this work, we provide a new schema of the pipeline of the generation process by classifying it into five modules.
Topological Sort for Sentence Ordering
Sentence ordering is the task of arranging the sentences of a given text in the correct order.
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language.
Politeness Transfer: A Tag and Generate Approach
This paper introduces a new task of politeness transfer which involves converting non-polite sentences to polite sentences while preserving the meaning.
Learning to Explore using Active Neural SLAM
The use of learning provides flexibility with respect to input modalities (in the SLAM module), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies).
Weakly-Supervised Reinforcement Learning for Controllable Behavior
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks.
A Closer Look at Accuracy vs. Robustness
Current methods for training robust networks lead to a drop in test accuracy, which has led prior works to posit that a robustness-accuracy tradeoff may be inevitable in deep learning.
On Emergent Communication in Competitive Multi-Agent Teams
Several recent works have found the emergence of grounded compositional language in the communication protocols developed by mostly cooperative multi-agent systems when learned end-to-end to maximize performance on a downstream task.
Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
In this paper, we show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks.
Differentiable Reasoning over a Virtual Knowledge Base
In particular, we describe a neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus.
Learning Not to Learn in the Presence of Noisy Labels
Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets.
Capsules with Inverted Dot-Product Attention Routing
We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote.
Think Locally, Act Globally: Federated Learning with Local and Global Representations
To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices.
Geometric Capsule Autoencoders for 3D Point Clouds
The pose encodes where the entity is, while the feature encodes what it is.
Worst Cases Policy Gradients
Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments.
Multiple Futures Prediction
Towards these goals, we introduce a probabilistic framework that efficiently learns latent variables to jointly model the multi-step future motions of agents in a scene.
Enhanced Convolutional Neural Tangent Kernels
An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel.
Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel
This new formulation gives us a better way to understand individual components of the Transformer{'}s attention, such as the better way to integrate the positional embedding.
Learning Data Manipulation for Augmentation and Weighting
In this work, we propose a new method that supports learning different manipulation schemes with the same gradient-based algorithm.
Complex Transformer: A Framework for Modeling Complex-Valued Sequence
While deep learning has received a surge of interest in a variety of fields in recent years, major deep learning models barely use complex numbers.
Ranked #2 on
Music Transcription
on MusicNet
On Universal Approximation by Neural Networks with Uniform Guarantees on Approximation of Infinite Dimensional Maps
Additionally, we provide the first lower-bound on the minimal number of input and output units required by a finite approximation to an infinite neural network to guarantee that it can uniformly approximate any nonlinear operator using samples from its inputs and outputs.
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance.
A Simple Approach to the Noisy Label Problem Through the Gambler's Loss
Learning in the presence of label noise is a challenging yet important task.
LSMI-Sinkhorn: Semi-supervised Mutual Information Estimation with Optimal Transport
To estimate the mutual information from data, a common practice is preparing a set of paired samples $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^n \stackrel{\mathrm{i. i. d.