Search Results for author: Fei Sha

Found 87 papers, 29 papers with code

Synthesize Policies for Transfer and Adaptation across Tasks and Environments

no code implementations NeurIPS 2018 Hexiang Hu, Liyu Chen, Boqing Gong, Fei Sha

The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.

DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems

no code implementations6 Feb 2024 Yair Schiff, Zhong Yi Wan, Jeffrey B. Parker, Stephan Hoyer, Volodymyr Kuleshov, Fei Sha, Leonardo Zepeda-Núñez

Learning dynamics from dissipative chaotic systems is notoriously difficult due to their inherent instability, as formalized by their positive Lyapunov exponents, which exponentially amplify errors in the learned dynamics.

A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

no code implementations1 Nov 2023 Tiwalayo Eisape, MH Tessler, Ishita Dasgupta, Fei Sha, Sjoerd van Steenkiste, Tal Linzen

A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises.

Logical Fallacies

The Impact of Depth and Width on Transformer Language Model Generalization

no code implementations30 Oct 2023 Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, Tal Linzen

To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways.

Language Modelling

WeatherBench 2: A benchmark for the next generation of data-driven global weather models

1 code implementation29 Aug 2023 Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, Fei Sha

WeatherBench 2 is an update to the global, medium-range (1-14 day) weather forecasting benchmark proposed by Rasp et al. (2020), designed with the aim to accelerate progress in data-driven weather modeling.

Weather Forecasting

Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories

1 code implementation ICCV 2023 Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, Vittorio Ferrari

Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13. 0% accuracy on our dataset.

Question Answering Retrieval +1

User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems

no code implementations13 Jun 2023 Marc Finzi, Anudhyan Boral, Andrew Gordon Wilson, Fei Sha, Leonardo Zepeda-Núñez

In this work, we develop a probabilistic approximation scheme for the conditional score function which provably converges to the true distribution as the noise level decreases.

Uncertainty Quantification

V2Meow: Meowing to the Visual Beat via Video-to-Music Generation

no code implementations11 May 2023 Kun Su, Judith Yue Li, Qingqing Huang, Dima Kuzmin, Joonseok Lee, Chris Donahue, Fei Sha, Aren Jansen, Yu Wang, Mauro Verzetti, Timo I. Denk

Video-to-music generation demands both a temporally localized high-quality listening experience and globally aligned video-acoustic signatures.

Music Generation

Policy-Induced Self-Supervision Improves Representation Finetuning in Visual RL

no code implementations12 Feb 2023 Sébastien M. R. Arnold, Fei Sha

Together with freezing the bottom layers, this objective results in significantly better representation than frozen, finetuned, and self-supervised alternatives on a wide range of benchmarks.

Evolve Smoothly, Fit Consistently: Learning Smooth Latent Dynamics For Advection-Dominated Systems

no code implementations25 Jan 2023 Zhong Yi Wan, Leonardo Zepeda-Núñez, Anudhyan Boral, Fei Sha

We present a data-driven, space-time continuous framework to learn surrogate models for complex physical systems described by advection-dominated partial differential equations.

ALMA: Hierarchical Learning for Composite Multi-Agent Tasks

1 code implementation27 May 2022 Shariq Iqbal, Robby Costales, Fei Sha

Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, we observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus on one subtask to the exclusion of all else in the environment.

Decision Making Inductive Bias +1

Possibility Before Utility: Learning And Using Hierarchical Affordances

1 code implementation ICLR 2022 Robby Costales, Shariq Iqbal, Fei Sha

Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history.

Hierarchical Reinforcement Learning Navigate +2

Policy Learning and Evaluation with Randomized Quasi-Monte Carlo

no code implementations16 Feb 2022 Sebastien M. R. Arnold, Pierre L'Ecuyer, Liyu Chen, Yi-fan Chen, Fei Sha

Reinforcement learning constantly deals with hard integrals, for example when computing expectations in policy evaluation and policy iteration.

Continuous Control Policy Gradient Methods +1

HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks

no code implementations NeurIPS Workshop DLDE 2021 Filipe de Avila Belbute-Peres, Yi-fan Chen, Fei Sha

Many types of physics-informed neural network models have been proposed in recent years as approaches for learning solutions to differential equations.

Visually Grounded Concept Composition

no code implementations Findings (EMNLP) 2021 BoWen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, Fei Sha

We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images.

Sentence

Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?

2 code implementations EMNLP 2021 Linlu Qiu, Hexiang Hu, BoWen Zhang, Peter Shaw, Fei Sha

We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding.

Systematic Generalization

ReadTwice: Reading Very Large Documents with Memories

no code implementations NAACL 2021 Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha

Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections.

Question Answering

Embedding Adaptation is Still Needed for Few-Shot Learning

no code implementations15 Apr 2021 Sébastien M. R. Arnold, Fei Sha

Constructing new and more challenging tasksets is a fruitful methodology to analyse and understand few-shot classification methods.

Clustering Few-Shot Learning +1

DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections

no code implementations EACL 2021 Yury Zemlyanskiy, Sudeep Gandhe, Ruining He, Bhargav Kanagal, Anirudh Ravula, Juraj Gottweis, Fei Sha, Ilya Eckstein

This enables a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision.

Knowledge Base Completion Natural Language Queries +4

A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus

no code implementations18 Nov 2020 BoWen Zhang, Hexiang Hu, Joonseok Lee, Ming Zhao, Sheide Chammas, Vihan Jain, Eugene Ie, Fei Sha

Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation.

Language Modelling Masked Language Modeling +3

Learning to Represent Image and Text with Denotation Graph

no code implementations EMNLP 2020 BoWen Zhang, Hexiang Hu, Vihan Jain, Eugene Ie, Fei Sha

Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images.

Attribute Image Retrieval +4

Drinking from a Firehose: Continual Learning with Web-scale Natural Language

1 code implementation18 Jul 2020 Hexiang Hu, Ozan Sener, Fei Sha, Vladlen Koltun

Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.

Continual Learning

Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation

no code implementations13 Jun 2020 Zhiyun Lu, Eugene Ie, Fei Sha

Many methods have been proposed to quantify the predictive uncertainty associated with the outputs of deep neural networks.

Out-of-Distribution Detection

Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning

2 code implementations7 Jun 2020 Shariq Iqbal, Christian A. Schroeder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, Fei Sha

Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities; however, common patterns of behavior often emerge among these agents/entities.

counterfactual Multi-agent Reinforcement Learning +3

BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps

1 code implementation ACL 2020 Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, Fei Sha

To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially.

Imitation Learning Navigate +1

Visual Storytelling via Predicting Anchor Word Embeddings in the Stories

no code implementations13 Jan 2020 Bowen Zhang, Hexiang Hu, Fei Sha

To narrate a sequence of images, we use the predicted anchor word embeddings and the image features as the joint input to a seq2seq model.

Visual Storytelling Word Embeddings

When MAML Can Adapt Fast and How to Assist When It Cannot

1 code implementation30 Oct 2019 Sébastien M. R. Arnold, Shariq Iqbal, Fei Sha

On the other hand, we have just started to understand and analyze how they are able to adapt fast to new tasks.

Meta-Learning Multi-Task Learning +1

Decoupling Adaptation from Modeling with Meta-Optimizers for Meta Learning

1 code implementation25 Sep 2019 Sébastien M.R. Arnold, Shariq Iqbal, Fei Sha

Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al, 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks.

Meta-Learning

Topic Augmented Generator for Abstractive Summarization

no code implementations19 Aug 2019 Melissa Ailem, Bo-Wen Zhang, Fei Sha

In this paper, we propose a new decoder where the output summary is generated by conditioning on both the input text and the latent topics of the document.

Abstractive Text Summarization

Neural Theorem Provers Do Not Learn Rules Without Exploration

1 code implementation17 Jun 2019 Michiel de Jong, Fei Sha

Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks.

Automated Theorem Proving

Amortized Inference of Variational Bounds for Learning Noisy-OR

no code implementations6 Jun 2019 Yiming Yan, Melissa Ailem, Fei Sha

Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds.

Variational Inference

Synthesized Policies for Transfer and Adaptation across Tasks and Environments

2 code implementations NeurIPS 2018 Hexiang Hu, Liyu Chen, Boqing Gong, Fei Sha

The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.

Hyper-parameter Tuning under a Budget Constraint

no code implementations1 Feb 2019 Zhiyun Lu, Chao-Kai Chiang, Fei Sha

We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint.

Decision Making

Classifier and Exemplar Synthesis for Zero-Shot Learning

1 code implementation16 Dec 2018 Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, Fei Sha

Zero-shot learning (ZSL) enables solving a task without the need to see its examples.

Denoising Zero-Shot Learning

Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions

5 code implementations CVPR 2020 Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, Fei Sha

Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels.

Few-Shot Image Classification Few-Shot Learning +3

Cross-Modal and Hierarchical Modeling of Video and Text

1 code implementation ECCV 2018 Bowen Zhang, Hexiang Hu, Fei Sha

Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story.

Action Recognition Retrieval +3

Actor-Attention-Critic for Multi-Agent Reinforcement Learning

3 code implementations ICLR 2019 Shariq Iqbal, Fei Sha

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings.

Multi-agent Reinforcement Learning reinforcement-learning +1

A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images

no code implementations EMNLP 2018 Melissa Ailem, Bo-Wen Zhang, Aurelien Bellet, Pascal Denis, Fei Sha

Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data.

Coreference Resolution Image Classification +5

Retrospective Encoders for Video Summarization

no code implementations ECCV 2018 Ke Zhang, Kristen Grauman, Fei Sha

The key idea is to complement the discriminative losses with another loss which measures if the predicted summary preserves the same information as in the original video.

Metric Learning Video Summarization

Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner

no code implementations CONLL 2018 Yury Zemlyanskiy, Fei Sha

There have been several attempts to define a plausible motivation for a chit-chat dialogue agent that can lead to engaging conversations.

Learning Answer Embeddings for Visual Question Answering

no code implementations CVPR 2018 Hexiang Hu, Wei-Lun Chao, Fei Sha

These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers.

Question Answering Transfer Learning +1

Cross-Dataset Adaptation for Visual Question Answering

no code implementations CVPR 2018 Wei-Lun Chao, Hexiang Hu, Fei Sha

Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model.

Domain Adaptation Question Answering +1

Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets

no code implementations NAACL 2018 Wei-Lun Chao, Hexiang Hu, Fei Sha

We apply the procedures to re-construct decoy answers for two popular Visual QA datasets as well as to create a new Visual QA dataset from the Visual Genome project, resulting in the largest dataset for this task.

Multiple-choice Question Answering +1

LabelBank: Revisiting Global Perspectives for Semantic Segmentation

1 code implementation29 Mar 2017 Hexiang Hu, Zhiwei Deng, Guang-Tong Zhou, Fei Sha, Greg Mori

We advocate that holistic inference of image concepts provides valuable information for detailed pixel labeling.

Segmentation Semantic Segmentation

Kernel Approximation Methods for Speech Recognition

no code implementations13 Jan 2017 Avner May, Alireza Bagheri Garakani, Zhiyun Lu, Dong Guo, Kuan Liu, Aurélien Bellet, Linxi Fan, Michael Collins, Daniel Hsu, Brian Kingsbury, Michael Picheny, Fei Sha

First, in order to reduce the number of random features required by kernel models, we propose a simple but effective method for feature selection.

feature selection speech-recognition +1

Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task

no code implementations22 Dec 2016 Nan Ding, Sebastian Goodman, Fei Sha, Radu Soricut

We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options.

Image Captioning Multi-Task Learning +1

Supervised Word Mover's Distance

1 code implementation NeurIPS 2016 Gao Huang, Chuan Guo, Matt J. Kusner, Yu Sun, Fei Sha, Kilian Q. Weinberger

Accurately measuring the similarity between text documents lies at the core of many real world applications of machine learning.

Document Classification General Classification +2

Recalling Holistic Information for Semantic Segmentation

no code implementations24 Nov 2016 Hexiang Hu, Zhiwei Deng, Guang-Tong Zhou, Fei Sha, Greg Mori

We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling.

Segmentation Semantic Segmentation

Attention Correctness in Neural Image Captioning

no code implementations31 May 2016 Chenxi Liu, Junhua Mao, Fei Sha, Alan Yuille

Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision.

Image Captioning

Video Summarization with Long Short-term Memory

1 code implementation26 May 2016 Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman

We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.

Domain Adaptation Structured Prediction +1

Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning

no code implementations ICCV 2017 Soravit Changpinyo, Wei-Lun Chao, Fei Sha

Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available.

Clustering Object +1

An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild

1 code implementation13 May 2016 Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, Fei Sha

Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.

Few-Shot Learning Generalized Zero-Shot Learning +1

Summary Transfer: Exemplar-based Subset Selection for Video Summarization

no code implementations CVPR 2016 Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman

Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.

Video Summarization

Synthesized Classifiers for Zero-Shot Learning

2 code implementations CVPR 2016 Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, Fei Sha

Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.

Object Zero-Shot Learning

Aligning where to see and what to tell: image caption with region-based attention and scene factorization

1 code implementation20 Jun 2015 Junqi Jin, Kun fu, Runpeng Cui, Fei Sha, Chang-Shui Zhang

In this paper, we propose an image caption system that exploits the parallel structures between images and sentences.

Image Captioning

Similarity Learning for High-Dimensional Sparse Data

1 code implementation10 Nov 2014 Kuan Liu, Aurélien Bellet, Fei Sha

A good measure of similarity between data points is crucial to many tasks in machine learning.

Dimensionality Reduction Metric Learning +1

Large-Margin Determinantal Point Processes

no code implementations6 Nov 2014 Boqing Gong, Wei-Lun Chao, Kristen Grauman, Fei Sha

Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.

Point Processes Video Summarization

Decorrelating Semantic Visual Attributes by Resisting the Urge to Share

no code implementations CVPR 2014 Dinesh Jayaraman, Fei Sha, Kristen Grauman

Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.

Attribute Multi-Task Learning

Two-Stage Metric Learning

no code implementations12 May 2014 Jun Wang, Ke Sun, Fei Sha, Stephane Marchand-Maillet, Alexandros Kalousis

This induces in the input data space a new family of distance metric with unique properties.

Metric Learning Vocal Bursts Valence Prediction

Sparse Compositional Metric Learning

no code implementations15 Apr 2014 Yuan Shi, Aurélien Bellet, Fei Sha

We propose a new approach for metric learning by framing it as learning a sparse combination of locally discriminative metrics that are inexpensive to generate from the training data.

General Classification Metric Learning

A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning

no code implementations9 Apr 2014 Aurélien Bellet, YIngyu Liang, Alireza Bagheri Garakani, Maria-Florina Balcan, Fei Sha

We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an $\epsilon$-approximate solution.

Sparse Learning

Reshaping Visual Datasets for Domain Adaptation

no code implementations NeurIPS 2013 Boqing Gong, Kristen Grauman, Fei Sha

By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.

Domain Adaptation Human Activity Recognition +1

Similarity Component Analysis

no code implementations NeurIPS 2013 Soravit Changpinyo, Kuan Liu, Fei Sha

Moreover, we show how SCA can be instrumental in exploratory analysis of data, where we gain insights about the data by examining patterns hidden in its latent components' local similarity values.

Link Prediction Metric Learning

Demystifying Information-Theoretic Clustering

no code implementations15 Oct 2013 Greg Ver Steeg, Aram Galstyan, Fei Sha, Simon DeDeo

We propose a novel method for clustering data which is grounded in information-theoretic principles and requires no parametric assumptions.

Clustering

Deformable Spatial Pyramid Matching for Fast Dense Correspondences

no code implementations CVPR 2013 Jaechul Kim, Ce Liu, Fei Sha, Kristen Grauman

We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.

Non-linear Metric Learning

no code implementations NeurIPS 2012 Dor Kedem, Stephen Tyree, Fei Sha, Gert R. Lanckriet, Kilian Q. Weinberger

On various benchmark data sets, we demonstrate these methods not only match the current state-of-the-art in terms of kNN classification error, but in the case of χ2-LMNN, obtain best results in 19 out of 20 learning settings.

Metric Learning

Semantic Kernel Forests from Multiple Taxonomies

no code implementations NeurIPS 2012 Sung Ju Hwang, Kristen Grauman, Fei Sha

When learning features for complex visual recognition problems, labeled image exemplars alone can be insufficient.

Object Object Recognition

Learning a Tree of Metrics with Disjoint Visual Features

no code implementations NeurIPS 2011 Kristen Grauman, Fei Sha, Sung Ju Hwang

Given a hierarchical taxonomy that captures semantic similarity between the objects, we learn a corresponding tree of metrics (ToM).

Attribute Metric Learning +2

Unsupervised Kernel Dimension Reduction

no code implementations NeurIPS 2010 Meihong Wang, Fei Sha, Michael. I. Jordan

In this framework, kernel-based measures of independence are used to derive low-dimensional representations that maximally capture information in covariates in order to predict responses.

Classification Clustering +2

DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification

no code implementations NeurIPS 2008 Simon Lacoste-Julien, Fei Sha, Michael. I. Jordan

By using the transformed topic mixture proportions as a new representation of documents, we obtain a supervised dimensionality reduction algorithm that uncovers the latent structure in a document collection while preserving predictive power for the task of classification.

Classification General Classification +2

Cannot find the paper you are looking for? You can Submit a new open access paper.