no code implementations • NeurIPS 2018 • Hexiang Hu, Liyu Chen, Boqing Gong, Fei Sha
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
no code implementations • 1 Nov 2023 • Tiwalayo Eisape, MH Tessler, Ishita Dasgupta, Fei Sha, Sjoerd van Steenkiste, Tal Linzen
A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises.
no code implementations • 30 Oct 2023 • Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, Tal Linzen
To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways.
1 code implementation • 29 Aug 2023 • Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, Fei Sha
WeatherBench 2 is an update to the global, medium-range (1-14 day) weather forecasting benchmark proposed by Rasp et al. (2020), designed with the aim to accelerate progress in data-driven weather modeling.
no code implementations • 24 Jun 2023 • Lizao Li, Rob Carver, Ignacio Lopez-Gomez, Fei Sha, John Anderson
The dominant approach to representing uncertainty in weather forecasting is to generate an ensemble of forecasts.
1 code implementation • ICCV 2023 • Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, Vittorio Ferrari
Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13. 0% accuracy on our dataset.
no code implementations • 13 Jun 2023 • Marc Finzi, Anudhyan Boral, Andrew Gordon Wilson, Fei Sha, Leonardo Zepeda-Núñez
In this work, we develop a probabilistic approximation scheme for the conditional score function which provably converges to the true distribution as the noise level decreases.
no code implementations • 11 May 2023 • Kun Su, Judith Yue Li, Qingqing Huang, Dima Kuzmin, Joonseok Lee, Chris Donahue, Fei Sha, Aren Jansen, Yu Wang, Mauro Verzetti, Timo I. Denk
Generating high quality music that complements the visual content of a video is a challenging task.
no code implementations • 12 Feb 2023 • Sébastien M. R. Arnold, Fei Sha
Together with freezing the bottom layers, this objective results in significantly better representation than frozen, finetuned, and self-supervised alternatives on a wide range of benchmarks.
no code implementations • 25 Jan 2023 • Zhong Yi Wan, Leonardo Zepeda-Núñez, Anudhyan Boral, Fei Sha
We present a data-driven, space-time continuous framework to learn surrogate models for complex physical systems described by advection-dominated partial differential equations.
no code implementations • 25 Jan 2023 • Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William Cohen
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks.
no code implementations • 15 Dec 2022 • Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks.
Ranked #3 on
Question Answering
on WebQuestions
no code implementations • COLING 2022 • Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
Then, it retrieves exemplars with outputs similar to the preliminary prediction which are used to generate a final prediction.
1 code implementation • 27 May 2022 • Shariq Iqbal, Robby Costales, Fei Sha
Work in MARL often focuses on solving tasks where agents interact with all other agents and entities in the environment; however, we observe that real-world tasks are often composed of several isolated instances of local agent interactions (subtasks), and each agent can meaningfully focus on one subtask to the exclusion of all else in the environment.
no code implementations • 24 May 2022 • Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling.
1 code implementation • ICLR 2022 • Robby Costales, Shariq Iqbal, Fei Sha
Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history.
no code implementations • 16 Feb 2022 • Sebastien M. R. Arnold, Pierre L'Ecuyer, Liyu Chen, Yi-fan Chen, Fei Sha
Reinforcement learning constantly deals with hard integrals, for example when computing expectations in policy evaluation and policy iteration.
2 code implementations • NAACL 2022 • Linlu Qiu, Peter Shaw, Panupong Pasupat, Paweł Krzysztof Nowak, Tal Linzen, Fei Sha, Kristina Toutanova
Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization.
no code implementations • 14 Dec 2021 • BoWen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, Fei Sha
We term this approach as Co-training Videos and Images for Action Recognition (CoVeR).
Ranked #6 on
Action Classification
on Moments in Time
(using extra training data)
1 code implementation • 9 Nov 2021 • Wang Zhu, Peter Shaw, Tal Linzen, Fei Sha
Neural network models often generalize poorly to mismatched domains or distributions.
no code implementations • NeurIPS Workshop DLDE 2021 • Filipe de Avila Belbute-Peres, Yi-fan Chen, Fei Sha
Many types of physics-informed neural network models have been proposed in recent years as approaches for learning solutions to differential equations.
1 code implementation • ICLR 2022 • Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, William Cohen
We propose to address this problem by integrating a semi-parametric representation of a large text corpus into a Transformer model as a source of factual knowledge.
Ranked #1 on
Passage Retrieval
on EntityQuestions
no code implementations • Findings (EMNLP) 2021 • BoWen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, Fei Sha
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images.
2 code implementations • EMNLP 2021 • Linlu Qiu, Hexiang Hu, BoWen Zhang, Peter Shaw, Fei Sha
We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding.
no code implementations • NAACL 2021 • Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections.
no code implementations • 15 Apr 2021 • Sébastien M. R. Arnold, Fei Sha
Constructing new and more challenging tasksets is a fruitful methodology to analyse and understand few-shot classification methods.
no code implementations • EACL 2021 • Yury Zemlyanskiy, Sudeep Gandhe, Ruining He, Bhargav Kanagal, Anirudh Ravula, Juraj Gottweis, Fei Sha, Ilya Eckstein
This enables a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision.
no code implementations • 18 Nov 2020 • BoWen Zhang, Hexiang Hu, Joonseok Lee, Ming Zhao, Sheide Chammas, Vihan Jain, Eugene Ie, Fei Sha
Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation.
1 code implementation • 23 Oct 2020 • Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, Eugene Ie
Summarization is the task of compressing source document(s) into coherent and succinct passages.
no code implementations • EMNLP 2020 • BoWen Zhang, Hexiang Hu, Vihan Jain, Eugene Ie, Fei Sha
Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images.
1 code implementation • 18 Jul 2020 • Hexiang Hu, Ozan Sener, Fei Sha, Vladlen Koltun
Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.
no code implementations • 13 Jun 2020 • Zhiyun Lu, Eugene Ie, Fei Sha
Many methods have been proposed to quantify the predictive uncertainty associated with the outputs of deep neural networks.
2 code implementations • 7 Jun 2020 • Shariq Iqbal, Christian A. Schroeder de Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, Fei Sha
Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities; however, common patterns of behavior often emerge among these agents/entities.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
1 code implementation • ACL 2020 • Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, Fei Sha
To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially.
no code implementations • 13 Jan 2020 • Bowen Zhang, Hexiang Hu, Fei Sha
To narrate a sequence of images, we use the predicted anchor word embeddings and the image features as the joint input to a seq2seq model.
Ranked #15 on
Visual Storytelling
on VIST
1 code implementation • 30 Oct 2019 • Sébastien M. R. Arnold, Shariq Iqbal, Fei Sha
On the other hand, we have just started to understand and analyze how they are able to adapt fast to new tasks.
1 code implementation • 25 Sep 2019 • Sébastien M.R. Arnold, Shariq Iqbal, Fei Sha
Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al, 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks.
no code implementations • 19 Aug 2019 • Melissa Ailem, Bo-Wen Zhang, Fei Sha
In this paper, we propose a new decoder where the output summary is generated by conditioning on both the input text and the latent topics of the document.
1 code implementation • 17 Jun 2019 • Michiel de Jong, Fei Sha
Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks.
no code implementations • 6 Jun 2019 • Yiming Yan, Melissa Ailem, Fei Sha
Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds.
1 code implementation • 28 May 2019 • Shariq Iqbal, Fei Sha
Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
2 code implementations • NeurIPS 2018 • Hexiang Hu, Liyu Chen, Boqing Gong, Fei Sha
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence.
no code implementations • 1 Feb 2019 • Zhiyun Lu, Chao-Kai Chiang, Fei Sha
We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint.
1 code implementation • 16 Dec 2018 • Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, Fei Sha
Zero-shot learning (ZSL) enables solving a task without the need to see its examples.
5 code implementations • CVPR 2020 • Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, Fei Sha
Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels.
1 code implementation • ECCV 2018 • Bowen Zhang, Hexiang Hu, Fei Sha
Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story.
3 code implementations • ICLR 2019 • Shariq Iqbal, Fei Sha
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
no code implementations • EMNLP 2018 • Melissa Ailem, Bo-Wen Zhang, Aurelien Bellet, Pascal Denis, Fei Sha
Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data.
no code implementations • ECCV 2018 • Ke Zhang, Kristen Grauman, Fei Sha
The key idea is to complement the discriminative losses with another loss which measures if the predicted summary preserves the same information as in the original video.
no code implementations • CONLL 2018 • Yury Zemlyanskiy, Fei Sha
There have been several attempts to define a plausible motivation for a chit-chat dialogue agent that can lead to engaging conversations.
no code implementations • COLING 2018 • Soravit Changpinyo, Hexiang Hu, Fei Sha
We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks.
no code implementations • CVPR 2018 • Hexiang Hu, Wei-Lun Chao, Fei Sha
These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers.
no code implementations • CVPR 2018 • Wei-Lun Chao, Hexiang Hu, Fei Sha
Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model.
no code implementations • NeurIPS 2017 • Maximilian Alber, Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, Fei Sha
Kernel machines as well as neural networks possess universal function approximation properties.
no code implementations • NAACL 2018 • Wei-Lun Chao, Hexiang Hu, Fei Sha
We apply the procedures to re-construct decoy answers for two popular Visual QA datasets as well as to create a new Visual QA dataset from the Visual Genome project, resulting in the largest dataset for this task.
1 code implementation • 29 Mar 2017 • Hexiang Hu, Zhiwei Deng, Guang-Tong Zhou, Fei Sha, Greg Mori
We advocate that holistic inference of image concepts provides valuable information for detailed pixel labeling.
no code implementations • 13 Jan 2017 • Avner May, Alireza Bagheri Garakani, Zhiyun Lu, Dong Guo, Kuan Liu, Aurélien Bellet, Linxi Fan, Michael Collins, Daniel Hsu, Brian Kingsbury, Michael Picheny, Fei Sha
First, in order to reduce the number of random features required by kernel models, we propose a simple but effective method for feature selection.
3 code implementations • CVPR 2017 • Hexiang Hu, Shiyi Lan, Yuning Jiang, Zhimin Cao, Fei Sha
Objects appear to scale differently in natural images.
no code implementations • 22 Dec 2016 • Nan Ding, Sebastian Goodman, Fei Sha, Radu Soricut
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options.
1 code implementation • NeurIPS 2016 • Gao Huang, Chuan Guo, Matt J. Kusner, Yu Sun, Fei Sha, Kilian Q. Weinberger
Accurately measuring the similarity between text documents lies at the core of many real world applications of machine learning.
no code implementations • 24 Nov 2016 • Hexiang Hu, Zhiwei Deng, Guang-Tong Zhou, Fei Sha, Greg Mori
We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling.
no code implementations • 31 May 2016 • Chenxi Liu, Junhua Mao, Fei Sha, Alan Yuille
Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision.
1 code implementation • 26 May 2016 • Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.
no code implementations • ICCV 2017 • Soravit Changpinyo, Wei-Lun Chao, Fei Sha
Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available.
1 code implementation • 13 May 2016 • Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, Fei Sha
Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.
no code implementations • 18 Mar 2016 • Zhiyun Lu, Dong Guo, Alireza Bagheri Garakani, Kuan Liu, Avner May, Aurelien Bellet, Linxi Fan, Michael Collins, Brian Kingsbury, Michael Picheny, Fei Sha
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition.
no code implementations • CVPR 2016 • Ke Zhang, Wei-Lun Chao, Fei Sha, Kristen Grauman
Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.
2 code implementations • CVPR 2016 • Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, Fei Sha
Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.
Ranked #1 on
Few-Shot Image Classification
on AWA - 0-Shot
1 code implementation • 20 Jun 2015 • Junqi Jin, Kun fu, Runpeng Cui, Fei Sha, Chang-Shui Zhang
In this paper, we propose an image caption system that exploits the parallel structures between images and sentences.
no code implementations • NeurIPS 2014 • Boqing Gong, Wei-Lun Chao, Kristen Grauman, Fei Sha
Video summarization is a challenging problem with great application potential.
no code implementations • 14 Nov 2014 • Zhiyun Lu, Avner May, Kuan Liu, Alireza Bagheri Garakani, Dong Guo, Aurélien Bellet, Linxi Fan, Michael Collins, Brian Kingsbury, Michael Picheny, Fei Sha
The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
1 code implementation • 10 Nov 2014 • Kuan Liu, Aurélien Bellet, Fei Sha
A good measure of similarity between data points is crucial to many tasks in machine learning.
no code implementations • 6 Nov 2014 • Boqing Gong, Wei-Lun Chao, Kristen Grauman, Fei Sha
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
no code implementations • CVPR 2014 • Dinesh Jayaraman, Fei Sha, Kristen Grauman
Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.
no code implementations • 12 May 2014 • Jun Wang, Ke Sun, Fei Sha, Stephane Marchand-Maillet, Alexandros Kalousis
This induces in the input data space a new family of distance metric with unique properties.
no code implementations • 15 Apr 2014 • Yuan Shi, Aurélien Bellet, Fei Sha
We propose a new approach for metric learning by framing it as learning a sparse combination of locally discriminative metrics that are inexpensive to generate from the training data.
no code implementations • 9 Apr 2014 • Aurélien Bellet, YIngyu Liang, Alireza Bagheri Garakani, Maria-Florina Balcan, Fei Sha
We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an $\epsilon$-approximate solution.
no code implementations • NeurIPS 2013 • Boqing Gong, Kristen Grauman, Fei Sha
By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain.
no code implementations • NeurIPS 2013 • Soravit Changpinyo, Kuan Liu, Fei Sha
Moreover, we show how SCA can be instrumental in exploratory analysis of data, where we gain insights about the data by examining patterns hidden in its latent components' local similarity values.
no code implementations • 15 Oct 2013 • Greg Ver Steeg, Aram Galstyan, Fei Sha, Simon DeDeo
We propose a novel method for clustering data which is grounded in information-theoretic principles and requires no parametric assumptions.
no code implementations • CVPR 2013 • Jaechul Kim, Ce Liu, Fei Sha, Kristen Grauman
We introduce a fast deformable spatial pyramid (DSP) matching algorithm for computing dense pixel correspondences.
no code implementations • NeurIPS 2012 • Sung Ju Hwang, Kristen Grauman, Fei Sha
When learning features for complex visual recognition problems, labeled image exemplars alone can be insufficient.
no code implementations • NeurIPS 2012 • Dor Kedem, Stephen Tyree, Fei Sha, Gert R. Lanckriet, Kilian Q. Weinberger
On various benchmark data sets, we demonstrate these methods not only match the current state-of-the-art in terms of kNN classification error, but in the case of χ2-LMNN, obtain best results in 19 out of 20 learning settings.
no code implementations • NeurIPS 2011 • Kristen Grauman, Fei Sha, Sung Ju Hwang
Given a hierarchical taxonomy that captures semantic similarity between the objects, we learn a corresponding tree of metrics (ToM).
no code implementations • NeurIPS 2010 • Meihong Wang, Fei Sha, Michael. I. Jordan
In this framework, kernel-based measures of independence are used to derive low-dimensional representations that maximally capture information in covariates in order to predict responses.
no code implementations • NeurIPS 2008 • Simon Lacoste-Julien, Fei Sha, Michael. I. Jordan
By using the transformed topic mixture proportions as a new representation of documents, we obtain a supervised dimensionality reduction algorithm that uncovers the latent structure in a document collection while preserving predictive power for the task of classification.