Search Results for author:
Found 19 papers, 9 papers with code
Adaptivity and Modularity for Efficient Generalization Over Task Complexity
We investigate how the use of a mechanism for adaptive and modular computation in transformers facilitates the learning of tasks that demand generalization over the number of sequential computation steps (i. e., the depth of the computation graph).
Diffusion Probabilistic Fields
Diffusion probabilistic models have quickly become a major approach for generative modeling of images, 3D geometry, video and other domains.
GAUDI: A Neural Architect for Immersive 3D Scene Generation
We introduce GAUDI, a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera.
Ranked #1 on
Image Generation
on ARKitScenes
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transformer models.
Exploring the Limits of Large Scale Pre-training
Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks.
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Gradual Domain Adaptation in the Wild: When Intermediate Distributions are Absent
It is shown that under the following two assumptions: (a) access to samples from intermediate distributions, and (b) samples being annotated with the amount of change from the source distribution; self-training can be successfully applied on gradually shifted samples to adapt the model toward the target distribution.
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
It has been shown that under the following two assumptions: (a) access to samples from intermediate distributions, and (b) samples being annotated with the amount of change from the source distribution, self-training can be successfully applied on gradually shifted samples to adapt the model toward the target distribution.
Long Range Arena : A Benchmark for Efficient Transformers
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity.
Long Range Arena: A Benchmark for Efficient Transformers
In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models.
Ranked #18 on
Long-range modeling
on LRA
(Pathfinder metric)
Transferring Inductive Biases through Knowledge Distillation
Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time.
Quantifying Attention Flow in Transformers
This makes attention weights unreliable as explanations probes.
A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings
The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP).
Blackbox Meets Blackbox: Representational Similarity \& Stability Analysis of Neural Language Models and Brains
In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models.
Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains
In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models.
Robust Evaluation of Language-Brain Encoding Experiments
Language-brain encoding experiments evaluate the ability of language models to predict brain responses elicited by language stimuli.
Incremental Reading for Question Answering
Current state-of-the-art question answering models reason over an entire passage, not incrementally.
Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity
We evaluate 8 different word embedding models on their usefulness for predicting the neural activation patterns associated with concrete nouns.
