Search Results for author:
Found 37 papers, 13 papers with code
Tradeoffs Between Alignment and Helpfulness in Language Models
Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks.
Generating Benchmarks for Factuality Evaluation of Language Models
FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements.
Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework
Here we propose $\textit{Align With Purpose}$, a $\textbf{general Plug-and-Play framework}$ for enhancing a desired property in models trained with the CTC criterion.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Fundamental Limitations of Alignment in Large Language Models
An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users.
In-Context Retrieval-Augmented Language Models
Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance.
Parallel Context Windows for Large Language Models
We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training.
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks.
Standing on the Shoulders of Giant Frozen Language Models
To demonstrate this, we introduce three novel methods for leveraging frozen models: input-dependent prompt tuning, frozen readers, and recursive LMs, each of which vastly improves on current frozen-model approaches.
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks
Recently, several works have demonstrated high gains by taking a straightforward approach for incorporating intermediate supervision in compounded natural language problems: the sequence-to-sequence LM is fed with an augmented input, in which the decomposed tasks' labels are simply concatenated to the original input.
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
We highlight a bias introduced by this common practice: we prove that the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples.
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
After their successful debut in natural language processing, Transformer architectures are now becoming the de-facto standard in many domains.
Neural tensor contractions and the expressive power of deep neural quantum states
We establish a direct connection between general tensor networks and deep feed-forward artificial neural networks.
The Depth-to-Width Interplay in Self-Attention
Our guidelines elucidate the depth-to-width trade-off in self-attention networks of sizes up to the scale of GPT3 (which we project to be too deep for its size), and beyond, marking an unprecedented width of 30K as optimal for a 1-Trillion parameter network.
On the Ethics of Building AI in a Responsible Manner
The AI-alignment problem arises when there is a discrepancy between the goals that a human designer specifies to an AI learner and a potential catastrophic outcome that does not reflect what the human designer really wants.
SenseBERT: Driving Some Sense into BERT
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding.
Ranked #11 on
Word Sense Disambiguation
on Words in Context
Deep autoregressive models for the efficient variational simulation of many-body quantum systems
Artificial Neural Networks were recently shown to be an efficient representation of highly-entangled many-body quantum states.
Quantum Entanglement in Deep Learning Architectures
Modern deep learning has enabled unprecedented achievements in various domains.
Benefits of Depth for Long-Term Memory of Recurrent Networks
We prove that deep recurrent networks support Start-End separation ranks which are exponentially higher than those supported by their shallow counterparts.
On the Long-Term Memory of Deep Recurrent Networks
A key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ability to model intricate long-term temporal dependencies.
Sum-Product-Quotient Networks
We present a novel tractable generative model that extends Sum-Product Networks (SPNs) and significantly boosts their power.
On a Formal Model of Safe and Scalable Self-driving Cars
In the second part we describe a design of a system that adheres to our safety assurance requirements and is scalable to millions of cars.
Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
Expressive efficiency refers to the ability of a network architecture to realize functions that require an alternative architecture to be much larger.
Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
This description enables us to carry a graph-theoretic analysis of a convolutional network, with which we demonstrate a direct control over the inductive bias of the deep network via its channel numbers, that are related to the min-cut in the underlying graph.
Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions
By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency.
On the Expressive Power of Overlapping Architectures of Deep Learning
Expressive efficiency refers to the relation between two architectures A and B, whereby any function realized by B could be replicated by A, but there exists functions realized by A, which cannot be replicated by B unless its size grows significantly larger.
Tensorial Mixture Models
Other methods, based on arithmetic circuits and sum-product networks, do allow tractable marginalization, but their performance is challenged by the need to learn the structure of a circuit.
Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
Second, the Markov Decision Process model often used in robotics is problematic in our case because of unpredictable behavior of other agents in this multi-agent scenario.
Learning a Metric Embedding for Face Recognition using the Multibatch Method
This work is motivated by the engineering task of achieving a near state-of-the-art face recognition on a minimal computing budget running on an embedded system.
Inductive Bias of Deep Convolutional Networks through Pooling Geometry
In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.
On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
We compare the end-to-end training approach to a modular approach in which a system is decomposed into semantically meaningful components.
Convolutional Rectifier Networks as Generalized Tensor Decompositions
Second, and more importantly, we show that depth efficiency is weaker with convolutional rectifier networks than it is with convolutional arithmetic circuits.
Long-term Planning by Short-term Prediction
We argue that dual versions of the MDP framework (that depend on the value function and the $Q$ function) are problematic for autonomous driving applications due to the non Markovian of the natural state space representation, and due to the continuous state and action spaces.
On the Expressive Power of Deep Learning: A Tensor Analysis
In this work we derive a deep network architecture based on arithmetic circuits that inherently employs locality, sharing and pooling.
Deep SimNets
We present a deep layered architecture that generalizes convolutional neural networks (ConvNets).
SimNets: A Generalization of Convolutional Networks
We present a deep layered architecture that generalizes classical convolutional neural networks (ConvNets).
ShareBoost: Efficient multiclass learning with feature sharing
We consider the problem of learning a multiclass predictor that uses only few features, and in particular, the number of used features should increase sub-linearly with the number of possible classes.
Introduction to Machine Learning: Class Notes 67577
Introduction to Machine learning covering Statistical Inference (Bayes, EM, ML/MaxEnt duality), algebraic and spectral methods (PCA, LDA, CCA, Clustering), and PAC learning (the Formal model, VC dimension, Double Sampling theorem).
