Search Results for author:
Found 26 papers, 11 papers with code
XGen-7B Technical Report
Most open-source LLMs, on the other hand, are limited in their ability to support longer sequence lengths, which is a key requirement for many tasks that require inference over an input context.
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages
In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill sampling, and, (4) data distributions.
Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization
Generative models have the ability to synthesize data points drawn from the data distribution, however, not all generated samples are high quality.
Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM).
BigIssue: A Realistic Bug Localization Benchmark
As machine learning tools progress, the inevitable question arises: How can machine learning help us write better code?
ProGen2: Exploring the Boundaries of Protein Language Models
Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial intelligence-driven protein design.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
To democratize this, we train and release a family of large language models up to 16. 1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER.
Ranked #89 on
Code Generation
on HumanEval
Long Document Summarization with Top-down and Bottom-up Inference
Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.
Ranked #1 on
Text Summarization
on Pubmed
MCMC Should Mix: Learning Energy-Based Model with Flow-Based Backbone
However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space.
Long Document Summarization with Top-Down and Bottom-Up Representation Inference
Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.
SCRIPT: Self-Critic PreTraining of Transformers
We introduce Self-CRItic Pretraining Transformers (SCRIPT) for representation learning of text.
Generative Text Modeling through Short Run Inference
It is initialized from the prior distribution of the latent variable and then runs a small number (e. g., 20) of Langevin dynamics steps guided by its posterior distribution.
Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
This paper proposes a latent space energy-based prior model for semi-supervised learning.
Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation
Open-domain dialogue generation has gained increasing attention in Natural Language Processing.
Learning Latent Space Energy-Based Prior Model
Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well.
MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC
Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm.
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM).
Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference
Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.
Flow Contrastive Estimation of Energy-Based Models
(2) The update of the flow model approximately minimizes the Jensen-Shannon divergence between the flow model and the data distribution.
Representation Learning: A Statistical Perspective
Learning representations of data is an important problem in statistics and machine learning.
A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
Recent work on mode connectivity in the loss landscape of deep neural networks has demonstrated that the locus of (sub-)optimal weight vectors lies on continuous paths.
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
We treat this non-convergent short-run MCMC as a learned generator model or a flow model.
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models
On the other hand, ConvNet potentials learned with non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images.
Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model.
Building a Telescope to Look Into High-Dimensional Image Spaces
However, characterizing a learned probability density to uncover the Hopfield memories of the model, encoded by the structure of the local modes, remains an open challenge.
Generative Hierarchical Learning of Sparse FRAME Models
This paper proposes a method for generative learning of hierarchical random field models.
