Search Results for author:
Found 66 papers, 18 papers with code
The Impact of Positional Encoding on Length Generalization in Transformers
In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE).
Keeping Up with the Language Models: Robustness-Bias Interplay in NLI Data and Models
Bias auditing is further complicated by LM brittleness: when a presumably biased outcome is observed, is it due to model bias or model brittleness?
Equivariant Few-Shot Learning from Pretrained Models
These weights are learned directly from the data using a small neural network, leading to excellent zero-shot and finetuned results that outperform equitune.
Enhancing Protein Language Models with Structure-based Encoder and Pre-training
Despite the ability to implicitly capture inter-residue contact information, transformer-based PLMs cannot encode protein structures explicitly for better structure-aware protein representations.
Physics-Inspired Protein Encoder Pre-Training via Siamese Sequence-Structure Diffusion Trajectory Prediction
DiffPreT guides the encoder to recover the native protein sequences and structures from the perturbed ones along the multimodal diffusion trajectory, which acquires the joint distribution of sequences and structures.
AI Maintenance: A Robustness Perspective
With the advancements in machine learning (ML) methods and compute resources, artificial intelligence (AI) empowered systems are becoming a prevailing technology.
Reprogramming Pretrained Language Models for Protein Sequence Representation Learning
To this end, we reprogram an off-the-shelf pre-trained English language transformer and benchmark it on a set of protein physicochemical prediction tasks (secondary structure, stability, homology, stability) as well as on a biomedically relevant set of protein function prediction tasks (antimicrobial, toxicity, antibody affinity).
Knowledge Graph Generation From Text
In this work we propose a novel end-to-end multi-stage Knowledge Graph (KG) generation system from textual inputs, separating the overall process into two stages.
Reducing Down(stream)time: Pretraining Molecular GNNs using Heterogeneous AI Accelerators
The demonstrated success of transfer learning has popularized approaches that involve pretraining models from massive data sources and subsequent finetuning towards a specific task.
Consistent Training via Energy-Based GFlowNets for Modeling Discrete Joint Distributions
We hypothesize that this can lead to incompatibility between the inductive optimization biases in training $R$ and in training the GFlowNet, potentially leading to worse samples and slow adaptation to changes in the distribution.
Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models
We also provide a novel group-theoretic definition for fairness in NLG.
SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data
Given a pretrained model, the representations of data synthesized from the Gaussian mixture are used to compare with our reference to infer the quality.
Reprogramming Large Pretrained Language Models for Antibody Sequence Infilling
We introduce Reprogramming for Protein Sequence Infilling, a framework in which pretrained natural language models are repurposed for protein sequence infilling via reprogramming, to infill protein sequence templates as a method of novel protein generation.
AlphaFold Distillation for Improved Inverse Protein Folding
In this work, we propose to perform knowledge distillation on the folding model's confidence metrics, e. g., pTM or pLDDT scores, to obtain a smaller, faster and end-to-end differentiable distilled model, which then can be included as part of the structure consistency regularized inverse folding model training.
Cloud-Based Real-Time Molecular Screening Platform with MolFormer
With the prospect of automating a number of chemical tasks with high fidelity, chemical language processing models are emerging at a rapid speed.
Active Sampling of Multiple Sources for Sequential Estimation
Additionally, each process $i\in\{1, \dots, K\}$ has a private parameter $\alpha_i$.
Causal Graphs Underlying Generative Models: Path to Learning with Limited Data
In this work, we provide a simple algorithm that relies on perturbation experiments on latent codes of a pre-trained generative autoencoder to uncover a causal graph that is implied by the generative model.
Accelerating Material Design with the Generative Toolkit for Scientific Discovery
With the growing availability of data within various scientific domains, generative models hold enormous potential to accelerate scientific discovery.
Learning Geometrically Disentangled Representations of Protein Folding Simulations
Massive molecular simulations of drug-target proteins have been used as a tool to understand disease mechanism and develop therapeutics.
Accelerating Inhibitor Discovery With A Deep Generative Foundation Model: Validation for SARS-CoV-2 Drug Targets
To perform target-aware design of novel inhibitor molecules, a protein sequence-conditioned sampling on the generative foundation model is performed.
Accurate Clinical Toxicity Prediction using Multi-task Deep Neural Nets and Contrastive Molecular Explanations
Additionally, our multi-task approach is comprehensive in the sense that it is comparable to state-of-the-art approaches for specific endpoints in in vitro, in vivo and clinical platforms.
Data-Efficient Graph Grammar Learning for Molecular Generation
This is a non-trivial task for neural network-based generative models since the relevant chemical knowledge can only be extracted and generalized from the limited training data.
Protein Representation Learning by Geometric Structure Pretraining
Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function.
Biological Sequence Design with GFlowNets
In this work, we propose an active learning algorithm leveraging epistemic uncertainty estimation and the recently proposed GFlowNets as a generator of diverse candidate solutions, with the objective to obtain a diverse batch of useful (as defined by some utility function, for example, the predicted anti-microbial activity of a peptide) and informative candidates after each round.
Towards Creativity Characterization of Generative Models via Group-based Subset Scanning
We propose group-based subset scanning to identify, quantify, and characterize creative processes by detecting a subset of anomalous node-activations in the hidden layers of the generative models.
Fourier Representations for Black-Box Optimization over Categorical Variables
In order to improve the performance and sample efficiency of such algorithms, we propose to use existing methods in conjunction with a surrogate model for the black-box evaluations over purely categorical variables.
Sample-Efficient Generation of Novel Photo-acid Generator Molecules using a Deep Generative Model
Photo-acid generators (PAGs) are compounds that release acids ($H^+$ ions) when exposed to light.
Best Arm Identification in Contaminated Stochastic Bandits
Owing to the adversarial contamination of the rewards, each arm's mean is only partially identifiable.
Mean-based Best Arm Identification in Stochastic Bandits under Reward Contamination
Owing to the adversarial contamination of the rewards, each arm's mean is only partially identifiable.
Benchmarking deep generative models for diverse antibody sequence design
Here we consider three recently proposed deep generative frameworks for protein design: (AR) the sequence-based autoregressive generative model, (GVP) the precise structure-based graph neural network, and Fold2Seq that leverages a fuzzy and scale-free representation of a three-dimensional fold, while enforcing structure-to-sequence (and vice versa) consistency.
Physics-enhanced deep surrogates for PDEs
We present a ''physics-enhanced deep-surrogate'' (''PEDS'') approach towards developing fast surrogate models for complex physical systems, which is described by partial differential equations (PDEs) and similar models.
ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models
Automatic construction of relevant Knowledge Bases (KBs) from text, and generation of semantically meaningful text from KBs are both long-standing goals in Machine Learning.
Towards Interpreting Zoonotic Potential of Betacoronavirus Sequences With Attention
Current methods for viral discovery target evolutionarily conserved proteins that accurately identify virus families but remain unable to distinguish the zoonotic potential of newly discovered viruses.
Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design
Designing novel protein sequences for a desired 3D topological fold is a fundamental yet non-trivial task in protein engineering.
Large-Scale Chemical Language Representations Capture Molecular Structure and Properties
Models based on machine learning can enable accurate and fast molecular property predictions, which is of interest in drug discovery and material design.
Predicting Deep Neural Network Generalization with Perturbation Response Curves
However, despite these successes, the recent Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020 competition suggests that there is a need for more robust and efficient measures of network generalization.
Augmenting Molecular Deep Generative Models with Topological Data Analysis Representations
Deep generative models have emerged as a powerful tool for learning useful molecular representations and designing novel molecules with desired properties, with applications in drug discovery and material design.
Gi and Pal Scores: Deep Neural Network Generalization Statistics
The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of regression, classification, and control tasks.
Towards creativity characterization of generative models via group-based subset scanning
We propose group-based subset scanning to quantify, detect, and characterize creative processes by detecting a subset of anomalous node-activations in the hidden layers of generative models.
ProGAE: A Geometric Autoencoder-based Generative Model for Disentangling Protein Dynamics
Empowered by the disentangled latent space learning, the extrinsic latent embedding is successfully used for classification or property prediction of different drugs bound to a specific protein.
Self-Progressing Robust Training
Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy machine learning systems.
Reprogramming Language Models for Molecular Representation Learning
Recent advancements in transfer learning have made it a promising approach for domain adaptation via transfer of learned representations.
Optimizing Molecules using Efficient Queries from Property Evaluations
Machine learning based methods have shown potential for optimizing existing molecules with more desirable properties, a critical step towards accelerating new chemical discovery.
DualTKB: A Dual Learning Bridge between Text and Knowledge Base
In this work, we present a dual learning approach for unsupervised text to path and path to text transfers in Commonsense Knowledge Bases (KBs).
Characterizing the Latent Space of Molecular Deep Generative Models with Persistent Homology Metrics
In this work, we propose a method for measuring how well the latent space of deep generative models is able to encode structural and chemical features of molecular datasets by correlating latent space metrics with metrics from the field of topological data analysis (TDA).
Explaining Chemical Toxicity using Missing Features
Chemical toxicity prediction using machine learning is important in drug development to reduce repeated animal and human testing, thus saving cost and time.
Optimizing Mode Connectivity via Neuron Alignment
Yet, current curve finding algorithms do not consider the influence of symmetry in the loss surface created by model weight permutations.
Active learning of deep surrogates for PDEs: Application to metasurface design
Surrogate models for partial-differential equations are widely used in the design of meta-materials to rapidly evaluate the behavior of composable components.
Combinatorial Black-Box Optimization with Expert Advice
We consider the problem of black-box function optimization over the boolean hypercube.
Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics
De novo therapeutic design is challenged by a vast chemical repertoire and multiple constraints, e. g., high broad-spectrum potency and low toxicity.
Learning Implicit Text Generation via Feature Matching
Generative feature matching network (GFMN) is an approach for training implicit generative models for images by performing moment matching on features from pre-trained neural networks.
Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness.
CogMol: Target-Specific and Selective Drug Design for COVID-19 Using Deep Generative Models
CogMol also includes insilico screening for assessing toxicity of parent molecules and their metabolites with a multi-task toxicity classifier, synthetic feasibility with a chemical retrosynthesis predictor, and target structure binding with docking simulations.
Nano-Material Configuration Design with Deep Surrogate Langevin Dynamics
We consider the problem of optimizing by sampling under multiple black-box constraints in nano-material design.
Improving Efficiency in Large-Scale Decentralized Distributed Training
Decentralized Parallel SGD (D-PSGD) and its asynchronous variant Asynchronous Parallel SGD (AD-PSGD) is a family of distributed learning algorithms that have been demonstrated to perform well for large-scale deep learning tasks.
Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $O\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.
A Decentralized Parallel Algorithm for Training Generative Adversarial Nets
Despite recent progress on decentralized algorithms for training deep neural networks, it remains unclear whether it is possible to train GANs in a decentralized manner.
SPROUT: Self-Progressing Robust Training
Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy and reliable machine learning systems.
Surrogate-Based Constrained Langevin Sampling With Applications to Optimal Material Configuration Design
We consider the problem of generating configurations that satisfy physical constraints for optimal material nano-pattern design, where multiple (and often conflicting) properties need to be simultaneously satisfied.
Optimizing Loss Landscape Connectivity via Neuron Alignment
Empirically, this initialization is critical for efficiently learning a simple, planar, low-loss curve between networks that successfully generalizes.
Interactive Visual Exploration of Latent Space (IVELS) for peptide auto-encoder model selection
We present the pipeline in an interactive visual tool to enable the exploration of the metrics, analysis of the learned latent space, and selection of the best model for a given task.
Toward A Neuro-inspired Creative Decoder
Creativity, a process that generates novel and meaningful ideas, involves increased association between task-positive (control) and task-negative (default) networks in the human brain.
PepCVAE: Semi-Supervised Targeted Design of Antimicrobial Peptide Sequences
Our model learns a rich latent space of the biological peptide context by taking advantage of abundant, unlabeled peptide sequences.
Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
important object pixels in an image) to justify its classification and analogously what should be minimally and necessarily \emph{absent} (viz.
Autism Classification Using Brain Functional Connectivity Dynamics and Machine Learning
The goal of the present study is to identify autism using machine learning techniques and resting-state brain imaging data, leveraging the temporal variability of the functional connections (FC) as the only information.
Neurology-as-a-Service for the Developing World
In this work, we present a cloud-based deep neural network approach to provide decision support for non-specialist physicians in EEG analysis and interpretation.
