Search Results for author:
Found 27 papers, 15 papers with code
Towards Generalist Prompting for Large Language Models by Mental Models
MeMo distills the cores of various prompting methods into individual mental models and allows LLMs to autonomously select the most suitable mental models for the problem, achieving or being near to the state-of-the-art results on diverse tasks such as STEM, logical reasoning, and commonsense reasoning in zero-shot settings.
Control Risk for Potential Misuse of Artificial Intelligence in Science
In this study, we aim to raise awareness of the dangers of AI misuse in science, and call for responsible AI development and use in this domain.
Inverse Design of Vitrimeric Polymers by Molecular Dynamics and Generative Modeling
Vitrimer is a new class of sustainable polymers with the ability of self-healing through rearrangement of dynamic covalent adaptive networks.
Overcoming the Barrier of Orbital-Free Density Functional Theory for Molecular Systems Using Deep Learning
Orbital-free density functional theory (OFDFT) is a quantum chemistry formulation that has a lower cost scaling than the prevailing Kohn-Sham DFT, which is increasingly desired for contemporary molecular research.
Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learning
In this paper, we introduce a novel deep learning framework, called Distributional Graphormer (DiG), in an attempt to predict the equilibrium distribution of molecular systems.
Invertible Rescaling Network and Its Extensions
To be specific, we develop invertible models to generate valid degraded images and meanwhile transform the distribution of lost contents to the fixed distribution of a latent variable during the forward degradation.
One Transformer Can Understand Both 2D & 3D Molecular Data
To achieve this goal, in this work, we develop a novel Transformer-based Molecular model called Transformer-M, which can take molecular data of 2D or 3D formats as input and generate meaningful semantic representations.
Ranked #4 on
Graph Regression
on PCQM4Mv2-LSC
From Static to Dynamic Structures: Improving Binding Affinity Prediction with a Graph-Based Deep Learning Model
Accurate prediction of the protein-ligand binding affinities is an essential challenge in the structure-based drug design.
Quantized Training of Gradient Boosting Decision Trees
Recent years have witnessed significant success in Gradient Boosting Decision Trees (GBDT) for a wide range of machine learning applications.
Your Transformer May Not be as Powerful as You Expect
Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications.
Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation.
An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation.
Do Transformers Really Perform Badly for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track.
Do Transformers Really Perform Bad for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
How could Neural Networks understand Programs?
Inspired by this, we propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition, which is indispensable for program understanding.
Revisiting Language Encoding in Learning Multilingual Representations
The language embedding can be either added to the word embedding or attached at the beginning of the sentence.
Modeling Lost Information in Lossy Image Compression
Specifically, ILC introduces an invertible encoding module to replace the encoder-decoder structure to produce the low dimensional informative latent representation, meanwhile, transform the lost information into an auxiliary latent variable that won't be further coded or stored.
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Pre-trained contextual representations (e. g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks.
Invertible Image Rescaling
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images.
Cross-Iteration Batch Normalization
We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied.
Ranked #180 on
Object Detection
on COCO test-dev
On Layer Normalization in the Transformer Architecture
This motivates us to remove the warm-up stage for the training of Pre-LN Transformers.
G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Capacity Control of ReLU Neural Networks by Basis-path Norm
Motivated by this, we propose a new norm \emph{Basis-path Norm} based on a group of linearly independent paths to measure the capacity of neural networks more accurately.
$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Asynchronous Stochastic Gradient Descent with Delay Compensation
We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD.
