Search Results for author:
Found 36 papers, 17 papers with code
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size.
Quantized Training of Gradient Boosting Decision Trees
Recent years have witnessed significant success in Gradient Boosting Decision Trees (GBDT) for a wide range of machine learning applications.
Do Transformers Really Perform Bad for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track.
Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation.
Deep Subdomain Adaptation Network for Image Classification
The adaptation can be achieved easily with most feed-forward network models by extending them with LMMD loss, which can be trained efficiently via back-propagation.
Invertible Image Rescaling
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images.
Uni-Mol: A Universal 3D Molecular Representation Learning Framework
Uni-Mol is composed of two models with the same SE(3)-equivariant transformer architecture: a molecular pretraining model trained by 209M molecular conformations; a pocket pretraining model trained by 3M candidate protein pocket data.
Ranked #1 on
Molecular Property Prediction
on MUV
Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+
Uni-Mol+ first generates a raw 3D molecule conformation from inexpensive methods such as RDKit.
Rethinking Positional Encoding in Language Pre-training
In this work, we investigate the positional encoding methods used in language pre-training (e. g., BERT) and identify several problems in the existing formulations.
How could Neural Networks understand Programs?
Inspired by this, we propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition, which is indispensable for program understanding.
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Pre-trained contextual representations (e. g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks.
Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space.
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space.
Revisiting Language Encoding in Learning Multilingual Representations
The language embedding can be either added to the word embedding or attached at the beginning of the sentence.
Light Multi-segment Activation for Model Compression
Inspired by the nature of the expressiveness ability in Neural Networks, we propose to use multi-segment activation, which can significantly improve the expressiveness ability with very little cost, in the compact student model.
Node-Aligned Graph-to-Graph (NAG2G): Elevating Template-Free Deep Learning Approaches in Single-Step Retrosynthesis
Single-step retrosynthesis (SSR) in organic chemistry is increasingly benefiting from deep learning (DL) techniques in computer-aided synthesis design.
A Communication-Efficient Parallel Algorithm for Decision Tree
After partitioning the training data onto a number of (e. g., $M$) machines, this algorithm performs both local voting and global voting in each iteration.
TabNN: A Universal Neural Network Solution for Tabular Data
Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all.
LightMC: A Dynamic and Efficient Multiclass Decomposition Algorithm
Instead of using fixed coding matrix and decoding strategy, LightMC uses a differentiable decoding strategy, which enables it to dynamically optimize the coding matrix and decoding strategy, toward increasing the overall accuracy of multiclass classification, via back propagation jointly with the training of base learners in an iterative way.
Taking Notes on the Fly Helps BERT Pre-training
In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization.
Taking Notes on the Fly Helps Language Pre-Training
In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization.
LazyFormer: Self Attention with Lazy Update
In each lazy block, the self-attention distribution is only computed once in the first layer and then is reused in all upper layers.
Transformers with Competitive Ensembles of Independent Mechanisms
In this work we explore a way in which the Transformer architecture is deficient: it represents each position with a large monolithic hidden representation and a single set of parameters which are applied over the entire hidden representation.
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
Do Transformers Really Perform Badly for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Boosted ab initio Cryo-EM 3D Reconstruction with ACE-EM
The central problem in cryo-electron microscopy (cryo-EM) is to recover the 3D structure from noisy 2D projection images which requires estimating the missing projection angles (poses).
3D Molecular Generation via Virtual Dynamics
Extensive experiment results on pocket-based molecular generation demonstrate that VD-Gen can generate novel 3D molecules to fill the target pocket cavity with high binding affinities, significantly outperforming previous baselines.
Do Deep Learning Methods Really Perform Better in Molecular Conformation Generation?
However, to our surprise, we design a simple and cheap algorithm (parameter-free) based on the traditional methods and find it is comparable to or even outperforms deep learning based MCG methods in the widely used GEOM-QM9 and GEOM-Drugs benchmarks.
Do Deep Learning Models Really Outperform Traditional Approaches in Molecular Docking?
What's more, they claim to perform better than traditional molecular docking, but the approach of comparison is not fair, since traditional methods are not designed for docking on the whole protein without a given pocket.
Uni-QSAR: an Auto-ML Tool for Molecular Property Prediction
Recently deep learning based quantitative structure-activity relationship (QSAR) models has shown surpassing performance than traditional methods for property prediction tasks in drug discovery.
End-to-End Crystal Structure Prediction from Powder X-Ray Diffraction
XtalNet represents a significant advance in CSP, enabling the prediction of complex structures from PXRD data without the need for external databases or manual intervention.
SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis
Recent breakthroughs in Large Language Models (LLMs) have revolutionized natural language understanding and generation, igniting a surge of interest in leveraging these technologies in the field of scientific literature analysis.
Uni-SMART: Universal Science Multimodal Analysis and Research Transformer
Scientific literature often includes a wide range of multimodal elements, such as molecular structure, tables, and charts, which are hard for text-focused LLMs to understand and analyze.
