Search Results for author:
Found 92 papers, 44 papers with code
Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics
Quantitatively measuring the impact-related aspects of scientific, engineering, and technological (SET) innovations is a fundamental problem with broad applications.
Long-context Protein Language Model
Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design.
Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens
Our first approach is Batch-Scheduled Sampling, where, during training, we stochastically choose between the ground-truth token from the dataset and the model's own generated token as input to predict the next token.
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification
In this paper, we propose a multi-task learning (MTL) framework that jointly fine-tunes PLMs on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals.
AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
On the other hand, there has been limited study on the misalignment between a web agent's observation/action representation and the pre-training data of the LLM it's based on.
Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling.
Inference Optimization of Foundation Models on AI Accelerators
Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI across various industries.
Learning to Generate Answers with Citations via Factual Consistency Models
Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations.
Hierarchical Compression of Text-Rich Graphs via Large Language Models
Large Language Models (LLMs), noted for their superior text understanding abilities, offer a solution for processing the text in graphs but face integration challenges due to their limitation for encoding graph structures and their computational complexities when dealing with extensive text in large neighborhoods of interconnected nodes.
GraphStorm: all-in-one graph machine learning framework for industry applications
GraphStorm has the following desirable properties: (a) Easy to use: it can perform graph construction and model training and inference with just a single command; (b) Expert-friendly: GraphStorm contains many advanced GML modeling techniques to handle complex graph data and improve model performance; (c) Scalable: every component in GraphStorm can operate on graphs with billions of nodes and can scale model training and inference to different hardware without changing any code.
GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning
In our GNN-RAG framework, the GNN acts as a dense subgraph reasoner to extract useful graph information, while the LLM leverages its natural language processing ability for ultimate KGQA.
Parameter-Efficient Tuning Large Language Models for Graph Representation Learning
Inspired by this, we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning with LLMs on text-rich graphs.
AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models
AutoGluon-Multimodal (AutoMM) is introduced as an open-source AutoML library designed specifically for multimodal learning.
Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization
PackLLM performs model fusion by solving an optimization problem for determining each LLM's importance, so that perplexity over the input prompt is minimized.
Pre-training Differentially Private Models with Limited Public Data
The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection.
Extreme Miscalibration and the Illusion of Adversarial Robustness
Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify.
OpenTab: Advancing Large Language Models as Open-domain Table Reasoners
Large Language Models (LLMs) trained on large volumes of data excel at various natural language tasks, but they cannot handle tasks requiring knowledge that has not been trained on previously.
SemPool: Simple, robust, and interpretable KG pooling for enhancing language models
Our method, termed SemPool, represents KG facts with pre-trained LMs, learns to aggregate their semantic information, and fuses it at different layers of the LM.
Zero redundancy distributed learning with differential privacy
Deep learning using large models have achieved great success in a wide range of domains.
Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection
Large Language Models (LLMs) can adapt to new tasks via in-context learning (ICL).
On the accuracy and efficiency of group-wise clipping in differentially private optimization
Recent advances have substantially improved the accuracy, memory cost, and training speed of differentially private (DP) deep learning, especially on large vision and language models with millions to billions of parameters.
Extending Input Contexts of Language Models through Training on Segmented Sequences
Our method benefits both models trained with absolute positional embeddings, by extending their input contexts, as well as popular relative positional embedding methods showing a reduced perplexity on sequences longer than they were trained on.
NameGuess: Column Name Expansion for Tabular Data
Recent advances in large language models have revolutionized many sectors, including the database industry.
Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space
Recent advances in tabular data generation have greatly enhanced synthetic data quality.
Coupling public and private gradient provably helps optimization
The success of large neural networks is crucially determined by the availability of data.
HYTREL: Hypergraph-enhanced Tabular Data Representation Learning
Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks.
DistTGL: Distributed Memory-Based Temporal Graph Neural Network Training
Evenworse, the tremendous overhead to synchronize the node memory make it impractical to be deployed to distributed GPU clusters.
Large Language Models of Code Fail at Completing Code with Potential Bugs
We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs.
Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion
To better solve this task, we query a program analyzer for information relevant to a given function call, and consider ways to provide the analyzer results to different code completion models during inference and training.
XTab: Cross-table Pretraining for Tabular Transformers
The success of self-supervised learning in computer vision and natural language processing has motivated pretraining methods on tabular data.
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs
Different from conventional knowledge distillation, GRAD jointly optimizes a GNN teacher and a graph-free student over the graph's nodes via a shared LM.
STREET: A Multi-Task Structured Reasoning and Explanation Benchmark
We introduce STREET, a unified multi-task and multi-domain natural language reasoning and explanation benchmark.
Multimodal Chain-of-Thought Reasoning in Language Models
Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach.
Ranked #4 on
Science Question Answering
on ScienceQA
OrthoReg: Improving Graph-regularized MLPs via Orthogonality Regularization
Graph Neural Networks (GNNs) are currently dominating in modeling graph-structure data, while their high reliance on graph structure for inference significantly impedes them from widespread applications.
SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost.
ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs
Our method, termed ReaRev, introduces a new way to KGQA reasoning with respect to both instruction decoding and execution.
Ranked #1 on
Semantic Parsing
on WebQuestionsSP
Injecting Domain Knowledge from Empirical Interatomic Potentials to Neural Networks for Predicting Material Properties
In recent years, neural network (NN)-based potentials trained on quantum mechanical (DFT-labeled) data have emerged as a more accurate alternative to conventional EIPs.
Predicting Cellular Responses with Variational Causal Inference and Refined Relational Information
Predicting the responses of a cell under perturbations may bring important benefits to drug discovery and personalized therapeutics.
Differentially Private Optimization on Large Model at Small Cost
Our implementation achieves state-of-the-art (SOTA) accuracy with very small extra cost: on GPT2 and at almost the same memory cost (<1% overhead), BK has 1. 03X the time complexity of the standard training (0. 83X training speed in practice), and 0. 61X the time complexity of the most efficient DP implementation (1. 36X training speed in practice).
Differentially Private Bias-Term Fine-tuning of Foundation Models
We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data.
Variational Causal Inference
Estimating an individual's potential outcomes under counterfactual treatments is a challenging task for traditional causal inference and supervised learning approaches when the outcome is high-dimensional (e. g. gene expressions, impulse responses, human faces) and covariates are relatively limited.
Efficient and effective training of language and graph neural network models
The effectiveness in our framework is achieved by applying stage-wise fine-tuning of the BERT model first with heterogenous graph information and then with a GNN model.
Nimble GNN Embedding with Tensor-Train Decomposition
This paper describes a new method for representing embedding tables of graph neural networks (GNNs) more compactly via tensor-train (TT) decomposition.
ScatterSample: Diversified Label Sampling for Data Efficient Graph Neural Network Learning
ScatterSample employs a sampling module termed DiverseUncertainty to collect instances with large uncertainty from different regions of the sample space for labeling.
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning
Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks.
Coarse-to-Fine Sparse Sequential Recommendation
We propose to model user dynamics from shopping intents and interacted items simultaneously.
TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphs
Our temporal parallel sampler achieves an average of 173x speedup on a multi-core CPU compared with the baselines.
Learning Personalized Item-to-Item Recommendation Metric via Implicit Feedback
We develop and investigate a personalizable deep metric model that captures both the internal contents of items and how they were interacted with by users.
Joint Learning of Hierarchical Community Structure and Node Representations: An Unsupervised Approach
To account for the structure in the node representations, Mazi generates node representations at each level of the hierarchy, and utilizes them to influence the node representations of the original graph.
TempoQR: Temporal Question Reasoning over Knowledge Graphs
The first computes a textual representation of a given question, the second combines it with the entity embeddings for entities involved in the question, and the third generates question-specific time embeddings.
Ranked #1 on
Question Answering
on CronQuestions
Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces
In this work, we benchmark a variety of single- and multi-task graph neural network (GNN) models against lower-bar and higher-bar traditional machine learning approaches employing human engineered molecular features.
Meta-learning via Language Model In-context Tuning
The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples.
Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
Consistency training is a popular method to improve deep learning models in computer vision and natural language processing.
Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
We aim to identify how different components in the KD pipeline affect the resulting performance and how much the optimal KD pipeline varies across different datasets/tasks, such as the data augmentation policy, the loss function, and the intermediate representation for transferring the knowledge between teacher and student.
HeMI: Multi-view Embedding in Heterogeneous Graphs
Many real-world graphs involve different types of nodes and relations between nodes, being heterogeneous by nature.
FaiREO: User Group Fairness for Equality of Opportunity in Course Recommendation
Course selection is challenging for students in higher educational institutions.
Trust your neighbors: A comprehensive survey of neighborhood-based methods for recommender systems
Collaborative recommendation approaches based on nearest-neighbors are still highly popular today due to their simplicity, their efficiency, and their ability to produce accurate and personalized recommendations.
Position-based Hash Embeddings For Scaling Graph Neural Networks
GNNs compute node representations by taking into account the topology of the node's ego-network and the features of the ego-network's nodes.
TraverseNet: Unifying Space and Time in Message Passing for Traffic Forecasting
This paper aims to unify spatial dependency and temporal dependency in a non-Euclidean space while capturing the inner spatial-temporal dependencies for traffic data.
DGL-LifeSci: An Open-Source Toolkit for Deep Learning on Graphs in Life Science
Graph neural networks (GNNs) constitute a class of deep learning methods for graph data.
Schema-Aware Deep Graph Convolutional Networks for Heterogeneous Graphs
To address this question, we propose our GCN framework 'Deep Heterogeneous Graph Convolutional Network (DHGCN)', which takes advantage of the schema of a heterogeneous graph and uses a hierarchical approach to effectively utilize information many hops away.
IACN: Influence-aware and Attention-based Co-evolutionary Network for Recommendation
The interaction modeling layer is responsible for updating the embedding of a user and an item when the user interacts with the item.
Universal Representation for Code
Learning from source code usually requires a large amount of labeled data.
Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning
Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair, which can capture the relationship between each node and its neighboring substructure in an unsupervised way.
Learning Student Interest Trajectory for MOOC Thread Recommendation
The projection operation learns to estimate future embedding of students and threads.
Learning over Families of Sets -- Hypergraph Representation Learning for Higher Order Tasks
In this work, we exploit the incidence structure to develop a hypergraph neural network to learn provably expressive representations of variable sized hyperedges which preserve local-isomorphism in the line graph of the hypergraph, while also being invariant to permutations of its constituent vertices.
An Empirical Comparison of Deep Learning Models for Knowledge Tracing on Large-Scale Dataset
Recent release of large-scale student performance dataset \cite{choi2019ednet} motivates the analysis of performance of deep learning approaches that have been proposed to solve KT.
Learning Student Interest Trajectory for MOOCThread Recommendation
The projection operation learns to estimate future embedding of students and threads.
Distant-Supervised Slot-Filling for E-Commerce Queries
Slot-filling refers to the task of annotating individual terms in a query with the corresponding intended product characteristics (product type, brand, gender, size, color, etc.).
DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs
To minimize the overheads associated with distributed computations, DistDGL uses a high-quality and light-weight min-cut graph partitioning algorithm along with multiple balancing constraints.
PanRep: Universal node embeddings for heterogeneous graphs
Learning unsupervised node embeddings facilitates several downstream tasks such as node classification and link prediction.
Heterogeneous Molecular Graph Neural Networks for Predicting Molecule Properties
As they carry great potential for modeling complex interactions, graph neural network (GNN)-based methods have been widely used to predict quantum mechanical properties of molecules.
Ranked #5 on
Formation Energy
on QM9
Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning
Motivated by this observation, we propose a graph representation learning method called Graph InfoClust (GIC), that seeks to additionally capture cluster-level information content.
Ranked #2 on
Link Prediction
on Citeseer
COVID-19 Knowledge Graph: Accelerating Information Retrieval and Discovery for Scientific Literature
Finally, we propose a document similarity engine that leverages low-dimensional graph embeddings from the CKG with semantic embeddings for similar article retrieval.
Few-shot link prediction via graph neural networks for Covid-19 drug-repurposing
This paper proposes an inductive RGCN for learning informative relation embeddings even in the few-shot learning regime.
PanRep: Graph neural networks for extracting universal node embeddings in heterogeneous graphs
Learning unsupervised node embeddings facilitates several downstream tasks such as node classification and link prediction.
Repurpose Open Data to Discover Therapeutics for COVID-19 using Deep Learning
While this study, by no means recommends specific drugs, it demonstrates a powerful deep learning methodology to prioritize existing drugs for further investigation, which holds the potential of accelerating therapeutic development for COVID-19.
DGL-KE: Training Knowledge Graph Embeddings at Scale
Experiments on knowledge graphs consisting of over 86M nodes and 338M edges show that DGL-KE can compute embeddings in 100 minutes on an EC2 instance with 8 GPUs and 30 minutes on an EC2 cluster with 4 machines with 48 cores/machine.
Distributed, Parallel, and Cluster Computing
Context-aware Non-linear and Neural Attentive Knowledge-based Models for Grade Prediction
Grade prediction for future courses not yet taken by students is important as it can help them and their advisers during the process of course selection as well as for designing personalized degree plans and modifying them based on their performance.
CAWA: An Attention-Network for Credit Attribution
Experiments on the credit attribution task on a variety of datasets show that the sentence class labels generated by CAWA outperform the competing approaches.
Boosting Item-based Collaborative Filtering via Nearly Uncoupled Random Walks
Item-based models are among the most popular collaborative filtering approaches for building recommender systems.
Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
Advancing research in the emerging field of deep graph learning requires new tools to support tensor computation over graphs.
Ranked #34 on
Node Classification
on Cora
Intent term selection and refinement in e-commerce queries
Moreover, for the tasks of identifying the important terms in a query and for predicting the additional terms that represent product intent, experiments illustrate that our approaches outperform the non-contextual baselines.
A Self-Attentive model for Knowledge Tracing
Knowledge tracing is the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities.
Will this Course Increase or Decrease Your GPA? Towards Grade-aware Course Recommendation
To this end, we propose two different grade-aware course recommendation approaches to recommend to each student his/her optimal set of courses.
Learning from Sets of Items in Recommender Systems
Our analysis of these ratings shows that though the majority of the users provide the average of the ratings on a set's constituent items as the rating on the set, there exists a significant number of users that tend to consistently either under- or over-rate the sets.
Sparse Neural Attentive Knowledge-based Models for Grade Prediction
Grade prediction for future courses not yet taken by students is important as it can help them and their advisers during the process of course selection as well as for designing personalized degree plans and modifying them based on their performance.
Feature-based factorized Bilinear Similarity Model for Cold-Start Top-n Item Recommendation
The user personalized non-collaborative methods based on item features can be used to address this item cold-start problem.
Adaptive Matrix Completion for the Users and the Items in Tail
In this work, we show that the skewed distribution of ratings in the user-item rating matrix of real-world datasets affects the accuracy of matrix-completion-based approaches.
Distributed representation of multi-sense words: A loss-driven approach
Word2Vec's Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words.
Text segmentation on multilabel documents: A distant-supervised approach
Segmenting text into semantically coherent segments is an important task with applications in information retrieval and text summarization.
Scalable Label Propagation for Multi-relational Learning on the Tensor Product of Graphs
In this paper, we generalize a widely used label propagation model to the normalized tensor product graph, and propose an optimization formulation and a scalable Low-rank Tensor-based Label Propagation algorithm (LowrankTLP) to infer multi-relations for two learning tasks, hyperlink prediction and multiple graph alignment.
