Search Results for author:
Found 113 papers, 42 papers with code
SlimPajama-DC: Understanding Data Combinations for LLM Training
This paper aims to understand the impacts of various data combinations (e. g., web text, wikipedia, github, books) on the training of large language models using SlimPajama.
Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
We release two open versions of the model -- the foundation Jais model, and an instruction-tuned Jais-chat variant -- with the aim of promoting research on Arabic LLMs.
Defending Against Malicious Behaviors in Federated Learning with Blockchain
In the era of deep learning, federated learning (FL) presents a promising approach that allows multi-institutional data owners, or clients, to collaboratively train machine learning models without compromising data privacy.
Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective
We present a new dataset condensation framework termed Squeeze, Recover and Relabel (SRe$^2$L) that decouples the bilevel optimization of model and synthetic data during training, to handle varying scales of datasets, model architectures and image resolutions for effective dataset condensation.
Identification of Nonlinear Latent Hierarchical Models
In this work, we investigate the identification problem for nonlinear latent hierarchical causal models in which observed variables are generated by a set of causally related latent variables, and some latent variables may not have observed children.
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks.
Weakly Supervised 3D Open-vocabulary Segmentation
Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research.
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
We propose DSR-LM, a Differentiable Symbolic Reasoning framework where pre-trained LMs govern the perception of factual knowledge, and a symbolic module performs deductive reasoning.
3D Semantic Segmentation in the Wild: Learning Generalized Models for Adverse-Condition Point Clouds
In addition, we design a domain randomization technique that alternatively randomizes the geometry styles of point clouds and aggregates their embeddings, ultimately leading to a generalizable model that can improve 3DSS under various adverse weather effectively.
KD-DLGAN: Data Limited Image Generation via Knowledge Distillation
The first is aggregated generative KD that mitigates the discriminator overfitting by challenging the discriminator with harder learning tasks and distilling more generalizable knowledge from the pre-trained models.
StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields
In addition, it transforms the grid features according to the reference style which directly leads to high-quality zero-shot style transfer.
Memory-adaptive Depth-wise Heterogenous Federated Learning
Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data.
The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning
Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations (or ``chain-of-thought'' (CoT)) for in-context learning.
MixMask: Revisiting Masking Strategy for Siamese ConvNets
These include (I) the inability to drop uninformative masked regions in ConvNets as they process data continuously, resulting in low training efficiency compared to ViT models; and (II) the mismatch between erase-based masking and the contrastive-based objective in Siamese ConvNets, which differs from the MIM approach.
AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness
Scaling up model sizes can lead to fundamentally new capabilities in many machine learning (ML) tasks.
Betty: An Automatic Differentiation Library for Multilevel Optimization
Gradient-based multilevel optimization (MLO) has gained attention as a framework for studying numerous problems, ranging from hyperparameter optimization and meta-learning to neural architecture search and reinforcement learning.
SDQ: Stochastic Differentiable Quantization with Mixed Precision
In order to deploy deep models in a computationally efficient manner, model quantization approaches have been frequently used.
Rare Gems: Finding Lottery Tickets at Initialization
Frankle & Carbin conjecture that we can avoid this by training "lottery tickets", i. e., special sparse subnetworks found at initialization, that can be trained to high accuracy.
Stochastic Neural Networks with Infinite Width are Deterministic
This work theoretically studies stochastic neural networks, a main type of neural network in use.
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
This paper explores the feasibility of finding an optimal sub-model from a vision transformer and introduces a pure vision transformer slimming (ViT-Slim) framework.
Multimodal Image Synthesis and Editing: The Generative AI Era
With superb power in modeling the interaction among multimodal information, multimodal image synthesis and editing has become a hot research topic in recent years.
Data-Free Neural Architecture Search via Recursive Label Calibration
We identify that the NAS task requires the synthesized data (we target at image domain here) with enough semantics, diversity, and a minimal domain gap from the natural images.
A Fast Knowledge Distillation Framework for Visual Recognition
In this study, we present a Fast Knowledge Distillation (FKD) framework that replicates the distillation training phase and generates soft labels using the multi-crop KD approach, while training faster than ReLabel since no post-processes such as RoI align and softmax operations are used.
Ranked #505 on
Image Classification
on ImageNet
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i. e., uniform strategy, due to its superior representational capacity.
Learning from Mistakes -- A Framework for Neural Architecture Search
We propose a novel machine learning method called Learning From Mistakes (LFM), wherein the learner improves its ability to learn by focusing more on the mistakes during revision.
Sliced Recursive Transformer
The proposed weight sharing mechanism by sliced recursion structure allows us to build a transformer with more than 100 or even 1000 shared layers with ease while keeping a compact size (13~15M), to avoid optimization difficulties when the model is too large.
Ranked #249 on
Image Classification
on ImageNet
Tradeoffs of Linear Mixed Models in Genome-wide Association Studies
Motivated by empirical arguments that are well-known from the genome-wide association studies (GWAS) literature, we study the statistical properties of linear mixed models (LMMs) applied to GWAS.
Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features
Machine learning has demonstrated remarkable prediction accuracy over i. i. d data, but the accuracy often drops when tested with data from another distribution.
Multi-task Learning of Order-Consistent Causal Graphs
We consider the problem of discovering $K$ related Gaussian directed acyclic graphs (DAGs), where the involved graph structures share a consistent causal order and sparse unions of supports.
Text Generation with Efficient (Soft) $Q$-Learning
We apply the approach to a wide range of text generation tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
Multi-modal Self-supervised Pre-training for Large-scale Genome Data
In this work, we propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT.
Towards Visual Question Answering on Pathology Images
In this paper, we aim to develop a pathological visual question answering framework to analyze pathology images and answer medical questions related to these images.
On the Generation of Medical Dialogs for COVID-19
Training complex dialog generation models on small datasets bears high risk of overfitting.
Prototypical Graph Contrastive Learning
However, since for a query, its negatives are uniformly sampled from all graphs, existing methods suffer from the critical sampling bias issue, i. e., the negatives likely having the same semantic structure with the query, leading to performance degradation.
Negational Symmetry of Quantum Neural Networks for Binary Pattern Classification
In this work, we provide some theoretical insight into the properties of QNNs by presenting and analyzing a new form of invariance embedded in QNNs for both quantum binary classification and quantum representation learning, which we term negational symmetry.
On Data Efficiency of Meta-learning
Meta-learning has enabled learning statistical models that can be quickly adapted to new prediction tasks.
Learning Robust Models by Countering Spurious Correlations
In this paper, we formally study the generalization error bound for this setup with the knowledge of how the spurious features are associated with the label.
On the Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations
Data augmentation is one of the most popular techniques for improving the robustness of neural networks.
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling
Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels.
AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning
Synchronization is a key step in data-parallel distributed machine learning (ML).
Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
Federated learning is typically approached as an optimization problem, where the goal is to minimize a global loss function by distributing computation across client devices that possess local data and specify different parts of the global objective.
Pathological Visual Question Answering
To deal with the issue that a publicly available pathology VQA dataset is lacking, we create PathVQA dataset.
On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks
Conversely, the probability of an interaction of $k$ variables surviving Dropout at rate $p$ is $\mathcal{O}((1-p)^k)$.
XRayGAN: Consistency-preserving Generation of X-ray Images from Radiology Reports
To address this problem, we develop methods to generate view-consistent, high-fidelity, and high-resolution X-ray images from radiology reports to facilitate radiology training of medical students.
On the Generation of Medical Dialogues for COVID-19
On these two datasets, we train several dialogue generation models based on Transformer, GPT, and BERT-GPT.
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports
In this work, we propose a novel framework that exploits the structure information between and within report sections for generating CXR imaging reports.
Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT Scans
Besides, these works require a large number of CTs to train accurate diagnosis models, which are difficult to obtain.
Learning from Imperfect Annotations
Many machine learning systems today are trained on large amounts of human-annotated data.
Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning
This drawback hinders the model from learning subtle variance and fine-grained information.
PathVQA: 30000+ Questions for Medical Visual Question Answering
To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer.
Methods for comparing uncertainty quantifications for material property predictions
These uncertainty estimates are instrumental for determining which materials to screen next, but there is not yet a standard procedure for judging the quality of such uncertainty estimates objectively.
Materials Science Computational Physics
Generalized Zero-shot ICD Coding
The International Classification of Diseases (ICD) is a list of classification codes for the diagnoses.
Learning from Imperfect Annotations: An End-to-End Approach
Many machine learning systems today are trained on large amounts of human-annotated data.
Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach
Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system.
Efficient Exploration via State Marginal Matching
The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective.
Regularizing Black-box Models for Improved Interpretability (HILL 2019 Version)
Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, which lack guarantees about their explanation quality.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Explaining a black-box using Deep Variational Information Bottleneck Approach
Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system.
Regularizing Black-box Models for Improved Interpretability
Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, whose explanation quality can be unpredictable.
ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language
Optimizing an expensive-to-query function is a common task in science and engineering, where it is beneficial to keep the number of queries to a minimum.
Data-to-Text Generation with Style Imitation
That is, the model learns to imitate the writing style of any given exemplar sentence, with automatic adaptions to faithfully describe the content record.
Text Infilling
Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting.
Connecting the Dots Between MLE and RL for Sequence Prediction
Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency.
Unsupervised Pseudo-Labeling for Extractive Summarization on Electronic Health Records
Extractive summarization is very useful for physicians to better manage and digest Electronic Health Records (EHRs).
On the Complexity of Exploration in Goal-Driven Navigation
Next, we propose to measure the complexity of each environment by constructing dependency graphs between the goals and analytically computing \emph{hitting times} of a random walk in the graph.
Multimodal Machine Learning for Automated ICD Coding
This study presents a multimodal machine learning model to predict ICD-10 diagnostic codes.
AutoLoss: Learning Discrete Schedules for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
AutoLoss: Learning Discrete Schedule for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
Missing Value Imputation Based on Deep Generative Models
In this paper, we propose a probabilistic framework based on deep generative models for MVI.
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
To our knowledge, this is the first successful case of the learned driving policy through reinforcement learning in the high-fidelity simulator, which performs better-than supervised imitation learning.
Geometric Generalization Based Zero-Shot Learning Dataset Infinite World: Simple Yet Powerful
Analogously, this paper introduces geometric generalization based zero-shot learning tests to measure the rapid learning ability and the internal consistency of deep generative models.
DiCE: The Infinitely Differentiable Monte Carlo Estimator
Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for estimators of higher-order derivatives.
Texar: A Modularized, Versatile, and Extensible Toolbox for Text Generation
The features make Texar particularly suitable for technique sharing and generalization across different text generation applications.
Nonoverlap-Promoting Variable Selection
Variable selection is a classic problem in machine learning (ML), widely used to find important explanatory factors, and improve generalization performance and interpretability of ML models.
A Neural Architecture for Automated ICD Coding
The International Classification of Diseases (ICD) provides a hierarchy of diagnostic codes for classifying diseases.
Deep Generative Models with Learnable Knowledge Constraints
The broad set of deep generative models (DGMs) has achieved remarkable advances.
Gated Path Planning Networks
Value Iteration Networks (VINs) are effective differentiable path planning modules that can be used by agents to perform navigation while still maintaining end-to-end differentiability of the entire architecture.
Self-Training for Jointly Learning to Ask and Answer Questions
The two tasks of question answering and question generation are usually tackled separately in the NLP literature.
Dynamic-structured Semantic Propagation Network
Moreoever, we demonstrate a universal segmentation model that is jointly trained on diverse datasets can surpass the performance of the common fine-tuning scheme for exploiting multiple domain knowledge.
Ranked #56 on
Semantic Segmentation
on Cityscapes test
Deep learning based supervised semantic segmentation of Electron Cryo-Subtomograms
Cellular Electron Cryo-Tomography (CECT) is a powerful imaging technique for the 3D visualization of cellular structure and organization at submolecular resolution.
Neural Architecture Search with Bayesian Optimisation and Optimal Transport
A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model.
Real-to-Virtual Domain Unification for End-to-End Autonomous Driving
In the spectrum of vision-based autonomous driving, vanilla end-to-end models are not interpretable and suboptimal in performance, while mediated perception models require additional intermediate representations such as segmentation masks or detection bounding boxes, whose annotation can be prohibitively expensive as we move to a larger scale.
Convolutional Neural Networks for Medical Diagnosis from Admission Notes
The hope is that the tool can be used to reduce mis-diagnosis.
On the Automatic Generation of Medical Imaging Reports
To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs.
Predicting Discharge Medications at Admission Time Based on Deep Learning
Predicting discharge medications right after a patient being admitted is an important clinical decision, which provides physicians with guidance on what type of medication regimen to plan for and what possible changes on initial medication may occur during an inpatient stay.
From Textbooks to Knowledge: A Case Study in Harvesting Axiomatic Knowledge from Textbooks to Solve Geometry Problems
These axioms are then parsed into rules that are used to improve the state-of-the-art in solving geometry problems.
Learning to Solve Geometry Problems from Natural Language Demonstrations in Textbooks
As a case study, we explore the task of learning to solve geometry problems using demonstrative solutions available in textbooks.
A Constituent-Centric Neural Architecture for Reading Comprehension
Reading comprehension (RC), aiming to understand natural texts and answer questions therein, is a challenging task.
Recurrent Estimation of Distributions
After, an RNN is used to compute the conditional distributions of the latent covariates.
Nonparametric Variational Auto-encoders for Hierarchical Representation Learning
In this work, we propose hierarchical nonparametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space.
Seeing the Forest from the Trees in Two Looks: Matrix Sketching by Cascaded Bilateral Sampling
Given a matrix A of size m by n, state-of-the-art randomized algorithms take O(m * n) time and space to obtain its low-rank decomposition.
Post-Inference Prior Swapping
However, we demonstrate that IS will fail for many choices of the target prior, depending on its parametric form and similarity to the false prior.
Harnessing Deep Neural Networks with Logic Rules
Combining deep neural networks with structured logic rules is desirable to harness flexibility and reduce uninterpretability of the neural models.
Ranked #61 on
Sentiment Analysis
on SST-2 Binary classification
Latent Variable Modeling with Diversity-Inducing Mutual Angular Regularization
On two popular latent variable models --- restricted Boltzmann machine and distance metric learning, we demonstrate that MAR can effectively capture long-tail patterns, reduce model complexity without sacrificing expressivity and improve interpretability.
Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
To investigate how to adapt existing frameworks to efficiently support distributed GPUs, we propose Poseidon, a scalable system architecture for distributed inter-machine communication in existing DL frameworks.
Distributed Machine Learning via Sufficient Factor Broadcasting
Matrix-parametrized models, including multiclass logistic regression and sparse coding, are used in machine learning (ML) applications ranging from computer vision to computational biology.
On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularization
Recently diversity-inducing regularization methods for latent variable models (LVMs), which encourage the components in LVMs to be diverse, have been studied to address several issues involved in latent variable modeling: (1) how to capture long-tail patterns underlying data; (2) how to reduce model complexity without sacrificing expressivity; (3) how to improve the interpretability of learned patterns.
Scalable Gaussian Processes for Characterizing Multidimensional Change Surfaces
We present a scalable Gaussian process model for identifying and characterizing smooth multidimensional changepoints, and automatically learning changes in expressive covariance structure.
Embarrassingly Parallel Variational Inference in Nonconjugate Models
We develop a parallel variational inference (VI) procedure for use in data-distributed settings, where each machine only has access to a subset of data and runs VI independently, without communicating with other machines.
Cauchy Principal Component Analysis
In this paper, we propose Cauchy Principal Component Analysis (Cauchy PCA), a very simple yet effective PCA method which is robust to various types of noise.
Large Scale Distributed Distance Metric Learning
In large scale machine learning and data mining problems with high feature dimensionality, the Euclidean distance between data points can be uninformative, and Distance Metric Learning (DML) is often desired to learn a proper similarity measure (using side information such as example data pairs being similar or dissimilar).
Fast Function to Function Regression
Function to function regression (FFR) covers a large range of interesting applications including time-series prediction problems, and also more general tasks like studying a mapping between two separate types of distributions.
Distributed Machine Learning via Sufficient Factor Broadcasting
Matrix-parametrized models, including multiclass logistic regression and sparse coding, are used in machine learning (ML) applications ranging from computer vision to computational biology.
Nonparametric Latent Tree Graphical Models: Inference, Estimation, and Structure Learning
Tree structured graphical models are powerful at expressing long range or hierarchical dependency among many variables, and have been widely applied in different areas of computer science and statistics.
Asymptotically Exact, Embarrassingly Parallel MCMC
This embarrassingly parallel algorithm allows each machine to act independently on a subset of the data (without communication) until the final combination stage.
Fast Distribution To Real Regression
We study the problem of distribution to real-value regression, where one aims to regress a mapping $f$ that takes in a distribution input covariate $P\in \mathcal{I}$ (for a non-parametric family of distributions $\mathcal{I}$) and outputs a real-valued response $Y=f(P) + \epsilon$.
