Search Results for author:
Found 57 papers, 35 papers with code
AcTune: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models
We develop AcTune, a new framework that improves the label efficiency of active PLM fine-tuning by unleashing the power of unlabeled data via self-training.
Generate Any Scene: Evaluating and Improving Text-to-Vision Generation with Scene Graph Programming
We introduce Generate Any Scene, a framework that systematically enumerates scene graphs representing a vast array of visual scenes, spanning realistic to imaginative compositions.
Template Matters: Understanding the Role of Instruction Templates in Multimodal Language Model Evaluation and Training
Models tuned on our augmented dataset achieve the best overall performance when compared with the same scale MLMs tuned on at most 75 times the scale of our augmented dataset, highlighting the importance of instruction templates in MLM training.
ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
Our multi-image instruction data leads to an 8% improvement on Mantis-Eval.
Ranked #105 on
Visual Question Answering
on MM-Vet
TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
While open-source multi-modal language models perform well on simple question answering tasks, they often fail on complex questions that require multiple capabilities, such as fine-grained recognition, visual grounding, and reasoning, and that demand multi-step solutions.
Ranked #61 on
Visual Question Answering
on MM-Vet
EcoAct: Economic Agent Determines When to Register What Action
Recent advancements have enabled Large Language Models (LLMs) to function as agents that can perform actions using external tools.
Language Model Preference Evaluation with Multiple Weak Evaluators
To improve model-based preference evaluation, we introduce GED (Preference Graph Ensemble and Denoise), a novel approach that leverages multiple model-based evaluators to construct preference graphs, and then ensemble and denoise these graphs for better, non-contradictory evaluation results.
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs.
Explaining Length Bias in LLM-Based Preference Evaluations
The use of large language models (LLMs) as judges, particularly in preference comparisons, has become widespread, but this reveals a notable bias towards longer responses, undermining the reliability of such evaluations.
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
In this work, we propose a data-centric framework, Biomedical Visual Instruction Tuning with Clinician Preference Alignment (BioMed-VITAL), that incorporates clinician preferences into both stages of generating and selecting instruction data for tuning biomedical multimodal foundation models.
DataComp-LM: In search of the next generation of training sets for language models
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models.
Adaptive In-conversation Team Building for Language Model Agents
Leveraging multiple large language model (LLM) agents has shown to be a promising approach for tackling complex tasks, while the effective design of multiple agents for a particular application remains an art.
Iterated Learning Improves Compositionality in Large Vision-Language Models
A fundamental characteristic common to both human vision and natural language is their compositional nature.
LLMs-based Few-Shot Disease Predictions using EHR: A Novel Approach Combining Predictive Agent Reasoning and Critical Agent Instruction
Electronic health records (EHRs) contain valuable patient data for health-related prediction tasks, such as disease prediction.
m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks
With m&m's, we evaluate 10 popular LLMs with 2 planning strategies (multi-step vs. step-by-step planning), 2 plan formats (JSON vs. code), and 3 types of feedback (parsing/verification/execution).
Offline Training of Language Model Agents with Functions as Learnable Weights
Researchers and practitioners have recently reframed powerful Large Language Models (LLMs) as agents, enabling them to automate complex tasks largely via the use of specialized functions.
Leveraging Large Language Models for Structure Learning in Prompted Weak Supervision
We propose a Structure Refining Module, a simple yet effective first approach based on the similarities of the prompts by taking advantage of the intrinsic structure in the embedding space.
EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records
Large language models (LLMs) have demonstrated exceptional capabilities in planning and tool utilization as autonomous agents, but few have been developed for medical problem-solving.
How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking
This paper presents LEMR (Label-Efficient Model Ranking) and introduces the MoraBench Benchmark.
EcoAssistant: Using LLM Assistant More Affordably and Accurately
Today, users ask Large language models (LLMs) as assistants to answer queries that require external knowledge; they ask about the weather in a specific city, about stock prices, and even about where specific locations are within their neighborhood.
NLPBench: Evaluating Large Language Models on Solving NLP Problems
Recent developments in large language models (LLMs) have shown promise in enhancing the capabilities of natural language processing (NLP).
Uncovering Neural Scaling Laws in Molecular Representation Learning
Molecular Representation Learning (MRL) has emerged as a powerful tool for drug and materials discovery in a variety of tasks such as virtual screening and inverse design.
When to Learn What: Model-Adaptive Data Augmentation Curriculum
Unlike previous work, MADAug selects augmentation operators for each input image by a model-adaptive policy varying between training stages, producing a data augmentation curriculum optimized for better generalization.
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
AutoGen is an open-source framework that allows developers to build LLM applications via multiple agents that can converse with each other to accomplish tasks.
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Most of the existing Large Language Model (LLM) benchmarks on scientific problem reasoning focus on problems grounded in high-school subjects and are confined to elementary algebraic operations.
Subclass-balancing Contrastive Learning for Long-tailed Recognition
We overcome these drawbacks by a novel ``subclass-balancing contrastive learning (SBCL)'' approach that clusters each head class into multiple subclasses of similar sizes as the tail classes and enforce representations to capture the two-layer class hierarchy between the original classes and their subclasses.
SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
In the last year alone, a surge of new benchmarks to measure compositional understanding of vision-language models have permeated the machine learning ecosystem.
Taming Small-sample Bias in Low-budget Active Learning
Instead of tuning the coefficient for each query round, which is sensitive and time-consuming, we propose the curriculum Firth bias reduction (CHAIN) that can automatically adjust the coefficient to be adaptive to the training process.
On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training
Pre-training datasets are critical for building state-of-the-art machine learning models, motivating rigorous study on their impact on downstream tasks.
MaskSearch: Querying Image Masks at Scale
Machine learning tasks over image databases often generate masks that annotate image content (e. g., saliency maps, segmentation maps, depth maps) and enable a variety of applications (e. g., determine if a model is learning spurious correlations or if an image was maliciously modified to mislead a model).
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
Label-Efficient Interactive Time-Series Anomaly Detection
Time-series anomaly detection is an important task and has been widely applied in the industry.
Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph
To answer this question, we theoretically study the concentration property of features obtained by neighborhood aggregation on homophilic and heterophilic graphs, introduce the single-pass augmentation-free graph contrastive learning loss based on the property, and provide performance guarantees for the minimizer of the loss on downstream tasks.
Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision
In particular, it is built on a mixture of Bayesian label models, each corresponding to a global pattern of correlation, and the coefficients of the mixture components are predicted by a Gaussian Process classifier based on instance features.
Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification
To obtain a large amount of training labels inexpensively, researchers have recently adopted the weak supervision (WS) paradigm, which leverages labeling rules to synthesize training labels rather than using individual annotations to achieve competitive results for natural language processing (NLP) tasks.
Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
Large Language Models have demonstrated remarkable few-shot performance, but the performance can be sensitive to the selection of few-shot instances.
Binary Classification with Positive Labeling Sources
Thus, in this work, we study the application of WS on binary classification tasks with positive labeling sources only.
Learning Hyper Label Model for Programmatic Weak Supervision
We train the model on synthetic data generated in the way that ensures the model approximates the analytical optimal solution, and build the model upon Graph Neural Network (GNN) to ensure the model prediction being invariant (or equivariant) to the permutation of LFs (or data points).
Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning
In this work, we demonstrate that the learned representation of the $Q$-network and its target $Q$-network should, in theory, satisfy a favorable distinguishable representation property.
Understanding Programmatic Weak Supervision via Source-aware Influence Function
Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end model.
Augmentation-Free Graph Contrastive Learning with Performance Guarantee
Graph contrastive learning (GCL) is the most representative and prevalent self-supervised learning approach for graph-structured data.
A Survey on Deep Graph Generation: Methods and Applications
In this paper, we conduct a comprehensive review on the existing literature of deep graph generation from a variety of emerging methods to its wide application areas.
Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming
Weak Supervision (WS) techniques allow users to efficiently create large training datasets by programmatically labeling data with heuristic sources of supervision.
A Survey on Programmatic Weak Supervision
Labeling training data has become one of the major roadblocks to using machine learning.
TaxoEnrich: Self-Supervised Taxonomy Completion via Structure-Semantic Representations
Taxonomies are fundamental to many real-world applications in various domains, serving as structural representations of knowledge.
AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Models
We propose {\ours}, a new framework that leverages unlabeled data to improve the label efficiency of active PLM fine-tuning.
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Creating Training Sets via Weak Indirect Supervision
Creating labeled training sets has become one of the major roadblocks in machine learning.
WRENCH: A Comprehensive Benchmark for Weak Supervision
To address these problems, we introduce a benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches.
Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Who Should Go First? A Self-Supervised Concept Sorting Model for Improving Taxonomy Expansion
Taxonomies have been widely used in various machine learning and text mining systems to organize knowledge and facilitate downstream tasks.
A Survey on Graph Structure Learning: Progress and Opportunities
Specifically, we first formulate a general pipeline of GSL and review state-of-the-art methods classified by the way of modeling graph structures, followed by applications of GSL across domains.
Taxonomy Completion via Triplet Matching Network
Previous approaches focus on the taxonomy expansion, i. e. finding an appropriate hypernym concept from the taxonomy for a new query concept.
Relation Learning on Social Networks with Multi-Modal Graph Edge Variational Autoencoders
While node semantics have been extensively explored in social networks, little research attention has been paid to profile edge semantics, i. e., social relations.
Neural Embedding Propagation on Heterogeneous Networks
While generalizing LP as a simple instance, NEP is far more powerful in its natural awareness of different types of objects and links, and the ability to automatically capture their important interaction patterns.
