Search Results for author:
Found 49 papers, 17 papers with code
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Large Language Models (LLMs) have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information from long-term interactions limits their effectiveness in applications requiring sustained personalization.
Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation
To address this, we propose Magnet, a principled framework for synthesizing high-quality training trajectories to enhance the function calling capability of large language model agents in multi-turn conversations with humans.
PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving
Recent agent frameworks and inference-time algorithms often struggle with complex planning problems due to limitations in verifying generated plans or reasoning and varying complexity of instances within a single task.
Universal Model Routing for Efficient LLM Inference
Large language models' significant advances in capabilities are accompanied by significant increases in inference costs.
When One LLM Drools, Multi-LLM Collaboration Rules
This position paper argues that in many realistic (i. e., complex, contextualized, subjective) scenarios, one LLM is not enough to produce a reliable output.
Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights.
Exemplar Masking for Multimodal Incremental Learning
Specifically, the non-important tokens are masked based on the attention weights and the correlation across different modalities, significantly reducing the storage size of an exemplar and consequently saving more exemplars under the same memory buffer.
Reverse Thinking Makes LLMs Stronger Reasoners
Reverse thinking plays a crucial role in human reasoning.
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Extensive experiments demonstrate that Model Swarms could flexibly adapt LLM experts to a single task, multi-task domains, reward models, as well as diverse human interests, improving over 12 model composition baselines by up to 21. 0% across tasks and contexts.
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
To address these limitations, we introduce Speculative Knowledge Distillation (SKD), a novel approach that leverages cooperation between student and teacher models to generate high-quality training data on-the-fly while aligning with the student's inference-time distribution.
Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval
Recent advances in large language models (LLMs) have enabled autonomous agents with complex reasoning and task-fulfillment capabilities using a wide range of tools.
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.
CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation
Grounded generation aims to equip language models (LMs) with the ability to produce more credible and accountable responses by accurately citing verifiable sources.
CodecLM: Aligning Language Models with Tailored Synthetic Data
To this end, we introduce CodecLM, a general framework for adaptively generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs.
Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction
In this new knowledge acquisition and extraction, we find a very intriguing fact that LLMs can accurately answer questions about the first sentence, but they struggle to extract information described in the middle or end of the documents used for fine-tuning.
Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding
We propose the Chain-of-Table framework, where tabular data is explicitly used in the reasoning chain as a proxy for intermediate thoughts.
Ranked #4 on
Table-based Fact Verification
on TabFact
LMDX: Language Model-based Document Information Extraction and Localization
The main obstacles to adopting LLMs for this task include the absence of layout encoding within LLMs, which is critical for high quality extraction, and the lack of a grounding mechanism to localize the predicted entities within the document.
Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models
Today, large language models (LLMs) are taught to use new tools by providing a few demonstrations of the tool's usage.
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction
In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss.
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80% of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100% of the dataset.
Multimodal Prompting with Missing Modalities for Visual Recognition
In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models.
Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval
Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image.
Ranked #1 on
Zero-shot Image Retrieval
on ImageNet-R
Neural Spline Search for Quantile Probabilistic Modeling
Accurate estimation of output quantiles is crucial in many use cases, where it is desired to model the range of possibility.
VRDU: A Benchmark for Visually-rich Document Understanding
Understanding visually-rich business documents to extract structured data and automate business workflows has been receiving attention both in academia and industry.
QueryForm: A Simple Zero-shot Form Entity Query Framework
Zero-shot transfer learning for document understanding is a crucial yet under-investigated scenario to help reduce the high cost involved in annotating document entities.
Prefix Conditioning Unifies Language and Label Supervision
In experiments, we show that this simple technique improves the performance in zero-shot image recognition accuracy and robustness to the image-level distribution shift.
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning
Continual learning aims to enable a single model to learn a sequence of tasks without catastrophic forgetting.
FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction
Sequence modeling has demonstrated state-of-the-art performance on natural language and document understanding tasks.
Towards Group Robustness in the presence of Partial Group Labels
Such a requirement is impractical in situations where the data labeling efforts for minority or rare groups are significantly laborious or where the individuals comprising the dataset choose to conceal sensitive information.
Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types
We define a distance function between images, each of which is represented as a bag of embeddings, by the Euclidean distance between weighted averaged embeddings.
Learning to Prompt for Continual Learning
The mainstream paradigm behind continual learning has been to adapt the model parameters to non-stationary data distributions, where catastrophic forgetting is the central challenge.
Invariant Learning with Partial Group Labels
Such a requirement is impractical in situations where the data labelling efforts for minority or rare groups is significantly laborious or where the individuals comprising the dataset choose to conceal sensitive information.
Unifying Distribution Alignment as a Loss for Imbalanced Semi-supervised Learning
While remarkable progress in imbalanced supervised learning has been made recently, less attention has been given to the setting of imbalanced semi-supervised learning (SSL) where not only is a few labeled data provided, but the underlying data distribution can be severely imbalanced.
ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction
Natural reading orders of words are crucial for information extraction from form-like documents.
Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection
We demonstrate our method on various unsupervised AD tasks with image and tabular data.
Learning from Weakly-labeled Web Videos via Exploring Sub-Concepts
To address this issue, we introduce a new method for pre-training video action recognition models using queried web videos.
Exploring Sub-Pseudo Labels for Learning from Weakly-Labeled Web Videos
To address this issue, we introduce a new method for pre-training video action recognition models using queried web videos.
Learning to Branch for Multi-Task Learning
Training multiple tasks jointly in one deep network yields reduced latency during inference and better performance over the single-task counterpart by sharing certain layers of a network.
A Simple Semi-Supervised Learning Framework for Object Detection
Semi-supervised learning (SSL) has a potential to improve the predictive performance of machine learning models using unlabeled data.
Ranked #13 on
Semi-Supervised Object Detection
on COCO 100% labeled data
(using extra training data)
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation
In this work, we connect two distinct concepts for unsupervised domain adaptation: feature distribution alignment between domains by utilizing the task-specific decision boundary and the Wasserstein metric.
Ranked #19 on
Image-to-Image Translation
on SYNTHIA-to-Cityscapes
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
Deep multitask networks, in which one neural network produces multiple predictive outputs, can offer better speed and performance than their single-task counterparts but are challenging to train properly.
RoomNet: End-to-End Room Layout Estimation
This paper focuses on the task of room layout estimation from a monocular RGB image.
Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
We present recursive recurrent neural networks with attention modeling (R$^2$AM) for lexicon-free optical character recognition in natural scene images.
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures.
Training Deeper Convolutional Networks with Deep Supervision
One of the most promising ways of improving the performance of deep convolutional neural networks is by increasing the number of convolutional layers.
Deeply-Supervised Nets
Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent.
Region-based Discriminative Feature Pooling for Scene Text Recognition
We present a new feature representation method for scene text recognition problem, particularly focusing on improving scene character recognition.
