Search Results for author:
Found 79 papers, 41 papers with code
DocQueryNet: Value Retrieval with Arbitrary Queries for Form-like Documents
Unlike previous methods that only address a fixed set of field items, our method predicts target value for an arbitrary query based on the understanding of the layout and semantics of a form.
MedAgentGym: Training LLM Agents for Code-Based Medical Reasoning at Scale
We introduce MedAgentGYM, the first publicly available training environment designed to enhance coding-based medical reasoning capabilities in large language model (LLM) agents.
BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models.
DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
Extensive experiments on image and video understanding tasks demonstrate that DyMU can reduce the average visual token count by 32%-85% while achieving comparable performance to full-length models across diverse VLM architectures, including the recently popularized AnyRes-based visual encoders.
Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration
Retrieval-Augmented Generation (RAG) systems often struggle to handle multi-hop question-answering tasks accurately due to irrelevant context retrieval and limited complex reasoning capabilities.
Could AI Trace and Explain the Origins of AI-Generated Images and Text?
AI-generated content is becoming increasingly prevalent in the real world, leading to serious ethical and societal concerns.
A Status Quo Investigation of Large Language Models towards Cost-Effective CFD Automation with OpenFOAMGPT: ChatGPT vs. Qwen vs. Deepseek
We evaluated the performance of OpenFOAMGPT incorporating multiple large-language models.
RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization
Large language models (LLMs) have achieved impressive performance but face high computational costs and latency, limiting their deployment in resource-constrained settings.
ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
Our multi-image instruction data leads to an 8% improvement on Mantis-Eval.
Ranked #107 on
Visual Question Answering
on MM-Vet
BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text.
SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains
Retrieval-augmented generation (RAG) enhances the question-answering (QA) abilities of large language models (LLMs) by integrating external knowledge.
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames.
Trust but Verify: Programmatic VLM Evaluation in the Wild
Next, to evaluate free-form model responses to queries in PROVE, we propose a programmatic evaluation strategy that measures both the helpfulness and truthfulness of a response within a unified scene graph-based framework.
xLAM: A Family of Large Action Models to Empower AI Agent Systems
By releasing the xLAM series, we aim to advance the performance of open-source LLMs for autonomous AI agents, potentially accelerating progress and democratizing access to high-performance models for agent tasks.
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions.
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs.
Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
Unlike traditional methods, which generate two responses before obtaining the preference label, RMBoost first generates one response and selects a preference label, followed by generating the second more (or less) preferred response conditioned on the pre-selected preference label and the first response.
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
Multimodal interleaved datasets featuring free-form interleaved sequences of images and text are crucial for training frontier large multimodal models (LMMs).
TACCO: Task-guided Co-clustering of Clinical Concepts and Patient Visits for Disease Subtyping based on EHR Data
In this work, we introduce TACCO, a novel framework that jointly discovers clusters of clinical concepts and patient visits based on a hypergraph modeling of EHR data.
MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases
The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization.
From Basic to Extra Features: Hypergraph Transformer Pretrain-then-Finetuning for Balanced Clinical Predictions on EHR
Electronic Health Records (EHRs) contain rich patient information and are crucial for clinical research and practice.
Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation
Continual Test-Time Adaptation (CTTA), which aims to adapt the pre-trained model to ever-evolving target domains, emerges as an important task for vision models.
MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning
Faced with the challenges of balancing model performance, computational resources, and data privacy, MedAdapter provides an efficient, privacy-preserving, cost-effective, and transparent solution for adapting LLMs to the biomedical domain.
BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers
Developing effective biomedical retrieval models is important for excelling at knowledge-intensive biomedical tasks but still challenging due to the deficiency of sufficient publicly annotated biomedical data and computational resources.
SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant
Recent advances in vision-language models have shown notable generalization in broad tasks through visual instruction tuning.
Ranked #117 on
Visual Question Answering
on MM-Vet
NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation
To address these challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15 distinct manipulation tasks, containing over 4500 episodes meticulously annotated with fine-grained language instructions.
FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability
This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats, a crucial yet underexamined capability for their application as AI agents.
RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records
We present RAM-EHR, a Retrieval AugMentation pipeline to improve clinical predictions on Electronic Health Records (EHRs).
Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM
While there has been a lot of research on representation learning of structured EHR data, the fusion of different types of EHR data (multimodal fusion) is not well studied.
Text2Data: Low-Resource Data Generation with Textual Control
Natural language serves as a common and straightforward signal for humans to interact seamlessly with machines.
MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
Embodied agents equipped with GPT as their brains have exhibited extraordinary decision-making and generalization abilities across various tasks.
EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records
Large language models (LLMs) have demonstrated exceptional capabilities in planning and tool utilization as autonomous agents, but few have been developed for medical problem-solving.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
To tackle these issues, we propose a continual self-supervised method, Adaptive Distribution Masked Autoencoders (ADMA), which enhances the extraction of target domain knowledge while mitigating the accumulation of distribution shifts.
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
To enable this framework, we devise a scalable pipeline that automatically generates high-quality, instruction-tuning datasets from readily available captioning data across different modalities, and contribute 24K QA data for audio and 250K QA data for 3D.
Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models
To address this challenge, we delve into synthetic clinical text generation using LLMs for clinical NLP tasks.
Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation
In this paper, we propose a distribution-aware tuning (DAT) method to make the semantic segmentation CTTA efficient and practical in real-world applications.
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
The massive successes of large language models (LLMs) encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs).
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
REX: Rapid Exploration and eXploitation for AI Agents
In this paper, we propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX.
Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
To address this challenge, we propose a weakly-supervised approach for scientific document classification using label names only.
A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises
This comprehensive review aims to provide an overview of the current state of Healthcare Knowledge Graphs (HKGs), including their construction, utilization models, and applications across various healthcare and biomedical research domains.
R-Mixup: Riemannian Mixup for Biological Networks
Biological networks are commonly used in biomedical and healthcare domains to effectively model the structure of complex biological systems with interactions linking biological entities.
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages.
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
It achieves a new SOTA of 50. 6% (top-1) on Objaverse-LVIS and 84. 7% (top-1) on ModelNet40 in zero-shot classification.
Ranked #10 on
3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations
In summary, an OV method learns task-specific information using strong supervision from base annotations and novel category information using weak supervision from image-captions pairs.
GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation
Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation.
HIVE: Harnessing Human Feedback for Instructional Visual Editing
Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences.
Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation
Accurately estimating 3D hand pose is crucial for understanding how humans interact with the world.
Neighborhood-Regularized Self-Training for Learning with Few Labels
Our further analysis demonstrates that our proposed data selection strategy reduces the noise of pseudo labels by 36. 8% and saves 57. 3% of the time when compared with the best baseline.
Hierarchical Point Attention for Indoor 3D Object Detection
By plugging our proposed modules into the state-of-the-art transformer-based 3D detectors, we improve the previous best results on both benchmarks, with more significant improvements on smaller objects.
LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer
Graphic layout designs play an essential role in visual communication.
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets.
Ranked #4 on
Training-free 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Tackling Data Heterogeneity in Federated Learning with Class Prototypes
We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization by combining the uniformity and semantics of class prototypes.
Learning Task-Aware Effective Brain Connectivity for fMRI Analysis with Graph Neural Networks
To better adapt GNNs for fMRI analysis, we propose TBDS, an end-to-end framework based on \underline{T}ask-aware \underline{B}rain connectivity \underline{D}AG (short for Directed Acyclic Graph) \underline{S}tructure generation for fMRI analysis.
Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
Large Language Models have demonstrated remarkable few-shot performance, but the performance can be sensitive to the selection of few-shot instances.
TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation
To address this deficiency, we develop a new method to generate high-quality and diverse QA pairs by explicitly utilizing the existing rich text available in the scene context of each image.
Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework
Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks.
SmartAdapt: Multi-Branch Object Detection Framework for Videos on Mobiles
In this paper, we ask, and answer, the wide-ranging question across all MBODFs: How to expose the right set of execution branches and then how to schedule the optimal one at inference time?
Virtuoso: Video-based Intelligence for real-time tuning on SOCs
First, the system does not consider energy consumption of the models while making a decision on which model to run.
Value Retrieval with Arbitrary Queries for Form-like Documents
Unlike previous methods that only address a fixed set of field items, our method predicts target value for an arbitrary query based on the understanding of the layout and semantics of a form.
Burn After Reading: Online Adaptation for Cross-domain Streaming Data
In the context of online privacy, many methods propose complex privacy and security preserving measures to protect sensitive data.
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs.
Field Extraction from Forms with Unlabeled Data
We propose a novel framework to conduct field extraction from forms with unlabeled data.
Robustness Evaluation of Transformer-based Form Field Extractors via Form Attacks
We propose a novel framework to evaluate the robustness of transformer-based form field extraction methods via form attacks.
MetaHistoSeg: A Python Framework for Meta Learning in Histopathology Image Segmentation
We also curate a histopathology meta dataset - a benchmark dataset for training and validating models on out-of-distribution performance across a range of cancer types.
JANUS: Benchmarking Commercial and Open-Source Cloud and Edge Platforms for Object and Anomaly Detection Workloads
In addition, we also look at the pros and cons of some of the proprietary deep-learning object detection packages, such as Amazon Rekognition, Google Vision, and Azure Cognitive Services, to contrast with open-source and tunable solutions, such as Faster R-CNN (FRCNN).
ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles
In this paper we introduce ApproxDet, an adaptive video object detection framework for mobile devices to meet accuracy-latency requirements in the face of changing content and resource contention scenarios.
WOAD: Weakly Supervised Online Action Detection in Untrimmed Videos
Online action detection in untrimmed videos aims to identify an action as it happens, which makes it very important for real-time applications.
Ranked #5 on
Online Action Detection
on THUMOS'14
Proposal Learning for Semi-Supervised Object Detection
two-stage object detectors) by training on both labeled and unlabeled data.
Context-aware Active Multi-Step Reinforcement Learning
Then given the selected samples, we propose the adaptive multi-step TD, which generalizes TD($\lambda$), but adaptively switch on/off the backups from future returns of different steps.
ApproxNet: Content and Contention-Aware Video Analytics System for Embedded Clients
None of the current approximation techniques for object classification DNNs can adapt to changing runtime conditions, e. g., changes in resource availability on the device, the content characteristics, or requirements from the user.
TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy
We introduce TempoCave, a novel visualization application for analyzing dynamic brain networks, or connectomes.
Human-Computer Interaction Neurons and Cognition
Collecting and Annotating the Large Continuous Action Dataset
We make available to the community a new dataset to support action-recognition research.
Human Action Segmentation With Hierarchical Supervoxel Consistency
Detailed analysis of human action, such as action classification, detection and localization has received increasing attention from the community; datasets like JHMDB have made it plausible to conduct studies analyzing the impact that such deeper information has on the greater action understanding problem.
Sequential Labeling with online Deep Learning
Deep learning has attracted great attention recently and yielded the state of the art performance in dimension reduction and classification problems.
Compositional Structure Learning for Action Understanding
The focus of the action understanding literature has predominately been classification, how- ever, there are many applications demanding richer action understanding such as mobile robotics and video search, with solutions to classification, localization and detection.
Actionness Ranking with Lattice Conditional Ordinal Random Fields
Action analysis in image and video has been attracting more and more attention in computer vision.
