Search Results for author:
Found 55 papers, 27 papers with code
DocQueryNet: Value Retrieval with Arbitrary Queries for Form-like Documents
Unlike previous methods that only address a fixed set of field items, our method predicts target value for an arbitrary query based on the understanding of the layout and semantics of a form.
SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant
Recent advancements in the vision-language model have shown notable generalization in vision-language tasks after visual instruction tuning.
Ranked #35 on
Visual Question Answering
on MM-Vet
NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation
To address these challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15 distinct manipulation tasks, containing over 4500 episodes meticulously annotated with fine-grained language instructions.
FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability
This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats, a crucial yet underexamined capability for their application as AI agents.
RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records
We present RAM-EHR, a Retrieval AugMentation pipeline to improve clinical predictions on Electronic Health Records (EHRs).
Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM
While there has been a lot of research on representation learning of structured EHR data, the fusion of different types of EHR data (multimodal fusion) is not well studied.
MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
Embodied agents equipped with GPT as their brain have exhibited extraordinary decision-making and generalization abilities across various tasks.
EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records
Large language models (LLMs) have demonstrated exceptional capabilities in planning and tool utilization as autonomous agents, but few have been developed for medical problem-solving.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
To tackle these issues, we propose a continual self-supervised method, Adaptive Distribution Masked Autoencoders (ADMA), which enhances the extraction of target domain knowledge while mitigating the accumulation of distribution shifts.
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
Vision-language pre-training and instruction tuning have demonstrated general-purpose capabilities in 2D visual reasoning tasks by aligning visual encoders with state-of-the-art large language models (LLMs).
Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models
Clinical natural language processing requires methods that can address domain-specific challenges, such as complex medical terminology and clinical contexts.
Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation
In this paper, we propose a distribution-aware tuning (DAT) method to make the semantic segmentation CTTA efficient and practical in real-world applications.
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
The massive successes of large language models (LLMs) encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs).
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
REX: Rapid Exploration and eXploitation for AI Agents
In this paper, we propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX.
Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
To address this challenge, we propose a weakly-supervised approach for scientific document classification using label names only.
A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises
This work aims to serve as a valuable resource for understanding the potential and opportunities of HKG in health research.
R-Mixup: Riemannian Mixup for Biological Networks
Biological networks are commonly used in biomedical and healthcare domains to effectively model the structure of complex biological systems with interactions linking biological entities.
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages.
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
Recent advancements in multimodal pre-training methods have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions.
Ranked #4 on
3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations
In summary, an OV method learns task-specific information using strong supervision from base annotations and novel category information using weak supervision from image-captions pairs.
GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation
Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation.
HIVE: Harnessing Human Feedback for Instructional Visual Editing
Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences.
Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation
Accurately estimating 3D hand pose is crucial for understanding how humans interact with the world.
Neighborhood-Regularized Self-Training for Learning with Few Labels
Our further analysis demonstrates that our proposed data selection strategy reduces the noise of pseudo labels by 36. 8% and saves 57. 3% of the time when compared with the best baseline.
Model-Agnostic Hierarchical Attention for 3D Object Detection
By plugging our proposed modules into the state-of-the-art transformer-based 3D detector, we improve the previous best results on both benchmarks, with the largest improvement margin on small objects.
LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer
Graphic layout designs play an essential role in visual communication.
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets.
Ranked #3 on
Training-free 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Tackling Data Heterogeneity in Federated Learning with Class Prototypes
We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization by combining the uniformity and semantics of class prototypes.
Learning Task-Aware Effective Brain Connectivity for fMRI Analysis with Graph Neural Networks
To better adapt GNNs for fMRI analysis, we propose TBDS, an end-to-end framework based on \underline{T}ask-aware \underline{B}rain connectivity \underline{D}AG (short for Directed Acyclic Graph) \underline{S}tructure generation for fMRI analysis.
Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
Large Language Models have demonstrated remarkable few-shot performance, but the performance can be sensitive to the selection of few-shot instances.
TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation
To address this deficiency, we develop a new method to generate high-quality and diverse QA pairs by explicitly utilizing the existing rich text available in the scene context of each image.
Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework
Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks.
SmartAdapt: Multi-Branch Object Detection Framework for Videos on Mobiles
In this paper, we ask, and answer, the wide-ranging question across all MBODFs: How to expose the right set of execution branches and then how to schedule the optimal one at inference time?
Virtuoso: Video-based Intelligence for real-time tuning on SOCs
First, the system does not consider energy consumption of the models while making a decision on which model to run.
Value Retrieval with Arbitrary Queries for Form-like Documents
Unlike previous methods that only address a fixed set of field items, our method predicts target value for an arbitrary query based on the understanding of the layout and semantics of a form.
Burn After Reading: Online Adaptation for Cross-domain Streaming Data
In the context of online privacy, many methods propose complex privacy and security preserving measures to protect sensitive data.
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs.
Robustness Evaluation of Transformer-based Form Field Extractors via Form Attacks
We propose a novel framework to evaluate the robustness of transformer-based form field extraction methods via form attacks.
Field Extraction from Forms with Unlabeled Data
We propose a novel framework to conduct field extraction from forms with unlabeled data.
MetaHistoSeg: A Python Framework for Meta Learning in Histopathology Image Segmentation
We also curate a histopathology meta dataset - a benchmark dataset for training and validating models on out-of-distribution performance across a range of cancer types.
JANUS: Benchmarking Commercial and Open-Source Cloud and Edge Platforms for Object and Anomaly Detection Workloads
In addition, we also look at the pros and cons of some of the proprietary deep-learning object detection packages, such as Amazon Rekognition, Google Vision, and Azure Cognitive Services, to contrast with open-source and tunable solutions, such as Faster R-CNN (FRCNN).
ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles
In this paper we introduce ApproxDet, an adaptive video object detection framework for mobile devices to meet accuracy-latency requirements in the face of changing content and resource contention scenarios.
WOAD: Weakly Supervised Online Action Detection in Untrimmed Videos
Online action detection in untrimmed videos aims to identify an action as it happens, which makes it very important for real-time applications.
Ranked #5 on
Online Action Detection
on THUMOS'14
Proposal Learning for Semi-Supervised Object Detection
two-stage object detectors) by training on both labeled and unlabeled data.
Context-aware Active Multi-Step Reinforcement Learning
Then given the selected samples, we propose the adaptive multi-step TD, which generalizes TD($\lambda$), but adaptively switch on/off the backups from future returns of different steps.
ApproxNet: Content and Contention-Aware Video Analytics System for Embedded Clients
None of the current approximation techniques for object classification DNNs can adapt to changing runtime conditions, e. g., changes in resource availability on the device, the content characteristics, or requirements from the user.
TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy
We introduce TempoCave, a novel visualization application for analyzing dynamic brain networks, or connectomes.
Human-Computer Interaction Neurons and Cognition
Collecting and Annotating the Large Continuous Action Dataset
We make available to the community a new dataset to support action-recognition research.
Human Action Segmentation With Hierarchical Supervoxel Consistency
Detailed analysis of human action, such as action classification, detection and localization has received increasing attention from the community; datasets like JHMDB have made it plausible to conduct studies analyzing the impact that such deeper information has on the greater action understanding problem.
Sequential Labeling with online Deep Learning
Deep learning has attracted great attention recently and yielded the state of the art performance in dimension reduction and classification problems.
Compositional Structure Learning for Action Understanding
The focus of the action understanding literature has predominately been classification, how- ever, there are many applications demanding richer action understanding such as mobile robotics and video search, with solutions to classification, localization and detection.
Actionness Ranking with Lattice Conditional Ordinal Random Fields
Action analysis in image and video has been attracting more and more attention in computer vision.
