Search Results for author:
Found 22 papers, 11 papers with code
DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection
This is followed by a fine-tuning stage that leverages a small number of high-resolution samples to further enhance detection performance.
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Attempting to complement this deficiency, we investigate layerwise properties of LoRA on fine-tuning tasks and observe an uncommon skewness of weight norms across different layers.
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself.
Ranked #29 on
Visual Question Answering
on MM-Vet
GradSafe: Detecting Unsafe Prompts for LLMs via Safety-Critical Gradient Analysis
In this study, we propose GradSafe, which effectively detects unsafe prompts by scrutinizing the gradients of safety-critical parameters in LLMs.
The Instinctive Bias: Spurious Images lead to Hallucination in MLLMs
In this paper, we identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers, causing MLLMs to suffer from hallucination.
SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level.
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs.
G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
We first analyze the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehending basic geometric elements and their relationships.
Plum: Prompt Learning using Metaheuristic
Since the emergence of large language models, prompt learning has become a popular method for optimizing and customizing these models.
PerceptionGPT: Effectively Fusing Visual Perception into LLM
In this paper, we present a novel end-to-end framework named PerceptionGPT, which efficiently and effectively equips the VLLMs with visual perception abilities by leveraging the representation power of LLMs' token embedding.
Mitigating the Alignment Tax of RLHF
Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different reward-tax trade-offs, we propose Adaptive Model Averaging (AMA) to adaptively find various combination ratios of model layers.
DetGPT: Detect What You Need via Reasoning
Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines.
Effective Bilevel Optimization via Minimax Reformulation
To address this issue, we propose a reformulation of bilevel optimization as a minimax problem, effectively decoupling the outer-inner dependency.
Probabilistic Bilevel Coreset Selection
The goal of coreset selection in supervised learning is to produce a weighted subset of data, so that training only on the subset achieves similar performance as training on the entire dataset.
Model Agnostic Sample Reweighting for Out-of-Distribution Learning
The overfitting issue is addressed by considering a bilevel formulation to search for the sample reweighting, in which the generalization complexity depends on the search space of sample weights instead of the model size.
DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics
Specifically, we first reserve a short trajectory of global model snapshots on the server.
Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimization
Since at each round, the number of tunable parameters optimized on the server side equals the number of participating clients (thus independent of the model size), we are able to train a global model with massive parameters using only a small amount of proxy data (e. g., around one hundred samples).
Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning
In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs.
G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation
In this paper, we investigate the knowledge distillation (KD) strategy for object detection and propose an effective framework applicable to both homogeneous and heterogeneous student-teacher pairs.
Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
For student morphism, weight inheritance strategy is adopted, allowing the student to flexibly update its architecture while fully utilize the predecessor's weights, which considerably accelerates the search; To facilitate dynamic distillation, an elastic teacher pool is trained via integrated progressive shrinking strategy, from which teacher detectors can be sampled without additional cost in subsequent searches.
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
In this work, we propose BONAS (Bayesian Optimized Neural Architecture Search), a sample-based NAS framework which is accelerated using weight-sharing to evaluate multiple related architectures simultaneously.
Multi-objective Neural Architecture Search via Predictive Network Performance Optimization
Inspired by the nature of the graph structure of a neural network, we propose BOGCN-NAS, a NAS algorithm using Bayesian Optimization with Graph Convolutional Network (GCN) predictor.
