Search Results for author:
Found 103 papers, 63 papers with code
TEST_POSITIVE at W-NUT 2020 Shared Task-3: Cross-task modeling
The competition of extracting COVID-19 events from Twitter is to develop systems that can automatically extract related events from tweets.
Extracting COVID-19 Events from Twitter
Language Modelling
+4
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution.
Ranked #11 on
Visual Question Answering
on MM-Vet
Are We on the Right Way for Evaluating Large Vision-Language Models?
We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.
InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI).
Ranked #5 on
Long-Context Understanding
on Ada-LEval (BestAnswer)
Long-CLIP: Unlocking the Long-Text Capability of CLIP
Contrastive Language-Image Pre-training (CLIP) has been the cornerstone for zero-shot classification, text-image retrieval, and text-image generation by aligning image and text modalities.
RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition
Notably, our approach demonstrates a significant improvement in performance on 5 fine-grained visual recognition benchmarks, 11 few-shot image recognition datasets, and the 2 object detection datasets under the zero-shot recognition setting.
AIGCs Confuse AI Too: Investigating and Explaining Synthetic Image-induced Hallucinations in Large Vision-Language Models
Remarkably, our findings shed light on a consistent AIGC \textbf{hallucination bias}: the object hallucinations induced by synthetic images are characterized by a greater quantity and a more uniform position distribution, even these synthetic images do not manifest unrealistic or additional relevant visual features compared to natural images.
Deep learning for multi-label classification of coral conditions in the Indo-Pacific via underwater photogrammetry
A dataset containing over 20, 000 high-resolution coral images of different health conditions and stressors was constructed based on the field survey.
SongComposer: A Large Language Model for Lyric and Melody Composition in Song Generation
We present SongComposer, an innovative LLM designed for song composition.
Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder
Reconstructing natural language from non-invasive electroencephalography (EEG) holds great promise as a language decoding technology for brain-computer interfaces (BCIs).
CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning
Automatic International Classification of Diseases (ICD) coding plays a crucial role in the extraction of relevant information from clinical notes for proper recording and billing.
DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models
We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal large language models (MLLMs) to enhance vision-language task performance.
VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models
Correspondingly, we propose a novel VQAttack model, which can iteratively generate both image and text perturbations with the designed modules: the large language model (LLM)-enhanced image attack and the cross-modal joint attack module.
SepRep-Net: Multi-source Free Domain Adaptation via Model Separation And Reparameterization
We consider multi-source free domain adaptation, the problem of adapting multiple existing models to a new domain without accessing the source data.
Position Paper: Assessing Robustness, Privacy, and Fairness in Federated Learning Integrated with Foundation Models
Federated Learning (FL), while a breakthrough in decentralized machine learning, contends with significant challenges such as limited data availability and the variability of computational resources, which can stifle the performance and scalability of the models.
Recent Advances in Predictive Modeling with Electronic Health Records
This survey systematically reviews recent advances in deep learning-based predictive models using EHR data.
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension.
Ranked #16 on
Visual Question Answering
on MM-Vet
Rethinking Personalized Federated Learning with Clustering-based Dynamic Graph Propagation
Subsequently, we design a precise personalized model distribution strategy to allow clients to obtain the most suitable model from the server side.
Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions
The widespread adoption of Electronic Health Record (EHR) systems in healthcare institutes has generated vast amounts of medical data, offering significant opportunities for improving healthcare services through deep learning techniques.
Vulnerabilities of Foundation Model Integrated Federated Learning Under Adversarial Threats
Federated Learning (FL) addresses critical issues in machine learning related to data privacy and security, yet suffering from data insufficiency and imbalance under certain circumstances.
Enhancing Evolving Domain Generalization through Dynamic Latent Representations
However, in non-stationary tasks where new domains evolve in an underlying continuous structure, such as time, merely extracting the invariant features is insufficient for generalization to the evolving new domains.
Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
Notably, in the realm of robot task planning, LLMs harness their advanced reasoning and language comprehension capabilities to formulate precise and efficient action plans based on natural language instructions.
TechGPT-2.0: A large language model project to solve the task of knowledge graph construction
Large language models have exhibited robust performance across diverse natural language processing tasks.
Understanding LLMs: A Comprehensive Overview from Training to Inference
Low-cost training and deployment of LLMs represent the future development trend.
S3Net: Innovating Stereo Matching and Semantic Segmentation with a Single-Branch Semantic Stereo Network in Satellite Epipolar Imagery
Stereo matching and semantic segmentation are significant tasks in binocular satellite 3D reconstruction.
Segment Change Model (SCM) for Unsupervised Change detection in VHR Remote Sensing Images: a Case Study of Buildings
The field of Remote Sensing (RS) widely employs Change Detection (CD) on very-high-resolution (VHR) images.
Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases
We conducted a series of structured experiments to evaluate their performance in various industrial application scenarios, offering a comprehensive perspective on their practical utility.
HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a Single Image
Extensive experiments demonstrate the effectiveness of HyperDreamer in modeling region-aware materials with high-resolution textures and enabling user-friendly editing.
OneLLM: One Framework to Align All Modalities with Language
In detail, we first train an image projection module to connect a vision encoder with LLM.
Ranked #73 on
Visual Question Answering
on MM-Vet
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents.
GPT4Point: A Unified Framework for Point-Language Understanding and Generation
Multimodal Large Language Models (MLLMs) have excelled in 2D image-text comprehension and image generation, but their understanding of the 3D world is notably deficient, limiting progress in 3D language understanding and generation.
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue, along with a rollback strategy that retrospects the presence of summary tokens in the previously generated tokens, and re-allocate the token selection if necessary.
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
Multimodal large language models have made significant advancements in recent years, yet they still suffer from a common issue known as the "hallucination problem", in which the models generate textual descriptions that inaccurately depict or entirely fabricate content from associated images.
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data.
Ranked #1 on
visual instruction following
on LLaVA-Bench
Adversarial Prompt Tuning for Vision-Language Models
With the rapid advancement of multimodal learning, pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capacities in bridging the gap between visual and language modalities.
AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
Despite making significant progress in multi-modal tasks, current Multi-modal Large Language Models (MLLMs) encounter the significant challenge of hallucinations, which may lead to harmful consequences.
Holistic Evaluation of GPT-4V for Biomedical Imaging
GPT-4V represents a breakthrough in artificial general intelligence (AGI) for computer vision, with applications in the biomedical domain.
Hierarchical Pretraining on Multimodal Electronic Health Records
Pretraining has proven to be a powerful technique in natural language processing (NLP), exhibiting remarkable success in various NLP downstream tasks.
ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data
Radiology report generation, as a key step in medical image analysis, is critical to the quantitative analysis of clinically informed decision-making levels.
MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation
Health risk prediction is one of the fundamental tasks under predictive modeling in the medical domain, which aims to forecast the potential health risks that patients may face in the future using their historical Electronic Health Records (EHR).
Co-modeling the Sequential and Graphical Routes for Peptide Representation Learning
Peptides are formed by the dehydration condensation of multiple amino acids.
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition
We propose InternLM-XComposer, a vision-language large model that enables advanced image-text comprehension and composition.
Ranked #9 on
Visual Question Answering (VQA)
on InfiMM-Eval
Intelligent machines work in unstructured environments by differential neuromorphic computing
By mimicking the intrinsic nature of human low-level perception mechanisms, the electronic memristive neuromorphic circuit-based method, presented here shows the potential for adapting to diverse sensing technologies and helping intelligent machines generate smart high-level decisions in the real world.
MLLM-DataEngine: An Iterative Refinement Approach for MLLM
Despite the great advance of Multimodal Large Language Models (MLLMs) in both instruction dataset building and benchmarking, the independence of training and evaluation makes current MLLMs hard to further improve their capability under the guidance of evaluation results with a relatively low human cost.
VIGC: Visual Instruction Generation and Correction
A practical solution to this problem would be to utilize the available multimodal large language models (MLLMs) to generate instruction data for vision-language tasks.
WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models
The rise in popularity of ChatGPT and GPT-4 has significantly accelerated the development of large models, leading to the creation of numerous impressive large language models(LLMs) and multimodal large language models (MLLMs).
FOLT: Fast Multiple Object Tracking from UAV-captured Videos Based on Optical Flow
Given the extracted flow, the flow-guided feature augmentation is designed to augment the object detection feature based on its optical flow, which improves the detection of small objects.
Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization
Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed.
Evaluating Large Language Models for Radiology Natural Language Processing
The rise of large language models (LLMs) has marked a pivotal shift in the field of natural language processing (NLP).
Efficient Prediction of Peptide Self-assembly through Sequential and Graphical Encoding
In recent years, there has been an explosion of research on the application of deep learning to the prediction of various peptide properties, due to the significant development and market potential of peptides.
MMBench: Is Your Multi-modal Model an All-around Player?
In response to these challenges, we propose MMBench, a novel multi-modality benchmark.
Ranked #1 on
Visual Question Answering
on MMBench
Real-time Workload Pattern Analysis for Large-scale Cloud Databases
This has led to the increasing volume of database workloads, which provides the opportunity for pattern analysis.
Review of Large Vision Models and Visual Prompt Engineering
This review aims to summarize the methods employed in the computer vision domain for large vision models and visual prompt engineering, exploring the latest advancements in visual prompt engineering.
OCBEV: Object-Centric BEV Transformer for Multi-View 3D Object Detection
Multi-view 3D object detection is becoming popular in autonomous driving due to its high effectiveness and low cost.
BUOL: A Bottom-Up Framework with Occupancy-aware Lifting for Panoptic 3D Scene Reconstruction From A Single Image
The 3D voxels are then refined and grouped into 3D instances according to the predicted 2D instance centers.
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers
Although extensively studied for unimodal models, the acceleration for multimodal models, especially the vision-language Transformers, is relatively under-explored.
Prompt Engineering for Healthcare: Methodologies and Applications
Prompt engineering is a critical technique in the field of natural language processing that involves designing and optimizing the prompts used to input information into models, aiming to enhance their performance on specific tasks.
ImpressionGPT: An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT
The 'Impression' section of a radiology report is a critical basis for communication between radiologists and other physicians, and it is typically written by radiologists based on the 'Findings' section.
V3Det: Vast Vocabulary Visual Detection Dataset
2) Hierarchical Category Organization: The vast vocabulary of V3Det is organized by a hierarchical category tree which annotates the inclusion relationship among categories, encouraging the exploration of category relationships in vast and open vocabulary object detection.
Dense Distinct Query for End-to-End Object Detection
Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments.
Ranked #3 on
Object Detection
on CrowdHuman (full body)
Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation
Referring image segmentation segments an image from a language expression.
Knowledge-Enhanced Semi-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoT
This is a new yet practical scenario in federated learning, i. e., labels-at-server semi-supervised federated learning (SemiFL).
Deep Generative Mixture Model for Robust Imbalance Classification
In this paper, a deep generative classifier is proposed to mitigate this issue via both model perturbation and data perturbation.
Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
Specifically, we construct a negative-sample-free triplet steam structure that is composed of an anchor stream without any masking, a spatial masking stream with Central Spatial Masking (CSM), and a temporal masking stream with Motion Attention Temporal Masking (MATM).
Siamese transformer with hierarchical concept embedding for fine-grained image recognition
In particular, one subnetwork is for coarse-scale patches to learn the discriminative regions with the aid of the innate multi-head self-attention mechanism of the transformer.
UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers
Real-world data contains a vast amount of multimodal information, among which vision and language are the two most representative modalities.
Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant
Semi-Supervised Semantic Segmentation aims at training the segmentation model with limited labeled data and a large amount of unlabeled data.
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation
Recent advances in modeling 3D objects mostly rely on synthetic datasets due to the lack of large-scale realscanned 3D databases.
Self-supervised Domain Adaptation for Breaking the Limits of Low-quality Fundus Image Quality Enhancement
Most of the existing image enhancement methods mainly focus on improving the image quality by leveraging the guidance of high-quality images, which is difficult to be collected in medical applications.
Multi-Level Logit Distillation
Through this framework, the prediction alignment is not only conducted at the instance level, but also at the batch and class level, through which the student model learns instance prediction, input correlation, and category correlation simultaneously.
DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition
Graph convolution networks (GCN) have been widely used in skeleton-based action recognition.
Ranked #7 on
Skeleton Based Action Recognition
on NTU RGB+D
In Differential Privacy, There is Truth: On Vote Leakage in Ensemble Private Learning
When learning from sensitive data, care must be taken to ensure that training algorithms address privacy concerns.
Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstruction
Previous methods based on neural volume rendering mostly train a fully implicit model with MLPs, which typically require hours of training for a single scene.
An Understanding-Oriented Robust Machine Reading Comprehension Model
In this paper, we propose an understanding-oriented machine reading comprehension model to address three kinds of robustness issues, which are over sensitivity, over stability and generalization.
Deep Amortized Relational Model with Group-Wise Hierarchical Generative Process
In this paper, we propose Deep amortized Relational Model (DaRM) with group-wise hierarchical generative process for community discovery and link prediction on relational data (e. g., graph, network).
What Are Expected Queries in End-to-End Object Detection?
As both sparse and dense queries are imperfect, then \emph{what are expected queries in end-to-end object detection}?
PYSKL: Towards Good Practices for Skeleton Action Recognition
The toolbox supports a wide variety of skeleton action recognition algorithms, including approaches based on GCN and CNN.
Ranked #19 on
Skeleton Based Action Recognition
on NTU RGB+D 120
MINI: Mining Implicit Novel Instances for Few-Shot Object Detection
The offline mining mechanism leverages a self-supervised discriminative model to collaboratively mine implicit novel instances with a trained FSOD network.
Deep Understanding based Multi-Document Machine Reading Comprehension
Most existing multi-document machine reading comprehension models mainly focus on understanding the interactions between the input question and documents, but ignore following two kinds of understandings.
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image.
Generalized Referring Expression Segmentation
Image Segmentation
+2
Few-Shot Object Detection via Association and DIscrimination
1) In the association step, in contrast to implicitly leveraging multiple base classes, we construct a compact novel class feature space via explicitly imitating a specific base class feature space.
FedTriNet: A Pseudo Labeling Method with Three Players for Federated Semi-supervised Learning
In particular, we propose to use three networks and a dynamic quality control mechanism to generate high-quality pseudo labels for unlabeled data, which are added to the training set.
UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer
Specifically, the CTrans module is an alternate of the U-Net skip connections, which consists of a sub-module to conduct the multi-scale Channel Cross fusion with Transformer (named CCT) and a sub-module Channel-wise Cross-Attention (named CCA) to guide the fused multi-scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity.
Ranked #2 on
Medical Image Segmentation
on GlaS
(IoU metric)
FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
Most existing FedSSL methods focus on the classical scenario, i. e, the labeled and unlabeled data are stored at the client side.
Adaptively Optimize Content Recommendation Using Multi Armed Bandit Algorithms in E-commerce
Second, we compare the accumulative rewards of the three MAB algorithms with more than 1, 000 trials using actual historical A/B test datasets.
Cluster-Wise Hierarchical Generative Model for Deep Amortized Clustering
In this paper, we propose Cluster-wise Hierarchical Generative Model for deep amortized clustering (CHiGac).
Interpretable Image Recognition by Constructing Transparent Embedding Space
This plug-in embedding space is spanned by transparent basis concepts which are constructed on the Grassmann manifold.
CARAFE++: Unified Content-Aware ReAssembly of FEatures
Feature reassembly, i. e. feature downsampling and upsampling, is a key operation in a number of modern convolutional network architectures, e. g., residual networks and feature pyramids.
CodeCMR: Cross-Modal Retrieval For Function-Level Binary Source Code Matching
We adopt Deep Pyramid Convolutional Neural Network (DPCNN) for source code feature extraction and Graph Neural Network (GNN) for binary code feature extraction.
TEST_POSITIVE at W-NUT 2020 Shared Task-3: Joint Event Multi-task Learning for Slot Filling in Noisy Text
The competition of extracting COVID-19 events from Twitter is to develop systems that can automatically extract related events from tweets.
Extracting COVID-19 Events from Twitter
Language Modelling
+5
Texture Memory-Augmented Deep Patch-Based Image Inpainting
By bringing together the best of both paradigms, we propose a new deep inpainting framework where texture generation is guided by a texture memory of patch samples extracted from unmasked regions.
Active Learning for Product Type Ontology Enhancement in E-commerce
Entity-based semantic search has been widely adopted in modern search engines to improve search accuracy by understanding users' intent.
Seesaw Loss for Long-Tailed Instance Segmentation
Instances of head classes dominate a long-tailed dataset and they serve as negative samples of tail categories.
MMFashion: An Open-Source Toolbox for Visual Fashion Analysis
This toolbox supports a wide spectrum of fashion analysis tasks, including Fashion Attribute Prediction, Fashion Recognition and Retrieval, Fashion Landmark Detection, Fashion Parsing and Segmentation and Fashion Compatibility and Recommendation.
FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC
In MEC, edge nodes would not like to voluntarily participate in learning, and they differ in the provision of multi-dimensional resources, both of which might deteriorate the performance of federated learning.
Side-Aware Boundary Localization for More Precise Object Detection
To tackle the difficulty of precise localization in the presence of displacements with large variance, we further propose a two-step localization scheme, which first predicts a range of movement through bucket prediction and then pinpoints the precise position within the predicted bucket.
On the Robustness of the Backdoor-based Watermarking in Deep Neural Networks
We focus on backdoor-based watermarking and propose two -- a black-box and a white-box -- attacks that remove the watermark.
MMDetection: Open MMLab Detection Toolbox and Benchmark
In this paper, we introduce the various features of this toolbox.
CARAFE: Content-Aware ReAssembly of FEatures
CARAFE introduces little computational overhead and can be readily integrated into modern network architectures.
Ranked #3 on
Feature Upsampling
on ImageNet
Hierarchical Attention Generative Adversarial Networks for Cross-domain Sentiment Classification
Cross-domain sentiment classification (CDSC) is an importance task in domain adaptation and sentiment classification.
Hybrid Task Cascade for Instance Segmentation
In exploring a more effective approach, we find that the key to a successful instance segmentation cascade is to fully leverage the reciprocal relationship between detection and segmentation.
Ranked #32 on
Object Detection
on COCO-O
Region Proposal by Guided Anchoring
State-of-the-art detectors mostly rely on a dense anchoring scheme, where anchors are sampled uniformly over the spatial domain with a predefined set of scales and aspect ratios.
Ranked #1 on
Region Proposal
on COCO test-dev
Optimizing Video Object Detection via a Scale-Time Lattice
High-performance object detection relies on expensive convolutional networks to compute features, often leading to significant challenges in applications, e. g. those that require detecting objects from video streams in real time.
