Search Results for author:
Found 37 papers, 27 papers with code
Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation
Adversarial Training (AT) is widely recognized as an effective approach to enhance the adversarial robustness of Deep Neural Networks.
AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning
The recent progress of large language model agents has opened new possibilities for automating tasks through graphical user interfaces (GUIs), especially in mobile environments where intelligent interaction can greatly enhance usability.
SPPSFormer: High-quality Superpoint-based Transformer for Roof Plane Instance Segmentation from Point Clouds
Transformers have been seldom employed in point cloud roof plane instance segmentation, which is the focus of this study, and existing superpoint Transformers suffer from limited performance due to the use of low-quality superpoints.
MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding
Video temporal understanding is crucial for multimodal large language models (MLLMs) to reason over events in videos.
Visual Abstract Thinking Empowers Multimodal Reasoning
Images usually convey richer detail than text, but often include redundant information which potentially downgrades multimodal reasoning performance.
ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editing
In contrast, small-scale models, including chart-domain models, struggle both with following editing instructions and generating overall chart images, underscoring the need for further development in this area.
Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model
To address this gap, we conduct a comprehensive and systematic safety evaluation of 11 MLRMs across 5 benchmarks and unveil prevalent safety degradation phenomena in most advanced models.
AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
Specifically, we first design mapping function between models to apply model merging on MLLMs with different architecture.
DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding
Human experts excel at fine-grained visual discrimination by leveraging domain knowledge to refine perceptual features, a capability that remains underdeveloped in current Multimodal Large Language Models (MLLMs).
Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition
These curated samples are subsequently used for large-scale multimodal pre-training, completing a self-learning cycle that strengthens the model's cognitive foundation.
DNA Origami Nanostructures Observed in Transmission Electron Microscopy Images can be Characterized through Convolutional Neural Networks
Artificial intelligence (AI) models remain an emerging strategy to accelerate materials design and development.
How Do Multimodal Large Language Models Handle Complex Multimodal Reasoning? Placing Them in An Extensible Escape Game
However, existing evaluations primarily assess the final task completion, often degrading assessments to isolated abilities such as visual grounding and visual question answering.
ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
: (1) Low executability and poor restoration of chart details in the generated code and (2) Lack of large-scale and diverse training data.
Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models
The recent advancement of Multimodal Large Language Models (MLLMs) has significantly improved their fine-grained perception of single images and general comprehension across multiple images.
Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency
Despite achieving some progress, they have a low attack success rate on commercial closed-source MLLMs.
LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer
To address this issue, we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window transformer that enables capturing diverse visual granularity by constructing and integrating a high-resolution feature pyramid.
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs.
PlaneSAM: Multimodal Plane Instance Segmentation Using the Segment Anything Model
Based on EfficientSAM, a fast version of SAM, we propose a plane instance segmentation network called PlaneSAM, which can fully integrate the information of the RGB bands (spectral bands) and the D band (geometric band), thereby improving the effectiveness of plane instance segmentation in a multimodal manner.
ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models
Active perception, a crucial human capability, involves setting a goal based on the current understanding of the environment and performing actions to achieve that goal.
Co-Fix3D: Enhancing 3D Object Detection with Collaborative Refinement
3D object detection in driving scenarios faces the challenge of complex road environments, which can lead to the loss or incompleteness of key features, thereby affecting perception performance.
PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformer
Amidst the surge in deep learning-based password guessing models, challenges of generating high-quality passwords and reducing duplicate passwords persist.
Goldfish: An Efficient Federated Unlearning Framework
To address the challenge of low validity in existing machine unlearning algorithms, we propose a novel loss function.
CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models
Multimodal large language models (MLLMs) have demonstrated promising results in a variety of tasks that combine vision and language.
Model Composition for Multimodal Large Language Models
In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model.
Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
With the bloom of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) that incorporate LLMs with pre-trained vision models have recently demonstrated impressive performance across diverse vision-language tasks.
Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge not only in natural language tasks, but also in some vision-language tasks such as open-domain knowledge-based visual question answering (OK-VQA).
Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
However, existing visual instruction tuning methods only utilize image-language instruction data to align the language and image modalities, lacking a more fine-grained cross-modal alignment.
Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learning
In this paper, we introduce a novel deep learning framework, called Distributional Graphormer (DiG), in an attempt to predict the equilibrium distribution of molecular systems.
Weakly Supervised Vision-and-Language Pre-training with Relative Representations
Weakly supervised vision-and-language pre-training (WVLP), which learns cross-modal representations with limited cross-modal supervision, has been shown to effectively reduce the data cost of pre-training while maintaining decent performance on downstream tasks.
CG-SSD: Corner Guided Single Stage 3D Object Detection from LiDAR Point Cloud
At present, the anchor-based or anchor-free models that use LiDAR point clouds for 3D object detection use the center assigner strategy to infer the 3D bounding boxes.
Efficient DNA-driven nanocavities for approaching quasi-deterministic strong coupling to a few fluorophores
Strong coupling between light and matter is the foundation of promising quantum photonic devices such as deterministic single photon sources, single atom lasers and photonic quantum gates, which consist of an atom and a photonic cavity.
Optics Quantum Physics
AtomSets -- A Hierarchical Transfer Learning Framework for Small and Large Materials Datasets
Here we leverage the transfer learning concept and the graph network deep learning framework and develop the AtomSets machine learning framework for consistent high model accuracy at both small and large materials data.
Feature Engineering
Transfer Learning
Materials Science
Mask-Align: Self-Supervised Neural Word Alignment
Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks.
Holistic Combination of Structural and Textual Code Information for Context based API Recommendation
APIRec-CST is a deep learning model that combines the API usage with the text information in the source code based on an API Context Graph Network and a Code Token Network that simultaneously learn structural and textual features for API recommendation.
Learning Properties of Ordered and Disordered Materials from Multi-fidelity Data
Predicting the properties of a material from the arrangement of its atoms is a fundamental goal in materials science.
Materials Science Disordered Systems and Neural Networks
Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
Similarly, we show that MEGNet models trained on $\sim 60, 000$ crystals in the Materials Project substantially outperform prior ML models in the prediction of the formation energies, band gaps and elastic moduli of crystals, achieving better than DFT accuracy over a much larger data set.
Ranked #5 on
Formation Energy
on Materials Project
Drug Discovery
Formation Energy
Materials Science
Computational Physics
Automated Generation and Ensemble-Learned Matching of X-ray Absorption Spectra
We report the development of XASdb, a large database of computed reference X-ray absorption spectra (XAS), and a novel Ensemble-Learned Spectra IdEntification (ELSIE) algorithm for the matching of spectra.
Materials Science
