Search Results for author:
Found 95 papers, 58 papers with code
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings.
Ranked #2 on
Visual Question Answering
on BenchLMM
Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation
Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches.
Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models
Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks.
Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions
Due to the inherent flexibility of prompting, foundation models have emerged as the predominant force in the fields of natural language processing and computer vision.
Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation
Mamba-UNet adopts a pure Visual Mamba (VMamba)-based encoder-decoder structure, infused with skip connections to preserve spatial information across different scales of the network.
AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?
With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater variability on many benchmark datasets.
Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey
In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm.
PokeBNN: A Binary Pursuit of Lightweight Accuracy
In order to enable joint optimization of the cost together with accuracy, we define arithmetic computation effort (ACE), a hardware- and energy-inspired cost metric for quantized and binarized networks.
Ranked #1 on
Binarization
on ImageNet
Binarized Neural Machine Translation
In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind.
MyGO: Discrete Modality Information as Fine-Grained Tokens for Multi-modal Knowledge Graph Completion
To overcome their inherent incompleteness, multi-modal knowledge graph completion (MMKGC) aims to discover unobserved knowledge from given MMKGs, leveraging both structural information from the triples and multi-modal information of the entities.
Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications
Recent work shows that it provides potential benefits for machine learning models by incorporating the physical prior and collected data, which makes the intersection of machine learning and physics become a prevailing paradigm.
PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs
In addition to providing a standardized means of assessing performance, PINNacle also offers an in-depth analysis to guide future research, particularly in areas such as domain decomposition methods and loss reweighting for handling multi-scale problems and complex geometry.
Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering
Deploying large language models (LLMs) to real scenarios for domain-specific question answering (QA) is a key thrust for LLM applications, which poses numerous challenges, especially in ensuring that responses are both accommodating to user requirements and appropriately leveraging domain-specific knowledge bases.
Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context
Conversations have an intrinsic one-to-many property, which means that multiple responses can be appropriate for the same dialog context.
Ranked #6 on
End-To-End Dialogue Modelling
on MULTIWOZ 2.0
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond
In this study, we explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents.
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
To address this, we introduce Embodied-Instruction-Evolution (EIE), an automatic framework for synthesizing instruction tuning examples in multimodal embodied environments.
Making Large Language Models Perform Better in Knowledge Graph Completion
In this paper, we explore methods to incorporate structural information into the LLMs, with the overarching goal of facilitating structure-aware reasoning.
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining
In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories.
How Robust is Google's Bard to Adversarial Image Attacks?
By attacking white-box surrogate vision encoders or MLLMs, the generated adversarial examples can mislead Bard to output wrong image descriptions with a 22% success rate based solely on the transferability.
FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations
We design an efficient FPGA-based accelerator for our novel BNN model that supports the fractional activations.
Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
The proposed approach is applicable to a variety of DNN architectures and significantly reduces the computational cost of DNN execution with almost no accuracy loss.
Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph
On the one hand, sparse KGs contain less information, which makes it difficult for the model to choose correct paths.
MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid
Multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) whose entities are associated with relevant images.
Ranked #1 on
Entity Alignment
on FBYG15k
(using extra training data)
Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection
Differing from vanilla prototypical networks simply computing event prototypes by averaging, which only consume event mentions once, our model is more robust and is capable of distilling contextual information from event mentions for multiple times due to the multi-hop mechanism of DMNs.
Rethinking Model Ensemble in Transfer-based Adversarial Attacks
It is widely recognized that deep learning models lack robustness to adversarial examples.
Autonomous Evaluation and Refinement of Digital Agents
We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control.
ML-Bench: Evaluating Large Language Models for Code Generation in Repository-Level Machine Learning Tasks
While Large Language Models (LLMs) have demonstrated proficiency in code generation benchmarks, translating these results into practical development scenarios - where leveraging existing repository-level libraries is the norm - remains challenging.
Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation
To solve the above problem, in this work, we propose a novel method called Adversarial and Contrastive Variational Autoencoder (ACVAE) for sequential recommendation.
Dual-Task Mutual Learning for Semi-Supervised Medical Image Segmentation
The success of deep learning methods in medical image segmentation tasks usually requires a large amount of labeled data.
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring
On the ALFRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%.
Tele-Knowledge Pre-training for Fault Analysis
In this work, we share our experience on tele-knowledge pre-training for fault analysis, a crucial task in telecommunication applications that requires a wide range of knowledge normally found in both machine log data and product documents.
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment
As a crucial extension of entity alignment (EA), multi-modal entity alignment (MMEA) aims to identify identical entities across disparate knowledge graphs (KGs) by exploiting associated visual information.
Ranked #1 on
Multi-modal Entity Alignment
on UMVM-oea-d-w-v2
(using extra training data)
A Probabilistic End-To-End Task-Oriented Dialog Model with Latent Belief States towards Semi-Supervised Learning
In this paper we aim at alleviating the reliance on belief state labels in building end-to-end dialog systems, by leveraging unlabeled dialog data towards semi-supervised learning.
Ranked #2 on
End-To-End Dialogue Modelling
on MULTIWOZ 2.1
Unrestricted Adversarial Attacks on ImageNet Competition
Many works have investigated the adversarial attacks or defenses under the settings where a bounded and imperceptible perturbation can be added to the input.
Logic Synthesis Meets Machine Learning: Trading Exactness for Generalization
If the function is incompletely-specified, the implementation has to be true only on the care set.
Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability
However, we find in experiments that many paths given by these models are actually unreasonable, while little works have been done on interpretability evaluation for them.
DANLI: Deliberative Agent for Following Natural Language Instructions
These reactive agents are insufficient for long-horizon complex tasks.
Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Driving
However, there still lacks a systematic understanding of the robustness of these vision-dependent BEV models, which is closely related to the safety of autonomous driving systems.
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding
However, evaluations only based on end task performance shed little light on machines' true ability in language understanding and reasoning.
Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion
To address the mentioned problems, we propose Adaptive Multi-modal Fusion and Modality Adversarial Training (AdaMF-MAT) to unleash the power of imbalanced modality information for MMKGC.
Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs
In this paper, we propose GraphEval to evaluate an LLM's performance using a substantially large test dataset.
CausE: Towards Causal Knowledge Graph Embedding
Knowledge graph embedding (KGE) focuses on representing the entities and relations of a knowledge graph (KG) into the continuous vector spaces, which can be employed to predict the missing triples to achieve knowledge graph completion (KGC).
Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework
In this paper, we instantiate our framework with an attack algorithm named Textual Projected Gradient Descent (T-PGD).
Understanding Hyperdimensional Computing for Parallel Single-Pass Learning
Using representation theory, we characterize which similarity matrices can be "expressed" by finite group VSA hypervectors, and we show how these VSAs can be constructed.
Diffusion Noise Feature: Accurate and Fast Generated Image Detection
DNF is extracted from the estimated noise generated during the inverse diffusion process.
The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework
In this work, to evaluate models' ability to accurately embed entities within MMKGs, we focus on two widely researched tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA).
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model.
Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding
Negative sampling (NS) is widely used in knowledge graph embedding (KGE), which aims to generate negative triples to make a positive-negative contrast during training.
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?
Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world.
Paraphrase Augmented Task-Oriented Dialog Generation
Neural generative models have achieved promising performance on dialog generation tasks if given a huge data set.
Interpretable and Low-Resource Entity Matching via Decoupling Feature Learning from Decision Making
Using a set of comparison features and a limited amount of annotated data, KAT Induction learns an efficient decision tree that can be interpreted by generating entity matching rules whose structure is advocated by domain experts.
D-Net: Dynamic Large Kernel with Dynamic Feature Fusion for Volumetric Medical Image Segmentation
D-Net is able to effectively utilize a multi-scale large receptive field and adaptively harness global contextual information.
Face Alignment Assisted by Head Pose Estimation
In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation.
MACO: A Modality Adversarial and Contrastive Framework for Modality-missing Multi-modal Knowledge Graph Completion
Nevertheless, existing methods emphasize the design of elegant KGC models to facilitate modality interaction, neglecting the real-life problem of missing modalities in KGs.
Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?
Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks.
U-Net-and-a-half: Convolutional network for biomedical image segmentation using multiple expert-driven annotations
If more than a single expert is involved in the annotation of the same images, then the inter-expert agreement is not necessarily perfect, and no single expert annotation can precisely capture the so-called ground truth of the regions of interest on all images.
A Latent Variable Approach to Gaussian Process Modeling with Qualitative and Quantitative Factors
Computer simulations often involve both qualitative and numerical inputs.
Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning
An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing.
Elastic CRFs for Open-ontology Slot Filling
Second, the one-hot encoding of slot labels ignores the semantic meanings and relations for slots, which are implicit in their natural language descriptions.
Data-Centric Mixed-Variable Bayesian Optimization For Materials Design
To this end, we present a data-centric, mixed-variable Bayesian Optimization framework that integrates data from literature, experiments, and simulations for knowledge discovery and computational materials design.
Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables
We present in this paper the integration of a novel latent-variable (LV) approach for mixed-variable GP modeling with the BO framework for materials design.
Covariance Estimation for Matrix-valued Data
Covariance estimation for matrix-valued data has received an increasing interest in applications.
Bridging 2D and 3D Segmentation Networks for Computation Efficient Volumetric Medical Image Segmentation: An Empirical Study of 2.5D Solutions
Despite these works lead to improvements on a variety of segmentation tasks, to the best of our knowledge, there has not previously been a large-scale empirical comparison of these methods.
Distributed Representations of Entities in Open-World Knowledge Graphs
DAN leverages neighbor context as the query vector to score the neighbors of an entity, thereby distributing the entity semantics only among its neighbor embeddings.
Cascaded Convolutional Neural Network for Automatic Myocardial Infarction Segmentation from Delayed-Enhancement Cardiac MRI
Automatic segmentation of myocardial contours and relevant areas like infraction and no-reflow is an important step for the quantitative evaluation of myocardial infarction.
Exploiting Shared Knowledge from Non-COVID Lesions for Annotation-Efficient COVID-19 CT Lung Infection Segmentation
In this paper, we propose a novel relation-driven collaborative learning model to exploit shared knowledge from non-COVID lesions for annotation-efficient COVID-19 CT lung infection segmentation.
Exact Recovery of Community Structures Using DeepWalk and Node2vec
Random-walk based network embedding algorithms like DeepWalk and node2vec are widely used to obtain Euclidean representation of the nodes in a network prior to performing downstream inference tasks.
Improving Conversational Recommendation System by Pretraining on Billions Scale of Knowledge Graph
We first construct a billion-scale conversation knowledge graph (CKG) from information about users, items and conversations, and then pretrain CKG by introducing knowledge graph embedding method and graph convolution network to encode semantic and structural information respectively. To make the CTR prediction model sensible of current state of users and the relationship between dialogues and items, we introduce user-state and dialogue-interaction representations based on pre-trained CKG and propose K-DCN. In K-DCN, we fuse the user-state representation, dialogue-interaction representation and other normal feature representations via deep cross network, which will give the rank of candidate items to be recommended. We experimentally prove that our proposal significantly outperforms baselines and show it's real application in Alime.
Deep Learning methods for automatic evaluation of delayed enhancement-MRI. The results of the EMIDEC challenge
The publicly available database consists of 150 exams divided into 50 cases with normal MRI after injection of a contrast agent and 100 cases with myocardial infarction (and then with a hyperenhanced area on DE-MRI), whatever their inclusion in the cardiac emergency department.
BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining
Neural network robustness has become a central topic in machine learning in recent years.
Adversarial Semantic Contour for Object Detection
To address this issue, we propose a novel method of Adversarial Semantic Contour (ASC) guided by object contour as prior.
SAU-Net: Efficient 3D Spine MRI Segmentation Using Inter-Slice Attention
Accurate segmentation of spine Magnetic Resonance Imaging (MRI) is highly demanded in morphological research, quantitative analysis, and diseases identification, such as spinal canal stenosis, disc herniation and degeneration.
Uncertainty-Guided Mutual Consistency Learning for Semi-Supervised Medical Image Segmentation
In this paper, we propose a novel uncertainty-guided mutual consistency learning framework to effectively exploit unlabeled data by integrating intra-task consistency learning from up-to-date predictions for self-ensembling and cross-task consistency learning from task-level regularization to exploit geometric shape information.
Perturbation Analysis of Randomized SVD and its Applications to High-dimensional Statistics
We first derive upper bounds for the $\ell_2$ (spectral norm) and $\ell_{2\to\infty}$ (maximum row-wise $\ell_2$ norm) distances between the approximate singular vectors of $\hat{\mathbf{M}}$ and the true singular vectors of the signal matrix $\mathbf{M}$.
Knowledge Graph Completion with Pre-trained Multimodal Transformer and Twins Negative Sampling
Twins negative sampling is suitable for multimodal scenarios and could align different embeddings for entities.
To Make Yourself Invisible with Adversarial Semantic Contours
In this paper, we propose Adversarial Semantic Contour (ASC), an MAP estimate of a Bayesian formulation of sparse attack with a deceived prior of object contour.
DA-VEGAN: Differentiably Augmenting VAE-GAN for microstructure reconstruction from extremely small data sets
Microstructure reconstruction is an important and emerging field of research and an essential foundation to improving inverse computational materials engineering (ICME).
Mastering Asymmetrical Multiplayer Game with Multi-Agent Asymmetric-Evolution Reinforcement Learning
We propose asymmetric-evolution training (AET), a novel multi-agent reinforcement learning framework that can train multiple kinds of agents simultaneously in AMP game.
Learning to Detect Novel and Fine-Grained Acoustic Sequences Using Pretrained Audio Representations
This work investigates pretrained audio representations for few shot Sound Event Detection.
Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Survey
Due to the flexibility of prompting, foundation models have become the dominant force in the domains of natural language processing and image generation.
Robust Detection of Lead-Lag Relationships in Lagged Multi-Factor Models
In multivariate time series systems, key insights can be obtained by discovering lead-lag relationships inherent in the data, which refer to the dependence between two time series shifted in time relative to one another, and which can be leveraged for the purposes of control, forecasting or clustering.
Dynamic Time Warping for Lead-Lag Relationships in Lagged Multi-Factor Models
In multivariate time series systems, lead-lag relationships reveal dependencies between time series when they are shifted in time relative to each other.
Competitive Ensembling Teacher-Student Framework for Semi-Supervised Left Atrium MRI Segmentation
Semi-supervised learning has greatly advanced medical image segmentation since it effectively alleviates the need of acquiring abundant annotations from experts and utilizes unlabeled data which is much easier to acquire.
Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectification
The Segment Anything Model (SAM) has recently emerged as a groundbreaking foundation model for prompt-driven image segmentation tasks.
SemiSAM: Exploring SAM for Enhancing Semi-Supervised Medical Image Segmentation with Extremely Limited Annotations
Semi-supervised learning has attracted much attention due to its less dependence on acquiring abundant annotations from experts compared to fully supervised methods, which is especially important for medical image segmentation which typically requires intensive pixel/voxel-wise labeling by domain experts.
Another Way to the Top: Exploit Contextual Clustering in Learned Image Coding
While convolution and self-attention are extensively used in learned image compression (LIC) for transform coding, this paper proposes an alternative called Contextual Clustering based LIC (CLIC) which primarily relies on clustering operations and local attention for correlation characterization and compact representation of an image.
Trainable Fixed-Point Quantization for Deep Learning Acceleration on FPGAs
Quantization is a crucial technique for deploying deep learning models on resource-constrained devices, such as embedded FPGAs.
Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science
Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines.
UMOEA/D: A Multiobjective Evolutionary Algorithm for Uniform Pareto Objectives based on Decomposition
Multiobjective optimization (MOO) is prevalent in numerous applications, in which a Pareto front (PF) is constructed to display optima under various preferences.
Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms
However, different metrics lead to divergent and even contradictory results in various contexts.
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens.
Ranked #1 on
Generalized Referring Expression Segmentation
on gRefCOCO
(using extra training data)
Causal Language Modeling
Generalized Referring Expression Segmentation
+2
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs.
Theoretical Bound-Guided Hierarchical VAE for Neural Image Codecs
Recent studies reveal a significant theoretical link between variational autoencoders (VAEs) and rate-distortion theory, notably in utilizing VAEs to estimate the theoretical upper bound of the information rate-distortion function of images.
Surveying Attitudinal Alignment Between Large Language Models Vs. Humans Towards 17 Sustainable Development Goals
This study conducts a comprehensive review and analysis of the existing literature on the attitudes of LLMs towards the 17 SDGs, emphasizing the comparison between their attitudes and support for each goal and those of humans.
Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Models
Medical image segmentation plays an important role in many image-guided clinical approaches.
