Search Results for author:
Found 294 papers, 137 papers with code
Chinese Grammatical Error Correction Based on Hybrid Models with Data Augmentation
A better Chinese Grammatical Error Diagnosis (CGED) system for automatic Grammatical Error Correction (GEC) can benefit foreign Chinese learners and lower Chinese learning barriers.
DoTAT: A Domain-oriented Text Annotation Tool
Secondly, the tool provides annotation of events, nested event and nested entity, which are frequently required in domain-related text structuring tasks.
Make Your Training Flexible: Towards Deployment-Efficient Video Models
By making the sampling grid flexible and leveraging token selection, it is easily adopted in most popular video training frameworks, boosting model robustness with nearly no additional cost.
Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation
Earth observation (EO) data features diverse sensing platforms with varying spectral bands, spatial resolutions, and sensing modalities.
GeoLangBind: Unifying Earth Observation with Agglomerative Vision-Language Foundation Models
Earth observation (EO) data, collected from diverse sensors with varying imaging principles, present significant challenges in creating unified analytical frameworks.
CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation
The construction of interleaved multimodal multi-step reasoning chains, which utilize critical visual region tokens, extracted from intermediate reasoning steps, as supervisory signals.
LMHLD: A Large-scale Multi-source High-resolution Landslide Dataset for Landslide Detection based on Deep Learning
However, DL models rely heavily on high-quality labeled landslide data for strong feature extraction capabilities.
WhatELSE: Shaping Narrative Spaces at Configurable Level of Abstraction for AI-bridged Interactive Storytelling
Generative AI significantly enhances player agency in interactive narratives (IN) by enabling just-in-time content generation that adapts to player actions.
OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework
OccProphet reduces 58\%$\sim$78\% of the computational cost with a 2. 6$\times$ speedup compared with the state-of-the-art Cam4DOcc.
Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
The burgeoning computational demands for training large language models (LLMs) necessitate efficient methods, including quantized training, which leverages low-bit arithmetic operations to reduce costs.
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing
Large Language Models (LLMs) are widely applied in decision making, but their deployment is threatened by jailbreak attacks, where adversarial users manipulate model behavior to bypass safety measures.
CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for Zero-Shot Customized Video Diffusion Transformers
Customized generation has achieved significant progress in image synthesis, yet personalized video generation remains challenging due to temporal inconsistencies and quality degradation.
Learning Generalizable Features for Tibial Plateau Fracture Segmentation Using Masked Autoencoder and Limited Annotations
Accurate automated segmentation of tibial plateau fractures (TPF) from computed tomography (CT) requires large amounts of annotated data to train deep learning models, but obtaining such annotations presents unique challenges.
Fuzzy-aware Loss for Source-free Domain Adaptation in Visual Emotion Recognition
Source-free domain adaptation in visual emotion recognition (SFDA-VER) is a highly challenging task that requires adapting VER models to the target domain without relying on source data, which is of great significance for data privacy protection.
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
This paper aims to improve the performance of video multimodal large language models (MLLM) via long and rich context (LRC) modeling.
Ranked #11 on
Referring Video Object Segmentation
on MeViS
DiffVSR: Enhancing Real-World Video Super-Resolution with Diffusion Models for Advanced Visual Quality and Temporal Consistency
Diffusion models have demonstrated exceptional capabilities in image generation and restoration, yet their application to video super-resolution faces significant challenges in maintaining both high fidelity and temporal consistency.
Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models
We present Vchitect-2. 0, a parallel transformer architecture designed to scale up video diffusion models for large-scale text-to-video generation.
MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning
Given visual segments and textual descriptions of events, MECD identifies the causal associations between these events to derive a comprehensive and structured event-level video causal graph explaining why and how the result event occurred.
Salient Region Matching for Fully Automated MR-TRUS Registration
The registration of magnetic resonance (MR) and transrectal ultrasound (TRUS) can provide guidance for the targeted biopsy of prostate cancer.
SELMA3D challenge: Self-supervised learning for 3D light-sheet microscopy image segmentation
Combined with the progress in large-scale data analysis, driven by deep learning, these innovations empower researchers to rapidly investigate the morphological and functional properties of diverse biological samples.
Stochastically Constrained Best Arm Identification with Thompson Sampling
We consider the problem of the best arm identification in the presence of stochastic constraints, where there is a finite number of arms associated with multiple performance measures.
Interpretable Load Forecasting via Representation Learning of Geo-distributed Meteorological Factors
However, the difference in MF collected in various locations within a region may be significant, which poses a challenge in selecting the appropriate MF from numerous locations.
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
This paper introduces a Hierarchical visual token Compression (HiCo) method designed for high-fidelity representation and a practical context modeling system VideoChat-Flash tailored for multimodal long-sequence processing.
Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals though they give comprehensive perception and reasoning in a spectrum of vision applications.
Tracking the Feature Dynamics in LLM Training: A Mechanistic Study
Understanding training dynamics and feature evolution is crucial for the mechanistic interpretability of large language models (LLMs).
LLM Agent for Fire Dynamics Simulations
FoamPilot provides three core functionalities: code insight, case configuration and simulation evaluation.
QSM-RimDS: A detection and segmentation tool for paramagnetic rim lesions in multiple sclerosis
Deep learning-based QSM-RimNet can provide automated PRL detection, but this method does not provide rim segmentation for microglial density quantification and requires precise QSM lesion masks.
Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel
In this paper, we introduce a Self-Refining Data Flywheel (SRDF) that generates high-quality and large-scale navigational instruction-trajectory pairs by iteratively refining the data pool through the collaboration between two models, the instruction generator and the navigator, without any human-in-the-loop annotation.
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
We introduce InternVL 2. 5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2. 0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality.
Ranked #1 on
Video Question Answering
on NExT-QA
SyncVIS: Synchronized Video Instance Segmentation
Recent DETR-based methods have advanced the development of Video Instance Segmentation (VIS) through transformers' efficiency and capability in modeling spatial and temporal information.
Point Cloud Understanding via Attention-Driven Contrastive Learning
Recently Transformer-based models have advanced point cloud understanding by leveraging self-attention mechanisms, however, these methods often overlook latent information in less prominent regions, leading to increased sensitivity to perturbations and limited global comprehension.
Analysis of the impact of heterogeneous platoon for mixed traffic flow: control strategy, fuel consumption and emissions
Finally, numerical experiments were conducted to calculate the average fuel consumption and pollutant emissions of mixed traffic flow under different spacing control strategies, and the impact of platoon spacing control strategies on traffic flow fuel consumption and pollutant emissions was further analyzed.
SignEye: Traffic Sign Interpretation from Vehicle First-Person View
Meanwhile, we develop a traffic guidance assistant (TGA) scenario application to re-explore the role of traffic signs in ADS as a complement to popular autonomous technologies (such as obstacle perception).
Unsupervised Congestion Status Identification Using LMP Data
The congestion part of LMPs is spanned by certain row vectors of the power transfer distribution factor (PTDF) matrix, and the subspace attributes of an LMP vector uniquely are found to reflect the instantaneous congestion status of all the transmission lines.
A Novel Combined Data-Driven Approach for Electricity Theft Detection
One technique is the Maximum Information Coefficient (MIC), which can find the correlations between the non-technical loss (NTL) and a certain electricity behavior of the consumer.
Coherent Hierarchical Probabilistic Forecasting of Electric Vehicle Charging Demand
The growing penetration of electric vehicles (EVs) significantly changes typical load curves in smart grids.
ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives
By looking inside bytes, the bit-level details of file fragments can be accessed, enabling a more accurate classification.
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format.
Ranked #7 on
Moment Retrieval
on Charades-STA
(using extra training data)
Multi-Level Speaker Representation for Target Speaker Extraction
In this work, we propose a multi-level speaker representation approach, from raw features to neural embeddings, to serve as the speaker reference cue.
Implicit neural representation for free-breathing MR fingerprinting (INR-MRF): co-registered 3D whole-liver water T1, water T2, proton density fat fraction, and R2* mapping
Purpose: To develop an MRI technique for free-breathing 3D whole-liver quantification of water T1, water T2, proton density fat fraction (PDFF), R2*.
Open World Object Detection: A Survey
Exploring new knowledge is a fundamental human ability that can be mirrored in the development of deep neural networks, especially in the field of object detection.
FiRework: Field Refinement Framework for Efficient Enhancement of Deformable Registration
In FiRework, we redesign the continuous deformation framework to mitigate the aforementioned errors.
Task-oriented Time Series Imputation Evaluation via Generalized Representers
Time series analysis is widely used in many fields such as power energy, economics, and transportation, including different tasks such as forecasting, anomaly detection, classification, etc.
Leverage Knowledge Graph and Large Language Model for Law Article Recommendation: A Case Study of Chinese Criminal Law
To address this issue, we propose an efficient law article recommendation approach utilizing a Knowledge Graph (KG) and a Large Language Model (LLM).
Evolving Multi-Scale Normalization for Time Series Forecasting under Distribution Shifts
Complex distribution shifts are the main obstacle to achieving accurate long-term time series forecasting.
XTRUST: On the Multilingual Trustworthiness of Large Language Models
Large language models (LLMs) have demonstrated remarkable capabilities across a range of natural language processing (NLP) tasks, capturing the attention of both practitioners and the broader public.
Towards Real-world Deployment of NILM Systems: Challenges and Practices
Non-intrusive load monitoring (NILM), as a key load monitoring technology, can much reduce the deployment cost of traditional power sensors.
Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras
However, adapting 3DGS to different camera models, particularly fisheye lenses, poses challenges due to the unique 3D to 2D projection calculation.
NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation
To alleviate the labor-intensive requirement of manual prompts, we introduce a Gaussian-Kernel Prompt Encoder (GKP-Encoder) to generate density maps driven by a single point, which guides segmentation predictions by mixing position prompts and semantic prompts.
A Survey of Embodied Learning for Object-Centric Robotic Manipulation
Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI.
Xinyu: An Efficient LLM-based System for Commentary Generation
To address the advanced requirements, we present an argument ranking model for arguments and establish a comprehensive evidence database that includes up-to-date events and classic books, thereby strengthening the substantiation of the evidence with retrieval augmented generation (RAG) technology.
HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model
Specifically, to achieve layer-wise clothing generation, we propose a dual-representation decoupling framework for generating clothing decoupled from the human body, in conjunction with an innovative multi-layer fusion volume rendering method.
G2Face: High-Fidelity Reversible Face Anonymization via Generative and Geometric Priors
This paper introduces G\textsuperscript{2}Face, which leverages both generative and geometric priors to enhance identity manipulation, achieving high-quality reversible face anonymization without compromising data utility.
PADetBench: Towards Benchmarking Physical Attacks against Object Detection
Moreover, physical dynamics and cross-domain transformation are challenging to strictly regulate in the real world, leading to unaligned evaluation and comparison, severely hindering the development of physically robust models.
FlashGS: Efficient 3D Gaussian Splatting for Large-scale and High-resolution Rendering
This work introduces FlashGS, an open-source CUDA Python library, designed to facilitate the efficient differentiable rasterization of 3D Gaussian Splatting through algorithmic and kernel-level optimizations.
SpectralEarth: Training Hyperspectral Foundation Models at Scale
To close this gap, we introduce SpectralEarth, a large-scale multi-temporal dataset designed to pretrain hyperspectral foundation models leveraging data from the Environmental Mapping and Analysis Program (EnMAP).
Context-aware knowledge graph framework for traffic speed forecasting using graph neural network
This study proposes a novel context-aware knowledge graph (CKG) framework to enhance traffic speed forecasting by effectively modeling spatial and temporal contexts.
ESP-MedSAM: Efficient Self-Prompting SAM for Universal Domain-Generalized Medical Image Segmentation
Finally, we design the Query-Decoupled Modality Decoder (QDMD) that leverages a one-to-one strategy to provide an independent decoding channel for every modality.
Internal Consistency and Self-Feedback in Large Language Models: A Survey
In this paper, we use a unified perspective of internal consistency, offering explanations for reasoning deficiencies and hallucinations.
ViLLa: Video Reasoning Segmentation with Large Language Model
To bridge the gap between image and video, in this work, we propose a new video segmentation task - video reasoning segmentation.
SGIFormer: Semantic-guided and Geometric-enhanced Interleaving Transformer for 3D Instance Segmentation
Specifically, the principle of our SMQ initialization scheme is to leverage the predicted voxel-wise semantic information to implicitly generate the scene-aware query, yielding adequate scene prior and compensating for the learnable query set.
MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis
Auto-regressive models have made significant progress in the realm of language generation, yet they do not perform on par with diffusion models in the domain of image synthesis.
Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition
Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; and the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets.
A Narrative Review of Image Processing Techniques Related to Prostate Ultrasound
In consequence, this survey provides a \textcolor{blue}{narrative } analysis of this field, outlining the evolution of image processing methods in the context of TRUS image analysis and meanwhile highlighting their relevant contributions.
Towards Multi-modality Fusion and Prototype-based Feature Refinement for Clinically Significant Prostate Cancer Classification in Transrectal Ultrasound
In this study, we propose a novel learning framework for clinically significant prostate cancer (csPCa) classification using multi-modality TRUS.
Symmetric Multi-Similarity Loss for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2024
In this report, we present our champion solution for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge in CVPR 2024.
Ranked #1 on
Multi-Instance Retrieval
on EPIC-KITCHENS-100
Encoding Matching Criteria for Cross-domain Deformable Image Registration
Most existing deep learning-based registration methods are trained on single-type images to address same-domain tasks. However, cross-domain deformable registration remains challenging. We argue that the tailor-made matching criteria in traditional registration methods is one of the main reason they are applicable in different domains. Motivated by this, we devise a registration-oriented encoder to model the matching criteria of image features and structural features, which is beneficial to boost registration accuracy and adaptability. Specifically, a general feature encoder (Encoder-G) is proposed to capture comprehensive medical image features, while a structural feature encoder (Encoder-S) is designed to encode the structural self-similarity into the global representation. Extensive experiments on images from three different domains prove the efficacy of the proposed method.
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset.
Data-driven Power Flow Linearization: Simulation
Subsequently, this paper evaluates a total of 44 methods, containing over 30 existing DPFL approaches, some innovative DPFL techniques, and several classic physics-driven power flow linearization methods for benchmarking.
Data-driven Power Flow Linearization: Theory
Further, this tutorial implements extensive numerical comparisons of all existing DPFL methods (40 methods in total) and four classic physics-driven approaches, focusing on their generalizability, applicability, accuracy, and computational efficiency.
Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining
Second, we revisit and explore cross-domain continual pretraining for both multispectral and SAR imagery, building efficient EO foundation models from strongest vision models such as DINOv2.
MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding
We introduce MM-Mixing, a multi-modal mixing alignment framework for 3D understanding.
Deep learning improved autofocus for motion artifact reduction and its application in quantitative susceptibility mapping
Purpose: To develop a pipeline for motion artifact correction in mGRE and quantitative susceptibility mapping (QSM).
StoryVerse: Towards Co-authoring Dynamic Plot with LLM-based Character Simulation via Narrative Planning
The process creates "living stories" that dynamically adapt to various game world states, resulting in narratives co-created by the author, character simulation, and player.
Prompt When the Animal is: Temporal Animal Behavior Grounding with Positional Recovery Training
Temporal grounding is crucial in multimodal learning, but it poses challenges when applied to animal behavior data due to the sparsity and uniform distribution of moments.
A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective
A comprehensive list of studies in this survey is publicly available in an active repository that continuously collects the latest work: https://github. com/HuaiyuanXu/3D-Occupancy-Perception.
On the Foundations of Earth and Climate Foundation Models
Foundation models have enormous potential in advancing Earth and climate sciences, however, current approaches may not be optimal as they focus on a few basic features of a desirable Earth and climate foundation model.
CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation
We study the potential of noisy labels y to pretrain semantic segmentation models in a multi-modal learning framework for geospatial applications.
Towards Cross-Scale Attention and Surface Supervision for Fractured Bone Segmentation in CT
The proposed method achieves an average DSC of 93. 36%, ASSD of 0. 85mm, 95HD of 7. 51mm.
Federated Graph Learning for EV Charging Demand Forecasting with Personalization Against Cyberattacks
To address these challenges, a federated graph learning approach involving multiple charging stations is proposed to collaboratively train a more generalized deep learning model for demand forecasting while capturing spatial correlations among various stations and enhancing robustness against potential attacks.
A Survey on Diffusion Models for Time Series and Spatio-Temporal Data
Conditioned models, on the other hand, utilize extra information to enhance performance and are similarly divided for both predictive and generative tasks.
ComposerX: Multi-Agent Symbolic Music Composition with LLMs
Music composition represents the creative side of humanity, and itself is a complex task that requires abilities to understand and generate information with long dependency and harmony constraints.
Exploring Kinetic Curves Features for the Classification of Benign and Malignant Breast Lesions in DCE-MRI
In this study, we propose to fully leverage the dynamic characteristics from the kinetic curves as well as the radiomic features to boost the classification accuracy of benign and malignant breast lesions.
Video sentence grounding with temporally global textual knowledge
Temporal sentence grounding involves the retrieval of a video moment with a natural language query.
Texture Classification Network Integrating Adaptive Wavelet Transform
Graves' disease is a common condition that is diagnosed clinically by determining the smoothness of the thyroid texture and its morphology in ultrasound images.
Contextual Embedding Learning to Enhance 2D Networks for Volumetric Image Segmentation
Accordingly, we propose a contextual embedding learning approach to facilitate 2D CNNs capturing spatial information properly.
ModeTv2: GPU-accelerated Motion Decomposition Transformer for Pairwise Optimization in Medical Image Registration
Deformable image registration plays a crucial role in medical imaging, aiding in disease diagnosis and image-guided interventions.
F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning
In this paper, we propose an exemplar-free approach--Forward-only Online Analytic Learning (F-OAL).
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue.
Ranked #1 on
Action Classification
on MIT
Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
The development of foundation models has revolutionized our ability to interpret the Earth's surface using satellite observational data.
Recurrent Drafter for Fast Speculative Decoding in Large Language Models
We present Recurrent Drafter (ReDrafter), an advanced speculative decoding approach that achieves state-of-the-art speedup for large language models (LLMs) inference.
Non-Intrusive Load Monitoring in Smart Grids: A Comprehensive Review
Non-Intrusive Load Monitoring (NILM) is pivotal in today's energy landscape, offering vital solutions for energy conservation and efficient management.
VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain.
Ranked #56 on
Action Classification
on Kinetics-400
Learning to Maximize Mutual Information for Chain-of-Thought Distillation
Knowledge distillation, the technique of transferring knowledge from large, complex models to smaller ones, marks a pivotal step towards efficient AI deployment.
AIO2: Online Correction of Object Labels for Deep Learning with Incomplete Annotation in Remote Sensing Image Segmentation
AIO2 utilizes a mean teacher model to enhance training robustness with noisy labels to both stabilize the training accuracy curve for fitting in ACT and provide pseudo labels for correction in O2C.
Task Specific Pretraining with Noisy Labels for Remote Sensing Image Segmentation
Compared to supervised deep learning, self-supervision provides remote sensing a tool to reduce the amount of exact, human-crafted geospatial annotations.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.
Multi-modality transrectal ultrasound video classification for identification of clinically significant prostate cancer
With the aim of effectively identifying prostate cancer, we propose a framework for the classification of clinically significant prostate cancer (csPCa) from multi-modality TRUS videos.
Pyramid Attention Network for Medical Image Registration
The advent of deep-learning-based registration networks has addressed the time-consuming challenge in traditional iterative methods. However, the potential of current registration networks for comprehensively capturing spatial relationships has not been fully explored, leading to inadequate performance in large-deformation image registration. The pure convolutional neural networks (CNNs) neglect feature enhancement, while current Transformer-based networks are susceptible to information redundancy. To alleviate these issues, we propose a pyramid attention network (PAN) for deformable medical image registration. Specifically, the proposed PAN incorporates a dual-stream pyramid encoder with channel-wise attention to boost the feature representation. Moreover, a multi-head local attention Transformer is introduced as decoder to analyze motion patterns and generate deformation fields. Extensive experiments on two public brain magnetic resonance imaging (MRI) datasets and one abdominal MRI dataset demonstrate that our method achieves favorable registration performance, while outperforming several CNN-based and Transformer-based registration networks. Our code is publicly available at https://github. com/JuliusWang-7/PAN.
Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks
Our study thus provides first-hand insights into using ChatGPT to fulfill software engineering tasks with real-world developers and motivates the need for novel interaction mechanisms that help developers effectively work with large language models to achieve desired outcomes.
Learning the Market: Sentiment-Based Ensemble Trading Agents
We propose and study the integration of sentiment analysis and deep reinforcement learning ensemble algorithms for stock trading by evaluating strategies capable of dynamically altering their active agent given the concurrent market environment.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Explaining Time Series via Contrastive and Locally Sparse Perturbations
Explaining multivariate time series is a compound challenge, as it requires identifying important locations in the time series and matching complex temporal patterns.
One for All: Toward Unified Foundation Models for Earth Vision
Current remote sensing foundation models typically specialize in a single modality or a specific spatial resolution range, limiting their versatility for downstream datasets.
HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling for Long-Term Forecasting
Time series forecasting is crucial and challenging in the real world.
Ranked #18 on
Time Series Forecasting
on ETTh1 (336) Multivariate
RimSet: Quantitatively Identifying and Characterizing Chronic Active Multiple Sclerosis Lesion on Quantitative Susceptibility Maps
We validated RimSet using simulated QSM images and an in vivo dataset of 172 MS subjects with 177 rim+ and 3986 rim-lesions.
Seamless and multi-resolution energy forecasting
The relationship between (i) errors in both time and frequency domains and (ii) operational value of the forecasts is analysed.
Guidelines in Wastewater-based Epidemiology of SARS-CoV-2 with Diagnosis
With the global spread and increasing transmission rate of SARS-CoV-2, more and more laboratories and researchers are turning their attention to wastewater-based epidemiology (WBE), hoping it can become an effective tool for large-scale testing and provide more ac-curate predictions of the number of infected individuals.
Dataset Distillation via Adversarial Prediction Matching
This ensures the memory efficiency of our method and provides a flexible tradeoff between time and memory budgets, allowing us to distil ImageNet-1K using a minimum of only 6. 5GB of GPU memory.
TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation
What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity.
QuickQuakeBuildings: Post-earthquake SAR-Optical Dataset for Quick Damaged-building Detection
Quick and automated earthquake-damaged building detection from post-event satellite imagery is crucial, yet it is challenging due to the scarcity of training data required to develop robust algorithms.
Layered 3D Human Generation via Semantic-Aware Diffusion Model
To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing that can eliminate the non-clothing content generated by the model.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models.
Multi-delay arterial spin-labeled perfusion estimation with biophysics simulation and deep learning
The trained network was further tested in a synthetic brain ASL image based on vasculature network extracted from magnetic resonance (MR) angiography.
Load Data Valuation in Multi-Energy Systems: An End-to-End Approach
Accurate load forecasting serves as the foundation for the flexible operation of multi-energy systems (MES).
Goal-Oriented Wireless Communication Resource Allocation for Cyber-Physical Systems
We formulate a bandwidth allocation problem aimed at maximizing the information utility gain of transmitted data brought to CPS operation goals.
Harvest Video Foundation Models via Efficient Post-Pretraining
Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets.
Feature Guided Masked Autoencoder for Self-supervised Learning in Remote Sensing
Self-supervised learning guided by masked image modelling, such as Masked AutoEncoder (MAE), has attracted wide attention for pretraining vision transformers in remote sensing.
Ranked #1 on
Multi-Label Image Classification
on BigEarthNet-S1 (official test set)
(using extra training data)
Video Referring Expression Comprehension via Transformer with Content-conditioned Query
Video Referring Expression Comprehension (REC) aims to localize a target object in videos based on the queried natural language.
Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook
In this survey, we offer a comprehensive and up-to-date review of large models tailored (or adapted) for time series and spatio-temporal data, spanning four key facets: data types, model categories, model scopes, and application areas/tasks.
Boosting High Resolution Image Classification with Scaling-up Transformers
We present a holistic approach for high resolution image classification that won second place in the ICCV/CVPPA2023 Deep Nutrient Deficiency Challenge.
LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model.
Ranked #4 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
PlotMap: Automated Layout Design for Building Game Worlds
We develop a method of generating datasets of facility layout tasks, create a gym-like environment for experimenting with and evaluating different methods, and further analyze the two methods with comprehensive experiments, aiming to provide insights for solving facility layout tasks.
Bitstream-Corrupted Video Recovery: A Novel Benchmark Dataset and Method
The past decade has witnessed great strides in video recovery by specialist technologies, like video inpainting, completion, and error concealment.
OccluTrack: Rethinking Awareness of Occlusion for Enhancing Multiple Pedestrian Tracking
Particularly, the improvements on IDF1, IDSw, AssA, and AssR demonstrate the effectiveness of our OccluTrack on tracking and association performance.
Decoupling Common and Unique Representations for Multimodal Self-supervised Learning
The increasing availability of multi-sensor data sparks wide interest in multimodal self-supervised learning.
Representation Learning for Sequential Volumetric Design Tasks
We develop the preference model by estimating the density of the learned representations whereas we train an autoregressive transformer model for sequential design generation.
Joint Oscillation Damping and Inertia Provision Service for Converter-Interfaced Generation
To address this gap, this paper defines the joint oscillation damping and inertia provision services at the system level, seeking to encourage converter-interfaced generation to provide enhanced damping and fast frequency response capabilities.
Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts
Cross-scene generalizable NeRF models, which can directly synthesize novel views of unseen scenes, have become a new spotlight of the NeRF field.
Deep Semantic Model Fusion for Ancient Agricultural Terrace Detection
Discovering ancient agricultural terraces in desert regions is important for the monitoring of long-term climate changes on the Earth's surface.
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
R2Det: Redemption from Range-view for Accurate 3D Object Detection
LiDAR-based 3D object detection is of paramount importance for autonomous driving.
Benchmarks and Custom Package for Energy Forecasting
However, there are many differences between energy forecasting and traditional time series forecasting.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Specifically, we utilize a multi-scale approach to generate video-related descriptions.
SimPLe: Similarity-Aware Propagation Learning for Weakly-Supervised Breast Cancer Segmentation in DCE-MRI
The network first utilizes the pseudo-masks generated using the extreme points to train itself, by minimizing a contrastive loss, which encourages the network to learn more representative features for cancerous voxels.
Hyper-parameter Adaptation of Conformer ASR Systems for Elderly and Dysarthric Speech Recognition
Automatic recognition of disordered and elderly speech remains highly challenging tasks to date due to data scarcity.
Semi-Supervised Learning for hyperspectral images by non parametrically predicting view assignment
Recently, to effectively train the deep learning models with minimal labelled samples, the unlabeled samples are also being leveraged in self-supervised and semi-supervised setting.
SSL4EO-L: Datasets and Foundation Models for Landsat Imagery
The Landsat program is the longest-running Earth observation program in history, with 50+ years of data acquisition by 8 satellites.
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs).
Ranked #3 on
Temporal/Casual QA
on NExT-QA
(using extra training data)
SaDI: A Self-adaptive Decomposed Interpretable Framework for Electric Load Forecasting under Extreme Events
In this paper, we propose a novel forecasting framework, named Self-adaptive Decomposed Interpretable framework~(SaDI), which ensembles long-term trend, short-term trend, and period modelings to capture temporal characteristics in different components.
Top-Down Framework for Weakly-supervised Grounded Image Captioning
Weakly-supervised grounded image captioning (WSGIC) aims to generate the caption and ground (localize) predicted object words in the input image without using bounding box supervision.
ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer
The Transformer structures have been widely used in computer vision and have recently made an impact in the area of medical image registration.
DiffLoad: Uncertainty Quantification in Electrical Load Forecasting with the Diffusion Model
The uncertainties in load forecasting can be divided into two types: epistemic uncertainty and aleatoric uncertainty.
GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data
Towards a fair and comprehensive analysis of existing methods, the proposed benchmark consists of 1) a large-scale dataset including co-registered RGB and nDSM pairs and pixel-wise semantic labels; 2) a comprehensive evaluation and analysis of existing multi-modal fusion strategies for both convolutional and Transformer-based networks on remote sensing data.
Ranked #1 on
Semantic Segmentation
on GAMUS
VideoLLM: Modeling Video Sequence with Large Language Models
Building upon this insight, we propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding.
VideoChat: Chat-Centric Video Understanding
In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat.
Ranked #3 on
Question Answering
on NExT-QA (Open-ended VideoQA)
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+5
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2.
Physics-based network fine-tuning for robust quantitative susceptibility mapping from high-pass filtered phase
Purpose: To improve the generalization ability of convolutional neural network (CNN) based prediction of quantitative susceptibility mapping (QSM) from high-pass filtered phase (HPFP) image.
ScatterFormer: Locally-Invariant Scattering Transformer for Patient-Independent Multispectral Detection of Epileptiform Discharges
Patient-independent detection of epileptic activities based on visual spectral representation of continuous EEG (cEEG) has been widely used for diagnosing epilepsy.
Label-free timing analysis of SiPM-based modularized detectors with physics-constrained deep learning
We mathematically demonstrate the existence of the optimal function desired by the method, and give a systematic algorithm for training and calibration of the model.
SSN: Stockwell Scattering Network for SAR Image Change Detection
Recently, synthetic aperture radar (SAR) image change detection has become an interesting yet challenging direction due to the presence of speckle noise.
Maximum Spherical Mean Value (mSMV) Filtering for Whole Brain Quantitative Susceptibility Mapping
To develop a tissue field filtering algorithm, called maximum Spherical Mean Value (mSMV), for reducing shadow artifacts in quantitative susceptibility mapping (QSM) of the brain without requiring brain tissue erosion. Residual background field is a major source of shadow artifacts in QSM.
Bitstream-Corrupted JPEG Images are Restorable: Two-stage Compensation and Alignment Framework for Image Restoration
In this paper, we study a real-world JPEG image restoration problem with bit errors on the encrypted bitstream.
A Byte Sequence is Worth an Image: CNN for File Fragment Classification Using Bit Shift and n-Gram Embeddings
File fragment classification (FFC) on small chunks of memory is essential in memory forensics and Internet security.
mcLARO: Multi-Contrast Learned Acquisition and Reconstruction Optimization for simultaneous quantitative multi-parametric mapping
Results: The retrospective ablation study showed improved image sharpness of mcLARO compared to the baseline network without multi-contrast sampling pattern optimization or image feature fusion, and negligible bias and narrow 95% limits of agreement on regional T1, T2, T2* and QSM values were obtained by the under-sampled reconstructions compared to the fully sampled reconstruction.
Efficient automatic segmentation for multi-level pulmonary arteries: The PARSE challenge
Efficient automatic segmentation of multi-level (i. e. main and branch) pulmonary arteries (PA) in CTPA images plays a significant role in clinical applications.
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90. 0% on K400 and 89. 9% on K600) and Something-Something (68. 7% on V1 and 77. 0% on V2).
Ranked #1 on
Self-Supervised Action Recognition
on UCF101
(using extra training data)
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain.
Ranked #1 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
NeRFLiX: High-Quality Neural View Synthesis by Learning a Degradation-Driven Inter-viewpoint MiXer
Neural radiance fields (NeRF) show great success in novel view synthesis.
Ranked #1 on
Novel View Synthesis
on LLFF
PointPatchMix: Point Cloud Mixing with Patch Scoring
With Point-MAE as our baseline, our model surpasses previous methods by a significant margin, achieving 86. 3% accuracy on ScanObjectNN and 94. 1% accuracy on ModelNet40.
Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition
Experiments conducted on the UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest TDNN and Conformer ASR systems integrated domain adapted wav2vec2. 0 models consistently outperform the standalone wav2vec2. 0 models by statistically significant WER reductions of 8. 22% and 3. 43% absolute (26. 71% and 15. 88% relative) on the two tasks respectively.
Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognition
The spiking neural network (SNN) using leaky-integrated-and-fire (LIF) neurons has been commonly used in automatic speech recognition (ASR) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Rate-Perception Optimized Preprocessing for Video Coding
In the past decades, lots of progress have been done in the video compression field including traditional video codec and learning-based video codec.
Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision
The former aims to infer all masked entities in the caption given the group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities.
Open Vocabulary Semantic Segmentation
Open-Vocabulary Semantic Segmentation
+1
UniFormerV2: Unlocking the Potential of Image ViTs for Video Understanding
The prolific performances of Vision Transformers (ViTs) in image tasks have prompted research into adapting the image ViTs for video tasks.
Boosting Accuracy and Robustness of Student Models via Adaptive Adversarial Distillation
Thus, recent studies concern about adversarial distillation (AD) that aims to inherit not only prediction accuracy but also adversarial robustness of a robust teacher model under the paradigm of robust optimization.
NeuralLift-360: Lifting an In-the-Wild 2D Photo to a 3D Object With 360deg Views
In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360deg views that corresponds well with the given reference image.
Pixels, Regions, and Objects: Multiple Enhancement for Salient Object Detection
A multi-level hybrid loss is firstly designed to guide the network to learn pixel-level, region-level, and object-level features.
A Survey of Face Recognition
Recent years witnessed the breakthrough of face recognition with deep convolutional neural networks.
InternVideo: General Video Foundation Models via Generative and Discriminative Learning
Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications.
Ranked #1 on
Action Recognition
on Something-Something V1
(using extra training data)
Inconsistency Ranking-based Noisy Label Detection for High-quality Data
We apply this technique to the automatic speaker verification (ASV) task as a proof of concept.
NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views
In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360{\deg} views that correspond well with the given reference image.
CMC v2: Towards More Accurate COVID-19 Detection with Discriminative Video Priors
This paper presents our solution for the 2nd COVID-19 Competition, occurring in the framework of the AIMIA Workshop at the European Conference on Computer Vision (ECCV 2022).
A Particle-based Sparse Gaussian Process Optimizer
Task learning in neural networks typically requires finding a globally optimal minimizer to a loss function objective.
Adjacent Slice Feature Guided 2.5D Network for Pulmonary Nodule Segmentation
In this paper, we design an adjacent slice feature fusion model to introduce information from adjacent slices.
UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer
UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format.
InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges
In this report, we present our champion solutions to five tracks at Ego4D challenge.
Ranked #1 on
State Change Object Detection
on Ego4D
SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation
Self-supervised pre-training bears potential to generate expressive representations without human annotation.
Ranked #1 on
Multi-Label Image Classification
on BigEarthNet (official test set)
(using extra training data)
LARO: Learned Acquisition and Reconstruction Optimization to accelerate Quantitative Susceptibility Mapping
In this paper, we present our new framework, called Learned Acquisition and Reconstruction Optimization (LARO), which aims to accelerate the multi-echo gradient echo (mGRE) pulse sequence for QSM.
Exploiting prompt learning with pre-trained language models for Alzheimer's Disease detection
Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and to delay further progression.
PalGAN: Image Colorization with Palette Generative Adversarial Networks
Multimodal ambiguity and color bleeding remain challenging in colorization.
Non-Iterative Scribble-Supervised Learning with Pacing Pseudo-Masks for Medical Image Segmentation
To address these issues, we propose a non-iterative method where a stream of varying (pacing) pseudo-masks teach a network via consistency training, named PacingPseudo.
EarthNets: Empowering AI in Earth Observation
Furthermore, a new platform for EO, termed EarthNets, is released to achieve a fair and consistent evaluation of deep learning methods on remote sensing data.
Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?
Vision Transformers (ViTs) have proven to be effective, in solving 2D image understanding tasks by training over large-scale image datasets; and meanwhile as a somehow separate track, in modeling the 3D visual world too such as voxels or point clouds.
A multi view multi stage and multi window framework for pulmonary artery segmentation from CT scans
In addition, in order to improve the segmentation performance, we adopt multi-view and multi-window level method, at the same time we employ a fine-tune strategy to mitigate the impact of inconsistent labeling.
PulseDL-II: A System-on-Chip Neural Network Accelerator for Timing and Energy Extraction of Nuclear Detector Signals
Recent literature reveals that deep learning models, especially one-dimensional convolutional neural networks, are promising when dealing with digital signals from nuclear detectors.
Quality-Constant Per-Shot Encoding by Two-Pass Learning-based Rate Factor Prediction
With the help of first pass predicted RF and corresponding actual quality as feedback, the second pass prediction will be highly accurate.
Exploring linguistic feature and model combination for speech recognition based automatic AD detection
Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and delay progression.
Self-supervised Learning in Remote Sensing: A Review
In deep learning research, self-supervised learning (SSL) has received great attention triggering interest within both the computer vision and remote sensing communities.
Ranked #3 on
Multi-Label Image Classification
on BigEarthNet
1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)
Our model consists of three modules: the candidate waypoints predictor (CWP), the history enhanced planner and the tryout controller.
Conformer Based Elderly Speech Recognition System for Alzheimer's Disease Detection
Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care to delay further progression.
WOLONet: Wave Outlooker for Efficient and High Fidelity Speech Synthesis
Recently, GAN-based neural vocoders such as Parallel WaveGAN, MelGAN, HiFiGAN, and UnivNet have become popular due to their lightweight and parallel structure, resulting in a real-time synthesized waveform with high fidelity, even on a CPU.
Monitoring Urban Forests from Auto-Generated Segmentation Maps
We present and evaluate a weakly-supervised methodology to quantify the spatio-temporal distribution of urban forests based on remotely sensed data with close-to-zero human interaction.
UMSNet: An Universal Multi-sensor Network for Human Activity Recognition
Human activity recognition (HAR) based on multimodal sensors has become a rapidly growing branch of biometric recognition and artificial intelligence.
Beam Training and Tracking in MmWave Communication: A Survey
This article provides an overview of the beam training and tracking technologies on mmWave bands and reveals the insights for future research in the 6th Generation (6G) mobile network.
Long-run User Value Optimization in Recommender Systems through Content Creation Modeling
Content recommender systems are generally adept at maximizing immediate user satisfaction but to optimize for the \textit{long-run} user value, we need more statistically sophisticated solutions than off-the-shelf simple recommender algorithms.
Self-supervised Vision Transformers for Joint SAR-optical Representation Learning
Experimental results employing the BigEarthNet-MM dataset demonstrate the benefits of both, the ViT backbones and the proposed multimodal SSL algorithm DINO-MM.
A Global Modeling Approach for Load Forecasting in Distribution Networks
Efficient load forecasting is needed to ensure better observability in the distribution networks, whereas such forecasting is made possible by an increasing number of smart meter installations.
MAT: Mask-Aware Transformer for Large Hole Image Inpainting
Recent studies have shown the importance of modeling long-range interactions in the inpainting problem.
Ranked #2 on
Image Inpainting
on CelebA-HQ
