Search Results for author:
Found 74 papers, 31 papers with code
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
In this paper, we propose PhysGame as a pioneering benchmark to evaluate physical commonsense violations in gameplay videos.
MuSiCNet: A Gradual Coarse-to-Fine Framework for Irregularly Sampled Multivariate Time Series Analysis
Irregularly sampled multivariate time series (ISMTS) are prevalent in reality.
Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
In terms of methodology, we propose Continual LLaVA, a rehearsal-free method tailored for continual instruction tuning in LVLMs.
Synth4Seg -- Learning Defect Data Synthesis for Defect Segmentation using Bi-level Optimization
Synthetic defect data generation is a popular approach for mitigating data challenges.
How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?
Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts.
ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
To bridge this gap, we present ING-VP, the first INteractive Game-based Vision Planning benchmark, specifically designed to evaluate the spatial imagination and multi-step reasoning abilities of MLLMs.
TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance.
Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats.
Contrastive Localized Language-Image Pre-Training
Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications.
Data Augmentation for Sparse Multidimensional Learning Performance Data Using Generative AI
Second, a tensor factorization method is used to impute missing values in sparse tensors of collected learner data, thereby grounding the imputation on knowledge tracing tasks that predict missing performance values based on real observations.
MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval
Text-Video Retrieval (TVR) aims to align and associate relevant video content with corresponding natural language queries.
Apple Intelligence Foundation Language Models
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute.
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Ultimately, MMAU not only sheds light on the capabilities and limitations of LLM agents but also enhances the interpretability of their performance.
SLRL: Structured Latent Representation Learning for Multi-view Clustering
Subsequently, to exploit the structural information among samples, a k-nearest neighbor graph is constructed from this common latent representation.
Integrating Attentional Factors and Spacing in Logistic Knowledge Tracing Models to Explore the Impact of Training Sequences on Category Learning
Although a recent study underscores the joint influence of memory and attentional factors on sequencing effects, there remains a scarcity of effective computational models integrating both attentional and memory considerations to comprehensively understand the effect of training sequences on students' performance.
Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps
Real-time detection and prediction of extreme weather protect human lives and infrastructure.
Textual Inversion and Self-supervised Refinement for Radiology Report Generation
Existing mainstream approaches follow the encoder-decoder paradigm for generating radiology reports.
Uncertainty-aware sign language video retrieval with probability distribution modeling
Sign language video retrieval plays a key role in facilitating information access for the deaf community.
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter
Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries.
Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation
Recent developments underscore the potential of textual information in enhancing learning models for a deeper understanding of medical visual semantics.
Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations
State-of-the-art language models (LMs) sometimes generate non-factual hallucinations that misalign with world knowledge.
depyf: Open the Opaque Box of PyTorch Compiler for Machine Learning Researchers
PyTorch \texttt{2. x} introduces a compiler designed to accelerate deep learning programs.
Predicting Learning Performance with Large Language Models: A Study in Adult Literacy
This research is motivated by the potential of LLMs to predict learning performance based on its inherent reasoning and computational capabilities.
Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation
In this paper, we propose a method to evaluate the response preference by using the output probabilities of response pairs under contrastive prompt pairs, which could achieve better performance on LLaMA2-7B and LLaMA2-13B compared to RLAIF.
Recommendation Fairness in Social Networks Over Time
We further study how interventions on network properties influence fairness by examining counterfactual scenarios with alternative evolution outcomes and differing network properties.
3DG: A Framework for Using Generative AI for Handling Sparse Learner Performance Data From Intelligent Tutoring Systems
Learning performance data (e. g., quiz scores and attempts) is significant for understanding learner engagement and knowledge mastery level.
Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
We investigate this approach under two different settings: one where the policy model is smaller and is paired with a more powerful critic model, and another where a single language model fulfills both roles.
Real-Time Exposure Correction via Collaborative Transformations and Adaptive Sampling
Specifically the global transformation adjusts the overall appearance using image-adaptive 3D LUTs to provide decent global contrast and sharp details while the pixel transformation compensates for local context.
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study
Large Multimodal Models (LMMs) have demonstrated impressive performance across various vision and language tasks, yet their potential applications in recommendation tasks with visual assistance remain unexplored.
Successor Features for Efficient Multisubject Controlled Text Generation
While large language models (LLMs) have achieved impressive performance in generating fluent and realistic text, controlling the generated text so that it exhibits properties such as safety, factuality, and non-toxicity remains challenging.
Video Referring Expression Comprehension via Transformer with Content-conditioned Query
Video Referring Expression Comprehension (REC) aims to localize a target object in videos based on the queried natural language.
Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model
In the CPT and SFT phases, Qilin-Med achieved 38. 4% and 40. 0% accuracy on the CMExam test set, respectively.
VeCLIP: Improving CLIP Training via Visual-enriched Captions
For example, VeCLIP achieves up to +25. 2% gain in COCO and Flickr30k retrieval tasks under the 12M setting.
G2L: Semantically Aligned and Uniform Video Grounding via Geodesic and Game Theory
Due to two annoying issues in video grounding: (1) the co-existence of some visual entities in both ground truth and other moments, \ie semantic overlapping; (2) only a few moments in the video are annotated, \ie sparse annotation dilemma, vanilla contrastive learning is unable to model the correlations between temporally distant moments and learned inconsistent video representations.
Improving Retrieval-Augmented Large Language Models via Data Importance Learning
There are exponentially many terms in the multilinear extension, and one key contribution of this paper is a polynomial time algorithm that computes exactly, given a retrieval-augmented model with an additive utility function and a validation set, the data importance of data points in the retrieval corpus using the multilinear extension of the model's utility function.
Improving Reference-based Distinctive Image Captioning with Contrastive Rewards
A recent DIC method proposes to generate distinctive captions by comparing the target image with a set of semantic-similar reference images, i. e., reference-based DIC (Ref-DIC).
VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON
We introduce the VISION Datasets, a diverse collection of 14 industrial inspection datasets, uniquely poised to meet these challenges.
Efficient ConvBN Blocks for Transfer Learning and Beyond
Subsequently, we propose a novel Tune mode to bridge the gap between Eval mode and Deploy mode.
Systematic Rectification of Language Models via Dead-end Analysis
Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse.
RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection
In this paper, we propose a Robust GAN-inversion (RGI) method with a provable robustness guarantee to achieve image restoration under unknown \textit{gross} corruptions, where a small fraction of pixels are completely corrupted.
Learning with Rejection for Abstractive Text Summarization
State-of-the-art abstractive summarization systems frequently hallucinate content that is not supported by the source document, mainly due to noise in the training dataset.
Exploiting Auxiliary Caption for Video Grounding
Video grounding aims to locate a moment of interest matching the given query sentence from an untrimmed video.
Iterative Proposal Refinement for Weakly-Supervised Video Grounding
Weakly-Supervised Video Grounding (WSVG) aims to localize events of interest in untrimmed videos with only video-level annotations.
Video Referring Expression Comprehension via Transformer with Content-aware Query
To this end, we introduce two new datasets (i. e., VID-Entity and VidSTG-Entity) by augmenting the VIDSentence and VidSTG datasets with the explicitly referred words in the whole sentence, respectively.
Adversarial Inter-Group Link Injection Degrades the Fairness of Graph Neural Networks
We hope our findings raise awareness about this issue in our community and lay a foundation for the future development of GNN models that are more robust to such attacks.
Latent Heterogeneous Graph Network for Incomplete Multi-View Learning
Multi-view learning has progressed rapidly in recent years.
LocVTP: Video-Text Pre-training for Temporal Localization
To further enhance the temporal reasoning ability of the learned feature, we propose a context projection head and a temporal aware contrastive loss to perceive the contextual relationships.
Correspondence Matters for Video Referring Expression Comprehension
Extensive experiments demonstrate that our DCNet achieves state-of-the-art performance on both video and image REC benchmarks.
A Survey on Neural Abstractive Summarization Methods and Factual Consistency of Summarization
Automatic summarization is the process of shortening a set of textual data computationally, to create a subset (a summary) that represents the most important pieces of information in the original text.
Jacobian Norm for Unsupervised Source-Free Domain Adaptation
Unsupervised Source (data) Free domain adaptation (USFDA) aims to transfer knowledge from a well-trained source model to a related but unlabeled target domain.
PAEDID: Patch Autoencoder Based Deep Image Decomposition For Pixel-level Defective Region Segmentation
To combine the best of both worlds, we present an unsupervised patch autoencoder based deep image decomposition (PAEDID) method for defective region segmentation.
Unsupervised Pre-training for Temporal Action Localization Tasks
These pre-trained models can be sub-optimal for temporal localization tasks due to the inherent discrepancy between video-level classification and clip-level localization.
Synthetic Defect Generation for Display Front-of-Screen Quality Inspection: A Survey
Display front-of-screen (FOS) quality inspection is essential for the mass production of displays in the manufacturing process.
Information Gain Propagation: a new way to Graph Active Learning with Soft Labels
Graph Neural Networks (GNNs) have achieved great success in various tasks, but their performance highly relies on a large number of labeled nodes, which typically requires considerable human effort.
Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
On active learning task, our method achieves 97. 0% Top-1 Accuracy on CIFAR10 with 0. 1% annotated data, and 83. 9% Top-1 Accuracy on CIFAR100 with 10% annotated data.
RIM: Reliable Influence-based Active Learning on Graphs
Message passing is the core of most graph models such as Graph Convolutional Network (GCN) and Label Propagation (LP), which usually require a large number of clean labeled data to smooth out the neighborhood over the graph.
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
State-of-the-art abstractive summarization systems often generate \emph{hallucinations}; i. e., content that is not directly inferable from the source text.
Abstractive Text Summarization
Reinforcement Learning (RL)
+1
Deep Motion Prior for Weakly-Supervised Temporal Action Localization
In this paper, we analyze that the motion cues behind the optical flow features are complementary informative.
Optical Flow Estimation
Weakly-supervised Temporal Action Localization
+1
UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing
Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
All You Need is a Second Look: Towards Arbitrary-Shaped Text Detection
Compared to the traditional single-stage segmentation network, our NASK conducts the detection in a coarse-to-fine manner with the first stage segmentation spotting the rectangle text proposals and the second one retrieving compact representations.
BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
While single-shot quantized neural architecture search enjoys flexibility in both model architecture and quantization policy, the combined search space comes with many challenges, including instability when training the weight-sharing supernet and difficulty in navigating the exponentially growing search space.
Hardware Aware Neural Architecture Search
Model Optimization
+2
Video Frame Interpolation via Structure-Motion based Iterative Fusion
Inspired by the observation that audiences have different visual preferences on foreground and background objects, we for the first time propose to use saliency masks in the evaluation processes of the task of video frame interpolation.
RR-Net: Injecting Interactive Semantics in Human-Object Interaction Detection
Upon the frame, an Interaction Intensifier Module and a Correlation Parsing Module are carefully designed, where: a) interactive semantics from humans can be exploited and passed to objects to intensify interactions, b) interactive correlations among humans, objects and interactions are integrated to promote predictions.
CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
In this paper, we argue that learning by comparing helps identify these hard snippets and we propose to utilize snippet Contrastive learning to Localize Actions, CoLA for short.
Quantum error-correcting codes from matrix-product codes related to quasi-orthogonal and quasi-unitary matrices
The construction of matrix-product codes with certain self-orthogonality over finite fields is an effective way to obtain good $q$-ary quantum codes of large length.
Information Theory Information Theory Quantum Physics
Factual Error Correction for Abstractive Summarization Models
Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset.
TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion
Our analysis also reveals important sources of variability both within and across TKG datasets, and we introduce several simple but strong baselines that outperform the prior state of the art in certain settings.
An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department
In order to verify performance in a real clinical setting, we silently deployed a preliminary version of the deep neural network at New York University Langone Health during the first wave of the pandemic, which produced accurate predictions in real-time.
Task-agnostic Temporally Consistent Facial Video Editing
Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
All you need is a second look: Towards Tighter Arbitrary shape text detection
Specifically, \textit{NASK} consists of a Text Instance Segmentation network namely \textit{TIS} (\(1^{st}\) stage), a Text RoI Pooling module and a Fiducial pOint eXpression module termed as \textit{FOX} (\(2^{nd}\) stage).
Unsupervised Domain Adaptation Through Transferring both the Source-Knowledge and Target-Relatedness Simultaneously
Unsupervised domain adaptation (UDA) is an emerging research topic in the field of machine learning and pattern recognition, which aims to help the learning of unlabeled target domain by transferring knowledge from the source domain.
Referring Expression Generation Using Entity Profiles
Referring Expression Generation (REG) is the task of generating contextually appropriate references to entities.
