Search Results for author:
Found 70 papers, 20 papers with code
MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval
This intrinsic modality imbalance leaves a considerable portion of visual information remaining unaligned with text.
Generative Video Diffusion for Unseen Cross-Domain Video Moment Retrieval
We address two problems in video editing for optimising unseen domain VMR: (1) generation of high-quality simulation videos of different moments with subtle distinctions, (2) selection of simulation videos that complement existing source training videos without introducing harmful noise or unnecessary repetitions.
Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory
This observation motivates us to design an HOI detector that can be trained even with long-tailed labeled data and can leverage existing knowledge from pre-trained models.
Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training
The correlation between the vision and text is essential for video moment retrieval (VMR), however, existing methods heavily rely on separate pre-training feature extractors for visual and textual understanding.
Moment Detection in Long Tutorial Videos
To study this problem, we propose the first dataset of untrimmed, long-form tutorial videos for the task of Moment Detection called the Behance Moment Detection (BMD) dataset.
LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos
Livestream videos have become a significant part of online learning, where design, digital marketing, creative painting, and other skills are taught by experienced experts in the sessions, making them valuable materials.
Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment
Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding.
Video Activity Localisation with Uncertainties in Temporal Boundary
Such uncertainties in temporal labelling are currently ignored in model training, resulting in learning mis-matched video-text correlation with poor generalisation in test.
MHMS: Multimodal Hierarchical Multimedia Summarization
Multimedia summarization with multimodal output can play an essential role in real-world applications, i. e., automatically generating cover images and titles for news articles or providing introductions to online videos.
StyleBabel: Artistic Style Tagging and Captioning
We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools.
Cross Modal Retrieval with Querybank Normalisation
In this work we first show that, despite their effectiveness, state-of-the-art joint embeddings suffer significantly from the longstanding "hubness problem" in which a small number of gallery embeddings form the nearest neighbours of many queries.
Ranked #5 on
Video Retrieval
on QuerYD
Time-Equivariant Contrastive Video Representation Learning
We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos.
Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
StreamHover: Livestream Transcript Summarization and Annotation
With the explosive growth of livestream broadcasting, there is an urgent need for new summarization technology that enables us to create a preview of streamed content and tap into this wealth of knowledge.
Font Completion and Manipulation by Cycling Between Multi-Modality Representations
The novel graph constructor maps a glyph's latent code to its graph representation that matches expert knowledge, which is trained to help the translation task.
Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation
Video activity localisation has recently attained increasing attention due to its practical values in automatically localising the most salient visual segments corresponding to their language descriptions (sentences) from untrimmed and unstructured videos.
Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study
Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale, and/or depends on the access to original training data as well as the trained model parameters.
Compositional Sketch Search
We present an algorithm for searching image collections using free-hand sketches that describe the appearance and relative positions of multiple objects.
Magic Layouts: Structural Prior for Component Detection in User Interface Designs
We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts.
A Multi-Implicit Neural Representation for Fonts
Fonts are ubiquitous across documents and come in a variety of styles.
TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval
In recent years, considerable progress on the task of text-video retrieval has been achieved by leveraging large-scale pretraining on visual and audio datasets to construct powerful video encoders.
ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similarity
We present ALADIN (All Layer AdaIN); a novel architecture for searching images based on the similarity of their artistic style.
Video Question Answering on Screencast Tutorials
This paper presents a new video question answering task on screencast tutorials.
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
We propose Geo-PIFu, a method to recover a 3D mesh from a monocular color image of a clothed person.
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics
Based on this criterion, we introduce a novel image transformation that we call limited context inpainting (LCI).
Superpixel Segmentation with Fully Convolutional Networks
In computer vision, superpixels have been widely used as an effective way to reduce the number of image primitives for subsequent processing.
Neural Architecture Search for Deep Image Prior
We present a neural architecture search (NAS) technique to enhance the performance of unsupervised image de-noising, in-painting and super-resolution under the recently proposed Deep Image Prior (DIP).
Is There Mode Collapse? A Case Study on Face Generation and Its Black-box Calibration
Generative adversarial networks (GANs) nowadays are capable of producing im-ages of incredible realism.
An Internal Learning Approach to Video Inpainting
We propose a novel video inpainting algorithm that simultaneously hallucinates missing appearance and motion (optical flow) information, building upon the recent 'Deep Image Prior' (DIP) that exploits convolutional network architectures to enforce plausible texture in static images.
Large-scale Tag-based Font Retrieval with Generative Feature Learning
In this paper, we address the problem of large-scale tag-based font retrieval which aims to bring semantics to the font selection process and enable people without expert knowledge to use fonts effectively.
Privacy-Preserving Deep Action Recognition: An Adversarial Learning Framework and A New Dataset
We first discuss an innovative heuristic of cross-dataset training and evaluation, enabling the use of multiple single-task datasets (one with target task labels and the other with privacy labels) in our problem.
Learning Video Representations from Correspondence Proposals
In particular, it can effectively learn representations for videos by mixing appearance and long-range motion with an RGB-only input.
Creative Procedural-Knowledge Extraction From Web Design Tutorials
Complex design tasks often require performing diverse actions in a specific order.
LiveSketch: Query Perturbations for Guided Sketch-based Visual Search
LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries.
Visual Font Pairing
Third, font pairing is an asymmetric problem in that the roles played by header and body fonts are not interchangeable.
Interactive Boundary Prediction for Object Selection
Instead of relying on pre-defined low-level image features, our method adaptively predicts object boundaries according to image content and user interactions.
What do I Annotate Next? An Empirical Study of Active Learning for Action Localization
In this paper, we introduce a novel active learning framework for temporal localization that aims to mitigate this data dependency issue.
Synthetically Supervised Feature Learning for Scene Text Recognition
We propose to leverage the parameters that lead to the output images to improve image feature learning.
``Factual'' or ``Emotional'': Stylized Image Captioning with Adaptive Learning and Attention
It uses two groups of matrices to capture the factual and stylized knowledge, respectively, and automatically learns the word-level weights of the two groups based on previous context.
Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study
This paper aims to improve privacy-preserving visual recognition, an increasingly demanded feature in smart camera applications, by formulating a unique adversarial training framework.
"Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention
It uses two groups of matrices to capture the factual and stylized knowledge, respectively, and automatically learns the word-level weights of the two groups based on previous context.
Multi-Task Adversarial Network for Disentangled Feature Learning
The encoder and the discriminators are trained cooperatively on factors of interest, but in an adversarial way on factors of distraction.
Disentangling Structure and Aesthetics for Style-Aware Image Completion
Content-aware image completion or in-painting is a fundamental tool for the correction of defects or removal of objects in images.
Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
We propose a fast feed-forward network for arbitrary style transfer, which can generate stylized image for previously unseen content and style image pairs.
Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions
In this work, we propose two different models, which employ different schemes for injecting sentiments into image captions.
Exploring Asymmetric Encoder-Decoder Structure for Context-based Sentence Representation Learning
Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language.
Speeding up Context-based Sentence Representation Learning with Non-autoregressive Convolutional Decoding
We carefully designed experiments to show that neither an autoregressive decoder nor an RNN decoder is required.
Sketching With Style: Visual Search With Sketches and Aesthetic Context
We propose a novel measure of visual similarity for image retrieval that incorporates both structural and aesthetic (style) constraints.
Spatial-Semantic Image Search by Visual Feature Synthesis
We train a convolutional neural network to synthesize appropriate visual features that captures the spatial-semantic constraints from the user canvas query.
Rethinking Skip-thought: A Neighborhood based Approach
We train our skip-thought neighbor model on a large corpus with continuous sentences, and then evaluate the trained model on 7 tasks, which include semantic relatedness, paraphrase detection, and classification benchmarks.
BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography
Furthermore, we carry out baseline experiments to show the value of this dataset for artistic style prediction, for improving the generality of existing object classifiers, and for the study of visual domain adaptation.
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection.
Composition-Preserving Deep Photo Aesthetics Assessment
Deep convolutional neural network (ConvNet) methods have recently shown promising results for aesthetics assessment.
Ranked #6 on
Aesthetics Quality Assessment
on AVA
Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark
We hope that this data set encourages further research on visual emotion analysis.
Image Captioning with Semantic Attention
Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing.
Multi-Instance Visual-Semantic Embedding
Visual-semantic embedding models have been recently proposed and shown to be effective for image classification and zero-shot learning, by mapping images into a continuous semantic label space.
Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks
Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis.
DeepFont: Identify Your Font from An Image
As font is one of the core design concepts, automatic font identification and similar font suggestion from an image or photo has been on the wish list of many designers.
Ranked #1 on
Font Recognition
on VFR-Wild
Fine-Grained Recognition Without Part Annotations
Scaling up fine-grained recognition to all domains of fine-grained objects is a challenge the computer vision community will need to face in order to realize its goal of recognizing all object categories.
Real-World Font Recognition Using Deep Network and Domain Adaptation
We address a challenging fine-grain classification problem: recognizing a font style from an image of text.
Collaborative Feature Learning from Social Media
We validate our feature learning paradigm on this dataset and find that the learned feature significantly outperforms the state-of-the-art image features in learning better image similarities.
Decomposition-Based Domain Adaptation for Real-World Font Recognition
We present a domain adaption framework to address a domain mismatch between synthetic training and real-world testing data.
Fast Edge-Preserving PatchMatch for Large Displacement Optical Flow
We present a fast optical flow algorithm that can handle large displacement motions.
Large-Scale Visual Font Recognition
This paper addresses the large-scale visual font recognition (VFR) problem, which aims at automatic identification of the typeface, weight, and slope of the text in an image or photo without any knowledge of content.
Ranked #1 on
Font Recognition
on VFR-447
GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
The ability to train large-scale neural networks has resulted in state-of-the-art performance in many areas of computer vision.
Specular Reflection Separation Using Dark Channel Prior
Our method is based on a novel observation that for most natural images the dark channel can provide an approximate specular-free image.
Plane-Based Content Preserving Warps for Video Stabilization
Recently, a new image deformation technique called content-preserving warping (CPW) has been successfully employed to produce the state-of-the-art video stabilization results in many challenging cases.
Large Displacement Optical Flow from Nearest Neighbor Fields
We use approximate nearest neighbor fields to compute an initial motion field and use a robust algorithm to compute a set of similarity transformations as the motion candidates for segmentation.
