Search Results for author:
Found 93 papers, 47 papers with code
Text to Multi-level MindMaps: A Novel Method for Hierarchical Visual Abstraction of Natural Language Text
This work firstly introduces MindMap Multilevel Visualization concept which is to jointly visualize and summarize textual information.
Generalized Twin Gaussian Processes using Sharma-Mittal Divergence
In this paper, we present a generalized structured regression framework based on Shama-Mittal divergence, a relative entropy measure, which is introduced to the Machine Learning community in this work.
Learning Hypergraph-regularized Attribute Predictors
Then the attribute prediction problem is casted as a regularized hypergraph cut problem in which HAP jointly learns a collection of attribute projections from the feature space to a hypergraph embedding space aligned with the attribute space.
Tell and Predict: Kernel Classifier Prediction for Unseen Visual Classes from Unstructured Text Descriptions
In this paper we propose a framework for predicting kernelized classifiers in the visual domain for categories with no training images where the knowledge comes from textual description about these categories.
Digging Deep into the layers of CNNs: In Search of How CNNs Achieve View Invariance
How does fine-tuning of a pre-trained CNN on a multi-view dataset affect the representation at each layer of the network?
Sherlock: Scalable Fact Learning in Images
We show that learning visual facts in a structured way enables not only a uniform but also generalizable visual understanding.
Convolutional Models for Joint Object Categorization and Pose Estimation
In the task of Object Recognition, there exists a dichotomy between the categorization of objects and estimating object pose, where the former necessitates a view-invariant representation, while the latter requires a representation capable of capturing pose information over different categories of objects.
Zero-Shot Event Detection by Multimodal Distributional Semantic Embedding of Videos
To our knowledge, this is the first Zero-Shot event detection model that is built on top of distributional semantics and extends it in the following directions: (a) semantic embedding of multimodal information in videos (with focus on the visual modalities), (b) automatically determining relevance of concepts/attributes to a free text query, which could be useful for other applications, and (c) retrieving videos by free text event query (e. g., "changing a vehicle tire") based on their content.
Write a Classifier: Predicting Visual Classifiers from Unstructured Text
Then, we propose a new constrained optimization formulation that combines a regression function and a knowledge transfer function with additional constraints to predict the parameters of a linear classifier.
Automatic Annotation of Structured Facts in Images
Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions.
SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-Grained Recognition
In this paper, we propose a new CNN architecture that integrates semantic part detection and abstraction (SPDA-CNN) for fine-grained classification.
Overlapping Cover Local Regression Machines
We present the Overlapping Domain Cover (ODC) notion for kernel machines, as a set of overlapping subsets of the data that covers the entire training set and optimized to be spatially cohesive as possible.
CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms
We argue that such networks are limited in their ability to generate creative products in their original design.
Relationship Proposal Networks
We demonstrate the ability of our Rel-PN to localize relationships with only a few thousand proposals.
Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision
We propose a learning framework that is able to connect text terms to its relevant parts and suppress connections to non-visual text terms without any part-text annotations.
Memory Aware Synapses: Learning what (not) to forget
We show state-of-the-art performance and, for the first time, the ability to adapt the importance of the parameters based on unlabeled data towards what the network needs (not) to forget, which may vary depending on test conditions.
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts
Most existing zero-shot learning methods consider the problem as a visual semantic embedding one.
DeSIGN: Design Inspiration from Generative Networks
Can an algorithm create original and compelling fashion designs to serve as an inspirational assistant?
Large-Scale Visual Relationship Understanding
Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of <subject, relation, object> triples.
Choose Your Neuron: Incorporating Domain Knowledge through Neuron-Importance
Our approach, which we call Neuron Importance-AwareWeight Transfer (NIWT), learns to map domain knowledge about novel "unseen" classes onto this dictionary of learned concepts and then optimizes for network parameters that can effectively combine these concepts - essentially learning classifiers by discovering and composing learned semantic concepts in deep networks.
Uncertainty-guided Lifelong Learning in Bayesian Networks
Sequentially learning of tasks arriving in a continuous stream is a complex problem and becomes more challenging when the model has a fixed capacity.
Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting
A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels.
GDPP: Learning Diverse Generations Using Determinantal Point Process
Generative models have proven to be an outstanding tool for representing high-dimensional probability distributions and generating realistic-looking images.
Efficient Lifelong Learning with A-GEM
In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task.
Ranked #6 on
Continual Learning
on ASC (19 tasks)
Exploring the Challenges towards Lifelong Fact Learning
So far life-long learning (LLL) has been studied in relatively small-scale and relatively artificial setups.
On Tiny Episodic Memories in Continual Learning
But for a successful knowledge transfer, the learner needs to remember how to perform previous tasks.
Ranked #7 on
Class Incremental Learning
on cifar100
Semi-Supervised Few-Shot Learning with Prototypical Random Walks
We develop a random walk semi-supervised loss that enables the network to learn representations that are compact and well-separated.
Creativity Inspired Zero-Shot Learning
We relate ZSL to human creativity by observing that zero-shot learning is about recognizing the unseen and creativity is about creating a likable unseen.
Learning Diverse Generations using Determinantal Point Processes
Embedded in an adversarial training and variational autoencoder, our Generative DPP approach shows a consistent resistance to mode-collapse on a wide-variety of synthetic data and natural image datasets including MNIST, CIFAR10, and CelebA, while outperforming state-of-the-art methods for data-efficiency, convergence-time, and generation quality.
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Continual learning aims to learn new tasks without forgetting previously learned ones.
Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction
Better machine understanding of pedestrian behaviors enables faster progress in modeling interactions between agents such as autonomous vehicles and humans.
Ranked #3 on
Trajectory Prediction
on ETH
Exploring Long Tail Visual Relationship Recognition with Large Vocabulary
We use these benchmarks to study the performance of several state-of-the-art long-tail models on the LTVRR setup.
Efficient long-distance relation extraction with DG-SpanBERT
In natural language processing, relation extraction seeks to rationally understand unstructured text.
Ranked #16 on
Relation Extraction
on TACRED
Compositional Language Continual Learning
It also shows significant improvement in machine translation task.
Inner Ensemble Networks: Average Ensemble as an Effective Regularizer
Also, we show empirically and theoretically that IENs lead to a greater variance reduction in comparison with other similar approaches such as dropout and maxout.
Class Normalization for (Continual)? Generalized Zero-Shot Learning
Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime.
Temporal Positive-unlabeled Learning for Biomedical Hypothesis Generation via Risk Estimation
Then, the key is to capture the temporal evolution of node pair (term pair) relations from just the positive and unlabeled data.
Adversarial Generation of Continuous Images
In most existing learning systems, images are typically viewed as 2D pixel arrays.
Ranked #12 on
Image Generation
on LSUN Churches 256 x 256
CIZSL++: Creativity Inspired Generative Zero-Shot Learning
To improve the discriminative power of ZSL, we model the visual learning process of unseen categories with inspiration from the psychology of human creativity for producing novel art.
HalentNet: Multimodal Trajectory Forecasting with Hallucinative Intents
Our model's learned representation leads to better and more semantically meaningful coverage of the trajectory distribution.
Motion Forecasting with Unlikelihood Training
We propose a new objective, unlikelihood training, which forces generated trajectories that conflicts with contextual information to be assigned a lower probability by our model.
Class Normalization for Zero-Shot Learning
Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime.
Gradient Descent Resists Compositionality
In this paper, we argue that gradient descent is one of the reasons that make compositionality learning hard during neural network optimization.
Transferability of Compositionality
To our best knowledge, this is the first work to focus on the transferability of compositionality, and it is orthogonal to existing efforts of learning compositional representations in training distribution.
ArtEmis: Affective Language for Visual Art
We present a novel large-scale dataset and accompanying machine learning models aimed at providing a detailed understanding of the interplay between visual content, its emotional effect, and explanations for the latter in language.
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data.
Aligning Latent and Image Spaces to Connect the Unconnectable
In this work, we develop a method to generate infinite high-resolution images with diverse and complex content.
Ranked #1 on
Infinite Image Generation
on LHQ
Imaginative Walks: Generative Random Walk Deviation Loss for Improved Unseen Learning Representation
By generating representations of unseen classes based on their semantic descriptions, e. g., attributes or text, generative ZSL attempts to differentiate unseen from seen categories.
RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognition
This paper shows that modeling an effective message-passing flow through an attention mechanism can be critical to tackling the compositionality and long-tail challenges in VRR.
CausalDyna: Improving Generalization of Dyna-style Reinforcement Learning via Counterfactual-Based Data Augmentation
Deep reinforcement learning agents trained in real-world environments with a limited diversity of object properties to learn manipulation tasks tend to suffer overfitting and fail to generalize to unseen testing environments.
HyperCGAN: Text-to-Image Synthesis with HyperNet-Modulated Conditional Generative Adversarial Networks
To the best of our knowledge, our work is the first one which explores text-controllable continuous image generation.
Domain-Aware Continual Zero-Shot Learning
Accordingly, we propose a Domain-Invariant Network (DIN) to learn factorized features for shifting domains and improved textual representation for unseen classes.
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2
We build our model on top of StyleGAN2 and it is just ${\approx}5\%$ more expensive to train at the same resolution while achieving almost the same image quality.
Efficiently Disentangle Causal Representations
This paper proposes an efficient approach to learning disentangled representations with causal mechanisms based on the difference of conditional probabilities in original and new distributions.
Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classification
The main question we address in this paper is how to scale up visual recognition of unseen classes, also known as zero-shot learning, to tens of thousands of categories as in the ImageNet-21K benchmark.
Social-Implicit: Rethinking Trajectory Prediction Evaluation and The Effectiveness of Implicit Maximum Likelihood Estimation
AMD is a metric that quantifies how close the whole generated samples are to the ground truth.
Ranked #1 on
Trajectory Prediction
on Stanford Drone
(ADE (in world coordinates) metric)
It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection
As a step in this direction, the ArtEmis dataset was recently introduced as a large-scale dataset of emotional reactions to images along with language explanations of these chosen emotions.
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
After finetuning the pretrained LoMaR on 384$\times$384 images, it can reach 85. 4% top-1 accuracy, surpassing MAE by 0. 6%.
Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning
In some complex environments with continuous state-action spaces, sparse rewards, and/or long temporal horizons, learning a good policy in the original environments can be difficult.
PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
In this work, we revisit the classical PointNet++ through a systematic study of model training and scaling strategies, and offer two major contributions.
Ranked #3 on
3D Semantic Segmentation
on OpenTrench3D
A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning
With the success of pretraining techniques in representation learning, a number of continual learning methods based on pretrained models have been proposed.
ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture
This paper introduces ArtELingo, a new benchmark and dataset, designed to encourage work on diversity across languages and cultures.
Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding
The main question we address in this paper is "can we consolidate the 3D visual stream by 2D clues synthesized from point clouds and efficiently utilize them in training and testing?".
Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only
Semantic segmentation is a crucial task in computer vision that involves segmenting images into semantically meaningful regions at the pixel level.
FishNet: A Large-scale Dataset and Benchmark for Fish Recognition, Detection, and Functional Trait Prediction
Aquatic species are essential components of the world's ecosystem, and the preservation of aquatic biodiversity is crucial for maintaining proper ecosystem functioning.
Guiding Online Reinforcement Learning with Action-Free Offline Pretraining
In this paper, we investigate the potential of using action-free offline datasets to improve online reinforcement learning, name this problem Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL).
ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions
By keeping acquiring new visual information from BLIP-2's answers, ChatCaptioner is able to generate more enriched image descriptions.
MoStGAN-V: Video Generation with Temporal Motion Styles
Video generation remains a challenging task due to spatiotemporal complexity and the requirement of synthesizing diverse motions with temporal consistency.
Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions
Video captioning aims to convey dynamic scenes from videos using natural language, facilitating the understanding of spatiotemporal information within our environment.
ImageCaptioner$^2$: Image Captioner for Image Captioning Bias Amplification Assessment
In addition, we design a formulation for measuring the bias of generated captions as prompt-based image captioning instead of using language classifiers.
HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models
A human evaluation aligned with 95% of our evaluations on average was conducted to probe the effectiveness of HRS-Bench.
LLM as A Robotic Brain: Unifying Egocentric Memory and Control
In this paper, we propose a novel and generalizable framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control.
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts.
Ranked #9 on
Visual Question Answering
on BenchLMM
Exploring Open-Vocabulary Semantic Segmentation without Human Labels
Although acquired extensive knowledge of visual concepts, it is non-trivial to exploit knowledge from these VL models to the task of semantic segmentation, as they are usually trained at an image level.
MammalNet: A Large-scale Video Benchmark for Mammal Recognition and Behavior Understanding
Monitoring animal behavior can facilitate conservation efforts by providing key insights into wildlife health, population status, and ecosystem function.
SLAMB: Accelerated Large Batch Training with Sparse Communication
Distributed training of large deep neural networks requires frequent exchange of massive data between machines, thus communication efficiency is a major concern.
OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?
Hence, humour generation and understanding can serve as a new task for evaluating the ability of deep-learning methods to process abstract and subjective information.
Continual Zero-Shot Learning through Semantically Guided Generative Random Walks
The GRW loss augments the training by continually encouraging the model to generate realistic and characterized samples to represent the unseen space.
Overcoming Generic Knowledge Loss with Selective Parameter Update
Our method achieves improvements on the accuracy of the newly learned tasks up to 7% while preserving the pretraining knowledge with a negligible decrease of 0. 9% on a representative control set accuracy.
Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations
We introduce Affective Visual Dialog, an emotion explanation and reasoning task as a testbed for research on understanding the formation of emotions in visually grounded conversations.
CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding
To this end, we formulate the 3D visual grounding problem as a sequence-to-sequence task by first predicting a chain of anchors and then the final target.
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others.
Ranked #10 on
Visual Question Answering
on BenchLMM
A Hybrid Graph Network for Complex Activity Detection in Video
Attention is then applied to this graph to obtain an overall representation of the local dynamic scene.
3DCoMPaT$^{++}$: An improved Large-scale 3D Vision Dataset for Compositional Recognition
In this work, we present 3DCoMPaT$^{++}$, a multimodal 2D/3D dataset with 160 million rendered views of more than 10 million stylized 3D shapes carefully annotated at the part-instance level, alongside matching RGB point clouds, 3D textured meshes, depth maps, and segmentation masks.
ToddlerDiffusion: Flash Interpretable Controllable Diffusion Model
Diffusion-based generative models excel in perceptually impressive synthesis but face challenges in interpretability.
Label Delay in Continual Learning
We introduce a new continual learning framework with explicit modeling of the label delay between data and label streams over time steps.
StoryGPT-V: Large Language Models as Consistent Story Visualizers
Therefore, we introduce \textbf{StoryGPT-V}, which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters grounded on given story descriptions.
Uni3DL: Unified Model for 3D and Language Understanding
In this work, we present Uni3DL, a unified model for 3D and Language understanding.
AI Art Neural Constellation: Revealing the Collective and Contrastive State of AI-Generated and Human Art
Our comparative analysis is based on an extensive dataset, dubbed ``ArtConstellation,'' consisting of annotations about art principles, likability, and emotions for 6, 000 WikiArt and 3, 200 AI-generated artworks.
MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens
This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM) designed specifically for video understanding.
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes
Due to the scarcity and unsuitability of existent 3D-oriented linguistic resources for this task, we first develop two large-scale and complementary visio-linguistic datasets: i) extbf{ extit{Sr3D}}, which contains 83. 5K template-based utterances leveraging extit{spatial relations} with other fine-grained object classes to localize a referred object in a given scene, and ii) extbf{ extit{Nr3D}} which contains 41. 5K extit{natural, free-form}, utterances collected by deploying a 2-player object reference game in 3D scenes.
