Search Results for author: James Hays

Found 57 papers, 29 papers with code

I Can't Believe It's Not Scene Flow!

1 code implementation7 Mar 2024 Ishan Khatri, Kyle Vedder, Neehar Peri, Deva Ramanan, James Hays

Current scene flow methods broadly fail to describe motion on small objects, and current scene flow evaluation protocols hide this failure by averaging over many points, with most drawn larger objects.

MAGMA: Music Aligned Generative Motion Autodecoder

no code implementations3 Sep 2023 Sohan Anisetty, Amit Raj, James Hays

Mapping music to dance is a challenging problem that requires spatial and temporal coherence along with a continual synchronization with the music's progression.

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

no code implementations17 Aug 2023 Shengcao Cao, Mengtian Li, James Hays, Deva Ramanan, Yi-Xiong Wang, Liang-Yan Gui

To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt.

Edge-computing Instance Segmentation +5

LANe: Lighting-Aware Neural Fields for Compositional Scene Synthesis

no code implementations6 Apr 2023 Akshay Krishnan, Amit Raj, Xianling Zhang, Alexandra Carlson, Nathan Tseng, Sandhya Sridhar, Nikita Jaipuria, James Hays

Specifically, we learn a scene representation that disentangles the static background and transient elements into a world-NeRF and class-specific object-NeRFs to allow compositional synthesis of multiple objects in the scene.

Object

Modulating Pretrained Diffusion Models for Multimodal Image Synthesis

no code implementations24 Feb 2023 Cusuh Ham, James Hays, Jingwan Lu, Krishna Kumar Singh, Zhifei Zhang, Tobias Hinz

We show that MCM enables user control over the spatial layout of the image and leads to increased control over the image generation process.

Image Generation Semantic Segmentation

PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images

no code implementations5 Jan 2023 Patrick Grady, Jeremy A. Collins, Chengcheng Tang, Christopher D. Twigg, Kunal Aneja, James Hays, Charles C. Kemp

We present a novel approach that enables diverse data to be captured with only an RGB camera and a cooperative participant.

Mixed Reality

Trust, but Verify: Cross-Modality Fusion for HD Map Change Detection

2 code implementations14 Dec 2022 John Lambert, James Hays

High-definition (HD) map change detection is the task of determining when sensor data and map data are no longer in agreement with one another due to real-world changes.

Change Detection

Far3Det: Towards Far-Field 3D Detection

no code implementations25 Nov 2022 Shubham Gupta, Jeet Kanjani, Mengtian Li, Francesco Ferroni, James Hays, Deva Ramanan, Shu Kong

We focus on the task of far-field 3D detection (Far3Det) of objects beyond a certain distance from an observer, e. g., $>$50m.

Autonomous Vehicles Philosophy

Soft Augmentation for Image Classification

1 code implementation CVPR 2023 Yang Liu, Shen Yan, Laura Leal-Taixé, James Hays, Deva Ramanan

We draw inspiration from human visual classification studies and propose generalizing augmentation with invariant transforms to soft augmentation where the learning target softens non-linearly as a function of the degree of the transform applied to the sample: e. g., more aggressive image crop augmentations produce less confident learning targets.

Classification Data Augmentation +1

A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch

no code implementations5 Aug 2022 Patsorn Sangkloy, Wittawat Jitkrittum, Diyi Yang, James Hays

We empirically demonstrate that using an input sketch (even a poorly drawn one) in addition to text considerably increases retrieval recall compared to traditional text-based image retrieval.

Image Retrieval Retrieval

DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars

no code implementations29 Mar 2022 Amit Raj, Umar Iqbal, Koki Nagano, Sameh Khamis, Pavlo Molchanov, James Hays, Jan Kautz

In this work, we present, DRaCoN, a framework for learning full-body volumetric avatars which exploits the advantages of both the 2D and 3D neural rendering techniques.

Neural Rendering

PressureVision: Estimating Hand Pressure from a Single RGB Image

1 code implementation19 Mar 2022 Patrick Grady, Chengcheng Tang, Samarth Brahmbhatt, Christopher D. Twigg, Chengde Wan, James Hays, Charles C. Kemp

We also show that the output of our model depends on the appearance of the hand and cast shadows near contact regions.

Contact mechanics

CoGS: Controllable Generation and Search from Sketch and Style

1 code implementation17 Mar 2022 Cusuh Ham, Gemma Canet Tarres, Tu Bui, James Hays, Zhe Lin, John Collomosse

CoGS enables exploration of diverse appearance possibilities for a given sketched object, enabling decoupled control over the structure and the appearance of the output.

Object

MSeg: A Composite Dataset for Multi-domain Semantic Segmentation

2 code implementations CVPR 2020 John Lambert, Zhuang Liu, Ozan Sener, James Hays, Vladlen Koltun

We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model's robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions.

Computational Efficiency Instance Segmentation +3

Pixel-Aligned Volumetric Avatars

no code implementations CVPR 2021 Amit Raj, Michael Zollhofer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, Stephen Lombardi

Volumetric models typically employ a global code to represent facial expressions, such that they can be driven by a small set of animation parameters.

Generalizable Novel View Synthesis

PVA: Pixel-aligned Volumetric Avatars

no code implementations7 Jan 2021 Amit Raj, Michael Zollhoefer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, Stephen Lombardi

Volumetric models typically employ a global code to represent facial expressions, such that they can be driven by a small set of animation parameters.

ANR: Articulated Neural Rendering for Virtual Avatars

no code implementations CVPR 2021 Amit Raj, Julian Tanke, James Hays, Minh Vo, Carsten Stoll, Christoph Lassner

The combination of traditional rendering with neural networks in Deferred Neural Rendering (DNR) provides a compelling balance between computational complexity and realism of the resulting images.

Neural Rendering

Scene Flow from Point Clouds with or without Learning

no code implementations31 Oct 2020 Jhony Kaesemodel Pontes, James Hays, Simon Lucey

Our proposed objective function can be used with or without learning---as a self-supervisory signal to learn scene flow representations, or as a non-learning-based method in which the scene flow is optimized during runtime.

Motion Segmentation Self-Supervised Learning

3D for Free: Crossmodal Transfer Learning using HD Maps

no code implementations24 Aug 2020 Benjamin Wilson, Zsolt Kira, James Hays

In this work, we address the long-tail problem by leveraging both the large class-taxonomies of modern 2D datasets and the robustness of state-of-the-art 2D detection methods.

3D Object Detection Autonomous Driving +5

TIDE: A General Toolbox for Identifying Object Detection Errors

2 code implementations ECCV 2020 Daniel Bolya, Sean Foley, James Hays, Judy Hoffman

We introduce TIDE, a framework and associated toolbox for analyzing the sources of error in object detection and instance segmentation algorithms.

Instance Segmentation object-detection +2

Argoverse: 3D Tracking and Forecasting with Rich Maps

3 code implementations CVPR 2019 Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, James Hays

In our baseline experiments, we illustrate how detailed map information such as lane direction, driveable area, and ground height improves the accuracy of 3D object tracking and motion forecasting.

3D Object Tracking Autonomous Vehicles +3

Towards Markerless Grasp Capture

no code implementations17 Jul 2019 Samarth Brahmbhatt, Charles C. Kemp, James Hays

However, grasp capture - capturing the pose of a hand grasping an object, and orienting it w. r. t.

Hand Pose Estimation Object

Kernel Mean Matching for Content Addressability of GANs

1 code implementation14 May 2019 Wittawat Jitkrittum, Patsorn Sangkloy, Muhammad Waleed Gondal, Amit Raj, James Hays, Bernhard Schölkopf

We propose a novel procedure which adds "content-addressability" to any given unconditional implicit model e. g., a generative adversarial network (GAN).

Generative Adversarial Network Image Generation

ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging

2 code implementations CVPR 2019 Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, James Hays

We present ContactDB, a novel dataset of contact maps for household objects that captures the rich hand-object contact that occurs during grasping, enabled by use of a thermal camera.

Grasp Contact Prediction Object +1

ContactGrasp: Functional Multi-finger Grasp Synthesis from Contact

4 code implementations7 Apr 2019 Samarth Brahmbhatt, Ankur Handa, James Hays, Dieter Fox

Using a dataset of contact demonstrations from humans grasping diverse household objects, we synthesize functional grasps for three hand models and two functional intents.

Object

Let's Transfer Transformations of Shared Semantic Representations

2 code implementations2 Mar 2019 Nam Vo, Lu Jiang, James Hays

In this work we show how one can learn transformations with no training examples by learning them on another domain and then transfer to the target domain.

Attribute Image Retrieval +3

Composing Text and Image for Image Retrieval - An Empirical Odyssey

4 code implementations CVPR 2019 Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, James Hays

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image.

Image Retrieval Image Retrieval with Multi-Modal Query +1

Informative Features for Model Comparison

3 code implementations NeurIPS 2018 Wittawat Jitkrittum, Heishiro Kanagawa, Patsorn Sangkloy, James Hays, Bernhard Schölkopf, Arthur Gretton

Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models.

SwapNet: Garment Transfer in Single View Images

1 code implementation ECCV 2018 Amit Raj, Patsorn Sangkloy, Huiwen Chang, Jingwan Lu, Duygu Ceylan, James Hays

Garment transfer is a challenging task that requires (i) disentangling the features of the clothing from the body pose and shape and (ii) realistic synthesis of the garment texture on the new body.

 Ranked #1 on Virtual Try-on on FashionIQ (using extra training data)

Virtual Try-on

Generalization in Metric Learning: Should the Embedding Layer be the Embedding Layer?

1 code implementation8 Mar 2018 Nam Vo, James Hays

This work studies deep metric learning under small to medium scale data as we believe that better generalization could be a contributing factor to the improvement of previous fine-grained image retrieval methods; it should be considered when designing future techniques.

Image Retrieval Metric Learning +1

Let's Dance: Learning From Online Dance Videos

1 code implementation23 Jan 2018 Daniel Castro, Steven Hickson, Patsorn Sangkloy, Bhavishya Mittal, Sean Dai, James Hays, Irfan Essa

We present a comparison of numerous state-of-the-art techniques on our dataset using three different representations (video, optical flow and multi-person pose data) in order to analyze these approaches.

Action Recognition Optical Flow Estimation +1

Geometry-Aware Learning of Maps for Camera Localization

1 code implementation CVPR 2018 Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz

Maps are a key component in image-based camera localization and visual SLAM systems: they are used to establish geometric constraints between images, correct drift in relative pose estimation, and relocalize cameras after lost tracking.

Camera Localization Visual Localization

On Convergence and Stability of GANs

8 code implementations ICLR 2018 Naveen Kodali, Jacob Abernethy, James Hays, Zsolt Kira

We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions.

Revisiting IM2GPS in the Deep Learning Era

no code implementations ICCV 2017 Nam Vo, Nathan Jacobs, James Hays

The recent state-of-the-art approach to this problem is a deep image classification approach in which the world is spatially divided into cells and a deep network is trained to predict the correct cell for a given image.

Classification Density Estimation +3

DeepNav: Learning to Navigate Large Cities

no code implementations CVPR 2017 Samarth Brahmbhatt, James Hays

We present DeepNav, a Convolutional Neural Network (CNN) based algorithm for navigating large cities using locally visible street-view images.

Navigate

Super-resolution Using Constrained Deep Texture Synthesis

no code implementations26 Jan 2017 Libin Sun, James Hays

Hallucinating high frequency image details in single image super-resolution is a challenging task.

Image Super-Resolution Texture Synthesis

Complex Event Recognition from Images with Few Training Examples

no code implementations17 Jan 2017 Unaiza Ahsan, Chen Sun, James Hays, Irfan Essa

We propose to leverage concept-level representations for complex event recognition in photographs given limited training examples.

Scribbler: Controlling Deep Image Synthesis with Sketch and Color

1 code implementation CVPR 2017 Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, James Hays

In this paper, we propose a deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces.

Colorization Image Generation

StuffNet: Using 'Stuff' to Improve Object Detection

1 code implementation19 Oct 2016 Samarth Brahmbhatt, Henrik I. Christensen, James Hays

Through experiments on Pascal VOC 2007 and 2012, we demonstrate the effectiveness of this method and show that StuffNet also significantly improves object detection performance on such datasets.

Object object-detection +3

Localizing and Orienting Street Views Using Overhead Imagery

no code implementations30 Jul 2016 Nam Vo, James Hays

In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e. g. satellite) images.

General Classification Retrieval

Solving Small-Piece Jigsaw Puzzles by Growing Consensus

no code implementations CVPR 2016 Kilho Son, Daniel Moreno, James Hays, David B. Cooper

To reconstruct such challenging puzzles, we aim to search for piece configurations which maximize the size of consensus (i. e. grid or loop) configurations which represent a geometric consensus or agreement among pieces.

Learning to Match Aerial Images With Deep Attentive Architectures

no code implementations CVPR 2016 Hani Altwaijry, Eduard Trulls, James Hays, Pascal Fua, Serge Belongie

We demonstrate that our models outperform the state-of-the-art on ultra-wide baseline matching and approach human accuracy.

Lens Factory: Automatic Lens Generation Using Off-the-shelf Components

no code implementations30 Jun 2015 Libin Sun, Brian Guenter, Neel Joshi, Patrick Therien, James Hays

Unfortunately, custom lens design is costly (thousands to tens of thousands of dollars), time consuming (10-12 weeks typical lead time), and requires specialized optics design expertise.

Learning Deep Representations for Ground-to-Aerial Geolocalization

no code implementations CVPR 2015 Tsung-Yi Lin, Yin Cui, Serge Belongie, James Hays

Most approaches predict the location of a query image by matching to ground-level images with known locations (e. g., street-view data).

Face Verification

Microsoft COCO: Common Objects in Context

33 code implementations1 May 2014 Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, Piotr Dollár

We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding.

Instance Segmentation Object +5

FrameBreak: Dramatic Image Extrapolation by Guided Shift-Maps

no code implementations CVPR 2013 Yinda Zhang, Jianxiong Xiao, James Hays, Ping Tan

We analyze the self-similarity of the guide image to generate a set of allowable local transformations and apply them to the input image.

Image Generation

Cross-View Image Geolocalization

no code implementations CVPR 2013 Tsung-Yi Lin, Serge Belongie, James Hays

On the other hand, there is no shortage of visual and geographic data that densely covers the Earth we examine overhead imagery and land cover survey data but the relationship between this data and ground level query photographs is complex.

Dating historical color images

no code implementations ECCV 2012 Frank Palermo, James Hays, Alexei A. Efros

We introduce the task of automatically estimating the age of historical color photographs.

Age Estimation

Cannot find the paper you are looking for? You can Submit a new open access paper.