no code implementations • 21 May 2024 • Cusuh Ham, Matthew Fisher, James Hays, Nicholas Kolkin, Yuchen Liu, Richard Zhang, Tobias Hinz
We present personalized residuals and localized attention-guided sampling for efficient concept-driven generation using text-to-image diffusion models.
1 code implementation • 7 Mar 2024 • Ishan Khatri, Kyle Vedder, Neehar Peri, Deva Ramanan, James Hays
Current scene flow methods broadly fail to describe motion on small objects, and current scene flow evaluation protocols hide this failure by averaging over many points, with most drawn larger objects.
1 code implementation • 19 Oct 2023 • Abhinav Agarwalla, Xuhua Huang, Jason Ziglar, Francesco Ferroni, Laura Leal-Taixé, James Hays, Aljoša Ošep, Deva Ramanan
Our network is modular by design and optimized for all aspects of both the panoptic segmentation and tracking task.
no code implementations • 5 Oct 2023 • Mengyu Yang, Patrick Grady, Samarth Brahmbhatt, Arun Balajee Vasudevan, Charles C. Kemp, James Hays
How easy is it to sneak up on a robot?
no code implementations • 3 Sep 2023 • Sohan Anisetty, Amit Raj, James Hays
Mapping music to dance is a challenging problem that requires spatial and temporal coherence along with a continual synchronization with the music's progression.
no code implementations • 17 Aug 2023 • Shengcao Cao, Mengtian Li, James Hays, Deva Ramanan, Yi-Xiong Wang, Liang-Yan Gui
To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt.
no code implementations • 8 Aug 2023 • Neehar Peri, Mengtian Li, Benjamin Wilson, Yu-Xiong Wang, James Hays, Deva Ramanan
LiDAR-based 3D detection plays a vital role in autonomous navigation.
1 code implementation • 17 May 2023 • Kyle Vedder, Neehar Peri, Nathaniel Chodosh, Ishan Khatri, Eric Eaton, Dinesh Jayaraman, Yang Liu, Deva Ramanan, James Hays
Scene flow estimation is the task of describing the 3D motion field between temporally successive point clouds.
Ranked #3 on Self-supervised Scene Flow Estimation on Argoverse 2
no code implementations • 6 Apr 2023 • Akshay Krishnan, Amit Raj, Xianling Zhang, Alexandra Carlson, Nathan Tseng, Sandhya Sridhar, Nikita Jaipuria, James Hays
Specifically, we learn a scene representation that disentangles the static background and transient elements into a world-NeRF and class-specific object-NeRFs to allow compositional synthesis of multiple objects in the scene.
no code implementations • 24 Feb 2023 • Cusuh Ham, James Hays, Jingwan Lu, Krishna Kumar Singh, Zhifei Zhang, Tobias Hinz
We show that MCM enables user control over the spatial layout of the image and leads to increased control over the image generation process.
no code implementations • 5 Jan 2023 • Patrick Grady, Jeremy A. Collins, Chengcheng Tang, Christopher D. Twigg, Kunal Aneja, James Hays, Charles C. Kemp
We present a novel approach that enables diverse data to be captured with only an RGB camera and a cooperative participant.
1 code implementation • Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 2021 • Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, James Hays
Models are tasked with the prediction of future motion for "scored actors" in each scenario and are provided with track histories that capture object location, heading, velocity, and category.
2 code implementations • 14 Dec 2022 • John Lambert, James Hays
High-definition (HD) map change detection is the task of determining when sensor data and map data are no longer in agreement with one another due to real-world changes.
no code implementations • 25 Nov 2022 • Shubham Gupta, Jeet Kanjani, Mengtian Li, Francesco Ferroni, James Hays, Deva Ramanan, Shu Kong
We focus on the task of far-field 3D detection (Far3Det) of objects beyond a certain distance from an observer, e. g., $>$50m.
1 code implementation • CVPR 2023 • Yang Liu, Shen Yan, Laura Leal-Taixé, James Hays, Deva Ramanan
We draw inspiration from human visual classification studies and propose generalizing augmentation with invariant transforms to soft augmentation where the learning target softens non-linearly as a function of the degree of the transform applied to the sample: e. g., more aggressive image crop augmentations produce less confident learning targets.
1 code implementation • European Conference on Computer Vision 2022 • John Lambert, Yuguang Li, Ivaylo Boyadzhiev, Lambert Wixson, Manjunath Narayana, Will Hutchcroft, James Hays, Frank Dellaert, Sing Bing Kang
We propose a new system for automatic 2D floorplan reconstruction that is enabled by SALVe, our novel pairwise learned alignment verifier.
no code implementations • 5 Aug 2022 • Patsorn Sangkloy, Wittawat Jitkrittum, Diyi Yang, James Hays
We empirically demonstrate that using an input sketch (even a poorly drawn one) in addition to text considerably increases retrieval recall compared to traditional text-based image retrieval.
no code implementations • 29 Mar 2022 • Amit Raj, Umar Iqbal, Koki Nagano, Sameh Khamis, Pavlo Molchanov, James Hays, Jan Kautz
In this work, we present, DRaCoN, a framework for learning full-body volumetric avatars which exploits the advantages of both the 2D and 3D neural rendering techniques.
1 code implementation • 19 Mar 2022 • Patrick Grady, Chengcheng Tang, Samarth Brahmbhatt, Christopher D. Twigg, Chengde Wan, James Hays, Charles C. Kemp
We also show that the output of our model depends on the appearance of the hand and cast shadows near contact regions.
1 code implementation • 17 Mar 2022 • Cusuh Ham, Gemma Canet Tarres, Tu Bui, James Hays, Zhe Lin, John Collomosse
CoGS enables exploration of diverse appearance possibilities for a given sketched object, enabling decoupled control over the structure and the appearance of the output.
2 code implementations • CVPR 2020 • John Lambert, Zhuang Liu, Ozan Sener, James Hays, Vladlen Koltun
We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model's robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions.
Ranked #8 on Semantic Segmentation on ScanNetV2
no code implementations • CVPR 2021 • Amit Raj, Michael Zollhofer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, Stephen Lombardi
Volumetric models typically employ a global code to represent facial expressions, such that they can be driven by a small set of animation parameters.
Ranked #5 on Generalizable Novel View Synthesis on ZJU-MoCap
no code implementations • 7 Jan 2021 • Amit Raj, Michael Zollhoefer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, Stephen Lombardi
Volumetric models typically employ a global code to represent facial expressions, such that they can be driven by a small set of animation parameters.
no code implementations • CVPR 2021 • Amit Raj, Julian Tanke, James Hays, Minh Vo, Carsten Stoll, Christoph Lassner
The combination of traditional rendering with neural networks in Deferred Neural Rendering (DNR) provides a compelling balance between computational complexity and realism of the resulting images.
no code implementations • 31 Oct 2020 • Jhony Kaesemodel Pontes, James Hays, Simon Lucey
Our proposed objective function can be used with or without learning---as a self-supervisory signal to learn scene flow representations, or as a non-learning-based method in which the scene flow is optimized during runtime.
no code implementations • 24 Aug 2020 • Benjamin Wilson, Zsolt Kira, James Hays
In this work, we address the long-tail problem by leveraging both the large class-taxonomies of modern 2D datasets and the robustness of state-of-the-art 2D detection methods.
2 code implementations • ECCV 2020 • Daniel Bolya, Sean Foley, James Hays, Judy Hoffman
We introduce TIDE, a framework and associated toolbox for analyzing the sources of error in object detection and instance segmentation algorithms.
2 code implementations • ECCV 2020 • Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, James Hays
We introduce ContactPose, the first dataset of hand-object contact paired with hand pose, object pose, and RGB-D images.
Ranked #1 on Grasp Contact Prediction on ContactPose
3 code implementations • CVPR 2019 • Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, James Hays
In our baseline experiments, we illustrate how detailed map information such as lane direction, driveable area, and ground height improves the accuracy of 3D object tracking and motion forecasting.
no code implementations • 17 Jul 2019 • Samarth Brahmbhatt, Charles C. Kemp, James Hays
However, grasp capture - capturing the pose of a hand grasping an object, and orienting it w. r. t.
1 code implementation • 14 May 2019 • Wittawat Jitkrittum, Patsorn Sangkloy, Muhammad Waleed Gondal, Amit Raj, James Hays, Bernhard Schölkopf
We propose a novel procedure which adds "content-addressability" to any given unconditional implicit model e. g., a generative adversarial network (GAN).
2 code implementations • CVPR 2019 • Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, James Hays
We present ContactDB, a novel dataset of contact maps for household objects that captures the rich hand-object contact that occurs during grasping, enabled by use of a thermal camera.
Ranked #1 on Grasp Contact Prediction on ContactDB
4 code implementations • 7 Apr 2019 • Samarth Brahmbhatt, Ankur Handa, James Hays, Dieter Fox
Using a dataset of contact demonstrations from humans grasping diverse household objects, we synthesize functional grasps for three hand models and two functional intents.
2 code implementations • 2 Mar 2019 • Nam Vo, Lu Jiang, James Hays
In this work we show how one can learn transformations with no training examples by learning them on another domain and then transfer to the target domain.
4 code implementations • CVPR 2019 • Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, James Hays
In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image.
Ranked #2 on Image Retrieval with Multi-Modal Query on MIT-States
3 code implementations • NeurIPS 2018 • Wittawat Jitkrittum, Heishiro Kanagawa, Patsorn Sangkloy, James Hays, Bernhard Schölkopf, Arthur Gretton
Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models.
1 code implementation • ECCV 2018 • Amit Raj, Patsorn Sangkloy, Huiwen Chang, Jingwan Lu, Duygu Ceylan, James Hays
Garment transfer is a challenging task that requires (i) disentangling the features of the clothing from the body pose and shape and (ii) realistic synthesis of the garment texture on the new body.
Ranked #1 on Virtual Try-on on FashionIQ (using extra training data)
1 code implementation • 8 Mar 2018 • Nam Vo, James Hays
This work studies deep metric learning under small to medium scale data as we believe that better generalization could be a contributing factor to the improvement of previous fine-grained image retrieval methods; it should be considered when designing future techniques.
1 code implementation • 23 Jan 2018 • Daniel Castro, Steven Hickson, Patsorn Sangkloy, Bhavishya Mittal, Sean Dai, James Hays, Irfan Essa
We present a comparison of numerous state-of-the-art techniques on our dataset using three different representations (video, optical flow and multi-person pose data) in order to analyze these approaches.
1 code implementation • CVPR 2018 • Wengling Chen, James Hays
Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision.
1 code implementation • CVPR 2018 • Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz
Maps are a key component in image-based camera localization and visual SLAM systems: they are used to establish geometric constraints between images, correct drift in relative pose estimation, and relocalize cameras after lost tracking.
Ranked #5 on Visual Localization on Oxford RobotCar Full
2 code implementations • CVPR 2018 • Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang, Fisher Yu, James Hays
In this paper, we investigate deep image synthesis guided by sketch, color, and texture.
Ranked #2 on Image Reconstruction on Edge-to-Shoes
8 code implementations • ICLR 2018 • Naveen Kodali, Jacob Abernethy, James Hays, Zsolt Kira
We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions.
no code implementations • ICCV 2017 • Nam Vo, Nathan Jacobs, James Hays
The recent state-of-the-art approach to this problem is a deep image classification approach in which the world is spatially divided into cells and a deep network is trained to predict the correct cell for a given image.
Ranked #1 on Photo geolocation estimation on Im2GPS (Reference images metric)
no code implementations • CVPR 2017 • Samarth Brahmbhatt, James Hays
We present DeepNav, a Convolutional Neural Network (CNN) based algorithm for navigating large cities using locally visible street-view images.
no code implementations • 26 Jan 2017 • Libin Sun, James Hays
Hallucinating high frequency image details in single image super-resolution is a challenging task.
no code implementations • 17 Jan 2017 • Unaiza Ahsan, Chen Sun, James Hays, Irfan Essa
We propose to leverage concept-level representations for complex event recognition in photographs given limited training examples.
1 code implementation • CVPR 2017 • Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, James Hays
In this paper, we propose a deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces.
1 code implementation • 19 Oct 2016 • Samarth Brahmbhatt, Henrik I. Christensen, James Hays
Through experiments on Pascal VOC 2007 and 2012, we demonstrate the effectiveness of this method and show that StuffNet also significantly improves object detection performance on such datasets.
1 code implementation • ECCV 2016 • Genevieve Patterson, James Hays
In this paper, we discover and annotate visual attributes for the COCO dataset.
no code implementations • 30 Jul 2016 • Nam Vo, James Hays
In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e. g. satellite) images.
no code implementations • CVPR 2016 • Kilho Son, Daniel Moreno, James Hays, David B. Cooper
To reconstruct such challenging puzzles, we aim to search for piece configurations which maximize the size of consensus (i. e. grid or loop) configurations which represent a geometric consensus or agreement among pieces.
no code implementations • CVPR 2016 • Hani Altwaijry, Eduard Trulls, James Hays, Pascal Fua, Serge Belongie
We demonstrate that our models outperform the state-of-the-art on ultra-wide baseline matching and approach human accuracy.
no code implementations • 30 Jun 2015 • Libin Sun, Brian Guenter, Neel Joshi, Patrick Therien, James Hays
Unfortunately, custom lens design is costly (thousands to tens of thousands of dollars), time consuming (10-12 weeks typical lead time), and requires specialized optics design expertise.
no code implementations • CVPR 2015 • Tsung-Yi Lin, Yin Cui, Serge Belongie, James Hays
Most approaches predict the location of a query image by matching to ground-level images with known locations (e. g., street-view data).
35 code implementations • 1 May 2014 • Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, Piotr Dollár
We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding.
no code implementations • CVPR 2013 • Tsung-Yi Lin, Serge Belongie, James Hays
On the other hand, there is no shortage of visual and geographic data that densely covers the Earth we examine overhead imagery and land cover survey data but the relationship between this data and ground level query photographs is complex.
no code implementations • CVPR 2013 • Yinda Zhang, Jianxiong Xiao, James Hays, Ping Tan
We analyze the self-similarity of the guide image to generate a set of allowable local transformations and apply them to the input image.
no code implementations • ECCV 2012 • Frank Palermo, James Hays, Alexei A. Efros
We introduce the task of automatically estimating the age of historical color photographs.