Search Results for author:
Found 115 papers, 48 papers with code
Two Stream Active Query Suggestion for Active Learning in Connectomics
As a use case, we build an end-to-end active learning framework with our query suggestion method for 3D synapse detection and mitochondria segmentation in connectomics.
Visual Acoustic Fields
Meanwhile, the sound localization module enables querying the 3D scene, represented by the feature-augmented 3DGS, to localize hitting positions based on the sound sources.
4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models
To address these challenges, we propose 4D LangSplat, which learns 4D language fields to handle time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes efficiently.
Generalization of CNNs on Relational Reasoning with Bar Charts
We then test the generalization performance of CNNs on a classic relational reasoning task: estimating bar length ratios in a bar chart, by progressively perturbing the standard visualizations.
S-LoRA: Scalable Low-Rank Adaptation for Class Incremental Learning
Continual Learning with foundation models has recently emerged as a promising approach to harnessing the power of pre-trained models for sequential tasks.
Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
As a common image editing operation, image composition involves integrating foreground objects into background scenes.
GPT-2 Through the Lens of Vector Symbolic Architectures
Understanding the general priniciples behind transformer models remains a complex endeavor.
Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Models
Vision Language Models (VLMs) have been successful at many chart comprehension tasks that require attending to both the images of charts and their accompanying textual descriptions.
SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization
The manual prompt design process for LLMs at the reasoning phase is tedious and an automated prompt optimization method is desired.
Ranked #1 on
Visual Social Relationship Recognition
on PIPA
Multimodal Learning for Embryo Viability Prediction in Clinical IVF
In clinical In-Vitro Fertilization (IVF), identifying the most viable embryo for transfer is important to increasing the likelihood of a successful pregnancy.
Tree of Attributes Prompt Learning for Vision-Language Models
To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a "concept - attribute - description" structure for each category, and then learn the hierarchy with vision and text prompt tokens.
Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models
To address this, we propose Concept2Concept, a framework where we characterize conditional distributions of vision language models using interpretable concepts and metrics that can be defined in terms of these concepts.
Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuning
Additionally, based on our exploration of TSD, we focus on an important issue in PEFT: the initialization of LoRA.
MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality
MoRA projects each input to a low intrinsic dimension but uses different modality-aware up-projections for modality-specific adaptation in cases of missing modalities.
Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment
Second, we introduce new evaluation metrics to rigorously assess sound source localization methods, focusing on accurately evaluating both localization performance and cross-modal interaction ability.
Lite2Relight: 3D-aware Single Image Portrait Relighting
Achieving photorealistic 3D view synthesis and relighting of human portraits is pivotal for advancing AR/VR applications.
They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias
Vision Language Models (VLMs) such as CLIP are powerful models; however they can exhibit unwanted biases, making them less safe when deployed directly in applications such as text-to-image, text-to-video retrievals, reverse search, or classification tasks.
Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex
Using \textit{MacaqueITBench}, we investigated the impact of distribution shifts on models predicting neural activity by dividing the images into Out-Of-Distribution (OOD) train and test splits.
Learning Gaze-aware Compositional GAN
In this work, we present a generative framework to create annotated gaze data by leveraging the benefits of labeled and unlabeled data sources.
Frenet-Serret Frame-based Decomposition for Part Segmentation of 3D Curvilinear Structures
Accurately segmenting 3D curvilinear structures in medical imaging remains challenging due to their complex geometry and the scarcity of diverse, large-scale datasets for algorithm development and evaluation.
R^2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding
Video temporal grounding (VTG) is a fine-grained video understanding problem that aims to ground relevant clips in untrimmed videos given natural language queries.
Joint-Task Regularization for Partially Labeled Multi-Task Learning
Multi-task learning has become increasingly popular in the machine learning field, but its practicality is hindered by the need for large, labeled datasets.
$R^2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding
Video temporal grounding (VTG) is a fine-grained video understanding problem that aims to ground relevant clips in untrimmed videos given natural language queries.
Ranked #6 on
Highlight Detection
on QVHighlights
Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge
The resulting FracNet+ demonstrates competitive performance in rib fracture detection, which lays a foundation for further research and development in AI-assisted rib fracture detection and diagnosis.
Measuring and Controlling Instruction (In)Stability in Language Model Dialogs
System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction.
GenLens: A Systematic Evaluation of Visual GenAI Model Outputs
This research underscores the importance of robust early-stage evaluation tools in GenAI development, contributing to the advancement of fair and high-quality GenAI models.
The Indoor-Training Effect: unexpected gains from distribution shifts in the transition function
We refer to this process as Noise Injection and the resulting environments as {\delta}-environments.
TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images
While imaging techniques at macro and mesoscales have garnered substantial attention and resources, microscale Volume Electron Microscopy (vEM) imaging, capable of revealing intricate vascular details, has lacked the necessary benchmarking infrastructure.
LangSplat: 3D Language Gaussian Splatting
Humans live in a 3D world and commonly use natural language to interact with a 3D scene.
Unraveling the Temporal Dynamics of the Unet in Diffusion Models
We propose an analytical method to systematically assess the impact of time steps and core Unet components on the final output.
S$^3$-TTA: Scale-Style Selection for Test-Time Augmentation in Biomedical Image Segmentation
Deep-learning models have been successful in biomedical image segmentation.
Structure-Preserving Instance Segmentation via Skeleton-Aware Distance Transform
Objects with complex structures pose significant challenges to existing instance segmentation methods that rely on boundary or affinity maps, which are vulnerable to small errors around contacting pixels that cause noticeable connectivity change.
Sound Source Localization is All about Cross-Modal Alignment
However, prior arts and existing benchmarks do not account for a more important aspect of the problem, cross-modal semantic understanding, which is essential for genuine sound source localization.
CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation
Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability.
The time is ripe to reverse engineer an entire nervous system: simulating behavior from neural interactions
Here, we argue that such completeness is possible in C. elegans; a complete description that goes all the way from the activity of every neuron to predict behavior.
A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision
Existing model evaluation tools mainly focus on evaluating classification models, leaving a gap in evaluating more complex models, such as object detection.
Domain-Scalable Unpaired Image Translation via Latent Space Anchoring
However, given a UNIT model trained on certain domains, it is difficult for current methods to incorporate new domains because they often need to train the full model on both existing and new domains.
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
This intervention significantly improves the performance of LLaMA models on the TruthfulQA benchmark.
Masked Image Training for Generalizable Deep Image Denoising
To address this issue, we present a novel approach to enhance the generalization performance of denoising networks, known as masked training.
The XPRESS Challenge: Xray Projectomic Reconstruction -- Extracting Segmentation with Skeletons
In this task, we provide volumetric XNH images of cortical white matter axons from the mouse brain along with ground truth annotations for axon trajectories.
An Out-of-Domain Synapse Detection Challenge for Microwasp Brain Connectomes
The size of image stacks in connectomics studies now reaches the terabyte and often petabyte scales with a great diversity of appearance across brain regions and samples.
QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity
The mechanism of existing style transfer algorithms is by minimizing a hybrid loss function to push the generated image toward high similarities in both content and style.
ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal
Recent deep learning methods have achieved promising results in image shadow removal.
Ranked #9 on
Shadow Removal
on ISTD+
Can Machines Imitate Humans? Integrative Turing Tests for Vision and Language Demonstrate a Narrowing Gap
To address this question, we turn to the Turing test and systematically benchmark current AIs in their abilities to imitate humans in three language tasks (Image captioning, Word association, and Conversation) and three vision tasks (Object detection, Color estimation, and Attention prediction).
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Language models show a surprising range of capabilities, but the source of their apparent competence is unclear.
RibSeg v2: A Large-scale Benchmark for Rib Labeling and Anatomical Centerline Extraction
Based on the RibSeg v2, we develop a pipeline including deep learning-based methods for rib labeling, and a skeletonization-based method for centerline extraction.
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training.
Improving generalization by mimicking the human visual diet
We present a new perspective on bridging the generalization gap between biological and computer vision -- mimicking the human visual diet.
Diagnosing Ensemble Few-Shot Classifiers
The base learners and labeled samples (shots) in an ensemble few-shot classifier greatly affect the model performance.
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction
Existing leading methods for spectral reconstruction (SR) focus on designing deeper or wider convolutional neural networks (CNNs) to learn the end-to-end mapping from the RGB image to its hyperspectral image (HSI).
Ranked #1 on
Spectral Reconstruction
on ARAD-1K
Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training
Additionally, for better noise fitting, we present an efficient architecture Simple Multi-scale Network (SMNet) as the generator.
Ranked #1 on
Noise Estimation
on SIDD
Instance Segmentation of Unlabeled Modalities via Cyclic Segmentation GAN
Instance segmentation for unlabeled imaging modalities is a challenging but essential task as collecting expert annotation can be expensive and time-consuming.
Revisiting RCAN: Improved Training for Image Super-Resolution
Image super-resolution (SR) is a fast-moving field with novel architectures attracting the spotlight.
Texture-Based Error Analysis for Image Super-Resolution
Our key contribution is to leverage a texture classifier, which enables us to assign patches with semantic labels, to identify the source of SR errors both globally and locally.
YouMVOS: An Actor-Centric Multi-Shot Video Object Segmentation Dataset
Many video understanding tasks require analyzing multi-shot videos, but existing datasets for video object segmentation (VOS) only consider single-shot videos.
PyTorch Connectomics: A Scalable and Flexible Segmentation Framework for EM Connectomics
We present PyTorch Connectomics (PyTC), an open-source deep-learning framework for the semantic and instance segmentation of volumetric microscopy images, built upon PyTorch.
Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation
Motivated by this, we propose a video instance segmentation method that alleviates the problem due to missing detections.
Ranked #31 on
Video Instance Segmentation
on YouTube-VIS validation
Three approaches to facilitate DNN generalization to objects in out-of-distribution orientations and illuminations
While humans have a remarkable capability of recognizing objects in out-of-distribution (OoD) orientations and illuminations, Deep Neural Networks (DNNs) severely suffer in this case, even when large amounts of training examples are available.
MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification
We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D.
GenNI: Human-AI Collaboration for Data-Backed Text Generation
These controls allow users to globally constrain model generations, without sacrificing the representation power of the deep learning models.
Scope2Screen: Focus+Context Techniques for Pathology Tumor Assessment in Multivariate Image Data
Inspection of tissues using a light microscope is the primary method of diagnosing many diseases, notably cancer.
Intervention-based Recurrent Casual Model for Non-stationary Video Causal Discovery
Non-stationary casual structures are prevalent in real-world physical systems.
Emergent Neural Network Mechanisms for Generalization to Objects in Novel Orientations
The capability of Deep Neural Networks (DNNs) to recognize objects in orientations outside the distribution of the training data is not well understood.
Asymmetric 3D Context Fusion for Universal Lesion Detection
The proposed A3D consistently outperforms symmetric context fusion operators by considerable margins, and establishes a new \emph{state of the art} on DeepLesion.
RibSeg Dataset and Strong Point Cloud Baselines for Rib Segmentation from CT Scans
Manual rib inspections in computed tomography (CT) scans are clinically critical but labor-intensive, as 24 ribs are typically elongated and oblique in 3D volumes.
Developmental Stage Classification of EmbryosUsing Two-Stream Neural Network with Linear-Chain Conditional Random Field
In this work, we propose a two-stream model for developmental stage classification.
NucMM Dataset: 3D Neuronal Nuclei Instance Segmentation at Sub-Cubic Millimeter Scale
Segmenting 3D cell nuclei from microscopy image volumes is critical for biological and clinical analysis, enabling the study of cellular expression patterns and cell lineages.
AxonEM Dataset: 3D Axon Instance Segmentation of Brain Cortical Regions
Electron microscopy (EM) enables the reconstruction of neural circuits at the level of individual synapses, which has been transformative for scientific discoveries.
In-distribution adversarial attacks on object recognition models using gradient-free search
Past works attribute these errors to dataset bias, claiming that models fail on these perturbed samples as they do not belong to the training data distribution.
VICE: Visual Identification and Correction of Neural Circuit Errors
A connectivity graph of neurons at the resolution of single synapses provides scientists with a tool for understanding the nervous system in health and disease.
Discrete Cosine Transform Network for Guided Depth Map Super-Resolution
Guided depth super-resolution (GDSR) is an essential topic in multi-modal image processing, which reconstructs high-resolution (HR) depth maps from low-resolution ones collected with suboptimal conditions with the help of HR RGB images of the same scene.
PhotoApp: Photorealistic Appearance Editing of Head Portraits
We present an approach for high-quality intuitive editing of the camera viewpoint and scene illumination in a portrait image.
Consistent Recurrent Neural Networks for 3D Neuron Segmentation
We present a recurrent network for the 3D reconstruction of neurons that sequentially generates binary masks for every object in an image with spatio-temporal consistency.
On the Capability of CNNs to Generalize to Unseen Category-Viewpoint Combinations
We find that learning category and viewpoint in separate networks compared to a shared one leads to an increase in selectivity and invariance, as separate networks are not forced to preserve information about both category and viewpoint.
Context Reasoning Attention Network for Image Super-Resolution
However, the basic convolutional layer in CNNs is designed to extract local patterns, lacking the ability to model global context.
Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution
Specifically, we propose a dynamic high-pass filtering (HPF) module that locally applies adaptive filter weights for each spatial location and channel group to preserve high-frequency signals.
Learning Vector Quantized Shape Code for Amodal Blastomere Instance Segmentation
To alleviate the problem, we propose to classify input features into intermediate shape codes and recover complete object shapes from them.
MitoEM Dataset: Large-scale 3D Mitochondria Instance Segmentation from EM Images
On MitoEM, we find existing instance segmentation methods often fail to correctly segment mitochondria with complex shapes or close contacts with other instances.
Ranked #2 on
3D Instance Segmentation
on MitoEM
(AP75-R-Test metric)
Monocular Reconstruction of Neural Face Reflectance Fields
The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing.
When and how CNNs generalize to out-of-distribution category-viewpoint combinations
In this paper, we investigate when and how such OOD generalization may be possible by evaluating CNNs trained to classify both object category and 3D viewpoint on OOD combinations, and identifying the neural mechanisms that facilitate such OOD generalization.
Automated Measurements of Key Morphological Features of Human Embryos for IVF
A major challenge in clinical In-Vitro Fertilization (IVF) is selecting the highest quality embryo to transfer to the patient in the hopes of achieving a pregnancy.
A New Age of Computing and the Brain
In December 2014, a two-day workshop supported by the Computing Community Consortium (CCC) and the National Science Foundation's Computer and Information Science and Engineering Directorate (NSF CISE) was convened in Washington, DC, with the goal of bringing together computer scientists and brain researchers to explore these new opportunities and connections, and develop a new, modern dialogue between the two research communities.
A Topological Nomenclature for 3D Shape Analysis in Connectomics
Next, we develop nomenclature rules for pyramidal neurons and mitochondria from the reduced graph and finally learn the feature embedding for shape manipulation.
White-Box Adversarial Defense via Self-Supervised Data Estimation
In this paper, we study the problem of how to defend classifiers against adversarial attacks that fool the classifiers using subtly modified input data.
FDive: Learning Relevance Models using Pattern-based Similarity Measures
The detection of interesting patterns in large high-dimensional datasets is difficult because of their dimensionality and pattern complexity.
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference
Automation of tasks can have critical consequences when humans lose agency over decision processes.
Fast Mitochondria Detection for Connectomics
High-resolution connectomics data allows for the identification of dysfunctional mitochondria which are linked to a variety of diseases such as autism or bipolar.
Debugging Sequence-to-Sequence Models with Seq2Seq-Vis
Neural attention-based sequence-to-sequence models (seq2seq) (Sutskever et al., 2014; Bahdanau et al., 2014) have proven to be accurate and robust for many sequence prediction tasks.
Parallel Separable 3D Convolution for Video and Volumetric Data Understanding
For video and volumetric data understanding, 3D convolution layers are widely used in deep learning, however, at the cost of increasing computation and training time.
Synthetically Trained Icon Proposals for Parsing and Summarizing Infographics
While automatic text extraction works well on infographics, computer vision approaches trained on natural images fail to identify the stand-alone visual elements in infographics, or `icons'.
Detecting Synapse Location and Connectivity by Signed Proximity Estimation and Pruning with Deep Nets
The few methods that computes direction along with contact location have only been demonstrated to work on either dyadic (most common in vertebrate brain) or polyadic (found in fruit fly brain) synapses, but not on both types.
Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
In this work, we present a visual analysis tool that allows interaction with a trained sequence-to-sequence model through each stage of the translation process.
Learning to Super-Resolve Blurry Face and Text Images
We present an algorithm to directly restore a clear high-resolution image from a blurry low-resolution input.
Understanding Infographics through Textual and Visual Tag Prediction
And second, we use these predicted text tags as a supervisory signal to localize the most diagnostic visual elements from within the infographic i. e. visual hashtags.
Learning Visual Importance for Graphic Designs and Data Visualizations
Our models are neural networks trained on human clicks and importance annotations on hundreds of designs.
Anisotropic EM Segmentation by 3D Affinity Learning and Agglomeration
The field of connectomics has recently produced neuron wiring diagrams from relatively large brain regions from multiple animals.
Personalizing Gesture Recognition Using Hierarchical Bayesian Neural Networks
Leveraging recent work on learning Bayesian neural networks, we build fast, scalable algorithms for inferring the posterior distribution over all network weights in the hierarchy.
Criteria Sliders: Learning Continuous Database Criteria via Interactive Ranking
Large databases are often organized by hand-labeled metadata, or criteria, which are expensive to collect.
Morphological Error Detection in 3D Segmentations
Deep learning algorithms for connectomics rely upon localized classification, rather than overall morphology.
Guided Proofreading of Automatic Segmentations for Connectomics
Automatic cell image segmentation methods in connectomics produce merge and split errors, which require correction through proofreading.
BubbleView: an interface for crowdsourcing image importance maps and tracking visual attention
In this paper, we present BubbleView, an alternative methodology for eye tracking using discrete mouse clicks to measure which information people consciously choose to examine.
A Multi-Pass Approach to Large-Scale Connectomics
The field of connectomics faces unprecedented "big data" challenges.
RhoanaNet Pipeline: Dense Automatic Neural Annotation
The best segmentation results obtained gave $V^\text{Info}_\text{F-score}$ scores of 0. 9054 and 09182 for the cortex datasets, 0. 9438 for LGN, and 0. 9150 for Cerebellum.
Icon: An Interactive Approach to Train Deep Neural Networks for Segmentation of Neuronal Structures
We present an interactive approach to train a deep neural network pixel classifier for the segmentation of neuronal structures.
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
In this work, we present LSTMVIS, a visual analysis tool for recurrent neural networks with a focus on understanding these hidden state dynamics.
Blind Image Deblurring Using Dark Channel Prior
Therefore, enforcing the sparsity of the dark channel helps blind deblurring on various scenarios, including natural, face, text, and low-illumination images.
Ranked #11 on
Deblurring
on RealBlur-R (trained on GoPro)
Context-guided diffusion for label propagation on graphs
Existing approaches for diffusion on graphs, e. g., for label propagation, are mainly focused on isotropic diffusion, which is induced by the commonly-used graph Laplacian regularizer.
Local High-order Regularization on Data Manifolds
The iterated graph Laplacian enables high-order regularization, but it has a high computational complexity and so cannot be applied to large problems.
Semi-supervised Learning with Explicit Relationship Regularization
In many learning tasks, the structure of the target space of a function holds rich information about the relationships between evaluations of functions on different data points.
Layered RGBD Scene Flow Estimation
As consumer depth sensors become widely available, estimating scene flow from RGBD sequences has received increasing attention.
Local Layering for Joint Motion Estimation and Occlusion Detection
To handle such situations, we propose a local layering model where motion and occlusion relationships are inferred jointly.
VESICLE: Volumetric Evaluation of Synaptic Interfaces using Computer vision at Large Scale
An open challenge problem at the forefront of modern neuroscience is to obtain a comprehensive mapping of the neural pathways that underlie human brain function; an enhanced understanding of the wiring diagram of the brain promises to lead to new breakthroughs in diagnosing and treating neurological disorders.
A Fully-Connected Layered Model of Foreground and Background Flow
Layered models allow scene segmentation and motion estimation to be formulated together and to inform one another.
Reconstructing Loopy Curvilinear Structures Using Integer Programming
We propose a novel approach to automated delineation of linear structures that form complex and potentially loopy networks.
Large-Scale Automatic Reconstruction of Neuronal Processes from Electron Microscopy Images
Segmentation of large-scale electron microscopy data is the main bottleneck in the analysis of these data sets.
