Search Results for author:
Found 70 papers, 46 papers with code
BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency
As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to project various sensory modalities into a shared feature space.
MSINet: Twins Contrastive Search of Multi-Scale Interaction for Object ReID
Neural Architecture Search (NAS) has been increasingly appealing to the society of object Re-Identification (ReID), for that task-specific architectures significantly improve the retrieval performance.
Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models
To address this challenge, we propose a novel method ZSCL to prevent zero-shot transfer degradation in the continual learning of vision-language models in both feature and parameter space.
DiM: Distilling Dataset into Generative Model
To the best of our knowledge, we are the first to achieve higher accuracy on complex architectures than simple ones, such as 75. 1\% with ResNet-18 and 72. 6\% with ConvNet-3 on ten images per class of CIFAR-10.
InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning
We train the full data in the last few epochs to improve the performance of our method, which further reduces the bias of the total update.
CRIN: Rotation-Invariant Point Cloud Analysis and Rotation Estimation via Centrifugal Reference Frame
CRIN directly takes the coordinates of points as input and transforms local points into rotation-invariant representations via centrifugal reference frames.
DREAM: Efficient Dataset Distillation by Representative Matching
Although there are various matching objectives, currently the strategy for selecting original images is limited to naive random sampling.
Robust Robot Planning for Human-Robot Collaboration
Based on this, we propose two contributions: 1) an approach to automatically generate an uncertain human behavior (a policy) for each given objective function while accounting for possible robot behaviors; and 2) a robot planning algorithm that is robust to the above-mentioned uncertainties and relies on solving a partially observable Markov decision process (POMDP) obtained by reasoning on a distribution over human behaviors.
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
To address these challenges, we introduce a system that can jointly optimize distributed execution and gradient checkpointing plans.
Adaptive Computation with Elastic Input Sequence
However, most standard neural networks have the same function type and fixed computation budget on different samples regardless of their nature and difficulty.
MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs
The vision of this paper is to provide a more comprehensive and practical benchmark study for MIG in order to eliminate the need for tedious manual benchmarking and tuning efforts.
On the Inconsistencies of Conditionals Learned by Masked Language Models
We empirically quantify such inconsistencies in the simple scenario of bigrams for two common styles of masked language models: T5-style and BERT-style.
GPTR: Gestalt-Perception Transformer for Diagram Object Detection
These lead to the fact that traditional data-driven detection model is not suitable for diagrams.
Elixir: Train a Large Language Model on a Small GPU Cluster
People who are inaccessible to a large number of GPUs resort to heterogeneous training systems for storing model parameters in CPU memory.
One-Shot General Object Localization
In contrast, our proposed OneLoc algorithm efficiently finds the object center and bounding box size by a special voting scheme.
Go Beyond Point Pairs: A General and Accurate Sim2Real Object Pose Voting Method with Efficient Online Synthetic Training
In this paper, we propose a novel method for sim-to-real pose estimation, which is effective on both instance-level and category-level settings.
SentBS: Sentence-level Beam Search for Controllable Summarization
A wide range of control perspectives have been explored in controllable text generation.
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Although the AI community has expanded the model scale to the trillion parameter level, the practical deployment of 10-100 billion parameter models is still uncertain due to the latency, throughput, and memory constraints.
Prompt Vision Transformer for Domain Generalization
In this paper, we propose a novel approach DoPrompt based on prompt learning to embed the knowledge of source domains in domain prompts for target domain prediction.
A Frequency-aware Software Cache for Large Recommendation System Embeddings
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies.
Active-Learning-as-a-Service: An Automatic and Efficient MLOps System for Data-Centric AI
In data-centric AI, active learning (AL) plays a vital role, but current AL tools 1) require users to manually select AL strategies, and 2) can not perform AL tasks efficiently.
Divide to Adapt: Mitigating Confirmation Bias for Domain Adaptation of Black-Box Predictors
Domain Adaptation of Black-box Predictors (DABP) aims to learn a model on an unlabeled target domain supervised by a black-box predictor trained on a source domain.
FaceMAE: Privacy-Preserving Face Recognition via Masked Autoencoders
Firstly, randomly masked face images are used to train the reconstruction module in FaceMAE.
Deeper vs Wider: A Revisit of Transformer Configuration
In this paper, we revisit these conventional configurations.
Ranked #88 on
Image Classification
on ImageNet
Reliable Label Correction is a Good Booster When Learning with Extremely Noisy Labels
This paradigm, however, is prone to significant degeneration under heavy label noise, as the number of clean samples is too small for conventional methods to behave well.
CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU
Our experiments show that previous scaling rules fail in the training of CTR prediction neural networks.
Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation
Text-based video segmentation aims to segment the target object in a video based on a describing sentence.
Ranked #7 on
Referring Expression Segmentation
on A2D Sentences
Optical Flow Estimation
Referring Expression Segmentation
+2
CPPF: Towards Robust Category-Level 9D Pose Estimation in the Wild
Drawing inspirations from traditional point pair features (PPFs), in this paper, we design a novel Category-level PPF (CPPF) voting method to achieve accurate, robust and generalizable 9D pose estimation in the wild.
Ranked #1 on
6D Pose Estimation using RGBD
on REAL275
(mAP 15, 5cm metric)
Towards Efficient and Scalable Sharpness-Aware Minimization
Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers.
CAFE: Learning to Condense Dataset by Aligning Features
Dataset condensation aims at reducing the network training effort through condensing a cumbersome training set into a compact synthetic one.
FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours
In this work, we present FastFold, an efficient implementation of AlphaFold for both training and inference.
Sky Computing: Accelerating Geo-distributed Computing in Federated Learning
In this paper, we proposed Sky Computing, a load-balanced model parallelism framework to adaptively allocate the weights to devices.
Crafting Better Contrastive Views for Siamese Representation Learning
For high performance Siamese representation learning, one of the keys is to design good contrastive pairs.
One Student Knows All Experts Know: From Sparse to Dense
Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts.
Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
Pixel-level 2D object semantic understanding is an important topic in computer vision and could help machine deeply understand objects (e. g. functionality and affordance) in our daily life.
Large-Scale Deep Learning Optimizations: A Comprehensive Survey
Deep learning have achieved promising results on a wide spectrum of AI applications.
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
The success of Transformer models has pushed the deep learning model scale to billions of parameters.
MReD: A Meta-Review Dataset for Structure-Controllable Text Generation
A more useful text generator should leverage both the input text and the control signal to guide the generation, which can only be built with a deep understanding of the domain knowledge.
Sharpness-Aware Minimization in Large-Batch Training: Training Vision Transformer In Minutes
Large-batch training is an important direction for distributed machine learning, which can improve the utilization of large-scale clusters and therefore accelerate the training process.
Solving infinite-horizon Dec-POMDPs using Finite State Controllers within JESP
This paper looks at solving collaborative planning problems formalized as Decentralized POMDPs (Dec-POMDPs) by searching for Nash equilibria, i. e., situations where each agent's policy is a best response to the other agents' (fixed) policies.
Cross-token Modeling with Conditional Computation
Mixture-of-Experts (MoE), a conditional computation architecture, achieved promising performance by scaling local module (i. e. feed-forward network) of transformer.
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
PatrickStar uses the CPU-GPU heterogeneous memory space to store the model data.
Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters
ONES automatically manages the elasticity of each job based on the training batch size, so as to maximize GPU utilization and improve scheduling efficiency.
Go Wider Instead of Deeper
To achieve better performance with fewer trainable parameters, recent methods are proposed to go shallower by parameter sharing or model compressing along with the depth.
Ranked #577 on
Image Classification
on ImageNet
Concurrent Adversarial Learning for Large-Batch Training
Current methods usually use extensive data augmentation to increase the batch size, but we found the performance gain with data augmentation decreases as batch size increases, and data augmentation will become insufficient after certain point.
Maximizing Parallelism in Distributed Training for Huge Neural Networks
Our work is the first to introduce a 3-dimensional model parallelism for expediting huge language models.
Tesseract: Parallelize the Tensor Parallelism Efficiently
It increases efficiency by reducing communication overhead and lowers the memory required for each GPU.
Sequence Parallelism: Long Sequence Training from System Perspective
That is, with sparse attention, our sequence parallelism enables us to train transformer with infinite long sequence.
An Efficient Training Approach for Very Large Scale Face Recognition
This method adopts Dynamic Class Pool (DCP) for storing and updating the identities features dynamically, which could be regarded as a substitute for the FC layer.
Ranked #1 on
Face Verification
on IJB-C
(training dataset metric)
An Efficient 2D Method for Training Super-Large Deep Learning Models
However, due to memory constraints, model parallelism must be utilized to host large models that would otherwise not fit into the memory of a single device.
Skeleton Merger: an Unsupervised Aligned Keypoint Detector
In this paper, we propose an unsupervised aligned keypoint detector, Skeleton Merger, which utilizes skeletons to reconstruct objects.
PRIN/SPRIN: On Extracting Point-wise Rotation Invariant Features
Spherical Voxel Convolution and Point Re-sampling are proposed to extract rotation invariant features for each point.
Canonical Voting: Towards Robust Oriented Bounding Box Detection in 3D Scenes
In the work, we disentangle the direct offset into Local Canonical Coordinates (LCC), box scales and box orientations.
UKPGAN: A General Self-Supervised Keypoint Detector
Keypoint detection is an essential component for the object registration and alignment.
Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour
EfficientNets are a family of state-of-the-art image classification models based on efficiently scaled convolutional neural networks.
How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy.
LOCx2, a Low-latency, Low-overhead, 2 x 5.12-Gbps Transmitter ASIC for the ATLAS Liquid Argon Calorimeter Trigger Upgrade
In this paper, we present the design and test results of LOCx2, a transmitter ASIC for the ATLAS Liquid Argon Calorimeter trigger upgrade.
Instrumentation and Detectors
The Limit of the Batch Size
For the first time we scale the batch size on ImageNet to at least a magnitude larger than all previous work, and provide detailed studies on the performance of many state-of-the-art optimization schemes under this setting.
Semantic Correspondence via 2D-3D-2D Cycle
Visual semantic correspondence is an important topic in computer vision and could help machine understand objects in our daily life.
KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotations
Detecting 3D objects keypoints is of great interest to the areas of both graphics and computer vision.
Human Correspondence Consensus for 3D Object Semantic Understanding
Semantic understanding of 3D objects is crucial in many applications such as object manipulation.
Auto-Precision Scaling for Distributed Deep Learning
Our experimental results show that for many applications, APS can train state-of-the-art models by 8-bit gradients with no or only a tiny accuracy loss (<0. 05%).
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
Large-Batch Training for LSTM and Beyond
LEGW enables Sqrt Scaling scheme to be useful in practice and as a result we achieve much better results than the Linear Scaling learning rate scheme.
Combinational Q-Learning for Dou Di Zhu
Deep reinforcement learning (DRL) has gained a lot of attention in recent years, and has been proven to be able to play Atari games and Go at or above human levels.
Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution
Point cloud analysis without pose priors is very challenging in real applications, as the orientations of point clouds are often unknown.
Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling
Using LARS, we scaled AlexNet and ResNet-50 to a batch size of 16K.
ImageNet Training in Minutes
If we can make full use of the supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in one minute.
Large Batch Training of Convolutional Networks
Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.
Asynchronous Parallel Greedy Coordinate Descent
n this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints.
