Search Results for author:
Found 31 papers, 16 papers with code
Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs
In contrast, training on Japanese text could improve question-answering tasks about Japanese knowledge and English-Japanese translation, which indicates that abilities for solving these two tasks can be regarded as \textit{Japanese abilities} for LLMs.
Lion Cub: Minimizing Communication Overhead in Distributed Lion
Communication overhead is a key challenge in distributed deep learning, especially on slower Ethernet interconnects, and given current hardware trends, communication is likely to become a major bottleneck.
Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
In large language model (LLM) training, several parallelization strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), as well as Sequence Parallelism (SP) and Context Parallelism (CP), are employed to distribute model parameters, activations, and optimizer states across devices.
Variational Low-Rank Adaptation Using IVON
We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models.
NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA
The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing.
Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
We demonstrate that, in specific standard settings, PC in the infinite-width limit behaves more similarly to the first-order gradient.
Formula-Supervised Visual-Geometric Pre-training
Throughout the history of computer vision, while research has explored the integration of images (visual) and point clouds (geometric), many advancements in image and 3D object recognition have tended to process these modalities separately.
Rethinking Image Super-Resolution from Training Data Perspectives
In this work, we investigate the understudied effect of the training data used for image super-resolution (SR).
Scaling Backwards: Minimal Synthetic Pre-training?
To this end, we search for a minimal, purely synthetic pre-training dataset that allows us to achieve performance similar to the 1 million images of ImageNet-1k.
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs).
Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.
Building a Large Japanese Web Corpus for Large Language Models
Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR.
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Despite these efforts, such models encounter challenges such as limited multilingual capabilities, risks of catastrophic forgetting during continual pretraining, and the high costs of training models from scratch, alongside the need to align with AI safety standards and regulatory frameworks.
Variational Learning is Effective for Large Deep Networks
We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks.
SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning
SegRCDB has a high potential to contribute to semantic segmentation pre-training and investigation by enabling the creation of large datasets without manual annotation.
Pre-training Vision Transformers with Very Limited Synthesized Images
Prior work on FDSL has shown that pre-training vision transformers on such synthetic datasets can yield competitive accuracy on a wide range of downstream tasks.
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms.
Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves
Unlike JFT-300M which is a static dataset, the quality of synthetic datasets will continue to improve, and the current work is a testament to this possibility.
Informative Sample-Aware Proxy for Deep Metric Learning
Among various supervised deep metric learning methods proxy-based approaches have achieved high retrieval accuracies.
Empirical Study on Optimizer Selection for Out-of-Distribution Generalization
Modern deep learning systems do not generalize well when the test data distribution is slightly different to the training data distribution.
Replacing Labeled Real-image Datasets with Auto-generated Contours
In the present work, we show that the performance of formula-driven supervised learning (FDSL) can match or even exceed that of ImageNet-21k without the use of real images, human-, and self-supervision during the pre-training of Vision Transformers (ViTs).
Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime
Generalization measures are intensively studied in the machine learning community for better modeling generalization gaps.
OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching
Inverse Reinforcement Learning (IRL) is attractive in scenarios where reward engineering can be tedious.
RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering
Furthermore, we utilize differentiable Levenberg-Marquardt (LM) optimization to refine a pose fast and accurately by minimizing the feature-metric error between the input and rendered image representations without the need of zooming in.
Ranked #5 on
6D Pose Estimation using RGB
on LineMOD
Epipolar-Guided Deep Object Matching for Scene Change Detection
To cope with the difficulty, we introduce a deep graph matching network that establishes object correspondence between an image pair.
Scalable and Practical Natural Gradient for Large-Scale Deep Learning
Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size.
Effect of Mixed Precision Computing on H-Matrix Vector Multiplication in BEM Analysis
Hierarchical Matrix (H-matrix) is an approximation technique which splits a target dense matrix into multiple submatrices, and where a selected portion of submatrices are low-rank approximated.
Mathematical Software Distributed, Parallel, and Cluster Computing
Practical Deep Learning with Bayesian Principles
Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning performance is boosted.
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size.
Extreme Scale FMM-Accelerated Boundary Integral Equation Solver for Wave Scattering
With distributed memory optimizations, on the other hand, we report near-optimal efficiency in the weak scalability study with respect to both the logarithmic communication complexity as well as the theoretical scaling complexity of FMM.
Performance Computational Engineering, Finance, and Science Mathematical Software
Asynchronous Execution of the Fast Multipole Method Using Charm++
Fast multipole methods (FMM) on distributed mem- ory have traditionally used a bulk-synchronous model of com- municating the local essential tree (LET) and overlapping it with computation of the local data.
Distributed, Parallel, and Cluster Computing 70F10 D.1.2; D.1.3; G.1.0; G.1.2
