A Continual Development Methodology for Large-scale Multitask Dynamic ML Systems
The traditional Machine Learning (ML) methodology requires to fragment the development and experimental process into disconnected iterations whose feedback is used to guide design or tuning choices. This methodology has multiple efficiency and scalability disadvantages, such as leading to spend significant resources into the creation of multiple trial models that do not contribute to the final solution.The presented work is based on the intuition that defining ML models as modular and extensible artefacts allows to introduce a novel ML development methodology enabling the integration of multiple design and evaluation iterations into the continuous enrichment of a single unbounded intelligent system. We define a novel method for the generation of dynamic multitask ML models as a sequence of extensions and generalizations. We first analyze the capabilities of the proposed method by using the standard ML empirical evaluation methodology. Finally, we propose a novel continuous development methodology that allows to dynamically extend a pre-existing multitask large-scale ML system while analyzing the properties of the proposed method extensions. This results in the generation of an ML model capable of jointly solving 124 image classification tasks achieving state of the art quality with improved size and compute cost.
PDF AbstractCode
Datasets
CIFAR-10
CIFAR-100
MNIST
SVHN
Fashion-MNIST
STL-10
DTD
Food-101
Caltech-101
iNaturalist
EuroSAT
ImageNet-A
EMNIST
Stanford Online Products
ImageNet-Sketch
ImageNet-LT
Oxford-IIIT Pet Dataset
PlantVillage
Places365
Stanford Dogs
Oxford-IIIT Pets
CARS196
Imagenette
UC Merced Land Use Dataset
Cats and Dogs
Malaria Dataset
