A Continual Development Methodology for Large-scale Multitask Dynamic ML Systems

15 Sep 2022  ·  Andrea Gesmundo ·

The traditional Machine Learning (ML) methodology requires to fragment the development and experimental process into disconnected iterations whose feedback is used to guide design or tuning choices. This methodology has multiple efficiency and scalability disadvantages, such as leading to spend significant resources into the creation of multiple trial models that do not contribute to the final solution.The presented work is based on the intuition that defining ML models as modular and extensible artefacts allows to introduce a novel ML development methodology enabling the integration of multiple design and evaluation iterations into the continuous enrichment of a single unbounded intelligent system. We define a novel method for the generation of dynamic multitask ML models as a sequence of extensions and generalizations. We first analyze the capabilities of the proposed method by using the standard ML empirical evaluation methodology. Finally, we propose a novel continuous development methodology that allows to dynamically extend a pre-existing multitask large-scale ML system while analyzing the properties of the proposed method extensions. This results in the generation of an ML model capable of jointly solving 124 image classification tasks achieving state of the art quality with improved size and compute cost.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Fine-Grained Image Classification Caltech-101 µ2Net+ (ViT-L/16) Top-1 Error Rate 4.06% # 4
Image Classification CARS196 µ2Net+ (ViT-L/16) Accuracy 87.18 # 1
Image Classification Cats and Dogs µ2Net+ (ViT-L/16) Accuracy 99.83 # 1
Image Classification DTD µ2Net+ (ViT-L/16) Accuracy 82.23 # 2
Image Classification EMNIST-Letters µ2Net+ (ViT-L/16) Accuracy 95.03 # 5
Image Classification EuroSAT µ2Net+ (ViT-L/16) Accuracy (%) 99.22 # 1
Fine-Grained Image Classification Food-101 µ2Net+ (ViT-L/16) Accuracy 91.47 # 8
Domain Generalization ImageNet-A µ2Net+ (ViT-L/16) Top-1 accuracy % 84.53 # 3
Long-tail Learning ImageNet-LT µ2Net+ (ViT-L/16) Top-1 Accuracy 82.5 # 1
Image Classification ImageNet-Sketch µ2Net+ (ViT-L/16) Accuracy 88.6 # 1
Image Classification Imagenette µ2Net+ (ViT-L/16) Accuracy 100 # 1
Image Classification iNaturalist 2018 µ2Net+ (ViT-L/16) Top-1 Accuracy 80.97% # 12
Image Classification Malaria Dataset µ2Net+ (ViT-L/16) Acc. (test) 97.46% # 2
Fine-Grained Image Classification Oxford-IIIT Pets µ2Net+ (ViT-L/16) Accuracy 95.5 # 3
Image Classification Places365 µ2Net+ (ViT-L/16) Top 1 Accuracy 59.15 # 3
Image Classification PlantVillage µ2Net+ (ViT-L/16) Accuracy 99.89 # 2
Fine-Grained Image Classification Stanford Dogs µ2Net+ (ViT-L/16) Accuracy 93.5% # 2
Image Classification Stanford Online Products µ2Net+ (ViT-L/16) Accuracy 89.47 # 1
Image Classification STL-10 µ2Net+ (ViT-L/16) Percentage correct 99.64 # 2
Scene Classification UC Merced Land Use Dataset µ2Net+ (ViT-L/16) Accuracy (%) 100 # 1

Methods


No methods listed for this paper. Add relevant methods here