Transfer learning is an effective technique for tuning a deep learning model when training data or computational resources are limited. It adjusts the parameters of an existing “base model” for a new task instead of training a new model from scratch. The accuracy of such a fine-tuned model depends on the choice of the right base model. Model search automates the selection of the base model for transfer learning by evaluating candidate models for a specific task. With model stores holding thousands to millions of models, this process is computationally expensive, as it requires running inference on each model with representative data, making it a bottleneck in transfer learning.
In this work, we present Alsatian, a novel model search system. Based on the observation that many candidate models overlap to a significant extent and based on a careful bottleneck analysis, we propose optimization techniques that are applicable to many model search methods and other tasks that require inference with a large number of similar models. The optimizations include: (i) splitting models into individual blocks that can be shared across models, (ii) caching of intermediate inference results and model blocks, and (iii) selecting a beneficial search order for models to increase reuse. In our evaluation on state-of-the-art deep learning models from computer vision and natural language processing, we show that Alsatian outperforms related systems by up to ~14×.