What is Distillation?
It is the process of transferring the information of a large and complex artificial intelligence model to a smaller model.
Overview
Distillation is the process of transferring the information of a very large and complex artificial intelligence model to a smaller and more efficient model. This method allows us to get results close to the performance of giant models by spending less resources.
How it works
The larger model acts as a teacher and checks the predictions of the smaller model. The little model becomes an expert on its own over time, learning to produce results that resemble the teacher's answers.
Where it is used
It is preferred to get fast and effective results in places where huge models cannot fit, such as smartphones or low-end devices.
Commonly confused with
It is confused with quantization; But while quantization compresses the model's weights, distillation involves training an entirely new, smaller model.
Frequently asked questions
Will the smaller model be as successful as the larger model?
There is usually some loss of performance, but the speed and cost advantage offset this loss.
Can every model be distilled?
Yes, but fitting the knowledge of complex models into a small structure requires a very difficult training process.
Related terms
This explanation was written in plain language for TreScout and machine-translated from the Turkish original · the Turkish version prevails. If something looks wrong or missing, write to hello@trescout.com. Read in Turkish →