AI companies race to use ‘distillation’ to produce cheaper models

Ars Technica

Center

Factuality

Ownership

AI firms follow DeepSeek’s lead, create cheaper models with “distillation”

Leading artificial intelligence firms including OpenAI, Microsoft, and Meta are turning to a process called “distillation” in the global race to create AI models that are cheaper for consumers and businesses to adopt. The technique caught widespread attention after China’s DeepSeek used it to build powerful and efficient AI models based on open source systems released by competitors Meta and Alibaba. The breakthrough rocked confidence in Silicon…

Financial Times

Australian Financial Review

Lean Right

View article

Reposted by

Australian Financial Review

Center

Factuality

Ownership

AI companies race to use ‘distillation’ to produce cheaper models

DeepSeek used technique to create smaller powerful models based on the technology of competitors such as Meta

Globo

Factuality

Ownership

Following DeepSeek's example, companies such as OpenAI, Microsoft and Meta are using the “distillation” process, which generates more accessible AI models, according to the Financial Times report.

Towards AI — Multidisciplinary Science Journal

Factuality

Ownership

Practical Guide to Distilling Large Models into Small Models: A Novel Approach with Extended Distillation

Last Updated on March 3, 2025 by Editorial Team Author(s): Shenggang Li Originally published on Towards AI. Comparing Traditional and Enhanced Step-by-Step Distillation: Adaptive Learning, Cosine Similarity, and Curriculum-Based Rationale Supervision This member-only story is on us. Upgrade to access all of Medium. Photo by Thorium on Unsplash In this paper, I will uncover the secrets behind transferring “big model” intelligence to smaller, more…

Edge AI and Vision Alliance

Factuality

Ownership

Fine-tuning LLMs for Cost-effective GenAI Inference at Scale

This article was originally published at Tryolabs’ website. It is reprinted here with the permission of Tryolabs. Data is the new oil, fueling the AI revolution. From user-tailored shopping assistants to AI researchers, to recreating the King, the applicability of AI models knows no bounds. Yet these models are only as good as the data we feed them. State-of-the-art generative AI models offer the potential to automate information extraction proc…

LJ infoDOCKET

Factuality

Ownership

Small Language Models: “The Big Shrink in LLMs”

From Communications of the ACM: …smaller models and datasets have emerged as a solution to some of their larger cousins’ drawbacks. Techniques such as knowledge distillation, transferring knowledge from large to smaller models, and pruning, removing model parameters (such as weight or temperature) without degrading accuracy, are also supporting the shrink. This and developments in edge computing enabled by smaller models that can run ‘on device’…