Efficient Language Modeling using Knowledge Distillation
Loading...
Citations
Altmetric:
Altmetric::
Download
Associated Records
Authors
File Licence
Editors
Date
2026
Source Title
Journal Title
Journal ISSN
Volume Title
Publisher
Melbourne Authors
Affiliation
Computing and Information Systems
Document Type
PhD thesis
Subject Code
Citation
Access Status
Open Access
DOI
Access URL
Open Access URL
Open Access at PMC
Linked Resource URL
NHMRC Grant Code
ARC Grant Code
Description
© 2026 Sayantan Dasgupta
Abstract
Large language models achieve strong performance but are increasingly costly to train, fine-tune, and deploy. For efficient inference, especially on edge devices in real time, smaller language models (SLMs) are essential. While SLMs may lack sufficient standalone performance, they can be enhanced through knowledge transfer from larger models. Knowledge distillation (KD) provides a principled method for this by transferring predictive distributions and internal representations from a large teacher model to a smaller student. However, standard KD pipelines are often inefficient: they require task-specific teacher fine-tuning, constrain the student architecture to match the teacher's hidden dimensionality, and use divergences ill-suited to the highly peaked, long-tailed token distributions typical of causal language modeling. This thesis develops computationally efficient, architecture-flexible distillation methods for transformer language models, emphasizing the preservation of the teacher's predictive distribution and the evaluation of end-to-end efficiency.
The thesis makes three contributions. First, we propose a task-agnostic distillation objective derived from a Taylor-series expansion of the KD loss around the Bayes-optimal solution. This mitigates the impact of using a non-fine-tuned teacher and eliminates the need to fine-tune it for each dataset. On GLUE benchmarks, our method matches or exceeds task-specific approaches on many low-resource datasets while requiring only a single GPU. Second, we introduce a representation-level distillation method based on Centered Kernel Alignment (CKA) that enables cross-width transfer of hidden states. Unlike projection-based approaches that require matching dimensionality, CKA permits students that are both shallower and narrower than the teacher, achieving higher compression ratios ($> 3 \times$) without significant degradation relative to dimension-matched baselines on generative tasks such as summarization and translation. Third, for decoder-only pretraining, we develop Tail-Aware Distillation (TAD), an objective that decouples the teacher's output distribution into a head and a tail component. By explicitly supervising low-probability tokens that standard KL-based distillation underweights, TAD better preserves the teacher's soft probability distribution. Empirically, TAD achieves up to 2.2 points higher average accuracy than vanilla KD across eight reasoning benchmarks and outperforms prior baselines such as MiniPLM by $50$–$120\%$ in relative improvement while maintaining the same computational budget. Collectively, this thesis demonstrates that knowledge distillation can serve as a sustainable and practical pipeline for training and deploying efficient language models—provided objectives are designed to preserve what matters: uncertainty structure, internal representations, and practical efficiency.
Notes
Terms of Use
Keywords
Language Models, Knowledge Distillation, Encoder, BERT, Decoder, Llama, Phi2, BART, T5, Mathematical Reasoning, GSM8K