Tag: Knowledge Distillation
-
ViT vs DeiT Review: Pure Attention Meets Distillation
ViT vs DeiT: how distillation and data augmentation let DeiT match ViT accuracy without 300M pretraining images. Training logs and ablation results included.
-
Knowledge Distillation ์ค์ : LLM ์์ถ๊ณผ TinyLlama ์ฌ๋ก
Compress BERT to 40% size with 97% accuracy retained. Knowledge Distillation guide: DistilBERT and TinyLlama cases with PyTorch implementation.