statement

Researchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.

A statement by the post author. Statements are as reported by the cited publishers — not verified by DecisionGateway.

אפשר למחוק כ-75% מראשי ה-attention במודל מאומן והוא ימשיך לעבוד כמעט אותו דבר
mathyaiwithmikePublished 2026-10-21 Read source

Discussion

0/2000