statement
Researchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.
A statement by the post author. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אפשר למחוק כ-75% מראשי ה-attention במודל מאומן והוא ימשיך לעבוד כמעט אותו דבר