statementSaid on 2026-10-19
The exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
האקספוננט עושה בדיוק את זה, ואחרי נרמול קיבלתם סופטמקס, כלומר attention