On the Role of the Activation Function in Transformer Self-Attention: An Empirical Investigation
中文摘要
这项研究探讨了 Transformer 自注意力机制中激活函数的作用，分析了其多项式表达能力与动态权重生成的结构视角。
English Summary
This research investigates the role of activation functions in Transformer self-attention, focusing on polynomial expressiveness and dynamic weight generation.
Original Excerpt
A Structural View of Polynomial Expressiveness and Dynamic Weight Generation Continue reading on Medium »