Bingqing Song

Oct 21, 2024 | amazon.science | Boran Han |Shuai Zhang |Jie Ding |Bingqing Song

While the Transformer architecture has achieved remarkable success across various domains, a thorough theoretical foundation explaining its optimization dynamics is yet to be fully developed. In this study, we aim to bridge this understanding gap by answering the following two core questions: (1) Which types of Transformer architectures allow Gradient Descent (GD) to achieve guaranteed convergence?