t\n> that\n> a\n> majority\n> of\n> American\n> governments\n> have\n> passed\n> new\n> laws\n> since\n> 2009\n> making\n> the\n> registration\n> or\n> voting\n> process\n> more\n> difficult\n> .\n> \n> \n> \n> \n> \n> \n> \n> It\n> is\n> in\n> this\n> spirit\n> that\n> a\n> majority\n> of\n> American\n> governments\n> have\n> passed\n> new\n> laws\n> since\n> 2009\n> making\n> the\n> registration\n> or\n> voting\n> process\n> more\n> difficult\n> .\n> \n> \n> \n> \n> \n> \n> \n\nFigure 3: An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of the verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for the word ‘making’. Different colors represent different heads. Best viewed in color. 13 Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\n# Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\nFigure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5 and 6. Note that the attentions are very sharp for this word. 14 Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\n# Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\nFigure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the sentence. We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6. The heads clearly learned to perform different tasks. 15"}1c:T1d6f,{"title":"Transformer: 重塑序列模型的未来","description":"欢迎来到我们的最新一集，今天我们将深入探讨Transformer模型，这个革命性的AI架构是如何彻底改变机器翻译和其他序列任务的。准备好迎接一个充满创新和技术的旅程吧！","roles":[{"name":"speaker1","title":"主持人","gender":"male"},{"name":"speaker2","title":"联合主持人","gender":"female"}],"topics":["Transformer模型的诞生","自注意力机制的原理","与RNN和CNN的比较","Transformer在机器翻译中的应用","多头注意力机制的优势","Transformer的训练细节","Transformer的性能和成本","Transformer在其他任务中的应用","未来的挑战和展望","实际案例和可视化"],"scripts":[{"name":"speaker1","content":"大家好，欢迎来到今天的

Text

Podcast Information

Roles

Scripts