vernments\n> have\n> passed\n> new\n> laws\n> since\n> 2009\n> making\n> the\n> registration\n> or\n> voting\n> process\n> more\n> difficult\n> .\n> \n> \n> \n> \n> \n> \n> \n> It\n> is\n> in\n> this\n> spirit\n> that\n> a\n> majority\n> of\n> American\n> governments\n> have\n> passed\n> new\n> laws\n> since\n> 2009\n> making\n> the\n> registration\n> or\n> voting\n> process\n> more\n> difficult\n> .\n> \n> \n> \n> \n> \n> \n> \n\nFigure 3: An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of the verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for the word ‘making’. Different colors represent different heads. Best viewed in color. 13 Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\n# Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\nFigure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5 and 6. Note that the attentions are very sharp for this word. 14 Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\n# Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\nFigure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the sentence. We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6. The heads clearly learned to perform different tasks. 15"}1c:T1146,{"title":"探索变换器模型的未来","description":"本期播客我们将深入讨论变换器模型的架构、应用以及未来的发展方向。","summary":"我们将探讨变换器模型如何改变序列转导任务的格局，并展望它在机器翻译和其他语言处理任务中的潜力。","key_points":["变换器模型的基本架构及其优势","机器翻译中的应用效果及其性能","未来可能的研究方向和应用场景"],"roles":[{"role":"host","name":"Leo","title":"播客主持人","gender":"male"},{"role":"guest","name":"Niki","title":"机器学习专家","gender":"female"}],"topics":["变换器模型","机器翻译","自注意力机制"],"scripts":[{"role":"host","name":"Leo","gender":"male","topic":"变换器模型","content":"大家好，欢迎收听本期播客！今天我们将深入探讨一个非常前沿的主题—

Text

Podcast Information

Roles

Scripts