ernments\n> have\n> passed\n> new\n> laws\n> since\n> 2009\n> making\n> the\n> registration\n> or\n> voting\n> process\n> more\n> difficult\n> .\n> \n> \n> \n> \n> \n> \n> \n> It\n> is\n> in\n> this\n> spirit\n> that\n> a\n> majority\n> of\n> American\n> governments\n> have\n> passed\n> new\n> laws\n> since\n> 2009\n> making\n> the\n> registration\n> or\n> voting\n> process\n> more\n> difficult\n> .\n> \n> \n> \n> \n> \n> \n> \n\nFigure 3: An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of the verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for the word ‘making’. Different colors represent different heads. Best viewed in color. 13 Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\n# Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\nFigure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5 and 6. Note that the attentions are very sharp for this word. 14 Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\n# Input-Input Layer5 \n\n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n> The\n> Law\n> will\n> never\n> be\n> perfect\n> ,\n> but\n> its\n> application\n> should\n> be\n> just\n> -\n> this\n> is\n> what\n> we\n> are\n> missing\n> ,\n> in\n> my\n> opinion\n> .\n> \n> \n\nFigure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the sentence. We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6. The heads clearly learned to perform different tasks. 15"}1c:T1c7a,{"title":"Transformer设计灵感揭秘","description":"欢迎来到我们的技术深度解析播客!今天,我们将深入探讨Transformers的设计灵感,以及它如何改变了机器翻译和自然语言处理的格局。我是主持人,将带领大家一起探索这一革命性的模型。","roles":[{"name":"speaker1","title":"主持人","gender":"male"},{"name":"speaker2","title":"共同主持人","gender":"female"}],"topics":["Transformer的设计理念","自注意力机制的引入","模型并行化的优势","序列长度与模型复杂度的关系","多头注意力机制","位置编码的重要性","Transformer在机器翻译中的应用","训练成本与性能的平衡","模型的可解释性","未来研究方向"],"scripts":[{"name":"speaker1","content":"欢迎来到我们的技术深度解析播客!我是主持人,今天非常荣幸邀请到我的

Text

Podcast Information

Roles

Scripts