{"id":473,"date":"2022-07-01T15:10:42","date_gmt":"2022-07-01T15:10:42","guid":{"rendered":"https:\/\/blog.liguanxin.cn\/?p=473"},"modified":"2022-07-01T15:10:42","modified_gmt":"2022-07-01T15:10:42","slug":"%e8%ae%ba%e6%96%87%e7%ac%94%e8%ae%b0-neurips-2021focal-self-attention-for-local-global-interactions-in-vision-transformers","status":"publish","type":"post","link":"https:\/\/blog.liguanxin.cn\/index.php\/2022\/07\/01\/%e8%ae%ba%e6%96%87%e7%ac%94%e8%ae%b0-neurips-2021focal-self-attention-for-local-global-interactions-in-vision-transformers\/","title":{"rendered":"\u8bba\u6587\u7b14\u8bb0\u2014\u2014[NeurIPS 2021]Focal Self-attention for Local-Global Interactions in Vision Transformers"},"content":{"rendered":"<p><strong>\u521b\u65b0\u70b9\uff1a<br \/>\n\u2460\u901a\u8fc7\u8ddd\u79bb\u6765\u7b97\u81ea\u6ce8\u610f\u529b\uff0c\u8ddd\u79bb\u8fd1\u7684patch\u6bd4\u8f83\u7cbe\u7ec6\uff0c\u8ddd\u79bb\u8fdc\u7684\u6bd4\u8f83\u7c97\u7cd9\uff0c\u4ece\u800c\u51cf\u5c11\u5927\u5206\u8fa8\u7387\u56fe\u50cf\u7684\u8ba1\u7b97\u91cf<\/strong><br \/>\n<img src=\"https:\/\/blog.liguanxin.cn\/wp-content\/uploads\/2022\/07\/\u5fae\u4fe1\u622a\u56fe_20220701221616.png\" alt=\"\" \/><\/p>\n<h2>\u603b\u4f53\u7ed3\u6784<\/h2>\n<p><img src=\"https:\/\/blog.liguanxin.cn\/wp-content\/uploads\/2022\/07\/\u5fae\u4fe1\u622a\u56fe_20220701221646.png\" alt=\"\" \/><br \/>\n\u603b\u4f53\u7ed3\u6784\u4e0e\u4f20\u7edf\u7684Vit\u76f8\u5dee\u4e0d\u5927\uff0c\u6bcf\u4e2apatch\u5927\u5c0f\u4e3a4\u00d74\uff0c\u6bcf\u4e2astage\u7ed3\u675f\u8fdb\u884c\u4e00\u6b21Patch Embedding\u3002<\/p>\n<h2>Focal self-attention<\/h2>\n<p><img src=\"https:\/\/blog.liguanxin.cn\/wp-content\/uploads\/2022\/07\/\u5fae\u4fe1\u622a\u56fe_20220701223909.png\" alt=\"\" \/><br \/>\n\u5206\u4e09\u4e2a\u5c42\u6b21\u6c42Q, K, V\uff0c\u5047\u8bbe\u8981\u6c42\u4e2d\u95f44*4\u533a\u57df\u7684\u81ea\u6ce8\u610f\u529b\uff1a<\/p>\n<ul>\n<li>\u7b2c\u4e00\u5c42\u627e\u5468\u56f4\u8ddd\u79bb\u4e3a8\u7684\u533a\u57df\uff0c\u7ec4\u62108*8\u7684\u77e9\u9635<\/li>\n<li>\u7b2c\u4e8c\u5c42\u627e\u5468\u56f4\u8ddd\u79bb\u4e3a4\u7684\u533a\u57df\uff0c\u7ec4\u621012<em>12\u7684\u77e9\u9635\u6c60\u5316\u4e3a6<\/em>6<\/li>\n<li>\u7b2c\u4e09\u5c42\u627e\u5468\u56f4\u8ddd\u79bb\u4e3a8\u7684\u533a\u57df\uff0c\u7ec4\u621020<em>20\u7684\u77e9\u9635\u6c60\u5316\u4e3a5<\/em>5<\/li>\n<\/ul>\n<p>\u628a\u4e09\u4e2a\u5c42\u6b21\u7684\u7ed3\u679cflatten\u5e76cancat\u6210\u957f\u5ea6\u4e3a125\u7684tokens\uff0c\u901a\u8fc7\u4e24\u4e2a\u7ebf\u7a0b\u6295\u5f71\u5f97\u5230V\u548cK\uff0c\u8ba9\u539f\u59cb\u76844*4\u76f4\u63a5\u6295\u5f71\u5f97\u5230Q\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u521b\u65b0\u70b9\uff1a \u2460\u901a\u8fc7\u8ddd\u79bb\u6765\u7b97\u81ea\u6ce8\u610f\u529b\uff0c\u8ddd\u79bb\u8fd1\u7684patch\u6bd4\u8f83\u7cbe\u7ec6\uff0c\u8ddd\u79bb\u8fdc\u7684\u6bd4\u8f83\u7c97\u7cd9\uff0c\u4ece\u800c\u51cf\u5c11\u5927\u5206\u8fa8\u7387\u56fe\u50cf\u7684\u8ba1\u7b97\u91cf  [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[6],"tags":[13,17,11],"_links":{"self":[{"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/posts\/473"}],"collection":[{"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/comments?post=473"}],"version-history":[{"count":0,"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/posts\/473\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/media?parent=473"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/categories?post=473"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.liguanxin.cn\/index.php\/wp-json\/wp\/v2\/tags?post=473"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}