Attention heads dump (n_head=32, d_head=128)

OV and QK matrices are computed as described in the framework paper. However, becasue the matrices are so large and we can't just look at the entire thing, a few heuristic decisions had to be made in how we present and interpret them. The following results are totally automated and not hand-picked by humans, but they are heuristically selected to be interesting or important entries in the matrices. The heuristic decisions made are as follows:

Should we "pivot" the table on the key or the query? We chose the pivot on the key (source) token. (Looking at the table will probably be helpful for understanding what we mean by this: note that there's only one key entry per row.) By this we will inspect the table by fixing various keys and looking at the corresponding queries and outputs. This is nice because the key or source is the value shared between the OV and QK matrices. However, it has the downside that QK values are not immediately comparable for different queries.

How do we normalize QK/OV values? Since QK values for a given query are fed into a softmax, a constant amount can be added to them without effecting the result. At least in theory, this means that QK values aren't comparable between different query values without some kind of normalization. But we do want to compare them (since we chose to pivot on the key). For this model, we've subtracted off the mean QK value foreach query, since models seem to use it as a kind of default value. (This is different than for the other model, since one head gives very high QK values, and would make the heuristic we use to sort keys negative.)

How do we normalize OV values? This is less important, since OV values for a given source token are comparable. But it is the case that, since the OV values are fed into a softmax, a contstant amount can be added to them without effecting the result. We chose to subtract off the mean to make values more consistent across different source tokens.

How do we chose which keys to show entries for? Since the matrices are massive, we need a heuristic for selecting "interesting" or "important" keys to look at entries for. For this page, the heuristic is QK.max(0) * OV.max(0) * token_prob**0.1. That is, we favor keys with queries that strongly prefer them, that have a large afffect on the output, and slightly prefer probable keys since they will occcur more frequently and have more opportunities to effect model behavior. The specific adjustment for probability is pretty arbitrary. Varying it (or even dropping it) will reveal different interesting entries.

How do we pick which queries and output tokens to show? Outputs are simply the tokens with the largest values. Queries are the tokens with the largest QK[:, src]*token_prob**0.1 values. (We upweight probable tokens heuristically because they're more likely to occur in the future context, and thus more likely to end up attending back to this token.)

Head 0:0

'—'

' _'

' ...'

'Advertisement'

'!'

' –'

'...'

' —'

'####'

'###'

'....'

'##'

'--'

':_'

' -'

"!'"

".'"

'..'

'".'

'\n\n   '

'’'

"'"

'['

'”,'

',"'

' [@'

'"\''

'[]'

'<META_START>'

'...,'

'”'

'.['

' "'

' {#'

'Read'

' :'

'\n   '

' ['

'\n\n '

' ).'

'**'

'============'

'View'

'"""'

' ...)'

'\n\n\n\n\n'

'#'

'\n\n'

",'"

'_**'

' '

' ‘'

'Labels'

'!"'

'http'

'Contents'

' i'

' (\\['

',_'

'More'

'self'

'*'

"';"

',”'

'\n           ...'

'...)'

'\n '

"'."

':'

' self'

"'s"

'\n\n       '

' *'

'\n       '

'\n\n\n'

'";'

'\r\n   '

'.:'

'.-'

' “'

'!!!'

' #'

',['

'[['

' )'

" '"

'\n           '

'"'

' \\['

'\n     '

'”.'

'..."'

'######'

'.)'

'*]{}'

'–'

'www'

' **'

' ..'

"...'"

':-'

'!!'

'\t'

'on'

'!)'

' !'

' http'

' \n'

'\n           ...'

'edit'

'#####'

'\\['

'***'

's'

'**_'

'---'

"'''"

'\n           ...'

' ):'

'\n \n'

' ]'

'\r\n       '

':['

' ([@'

',('

'];'

'=='

'.".'

';'

'\n           ...'

' https'

'._'

' );'

'('

'I'

'_.'

'——'

');'

' [['

'."'

' gonna'

'."""'

'\r\n\r\n       '

"');"

'*.'

'«'

'):'

'comments'

"\\'"

' ='

'.</'

' NOT'

'\r\n\r\n   '

'\n\n\t'

' pp'

'Comments'

'\n\n          ...'

'\n           ...'

' --'

" '."

'.<'

'https'

' .'

' AM'

'\n           ...'

' PM'

'();'

'?)'

'.]'

'was'

' \n   '

"'d"

'?'

'skip'

' p'

'?",'

'.**'

' __'

'Skip'

'01'

' J'

'2018'

'\n\n\n\n'

'_"'

' :)'

'\n           ...'

' <'

'Copyright'

'‘'

'Menu'

'have'

'About'

' dont'

" ''"

'References'

'\n\n        '

'\n           ...'

'\n         '

'possible'

'\n           ...'

Head 0:1

' Dr'

' M'

' G'

' C'

' U'

' L'

' D'

' N'

' E'

' J'

' B'

'S'

'B'

'R'

' R'

'G'

' P'

' Re'

'M'

' Ph'

'Dr'

'L'

' K'

'J'

' T'

'Ph'

'P'

'Doc'

' S'

' Mr'

'Col'

' F'

' V'

'N'

' A'

'V'

' New'

'Ann'

'Bar'

'C'

' O'

' Art'

'D'

'T'

'E'

'U'

' H'

'K'

'Mark'

'Lib'

' Char'

' Math'

' W'

' Ch'

'St'

'Sun'

'Ben'

' Ann'

' Z'

' St'

'Mc'

'Base'

'Mac'

' Morgan'

'New'

'ov'

'White'

'Char'

' agents'

' Y'

' An'

' Lib'

'Pat'

'H'

' Bill'

'Black'

' Mrs'

'Ch'

' Adam'

'Author'

'A'

'Car'

'Tr'

' John'

'Ar'

'Re'

' Al'

' Doc'

' ship'

'Pr'

'An'

' Paris'

'Net'

' Ben'

' Canada'

' Q'

'Win'

' Agent'

' boys'

'Ab'

'Mon'

' Japanese'

' Mac'

' Py'

'F'

' Sy'

' Canadian'

' Bar'

'ana'

'Ad'

' men'

'Mr'

'Al'

'iana'

' Alex'

'Mar'

' Daniel'

'Sy'

'Art'

' Ast'

' McC'

'Log'

' Michael'

'Mag'

' Dan'

'State'

'Red'

' Em'

'Ind'

'Rob'

' states'

'Q'

'ABC'

' Ed'

'ae'

' Mag'

'San'

'Y'

' Paul'

'Cl'

'John'

' Gen'

' State'

'Mal'

' Indian'

' London'

' Sh'

' Pol'

' horses'

' Sun'

' African'

' Jes'

' Jo'

' Miss'

'Max'

' Edward'

'https'

'Rad'

'Phil'

' Charles'

'#####'

' Tom'

' race'

' Tim'

' Bi'

' X'

' Col'

' Professor'

' Space'

' Trans'

'Ed'

' American'

' De'

' Navy'

' Ms'

' Russian'

'W'

' David'

'Co'

' Japan'

' Dam'

' Cal'

'Tom'

' Row'

'David'

'City'

'Z'

'inton'

' Stone'

' Min'

' Harry'

'Fri'

' War'

'Bro'

'Buff'

'By'

' Carol'

'Cam'

' Men'

' prison'

Head 0:2

' says'

'We'

' we'

' shit'

' guy'

'This'

' crowd'

'Our'

' everybody'

'Suppose'

' We'

' according'

' actually'

' woman'

' test'

' shown'

' However'

' characters'

' Suppose'

'Your'

' writes'

' narr'

' currently'

' which'

'The'

' our'

' Our'

' ought'

' See'

'My'

' guys'

' likely'

'However'

' amazing'

'She'

'And'

' she'

' stated'

' centre'

' you'

' indeed'

' weird'

' quite'

'They'

' somebody'

'He'

' said'

' perhaps'

' described'

' And'

' basically'

' etc'

' us'

' typically'

' group'

' thus'

' experiment'

' often'

' causing'

' Indeed'

' scenario'

' begins'

' behavior'

' This'

' hence'

' He'

'But'

' suitable'

' upon'

' But'

' apparently'

' however'

' show'

' its'

' Though'

' because'

' useful'

' Yet'

' She'

'On'

' present'

' noise'

' case'

' old'

' According'

' favour'

' My'

' following'

' extremely'

' he'

' evidence'

'Check'

' my'

' absolutely'

' very'

'check'

' probably'

' story'

'During'

' source'

'After'

' replied'

'If'

' interesting'

'See'

' this'

' asks'

' situation'

'Her'

' search'

'Each'

' objects'

' wrote'

' process'

' seems'

' enormous'

' somewhat'

'Test'

' Therefore'

' totally'

' usually'

' unlikely'

' towards'

' explains'

' though'

' called'

' essentially'

' fuck'

'According'

'For'

' planet'

' literally'

' Thus'

' noted'

' therefore'

' preferred'

' Use'

' stats'

' colour'

' nice'

'Note'

' excellent'

' but'

' really'

' suppose'

' approximately'

' suggests'

'information'

' relevant'

' audience'

' needed'

' reported'

' behaviour'

'When'

' Read'

' themes'

' results'

' I'

'from'

' completely'

'Since'

'Hi'

' current'

' table'

' event'

' or'

'Thanks'

' Determine'

'Now'

' scene'

' latter'

' Note'

' initially'

' kinds'

' So'

' pretty'

' automatically'

' hypothesis'

' persons'

' Moreover'

' authors'

' upgrade'

' Hence'

'Got'

'Then'

'Once'

' generally'

' truly'

' provided'

' discussed'

'see'

' estimate'

' novel'

'In'

' used'

' world'

' participants'

'Like'

' fairly'

' For'

Head 0:3

' Peter'

' Phil'

' Bob'

' Jacob'

' Ray'

' Smith'

' Russian'

' Man'

' Doug'

' Russia'

' Kent'

' Bell'

' Mart'

' Stan'

' Mac'

' Simon'

' Lewis'

' Moscow'

' David'

' Taylor'

' Russ'

' Sam'

' Albert'

' Ben'

' Lee'

' Tom'

' Jack'

' Lin'

' Gal'

' Perry'

' Israel'

' Apple'

' Rod'

' Alexander'

' Fox'

' Roy'

' Johnson'

' Williams'

' Dan'

' Bern'

' Ken'

' Prince'

' Philip'

' Bol'

' Chinese'

' Franklin'

' Wilson'

' Scott'

' Rich'

' Stephen'

' Rob'

' Jim'

' Joe'

' SD'

' Kim'

' Thomas'

' Andrew'

' Tony'

' James'

' Gil'

' Ross'

' Daniel'

' Brad'

' Israeli'

' Ted'

' Hamilton'

' Roman'

' Mil'

' Bill'

' Indian'

' Jewish'

' Ty'

' Churchill'

' Harry'

' Roger'

' Walter'

' Iran'

' Martin'

' Ham'

' Sher'

' White'

' Christian'

' Hal'

' Lu'

' Lou'

' Michael'

' Chris'

' HT'

' Victor'

' China'

' Clinton'

' Tor'

' SM'

' Charles'

' Core'

' Marg'

' Tim'

' George'

' Row'

' Leon'

' Jon'

' Irish'

' India'

' Tai'

' Lincoln'

' Mike'

' Frank'

' Pakistan'

' Howard'

' Robin'

' Japanese'

' North'

' Mad'

' Star'

' Richard'

' Greg'

' Louis'

' Mary'

' Ron'

' East'

' Arab'

' Brit'

' Ukraine'

' Kell'

' John'

' Henry'

' Paul'

' Greek'

' Wil'

' Ukrainian'

' Adam'

' Super'

' Hen'

' Ireland'

' Turkey'

' Harris'

' Korea'

' Angel'

' Afghan'

' Mah'

' Grand'

' Matt'

' Rome'

' French'

' Lat'

' Naz'

' Muslim'

' Tur'

' Bul'

' Robert'

' Lem'

' Italian'

' Jeff'

' Palestinian'

' Jimmy'

' Jean'

'Phil'

' Black'

' Ram'

' Fran'

' Francis'

' MS'

' German'

' Pan'

' BC'

' Japan'

' Dave'

' gp'

' Luc'

'Lin'

' Allen'

' Egypt'

' Jew'

' Ren'

' Jews'

' Jac'

' Aust'

' Sy'

' Canadian'

' Carol'

' Edward'

' Microsoft'

'Ben'

' Ed'

' Mark'

' Catholic'

' Ali'

'oscow'

' Gi'

' micro'

' King'

' France'

' Jose'

' NT'

' Poland'

' Morgan'

' Dra'

' Wol'

' Berlin'

' Julie'

Head 0:4

' i'

't'

' op'

'<'

'j'

' val'

'by'

'i'

'>'

' j'

' u'

'['

' e'

' pos'

' orig'

' lang'

' vol'

' or'

'El'

'[('

' s'

' arg'

'is'

'_['

' she'

' v'

' n'

' o'

'v'

' dic'

' al'

'o'

' es'

' k'

' van'

'She'

'_"'

' y'

'res'

' ll'

'orig'

' et'

' im'

'Dic'

' course'

'ait'

'dic'

'arg'

' t'

' ap'

' dev'

'ä'

' ed'

' we'

'she'

'Code'

' te'

'Definition'

' and'

'Im'

' de'

'é'

'lang'

' section'

'La'

' z'

' opt'

' he'

'Le'

'ø'

'are'

' el'

' &'

're'

'ö'

'è'

' dat'

' She'

'ij'

'te'

'â'

'á'

'[-'

'ü'

' par'

'dat'

'uid'

'å'

'k'

'Arg'

' uid'

'Str'

'Val'

' res'

' em'

' pt'

'pers'

' ma'

'ais'

'aren'

' doc'

'Der'

'Si'

'sys'

' sys'

'z'

' aux'

'í'

' d'

'æ'

'Es'

' du'

'Her'

"['"

' le'

'ie'

'seg'

' Je'

' not'

'I'

'én'

'n'

' ai'

'["'

' We'

'�' (\xc3)

'Je'

'Contents'

'ă'

'per'

'ô'

' He'

'är'

'és'

' argparse'

' her'

' Jr'

' Pat'

' om'

'.\\'

'He'

'We'

' result'

's'

'De'

' fs'

'ten'

'iam'

' dir'

'sen'

'ée'

'Par'

'val'

'Al'

'e'

' Her'

' å'

'nom'

' ya'

'ó'

'\\['

' av'

' ti'

'By'

' sch'

' na'

'elle'

'anti'

' Inc'

' Val'

'ham'

'ii'

' status'

' him'

' ter'

'à'

' they'

' det'

'aire'

'Am'

' ne'

' girl'

'ido'

' I'

'Sch'

'wise'

'Ter'

'ai'

'ür'

'var'

' dem'