--- type: WebPage title: **Instructions:** description: "" resource: https://arxiv.org/pdf/2602.14080 tags: [] timestamp: 2026-09-02T22:59:07.065473Z --- | Empty | | Shelves | | or | Lost | Keys? | | Recall | | Is | the | | | | ---------- | --- | ------- | --- | ---------- | ---- | ----- | ---------- | ------ | --- | --- | --- | --- | --- | | Bottleneck | | | for | Parametric | | | Factuality | | | | | | | NitayCalderon1,2,EyalBen-David1,ZorikGekhman1,EranOfek1 andGalYona1 1GoogleResearch,2Technion–IsraelInstituteofTechnology Standard factuality evaluations of LLMs treat all errors alike, obscuring whether failures arise from missing knowledge (empty shelves) or from limited access to encoded facts (lost keys). We propose a behavioral framework that profiles factual knowledge at the level of facts rather than questions, characterizingeachfactbywhetheritisencoded,andthenbyhowaccessibleitis: cannotberecalled, canbedirectlyrecalled,orcanonlyberecalledwithinference-timecomputation(thinking). Tosupport 6202 nuJ 91 ]LC.sc[ 2v08041.2062:viXra suchprofiling,weintroduceWikiProfile,anewbenchmarkconstructedviaanautomatedpipelinewitha promptedLLMgroundedinwebsearch. Across4millionresponsesfrom13LLMs,wefindthatencoding isnearlysaturatedinfrontiermodelsonourbenchmark,withGPT-5andGemini-3encoding95–98% of facts. However, recall remains a major bottleneck: many errors previously attributed to missing knowledgeinsteadstemfromfailurestoaccessit. Thesefailuresaresystematicanddisproportionately affect long-tail facts and reverse questions. Finally, we show that thinking improves recall and can recoverasubstantialfractionoffailures,indicatingthatfuturegainsmayrelylessonscalingandmore +thinking onmethodsthatimprovehowmodelsutilizewhattheyalreadyencode. +thinking | | | | | | | | 1. Introduction | | | | | | | | --- | ----------------- | --- | --- | --------------- | --- | --- | --------------- | --- | --- | --- | --- | --- | --- | | | Encoding Failure | | | Recall Failure | | | | | | | | | | | | (Empty Shelves) | | | (Lost Keys) | | | | | | | | | | LargeLanguageModels(LLMs)encodeavastamount offactualinformationintheirparameters(Joshietal., 2017;Kwiatkowskietal.,2019;Mallenetal.,2023; Not encoded and Inaccessible facts: Petronietal.,2019),yetfactualerrorsremainapersis- not known. Encoded but not known. tentchallenge(Haasetal.,2025;Ravichanderetal., | | Inference | | | | | | 2025;Weietal.,2024a). | | | | Whileexistingevaluations | | | | --- | ---------------- | --- | ------------- | --- | ------------- | --- | --------------------- | --- | --- | --- | ------------------------ | --- | --- | | | | | Recall | | Direct | | | | | | | | | | | without Encoding | | with Thinking | | Recall | | | | | | | | | consideraresponseaseithercorrectorincorrect,ac- curacyaloneprovideslittleinsightintothesourceof | | | | | | | | theerror. | IfanLLMincorrectlyanswersthequestion | | | | | | | --- | --- | --- | --- | --- | --- | --- | --------- | ------------------------------------ | --- | --- | --- | --- | --- | “WhichfamousbandplayedtheirfirstgigattheBoard- | | | | | | | | walk | club?” | (the answer | | is Oasis), | can we | tell why? | | --- | ---------------------- | --- | ------------------ | ------------------- | --- | --- | ---- | ------ | ----------- | --- | ---------- | ------ | --------- | | | Not encoded but infe- | | Low-accessibility | High-accessibility | | | | | | | | | | rable with thinking facts: Encoded but facts: Encoded and does not encode | | | | | | | | One | possibility | is that | the | LLM | | this | | --- | ------------------------ | --- | ---------------- | -------------- | --- | --- | --- | ----------- | ------- | --- | --- | --- | ---- | | | (e.g., via multi-hop or | | known only with | known without | | | | | | | | | | educated guesses). informationinitsparameters(e.g.,duetolimiteddata | | | | thinking. | thinking. | | | | | | | | | | | --- | --- | --- | ---------- | --------- | --- | --- | ------------------------- | --- | --- | --- | ---------------------- | --- | --- | | | | | | | | | coverageormodelcapacity). | | | | Alternatively,afactmay | | | Gemini 10.9 14.9 72.2 Knows:87.5 beencodedbutnotaccessibleunderdifferentconditions 3-Pro GPT-5 12.2 21.5 61.6 83.8 thanhowitwaslearned. Oneexampleisthereversal GPT-5 14.0 19.0 20.3 43.5 67.0 curse(Berglundetal.,2024;Linetal.,2024c),where mini Gemma3 23.1 31.6 8.6 36.2 anLLMcananswerthatOasisplayedtheirfirstgigat | | 27b | | | | | 45.3 | | | | | | | | | ------ | -------- | ---------- | --- | -------------- | -------- | ---- | ------------------- | --- | ----- | --------- | --------- | --- | ------ | | | | | | | | | the Boardwalk | | club, | yet fails | to answer | who | played | | Figure | 1 | Top: | We propose | | five knowledge | profiles | | theirfirstgigthere. | | | | | | | Bottom: thatcharacterizefacts. Percentagesofthese Encodingandrecallfailuresareindistinguishable | profilesacrossselectedLLMs,revealing: | | | | | (1)Scaling | | | | | | | | | | ------------------------------------- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | underaccuracymetrics,yettheyimplydifferentlimi- fills “empty shelves” by reducing encoding failures: tationsandsolutions. Encodingfailurescallforpre- | frontier | LLMs | encode | nearly | all facts | | | | | | | | | | | -------- | ---- | ------ | ------ | --------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | in our data. traininginterventions,suchasscalingmodelsizeor (2)Recallfailuresremainabundantdespitescaling, | | | | | | | | data | coverage. | Recall | failures | suggest | post-training | | | --- | --- | --- | --- | --- | --- | --- | ---- | --------- | ------ | -------- | ------- | ------------- | --- | (3)Think- leavingsubstantialroomforimprovement. interventionsthatoftenimprovehowmodelsutilize ingactsasarecoverymechanismoffactsthatwould whattheyalreadyencodeGekhmanetal.(2024);Lin otherwiseremain“lost”. etal.(2024a);Zhouetal.(2023). Correspondingauthor(s):nitayc@google.com,galyona@google.com.PublishedintheProceedingsofthe43𝑟𝑑 InternationalConferenceon MachineLearning(ICML2026).WikiProfileisavailableathttps://huggingface.co/datasets/google/WikiProfile. © 2026Google.Allrightsreserved EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality We therefore propose shifting the unit of analy- cases: factpopularityandthereversalcurse. While facts. sis from individual questions to We introduce theencodinggapbetweenpopularandlong-tailfacts knowledge profiling: a framework that categorizes forfrontierLLMsissmall(afewpercentagepoints), facts into one of five profiles (see Figure 1), based therecallgapissubstantiallylarger(oftenexceeding onwhetherthefactisencoded,andbyhowaccessible 25%). Reversequestionsrevealasurprisingpattern: itis: cannotberecalled, canbedirectlyrecalled, or LLMsstruggletoanswerreversequestionscompared canberecalledonlywithinference-timecomputation to direct ones, despite showing no disadvantage in (thinking). Weusethinkingtorefertoinference-time recognizingthecorrectanswerundermultiple-choice | techniquesthatelicitintermediatecomputationsbe- | | | | | | evaluation. | | | | | | | | ----------------------------------------------- | --- | --- | --- | --- | --- | ----------- | --- | --- | --- | --- | --- | --- | forethefinalanswer,includingbothchain-of-thought Together,ourresultsrefineprioraccountsoflong- (CoT)promptinganddedicatedthoughttokensgener- | | | | | | | tail errors | and | the reversal | | curse (Berglund | | et al., | | ----------------------------- | --- | -------- | --- | ------------------- | --- | ------------ | ------- | ---------------------------------- | ------- | --------------- | ----- | ------- | | atedbythinking-optimizedLLMs. | | | | Weoperationalize | | | | | | | | | | | | | | | | 2024; He | et al., | 2025; | Kandpal | et al., | 2023; | Mallen | | the definitions | of | encoding | and | recall behaviorally | | | | | | | | | | | | | | | | etal.,2023). | | Whereasthesefailuresareofteninter- | | | | | Wemeasureencodingby (HoltzmanandTan,2025). pretedaslimitationsofmemorizationorbidirectional | priming | the LLM | to complete | | a factual | proposition | | | | | | | | | ------- | ------- | ----------- | --- | --------- | ----------- | -------------------------------------------- | --- | --- | --- | --- | --- | ---- | | | | | | | | encoding,ourresultssuggestadifferentpicture: | | | | | | rare | usingacontextthatresembleshowthefactwaslikely factsareoftenencodedbutinaccessible,andreverse | tobeencounteredduringpre-training. | | | | Wemeasure | | | | | | | | | | ---------------------------------- | --- | --- | --- | --------- | --- | --------- | ------------- | --- | ---- | ---- | ---- | --------- | | | | | | | | facts can | be recognized | | even | when | they | cannot be | recall1 byaskingquestionsabouttheseencodedfacts | | | | | | | generated. | Thisreframesbothphenomenaasrecall | | | | | | | ---------------- | --- | --------- | --------- | --------- | ------ | ---------- | --------------------------------- | --- | --- | --- | --- | --- | | across different | | contexts, | including | phrasings | or re- | | | | | | | | failuresratherthan‘missingknowledge’. Ourresults | lationalorderings. | | Thisenablesthestudyoffrontier | | | | | | | | | | | | ------------------ | --- | ----------------------------- | --- | --- | --- | --------------- | --- | ----- | ---- | ---------- | ---------- | --- | | | | | | | | also complement | | prior | work | on ‘hidden | knowledge’ | | LLMsdespitetheinaccessibilityoftheirweightsand | | | | | | | (Gekhmanetal.,2025;Orgadetal.,2025): | | | | | | whatap- | | --- | --- | --- | --- | --- | --- | ------------------------------------ | --- | --- | --- | --- | --- | ------- | trainingdata. pearstobeknowledgeaccessibleonlythroughmodel To support knowledge profiling, we introduce internals may reflect encoded facts whose recall is WikiProfile,anewbenchmarkcomprising2,150 context-dependent,suchthattherightpromptorcon- facts, each paired with 10 questions that probe textcanelicit(Jiangetal.,2020). | encoding, | recall, | and recognition | | (multiple-choice). | | | | | | | | | | --------- | ------- | --------------- | --- | ------------------ | --- | --- | --- | --- | --- | --- | --- | --- | GiventhatencodinginfrontierLLMsisnearingsat- | WikiProfile | is | constructed | | via a fully | automated | | | | | | | | | ----------- | --- | ----------- | --- | ----------- | --------- | --- | --- | --- | --- | --- | --- | --- | urationwhilesubstantialheadroomremainsforrecall, | multi-step | pipeline | that | employs | a prompted | LLM | | | | | | | | | ---------- | -------- | ---- | ------- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | futureimprovementsarelikelytocomefrombetter groundedinwebsearchtogenerate,refine,andver- utilizationofexistingknowledge.Onemechanismthat | ifyquestions. | Unlikeotherbenchmarksthatrelyon | | | | | | | | | | | | | ------------- | ------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | alreadydemonstratesthisisthinking:ourresultsshow syntheticorschema-restrictedknowledge-basetriplets | | | | | | | that thinking | | recovers | 40–65% | of encoded-but-not- | | | | --- | --- | --- | --- | --- | --- | ------------- | --- | -------- | ------ | ------------------- | --- | --- | (Kumaretal.,2024;Luoetal.,2023;Wuetal.,2025b; | | | | | | | directly-knownfacts. | | | Thegainsaremostpronounced | | | | | --- | --- | --- | --- | --- | --- | -------------------- | --- | --- | ------------------------- | --- | --- | --- | Yuanetal.,2024),ourfactsareextractedfromnatural inchallengingsettings,suchaslong-tailfactsandre- | documents. | Thisenablesevaluationinmorerealistic | | | | | | | | | | | | | ---------- | ------------------------------------ | --- | --- | --- | --- | --------------- | --- | ----------------------------------- | --- | --- | --- | --- | | | | | | | | versequestions. | | Thisparallelsthe“tip-of-the-tongue” | | | | | settings,wherefactsareexpressedindirectly,andmay phenomenoninhumanmemory(BrownandMcNeill, involvecomplex,context-dependentrelations(Wang 1966;Schwartz,2002),inwhichadditionaleffortcan etal.,2025b). surfaceknowledgethatispresentbutmomentarilyin- Using WikiProfile,weevaluate13LLMs,result- accessible. Post-trainingandinference-timemethods inginmorethan4milliongradedanswers. Wefind maythusplayacentralroleinfactualitybyimproving that scaling primarily improves encoding, while re- howmodelsutilizewhattheyalreadyencode. | call remains | a major | bottleneck. | | For frontier | LLMs, | | | | | | | | | ------------ | ------- | ----------- | --- | ------------ | ----- | --- | --- | --- | --- | --- | --- | --- | Insummary,ourcontributions:(1)Weproposeabe- includingGemini-3-ProandGPT-5,encodingisnearly havioralframeworkthatcharacterizesfactualknowl- | saturated, | with | 95–98% | of facts | encoded. | Yet the | | | | | | | | | ---------- | ---- | ------ | -------- | -------- | ------- | --- | --- | --- | --- | --- | --- | --- | edgeinLLMsbyseparatingencodingfromrecallvia samemodelsstillcannotrecall25–33%offactswith- | | | | | | | knowledgeprofiles;(2)Weintroduce | | | | | WikiProfile, | | | ------------ | --------------------------------- | --- | --- | --- | --- | -------------------------------- | --- | ------------- | --- | ---------- | ------------ | ------ | | outthinking. | Inaddition,weshowthatscalingmodel | | | | | | | | | | | | | | | | | | | a benchmark | | for knowledge | | profiling, | along | with a | sizealoneincreaseswhatthemodelencodesbutdoes fullyautomatedpipelineforconstructingsuchbench- notnecessarilyimproveitsrecall. | | | | | | | marks; and | (3) | Through | evaluation | | of 13 | LLMs on | | --- | --- | --- | --- | --- | --- | ---------- | --- | ------- | ---------- | --- | ----- | ------- | Why do LLMs struggle to access knowledge they Wikipedia-derivedfacts,weshowthatthebottleneck have encoded? Our findings suggest that recall is forfactualityisrecall,notencoding,andthatthinking tightly coupled to the conditions under which facts helpsrecoverotherwiseinaccessibleknowledge. werelearned,degradingwhenqueriesdivergefrom | training-timepatterns. | | Wemanifestthisthroughtwo | | | | | | | | | | | | ---------------------- | --- | ------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 1Byanalogytohumanmemory,weuseencodingtodenotepara- metricrepresentation,andrecalltoaccessingencodedfacts. 2 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | 2. Knowledge | | | Profiling | | | | | | | | | | | | ------------ | --- | --- | --------- | --- | --- | --- | --- | --------------------- | ------------------------------------ | --- | --- | --- | --- | | | | | | | | | | pre-training document | A fact is a proposition between two | | | | | entities. The subject entity appears before Wefocusonfactualknowledgeastheabilitytoanswer the object entity in a given document. | short-formquestionscorrectlywithoutexternaltools. | | | | | | | | SUBJECT | | | | | | | ------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | Oasis played their first gig OBJECT Our aim is to provide a systematic, model-agnostic at the Boardwalk club | characterization | | of | factual | behavior | that | applies | to | | | | | | | | ------------------------------ | --- | --- | ------- | -------- | ------------ | ------- | ----------------------------- | --- | --- | ------------------------------- | --- | --- | --- | | | | | | | | | The LLM encodes a fact if it | | | The LLM knows a fact if it can | | | | | bothclosed-andopen-weightLLMs. | | | | | Tothisend,we | | | | | | | | | | | | | | | | | can reproduce it in a | | | answer questions about it in | | | | introducetwocomplementaryoperationalnotions,de- | | | | | | | | pre-training like context. | | | different contexts. | | | | | --------------------------------------- | --- | --- | --- | --- | --- | -------- | --------------------------- | --------------------- | --- | -------------------- | --- | ------- | -------- | | finedpurelyintermsofobservablebehavior: | | | | | | encoding | | | | | | | | | andknowledge. | | | | | | | | pre-training document | | Direct | | SUBJECT | ? OBJECT | Together,theyallowustodetermine | | | | | | | | | Oasis are an English rock | | | Where did Oasis play their first gig? | | | | --- | --- | --- | --- | --- | --- | --- | --- | ---------------------------------- | --- | --- | ------------------------------------- | --- | --- | whethererrorsarisefromencodingorrecall,thereby band formed in Manchester in | | | | | | | | | 1991. The group initially | | | Where was Oasis's first show? | | | | --- | --- | --- | --- | --- | --- | --- | --- | -------------------------- | --- | --- | ----------------------------- | --- | --- | informingdifferentpathstoimprovement. consisted of Liam Gallagher Reverse | | | | | | | | | (lead vocals), ........................... | | | | OBJECT | ? SUBJECT | | --- | --- | --- | --- | --- | --- | --- | --- | ------------------------------------------ | --- | --- | --- | ------ | --------- | .................................................. | | | | | | | | | Oasis played their first gig on | | | Which famous band played their first | | | | --- | --- | --- | --- | --- | --- | --- | --- | -------------------------------- | --- | --- | ------------------------------------- | --- | --- | 2.1. OperationalizingEncodingandKnowledge gig at the Boardwalk club? 14 August 1991 at ________ Who played their first show at the Boardwalk club? | FactsandQuestions: | | | Wedefineafactasaproposi- | | | | | | | | | | | | ------------------ | --- | --- | ------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | Top: | tion2 | | | | | | | Figure | 2 | | We extract | facts | from | Wikipedia, | a | | --------------------------------- | --- | -------------------------------- | --- | --- | --- | -------- | ----------- | --- | ------------ | ------------ | ---- | ---------- | -------- | | involvinganorderedpairofentities: | | | | | | asubject | | | | | | | | | | | | | | | | predominant | | source of | pre-training | | data. | Left: We | | andanobject. | | Therolesofsubjectandobjectarede- | | | | | | | | | | | | measureencodingbypromptingtheLLMtoreproduce terminedbythesourcetextfromwhichthefactwas | | | | | | | | facts | within | their original | context, | | testing | whether | | ----------------------------------- | --- | --- | --- | --- | --- | ------------ | ------------------------------------ | ------ | -------------- | -------- | --- | ------- | --------- | | extracted(e.g.,aWikipediadocument): | | | | | | thesubjectis | | | | | | | | | | | | | | | | theyarestoredinthemodel’sparameters. | | | | | | Right: We | theentitythatappearsfirstinthetext,andtheobject measureknowledgebyaskingquestionsacrossvaried | appears | subsequently. | | A question | | whose | answer | is | | | | | | | | ------- | ------------- | --- | ---------- | --- | ----- | ------ | --- | --- | --- | --- | --- | --- | --- | direct question, phrasingsandrelationaldirections,withandwithout | the object | is | termed | a | | | while a | ques- | | | | | | | | ---------- | --- | ------ | --- | --- | --- | ------- | ----- | --- | --- | --- | --- | --- | --- | thinking. tionwhoseansweristhesubjectistermedareverse question. ExamplesareshowninFigure2. | | | | | | | | reproducethatfactinapre-training-likecontext. | | | | | | In- | | --- | --- | --- | --- | --- | --- | --- | --------------------------------------------- | --- | --- | --- | --- | --- | --- | Foreachquestion𝑞,wegenerate𝑛responsesfrom tuitively,successfulreproductionundertraining-like theLLManduseapromptedLLMgrader(autorater)to conditionsprovidessufficientevidencethatthefactis compareeachresponsetothegoldanswerandlabel | | | | | | | | encoded(thoughitmaynotbeknown). | | | | | Sinceencod- | | | ------------- | --- | ----------- | --- | ----- | ----- | ------ | ------------------------------------------------- | --- | --- | --- | --- | ----------- | --- | | it as correct | or | incorrect.3 | | Let 𝑐 | and 𝑖 | denote | the | | | | | | | | | | | | 𝑞 | | 𝑞 | ing,underthisdefinition,isassessedbehaviorally,we | | | | | | | numberofresponsestoquestion𝑞gradedascorrect operationalizeitusingtasksthatstronglyprimethe | and incorrect. | | We | define | the question | | grade | as the | | | | | | | | -------------- | --- | --- | ------ | ------------ | --- | ----- | --------------------------------- | --- | --- | --- | --- | --- | ---------- | | | | | | | | | modelwiththecontextinwhichthefact | | | | | 𝑓 | originally | accuracyovergradableresponses: appeared. Wedenotethesetofsuchencodingtasks | | | | | | | | | WesayanLLMencodesthefact | | | | 𝑓 | | | --- | --- | --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | --- | --- | --- | | | | | | 𝑐 | | | byE | 𝑓. | | | | if: | | 𝑞 | | | | 𝑔(𝑞) = | . | | | | | | | | | | | --- | --- | --- | ------ | ---- | --- | --- | --- | --- | ----- | ------ | --- | --- | --- | | | | | | 𝑐 +𝑖 | | | | | | | | | | | | | | | 𝑞 𝑞 | | | | | ∃𝑞∈ E | , 𝑔(𝑞) | >𝜏, | | | 𝑓 | | | | | | | | where𝜏isapredefinedthreshold. | | | | | Weset𝜏=0.5in | | | --------- | --------------------------------- | --- | --- | --- | --- | --- | ------------------------------------------ | --- | --- | --- | --- | ------------ | --- | | Encoding: | Thenotionofencodinghasbeendefined | | | | | | | | | | | | | | | | | | | | | allexperimentsasanaturalmajoritythreshold. | | | | | | We | inmultiplewaysinpriorwork,typicallyreferringto showinAppendixD.2thatourfindingsarerobustto informationstoredinthemodel’sinternalparametric thischoice. | representations. | | Existing | | approaches | to | measuring | | | | | | | | | ---------------- | --- | -------- | --- | ---------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- | Inoursetup,E encoding often rely on access to internal states, a 𝑓 consistsoftwotasks. Thefirsttask ispropositioncompletion(seeFigure2),inwhichthe | requirement | | that does | not | align | with | our focus | on | | | | | | | | ----------- | --- | --------- | --- | ----- | ---- | --------- | --- | --- | --- | --- | --- | --- | --- | modelisgiventheleftcontext(theentiresourcetext | evaluating | frontier | | LLMs. | Moreover, | the | extent | to | | | | | | | | ---------- | -------- | --- | ----- | --------- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | whichthesemethodsreliablycapturewhetheramodel truncatedimmediatelybeforetheobjectentity),and trulyencodesafactremainsanopenquestion(Chen isaskedtocompletethefactualstatement. Thistask | | | | | | | | directly | | mimics the pre-training | | objective | | for which | | ------------- | --- | ------ | ------- | ----- | ---- | ------- | -------- | --- | ----------------------- | --- | --------- | --- | --------- | | et al., 2025; | | Haller | et al., | 2025; | Hase | et al., | 2023; | | | | | | | Huangetal.,2024;Maetal.,2023;Weietal.,2024b). the LLM was optimized. Notably, the answer (the objectentity)doesnotappearintheleftcontext. Wethereforeadoptanencoding-via-memorization contextual perspective: anLLMencodesafactifitcancorrectly The second task, which we refer to as | | | | | | | | questioning, | | uses the same | left | context | but | replaces | | --- | --- | --- | --- | --- | --- | --- | ------------ | --- | ------------- | ---- | ------- | --- | -------- | 2Althoughfactsareoftendescribedassubject-relation-object thefinalsentencewithadirectquestionaboutthefact. triplets,inourdatatheymayinvolvemultipleorcomplexrelations Weincludebothtasksbecausecompletion-basedeval- andarethereforetreatedaspropositions. uationcanbeambiguousforpost-trainedLLMs,which 3Thegradermayassign‘partially’or‘other’labelswhenare- sponsecannotbereliablyevaluated.See§4. are optimized to answer questions rather than con- 3 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality tinuepre-trainingdocuments. LLMsmaycontinuea improvements beyond pre-training, such as in post- | sentenceinmultipleplausiblewaysthatdonotexplic- | | | | | | training. | | | | | ----------------------------------------------- | --- | ------------------------ | --- | --- | --- | --------- | --- | --- | --- | | itlystatethetargetfact. | | Thecontextualquestioning | | | | | | | | DirectRecall:Thefactisencodedandknownwithout | task mitigates | this | ambiguity | by | explicitly | eliciting | | | | | | -------------- | ---- | --------- | --- | ---------- | --------- | --- | --- | --- | --- | thetargetfactinquestionformwhilepreservingthe thinking. This straightforward recall reflects highly originalsourcecontext,whichprimesthemodel. accessibleknowledge,whichisdesirableinmanyap- plications(e.g.,whenintegratingfactsintolong-form Wedeliberatelyexcludethinkingfromtheencoding responses). | evaluation. | Correct | reproduction | | can arise | not only | | | | | | ----------- | ------- | ------------ | --- | --------- | -------- | --- | --- | --- | --- | fromdirectencodingbutalsofrominference,includ- RecallwithThinking: Thefactisencodedbutknown ingmulti-hopreasoningoreducatedguessingbased onlywiththinking. Thisindicatesthatthefactisnot onotherencodedfacts. Toavoidconflatingencoding readilyaccessibleandrequiresadditionalcomputation withinference,encodingisalwaysmeasuredwithout toberecalled. Thisprofilecanbeviewedasarecovery | thinking. | Knowledge, | in contrast, | | is evaluated | both | | | | | | --------- | ---------- | ------------ | --- | ------------ | ---- | --- | --- | --- | --- | mechanismforotherwiseinaccessiblefacts. withandwithoutthinking. InferencewithoutEncoding:Thefactisnotencoded Knowledge: We say that an LLM knows a fact if it butisknownwhenthinkingisenabled. Inthiscase, cancorrectlyanswerquestionsaboutitacrossarange correctness may arise from inference over other en- of semantically equivalent contexts. This definition codedfacts,suchasmulti-hopreasoningoreducated | | | | | | | guessing. | It is also possible | that the | fact is encoded | | --- | --- | --- | --- | --- | --- | --------- | ------------------- | -------- | --------------- | capturestheintuitionthatfactualknowledgeshould berobusttosuperficialchanges,suchasinphrasing butmissedbyourencodingtasks. Thisprofileisless andtoalternativerelationaldirections(Elazaretal., reliable;relyingonitmaypromotehallucinations,and 2021; Meng et al., 2022; Zucchet et al., 2025). Let itoccurslessfrequentlyinpractice. | Q denoteasetofquestionsassociatedwithafact | | | | | | 𝑓. | | | | | ------------------------------------------ | --- | --- | -------- | --- | --- | -------------------------------------------- | --- | --- | --------- | | 𝑓 | | | | | | Finally,wedonotdefineaprofileforfactsthatare | | | | | WesayanLLMknowsafact | | | 𝑓 if: | | | | | | | | | | | | | | notencodedbutknownwithoutthinking. | | | Whilesuch | | | ∀𝑞∈ | Q , | 𝑔(𝑞) >𝜏, | | | casesarepossible,theyoccurinfewerthan0.5%of | | | | 𝑓 instancesacrossallevaluatedmodelsandarelikely Thecontrastwithencodingisintentional: encoding attributable to sampling noise or grader error. We usesexistentialquantification(∃)becausereproduc- thereforeexcludethisprofile. ingafactinanyprimingcontextsufficesasevidence ofstorage,whereasknowledgeusesuniversalquantifi- | | | | | | | 3. The | WikiProfile | Benchmark | | | --- | --- | --- | --- | --- | --- | ------ | ----------- | --------- | --- | cation(∀)becauserobustrecallshouldnotdependon Inoursetup,Q | phrasingorquerydirection. | | | | | 𝑓 consists | | | | | | ------------------------- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | of four questions: two direct and two reverse, each Tooperationalizetheknowledgeprofiles,weneeda inadifferentphrasing. Thistestsinvariancetoboth benchmark that supports measuring both encoding surfaceformandrelationaldirection. andknowledge. WehenceintroduceWikiProfile,a benchmarkforevaluatingandprofilingfactualknowl- 2.2. KnowledgeProfiles edgeinLLMs. Sinceprofilinginvolvesmeasuringen- | | | | | | | coding, | we extract facts | from Wikipedia, | a trusted | | --- | --- | --- | --- | --- | --- | ------- | ---------------- | --------------- | --------- | referencethatpre-trainingpipelinesarelikelytopri- | Given our | operational | definitions, | | each | fact can be | | | | | | --------- | ----------- | ------------ | --- | ---- | ----------- | --- | --- | --- | --- | characterizedalongthreedimensions: whetheritis oritize,makingitbothastrongcandidateforencoded encoded, whether it is known without thinking, and knowledgeandareliablegroundtruth.WikiProfile whetheritisknownonlywiththinking. | | | | | Theinteraction | | consists | of 2150 facts, | each with ten | corresponding | | --- | --- | --- | --- | -------------- | --- | -------- | -------------- | ------------- | ------------- | ofthesedimensionsyieldsqualitativelydistinctknowl- questions. These include two questions for measur- edge profiles. In particular, we identify five profiles, ingencoding(proposition-completionandcontextual illustratedinFigure1. questions);fourquestionsformeasuringknowledge (directandreversequestions,eachinstantiatedwith Encoding Failure: The fact is neither encoded nor twodistinctphrasings);andfourmultiple-choicevari- | known,withandwithoutthinking. | | | | Ahighrateofen- | | | | | | | ----------------------------- | --- | --- | --- | -------------- | --- | --------------------------------------------- | --- | --- | ----- | | | | | | | | antsformeasuringfactverificationcapabilities. | | | Exam- | codingfailuresindicateslimitationsinmodelcapacity plesoftasksareinTable1. | or pre-training | data | coverage, | suggesting | | interven- | | | | | | --------------- | ---- | --------- | ---------- | --- | --------- | -------------- | ---------------------------- | --- | --- | | | | | | | | Allquestionsin | WikiProfilehaveasingle,unam- | | | tionssuchasincreasingmodelsizeorexpandingthe | | | | | | | biguous | gold answer, | enabling automatic | validation | | --- | --- | --- | --- | --- | --- | ------- | ------------ | ------------------ | ---------- | trainingset. | | | | | | | with an | LLM grader (see | §4). WikiProfile | is con- | | --- | --- | --- | --- | --- | --- | ------- | --------------- | ---------------- | ------- | Recall Failure: The fact is encoded but not known, structedthroughafullyautomatedpipeline(Gemini- even with thinking. This indicates that the fact is 2.5-Prowiththinking;Comanicietal.2025)thatmay storedbutinaccessibleatinferencetime,andcallsfor be applicable to other corpora for domain-specific 4 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Left-context Wikipedia Object entity Direct Question Box D.13 Box D.14 Documents Reverse Question Rephrasing Adding Context Box D.7 Box D.11 -Sounds natural and -Minor edits to Box D.1 Generating Direct Question Generating Reverse Question conversational improve fluency Document Categorization -High verbatim (same -Select the subject (e.g., History, Science, ...) words as the left-context) -Add context (disambiguate) Box D.10 Trivial Filtering reverse Box D.2 Box D.8-9 Box D.8-9 Grounded by Google Search Named Entity Recognition Refinement Refinement Not unique -Specification: The requir- Specification, Minimalism Direct Natural Box D.3 ed level of detail is clear Becomes Question Object Entity Selection -Minimalism: Uses only Box D.11 trivial -Non-trivial, non-guessable the information necessary Becomes Verification Reverse Natural -Not time sensitive for disambiguation trivial A valid direct-reverse pair Question -Left-context ? a single, (roles swapped): specific, and unique answer Box D.10 subject object Not valid Contextual Question Filterting Box D.15 Box D.4-6 Grounded by Google Search Box D.10 Generate Distractors Suitability Verification Discard if: Filtering -Same entity type Three prompts. -Clarification is reuqired Grounded by Google Search -Multiple answers Not unique Not unique - - P R l e a l u y s o ib n l e th a e lt c e o r n n t a e t x iv t es M Q u u l e t s ip ti l o e n -C s h ( o x i 4 c ) e Proposition Reverse Question -Thematically similar Completion Task Direct Question -Popular association Not suitable Figure 3 | The WikiProfile Creation Pipeline: We propose a fully automated pipeline based on prompted LLMs. Theyellowboxesdenotethespecificpromptsusedateachstep(see§E).Left(purple): Factextraction andconstructionofthepropositioncompletiontask. Center(redandblue): Constructionofdirectandreverse questionsviageneration,refinement,andfiltering(groundedbyGoogleSearch). Right(green): Creationof remainingquestions(naturalphrasing,contextual,andmultiple-choiceversions)basedonthedirect/reverse pairs. Facts are discarded throughout the pipeline if their associated questions are rejected. Full details in AppendixA.Theencodingtasksare‘PropositionCompletion’and‘ContextualQuestion’. Theknowledgetasks are‘Direct/Reverse’,‘Direct/ReverseNatural’questions. knowledgeprofiling. QuestionGeneration:Wegeneratedirectandreverse questionsthroughathree-stepprocessofgeneration, Pipeline Overview: The full pipeline spans dozens refinement,andfiltering,ensuringthateachquestion ofstepsandfifteenprompts;hereweprovideahigh- isunambiguous,specific,minimal,andhasaunique leveloverview(Figure3),withcompletedetailsand answer. Directquestionsaregeneratedfirst,andre- prompts in Appendix A and E. All prompts in the versequestionsarethenderivedbyswappingsubject pipeline were carefully developed through manual and object roles. Because our goal is to test encod- optimizationanderroranalysisonaheld-outsubset ing,weinitiallygeneratehigh-verbatimquestionsthat oftensofdocuments,fromwhichtheexamplesused closely match the source text; we then create natu- inthepromptsaredrawn. ralrephrasingsofeachtotestrobustnesstophrasing variation. Contextual questions are constructed by FactExtraction: Toextractfacts,wesample10,000 appendingthedirectquestiontoitsleftcontext,pro- vidingmaximalpriming. Foreachquestion,wealso Wikipediapages,recordtheirvisitcounts,andcate- generateamultiple-choicevariantwiththreeplausi- gorize them into nine topics (see Figures 9 and 10 bledistractorsmatchedbyentitytypeandthematic for distributions). Next, we perform NER to iden- similarity. Allquestionsundergofilteringgroundedin tifyentitiesandtheirtypes(e.g.,DATE,LOCATION, Google Search. We prompt an LLM with each ques- PERSON;seeFigure11). Foreachdocument,wese- tion and discard cases where multiple answers are lect up to 3 candidate object entities (not from the returned or clarification is needed. This filtering is firstsentences),eachofwhichdefinesadistinctfact. strict: if any question is rejected, the entire fact is Thesecandidatesareselectedbasedonstrictcriteria: discarded,leavinguswithapproximately2,200facts. anentityissuitableonlyifitrepresentsanon-trivial, non-guessable,andtime-independent(i.e.,notsubject Finally,wemanuallyvalidatedfactsandtheirassoci- tochange)completionofitsleftcontext. Furthermore, atedquestions,discardingfewerthan2%offactsand the left context must constrain the completion to a yieldingafinalsetof2,150facts. SeeAppendixA.4 single,unambiguousanswer. Afterafinalverification fordetails. step,wedownsampleto5,000candidates,balancing categoriesandentitytypes. Themedianis100words, themeanis116words(±56std). 5 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality 4. Experimental Setup Recall Encoding Inference Recall Direct LLMs: Weevaluate13LLMsfromfivefamilies,cover- Knows: Failure Failure w/o Encoding with Thinking Recall 1.8 14.431.945.356.964.767.075.676.379.483.886.087.5 ingcommercialandopen-weightmodels: Gemini-3 100% andGemini-2.5(ProandFlashvariants)(Comanici 13.0 28.9 32.8 31.6 28.1 26.9 19.0 17.6 19.6 18.1 12.2 11.4 10.9 etal.,2025);GPT-5.2,GPT-5,andGPT-5-mini(Ope- 80% 6.8 12.6 14.9 nAI, 2025b); GPT-4.1 and GPT-4.1-mini (OpenAI, 14.9 8.4 14.0 17.9 10.3 13.0 21.5 60% 2025a); and Gemma3 (1B, 4B, 12B, 27B) (Kamath 23.1 19.2 20.3 35.3 14.6 etal.,2025). Eachmodelisevaluatedwithandwith- 40% 85.2 56.7 8.6 out thinking. Gemini-2.5, Gemini-3, and GPT-5 are 7.4 56.6 65.7 66.4 61.6 73.2 72.2 thinking-optimized; we use default settings when 20% 36.2 41.1 45.0 43.5 thinking is enabled and set effort/budget to zero to 23.6 disable it. For GPT-4.1 and Gemma3, we use CoT 10.4 0% p R r e o s m po p n ti s n e g. Generation: Foreachexample(anincom- Ge m m G a e 3 m -1 m G b e a m 3- m 4 G b a e 3 m -1 m 2b a G 3 P G - T 2 e - 7 m 4 b . i 1 n - i m -2 i . n 5 i -F G la P s T h -5- min G i PT-5 G . G e 2 m PT i - n 4 i- .1 2.5-P G r e o m G i P n T i- - 3 5 G - e F m la i s n h i-3-Pro pletetextoraquestion),wegenerateeightresponses Figure4|KnowledgeProfiles:Distributionofthefive perLLMwithatemperatureof1. InAppendixD.4,we profilesacross13LLMs(percentages). Theblackline justifythechoiceofeightresponses. Weemploythree markspotentialknowledge: thefractionoffactsknown prompts,oneforeachtaskformat(completion,closed- withorwithoutthinking(‘DirectRecall’+‘Recallwith bookQA,andmultiple-choiceQA).Eachpromptin- Thinking’+‘InferencewithoutEncoding’). Asshown, structs the LLM to produce concise, answer-only re- encodingfailuresdecreasesharplywithscale,while sponses (see Box E.17). For multiple-choice ques- recallfailurespersisteveninfrontiermodels. tions,answerchoicesareshuffledeachtime,ensuring thateachoptionisthecorrectanswerexactlytwice. 5. Results In total, we evaluate ≈4.5 million responses (13×2 LLMs×2150facts×10tasks×8responses). Ourresultssuggestthatfactualerrorsarisenotfrom Response Evaluation: We use two prompted LLM missingknowledge(emptyshelves),butfrominaccessi- bleknowledge(lostkeys). Wefirstcharacterizethisre- graders(autoraters)(Weietal.,2024a): oneforthe callbottleneckbyexaminingknowledgeprofilesacross completiontasks(BoxE.18)andonefortheclosed- all13evaluatedmodels(§5.1). Wethenaskwhyre- bookQAtasks(BoxE.19). BothgradersuseGemini- call fails, identifying two systematic patterns, fact 2.5-Prowiththinking. Thegradersassignoneoffour popularityandquestiondirectionality,whereencoded labels: CORRECT, INCORRECT (factual contradic- knowledgeremainsdifficulttoaccess(§5.2). Finally, tion), PARTIALLY (correct but at a different granu- we examine thinking as a mechanism that compen- larity;Yonaetal.2024),orOTHER(unattemptedor satesfortheseweaknesses,recoveringfactsthatdirect unverifiableanswers). IncludingPARTIALLYimproved generation cannot surface (§5.3). Due to the analy- graderstabilityinpreliminaryexperiments. sis scale, we occasionally report results for frontier Asdefinedin§2.1,thequestiongradeiscomputed LLMsonly;completeresultsanddetailsareprovided usingCORRECTandINCORRECTresponsesonly,ex- inAppendixC. cludingPARTIALLYandOTHER,sinceassigningthem weightsissubjective(Kalaietal.,2025). Inpractice, 5.1. TheRecallBottleneck suchresponsesarerare(lessthan5%; seeTable2). When all responses are PARTIALLY or OTHER, the Webeginbyexaminingknowledgefailures,namely, factisexcludedfromaggregatestatistics. Alternative facts that are not known. These correspond to two handling strategies yield negligible differences (see profiles: EncodingFailuresandRecallFailures. Figure4 ouranalysisinAppendixD.1). shows the distribution of all five knowledge profiles across13LLMs. Finally, in Appendix D.3, we assess cross-family grader consistency by comparing graders based on Encoding is saturated; recall is not. For frontier Gemini-2.5-Pro and GPT-5, and find 98.2% agree- LLMssuchasGemini-3-ProandGPT-5,factualencod- ment. Theremainingdisagreementsprimarilyinvolve ingisnearsaturation,with95-98%offactsencoded. OTHERorPARTIALLYlabels,whichareexcludedfrom Yetthesemodelsfailtodirectlyrecall26–34%ofthe ouranalysis;thus,ourresultsarenotsensitivetothe facts,or11–12%evenwiththinking. Accordingly,re- choiceofunderlyingLLM. callfailuresaccountformorethan70%ofGPT-5.2’s errorsandalargershareinstrongermodels,suggest- 6 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality ingrecallisindeedabottleneck. frontierLLMs. ForsmallermodelslikeGemma3-4B, encodinggapsaremuchlarger;thisisconsistentwith Scalingimprovesencodingmorethanrecall. | | | | | | | The | our broader | finding | that | scaling | primarily | improves | | ---------------------------------------- | --- | --- | --- | --- | --- | -------- | ----------- | ------- | ---- | ------- | --------- | -------- | | Gemma3modelfamilyillustratesthisclearly. | | | | | | Increas- | | | | | | | encoding. | ing model | capacity | and | size | from 1B | to 27B | param- | | | | | | | | --------- | -------- | --- | ---- | ------- | ------ | ------ | --- | --- | --- | --- | --- | --- | Reframingthelong-tailproblem. eters reduces encoding failures from 85% to 23%. Priorworkestab- However,asmodelscaleincreases,agrowingfraction lishedthatLLMsstrugglewithlong-tail(rare)facts, oferrorsshiftsfromencodingfailurestorecallfailures, typicallyinterpretedasacapacitylimitation(Kandpal peakingat33%(withthinking)and40%(without). etal.,2023;Mallenetal.,2023;Sunetal.,2024). Our Inthisfamily,scalingincreaseswhatthemodelstores; results suggest a complementary picture: rare facts areencoded;however,recallingthemisthechallenge. itdoesnotnecessarilyimprovewhatitcanaccessat inferencetime. Currently,thebottleneckhasshiftedfromknowledge | | | | | | | | acquisition | to utilization. | | Consequently, | | improving | | --- | --- | --- | --- | --- | --- | --- | ----------- | --------------- | --- | ------------- | --- | --------- | 5.2. WhyRecallFails factualityinfrontierLLMswilllikelyrequireinterven- tionsthatenhancerecall,notonlyscale. | Having | established | that | recall | is the | bottleneck, | we | | | | | | | | --------- | ------------- | --------- | ------------- | ---------- | ----------- | --------------- | -------------- | ----- | --- | ------- | --- | ------- | | now ask: | what | makes | it difficult? | | To answer | this, | | | | | | | | | | | | | | | Gemini-3-Flash | | | GPT-5 | | GPT-4.1 | | w e n e x | t f o c u s o | n e n c o | d e d f a | ct s . W e | s h o w | t h a t r e c a | l l 1 0 0 ∆ = | -0 .3 | | | | | | | | | | | | | | | ∆ = | + 4 . 7 | ∆ = | + 6 . 9 | r e m a i n s t ie d t o t he c o n d it io n s u n d e r w h ic h f a c t s w e r e 9 6 .2 96 .0 ∆ = -5 .8 ∆ = -8.5 | | | | | | | | | | | ∆ = -9.0 | | 9 2 . 0 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -------- | --- | ------- | a c qu i r e d . D u r i n g p r e - t r a i n i n g , f a c ts a r e e n c o u n t e r e d 8 8 .9 9 0 . 7 | | | | | | | | 8 0 | | 8 6 | .0 | 8 5 .1 | 8 5 .9 | | ------------------------------------------ | --- | --- | --- | --- | --- | ---- | --- | --- | --- | ------ | ------ | ------ | | | | | | | | | | 83 | .0 | 8 3 .0 | | | | inspecificcontexts,phrasings,andorderings. | | | | | | When | | | | | | 77.4 | 74.0 queriesdivergefromthetraining-timepatterns,recall 60 | becomesharder. | | Weexaminetwomanifestationsof | | | | | | | | | | | | -------------- | --- | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | this: factpopularityandquestiondirectionality. | | | | | | | | 40 Verify | Generate | Verify | Generate | Verify | Generate | | -------------- | --- | --- | ----- | --- | ------- | --- | --------- | -------- | ------ | -------- | ------- | -------- | | | | | | | | | (MCQA) | (CBQA) | | | | | | | | | | | | | | | | Direct | Reverse | | | Gemini-3-Flash | | | GPT-5 | | GPT-4.1 | | | | | | | | ∆=5.1 | | | | ∆=7.6 | | ∆=7.3 | | | | | | | | | --- | --- | --- | ----- | --- | ----- | --- | --- | --- | --- | --- | --- | --- | 1 0 0 ∆ = 21.4 F ig u r e 6 | T h e R e v e r s a l C u r s e : W e c o m p a r e d i r e c t | 99.5 | 94.5 | 98.4 | | ∆=24.9 | 99.1 | ∆=19.5 | | | | | | | | ---- | ---- | ---- | ---- | ------ | ---- | ------ | --- | --- | --- | --- | --- | --- | | 8 0 | | | 90.7 | | 91.8 | | | | | | | | 8 4 .7 re ca l l ra te s ( kn o w i n g e n c o d e d f a ct s wi t h ou t t h in k i n g ) | | | | | 77.8 | | 78.7 | | | | | | | | --- | --- | ---- | --- | ---- | --- | ---- | -------------------------------------------------- | --- | --- | --- | --- | --- | | 60 | | | | | | | ondirectandreversequestionsacrosstwotasks:verifi- | | | | | | | | | 63.3 | | | | 59.2 | | | | | | | | | | | | 52.9 | | | cation(multiple-choice)andgeneration(closed-book). | | | | | | 40 | 20 | | | | | | | TheΔdenotesthegapbetweenthedirectandreverse | | | | | | | --- | --- | --- | --- | --- | --- | --- | ------------------------------------------- | ------------------------------------ | --- | --- | --- | --- | | | | | | | | | settings. | WefindthatLLMshandlereversequestions | | | | | 0 | Encodes | Recalls | Encodes | | Recalls | Encodes | Recalls | | | | | | | | ------- | ---------------------- | ------- | --- | --------------------- | ------- | ------- | ----------- | --------------- | --- | ------------ | --- | ----------- | | | | | | | | | effectively | in verification | | but struggle | in | generation. | | | High Popularity (>80%) | | | Low Popularity (<20%) | | | | | | | | | Figure16presentsresultsforallLLMs. Fact Popularity: | Figure | 5 | | | | We compare | | two popu- | | | | | | | | ------ | --- | --- | --- | ---------- | --- | --------- | --- | --- | --- | --- | --- | --- | Reversequestionsareverifiablebuthardtorecall. | larity tiers | (bottom | 20% | vs. | top 20%) | in | terms of | | | | | | | | ------------ | ------- | --- | --- | -------- | --- | -------- | --- | --- | --- | --- | --- | --- | Figure6showsthatallLLMs(exceptforGemini-2.5- | encoding | rates | and direct | recall | rates | (knowing | en- | | | | | | | | -------- | ----- | ---------- | ------ | ----- | -------- | --- | --- | --- | --- | --- | --- | --- | Flash)exhibitlowerrecallratesforreversequestions | coded facts | without | thinking). | | The | Δ indicates | the | | | | | | | | ---------------- | ------- | ----------------------------- | --- | --- | ----------- | --- | ------------- | ----- | ------ | ------- | --------- | ---- | | | | | | | | | (e.g., GPT-5: | 82.9% | direct | vs. 74% | reverse). | This | | gapbetweentiers. | | Asshown,itisnarrowforencoding | | | | | | | | | | | asymmetryisconsistentwiththe“reversalcurse”doc- | butwideforrecall. | | Figure16presentsallLLMs. | | | | | | | | | | | | ----------------- | --- | ------------------------ | --- | --- | --- | --- | ----------------------------- | --- | --- | --- | -------------- | --- | | | | | | | | | umentedbyBerglundetal.(2024). | | | | Ourresultssug- | | Rarefactsareencodedbuthardtorecall. Wecom- gest a refinement of this view. If LLMs truly lack parefactsfromlow-popularityWikipediapages(bot- bidirectional knowledge, they should struggle with tom20%bypageviews,usedasaproxyforrarefacts) reversequestionsregardlessofformat. Totestthis,we withthosefromhigh-popularitypages(top20%). Fig- compareclosed-bookgenerationwithmultiple-choice ure5presents,forthetwotiers,thepercentageofen- questions, in which the correct answer is presented codedfactsand,amongencodedfacts,thepercentage among distractors (verification). In verification, re- that are directly recalled. Conditioning on encoded versequestionsarenoharderthandirectones: for9 factsisolatesrecallfromencoding,allowingustoas- models, reverseisactuallyeasier; fortheremaining sesswhetherrarefactsarehardertorecallevenwhen 4,performanceissimilar. Thisdissociationsuggests thatLLMsareawareofthebidirectionalassociationof | they are | encoded. | First, | we | find that | rare | facts are | | | | | | | | -------- | -------- | ------ | --- | --------- | ---- | --------- | --- | --- | --- | --- | --- | --- | encodedathighratesapproachingthoseofpopular thefact;theycanrecognizethecorrectanswerwhen | | | | encoding | gap | | | | | | | | | | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | facts. Second, while the between pop- presentedwithit. Theirfailureliesinrecallingitwhen ularandrarefactsismodest(e.g.,forGemini-3-Pro, thedirectiondoesnotmatchthetrainingdata. Inthis 99.5% vs. 94.5%, Δ=5), the recall gap is far larger view,thereversalcurseisarecallphenomenonrather (84.6%vs. 63.3%,Δ=21.3);thispatternholdsforall thanalackofbidirectionalassociation. 7 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality High Pop +Thinking Low Pop +Thinking Direct +Thinking Reverse +Thinking 100 ∆ ∆ T = = 2 1 1 2 .4 .5 ∆ ∆ T = = 2 1 1 1 .4 .7 ∆ ∆ T = = 2 1 4 4 .9 .5 ∆ ∆ T = = 2 1 2 6 .7 .6 ∆ ∆ T = = 1 1 9 6 .5 .0 90 ∆ 8 ∆ . T 6 = = 5 2 1 . 2 7 .0 .3 ∆ 6 ∆ . T 7 = = 5 2 1 . 0 8 .2 .3 1 ∆ 2 ∆ T . = 0 = 9 2 .0 .0 ∆ ∆ T = = 10 5 . . 6 3 ∆ 6 ∆ . T 9 = = 8 7 .5 .2 80 11.3 7.7 16.0 15.2 8.6 80 19.0 15.7 21.1 8.2 60 82.5 20.1 84.7 17.3 77.8 26.4 73.7 21.4 78.7 12.1 6 7 0 0 86.9 81.2 88.9 83.0 83.0 74.0 75.3 85.9 77.4 40 61.2 63.3 52.9 51.0 59.2 50 64.7 40 Gemini-3 Gemini-3 GPT-5 GPT-5.2 GPT-4.1 Gemini-3 Gemini-3 GPT-5 GPT-5 GPT-4.1 Pro Flash Pro Flash mini Figure7 | ThinkingandRecall: Weexaminetheimpactofthinkingonrecall(knowingencodedfacts). On theleft,wecomparetwopopularitytiers(bottom20%vs. top20%); Ontheright,wecomparedirectand reversequestions. ThepopularityordirectionalgapsaredenotedbyΔ(nothinking)andΔ𝑇 (withthinking). Asshown,thinkingnarrowsthegaps(Δ𝑇< Δ). Figure17presentsallLLMs. Phrasingisnotafactor. Ourdefinitionofknowsre- sponse diversity: thinking increases output variance, quirescorrectnessacrossphrasingsanddirections. We improvingthechancethatatleastonesampleiscor- verifythatphrasingdoesnotconfoundourresultsby rect (a sampling effect); (ii) Inference: thinking en- conducting hypothesis tests comparing the distribu- ablesmulti-stepderivationfromotherencodedfacts; tionsofperformanceunderhigh-verbatimandnatural (iii) Recall facilitation: thinking helps the model lo- phrasings. Across104hypothesistests(13LLMs,with cate and retrieve a specific encoded fact that direct andwithoutthinking,acrossfourtaskpairs)andaf- generationfailstoaccess. terFDRcorrection,wefindnosignificanteffects(see Our analysis in Appendix B.1 rules out response AppendixC.3fordetails). diversity as the main mechanism that improves fac- tualityforthinking-optimizedLLMs. Whenthinking 5.3. ThinkingasaRecoveryMechanism is enabled, models become more consistent in their correctness, producing a higher fraction of correct Wenowturntothequestionofwhatenablestherecov- samplesforthesamequestion. Thispatternisincom- eryofotherwiseinaccessibleknowledge. Tothisend, patiblewithincreasedoutputvariance. Wetherefore weexaminethepotentialofthinkingtofillthisrole. believethatthinkingprimarilyfacilitatesrecallofen- Thinking-optimizedLLMssuchasGemini-3,Gemini- coded knowledge. Figure 8 provides evidence that 2.5,andGPT-5modelsallocateadditionalcomputa- supportsthis: thinkingrecovers40–65%ofencoded tiontothinkingbydefault;fornon-thinkingmodels (Gemma3 and GPT-4.1), we elicit similar behavior throughCoTprompting. % of Facts that Become Known with Thinking 70% Thinking targets recall weaknesses. Figure 7 60% G G i i v v e e n n t t h h e e y y a a r r e e e N n O c T o d e e n d coded presentstheimprovementfromthinkingonencoded 50% facts (i.e., recall gains), broken down by popular- 40% ity and directionality. Thinking yields larger gains preciselywheredirectrecallisweakest. ForGemini- 30% 3-Pro, thinking improves the recall of rare facts by 20% 20.1points,versus11.3forpopularfacts;forreverse 10% questions, GPT-5 gains 19 points, versus 12 for di- 0% r t d h e ir c e e t c p q t o u io p e n u s a t l i a l o i r t n i y t s y g . a C g p a o p n (f s r ( e o f q r m o u m e Δ n = Δ tl 9 y = , t t 2 o h 1 i 2 n .4 ) k . i t n o g 1 n 2 a .5 rr ) ow an s d bo th th e Gemm G a e 3 m -1 G m b e a m 3- m 4 G b a e 3 m -1 m 2 G b a3 P - T 2 -4 7 . b 1 G -m em i G n i i n P i T - G - 2 4 . e . 5 1 m -F in la i- s 2 h .5-P G ro P G T G - P 5 T e . - 2 m 5- i m ni i - G n 3 i - e F m la in sh i-3-Pro GPT-5 Figure 8 | Recovering Depends on Encoding: We Whydoesthinkinghelp? Thinkinggainsareremark- reportthepercentageofnot-knownfactsthatbecome able because our facts are single-hop; they require knownwiththinking,conditionedonwhetherthefact nocomplexmulti-stepreasoning. Whyshouldthink- isencoded(red)ornot(yellow).Thinkingrecovers40– inghelprecallasimplefactsuchas“WheredidOasis 65%ofencodedfactsinthinking-optimizedLLMs,but play their first show?” We consider three candidate only5–15%ofnon-encodedfacts. Thisdependence mechanisms(withadditionaldetailsinAppendixB.1) onencodingisconsistentwiththerecallfacilitation throughwhichthinkingimprovesperformance: (i)Re- mechanism. 8 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality but not directly known facts in thinking-optimized temsinwhichtheconditionsforstoringinformation LLMs,butonly5–20%ofnon-encodedfacts. Thisde- divergefromtheconditionsforrecallingit. pendenceonencodingstatusisexpectedunderrecall | facilitation. | It is | also partially | expectedunder | | infer- | | | | | | | | ------------------------------------- | ----------- | -------------- | -------------- | --- | -------- | ---------- | ---- | --- | --- | --- | --- | | | | | | | | 6. Related | Work | | | | | | ence, since | non-encoded | | facts may lack | the | related | | | | | | | | encodedpremisesthatinferencerequires. | | | | | Nonethe- | | | | | | | less,ourfactsaresingle-hop,andcomplexdeductive EncodingandMemorization: Ourbehavioraldefini- reasoningisunnecessarytoderivethecorrectanswer. tionofencodingconnectstoworkonmemorization Forsingle-hopfacts,recallfacilitationisthemorepar- inLLMs(Carlinietal.,2023;Huangetal.,2024;Lu simoniousexplanation: themodelalreadystoresthe etal.,2024;Mentaetal.,2025;Wangetal.,2025a).In answerandneednotderiveitfromotherknowledge. particular,ourencodingcriterionalignswiththeprob- Wethereforeconcludethatthinkingprimarilyhelps abilisticframeworkofHayesetal.(2025). Memoriza- modelsaccessknowledgethattheyhavealreadyen- tionworktypicallyfocusesonprivacyordataleakage coded. Gekhmanetal.(2026)concurrentlyshowthat concerns,whereasweuseencoding-via-memorization thinkingexpandsparametricrecallthroughcomputa- forknowledgeprofiling. tionalbufferingandpriming. | | | | | | | LatentKnowledge: | Aseparatelineofworkusesinter- | | | | | | ------------------ | --- | ---------------------------- | --- | --- | --- | ---------------- | ----------------------------- | --- | --- | --- | --- | | Thecostofthinking. | | Whilethinkingimprovesrecall, | | | | | | | | | | nalrepresentationstoarguethatLLMspossess“latent itiscomputationallyexpensive. Acrossfrontiermod- knowledge” or “hidden knowledge”: facts they en- els,10–20%offactsareaccessibleonlyviathinking. code but fail to express in generation (Burns et al., Determiningexactlywhentoinvokethinkingisfun- 2023; Gekhman et al., 2025; Gottesman and Geva, damentally a metacognitive problem, as it requires 2024;Kadavathetal.,2022;Orgadetal.,2025;Simhi | the model | to recognize | | when direct | recall | is insuffi- | | | | | | | | --------- | ------------ | --- | ----------- | ------ | ----------- | -------------- | ------------ | --- | ------ | ------- | --------- | | | | | | | | et al., 2025). | One possible | | reason | is that | the post- | cient before triggering more intensive computation training phase may introduce new knowledge that Yonaetal.(2026). Improvingdirectrecallcouldhelp undermines the use of the knowledge acquired dur- side-step this issue until models develop improved ingpre-training(Gekhmanetal.,2024;Kaplanetal., metacognitiveabilities. 2026). Ourresultsofferacomplementarybehavioral perspective:whatappearstobelatentknowledgemay 5.4. ConnectionstoHumanCognition reflectcontext-dependentrecall. Factssurfacewhen queriesmatchtrainingconditions(asinourencoding Theroleofthinkinginsupportingrecallresonateswith tasks)butbecomehardertoaccesswhentheydiverge well-studiedphenomenainhumanmemory. Thetip- (asinourknowledgetasks). of-the-tonguephenomenondescribesstatesinwhicha | | | | | | | The Reversal | Curse: | Berglund | et al. | (2024) | docu- | | --- | --- | --- | --- | --- | --- | ------------ | ------ | -------- | ------ | ------ | ----- | personisconfidenttheyknowsomethingbutcannot immediately produce it (Brown and McNeill, 1966; mented that LLMs trained on “A is B” often fail to | | | | | | | answer“WhatisB?”. | Subsequentworkhasattributed | | | | | | --------- | ------ | ---- | ---------- | ----- | -------- | ----------------- | --------------------------- | --- | --- | --- | --- | | Schwartz, | 2002). | Such | states are | often | resolved | | | | | | | throughdeliberateeffort: thinkingofrelatedconcepts thistotheautoregressiveobjective(Guoetal.,2024; Kitounietal.,2024;Lvetal.,2024),trainingdynamics ormentallyretracingthecontextinwhichtheinfor- (Zhuetal.,2024),ordataasymmetries(Golovneva | mationwasencountered. | | | Theparalleltoourfindings | | | | | | | | | | --------------------- | --- | --- | ------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | issuggestive: encoded-but-not-knownfactsinLLMs etal.,2024;Heetal.,2025),withproposedsolutions | | | | | | | focusing | on pre-training | modifications | | (Golovneva | | | ------------- | --- | ---------- | --------- | -------- | --- | -------- | --------------- | ------------- | --- | ---------- | --- | | may represent | a | functional | analogue, | in which | the | | | | | | | informationisstoredbutthequeryisinsufficientto etal.,2024;Yuetal.,2025)orarchitecturalchanges | | | | | | | (WangandSun,2025). | | Bycomparingverificationto | | | | | -------------- | --------------------------------- | --- | --- | --- | --- | ------------------ | --- | ------------------------- | --- | --- | --- | | triggerrecall. | Thinkingprovidestheadditionalcom- | | | | | | | | | | | generation,weshowthatLLMscanrecognizereverse putationthat,likedeliberateeffortinhumans,bridges | thegap. | | | | | | answers | they cannot | produce, | suggesting | | the curse | | ------- | --- | --- | --- | --- | --- | ---------- | ---------------- | -------- | ---------- | --- | ---------- | | | | | | | | reflects a | recall asymmetry | rather | than | a | failure to | Separately, our verification results (in multiple- learn bidirectional associations. Thinking can miti- choicesettings,§5.2)echothefeeling-of-knowingphe- gatethecurse,providingapathtoimprovementthat nomenon: peopleoftenpredicttheywillrecognizean priorworkhasnotexplored. answerevenwhentheycannotrecallit(Hart,1965; NelsonandNarens,1990). Thedissociation,inwhich ThinkingandFactuality: TheeffectofCoTprompt- LLMscanverify(recognize)answerstoreverseques- ing (Wang et al., 2024b; Wei et al., 2022) and tionsbutcannotgenerate(recall)them,mirrorsthis reasoning-optimizedmodels(Marjanovicetal.,2025) asymmetryinhumanmemory. Whilewedonotclaim onfactualityremainscontested. Somestudiesreport thatLLMsimplementhuman-likecognitiveprocesses, improvedaccuracyviaincreaseddiversity(Wangetal., thefunctionalparallelssuggestthatthegapbetween 2023;Yangetal.,2025),whileothersfindhigherhal- encodingandrecallmaybeageneralpropertyofsys- lucinationrates(Wangetal.,2024a;Yaoetal.,2025) 9 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality orattributegainstomulti-hopreasoning(Zhangetal., Our study has several limitations worth noting. 2025). Effects on consistency are similarly mixed, WikiProfile is scoped to Wikipedia, an encyclopedic withsomestudiesreportingimprovements(Kimetal., domainofrelativelyprominentfacts,andourfindings 2025;Wuetal.,2025a)andothersreportingdegra- maynotgeneralizetoothercorpora. Ouroperational dation(Chengetal.,2025a,b). Acontemporaneous definitionofknowledgeislikewisebounded: wemea- work by Gekhman et al. (2026) also demonstrated surerecallalongtwoaxes,directionalityandphrasing, thatthinkingcanexpandtheboundaryofparametric forsingle-hopfacts,anddonotclaimtoexhaustively knowledge recall, identifying two key mechanisms characterizeallformsinwhichafactcanbequeried. underlyingthiseffect: acomputationalbufferandfac- Finally,fullyprofilingafrontiermodelonWikiProfile tualpriming. Ourworkoffersaunifyingperspective: costs±$500,thoughthiscanbereducedsubstantially thinkingfunctionsasarecallmechanism,notjusta byomittingmultiple-choicequestionsorusingfewer reasoningmechanism. Itseffectismostpronounced samplesperquestion. underchallengingconditions,suchasrarefactsand | reversequestions. | | | | | | | Impact Statement | | | | | | ----------------- | --- | --- | --- | --- | --- | --- | ---------------- | --- | --- | --- | --- | Thispaperpresentsworkaimedatunderstandingfac- 7. Discussion | | | | | | | | tualerrorsinLLMs. | | Ourfindingscouldinformdeploy- | | | | --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ----------------------------- | --- | --- | mentdecisionsbyclarifyingwhenmodelsarelikely Inthiswork,weproposeaframeworkforknowledge tofail(e.g.,withrarefactsorreversequestions)and profilingandapplyittoprofile13LLMsacrossabench- wheninference-timecomputationmaybebeneficial. | markofnaturallyoccurringfactsfromWikipedia. | | | | | | Our | | | | | | | ------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | Wedonotforeseenegativesocietalconsequencesspe- | results | point to recall, | not | encoding, | | as the primary | | cifictothiswork. | | | | | | -------------------------------------- | ---------------- | --- | --------- | --- | -------------- | --- | ---------------- | --- | --- | --- | --- | | bottleneckforfactualityinfrontierLLMs. | | | | | Acrossthe | | | | | | | modelsweevaluated,encodingisapproachingsatura- References tion;thestrongestLLMsencodeover95%offactsin | ourbenchmark. | Thechallengeliesinaccessingwhat | | | | | | | | | | | | ------------- | ------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | has been learned. Recall failures are not random; Y.BenjaminiandD.Yekutieli. Thecontrolofthefalse theyconcentrateonrarefactsandreversequestions, discoveryrateinmultipletestingunderdependency. conditionswherequeriesdivergefromtraining-time Annalsofstatistics,pages1165–1188,2001. | contexts. | ThispatternsuggeststhatLLMsstorefacts | | | | | | | | | | | | --------- | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | L.Berglund,M.Tong,M.Kaufmann,M.Balesni,A.C. inwaysthatremaintiedtotheiracquisition,making | | | | | | | | Stickland, | T. Korbak, | and | O. Evans. The | reversal | | --- | --- | --- | --- | --- | --- | --- | ---------- | ---------- | --- | ------------- | -------- | recallsensitivetosurface-levelquerycharacteristics. | | | | | | | | curse: Llmstrainedon"aisb"failtolearn"bisa". | | | | | | --- | --- | --- | --- | --- | --- | --- | -------------------------------------------- | --- | --- | --- | --- | Thisshiftfromencodingtorecallhasimplications InTheTwelfthInternationalConferenceonLearning beyondshort-formfactuality. Distinguishingbetween Representations, ICLR 2024, Vienna, Austria, May encoding and recall failures provides a framework 7-11, 2024. OpenReview.net, 2024. URL https: that could extend to long-form generation, multi- //openreview.net/forum?id=GPKTIktA0k. stepreasoning,andotherdomainswhereaccessing | | | | | | | | R.BrownandD.McNeill. | | The“tipofthetongue”phe- | | | | --------------------------- | --- | --- | --- | -------------------- | --- | --- | -------------------- | --- | ----------------------- | --- | --- | | learnedknowledgeiscritical. | | | | Intheageofretrieval- | | | | | | | | JournalofVerbalLearningandVerbalBe- | augmentedgeneration(RAG)andtool-usingagents, | | | | | | | nomenon. | | | | | | -------------------------------------------- | --- | --------------- | --- | --------- | --- | ------- | ------------------------- | --- | --- | ------------------- | --- | | | | | | | | | havior,5(4):325–337,1966. | | | doi: 10.1016/S0022- | | | it is tempting | to | view parametric | | knowledge | | as sec- | | | | | | 5371(66)80040-3. | ondary,somethingRAGcancompensatefor. | | | | | Butpara- | | | | | | | | ------------------------------------ | --- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | metricknowledgeisessentialforfluency,speed,and C.Burns,H.Ye,D.Klein,andJ.Steinhardt. Discover- | integration | across | contexts. | Improving | | how models | | | | | | | | ----------- | ------ | --------- | --------- | --- | ---------- | --- | --- | --- | --- | --- | --- | inglatentknowledgeinlanguagemodelswithout accesswhattheyalreadyencodeis,therefore,amean- | | | | | | | | supervision. | InICLR,2023. | | | | | --- | --- | --- | --- | --- | --- | --- | ------------ | ------------ | --- | --- | --- | ingfultarget. N.Carlini,D.Ippolito,M.Jagielski,K.Lee,F.Tramèr, | The gap | between | encoding | | and | recall might | be | | | | | | | ------- | ------- | -------- | --- | --- | ------------ | --- | ------------- | ----------- | --- | ------------ | ------ | | | | | | | | | and C. Zhang. | Quantifying | | memorization | across | bridgedthroughinterventionduringpre-training,for InICLR,2023. neurallanguagemodels. | example, | by augmenting | | data | with | self-generated | | | | | | | | -------- | ------------- | --- | ---- | ---- | -------------- | --- | --- | --- | --- | --- | --- | question-answer pairs Lin et al. (2025), or during Y.Chen,P.Cao,Y.Chen,K.Liu,andJ.Zhao. Knowl- post-training,asrecentstudiessuggestthatalignment | | | | | | | | edge localization: | | Mission | not accomplished? | en- | | --- | --- | --- | --- | --- | --- | --- | ------------------ | --- | ------- | ----------------- | --- | teaches models how to better utilize knowledge ac- ter query localization! In The Thirteenth Inter- | | | | | | | | national | Conference | on | Learning Representations, | | | ------ | ------------------- | --- | ------- | --- | -------------- | --- | -------- | ---------- | --- | ------------------------- | --- | | quired | during pre-training | | Gekhman | | et al. (2024); | | | | | | | Linetal.(2024a);Zhouetal.(2023). Ourfindings ICLR 2025, Singapore, April 24-28, 2025. Open- also suggest that inference-time techniques, such as Review.net,2025. URLhttps://openreview.net/ forum?id=tfyHbvFZ0K. thinking,canbeeffectiveinaidingparametricrecall. 10 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality J.Cheng,T.Su,J.Yuan,G.He,J.Liu,X.Tao,J.Xie, Z.Gekhman,G.Yona,R.Aharoni,M.Eyal,A.Feder, and H. Li. Chain-of-thought prompting obscures R. Reichart, and J. Herzig. Does fine-tuning llms hallucinationcuesinlargelanguagemodels:Anem- on new knowledge encourage hallucinations? In CoRR,abs/2506.17088,2025a. piricalevaluation. Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, doi: 10.48550/ARXIV.2506.17088. URL https: Proceedings of the 2024 Conference on Empirical //doi.org/10.48550/arXiv.2506.17088. Methods in Natural Language Processing, EMNLP | | | | | | | | 2024, Miami, | | FL, USA, | November | 12-16, | 2024, | | ----------------------------- | --- | --- | --- | -------------- | --- | --- | ------------ | --- | -------- | -------- | ------ | ----- | | X.Cheng,J.Li,X.Zhao,andJ.Wen. | | | | Thinkmore,hal- | | | | | | | | | pages7765–7784.AssociationforComputational | lucinateless: | Mitigatinghallucinationsviadualpro- | | | | | | | | | | | | | ------------- | ----------------------------------- | --- | --- | --- | --- | --- | ----------------- | --- | ---- | ----------------------- | --- | --- | | | | | | | | | Linguistics,2024. | | doi: | 10.18653/V1/2024.EMNLP- | | | cessoffastandslowthinking.InW.Che,J.Nabende, | | | | | | | | MAIN.444. | URL | https://doi.org/10.18653/v1/ | | | | | --- | --- | --- | --- | --- | --- | --- | --------- | --- | ---------------------------- | --- | --- | --- | E.Shutova,andM.T.Pilehvar,editors,Findingsof 2024.emnlp-main.444. | the Association | | for Computational | | Linguistics, | | ACL | | | | | | | | --------------- | --- | ----------------- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- | 2025, Vienna, Austria, July 27 - August 1, 2025, Z.Gekhman,E.Ben-David,H.Orgad,E.Ofek,Y.Be- pages7979–7990.AssociationforComputational linkov,I.Szpektor,J.Herzig,andR.Reichart.Inside- Linguistics, 2025b. URL https://aclanthology. out: Hidden factual knowledge in llms. CoRR, org/2025.findings-acl.417/. | | | | | | | | abs/2503.15299, | | 2025. | doi: | 10.48550/ARXIV. | | | ------------ | --- | ------- | ---------------- | --- | --- | ----- | --------------- | --- | ----- | ------------------------- | --------------- | --- | | | | | | | | | 2503.15299. | | URL | https://doi.org/10.48550/ | | | | G. Comanici, | E. | Bieber, | M. Schaekermann, | | I. | Pasu- | | | | | | | arXiv.2503.15299. pat,N.Sachdeva,I.S.Dhillon,M.Blistein,O.Ram, D.Zhang,E.Rosen,L.Marris,S.Petulla,C.Gaffney, Z.Gekhman,R.Aharoni,E.Ofek,M.Geva,R.Reichart, | A. Aharoni, | N. | Lintz, | T. C. | Pais, H. | Jacobsson, | | | | | | | | | ----------- | --- | ------ | ----- | -------- | ---------- | --- | --- | --- | --- | --- | --- | --- | andJ.Herzig.Thinkingtorecall:Howreasoningun- I. Szpektor, N. Jiang, K. Haridasan, A. Omran, locksparametricknowledgeinllms. arXivpreprint N. Saunshi, D. Bahri, G. Mishra, E. Chu, T. Boyd, arXiv:2603.09906,2026. | B. Hekman, | | A. Parisi, | C. Zhang, | K. | Kawintira- | | | | | | | | | ---------- | --- | ---------- | --------- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | non, T. Bedrax-Weiss, O. Wang, Y. Xu, O. Purkiss, O. Golovneva, Z. Allen-Zhu, J. Weston, and U. Mendlovic, I. Deutel, N. Nguyen, A. Langley, S. Sukhbaatar. Reverse training to nurse the re- CoRR, F. Korn, L. Rossazza, A. Ramé, S. Waghmare, versal curse. abs/2403.13799, 2024. doi: H.Miller,N.Byrd, A.Sheshan,R.H.S.Bhardwaj, 10.48550/ARXIV.2403.13799. URL https://doi. P. Janus, T. Rissa, D. Horgan, S. Silver, A. Wahid, org/10.48550/arXiv.2403.13799. | S. Brin, | Y. Raimond, | | K. Kloboves, | C. | Wang, | N. B. | | | | | | | | -------- | ----------- | --- | ------------ | --- | ----- | ----- | --- | --- | --- | --- | --- | --- | Gundavarapu,I.Shumailov,B.Wang,M.Pajarskas, D.GottesmanandM.Geva. Estimatingknowledgein largelanguagemodelswithoutgeneratingasingle J.Heyward,M.Nikoltchev,M.Kula,H.Zhou,Z.Gar- | | | | | | | | token. In | Y. Al-Onaizan, | | M. | Bansal, | and Y. Chen, | | --- | --- | --- | --- | --- | --- | --- | --------- | -------------- | --- | --- | ------- | ------------ | rett,S.Kafle,S.Arik,A.Goel,M.Yang,J.Park,K.Ko- editors,Proceedingsofthe2024ConferenceonEmpir- | jima, P. | Mahmoudieh, | | K. Kavukcuoglu, | | G. | Chen, | | | | | | | | -------- | ----------- | --- | --------------- | --- | --- | ----- | --- | --- | --- | --- | --- | --- | icalMethodsinNaturalLanguageProcessing,EMNLP D.Fritz,A.Bulyenov,S.Roy,D.Paparas,H.Shem- | | | | | | | | 2024, Miami, | | FL, USA, | November | 12-16, | 2024, | | --- | --- | --- | --- | --- | --- | --- | ------------ | --- | -------- | -------- | ------ | ----- | tov,B.Chen,R.Strudel,D.Reitter,A.Roy,A.Vlasov, pages3994–4019.AssociationforComputational C.Ryu,C.Leichner,H.Yang,Z.Mariet,D.Vnukov, | | | | | | | | Linguistics,2024. | | doi: | 10.18653/V1/2024.EMNLP- | | | | --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ---- | ----------------------- | --- | --- | T.Sohn,A.Stuart,W.Liang,M.Chen,P.Rawlani, https://doi.org/10.18653/v1/ | C. Koh, | J. Co-Reyes, | | G. Lai, | P. Banzal, | D. | Vytin- | MAIN.232. | URL | | | | | | ------- | ------------ | --- | ------- | ---------- | --- | ------ | --------- | --- | --- | --- | --- | --- | 2024.emnlp-main.232. | iotis, J. | Mei, | and M. | Cai. Gemini | | 2.5: Pushing | | | | | | | | | --------- | ---- | ------ | ----------- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | thefrontierwithadvancedreasoning,multimodal- Q.Guo,R.Wang,J.Guo,X.Tan,J.Bian,andY.Yang. | ity, long | context, | and | next generation | | agentic | ca- | | | | | | | | -------------------------- | -------- | --------------- | --------------- | --- | ------------ | ---- | ---------------------- | -------- | ----- | ------------ | -------------- | ------------ | | | CoRR, | | | | | | Mitigating | reversal | curse | in | large language | mod- | | pabilities. | | abs/2507.06261, | | | 2025. | doi: | | | | | | | | | | | | | | | els via semantic-aware | | | permutation | | training. In | | 10.48550/ARXIV.2507.06261. | | | | URL | https://doi. | | | | | | | | | | | | | | | | L. Ku, A. | Martins, | and | V. Srikumar, | editors, | Find- | org/10.48550/arXiv.2507.06261. ingsoftheAssociationforComputationalLinguistics, Y. Deng, W. Zhang, Z. Chen, and Q. Gu. Rephrase ACL2024,Bangkok,Thailandandvirtualmeeting, August11-16,2024,pages11453–11464.Associa- | andrespond: | | Letlargelanguagemodelsaskbetter | | | | | | | | | | | | ----------- | --- | ------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | questionsforthemselves. CoRR,abs/2311.04205, tionforComputationalLinguistics,2024. doi: 10. | | | | | | | | 18653/V1/2024.FINDINGS-ACL.680. | | | | | URLhttps: | | ----- | ------------------------------- | --- | --- | --- | --- | --- | ------------------------------- | --- | --- | --- | --- | --------- | | 2023. | doi: 10.48550/ARXIV.2311.04205. | | | | | URL | | | | | | | //doi.org/10.18653/v1/2024.findings-acl.680. https://doi.org/10.48550/arXiv.2311.04205. Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, L. Haas, G. Yona, G. D’Antonio, S. Goldshtein, and E.H.Hovy,H.Schütze,andY.Goldberg. Measuring D. Das. Simpleqa verified: A reliable factual- andimprovingconsistencyinpretrainedlanguage itybenchmarktomeasureparametricknowledge. models. Trans.Assoc.Comput.Linguistics,9:1012– CoRR, abs/2509.07968, 2025. doi: 10.48550/ 1031,2021. doi: 10.1162/TACL\_A\_00410. URL ARXIV.2509.07968. URL https://doi.org/10. https://doi.org/10.1162/tacl_a_00410. 48550/arXiv.2509.07968. 11 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality E. Habba, O. Arviv, I. Itzhak, Y. Perlitz, E. Bandel, A. Holtzman and C. Tan. Prompting as scientific CoRR, L.Choshen,M.Shmueli-Scheuer,andG.Stanovsky. inquiry. abs/2507.00163, 2025. doi: 10. DOVE:Alarge-scalemulti-dimensionalpredictions 48550/ARXIV.2507.00163. URLhttps://doi.org/ 10.48550/arXiv.2507.00163. | dataset | towards | meaningful | | LLM | evaluation. | In | | | | | | | | | ------- | ------- | ---------- | --- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- | W.Che,J.Nabende,E.Shutova,andM.T.Pilehvar, editors,FindingsoftheAssociationforComputational J.Huang,D.Yang,andC.Potts. Demystifyingverba- Linguistics, ACL 2025, Vienna, Austria, July 27 - tim memorization in large language models. In August 1, 2025, pages 11744–11763. Association Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, | | | | | | | | Proceedings | of | the 2024 | Conference | | on | Empirical | | --- | --- | --- | --- | --- | --- | --- | ----------- | --- | -------- | ---------- | --- | --- | --------- | https: | for Computational | | | Linguistics, | 2025. | URL | | | | | | | | | | ----------------- | --- | --- | ------------ | ----- | --- | --- | ------- | ---------- | --- | -------- | ----------- | --- | ----- | | | | | | | | | Methods | in Natural | | Language | Processing, | | EMNLP | //aclanthology.org/2025.findings-acl.611/. 2024,Miami,FL,USA,November12-16,2024,pages P.Haller,M.Ibrahim,P.Kirichenko,L.Sagun,andS.J. 10711–10732.AssociationforComputationalLin- | | | | | | | | guistics, | 2024. | doi: | 10.18653/V1/2024.EMNLP- | | | | | ---------------------------- | --- | --- | --- | ------------------ | --- | --- | --------- | ----- | ---- | ----------------------- | --- | --- | --- | | Bell. LLMknowledgeisbrittle: | | | | Truthfulnessrepre- | | | | | | | | | | https://doi.org/10.18653/v1/ | sentations | rely | on | superficial | resemblance. | | CoRR, | MAIN.598. | URL | | | | | | | ---------- | ---- | --- | ----------- | ------------ | --- | ----- | --------- | --- | --- | --- | --- | --- | --- | 2024.emnlp-main.598. | abs/2510.11905, | | 2025. | | doi: 10.48550/ARXIV. | | | | | | | | | | | --------------- | --- | ----- | ------------------------- | -------------------- | --- | --- | ----------------------------------- | --- | --- | --- | --- | --- | ------ | | 2510.11905. | | URL | https://doi.org/10.48550/ | | | | | | | | | | | | | | | | | | | Z.Jiang,F.F.Xu,J.Araki,andG.Neubig. | | | | | | Howcan | arXiv.2510.11905. | | | | | | | | | | | | | | Trans. As- | | --------- | ----------------------------------- | --- | --- | --- | --- | --- | ------------ | ---- | ------------ | ---------- | --- | ----- | ---------- | | | | | | | | | we know | what | language | models | | know. | | | | | | | | | | soc. Comput. | | Linguistics, | 8:423–438, | | 2020. | doi: | | J.T.Hart. | Memoryandthefeeling-of-knowingexpe- | | | | | | | | | | | | | 10.1162/TACL\_A\_00324.URLhttps://doi.org/ | rience. | Journal | of | Educational | Psychology, | | 56(4): | | | | | | | | | ------- | ------- | --- | ----------- | ----------- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | 10.1162/tacl_a_00324. | 208–216,1965. | | doi: | 10.1037/h0022263. | | | | | | | | | | | | ------------------------------------- | ------------ | ------------------ | ----------------- | --------------------- | ---------- | ------ | --------------------------------------------- | --------- | ------------------ | ----------- | ------- | ------------- | -------- | | | | | | | | | M.Joshi,E.Choi,D.S.Weld,andL.Zettlemoyer. | | | | | | Triv- | | P. Hase, | M. Bansal, | B. | Kim, | and A. Ghandeharioun. | | | | | | | | | | | | | | | | | | iaqa: Alargescaledistantlysupervisedchallenge | | | | | | | | Does | localization | | inform | editing? | surprising | | | | | | | | | | | | | | | | | datasetforreadingcomprehension. | | | | | InR.Barzilay | | | differences | | in causality-based | | localization | | vs. | | | | | | | | | | | | | | | | and M. | Kan, | editors, | Proceedings | | of the | 55th An- | | knowledge | | editing | in language | | models. | In | | | | | | | | | | | | | | | | nual Meeting | | of the Association | | for | Computational | | | A. Oh, | T. | Naumann, | A. | Globerson, | K. Saenko, | | | | | | | | | | | | | | | | | Linguistics, | ACL2017, | | Vancouver, | | Canada, | July30 | | M. Hardt, | | and S. | Levine, | editors, | Advances | in | | | | | | | | | | | | | | | | - August | 4, Volume | 1: | Long | Papers, | pages | 1601– | | NeuralInformationProcessingSystems36: | | | | | | Annual | | | | | | | | Conference on Neural Information Processing 1611. Association for Computational Linguistics, | | | | | | | | 2017. | doi: 10.18653/V1/P17-1147. | | | | URL | https: | | ------- | ----- | ------- | --- | --------- | -------- | --- | ----- | -------------------------- | --- | --- | --- | --- | ------ | | Systems | 2023, | NeurIPS | | 2023, New | Orleans, | LA, | | | | | | | | //doi.org/10.18653/v1/P17-1147. | USA, | December | 10 | - 16, | 2023, | | | | | | | | | | | ---- | -------- | --- | ----- | ----- | ----- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | 2023. | URL | | | | | | | | http://papers.nips.cc/paper_files/paper/ | | | | | | | | S. Kadavath, | T. | Conerly, | A. | Askell, | T. | Henighan, | | --- | --- | --- | --- | --- | --- | --- | ------------ | --- | -------- | --- | ------- | --- | --------- | 2023/hash/3927bbdcf0e8d1fa8aa23c26f358a281- | | | | | | | | D. Drain, | E. Perez, | N. | Schiefer, | Z. | Hatfield-Dodds, | | | --- | --- | --- | --- | --- | --- | --- | --------- | --------- | --- | --------- | --- | --------------- | --- | Abstract-Conference.html. | | | | | | | | N. DasSarma, | | E. Tran-Johnson, | | | S. Johnston, | S. E. | | --- | --- | --- | --- | --- | --- | --- | ------------ | --------- | ---------------- | ------- | --- | ------------ | -------- | | | | | | | | | Showk, | A. Jones, | N. | Elhage, | T. | Hume, | A. Chen, | J.Hayes,M.Swanberg,H.Chaudhari,I.Yona,I.Shu- Y.Bai,S.Bowman,S.Fort,D.Ganguli,D.Hernan- mailov,M.Nasr,C.A.Choquette-Choo,K.Lee,and | | | | | | | | dez, J. | Jacobson, | J. | Kernion, | S. | Kravec, | L. Lovitt, | | --------------------------------- | --- | ------------------------------- | -------- | ----------- | ------------- | ------ | ----------- | --------- | ---------- | ---------- | ---------- | -------- | ---------- | | A.F.Cooper. | | Measuringmemorizationinlanguage | | | | | | | | | | | | | | | | | | | | K. Ndousse, | | C. Olsson, | | S. Ringer, | D. | Amodei, | | modelsviaprobabilisticextraction. | | | | | InL.Chiruzzo, | | | | | | | | | | | | | | | | | T. Brown, | J. | Clark, | N. Joseph, | | B. Mann, | S. Mc- | | | | | | Proceedings | | of the | | | | | | | | | A. Ritter, | and | L. Wang, | editors, | | | | | | | | | | | | | | | | | | | Candlish, | C. | Olah, | and J. | Kaplan. | | Language | 2025ConferenceoftheNationsoftheAmericasChap- | | | | | | | | models | (mostly) | know | what | they | know. | CoRR, | | --- | --- | --- | --- | --- | --- | --- | ------ | -------- | ---- | ---- | ---- | ----- | ----- | teroftheAssociationforComputationalLinguistics: abs/2207.05221,2022. HumanLanguageTechnologies,NAACL2025-Vol- | ume1: | LongPapers,Albuquerque,NewMexico,USA, | | | | | | | | | | | | | | ----- | ------------------------------------- | -------- | --- | --- | --- | --- | ------------ | --- | ------- | --- | ----- | -------- | --- | | | | | | | | | A. T. Kalai, | O. | Nachum, | | S. S. | Vempala, | and | | April | 29 - May | 4, 2025, | | | | | | | | | | | | pages 9266–9291. Asso- E.Zhang. Whylanguagemodelshallucinate. CoRR, | ciation | for Computational | | | Linguistics, | 2025. | doi: | | | | | | | | | ------- | ----------------- | --- | --- | ------------ | ----- | ---- | --------------- | --- | ----- | --- | -------------------- | --- | --- | | | | | | | | | abs/2509.04664, | | 2025. | | doi: 10.48550/ARXIV. | | | URLhttps: 10.18653/V1/2025.NAACL-LONG.469. 2509.04664. URL https://doi.org/10.48550/ | //doi.org/10.18653/v1/2025.naacl-long.469. | | | | | | | arXiv.2509.04664. | | | | | | | | ------------------------------------------ | --- | --- | --- | --- | --- | --- | ----------------- | --- | --- | --- | --- | --- | --- | Y.He,B.He,Z.Ding,A.M.Lupidi,Y.Zhu,S.Chen, A.Kamath, J.Ferret,S.Pathak,N.Vieillard,R.Mer- C. Zhang, J. Chen, Y. Ma, V. Tresp, and I. Hor- hej, S. Perrin, T. Matejovicova, A. Ramé, M. Riv- rocks. Supposedly equivalent facts that aren’t? ière,L.Rouillard,T.Mesnard,G.Cideron,J.Grill, entity frequency in pre-training induces asymme- S.Ramos,E.Yvinec,M.Casbon,E.Pot,I.Penchev, try in llms. CoRR, abs/2503.22362, 2025. doi: G.Liu,F.Visin,K.Kenealy,L.Beyer,X.Zhai,A.Tsit- 10.48550/ARXIV.2503.22362. URL https://doi. sulin,R.Busa-Fekete,A.Feng,N.Sachdeva,B.Cole- org/10.48550/arXiv.2503.22362. man,Y.Gao,B.Mustafa,I.Barr,E.Parisotto,D.Tian, 12 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality M.Eyal,C.Cherry,J.Peter,D.Sinopalnikov,S.Bhu- S.Swayamdipta,D.Hoiem,andR.Schwartz. Why patiraju,R.Agarwal,M.Kazemi,D.Malkin,R.Ku- fine-tuningencourageshallucinationsandhowto mar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, fixit. arXivpreprintarXiv:2604.15574,2026. | A. Friesen, | A. | Sharma, | A. | Sharma, | A. | M. Gilady, | | | | | | | | | | ----------- | --- | ------- | --- | ------- | --- | ---------- | ------- | --------- | --- | -------- | --- | ------ | ------ | ------ | | | | | | | | | Y. Kim, | H. Jeong, | | S. Chen, | S. | S. Li, | M. Lu, | K. Al- | A.Goedeckemeyer,A.Saade,A.Kolesnikov,A.Ben- | | | | | | | | hamoud, | J. | Mun, | C. Grau, | M. | Jung, | R. Gameiro, | | | --- | --- | --- | --- | --- | --- | --- | ------- | --- | ---- | -------- | --- | ----- | ----------- | --- | debury,A.Abdagic,A.Vadi,A.György,A.S.Pinto, | | | | | | | | L. Fan, | E. Park, | T. | Lin, | J. Yoon, | W. | Yoon, | M. Sap, | | --------------- | ---------- | ----------- | ------------ | --------- | -------- | ----------- | ------------------------ | -------------- | --------- | ---- | -------------------- | ------- | ------------ | ------- | | A. Das, | A. Bapna, | | A. Miech, | A. | Yang, | A. Pater- | | | | | | | | | | | | | | | | | Y. Tsvetkov, | | P. Liang, | | X. Xu, | X. Liu, | D. | McDuff, | | son, | A. Shenoy, | A. | Chakrabarti, | | B. Piot, | B. Wu, | | | | | | | | | | | | | | | | | H. Lee, | H. | W. Park, | S. | Tulebaev, | and | C. Breazeal. | | | B. Shahriari, | | B. Petrini, | C. | Chen, | C. L. | Lan, C. A. | | | | | | | | | | | | | | | | | Medical | hallucinations | | | in foundation | | models | and | | Choquette-Choo, | | C. | Carey, | C. Brick, | | D. Deutsch, | | | | | | | | | | | | | | | | | theirimpactonhealthcare. | | | | CoRR,abs/2503.05777, | | | | D.Eisenbud,D.Cattle,D.Cheng,D.Paparas,D.S. | | | | | | | | 2025. | doi: | 10.48550/ARXIV.2503.05777. | | | | | URL | | --- | --- | --- | --- | --- | --- | --- | ----- | ---- | -------------------------- | --- | --- | --- | --- | --- | Sreepathihalli,D.Reid,D.Tran,D.Zelle,E.Noland, https://doi.org/10.48550/arXiv.2503.05777. E.Huizenga,E.Kharitonov,F.Liu,G.Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucinska, O. Kitouni, N. Nolte, A. Williams, M. Rabbat, H.Singh,H.Mehta,H.T.Lehri,H.Hazimeh,I.Bal- D. Bouchacourt, and M. Ibrahim. The factoriza- | lantyne, | I. Szpektor, | | I. Nardini, | | J. Pouget-Abadie, | | | | | | | | | | | -------- | ------------ | --- | ----------- | --- | ----------------- | --- | ---- | ------ | ----- | ------ | --- | ------- | --- | -------- | | | | | | | | | tion | curse: | Which | tokens | you | predict | | underlie | J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, the reversal curse and more. In A. Globersons, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. | K. Yu, | K. Hui, | K. | Vodrahalli, | | K. Greff, | L. Qiu, | | | | | | | Advances | in | | ------ | ------- | --- | ----------- | --- | --------- | ------- | -------- | --- | ------ | ------ | -------- | --- | -------- | --- | | | | | | | | | Tomczak, | | and C. | Zhang, | editors, | | | | M. Valentine, M. Coelho, M. Ritter, M. Hoffman, NeuralInformationProcessingSystems38: Annual M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, Conference on Neural Information Processing N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, Systems 2024, NeurIPS 2024, Vancouver, BC, N.Chauhan,O.Bunyan,P.Botarda,P.Caron,P.K. Canada, December 10 - 15, 2024, 2024. URL Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, http://papers.nips.cc/paper_files/paper/ P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, 2024/hash/cbcce87f745072c819204529be843d16- R.Pan,R.Rokni,R.Willoughby,R.Vallu,R.Mullins, Abstract-Conference.html. | S. Jerome, | S. | Smoot, | S. Girgin, | S. | Iqbal, | S. Reddy, | | | | | | | | | | ---------- | --- | ------ | ------------- | --- | ------ | ---------- | --------- | --- | ---- | -------- | --- | ---- | ------------- | --- | | | | | | | | | R. Kumar, | Y. | Kim, | S. Ravi, | H. | Sun, | C. Faloutsos, | | | S. Sheth, | S. | Põder, | S. Bhatnagar, | | S. | R. Panyam, | | | | | | | | | S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, R.Salakhutdinov,andM.Yoon. Automaticquestion- | | | | | | | | answergenerationforlong-tailknowledge. | | | | | | | CoRR, | | --------- | -------------- | -------- | ------- | ------------- | ----- | -------- | -------------------------------------- | --- | --- | ----- | ---- | --------------- | --- | ----- | | U. Kalra, | U. | Evci, V. | Misra, | V. Roseberry, | | V. Fein- | | | | | | | | | | | | | | | | | abs/2403.01382, | | | 2024. | doi: | 10.48550/ARXIV. | | | | berg, | V. Kolesnikov, | | W. Han, | W. | Kwon, | X. Chen, | | | | | | | | | https://doi.org/10.48550/ | | | | | | | | 2403.01382. | | URL | | | | | | | -------- | --- | ---- | ------- | --------- | --- | ----------- | ----------- | --- | --- | --- | --- | --- | --- | --- | | Y. Chow, | Y. | Zhu, | Z. Wei, | Z. Egyed, | | V. Cotruta, | | | | | | | | | arXiv.2403.01382. | M. Giang, | P. | Kirk, A. | Rao, | J. Lo, | E. Moreira, | L. G. | | | | | | | | | | --------- | --- | ----------- | ---- | --------- | ----------- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- | | Martins, | O. | Sanseviero, | L. | Gonzalez, | | Z. Gleicher, | | | | | | | | | T.Kwiatkowski,J.Palomaki,O.Redfield,M.Collins, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, | J. K. | Barral, | Z. Ghahramani, | | R. | Hadsell, | Y. Ma- | | | | | | | | | | ----- | ------- | -------------- | --- | --- | -------- | ------ | ---------- | --- | ---- | ------------- | --- | --- | ------ | ------- | | | | | | | | | J. Devlin, | K. | Lee, | K. Toutanova, | | L. | Jones, | M. Kel- | tias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, cey,M.Chang,A.M.Dai,J.Uszkoreit,Q.Le,and O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, S.Petrov. Naturalquestions: abenchmarkforques- C.Farabet,E.Buchatskaya,J.Alayrac,R.Anil,D.D. Trans.Assoc.Comput.Lin- tionansweringresearch. Lepikhin,S.Borgeaud,O.Bachem,A.Joulin,A.An- guistics,7:452–466,2019. doi: 10.1162/TACL\_A\ dreev, C. Hardin, R. Dadashi, and L. Hussenot. https://doi.org/10.1162/tacl_a_ | | | | | | | | _00276. | URL | | | | | | | | ---------------------- | --- | --- | --- | -------------------- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | --- | | Gemma3technicalreport. | | | | CoRR,abs/2503.19786, | | | | | | | | | | | 00276. | 2025. | doi: | 10.48550/ARXIV.2503.19786. | | | | URL | | | | | | | | | | ----- | ---- | -------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | https://doi.org/10.48550/arXiv.2503.19786. B. Y. Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, | | | | | | | | K. R. | Chandu, | C. | Bhagavatula, | | and | Y. Choi. | The | | ----------- | --- | ----- | ----------- | --- | ----------- | --- | ------------------------- | ------- | --- | ------------ | --- | ------------------- | -------- | --- | | N. Kandpal, | H. | Deng, | A. Roberts, | | E. Wallace, | and | | | | | | | | | | | | | | | | | unlockingspellonbasellms: | | | | | Rethinkingalignment | | | C. Raffel. Large language models struggle to viain-contextlearning. InTheTwelfthInternational learn long-tail knowledge. In A. Krause, E. Brun- ConferenceonLearningRepresentations,ICLR2024, skill, K. Cho, B. Engelhardt, S. Sabato, and Vienna,Austria,May7-11,2024.OpenReview.net, | | | | International | | Conference | on | | | | | | | | | | --- | --- | --- | ------------- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | J. Scarlett, editors, 2024a. URL https://openreview.net/forum?id= | Machine | Learning, | | ICML 2023, | | 23-29 | July 2023, | wxJ0eXwwda. | | | | | | | | | --------- | --------- | -------- | ---------- | --- | ----- | ---------- | ----------- | ---------- | --- | ----- | --- | ---- | --------- | --- | | Honolulu, | Hawaii, | | USA, | | | Proceed- | | | | | | | | | | | | | volume | | 202 | of | | | | | | | | | | | | | | | | | J. Lin, | V. Berges, | X. | Chen, | W. | Yih, | G. Ghosh, | and | | ings of | Machine | Learning | Research, | | pages | 15696– | | | | | | | | | 15707. PMLR, 2023. URL https://proceedings. B. Oguz. Learning facts at scale with active read- mlr.press/v202/kandpal23a.html. ing. CoRR, abs/2508.09494, 2025. doi: 10. | | | | | | | | 48550/ARXIV.2508.09494. | | | | URLhttps://doi.org/ | | | | | --- | --- | --- | --- | --- | --- | --- | ----------------------- | --- | --- | --- | ------------------- | --- | --- | --- | 10.48550/arXiv.2508.09494. | G. Kaplan, | Z. | Gekhman, | Z. | Zhu, L. | Rozner, | Y. Reif, | | | | | | | | | | ---------- | --- | -------- | --- | ------- | ------- | -------- | --- | --- | --- | --- | --- | --- | --- | --- | 13 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality S.Lin,L.Gao,B.Oguz,W.Xiong,J.Lin,S.Yih,and editors,Proceedingsofthe2024ConferenceonEmpir- icalMethodsinNaturalLanguageProcessing,EMNLP | X. Chen. | FLAME | | : Factuality-aware | | alignment | | | | | | | | | | -------- | ----- | --- | ------------------ | --- | --------- | --- | --- | --- | --- | --- | --- | --- | --- | for large language models. In A. Globersons, 2024,Miami,FL,USA,November12-16,2024,pages L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. 13603–13615.AssociationforComputationalLin- Tomczak, and C. Zhang, editors, Advances in guistics, 2024. doi: 10.18653/V1/2024.EMNLP- NeuralInformationProcessingSystems38: Annual MAIN.754. URL https://doi.org/10.18653/v1/ Conference on Neural Information Processing 2024.emnlp-main.754. | Systems | 2024, | NeurIPS | 2024, | Vancouver, | BC, | | | | | | | | | | ------- | ----- | ------- | ----- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | Canada, December 10 - 15, 2024, J. Ma, J. Gu, Z. Ling, Q. Liu, and C. Liu. Untying | | | | | 2024b. | URL | | | | | | | | | | --- | --- | --- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | http://papers.nips.cc/paper_files/paper/ thereversalcurseviabidirectionallanguagemodel 2024/hash/d16152d53088ad779ffa634e7bf66166- editing. CoRR, abs/2310.10322, 2023. doi: 10. Abstract-Conference.html. 48550/ARXIV.2310.10322. URLhttps://doi.org/ 10.48550/arXiv.2310.10322. | Z.Lin, Z.Fu, | K.Liu, | L.Xie, | B.Lin, | W.Wang, | D.Cai, | | | | | | | | | | ------------ | ------ | ----------- | ------ | ------------ | ------ | ---------- | --- | ----- | --------- | --- | ---- | ------------ | --- | | | | | | | | A. Mallen, | A. | Asai, | V. Zhong, | R. | Das, | D. Khashabi, | | | Y. Wu, | and J. | Ye. Delving | into | the reversal | curse: | | | | | | | | | How far can large language models generalize? and H. Hajishirzi. When not to trust language In A. Globersons, L. Mackey, D. Belgrave, A. Fan, models: Investigating effectiveness of parametric | | | | | | | and | non-parametric | | memories. | | In A. | Rogers, | J. L. | | ---------- | --- | ----------- | --- | --------- | -------- | --- | -------------- | --- | --------- | --- | ----- | ------- | ----- | | U. Paquet, | J. | M. Tomczak, | and | C. Zhang, | editors, | | | | | | | | | Proceed- AdvancesinNeuralInformationProcessingSystems Boyd-Graber, and N. Okazaki, editors, 38: Annual Conference on Neural Information ings of the 61st Annual Meeting of the Association | | | | | | | for | Computational | | Linguistics | | (Volume | 1: Long | Pa- | | --- | --- | --- | --- | --- | --- | --- | ------------- | --- | ----------- | --- | ------- | ------- | --- | ProcessingSystems2024,NeurIPS2024,Vancouver, BC,Canada,December10-15,2024,2024c. URL pers),ACL2023,Toronto,Canada,July9-14,2023, http://papers.nips.cc/paper_files/paper/ pages9802–9822.AssociationforComputational 2024/hash/36b6180f3dab4025ba763e853b044814- Linguistics, 2023. doi: 10.18653/V1/2023.ACL- Abstract-Conference.html. LONG.546. URL https://doi.org/10.18653/v1/ 2023.acl-long.546. | G. Lior, | E. Habba, | | S. Levy, | A. Caciularu, | and | | | | | | | | | | -------- | --------- | --- | -------- | ------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | G. Stanovsky. Reliableeval: A recipe for stochas- S. V. Marjanovic, A. Patel, V. Adlakha, M. Agha- | | | | | | CoRR, | johari, | P. | BehnamGhader, | | M. | Bhatia, | A. | Khan- | | --- | --- | --- | --- | --- | ----- | ------- | --- | ------------- | --- | --- | ------- | --- | ----- | ticLLMevaluationviamethodofmoments. abs/2505.22169, 2025. doi: 10.48550/ARXIV. delwal, A. Kraft, B. Krojer, X. H. Lù, N. Meade, https://doi.org/10.48550/ D. Shin, A. Kazemnejad, G. Kamath, M. Mos- | 2505.22169. | | URL | | | | | | | | | | | | | ----------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | arXiv.2505.22169. bach, K. Stanczak, and S. Reddy. Deepseek-r1 | | | | | | | thoughtology: | | Let’s | about | LLM | reasoning. | | CoRR, | | --- | --- | --- | --- | --- | --- | ------------- | --- | ----- | ----- | --- | ---------- | --- | ----- | X. Lu, X. Li, Q. Cheng, K. Ding, X. Huang, and abs/2504.07128, 2025. doi: 10.48550/ARXIV. https://doi.org/10.48550/ | X.Qiu. | Scalinglawsforfactmemorizationoflarge | | | | | 2504.07128. | | URL | | | | | | | -------- | ------------------------------------- | -------- | ----------------- | ------ | ----------- | ----------------- | --- | --- | --- | --- | --- | --- | --- | | language | models. | | In Y. Al-Onaizan, | | M. Bansal, | arXiv.2504.07128. | | | | | | | | | | | | Findings | of the | Association | | | | | | | | | | and Y. | Chen, | editors, | | | | | | | | | | | | for Computational Linguistics: EMNLP 2024, Mi- K. Meng, D. Bau, A. Andonian, and Y. Belinkov. ami, Florida, USA, November 12-16, 2024, pages Locating and editing factual associations in | | | | | | | GPT. | In | S. Koyejo, | | S. Mohamed, | | A. Agarwal, | | | --- | --- | --- | --- | --- | --- | ---- | --- | ---------- | --- | ----------- | --- | ----------- | --- | 11263–11282.AssociationforComputationalLin- Advances guistics,2024. doi: 10.18653/V1/2024.FINDINGS- D.Belgrave, K.Cho, andA.Oh, editors, URLhttps://doi.org/10.18653/v1/ in Neural Information Processing Systems 35: EMNLP.658. AnnualConferenceonNeuralInformationProcessing 2024.findings-emnlp.658. | | | | | | | Systems | 2022, | NeurIPS | | 2022, | New | Orleans, | LA, | | --- | --- | --- | --- | --- | --- | ------- | ----- | ------- | --- | ----- | --- | -------- | --- | L.Luo,T.Vu,D.Q.Phung,andG.Haffari. Systematic USA,November28-December9,2022,2022. URL https://papers.nips.cc/paper_files/paper/ assessmentoffactualknowledgeinlargelanguage models. In H. Bouamor, J. Pino, and K. Bali, edi- 2022/hash/6f1d43d5a82a37e89b0665b33bf3a182- tors,FindingsoftheAssociationforComputational Abstract-Conference.html. | Linguistics: | EMNLP2023,Singapore,December6-10, | | | | | | | | | | | | | | ------------ | --------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 2023,pages13272–13286.AssociationforCompu- T.R.Menta,S.Agrawal,andC.Agarwal. Analyzing | | | | | | | memorization | | in | large | language | models | through | | | ------------------------- | --- | --- | ---- | ----------------- | --- | ------------ | --- | --- | ----- | -------- | ------ | ------- | --- | | tationalLinguistics,2023. | | | doi: | 10.18653/V1/2023. | | | | | | | | | | FINDINGS-EMNLP.885. URLhttps://doi.org/10. thelensofmodelattribution. InL.Chiruzzo,A.Rit- 18653/v1/2023.findings-emnlp.885. ter, and L. Wang, editors, Proceedings of the 2025 ConferenceoftheNationsoftheAmericasChapterof A. Lv, K. Zhang, S. Xie, Q. Tu, Y. Chen, J. Wen, and the Association for Computational Linguistics: Hu- R.Yan. Ananalysisandmitigationofthereversal manLanguageTechnologies,NAACL2025-Volume 1: LongPapers,Albuquerque,NewMexico,USA,April | curse. | In Y. | Al-Onaizan, | M. Bansal, | and | Y. Chen, | | | | | | | | | | ------ | ----- | ----------- | ---------- | --- | -------- | --- | --- | --- | --- | --- | --- | --- | --- | 14 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality 29 - May 4, 2025, pages 10661–10689. Associ- Linguistics(Volume1: LongPapers),ACL2025,Vi- enna,Austria,July27-August1,2025,pages1402– | ation | for Computational | | | Linguistics, | 2025. | doi: | | | | | | | | ----- | ----------------- | --- | --- | ------------ | ----- | ---- | --- | --- | --- | --- | --- | --- | 10.18653/V1/2025.NAACL-LONG.535. URLhttps: 1425. Association for Computational Linguistics, //doi.org/10.18653/v1/2025.naacl-long.535. URLhttps://aclanthology.org/2025.acl- 2025. long.71/. M.Mizrahi,G.Kaplan,D.Malkin,R.Dror,D.Shahaf, and G. Stanovsky. State of what art? A call for Tip-of-the-TongueStates: Phenomenol- B.L.Schwartz. Trans.Assoc.Comput. multi-promptLLMevaluation. ogy, Mechanism, and Lexical Retrieval. Lawrence Linguistics,12:933–949,2024. doi: 10.1162/TACL\ ErlbaumAssociates,Mahwah,NJ,2002. | _A\_00681. | | URLhttps://doi.org/10.1162/tacl_ | | | | | | | | | | | | ---------- | --- | -------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | a_00681. | | | | | | | | A. Simhi, | J. Herzig, | I. Itzhak, | D. | Arad, Z. Gekhman, | | | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | ---------- | --- | ----------------- | --- | R.Reichart,F.Barez,G.Stanovsky,I.Szpektor,and | O. Nahum, | N. | Calderon, | O. | Keller, | I. Szpektor, | and | | | | | | | | --------- | --- | --------- | --- | ------- | ------------ | --- | ----------- | --- | --------------------------------- | --- | --- | --- | | | | | | | | | Y.Belinkov. | | HACK:hallucinationsalongcertainty | | | | R. Reichart. Are llms better than reported? de- andknowledgeaxes. CoRR,abs/2510.24222,2025. tecting label errors and mitigating their effect on doi: 10.48550/ARXIV.2510.24222. URLhttps:// modelperformance. CoRR,abs/2410.18889,2024. doi.org/10.48550/arXiv.2510.24222. https: | doi: | 10.48550/ARXIV.2410.18889. | | | | URL | | | | | | | | | ---- | -------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | //doi.org/10.48550/arXiv.2410.18889. K.Sun,Y.E.Xu,H.Zha,Y.Liu,andX.L.Dong. Head- | | | | | | | | to-tail: | How | knowledgeable | | are large | language | | ---------------------------- | --- | --- | --- | ----------- | ------------ | --------- | ------------- | ------------- | ------------------------------ | --- | --------- | -------- | | T.O.NelsonandL.Narens. | | | | Metamemory: | | Atheoret- | | | | | | | | | | | | | | | models(llms)? | | A.K.A.willllmsreplaceknowledge | | | | | icalframeworkandnewfindings. | | | | | InG.H.Bower, | | | InNAACL,2024. | | | | | graphs? | editor, | ThePsychologyofLearningandMotivation, | | | | | | | | | | | | | ------- | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | volume26,pages125–173.AcademicPress,New B. Wang and H. Sun. Is the reversal curse a bind- York,1990. | | | | | | | | ingproblem? | | uncoveringlimitationsoftransform- | | | | | ------- | ------- | ----- | ------- | --- | ------ | --------- | --------------- | --- | --------------------------------- | ---- | --------------- | ----- | | | | | | | | | ers from | a | basic generalization | | failure. | CoRR, | | OpenAI. | GPT-4.1 | model | series. | | Online | document, | | | | | | | | | | | | | | | abs/2504.01928, | | 2025. | doi: | 10.48550/ARXIV. | | https://openai.com/index/gpt-4- | 2025a. | URL | | | | | | | | https://doi.org/10.48550/ | | | | | ------ | --- | --- | --- | --- | --- | --- | ----------- | --- | ------------------------- | --- | --- | --- | | | | | | | | | 2504.01928. | | URL | | | | 1/. arXiv.2504.01928. | OpenAI. | GPT-5 | System | Card, | 2025b. | URL | https: | | | | | | | | ------- | ----- | ------ | ----- | ------ | --- | ------ | --- | --- | --- | --- | --- | --- | //openai.com/index/gpt-5-system-card/. J. Wang, Q. Sun, X. Li, and M. Gao. Boosting lan- guagemodelsreasoningwithchain-of-knowledge | | | | | | | | prompting. | | InL.Ku,A.Martins,andV.Srikumar, | | | | | --- | --- | --- | --- | --- | --- | --- | ---------- | --- | ------------------------------- | --- | --- | --- | H.Orgad,M.Toker,Z.Gekhman,R.Reichart,I.Szpek- editors,Proceedingsofthe62ndAnnualMeetingof | tor, H. | Kotek, | and | Y. Belinkov. | | Llms know | more | | | | | | | | ------- | --------------- | --- | ------------ | --------- | -------------- | ------ | --------------- | ------- | ----------------- | ----- | ----------- | ----- | | | | | | | | | the Association | | for Computational | | Linguistics | (Vol- | | than | they show: | On | the | intrinsic | representation | | | | | | | | | | | | | | | | ume | 1: Long | Papers), ACL | 2024, | Bangkok, | Thai- | | of LLM | hallucinations. | | | In The | Thirteenth | Inter- | | | | | | | national Conference on Learning Representations, land,August11-16,2024,pages4958–4981.Asso- | | | | | | | | ciationforComputationalLinguistics,2024a. | | | | | doi: | | ---- | ----- | ---------- | ----- | ------ | ----- | ----- | ----------------------------------------- | --- | --- | --- | --- | ---- | | ICLR | 2025, | Singapore, | April | 24-28, | 2025. | Open- | | | | | | | https: Review.net,2025. URLhttps://openreview.net/ 10.18653/V1/2024.ACL-LONG.271. URL forum?id=KRnsX5Em3W. //doi.org/10.18653/v1/2024.acl-long.271. F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. A.Bakhtin,Y.Wu,andA.H.Miller. Languagemod- Chi, S.Narang, A. Chowdhery, and D. Zhou. Self- elsasknowledgebases? InK.Inui,J.Jiang,V.Ng, consistency improves chain of thought reasoning and X. Wan, editors, Proceedings of the 2019 Con- inlanguagemodels. InTheEleventhInternational ference on Empirical Methods in Natural Language ConferenceonLearningRepresentations,ICLR2023, | | | | | | | | Kigali, | Rwanda, | May 1-5, | 2023. | | | | --- | --- | --- | --- | --- | --- | --- | ------- | ------- | -------- | ----- | --- | --- | Processing and the 9th International Joint Confer- OpenReview.net, enceonNaturalLanguageProcessing,EMNLP-IJCNLP 2023. URL https://openreview.net/forum?id= | 2019,HongKong,China,November3-7,2019,pages | | | | | | | 1PL1NIMMrw. | | | | | | | ------------------------------------------ | --- | --- | --- | --- | --- | --- | ----------- | --- | --- | --- | --- | --- | 2463–2473.AssociationforComputationalLinguis- X.Wang,A.Antoniades,Y.Elazar,A.Amayuelas,A.Al- | tics, 2019. | | doi: 10.18653/V1/D19-1250. | | | | URL | | | | | | | | ----------- | --- | -------------------------- | --- | --- | --- | --- | -------------------------- | --- | --- | --- | ------------------ | --- | | | | | | | | | balak,K.Zhang,andW.Y.Wang. | | | | Generalizationv.s. | | https://doi.org/10.18653/v1/D19-1250. | | | | | | | | memorization: | | Tracinglanguagemodels’capabili- | | | | | --- | --- | --- | --- | --- | --- | --- | ------------- | --- | ------------------------------- | --- | --- | --- | A. Ravichander, S. Ghela, D. Wadden, and Y. Choi. tiesbacktopretrainingdata. InTheThirteenthIn- ternationalConferenceonLearningRepresentations, | Halogen: | FantasticLLMhallucinationsandwhere | | | | | | | | | | | | | -------- | ---------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | to find them. In W. Che, J. Nabende, E. Shutova, ICLR2025,Singapore,April24-28,2025.OpenRe- andM.T.Pilehvar,editors,Proceedingsofthe63rd view.net, 2025a. URL https://openreview.net/ AnnualMeetingoftheAssociationforComputational forum?id=IQxBDLmVpT. 15 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Y. Wang, Y. Chen, W. Wen, Y. Sheng, L. Li, and Linguistics(Volume1: LongPapers),ACL2025,Vi- | | | | | | | | enna, | Austria, | July | 27 - August | 1, | 2025, | | | ------- | -------------------------------------- | --- | --- | --- | --- | --- | ----- | -------- | ---- | ----------- | --- | ----- | ----- | | D.Zeng. | Unveilingfactualrecallbehaviorsoflarge | | | | | | | | | | | | pages | languagemodelsthroughknowledgeneurons. In 18403–18419.AssociationforComputationalLin- URLhttps://aclanthology.org/ | Y. Al-Onaizan, | | M. | Bansal, | and | Y. Chen, | editors, | guistics,2025b. | | | | | | | | -------------- | --- | --- | ------- | --- | -------- | -------- | --------------- | --- | --- | --- | --- | --- | --- | Proceedings of the 2024 Conference on Empirical 2025.acl-long.901/. | Methods | in | Natural | Language | Processing, | | EMNLP | | | | | | | | | ------- | --- | ------- | -------- | ----------- | --- | ----- | --- | --- | --- | --- | --- | --- | --- | 2024,Miami,FL,USA,November12-16,2024,pages J. Yang, J. Tu, H. Liu, X. Wang, C. Zheng, Z. Zhang, 7388–7402. Association for Computational Lin- S. Cui, C. Chen, T. He, H. Wang, Y. Ong, and | | | | | | | | M.Huang. | | BARREL:boundary-awarereasoningfor | | | | | | --------------- | --- | -------------------------------- | ----------------------- | --- | --- | --- | ----------------------- | --- | --------------------------------- | -------------------- | --- | --- | --- | | guistics,2024b. | | doi: | 10.18653/V1/2024.EMNLP- | | | | | | | | | | | | | | | | | | | factualandreliablelrms. | | | CoRR,abs/2505.13529, | | | | | MAIN.420. | | URL https://doi.org/10.18653/v1/ | | | | | | | | | | | | 2024.emnlp-main.420. 2025. doi: 10.48550/ARXIV.2505.13529. URL https://doi.org/10.48550/arXiv.2505.13529. | Y. Wang, | C. Wan, | S. | Hu, Y. | Zhang, | X. Tian, | Y. Chen, | | | | | | | | | -------- | ------- | --- | ------ | ------ | -------- | -------- | --- | --- | --- | --- | --- | --- | --- | X.Shen,andJ.Ye. Tracinganddissectinghowllms Z.Yao,Y.Liu,Y.Chen,J.Chen,J.Fang,L.Hou,J.Li, recallfactualknowledgeforrealworldquestions.In andT.Chua. Arereasoningmodelsmoreproneto W.Che,J.Nabende,E.Shutova,andM.T.Pilehvar, hallucination? CoRR,abs/2505.23646,2025. doi: | editors,Proceedingsofthe63rdAnnualMeetingofthe | | | | | | | | | | | | https://doi. | | | ---------------------------------------------- | --- | --- | --- | --- | --- | --- | -------------------------- | --- | --- | --- | --- | ------------ | --- | | | | | | | | | 10.48550/ARXIV.2505.23646. | | | | URL | | | AssociationforComputationalLinguistics(Volume1: org/10.48550/arXiv.2505.23646. LongPapers),ACL2025,Vienna,Austria,July27- August 1, 2025, G. Yona, R. Aharoni, and M. Geva. Narrowing the | | | pages | 23246–23271. | | | Association | | | | | | | | | --- | --- | ----- | ------------ | --- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- | forComputationalLinguistics,2025b. URLhttps: knowledgeevaluationgap: Open-domainquestion //aclanthology.org/2025.acl-long.1133/. answeringwithmulti-granularityanswers. InL.Ku, | | | | | | | | A. Martins, | | and V. Srikumar, | | editors, | Proceedings | | | --- | --- | --- | --- | --- | --- | --- | ----------- | --- | ---------------- | --- | -------- | ----------- | --- | J.Wei,X.Wang,D.Schuurmans,M.Bosma,B.Ichter, of the 62nd Annual Meeting of the Association for F.Xia,E.H.Chi,Q.V.Le,andD.Zhou. Chain-of- ComputationalLinguistics(Volume1: LongPapers), thought prompting elicits reasoning in large lan- ACL2024,Bangkok,Thailand,August11-16,2024, InNeurIPS,2022. | guagemodels. | | | | | | | pages6737–6751.AssociationforComputational | | | | | | | | ------------ | ------- | --- | --------- | --- | -------- | --------- | ------------------------------------------ | --- | ---------- | --------------------- | --- | --- | --- | | | | | | | | | Linguistics, | | 2024. doi: | 10.18653/V1/2024.ACL- | | | | | J. Wei, N. | Karina, | H. | W. Chung, | Y. | J. Jiao, | S. Papay, | | | | | | | | https://doi.org/10.18653/v1/ | | | | | | | | LONG.365. | | URL | | | | | | ---------- | --- | --------- | --- | ------ | ------ | ------- | --------- | --- | --- | --- | --- | --- | --- | | A. Glaese, | J. | Schulman, | | and W. | Fedus. | Measur- | | | | | | | | 2024.acl-long.365. | ing short-form | | factuality | | in large | language | mod- | | | | | | | | | -------------- | ----- | --------------- | --- | -------- | -------- | -------- | --- | --- | --- | --- | --- | --- | --- | | els. | CoRR, | abs/2411.04368, | | | 2024a. | doi: 10. | | | | | | | | G.Yona,M.Geva,andY.Matias.Hallucinationsunder- URLhttps://doi.org/ | 48550/ARXIV.2411.04368. | | | | | | | | | | | | | arXiv | | ----------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | minetrust;metacognitionisawayforward. 10.48550/arXiv.2411.04368. preprintarXiv:2605.01428,2026. | Y. Wei, | X. Yu, | Y. Weng, | H. | Ma, Y. | Zhang, | J. Zhao, | | | | | | | | | ------- | ------ | -------- | --- | ------ | ------ | -------- | --- | --- | --- | --- | --- | --- | --- | S.Yu,Y.Xu,C.Du,Y.Zhou,M.Qiu,Q.Sun,H.Zhang, | andK.Liu. | | Doesknowledgelocalizationholdtrue? | | | | | | | | | | | | | --------- | --- | ---------------------------------- | --- | --- | --- | --- | --- | ------ | ------- | -------- | -------- | -------- | --- | | | | | | | | | and | J. Wu. | Reverse | modeling | in large | language | | surprisingdifferencesbetweenentityandrelation | | | | | | | | models. | In | L. Chiruzzo, | A. | Ritter, and | L. | Wang, | | ------------------------------------------ | --- | ----------- | --- | ------- | ----- | --------- | -------- | ----------- | ------------ | ----------- | ----------- | --- | ------ | | perspectives | | in language | | models. | In E. | Serra and | | | | | | | | | | | | | | | | | Proceedings | | of the 2025 | Conference | | of the | | F.Spezzano,editors,Proceedingsofthe33rdACM | | | | | | | editors, | | | | | | | NationsoftheAmericasChapteroftheAssociationfor InternationalConferenceonInformationandKnowl- | | | | | | | | ComputationalLinguistics: | | | HumanLanguageTech- | | | | | --- | --- | --- | --- | --- | --- | --- | ------------------------- | --- | --- | ------------------ | --- | --- | --- | edgeManagement,CIKM2024,Boise,ID,USA,Oc- | | | | | | | | nologies,NAACL2025-Volume2: | | | | ShortPapers,Al- | | | | --- | --- | --- | --- | --- | --- | --- | --------------------------- | --- | --- | --- | --------------- | --- | --- | tober21-25,2024,pages4118–4122.ACM,2024b. | | | | | | | | buquerque, | | New Mexico, | April | 29 - May | 4, | 2025, | | ----------------------------- | --- | --- | --- | --- | --- | -------- | ---------- | --- | ----------- | ----- | -------- | --- | ----- | | doi: 10.1145/3627673.3679900. | | | | | URL | https:// | | | | | | | | pages306–320.AssociationforComputationalLin- doi.org/10.1145/3627673.3679900. | | | | | | | | guistics, | 2025. | doi: | 10.18653/V1/2025.NAACL- | | | | | --- | --- | --- | --- | --- | --- | --- | --------- | ----- | ---- | ----------------------- | --- | --- | --- | F. Wu, W. Xuan, X. Lu, Z. Harchaoui, and Y. Choi. SHORT.27. URL https://doi.org/10.18653/v1/ 2025.naacl-short.27. | Theinvisibleleash: | | | WhyRLVRmaynotescapeits | | | | | | | | | | | | ------------------ | ----- | --------------- | ---------------------- | --- | ------ | -------- | --- | --- | --- | --- | --- | --- | --- | | origin. | CoRR, | abs/2507.14843, | | | 2025a. | doi: 10. | | | | | | | | 48550/ARXIV.2507.14843. URLhttps://doi.org/ J. Yuan, L. Pan, C. Hang, J. Guo, J. Jiang, B. Min, 10.48550/arXiv.2507.14843. P. Ng, and Z. Wang. Towards a holistic evalua- CoRR, | | | | | | | | tion | of llms | on factual | knowledge | recall. | | | | --- | --- | --- | --- | --- | --- | --- | ---- | ------- | ---------- | --------- | ------- | --- | --- | X.Wu,L.Pan,Y.Xie,R.Zhou,S.Zhao,Y.Ma,M.Du, | | | | | | | | abs/2404.16164, | | 2024. | doi: | 10.48550/ARXIV. | | | | --- | --- | --- | --- | --- | --- | --- | --------------- | --- | ----- | ---- | --------------- | --- | --- | R.Mao,A.T.Luu,andW.Y.Wang. Antileakbench: 2404.16164. URL https://doi.org/10.48550/ | Preventing | | data contamination | | | by automatically | | arXiv.2404.16164. | | | | | | | | ---------- | --- | ------------------ | --- | --- | ---------------- | --- | ----------------- | --- | --- | --- | --- | --- | --- | constructingbenchmarkswithupdatedreal-world knowledge. InW.Che,J.Nabende,E.Shutova,and M.Zhang,J.Bjerva,andR.Biswas. Followthepath: M.T.Pilehvar,editors,Proceedingsofthe63rdAn- Reasoningoverknowledgegraphpathstoimprove nual Meeting of the Association for Computational CoRR,abs/2505.11140,2025. | | | | | | | | llmfactuality. | | | | | | doi: | | --- | --- | --- | --- | --- | --- | --- | -------------- | --- | --- | --- | --- | --- | ---- | 16 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality 10.48550/ARXIV.2505.11140. URL https://doi. org/10.48550/arXiv.2505.11140. C.Zhou,P.Liu,P.Xu,S.Iyer,J.Sun,Y.Mao,X.Ma, A.Efrat,P.Yu,L.Yu,S.Zhang,G.Ghosh,M.Lewis, L.Zettlemoyer,andO.Levy. LIMA:lessismorefor alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, AdvancesinNeuralInformationProcessingSystems 36: Annual Conference on Neural Information Pro- cessingSystems2023,NeurIPS2023,NewOrleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.cc/paper_files/paper/ 2023/hash/ac662d74829e4407ce1d126477f4a03a- Abstract-Conference.html. H. Zhu, B. Huang, S. Zhang, M. I. Jordan, J. Jiao, Y. Tian, and S. J. Russell. Towards a theoretical understanding of the ’reversal curse’ via training dynamics. InA.Globersons,L.Mackey,D.Belgrave, A.Fan,U.Paquet,J.M.Tomczak,andC.Zhang,ed- itors,AdvancesinNeuralInformationProcessingSys- tems38: AnnualConferenceonNeuralInformation ProcessingSystems2024,NeurIPS2024,Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/paper_files/paper/ 2024/hash/a4b95476f673e6e538f80862f622ba2f- Abstract-Conference.html. N. Zucchet, J. Bornschein, S. C. Y. Chan, A. K. Lampinen, R. Pascanu, and S. De. How do lan- guage models learn facts? dynamics, curricula andhallucinations. CoRR,abs/2503.21676,2025. doi: 10.48550/ARXIV.2503.21676. URL https: //doi.org/10.48550/arXiv.2503.21676. 17 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality WikiProfileisconstructedusingafullyautomated pipelinepoweredbyapromptedLLM,Gemini-2.5-Pro Appendix | | | | | | | | with thinking | enabled | (Comanici | et | al., 2025). | Our | | --- | --- | --- | --- | --- | --- | --- | ------------- | ------- | --------- | --- | ----------- | --- | goalistoprovideaframeworkthatcanbereadilyap- pliedtoothercorpora,includingdomain-specificones, | | | | | | | | enabling | researchers | to identify | knowledge | | profiles | | --- | --- | --- | --- | --- | --- | --- | -------- | ----------- | ----------- | --------- | --- | -------- | A WikiProfile–FullDetails 18 andcharacterizeLLMbehavior. Promptsweredevel- opedthroughmanualoptimizationonasmallsubset | | A.1 | TasksandProperties | | | . . | . . . . 18 | | | | | | | | --- | --- | ------------------ | --- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- | ofapproximately100documentsandfacts,whichis | | A.2 | FactExtractionProcess | | | . | . . . . 19 | | | | | | | | --- | --- | ------------------------- | --- | --- | --- | ---------- | ---------------------------- | --- | --- | ------------------- | --- | --- | | | | | | | | | excludedfromthefinaldataset. | | | Thisprocessinvolved | | | | | A.3 | QuestionGenerationProcess | | | | . . 20 | | | | | | | iterativeerroranalysisandrefinement. | | A.4 | ManualQualityValidation | | | | . . . 21 | | | | | | | | --- | ------------------- | ----------------------- | --- | --- | --- | -------- | ----------------------- | --- | --- | --- | --- | --- | | | B ThinkingandRecall | | | | | 22 | A.1. TasksandProperties | | | | | | | | B.1 | ThreePossibleMechanisms | | | | . . 22 | | | | | | | Webeginbyoutliningthepropertiesthatguidedthe | | B.2 | Measuring | Availability | | and | Ro- | | | | | | | | --- | --- | ----------------------------- | ------------ | ----------------- | ----- | ---------- | -------------------- | --- | -------------------------- | --- | --- | --- | | | | bustness | . . | . . . . | . . . | . . . . 22 | designofWikiProfile. | | | | | | | | B.3 | RulingOutResponseDiversity | | | | . 23 | | | | | | | | | B.4 | EvidenceforRecallFacilitation | | | | 23 | | | | | | | | | | | | | | | RequiredProperties | | Allquestionsmustsatisfythe | | | | | | B.5 | The | Role of | Thinking-Specific | | | | | | | | | followingpropertiestoensuremethodologicalrigor | | | Training | . . | . . . . | . . . | . . . . 23 | | | | | | | | --- | --- | -------- | --- | ------- | ----- | ---------- | --- | --- | --- | --- | --- | --- | andenablecomparisonagainstasingleverifiablegold | | C AdditionalResults | | | | | 23 | answer: | | | | | | | --- | ------------------- | --- | --- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | C.1 Profiles. . . . . . . . . . . . . . 23 • Answerable: Acorrectanswerexists. | | C.2 | FactPopularity | | . . | . . . | . . . . 24 | Unambiguous: | | | | | | | --- | --- | -------------- | --- | ------- | ----- | ---------- | ----------------- | --- | ------------------------------ | --- | --- | --- | | | | | | | | | • | | Thequestionisself-containedand | | | | | | C.3 | Phrasing | . . | . . . . | . . . | . . . . 24 | preciselyphrased. | | | | | | C.4 ReverseQuestions . . . . . . . 25 • Specific: The expected answer type and level of C.5 Thinking . . . . . . . . . . . . . 25 detailareclearlyspecified. | | | | | | | | • Single-answer: | | Exactlyoneansweriscorrect. | | | | | --- | --- | --- | --- | --- | --- | --- | ---------------- | --- | -------------------------- | --- | --- | --- | D MethodologicalandDesignChoices 25 • Nottime-sensitive: Theanswerisstableovertime. | | D.1 | Handling | Partially | | and | Other | | | | | | | | --- | --- | ------------------- | --------- | ------- | ----- | ---------- | ----------------------------- | --- | ------------------------- | ------------------- | --- | --- | | | | Grades | . . . | . . . . | . . . | . . . . 25 | | | | | | | | | | | | | | | ControlledProperties | | Thesepropertiesdefinesys- | | | | | | D.2 | ThresholdSelection. | | | . . | . . . . 26 | | | | | | | | | | | | | | | tematicvariationsacrosstasks. | | | Bymanipulatingthem, | | | | | D.3 | GraderEvaluation. | | | . . . | . . . . 26 | | | | | | | weexaminefactorsthataffectfactualknowledge. | | D.4 | MultipleResponseGeneration | | | | . 27 | | | | | | | | --- | --------- | -------------------------- | --- | --- | --- | ---- | --------- | ----------- | ------------ | --- | ------------ | --- | | | | | | | | | • Format: | Completion, | closed-book, | | or multiple- | | | | E Prompts | | | | | 33 | | | | | | | choice. | | E.1 | FactsExtractionPrompts | | | | . . . . 33 | | | | | | | | --- | --- | ------------------------- | --- | --- | --- | ---------- | ---------- | ----------------------------------- | ---------------- | ------- | --- | -------- | | | | | | | | | • Context: | Theamountofauxiliaryinformationpro- | | | | | | | E.2 | QuestionGenerationPrompts | | | | . 43 | | | | | | | | | | | | | | | vided, | ranging | from full source | context | | (used to | E.3 Response Instructions and measureencoding)tominimalcontextrequiredfor | | | GraderPrompts | | . . | . . . | . . . . 58 | disambiguation. | | | | | | | --- | --- | ------------- | --- | --- | ----- | ---------- | --------------- | --- | --- | --- | --- | --- | Verbatim: | | | | | | | | • | The | degree of linguistic | | overlap | with | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -------------------- | --- | ------- | ---- | thesourcetext,spanningexactphrasing(forcom- pletiontasks),high-verbatimquestionsthatreuse | A. | WikiProfile | | – Full | Details | | | | | | | | | | --- | ----------- | --- | ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | sourcephrasesandterms,andlow-verbatim,natu- raluser-likequeries. | In this | section, | we | describe | WikiProfile | | and its con- | Direction: | | | | | | | ------------------ | -------- | --- | ------------------------------- | ----------- | --- | ------------ | ---------- | ------------------------------------ | --- | --- | --- | --- | | | | | | | | | • | Therelationaldirectionofthequestion, | | | | | | structionindetail. | | | WikiProfilecomprisestentasksde- | | | | | | | | | | definedbytheorderingoftheentitiesinthesource | finedforeachfactextractedfromaWikipediasource | | | | | | | text. | | | | | | | --------------------------------------------- | --- | --- | --- | --- | --- | --- | ----- | --- | --- | --- | --- | --- | text;theirpropertiesandexamplesaresummarizedin Table1. Thetasksspanthreeformats:textcompletion, closed-bookquestionanswering,andmultiple-choice Tasks Eachfactisrepresentedasarelationbetween question answering. Two tasks measure encoding asubjectandanobjectentity,wherethesubjectpre- | | | | | | | | | | | | left | context | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ------- | (propositioncompletionandacontextualquestion), cedes the object in the source text. The andfourtasksmeasureknowledge(directandreverse consistsofalltextprecedingtheobjectentity. | questions,eachwithtwophrasings). | | | | | | Eachknowledge | | | | | | | | -------------------------------- | --- | --- | --- | --- | --- | ------------- | --- | --- | --- | --- | --- | --- | Weusetwotaskstomeasureencoding: taskadditionallyhasamultiple-choicevariant. 18 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Percentage of Wiki Page Creation Years 17.5 15.0 )%( egatnecreP 12.5 10.0 7.5 5.0 2.5 0.0 | | | 1 2 | 3 4 5 | 6 7 8 | 9 0 1 | 2 3 4 | 5 6 | 7 8 9 | 0 1 2 | 3 4 | 5 | | | --- | --- | ----------- | ----------- | ----------- | ------- | ----------- | ----------- | ------- | ----------- | ----------- | --- | --- | | | | 0 0 0 | 0 0 0 | 0 0 0 | 1 1 | 1 1 1 | 1 1 | 1 1 1 | 2 2 2 | 2 2 2 | | | | | | 2 0 2 0 2 0 | 2 0 2 0 2 0 | 2 0 2 0 2 0 | 2 0 2 0 | 2 0 2 0 2 0 | 2 0 2 0 2 0 | 2 0 2 0 | 2 0 2 0 2 0 | 2 0 2 0 2 0 | | | Figure9 | DistributionofWikipediapagesaccordingtotheiryearofcreation. • Completion: Propositioncompletionoftheleftcon- tosimplestructuralrelationsandmayinvolvemore | text. | | | | | | complexones(e.g.,“Whatistheconditionthatcaused | | | | | | | | ------------------- | ------------------------------- | --- | --- | --- | --- | ---------------------------------------------- | --- | --- | --- | --- | --- | --- | | Contextual: | | | | | | ErikaNordbytospendtwohourswithoutaheartbeat | | | | | | | | • | Ahigh-verbatimquestionposedover | | | | | | | | | | | | | thesameleftcontext. | | | | | | beforeshewasrevived?”). | | | | | | | Completioncanbeambiguousforpost-trainedLLMs, We therefore extract facts from Wikipedia pages, whichmaygenerateplausiblecontinuationswithout whichwerelikelyakeycomponentofLLMpre-training stating the target fact. The contextual question ex- data. Wesample10,000Wikipediapagesummaries plicitly elicits the fact while preserving the original (firstparagraphs),alongwithpagepopularitystatis- context,mitigatingthisissue. tics.4 Weretainonlydocumentswithatleast50words. ThedistributionofpagecreationdatesisshowninFig- Weusefourtaskstomeasureknowledge: | | | | | | | ure9. | PagesarethencategorizedintonineWikipedia | | | | | | | --- | --- | --- | --- | --- | --- | ----- | ---------------------------------------- | --- | --- | --- | --- | --- | VitalArticlecategories.5 • Direct: A closed-book question, whose answer is Thedistributionofthecate- goriesispresentedinFigure10. theobjectentity. | • DirectNatural: | | Aconversationalrephrasingofthe | | | | | | | | | | | | ---------------- | --- | ------------------------------ | --- | --- | --- | ------------------------------------ | --- | --- | --- | --- | --- | --------- | | | | | | | | Factextractionproceedsinthreestages. | | | | | | First,the | Directquestion. LLMperformsNER,identifyinguptoeightentitiesper • Reverse: An inverted question where the object documentandassigningoneof19predefinedtypes becomesthesubject. | | | | | | | (Box | E.2). | The purpose | of | this stage | is to | generate | | ------- | -------- | --- | --- | --- | --- | ---- | ----- | ----------- | --- | ---------- | ----- | -------- | | Reverse | Natural: | | | | | | | | | | | | • A conversational rephrasing of a diverse set of entities that can serve as answers. theReversequestion. The distribution of the entity types is presented in Figure11. Finally,wereformulateeachofthefourclosed-book questionsasmultiple-choicequestions withfouran- Second,theLLMselectsuptothreeentitiessuitable sweroptions. Thedistractorsaredesignedtobeplau- forpropositioncompletion. Suitabilityisdefinedby sibleandrealistic. Thesetasksevaluatesensitivityto whether the left context constrains the answer to a task format and, crucially, the ability to verify facts. single,specific,anduniquecorrectentity,suchthatal- Thisshiftfromgenerationtoverificationprobesadis- ternativeanswerswouldbefactuallyinconsistent. In tinctformoffactualknowledge. addition,theentityisneithertime-sensitivenortrivial, makingtheproposition-completiontaskchallenging. | | | | | | | For | additional | details, | including | | exact criteria | and | | --- | --- | --- | --- | --- | --- | --- | ---------- | -------- | --------- | --- | -------------- | --- | A.2. FactExtractionProcess | | | | | | | examples,seethepromptinBoxE.3. | | | | | Thisselection | | | ----------- | ---------- | ------------------------- | ---------- | -------- | ------ | ------------------------------ | ------ | -------- | --------- | ---------- | ------------- | -------- | | | | | | | | step | yields | 33,709 | candidate | object | entities. | Third, | | We aim | to extract | facts that | are likely | encoded | by | | | | | | | | | LLMs (to | enable | measurementofencodingeven | | | for | | | | | | | | | | | | | | | 4Popularity | | is based | on page | views from | 2023-02 | to 2025- | | small LLMs) | yet | remain non-trivial. | | Notably, | we in- | | | | | | | | 06,accordingtohttps://analytics.wikimedia.org/published/ tentionallyavoidextractingfactsinaschema-based datasets/country_project_page/00_README.html | tripletformat(subject,relation,object). | | | | Ourgoalis | | | | | | | | | | --------------------------------------- | --- | --- | --- | --------- | --- | --- | --- | --- | --- | --- | --- | --- | 5https://en.wikipedia.org/wiki/Wikipedia:Vital_ | togeneratenon-trivialquestionsthatarenotlimited | | | | | | articles/Level/3 | | | | | | | | ----------------------------------------------- | --- | --- | --- | --- | --- | ---------------- | --- | --- | --- | --- | --- | --- | 19 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Table1 | ThepropertiesofthetentasksinWikiProfile, filtered. FromtheDirectquestion,wealsogenerate | | | | | | Direct | Natural Contextual | | | | | --- | --- | --- | --- | --- | ------ | ------------------ | --- | --- | --- | includingexamples. Alltasksweregeneratedfromthe the and questions, which Wikipediapagefor‘LoveandMoney’. Theextracted aresubsequentlyfiltered. Similarly,wegenerateand factis: ‘ThebandLoveandMoneywasformedbythree filtertheReverseNaturalquestions. Finally,wegener- formermembersofFriendsAgain,alongwiththebassist | | | | | | atethemultiple-choicevariants. | | Below,wedescribe | | | | --- | --- | --- | --- | --- | ------------------------------ | --- | ---------------- | --- | --- | BobbyPaterson’. the three-stage process used to construct the Direct question,andthenelaborateontheremainingtypes. | Task | Format | Context | Verbatim | Direction | | | | | | | ---------- | ---------- | ------- | -------- | --------- | --- | --- | --- | --- | --- | | Completion | Completion | Full | Exact | Direct | | | | | | “LoveandMoneyareaScottishrock/soul/funkband,formedin1985in Generation TheDirectquestionsaregeneratedby | Glasgow,Scotland. | Thebandwasformedbythreeformermembersof | | | | | | | | | | ----------------- | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | instructingtheLLMtodesignaquestionbasedonthe FriendsAgain(singer-songwriterandguitaristJamesGrant,drummerStu- leftcontextsuchthattheonlycorrectansweristhe artKerrandkeyboardistPaulMcGeechan)alongwithbassist...” Contextual Closed-book Full High Direct objectentity. Thequestionusesonlytheinformation “LoveandMoneyareaScottishrock/soul/funkband,formedin1985in necessaryfordisambiguation(minimalcontext)while Glasgow,Scotland. Thebandwasformedbythreeformermembersof preservingexactphrasingfromthesourcetext(high | FriendsAgain(singer-songwriterandguitaristJamesGrant,drummerStu- | | | | | verbatim). | | | | | | ---------------------------------------------------------------- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | Wealsospecifyasetofrequirementsinthe artKerrandkeyboardistPaulMcGeechan).Whatisthenameofthebassist prompt(BoxE.7),includingensuringthattheques- who,alongwiththem,formedthebandLoveandMoney?” Direct Closed-book Minimal High Direct tionisself-containedanddoesnotincludepronouns. “Whatisthenameofthebassistwho,alongwiththreeformermembersof We then refine the Direct questions using two addi- FriendsAgain,formedthebandLoveandMoney?” tionalprompts: onetoimprovespecificityandoneto | MCDirect | Multiple-choice | Minimal | High | Direct | | | | | | | -------- | --------------- | ------- | ---- | ------ | --- | --- | --- | --- | --- | enforceminimalism. A.BobbyPatersonB.JamesGrantC.NeilCunninghamD.StuartKerr | DirectNatural | Closed-book | Minimal | Natural | Direct | | | | | | | ------------- | ----------- | ------- | ------- | ------ | --- | --- | --- | --- | --- | “WhowasthebassistthatformedthebandLoveandMoneywiththethree Refinement | guysfromFriendsAgain?” | | | | | | Thespecificationprompt(BoxE.8)in- | | | | | ---------------------- | --- | --- | --- | --- | --- | --------------------------------- | --- | --- | --- | MCDirNat Multiple-choice Minimal Natural Direct structs the LLM to evaluate a question’s specificity: A.BobbyPatersonB.JamesGrantC.NeilCunninghamD.StuartKerr whether it clearly defines the required level of de- | Reverse | Closed-book | Minimal | High | Reverse | | | | | | | ------- | ----------- | ------- | ---- | ------- | ----------- | -------------- | --------- | ------------- | --- | | | | | | | tail in the | answer, and to | revise it | if necessary. | The | “WhatisthenameofthebandthatwasformedbybassistBobbyPaterson LLMshouldrecommendrejectingaquestionifitcan- andthreeformermembersofFriendsAgain?” MCReverse Multiple-choice Minimal High Reverse not be fixed, either because revising it would leak A.LoveandMoneyB.FriendsAgainC.SettheToneD.DeaconBlue a non-trivial clue that makes the question too easy RevNatural Closed-book Minimal Natural Reverse or because the question is too vague and would re- “WhichbandwasformedbyBobbyPatersonwiththreeguysfromFriends | | | | | | quireafullrewrite. | Thisstagerejected45questions, | | | | | --- | --- | --- | --- | --- | ------------------ | ----------------------------- | --- | --- | --- | Again?” | MCRevNat | | | | | leavingDirectquestionsfor4955facts. | | | Theminimal- | | | -------- | --------------- | ------- | ------- | ------- | ----------------------------------- | --- | --- | ----------- | --- | | | Multiple-choice | Minimal | Natural | Reverse | | | | | | A.LoveandMoneyB.FriendsAgainC.SettheToneD.DeaconBlue ism prompt (Box E.9) instructs the LLM to revise a questionifitcontainsnon-essentialinformationthat candidatesareverifiedusingthreeprompts,andonly canberemovedwhilepreservingdisambiguationand | those deemed | suitable | are retained | (Boxes | E.4, E.5, | specificity. | | | | | | ------------ | -------- | ------------ | ------ | --------- | ------------ | --- | --- | --- | --- | E.6). Thisverificationprocessresultsin12,031enti- | ties(35.6%). | Finally,wedownsamplethedatasetto | | | | | | | | | | ------------ | -------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | Filtering 5,000entitiesbybalancingfactcategoriesandentity Somequestionsmaystillbevagueoram- | types. | | | | | biguous, | or admit multiple | answers, | according | to | | ------ | --- | --- | --- | --- | ---------------------------------- | ----------------- | -------- | ------------- | --- | | | | | | | sourcesoutsidetheWikipediacontext. | | | Tofilterthese | | cases,weuseGemini-2.5-Prowiththinkinggrounded A.3. QuestionGenerationProcess | | | | | | in Google | Search. Each | question | is answered | with | | --- | --- | --- | --- | --- | --------- | ------------ | -------- | ----------- | ---- | We use three stages of generation, refinement, and search grounding, and a prompted LLM (Box E.10) | | | | | | evaluates | whether the question | is | suitable. | If the | | --- | --- | --- | --- | --- | --------- | -------------------- | --- | --------- | ------ | filteringtoensurethatquestionssatisfytherequired groundedanswerrequestsclarification,providesmul- | properties. | Therefinementstageresolvesambiguity | | | | | | | | | | ----------- | ----------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | and enforces specificity, for example, by clarifying tiplepossibleanswers,orvariesbyinterpretation,the | | | | | | questionisrejected. | Thisprocessremoves10.6%of | | | | | --- | --- | --- | --- | --- | ------------------- | ------------------------- | --- | --- | --- | therequiredanswertypeandaddingminimaldisam- | | | | | | Directquestions. | Filteringappliestoallfivequestion | | | | | --------------------------- | --- | --- | ------------------- | --- | ---------------- | --------------------------------- | --- | --- | --- | | biguatingcontextwhenneeded. | | | Thefilteringstageis | | | | | | | types: onceafactisdiscardedduetoanyofitsques- | grounded | in Google Search | to | recall plausible | alter- | | | | | | | -------------- | ---------------- | ------- | ---------------- | ---------- | -------------------------- | --- | ------------------ | --- | --- | | | | | | | tions,itisnotreconsidered. | | Thenumberofremoved | | | | native answers | and to | discard | questions | that admit | | | | | | multipleanswers. questionsisreportedintherelevantparagraphsbelow. | We first | apply these | three stages | to construct | the | | | | | | | -------- | ----------- | ------------ | ------------ | --- | --- | --- | --- | --- | --- | Directquestion. BasedontheDirectquestion,wegen- ReverseQuestions Forreversequestions,theobject eratetheReversequestion,whichisthenrefinedand entitybecomesthenewsubject,andthesubjectentity 20 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | | | | | Science | | | | | | ARTIFACT | | | | | --- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | -------- | --- | --- | --- | People | | | | | 12.6% | | | | | | 16.0% | | | | | --- | --- | --- | --- | ----- | --- | --- | --- | --- | --- | ----- | --- | --- | --- | DESCRIPTOR 11.9% | | | | | | | Arts | | 12.4% | | | | | PERSON | | --- | --- | --- | --- | --- | --- | ---- | --- | ----- | --- | --- | --- | --- | ------ | 16.6% 15.8% Society 11.7% | | | | | | | | LOCATION | 11.6% | | | | 2.8% | OTHER | | --- | --- | ----- | --- | --- | ---- | ---------- | -------- | ----- | --- | --- | --- | ---- | ------ | | | | | | | 7.3% | Philosophy | | | | | | 2.8% | | | | | | | | | & Religion | | | | | | | NUMBER | | | | 11.2% | | | | | | | | | | 5.2% | | Everyday Life | | | | | | 8.5% | | | | 11.5% | | | | DATE | | --- | --- | --- | --- | --- | ---- | --- | --- | --- | ----- | --- | --- | --- | ---- | 5.9% Geography | | | 11.1% | | | | | ORGANIZATION | | | | 6.6% | | | | --- | --- | ----- | --- | ---- | --- | --- | ------------ | --- | --- | ---- | ---- | -------- | --- | | | | | | 9.9% | | | | | | 8.5% | | BIO_CHEM | | Technology | | | | | History | | | | | EVENT | | CONCEPT | | | | ------------------------------------ | -------------------------------------- | --- | --- | ------- | --- | -------- | ----------------------------------------- | ------- | ----------- | --- | -------- | ------- | -------- | | | | | | | | | | Figure11|Distributionofobjectentitytypes. | | | | | | Forthis | | Figure10 | DistributionofWikipediapagecategories. | | | | | | | | | | | | | | | | | | | | | figure,wegroupedtogethersometypes: | | | | | | ARTIFACT | | | | | | | | | (WORK | OF ART, | PRODUCT), | | BIO_CHEM | | (ORGAN- | | oftheDirectquestionbecomestheanswer. | | | | | | Giventhe | | | | | | | | | | | | | | | | ISM, CONDITION, | | SUBSTANCE), | | | CONCEPT | (IDEA, | leftcontext,theDirectquestion,andthetargetanswer, weinstructtheLLMtoidentifythesubjectentityand LANGUAGE,LAW),DESCRIPTOR(CLASSIFICATION, GROUP,IDENTIFIER,TITLE). generateareversequestionthatsatisfiesallrequired | properties | (Box | E.11). | In | some cases, | additional | in- | | | | | | | | | ---------- | ---- | ------ | --- | ----------- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | thefinalincompletesentence)withthehigh-verbatim formationfromtheleftcontextisrequiredtoensure thatthereversequestionisunambiguousandhasa direct question, with minor edits for fluency and to | single | correct | answer. | For | example, | consider | the di- | avoidredundancy. | | | | | | | | ------ | ------- | ------- | --- | -------- | -------- | ------- | ---------------- | --- | --- | --- | --- | --- | --- | “Atwhicheventdidthefilm‘There’sStill rectquestion: WethenrepeattheGoogleSearch–basedfiltering | Tomorrow’premiere?” | | | withtheanswer‘The18thRome | | | | | | | | | | | | ------------------- | --- | --- | ------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | fortheDirectNatural,ReverseNatural,andContextual | FilmFestival’. | | Thesubjectentityisthefilm‘There’sStill | | | | | | | | | | | | | ---------------------------------------- | --------------------------------- | -------------------------------------- | ----- | ----- | ---- | --------- | -------------- | ----------------------------------- | --- | --------- | -------- | ---------- | ------- | | | | | | | | | questions. | Thisremovesanadditional5.5%offacts, | | | | | | | Tomorrow’. | Anaivereversequestionsuchas“Which | | | | | | | | | | | | | | | | | | | | | leaving | 2357 facts | for | which | all five | question | types | | filmpremieredatthe18thRomeFilmFestival?” | | | | | | isun- | | | | | | | | | | | | | | | | are available. | Finally, | | we remove | | facts with | lengthy | | suitable, | since | multiple | films | could | have | premiered | | | | | | | | questions(over40words)orshortleftcontexts(below | there. | The | LLM should | therefore | | incorporate | addi- | | | | | | | | | ------ | --- | ---------- | --------- | --- | ----------- | ----- | --- | --- | --- | --- | --- | --- | --- | 30words),yielding2200facts. tionalcontext,suchasthedirector’sname,yielding: “WhichfilmdirectedbyPaolaCortellesipremieredatthe | 18thRomeFilmFestival?” | | | | | | | A.4. ManualQualityValidation | | | | | | | | ---------------------- | --- | --- | --- | --- | --- | --- | ---------------------------- | --- | --- | --- | --- | --- | --- | TheLLMmayalsorejectgeneratingareverseques- To ensure high data quality, we manually validate | tion if | it cannot | do so | without | making | it | trivial (see | | | | | | | | | ------- | --------- | ----- | ------- | ------ | --- | ------------ | --------- | ----------- | --- | ------------ | --- | -------- | ------ | | | | | | | | | questions | by adapting | | the protocol | | of Nahum | et al. | thefourthexampleinBoxE.11). Thisoccursin2.2% (2024),whichprioritizesinstancesinwhichanLLM of cases. After generation, we apply the refinement ensemblefails. Specifically,wemanuallyexaminea stage,whichimprovesspecificationandenforcesmin- factanditstenassociatedquestionsifthefourfrontier imalism,rejectinganadditional3.6%ofReverseques- LLMs(Gemini-3-Pro/Flash,GPT-5,andGPT-4.1;eval- | tions. | Wethenproceedtofiltering. | | | | Beforeapplying | | | | | | | | | | ------ | ------------------------- | --- | --- | --- | -------------- | --- | ---------- | --- | ------- | --------- | --- | ----------- | ---- | | | | | | | | | uated with | and | without | thinking) | | unanimously | fail | GoogleSearch–groundedfiltering,weverifythatthe tocorrectlyansweratleastoneofthequestions,i.e., DirectandReversequestionsformavalidpairandare 𝑔(𝑞) ≤ 0.5. This process flagged 237 facts (10.7%). | non-trivial,usingthepromptinBoxE.12. | | | | | | Thisstep | | | | | | | | | ------------------------------------ | --- | --- | --- | --- | --- | -------- | ----------- | --- | ------- | ------------- | --- | -------- | ----- | | | | | | | | | Upon manual | | review, | we identified | | 43 (<2%) | facts | removes 31.5% of facts. Finally, the filtering stage andtheirassociatedquestionsaslowquality,andthey removesanadditional13.1%offacts. | | | | | | | | weresubsequentlyremovedfromthebenchmark. | | | | | | To | | --- | --- | --- | --- | --- | --- | --- | ---------------------------------------- | -------- | --- | -------- | -------- | --- | ----------- | | | | | | | | | further | validate | our | data, we | examined | | a subset of | 50factsinwhichthreeofthefourmodelsprovided | Natural | and | Contextual | | Questions | | | | | | | | | | | ------- | --- | ---------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | We instruct | | | | | | | | incorrectanswers. | | Wefoundonly3errorsinthissub- | | | | | | ------- | --- | ------- | ---- | ---------- | ------- | ----- | ----------------- | ----- | ---------------------------- | ------- | ------- | ---- | ----------- | | the LLM | to | rewrite | each | Direct and | Reverse | ques- | | | | | | | | | | | | | | | | set. Together, | these | | results | suggest | that | low-quality | tionasDirectNaturalandReverseNatural,following TheContextualquestionisconstructedus- questionsareuncommonintheremainingdataand BoxE.13. thatthebenchmarkmeetsahighstandard. | ingBoxE.14.6 | | Itcombinestheleftcontext(excluding | | | | | | | | | | | | | ------------ | --- | ---------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 6Wedonotgeneratecontextualversionsforreversequestions,as thesetendtobetooeasyandthecontextoftenincludestheanswer, frequentlythepagetitle. 21 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | B. Thinking | | and | Recall | | | | | | | | | | | | ----------- | --- | --- | ------ | --- | --- | --- | --------- | --- | ---------- | ------- | ------- | ---- | -------- | | | | | | | | | inference | | to produce | correct | answers | even | for non- | encodedfacts. In§5.3,wearguedthatthinkingprimarilyfacilitates recallofencodedknowledge. Herewepresentthefull (iii)Recallfacilitation. Thinkingmayhelpthemodel analysis,includingformaldefinitions,detailedresults locateandrecallafactthatisencodedbutnotacces- acrossallmodels,andadiscussionoflimitations. sible via direct generation. The fact is stored in the model’sparameters,asverifiedbytheencodingtasks, B.1. ThreePossibleMechanisms butthebarequestionfailstoactivateitsrecall. Think- ingprovidesadditionalcomputationthatbridgesthis | We consider | three | mechanisms | | by | which thinking | | gap. | | | | | | | | ----------- | ----- | ---------- | --- | --- | -------------- | --- | ---- | --- | --- | --- | --- | --- | --- | mayimprovefactualperformanceonourbenchmark. | | | | | | | | Example: | | An | LLM can | complete | the | relevant | | --- | --- | --- | --- | --- | --- | --- | -------- | --- | --- | ------- | -------- | --- | -------- | Eachmechanismmakesdistinctempiricalpredictions, Wikipedia proposition about Oasis’s first gig when whichweevaluateinthesubsequentsections. | | | | | | | | giventheleftcontext. | | | However,whenasked“Where | | | | | --------------------- | --- | ---------- | ------------------------- | --------- | ---- | ------ | ---------------------------- | ---------- | ------ | ----------------------- | ------------------ | --------- | --- | | | | | | | | | didOasisplaytheirfirstshow?” | | | | withoutcontext,di- | | | | (i)Responsediversity. | | | Thinkingmayaltertheoutput | | | | | | | | | | | | | | | | | | | rect | generation | fails. | With | thinking, | the model | can | | distribution | by | increasing | its | variance. | With | higher | | | | | | | | generateintermediatecontext(e.g.,recallingdetails | variance, | the probability | | that | at least | one of | 𝑛 sam- | | | | | | | | | --------- | --------------- | --- | ---- | -------- | ------ | ------ | --- | --- | --- | --- | --- | --- | --- | abouttheband’searlyhistoryinManchester)andper- plesiscorrectincreases–asamplingeffectanalogous formadditionalcomputationsthatprimerecallofthe | to the relationship | | between | | response | diversity | and | | | | | | | | | ------------------- | --- | ------- | --- | -------- | --------- | --- | --- | --- | --- | --- | --- | --- | --- | storedanswer. | pass@𝑘 | (Wang | et al., | 2023; | Yang et | al., 2025). | Im- | | | | | | | | | ------ | ----- | ------- | ----- | ------- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- | portantly,thismechanismdoesnotreflectimproved Prediction:Recallfacilitation,bydefinition,requires access to knowledge; rather, it is a statistical conse- thetargetfacttobeencoded. Itshouldthereforeben- quenceofbroaderexplorationoftheoutputspace. | | | | | | | | efitonlyencodedfacts. | | | Unlikeresponsediversity,it | | | | | -------- | --- | --- | --- | --- | --- | --- | --------------------- | ------- | ---- | -------------------------- | --- | ----------- | ----- | | Example: | | | | | | | can | improve | both | availability | and | robustness, | since | Withoutthinking,amodelasked“Where successfulrecallofastoredfactyieldsconsistentcor- | did Oasis | play their | first | show?” | might | produce | the | | | | | | | | | --------- | ---------- | ----- | ------ | ----- | ------- | --- | --- | --- | --- | --- | --- | --- | --- | rectness. | sameincorrectanswerin8outof8samples. | | | | | | Think- | | | | | | | | | ------------------------------------ | --- | --- | --- | --- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | ingtrajectories,byincorporatingadditionalsampling steps,mayleadtoadditionalanswers,somecorrect, | | | | | | | | B.2. | MeasuringAvailabilityandRobustness | | | | | | | --------- | --------- | ---- | ----- | ---- | ---------------- | --- | ---- | ---------------------------------- | --- | --- | --- | --- | --- | | most not, | yielding, | say, | 2 out | of 8 | correct samples. | | | | | | | | | Availabilityofthecorrectanswerhasimproved(from Tooperationalizethedistinctionbetweenavailability androbustness,weproposecorrect≥𝑝,ametricthat 0to2correct),butrobustnesshasnot:themodeldoes notreliablyconvergetothecorrectanswer. capturesbothaspectsalongacontinuousspectrum. It isdefinedastheproportionofquestionswhoseesti- | Prediction: | | | | | availabil- | | | | | | | | | | ----------- | -------- | --- | --------- | --------- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | | | Response | | diversity | increases | | | | | | | | | | matedprobabilityofcorrectnessacrosssamplesisat ity(thechanceofatleastonecorrectansweracross | | | | | | | | least | 𝑝 | | | | | | | -------- | ---------- | --- | -------- | ---------- | ----------- | --- | ----- | --- | --- | --- | --- | --- | --- | | samples) | but cannot | | increase | robustness | (consistent | | | | | | | | | Foreachquestion,wedraw𝑛independentsamples | correctness), | since | higher | variance | | cannot produce | | | | | | | | | | ------------- | ----- | ------ | -------- | --- | -------------- | --- | --- | --- | --- | --- | --- | --- | --- | reliableconvergencetothesamecorrectoutput. and observe 𝑘 correct responses, yielding an empir- | | | | | | | | ical | correctness | rate | ˆ𝑝 = | 𝑘/𝑛. We then | estimate | the | | -------------- | ---------------------------------- | --- | --- | --- | --- | --- | ----------- | ----------- | -------- | ---------- | ------------ | ------------- | --- | | (ii)Inference. | Thinkingmayenablemulti-stepderiva- | | | | | | | | | | | | | | | | | | | | | probability | | that, in | a repeated | run | of 𝑛 samples, | at | tionfromotherencodedfacts. Bychainingtogether leasta 𝑝fractionoftheanswerswouldbecorrect. Let relatedknowledge,theLLMcanproduceananswer 𝑚=⌈𝑝𝑛⌉. Usingabinomialmodel𝐾 ∼Binomial(𝑛,ˆ𝑝), | thatitdoesnotstoredirectly. | | | | Thismechanismrelies | | | | | | | | | | | --------------------------- | --- | --- | --- | ------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | then: | ontheavailabilityofrelatedencodedpremises, | | | | | | not | | | | | | | | | ------------------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ontheencodingstatusofthetargetfactitself. 𝑛 (cid:18)𝑛(cid:19) | | | | | | | | | | 𝑃ˆ(𝐾 | ∑︁ | ˆ𝑝𝑗(1−ˆ𝑝)𝑛−𝑗. | | | | ----------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | ------------- | --- | --- | | | | | | | | | | | ≥𝑚) | = | | | | | Example:AmodelmaynotencodethatOasisplayed | | | | | | | | | | | 𝑗 | | | 𝑗=𝑚 | their first | gig at | the | Boardwalk. | However, | through | | | | | | | | | | ----------- | ------ | --- | ---------- | -------- | ------- | --- | --- | --- | --- | --- | --- | --- | --- | thinking, it might chain together other related en- Aggregatingoverquestions,correct≥𝑝measuresthe coded facts: “Oasis formed in Manchester” → “the fractionofquestionsforwhichthisestimatedproba- Boardwalk was a famous music club in Manchester bilityexceeds0.5,i.e., 𝑃ˆ(𝐾 ≥ ⌈𝑝𝑛⌉) >0.5. | during | the 1990s” | → | “the Boardwalk | | hosted | early | | | | | | | | | ------ | ---------- | --- | -------------- | --- | ------ | ----- | --- | --- | --- | --- | --- | --- | --- | Themetricmakesthetrade-offbetweenavailability | gigsbyemergingbands.” | | | Thisinferenceproducesthe | | | | | | | | | | | | --------------------- | --- | --- | ------------------------ | --- | --- | --- | --- | ---------- | ------ | --- | ---------- | ---- | ---------- | | | | | | | | | and | robustness | clear. | At | low values | of 𝑝 | (e.g., 𝑝 = | correctanswerwithoutdirectrecall. 0.125, | | | | | | | | | corresponding | | to at | least 1 | correct | out of 8), | | --- | --- | --- | --- | --- | --- | --- | --- | ------------- | --- | ----- | ------- | ------- | ---------- | Prediction:Inferencegainsareregardlessofwhether themetricaskswhetherthemodelcanproduceany the specific target fact is encoded, since it operates correctanswer,whichisanalogoustopass@𝑘. Athigh overotherrelatedencodedfacts. Wethereforeexpect valuesof𝑝(e.g., 𝑝=0.875),itaskswhetherthemodel 22 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality iscorrectalmostallofthetime.7 • Thinkinggainsconcentratemostlyonencodedfacts, consistentwithrecallfacilitation(Figure8). | | | | | • The facts | in our benchmark | are single-hop: | com- | | --- | --- | --- | --- | ----------- | ---------------- | --------------- | ---- | B.3. RulingOutResponseDiversity plexdeductivereasoningisunnecessarytoderive | | | | | the correct | answer, | making inference | a less parsi- | | --- | --- | --- | --- | ----------- | ------- | ---------------- | ------------- | Figure12presentscorrect≥𝑝forall12LLMs,compar- moniousexplanationthanrecallofanencodedfact. ingperformancewithandwithoutthinkingacrossthe | full range | of thresholds. | At low thresholds | (𝑝<0.5, | | | | | | ---------- | -------------- | ----------------- | ------- | --- | --- | --- | --- | corresponding to Availability), thinking improves B.5. TheRoleofThinking-SpecificTraining | correct≥ | 𝑝 | | | | | | | | -------- | ------------- | ------------------ | -------- | --- | --- | --- | --- | | | for all LLMs. | This is consistent | with all | | | | | three mechanisms, including response diversity. At Ourevaluationpoolstwodistinctimplementationsof high thresholds (𝑝>0.5, corresponding to Robust- thinking: nativethinking,wherethemodelwasopti- ness),thepicturedivergesacrossmodelfamilies. | | | | For | mizedduringtrainingtousethinkingtokens(Gemini- | | | | | --- | --- | --- | --- | ---------------------------------------------- | --- | --- | --- | thinking-optimized LLMs (Gemini-3-Pro, Gemini-3- 3, Gemini-2.5, GPT-5 families), and CoT prompting, | | | | | where thinking | is elicited | at test time | without dedi- | | --- | --- | --- | --- | -------------- | ----------- | ------------ | ------------- | Flash,Gemini-2.5-Pro,GPT-5,GPT-5.2,GPT-5-mini), thinking also improves robustness: the +Thinking catedtraining(Gemma3,GPT-4.1). Theresultssug- curve remains above the Base curve across the full gestthattheseimplementationsyieldqualitativelydif- 𝑝. rangeof Thispatternisincompatiblewithresponse ferenteffects. Nativethinkingconsistentlyimproves diversityasthesoleexplanation,sinceincreasedout- bothavailabilityandrobustness,consistentwithrecall | | | | | facilitation. | CoTpromptingproducesmixedresults: | | | | ------------ | ------ | --------------------------- | --- | -------------------------------------------- | --------------------------------- | --- | --- | | put variance | cannot | produce greater consistency | in | | | | | | correctness. | | | | forGemma3models,itimprovesavailabilitybutde- | | | | gradesrobustness,consistentwitharesponse-diversity ForGemma3models(4B,12B,27B),thepatternre- thinking(elicitedviaCoTprompting)degrades effect;forGPT-4.1,ityieldsmodestgainsinboth. verses: robustness,withthecurvescrossingatintermediate Thisdistinctionhaspracticalimplications. There- 𝑝. GPT-4.1occupiesamiddleground: CoTprompting callfacilitationmechanism,inwhichthinkinghelps yieldsmodestgainsinrobustness. Ourresultssuggest themodelaccessencodedknowledge,maydependon that,unlikethinking-optimizedLLMs,CoTprompting thinking-specifictrainingratherthanmerelyonpro- mainlyincreasesresponsediversity,anditseffective- ducingadditionaltokensatinferencetime. Ifso,the nessinenhancinginferenceorrecallmaydependon benefitsofthinkingforfactualityarenotafreelunch modelcapacityortraining. obtainablethroughpromptingalone;theyrequireded- | | | | | icated training | investment. | We note, | however, that | | --- | --- | --- | --- | --------------- | ----------- | -------- | ------------- | ourexperimentaldesigncannotfullydisentanglethe B.4. EvidenceforRecallFacilitation effectsofthinking-specifictrainingfromotherdiffer- encesbetweenmodelfamilies(e.g.,modelsize,data, | Having | ruled out response | diversity | for thinking- | | | | | | ------ | ------------------ | --------- | ------------- | ------------------------- | --- | ---------------------- | --- | | | | | | post-trainingprocedures). | | Thepatternissuggestive | | optimizedLLMs,wenowaskwhethertheremaining butnotconclusive. gainsreflectinferenceorrecallfacilitation.Figure8re- ports,foreachmodel,thefractionoffactsthatbecome knownonlywiththinking,conditionedonencoding C. Additional Results status. The results show a strong dependence on encod- In this section, we clarify how each analysis, figure, ing: forthinking-optimizedLLMs,thinkingrecovers andtableiscomputed,andreportresultsforalleval- 40–65% of of encoded but not directly known facts, uatedLLMstosupportthemain-texttrends. | comparedtolessthan20%ofnon-encodedfacts. | | | This | | | | | | ---------------------------------------- | --- | --- | ---- | --- | --- | --- | --- | dependenceisexpectedunderrecallfacilitation;how- C.1. Profiles | ever,itdoesnot,byitself,ruleoutinference. | | | Nonethe- | | | | | | ----------------------------------------- | --- | --- | -------- | --- | --- | --- | --- | less,thecumulativeevidencefavorsrecallfacilitation Figure4presentstheprofiledistributionsforthe13 astheprimarymechanismbywhichthinkingimproves | | | | | LLMs, and | Figure 13 | provides a line plot | that high- | | --- | --- | --- | --- | --------- | --------- | -------------------- | ---------- | thefactualityofthinking-optimizedLLMs: lightstrends.Toestimatethedistributions,weexclude factswithnon-gradablequestions(seeAppendixD.1), • Thinkingimprovesrobustness,rulingoutdiversity | (§B.3). | | | | thenassigneachfactaprofileandcomputetheshare | | | | | ------- | --- | --- | --- | -------------------------------------------- | ----------------------------- | --- | --- | | | | | | ofeachprofile. | TheKnowsvaluesumstheDirectre- | | | 7With 𝑛=8samples, correct≥ 𝑝isinsensitivetochangesat call,recallwithThinking,andInferencewithoutThink- | verylowcorrectnessprobabilities(roughly | | 𝑝≲0.1), | sincesuch | | | | | | --------------------------------------- | --- | ------- | --------- | ------------- | ------------ | ---------------------- | --- | | | | | | ing profiles. | It therefore | slightly overestimates | the | casesfrequentlyyieldzerocorrectsamples.Thus,availabilitygains | | | | | LLM’s actual | performance | (with thinking | enabled), | | --- | --- | --- | --- | ------------ | ----------- | -------------- | --------- | intheextremelow-probabilitytailmaybeunderestimated,while robustnessathigh𝑝remainswell-estimated. sincesomefactsbecomeunknownwhenthinkingis 23 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | | Gemini-3-Pro | | | | Gemini-3-Flash | | | Gemini-2.5-Pro | | | | Gemma3-27b | | | --- | ------------ | --- | --- | ---- | -------------- | --- | --- | -------------- | --- | --- | --- | ---------- | --- | | 0.9 | | | | | | | 0.9 | | | | 0.7 | | | | p | | | | 0.90 | | | | | | | | | | ≥ | tcerroc | | | | | | | | | | | 0.6 | | | | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.8 | | | | | | | 0.8 | | | | | | | 0.85 0.5 | 0.7 | available | p | robust | | | | 0.7 | | | | | | | | --- | ---------- | --- | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ← → 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 | | | GPT-5 | | | | GPT-4.1 | | | GPT-5.2 | | | Gemma3-12b | | | --- | --- | ----- | --- | --- | --- | ------- | --- | --- | ------- | --- | --- | ---------- | --- | | | | | | | | | 0.9 | | | | 0.6 | | | 0.9 | p | | | | 0.85 | | | | | | | | | | | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ≥ 0.8 tcerroc 0.8 0.80 0.4 | 0.7 | | | | 0.75 | | | 0.7 | | | | | | | | --- | ---------- | --- | ------- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | available | p | robust | | | | | | | | | | | ← → 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 | | | GPT-5-mini | | | GPT-4.1-mini | | | Gemini-2.5-Flash | | | | Gemma3-4b | | | --- | --- | ---------- | --- | --- | ------------ | --- | --- | ---------------- | --- | --- | --- | --------- | --- | 0.8 | p 0.8 | | | | | | | 0.8 | | | | 0.4 | | | | ----- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ≥ tcerroc | 0.6 | | | | 0.6 | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | 0.6 | | | | 0.2 | | | available p robust available p robust available p robust available p robust | | | ← → | | | | ← → | | | ← → | | | ← → | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 | | | | | | | +Thinking | | | Base | | | | | | --- | --- | --- | --- | --- | --- | --------- | --- | --- | ---- | --- | --- | --- | --- | Figure12|Wereport𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑝for12LLMs,definedastheproportionofquestionswhoseestimatedprobability ≥ of correctness across samples is at least 𝑝. Low values of 𝑝 reflect availability (the model produces at least onecorrectanswer),whilehighervaluesreflectrobustness(themodeliscorrectconsistently). DifferentLLMs exhibitdistinctpatterns;eachcolumncorrespondstoadifferentpattern. C.2. FactPopularity | | Encoding Failure | | | Direct Retrieval | | Inf w/o Enc | | | | | | | | | ---- | ----------------- | --- | --- | ------------------- | --- | ----------- | ------------------------------------------- | --- | --- | --- | --- | --- | --- | | | Retrieval Failure | | | Retrieval +Thinking | | Knows | | | | | | | | | 100% | | | | | | | Popularitytiers(bottom20%,top20%)aredefined | | | | | | | 83.886.087.5 | | | | | | | | based | on | Wikipedia | page | visit counts | for | the pages | | --- | --- | --- | --- | --- | --- | --- | ----- | --- | --------- | ---- | ------------ | --- | --------- | 75.676.379.4 80% | | | | 64.767.0 | | | | fromwhichfactsareextracted. | | | | | Thisservesasarea- | | | --- | --- | --- | -------- | --- | --- | --- | --------------------------- | --- | ------------- | --- | --------- | ----------------- | --- | | 60% | | | 56.9 | | | | sonable | | but imperfect | | proxy for | fact incidence, | and | 45.3 | | | | | | | | we | expect | the observed | | trends | to be even | stronger | | --- | --- | --- | --- | --- | --- | --- | --- | ------ | ------------ | --- | ------ | ---------- | -------- | 40% | | | | | | | | withamoredirectmeasure. | | | | Figure16comparesthe | | | | --- | --- | --- | --- | ---- | --- | ------------ | ------------------------ | --- | ---------- | ---- | --------------------- | --- | ------------ | | 20% | | | | | | | encoding | | and recall | gaps | between | the | two popular- | | | | | | | | | itytiersacrossall13LLMs. | | | | Encodingperformanceis | | | | 0% | b b | b b | i h | i .2 | 1 o | 5 sh i-3-Pro | | | | | | | | 3 -1 3- 4 -1 2 2 7 m i n a s 5-mi n -5 -4 . P r P T - la m e a s u re d a s t he p ro p o r t i o n o f e n c o d e d f a c t s w i th in | Gemm a | m a a 3 | a 3- . 1 - | 5 -F l - | G P T G P | T 2 .5- G | 3 - F in | | | | | | | | | ------ | ------- | ---------- | -------- | --------- | --------- | -------- | --- | --- | --- | --- | --- | --- | --- | e m m m m m P T - 4 i -2 . G P T i n i- in i - m ea c h t ie r, w h il e re ca ll p e r f o r m a n c e i s m e a s u r e d a s th e | G | G e G e | G m i n | | e m | e m | G e | | | | | | | | | --- | ------- | ------- | --- | --- | --- | --- | ---------- | --- | -------- | ----- | ----------- | --- | ------------ | | | | G e | | G | G | | | | | | | | | | | | | | | | | proportion | | of known | facts | conditioned | | on being en- | coded.Thisconditioningisolatesrecallfromencoding, | Figure | 13 | Trends | | Identified | by | Profiles: | ‘Knows’ | | | | | | | | | ------ | ----------- | --- | ---------- | --- | --------- | ------- | -------- | --- | ------------ | ------- | ---- | ----- | ---------- | | | | | | | | | allowing | | us to assess | whether | rare | facts | are harder | equaltothesumof‘DirectRecall’(greenline),‘Recall torecallevenwhentheyareencoded. Asshown,for with Thinking’ (light green shade), and ‘Inference mostLLMs(10outof13;excludingGemma312b,4b, | withoutEncoding’(lightblueshade). | | | | | LLMsaresorted | | | | | | | | | | --------------------------------- | --- | --- | --- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- | --- | and1b),therecallgapissubstantiallylargerthanthe | accordingto‘Knows’. | | | Figure4presentsdistributions | | | | encodinggap. | | | | | | | | ------------------- | --- | --- | ---------------------------- | --- | --- | --- | ------------ | --- | --- | --- | --- | --- | --- | oftheprofiles. C.3. Phrasing AlargebodyofworksuggeststhatLLMsaresensitive enabled. Assuch,KnowscanbeviewedasanLLM’s topromptphrasing(Dengetal.,2023;Habbaetal., knowledgepotential. 2025;Linetal.,2024b;Lioretal.,2025;Mizrahietal., 24 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality 2024). Incontrast,in§5.2,wefindthatLLMsarein- tions. Asscaleincreases,theshareof‘both’decreases, sensitive to question rephrasing in our setup. Here and‘onlyreverse’becomesthedominantfailuremode we elaborate on this analysis. We conduct hypothe- (40-50%are‘onlyreverse’andanadditional15-30% | sistestscomparingthedistributionsofperformance | | | | | are‘both’). | | | | | | | | ---------------------------------------------- | --- | --- | --- | --- | ----------- | --- | --- | --- | --- | --- | --- | acrossphrasings,treatingsuccessasabinaryvariable | (1ifthequestiongradeexceeds0.5,0otherwise). | | | | In | | | | | | | | | ------------------------------------------- | --- | --- | --- | --- | ------------- | --- | --- | --- | --- | --- | --- | | | | | | | C.5. Thinking | | | | | | | total,weperform104tests,covering13LLMs(with | and without | thinking) | and four question | | pairs (di- | | | | | | | | | ----------- | --------- | ----------------- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | Figure17examinestheimpactofinference-timethink- rect vs. direct natural, reverse vs. reverse natural, ingonrecallacross13LLMs,focusingonpopularity | and their | multiple-choice | variants). | After | applying | | | | | | | | | --------- | --------------- | ---------- | ----- | -------- | ------------------------------- | --- | --- | --- | ----------------- | --- | --- | | | | | | | anddirectionality,respectively. | | | | Inbothanalyses,we | | | FalseDiscoveryRate(FDR)correctionusingtheBen- conditiononencodedfactsanddecomposerecallinto jamini–Yekutieliprocedurewith𝑞=0.05(Benjamini baselineperformance(factsknownwithoutthinking, andYekutieli,2001),noneofthetestsaresignificant. giventheyareencoded)andtheadditionalsharethat This indicates that, in our setup, phrasing does not becomesknownonlywiththinking. Asshown,think- confoundknowledgeassessment. ingmitigatesthepopularitygapfor8outof13LLMs | | | | | | (excluding | GPT-4.1-mini | | | and the | Gemma3 | models) | | --- | --- | --- | --- | --- | ---------- | ------------ | --- | --- | ------- | ------ | ------- | andthedirectionalgapfor9outof13LLMs(exclud- C.4. ReverseQuestions | | | | | | ingGemini-2.5-FlashandGemma3models). | | | | | | Inboth | | --- | --- | --- | --- | --- | ------------------------------------ | --- | --- | --- | --- | --- | ------ | Figure16reportsrecallperformanceconditionedon cases,themitigationissubstantialforfrontierLLMs, thatis,thestrongerproprietarymodels. | facts being | encoded, | for both multiple-choice | | and | | | | | | | | | ---------------------- | ---------- | ------------------------ | ----------- | --------- | ----------------- | --- | ------------ | --- | -------------- | --- | ----------- | | closed-book | questions, | under direct | and reverse | di- | | | | | | | | | rections,across13LLMs. | | Eachbarshowsthefraction | | | | | | | | | | | | | | | | D. Methodological | | | | and Design | | Choices | | of encoded | facts that | are known under | a | given di- | | | | | | | | | rectionandtaskformat. | | Afactiscountedasknown | | | | | | | | | | | | | | | | The goal | of | this section | | is to describe | | and justify | underthedirect(respectively,reverse)conditiononly ifthegradesforbothcorrespondingquestions(direct methodologicaldetailsanddesignchoicesthatinflu- and direct natural; reverse and reverse natural) ex- enceourevaluationandanalysis. | ceed𝑔(𝑞) | >0.5. Thesamecriterionisappliedtothe | | | | | | | | | | | | ------------------------ | ------------------------------------ | ------------------------ | --- | --- | ------------------------------------ | --- | --- | --- | --- | --- | --- | | multiple-choicevariants. | | Figure16showsthat,forall | | | | | | | | | | | | | | | | D.1. HandlingPartiallyandOtherGrades | | | | | | | LLMs,thegapbetweendirectandreversequestionsis muchsmallerforverificationthanforgeneration. For ThequestiongradeiscomputedasaccuracyoverCOR- 9outof13LLMs,reverseverificationisinfacteasier. RECTandINCORRECTresponsesonly,excludingre- sponseslabeledPARTIALLYorOTHER.Forexample, Ouranalysiscomparingdirectandreversequestions | | | | | | if a question | | receives | 3 | CORRECT, | 1 INCORRECT, | | | --- | --- | --- | --- | --- | ------------- | --- | -------- | --- | -------- | ------------ | --- | focusesonencodedfacts,asthisallowsustoisolate | | | | | | 2 PARTIALLY, | | and | 2 OTHER | responses, | | its grade is | | --------------- | --------------------------------- | --------- | -------- | ----- | ------------ | ------------------------------ | --- | ------- | ---------- | --- | ------------ | | recallfailures. | Onemightarguethatthisconditioning | | | | | | | | | | | | | | | | | 3 =0.75. | WeexcludePARTIALLYandOTHERfrom | | | | | | | introduces | selection bias, | since the | encoding | tasks | 3+1 | | | | | | | gradingbecauseassigningthemweightsissubjective | sharethesameanswerentityasdirectquestions. | | | | We | | | | | | | | | ------------------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | (Kalaietal.,2025). | thereforeconductseveralrobustnessanalyses. | | | | First, | | | | | | | | | ------------------------------------------ | --- | --- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | allkeyresultsreplicatewhenwedonotconditionon We next describe how we handle PARTIALLY and encoding: reverse generation remains substantially OTHERresponsesatthefactlevel. Weevaluatedthree harderthandirectgeneration,thegapismuchsmaller strategies;Table2reportstheresultingprofiledistri- or absent for verification (with reverse verification butionsandthefractionofexcludedfacts. | ofteneasier),andthinkingmitigatesthegap. | | | | Second, | | | | | | | | | ---------------------------------------- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | --- | Thefirststrategyexcludesafactifanyofitssixpro- wecontrolforanswertype(e.g.,person,location,etc) filingquestionsisnon-gradable,meaningallresponses bycomputingperformanceseparatelyforeachtype arePARTIALLYorOTHER.Thisapproachremovesa andthencomparingreversetodirectquestions;the substantialnumberoffacts,around10%forfrontier sametrendspersist. LLMsandover20%forGemma3models. | Finally, | we analyze | error attribution | over | all not- | | | | | | | | | -------- | ---------- | ----------------- | ---- | -------- | --- | --- | --- | --- | --- | --- | --- | Thesecondstrategygroupsquestionsintotaskpairs | known | facts and over | those that are | encoded. | We | | | | | | | | | ----- | -------------- | -------------- | -------- | --- | ---------- | ------- | --- | -------- | --- | -------- | ------ | | | | | | | (encoding, | direct, | and | reverse) | and | excludes | a fact | decomposethesourcesoferrorintothreecategories: | | | | | | only if | an entire | pair | is non-gradable. | | This | substan- | | --- | --- | --- | --- | --- | ------- | --------- | ---- | ---------------- | --- | ---- | -------- | failureondirectquestionsonly,failureonreverseques- tiallyreducesexclusionsto1–4%fornearlyallLLMs, | tionsonly,orfailureonboth. | | AsshowninFigure14, | | | | | | | | | | | -------------------------- | --- | ------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | whileproducingprofiledistributionsnearlyidentical acrossallLLMs,errorsfromreversequestionsconsis- tothefirststrategy. | tentlyexceedthosefromdirectquestions. | | | Forsmaller | | | | | | | | | | ------------------------------------- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- | LLMs,thedominanterrorsourceisfailing‘both’ques- Thethirdstrategyextendsthesecondbyassigning 25 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Error due to: Error due to: Direct Both Reverse Errors % Direct Both Reverse Errors % Gemini-3-Pro 27% 26% 47% 27.6 Gemini-3-Pro 27% 23% 51% 27.6 Gemini-3-Flash 29 22 49 26.8 Gemini-3-Flash 26 19 55 26.8 GPT-5 24 30 46 38.2 GPT-5 23 26 52 38.2 Gemini-2.5-Pro 26 25 49 33.4 Gemini-2.5-Pro 25 22 53 33.4 GPT-4.1 25 27 48 33.9 GPT-4.1 24 22 55 33.9 GPT-5.2 24 34 42 43.0 GPT-5.2 23 27 50 43.0 GPT-5-mini 21 45 33 56.2 GPT-5-mini 20 33 47 56.2 Gemini-2.5-Flash 30 40 31 54.9 Gemini-2.5-Flash 30 34 36 54.9 GPT-4.1-mini 23 45 33 58.5 GPT-4.1-mini 22 33 45 58.5 Gemma3-27b 22 47 31 63.2 Gemma3-27b 22 32 47 63.2 Gemma3-12b 20 56 24 76.2 Gemma3-12b 19 39 42 76.2 Gemma3-4b 12 73 15 89.5 Gemma3-4b 15 47 38 89.5 Gemma3-1b 5 89 50% 6 99.0 Gemma3-1b 10 59 50% 30 99.0 ← ← Figure14 | Wedecomposeerrorsintothreecategories: failuresononlydirectquestions,onlyreversequestions, orboth. Thedashedredlinemarks50%. TheleftbarplotshowsresultsforLLMswithoutthinkingonallfacts, whiletherightbarplotshowsresultsforencodedfacts. Asshown,errorsfromreversequestionsexceedthose fromdirectquestions,regardlessofencoding. PARTIALLYresponsesaweightof0.51,allowingfacts tobeconsideredknownevenwhenallresponsesare CORRECT PARTIALLY.Whilethisfurtherreducesexclusionsto below 1% and yields results very similar to the sec- ond strategy, it introduces subjective judgment that INCORRECT is harder to justify. We therefore adopt the second strategythroughoutthepaper. PARTIALLY Finally,in§2.2,wedonotdefineaprofileforfacts thatarenotencodedbutareknownwithoutthinking. OTHER Suchcasesarerare(lessthan0.5%acrossallmodels; seethe“DirectInference”columninTable2)andare l t i h k i e s ly pr d o u fi e le to d s o a e m s p n l o in t g ch n a o n is g e e o t r h g e ra re d s e u r l e ts r . ror;excluding CORRECT INCORRECT PARTIALLY OTHER GPT-5 D.2. ThresholdSelection Recallthatbothourdefinitionsofencodesandknows rely on a threshold 𝑔(𝑞) > 𝜏. A natural choice is 𝜏=0.5,whichcorrespondstocorrectnessbeingmore likelythanincorrectness. However,thethresholdisa designchoice,anddifferentvaluescanyielddifferent profiledistributions. To assess sensitivity to this choice, Figure 18 presents profile distributions for five additional rea- sonable values of 𝜏. While the exact proportions of profilesvarywith𝜏,ourmainconclusionsarerobust. Specifically,(1)encodingisnearlysaturatedforfron- tier LLMs, even under a conservative threshold of 𝜏=0.99;(2)recallfailuresaccountforasubstantial fractionoferrors,andtheirrelativeimpactincreases under stricter thresholds; and (3) thinking consis- tentlyactsasarecoverymechanism,withover10% offactsknownonlywiththinkingacrossallthreshold choices. orP-5.2-inimeG 78.58 0.05 0.19 0.00 0.06 16.75 0.00 0.07 0.35 0.31 1.25 0.01 0.00 0.56 0.10 1.71 Figure15 | Confusionmatrixbetweengradersbased onGemini-2.5-ProandGPT-5.Entriesarepercentages. Overallagreementis98.2%,withmostdisagreements occurring when Gemini-2.5-Pro predicts OTHER or PARTIALLY. D.3. GraderEvaluation Our evaluation relies on a prompted LLM grader, a common practice in short-form factuality evalua- tion (Haas et al., 2025; Wei et al., 2024a). We use Gemini-2.5-Prowiththinkingastheprimarygrader. The prompt is adapted from SimpleQA (Wei et al., 2024a) and refined through manual error analysis withGemini-2.5-Pro. Inthissubsection,weassessro- bustnesstothechoiceofthegrader’sunderlyingLLM bycomparingourgradertoaGPT-5-basedgraderwith thinking(mediumeffort). Weuniformlysample4,160 responsesacrosssixtasksandallevaluatedLLMs. The twogradersagreeon98.2%ofresponses. Figure15showstheconfusionmatrixbetweenthe twograders. MostdisagreementsoccurwhenGemini- 26 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | 2.5-Pro | assigns | OTHER | or | PARTIALLY, | while | GPT-5 | | ------------------------ | ------- | ----- | ----------------------- | ---------- | ----- | ----- | | assignsadefinitivelabel. | | | Tobetterunderstandthese | | | | disagreements,wemanuallyexaminethem;represen- | tativeexamplesareshowninBoxD.1. | | | | | Inmostcases, | | | ------------------------------- | --- | --- | --- | --- | ------------ | --- | Gemini-2.5-Proyieldsthemoreaccuratelabel,which | is unsurprising | | given | that | the prompt | is optimized | | | ------------------ | --- | ----------------------------- | ---- | ---------- | ------------ | --- | | forit. Inaddition, | | itoccasionallydrawsonexternal | | | | | knowledge. Overall,sincethegradersagreeonthevastmajority ofresponsesandtheremainingdisagreementsprimar- | ily involve | OTHER | or | PARTIALLY | labels, | which | are | | ----------- | ----- | --- | --------- | ------- | ----- | --- | excludedfromouranalysis,weconcludethatourre- | sults are | not sensitive | | to the | choice | of the grader’s | | | --------- | ------------- | --- | ------ | ------ | --------------- | --- | underlyingLLM. D.4. MultipleResponseGeneration Foreachtask,wegenerate𝑛=8responsesperquestion | for three | reasons. | First, | our | goal is | to assess | factual | | --------- | -------- | ------ | --- | ------- | --------- | ------- | knowledgeattheleveloffacts;asingleresponseisa noisyestimatorandmaymisclassifyafactasnoten- | codedornotknownduetosamplingvariance. | | | | | | Second, | | ------------------------------------- | --- | --- | --- | --- | --- | ------- | becauseourquestionsarenotsingle-tokenpredictions, probability-basedestimatesarenotwelldefined,and token-levellikelihoodsareinaccessibleformostpro- | prietaryLLMs, | | makingrepeatedsamplinganatural | | | | | | ------------- | -------------------------------------- | ------------------------------ | --- | --- | --- | --- | | alternative. | Finally,multipleresponsesarerequiredto | | | | | | distinguishavailability(atleastonecorrectresponse) fromrobustness(allresponsescorrect),whichispart ofouranalysisofinference-timethinking. Toassesstheeffectofmultiplesamplingandjustify ourchoiceof𝑛=8,weanalyzehowthefractionoffacts | 𝑔(𝑞) | >0.5 | | | 𝑛 | | | | ---- | ---- | ------ | ---- | ------ | ----------- | --- | | with | | varies | with | across | four tasks: | two | encodingtasks(completionandcontextual)andtwo | knowledgetasks(directandreverse). | | | | | Foreachtask | | | --------------------------------- | --- | ------- | ----- | --------- | ----------- | --- | | and model, | we | perform | 1,000 | bootstrap | resamples, | | eachtimesubsampling𝑛responsesperquestionand recomputingthefractionoffactswith𝑔(𝑞)>0.5. Fig- ure19showsthewidthofthe90%empiricalbootstrap confidenceinterval(95th–5thpercentiles)asafunc- | tion of 𝑛. | Across | the | evaluated | models, | the | interval | | ---------- | ------ | --- | --------- | ------- | --- | -------- | widthfallsbelow1%foralltasksat𝑛=8,indicating thattheestimatedfractionisstabletowithin±0.5%. Wethereforeuse𝑛=8throughoutthepaper. 27 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Table2 | ProfiledistributionsunderthreestrategiesforhandlingPARTIALLYandOTHERresponses. Thethree rightmostcolumnsreporttheshareof‘Excluded’facts,decomposedinto‘OTHERorPARTIALLY’responsesand ‘DirectInference’cases. Thecolumns‘Knows(+Think)’and‘Knows’arecomputedwith‘DirectInference’. PARTIALLY:AssignNaN.OTHER:AssignNaN.NaNsStrategy:Ignoretheentirefact. LLM Encodes Knows Knows Encoding Recall Direct Recallw/ Inferencew/o Excluded OTHERor Direct (+Think) Failure Failure Recall Thinking Encoding PARTIALLY Inference Gemini-3-Pro 98.3 87.4 72.4 1.5 11.2 72.3 14.9 0.1 5.2 5.1 0.1 Gemini-3-Flash 98.1 86.6 73.4 1.8 11.7 73.3 13.1 0.1 9.5 9.5 0.0 Gemini-2.5-Pro 98.1 79.7 66.1 1.9 18.5 66.0 13.5 0.1 7.4 7.3 0.1 Gemini-2.5-Flash 92.8 65.8 46.0 6.9 27.4 46.0 19.5 0.3 13.3 13.2 0.0 | GPT-5.2 | 92.8 75.7 | 56.8 6.3 | 18.1 56.6 | 18.0 | 0.9 8.2 | 8.0 0.2 | | ------- | --------- | -------- | --------- | ---- | ------- | ------- | | GPT-5 | 95.9 84.5 | 62.0 3.3 | 12.4 61.8 | 21.8 | 0.7 6.8 | 6.6 0.2 | GPT-5-mini 83.5 67.9 44.2 13.4 19.0 44.0 20.6 3.1 7.2 7.0 0.2 | GPT-4.1 | 96.0 77.3 | 64.9 3.7 | 19.1 64.9 | 12.0 | 0.4 9.2 | 9.1 0.1 | | ------- | --------- | -------- | --------- | ---- | ------- | ------- | GPT-4.1-mini 85.7 58.1 41.7 13.4 28.7 41.5 15.4 0.9 11.1 10.9 0.2 Gemma3-27b 80.1 47.3 37.3 19.5 33.4 37.0 9.7 0.4 21.0 20.7 0.3 Gemma3-12b 68.0 33.4 24.6 31.1 35.6 24.5 7.9 1.0 27.2 27.1 0.1 Gemma3-4b 46.3 16.0 11.0 53.5 30.6 10.9 4.9 0.2 27.2 27.1 0.1 | Gemma3-1b | 17.0 | 1.8 1.1 82.7 | 15.5 1.1 | 0.4 | 0.3 45.8 | 45.8 0.0 | | --------- | ---- | ------------ | -------- | --- | -------- | -------- | **Selected**PARTIALLY:AssignNaN.OTHER:AssignNaN.NaNsStrategy:IgnoreonlyifbothtasksareNaNs.**Selected** Knows Encoding Recall Direct Recallw/ Inferencew/o OTHERor Direct | LLM | Encodes | Knows | | | Excluded | | | --- | ------- | ----- | --- | --- | -------- | --- | (+Think) Failure Failure Recall Thinking Encoding PARTIALLY Inference Gemini-3-Pro 98.1 87.7 72.4 1.6 10.9 72.2 14.9 0.3 1.3 1.1 0.2 Gemini-3-Flash 97.2 86.1 73.2 2.5 11.4 73.2 12.6 0.2 3.2 3.1 0.1 Gemini-2.5-Pro 97.5 79.6 66.6 2.5 18.1 66.4 13.0 0.1 1.3 1.2 0.2 Gemini-2.5-Flash 91.1 64.8 45.1 8.4 26.9 45.0 19.2 0.5 1.9 1.8 0.1 | GPT-5.2 | 92.1 75.9 | 57.0 6.8 | 17.6 56.6 | 17.9 | 1.0 2.0 | 1.6 0.3 | | ------- | --------- | -------- | --------- | ---- | ------- | ------- | | GPT-5 | 95.3 84.1 | 61.8 4.0 | 12.2 61.6 | 21.5 | 0.8 1.6 | 1.3 0.2 | GPT-5-mini 82.8 67.3 43.8 14.0 19.0 43.5 20.3 3.2 1.8 1.5 0.3 | GPT-4.1 | 95.6 76.7 | 66.1 4.1 | 19.6 65.7 | 10.3 | 0.3 2.5 | 2.1 0.4 | | ------- | --------- | -------- | --------- | ---- | ------- | ------- | GPT-4.1-mini 83.8 57.3 41.5 14.9 28.1 41.1 14.6 1.2 2.6 2.2 0.3 | Gemma3-27b | 76.4 45.8 | 36.8 23.1 | 31.6 36.2 | 8.6 | 0.5 3.5 | 3.0 0.5 | | ---------- | --------- | ------------ | --------- | --- | -------- | ------- | | Gemma3-12b | 63.8 32.1 | 23.8 35.3 | 32.8 23.6 | 7.4 | 0.9 3.8 | 3.6 0.1 | | Gemma3-4b | 42.9 14.6 | 10.5 56.7 | 28.9 10.4 | 3.6 | 0.4 5.3 | 5.1 0.2 | | Gemma3-1b | 14.3 | 1.9 1.0 85.2 | 13.0 0.9 | 0.5 | 0.5 10.0 | 9.8 0.1 | PARTIALLY:0.51Weight.OTHER:AssignNaN.NaNsStrategy:IgnoreonlyifbothtasksareNaNs. LLM Encodes Knows Knows Encoding Recall Direct Recallw/ Inferencew/o Excluded OTHERor Direct (+Think) Failure Failure Recall Thinking Encoding PARTIALLY Inference Gemini-3-Pro 98.2 88.7 74.3 1.5 9.8 74.2 14.2 0.2 0.2 0.2 0.0 Gemini-3-Flash 97.5 87.0 75.4 2.3 10.9 75.2 11.4 0.2 0.5 0.3 0.2 Gemini-2.5-Pro 97.7 81.0 68.5 2.2 16.9 68.3 12.4 0.1 0.1 0.0 0.1 Gemini-2.5-Flash 92.3 67.1 47.2 7.3 25.8 47.0 19.5 0.5 0.8 0.6 0.2 | GPT-5.2 | 92.6 76.3 | 57.8 6.4 | 17.5 57.6 | 17.5 | 1.0 0.7 | 0.6 0.2 | | ------- | --------- | -------- | --------- | ---- | ------- | ------- | | GPT-5 | 95.7 85.3 | 63.4 3.5 | 11.3 63.2 | 21.1 | 0.7 0.4 | 0.1 0.2 | GPT-5-mini 84.1 68.7 45.1 12.9 18.8 44.7 20.6 3.0 0.8 0.4 0.4 | GPT-4.1 | 96.1 77.5 | 67.2 3.7 | 19.1 66.9 | 10.0 | 0.2 0.5 | 0.2 0.3 | | ------- | --------- | -------- | --------- | ---- | ------- | ------- | GPT-4.1-mini 85.2 58.4 42.7 13.5 28.5 42.4 14.4 1.3 1.0 0.6 0.4 | Gemma3-27b | 78.8 47.2 | 38.6 20.8 | 32.7 37.9 | 8.1 | 0.4 1.8 | 1.2 0.7 | | ---------- | --------- | ------------ | --------- | --- | ------- | ------- | | Gemma3-12b | 66.5 33.5 | 25.7 32.7 | 34.1 25.4 | 7.1 | 0.8 1.7 | 1.4 0.3 | | Gemma3-4b | 48.3 15.6 | 11.4 51.5 | 33.2 11.1 | 4.0 | 0.2 3.2 | 2.8 0.3 | | Gemma3-1b | 21.6 | 2.5 1.4 78.0 | 19.7 1.3 | 0.6 | 0.4 7.9 | 7.7 0.2 | 28 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Gemini-3-Pro Gemini-3-Flash Gemini-2.5-Pro Gemini-2.5-Flash GPT-5.2 | | ∆ | | ∆ | | ∆ | | | | | | | --- | ------- | ------ | ------ | ------ | ------ | ------ | ------ | ----------- | ------ | ------ | | 100 | = 3 . 5 | | = | 5 . 1 | = | 2 . 8 | ∆ = | 12.0 | ∆ = | 13.9 | | | 9 9. 3 | ∆=21.4 | 9 9. 5 | ∆=21.4 | 9 8. 8 | ∆=25.8 | | | | | | | 9 5 .8 | | | 9 4 .5 | | 9 6 .0 | 9 6 .5 | | 9 7 .6 | ∆=22.7 | | | | | | 84.7 | | | | 84.5 ∆=25.1 | | 83.8 | | | | 82.5 | | | | 79.1 | | | | | 73.7 | 50 | | 61.2 | | 63.3 | | | | 61.8 | | | | --- | --- | ---- | --- | ---- | --- | ---- | --- | ---- | --- | ---- | | | | | | | | 53.3 | | | | 51.0 | 36.7 0 Encodes Recalls Encodes Recalls Encodes Recalls Encodes Recalls Encodes Recalls | | GPT-5 | | GPT-5-mini | | | GPT-4.1 | GPT-4.1-mini | | Gemma3-27b | | | --- | ----- | ------ | ---------- | ------ | ----- | ------- | ------------ | ------ | ---------- | ------ | | | ∆=7.6 | | | | ∆=7.3 | | | | | | | 100 | | | ∆=11.6 | | | | ∆=11.0 | | ∆=22.7 | | | | 98.4 | ∆=24.9 | | | 99.1 | ∆=19.5 | | | | | | | 90.7 | | | | | 91.8 | | | | | | | | | 88.0 | ∆=21.1 | | | 89.5 | ∆=19.2 | 88.1 | | | | | 77.8 | | 76.4 | | 78.7 | | 78.6 | | ∆=24.0 | 65.4 | 50 | | | | 61.8 | | 59.2 | | 58.9 | | 56.5 | | --- | --- | --- | --- | ---- | --- | ---- | --- | ---- | --- | ---- | 52.9 | | | | | 40.6 | | | | 39.7 | | | | --- | --- | --- | --- | ---- | --- | --- | --- | ---- | --- | --- | 32.5 0 Encodes Recalls Encodes Recalls Encodes Recalls Encodes Recalls Encodes Recalls | | | | Gemma3-12b | | Gemma3-4b | | Gemma3-1b | | | | | --- | --- | --- | ---------- | --- | --------- | --- | --------- | --- | --- | --- | 100 | | | | ∆=24.6 | | | | | | Popularity: | | | --- | --- | --- | ------ | --- | ------ | --- | --- | --- | ----------- | --- | | | | | 74.2 | | ∆=24.5 | | | | High (>80%) | | ∆=22.5 | | | | 50 | | 56.2 | ∆=15.1 | | | Low (<20%) | | | --- | --- | --- | --- | --- | ---- | ------ | --- | --- | ---------- | --- | 49.6 47.3 ∆=6.7 | | | | | | | 31.7 32.9 | | ∆=4.8 | | | | --- | --- | --- | --- | ---- | --- | --------- | ---- | ----- | --- | --- | | | | | | 24.8 | | 17.8 | 17.5 | | | | 0 | | | | Encodes | Recalls | Encodes | Recalls | Encodes | Recalls | | | | --- | --- | --- | ------- | ------- | ------- | ------- | ------- | ------- | --- | --- | Gemini-3-Pro Gemini-3-Flash Gemini-2.5-Pro Gemini-2.5-Flash GPT-5.2 | | ∆=-2.3 | | ∆=-0.3 | | | | | | | | | --- | -------- | ------ | -------- | ------ | ------ | ------ | ------ | ----------- | ------ | ------ | | 100 | | | | ∆=-5.8 | ∆=-3.9 | | | | | | | | 97.495.1 | ∆=-5.7 | | | | ∆=-8.2 | ∆=+6.3 | | ∆=+5.6 | | | | | | 96.296.0 | | 94.0 | | | | | ∆=-9.1 | | | | | | 88.9 | | 90.1 | | | | | | 80 | | 86.9 | | 83.0 | | 85.0 | | 87.2 ∆=-1.5 | | 87.7 | | | | 81.2 | | | | | 80.8 | | 82.1 | 80.9 | 76.8 | | | | | | | | | 68.066.5 | | 71.8 | | --- | --- | --- | --- | --- | --- | --- | --- | -------- | --- | ---- | 60 Verify Generate Verify Generate Verify Generate Verify Generate Verify Generate | | (MCQA) | (CBQA) | | | | | | | | | | --- | ------ | ------ | ---------- | ------- | ------ | ------- | ------------ | ------ | ---------- | ------ | | | GPT-5 | | GPT-5-mini | | | GPT-4.1 | GPT-4.1-mini | | Gemma3-27b | | | 100 | | | | | ∆=+6.9 | | | | | | | | ∆=+4.7 | | | | | ∆=-8.5 | | | | | | | | ∆=-9.0 | ∆=+7.2 | | | | ∆=+7.2 | | ∆=+5.3 | | | | 90.7 | | | ∆=-10.6 | | 92.0 | | | | | | 80 | 86.0 | | | | 85.1 | 85.9 | | ∆=-9.4 | | ∆=-9.8 | | | | 83.0 | | 84.1 | | | | 84.4 | | 84.1 | | | | | 76.9 | 75.3 | | 77.4 | 77.2 | | 78.8 | | | | | 74.0 | | | | | | 72.2 | | 72.6 | | 60 | | | | 64.7 | | | | | | | | | | | | | | | | 62.8 | | 62.8 | Verify Generate Verify Generate Verify Generate Verify Generate Verify Generate | | | | Gemma3-12b | | Gemma3-4b | | Gemma3-1b | | | | | --- | --- | --- | ---------- | --- | --------- | --- | --------- | --- | --- | --- | ∆=+8.7 | | | | | 83.3∆=-11.5 | ∆=+3.3 | | | | | | | --- | --- | --- | ---- | ----------- | ------ | ------- | --- | --- | --- | ------ | | | | | 74.6 | | | ∆=-17.4 | | | | Direct | 64.6 | | | | 50 | | 59.662.9 | | ∆=-6.5 | | | Reverse | | --- | --- | --- | --- | ---- | -------- | ---- | ------ | ------- | --- | ------- | | | | | | 53.2 | | 54.3 | | ∆=-19.3 | | | 39.9 | | | | | | | 37.0 | | 33.4 35.4 | | | | --- | --- | --- | --- | --- | --- | ---- | --- | --------- | --- | --- | 16.0 0 | | | | Verify | Generate | Verify | Generate | Verify | Generate | | | | --- | --- | --- | ------ | -------- | ------ | -------- | ------ | -------- | --- | --- | | Top: Figure16 Wecomparetwopopularitytiers(bottom20%vs. top20%)intermsofencodingratesand directrecallrates(knowingencodedfactswithoutthinking). TheΔindicatesthegapbetweentiers. Bottom: Wecomparedirectrecallratesondirectandreversequestionsacrosstwotasks: verification(multiple-choice) andgeneration(closed-book). TheΔdenotesthegapbetweenthedirections. 29 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality ∆ ∆ T = = 2 1 1 2 .4 .5 ∆ ∆ T = = 2 1 1 1 .4 .7 ∆ ∆ T = = 2 1 5 7 .8 .2 ∆T=21.4 ∆ ∆ T = = 2 1 2 6 .7 .6 ∆ ∆ T = = 5 2 .7 .0 ∆ ∆ T = = 5 2 .8 .2 ∆ ∆ T = = 8 5 .2 .0 ∆ ∆ T = = 9 2 .0 .0 ∆ ∆ T = = 8 7 .5 .2 80 11.3 20.1 7.7 17.3 8.8 1 ∆ 7. = 8 25.1 15.2 80 8.6 12.3 6.7 10.3 7.6 10.9 12.0 19.0 6.9 8.2 60 17.4 21.4 40 82.5 61.2 84.7 63.3 79.1 61.8 21.5 73.7 60 86.9 81.2 88.9 83.0 85.0 76.8 83.0 74.0 85.9 77.4 53.3 51.0 36.7 20 40 Gemini-3 Gemini-3 Gemini-2.5 Gemini-2.5 GPT-5.2 Gemini-3 Gemini-3 Gemini-2.5 GPT-5 GPT-4.1 Pro Flash Pro Flash Pro Flash Pro 6 8 0 0 1 ∆ ∆ 6 T . = 0 = 2 1 4 2 4 . 6 9 . . 5 4 2 ∆ ∆ 2 T . = 5 = 2 1 1 2 8 . 4 1 . . 9 7 ∆ 8 ∆ . T 6 = = 1 1 9 1 6 . 2 5 . . 0 1 1 ∆ ∆ 6 T . = 7 = 1 2 9 1 .2 .1 1 ∆ ∆ 2 T . = 7 = 2 2 4 7 .0 .2 80 1 ∆ 2 ∆ . T 1 = = 9 5 1 .1 . 6 1 .1 1 ∆ 8 ∆ . T 1 = = 1 4 1 .5 . 5 0 .6 1 ∆ ∆ 5. T = 7 = 1 5 0 2 . . 1 3 6 .1 1 ∆ 3 ∆ . T 4 = = 9 8 1 .4 . 3 9 .9 9 ∆ ∆ .3 T = = 9 8 1 .8 . 0 9 .2 40 77.8 52.9 61.8 40.6 78.7 59.2 58.9 3 1 9 4 . . 7 8 56.5 3 9 2 . . 5 5 60 80.9 71.8 68.0 66.5 75.3 64.7 72.2 62.8 72.6 62.8 20 40 GPT-5 GPT-5 GPT-4.1 GPT-4.1 Gemma3 GPT-5.2 Gemini-2.5 GPT-5 GPT-4.1 Gemma3 mini mini 27b Flash mini mini 27b 60 ∆ ∆ T = = 2 2 2 6 .5 .1 1 ∆ ∆ 1 T . = 7 = 1 1 1 2 .5 .4 ∆ ∆ T = = 1 1 7 8 .4 .8 13.5 ∆ ∆ T = = 1 2 5 2 .1 .8 High Pop 60 10.8 10.9 ∆ ∆ T = = 1 2 9 7 .3 .2 Direct 40 10.1 +Thinking 40 9.4 13.5 +Thinking 47.3 9.9 Low Pop 64.6 Reverse 20 24.8 32.9 ∆ ∆ T = = 4 6 .8 .7 +Thinking 20 53.2 54.3 37.0 35.4 5.6 +Thinking 17.8 16.0 0 0 Gemma3 Gemma3 Gemma3 Gemma3 Gemma3 Gemma3 12b 4b 1b 12b 4b 1b Figure17 | Weexaminetheimpactofthinkingonretrieval(knowingencodedfacts). Ontheleft,wecompare twopopularitytiers(bottom20%vs. top20%),whileontheright,wecomparedirectandreversequestions. ThepopularityordirectionalgapsaredenotedbyΔ(nothinking)andΔ𝑇 (withthinking). τ encodes =0.01, τ knows =0.50 τ encodes =0.50, τ knows =0.50 Knows: Knows: 1.9 14.532.045.457.064.767.175.676.479.583.986.087.5 1.8 14.431.945.356.964.767.075.676.379.483.886.087.5 100% 100% 80% 22.1 37.7 39.5 35.3 34.8 30.7 26.4 21.0 21.6 19.4 14.0 1 1 2 2 . . 7 2 1 1 5 1 . . 2 5 80% 13.0 28.9 32.8 31.6 28.1 26.9 19.0 1 6 7 . . 8 6 19.6 18.1 12.2 1 1 2 1 . . 6 4 1 1 4 0 . . 9 9 60% 19.3 8.1 19.4 2 6 2 . . 5 9 18.6 10.5 13.0 22.1 60% 23.1 14.9 1 8 9 . . 4 2 2 1 0 4 . . 3 0 17.9 10.3 13.0 21.5 Recall 40% 76.0 47.8 2 7 8 . . 8 6 8.7 15.4 56.8 65.8 66.4 61.7 73.2 72.3 40% 85.2 56.7 3 7 5 . . 4 3 8.6 14.6 56.6 65.7 66.4 61.6 73.2 72.2 Failure 20% 36.3 41.3 45.1 43.6 20% 36.2 41.1 45.0 43.5 23.7 23.6 Encoding 0% 10.5 0% 10.4 Failure Gemma G 3 e - m 1b m G a e 3 m -4 m b a G 3 e - m 12 m b a G 3 P - G 2 T 7 e -4 b m .1 in -m i-2 in .5 i -F G la P s T h -5-min G i PT-5 G .2 G e P m T i - n 4 i . - 1 2.5-Pr G o e G m P in T i - - 5 3 G -F e l m as in h i-3-Pro Gemma G 3 e - m 1b m G a e 3 m -4 m b a G 3 e - m 12 m b a G 3 P - G 2 T 7 e -4 b m .1 in -m i-2 in .5 i -F G la P s T h -5-min G i PT-5 G .2 G e P m T i - n 4 i . - 1 2.5-Pr G o e G m P in T i - - 5 3 G -F e l m as in h i-3-Pro τ encodes =0.50, τ knows =0.99 τ encodes =0.99, τ knows =0.99 w I / n o f e E r n e c n o c d e i n g Knows: Knows: 0.7 9.0 23.636.441.345.656.163.465.070.573.280.280.3 0.6 8.9 23.236.341.145.555.863.365.070.473.280.180.2 100% 100% 6 8 0 0 % % 13.7 33.9 40.3 40.0 42.7 45.5 2 1 8 5 . . 1 9 2 7 9 . . 6 0 3 8 0 . . 2 4 1 2 7 6 . . 4 9 2 2 5 2 . . 8 1 1 1 0 7 . . 7 1 2 1 7 7 . . 2 9 6 8 0 0 % % 8.2 28.8 35.1 37.6 35.2 36.7 2 2 0 3 . . 7 5 2 1 5 1 . . 3 4 2 7 6 8 . . . 9 4 6 1 2 6 6 3 . . . 1 5 6 2 1 7 4 8 . . . 9 5 9 1 1 0 6 . . 6 4 2 1 5 6 . . 5 3 wit R h e T c h a i l n l k ing 40% 85.6 57.1 36.1 23.6 1 1 2 6 . . 7 0 1 8 6 . . 9 1 25.8 16.9 69.4 40% 91.2 62.2 41.7 26.1 1 2 2 3 . . 2 7 1 1 5 7 . . 5 8 22.6 16.5 69.3 Direct 20% 33.1 28.3 29.5 28.7 46.0 56.8 53.0 47.2 52.9 20% 33.0 28.1 29.4 28.3 45.8 56.7 52.8 47.2 52.8 Recall 20.4 20.0 7.9 7.8 0% 0% Gemma G 3 e - m 1b m G a e 3 m -4 m b a G 3 e - m 12 m b a G 3 P - G 2 T 7 e -4 b m .1 in -m i-2 in .5 i -F G la P s T h -5-min G i PT-5 G .2 G e P m T i - n 4 i . - 1 2.5-Pr G o e G m P in T i - - 5 3 G -F e l m as in h i-3-Pro Gemma G 3 e - m 1b m G a e 3 m -4 m b a G 3 e - m 12 m b a G 3 P - G 2 T 7 e -4 b m .1 in -m i-2 in .5 i -F G la P s T h -5-min G i PT-5 G .2 G e P m T i - n 4 i . - 1 2.5-Pr G o e G m P in T i - - 5 3 G -F e l m as in h i-3-Pro Figure18|Distributions(percentages)ofknowledgeprofilesacross13LLMsunderdifferent𝑔(𝑞) >𝜏thresholds forencodingandknowledge. Whiletheexactprofilepercentagesvarywith𝜏,ourmainfindingsarerobustto thechoiceofthreshold. 30 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | | Gemini-3-Pro | | Gemini-3-Flash | | | Gemini-2.5-Pro | | Gemini-2.5-Flash | | --- | ------------ | --- | -------------- | --- | --- | -------------- | --- | ---------------- | | 2.0 | | 2.0 | | | 2.0 | | 2.0 | | htgneL IC %09 | 1.5 | | 1.5 | | | 1.5 | | 1.5 | | | ----------------- | ------------ | --- | ---------- | --- | --- | ---------- | --- | --------- | | 1.0 | | 1.0 | | | 1.0 | | 1.0 | | | 0.5 | | 0.5 | | | 0.5 | | 0.5 | | | 0.0 | | 0.0 | | | 0.0 | | 0.0 | | | | GPT-5.2 | | GPT-5 | | | GPT-5-mini | | GPT-4.1 | | htgneL IC %09 2.0 | | 2.0 | | | 2.0 | | 2.0 | | | 1.5 | | 1.5 | | | 1.5 | | 1.5 | | | 1.0 | | 1.0 | | | 1.0 | | 1.0 | | | 0.5 | | 0.5 | | | 0.5 | | 0.5 | | | 0.0 | | 0.0 | | | 0.0 | | 0.0 | | | | GPT-4.1-mini | | Gemma3-27b | | | Gemma3-12b | | Gemma3-4b | | 2.0 | | 2.0 | | | 2.0 | | 2.0 | | htgneL IC %09 | 1.5 | | 1.5 | | | 1.5 | | 1.5 | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1.0 | | 1.0 | | | 1.0 | | 1.0 | | | 0.5 | | 0.5 | | | 0.5 | | 0.5 | | | 0.0 | | 0.0 | | | 0.0 | | 0.0 | | 2 4 6 8 10121416 2 4 6 8 10121416 2 4 6 8 10121416 2 4 6 8 10121416 | | N | | N | | | N | | N | | --- | --- | --- | ---------- | ---------- | --- | -------------- | --- | --- | | | | | Completion | Contextual | | Direct Reverse | | | | Sensitivityofthefractionoffactswith𝑔(𝑞)>0.5tothenumberofsampledresponses𝑛,shownas Figure19 thewidthofthe90%empiricalbootstrapconfidenceinterval(95th–5thpercentiles)acrosseightLLMs. As shown,theintervalwidthdropsbelow1%at𝑛=8. 31 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality BoxD.1: AnalysisofDisagreementsBetweenGraders DirectQuestion: WhodevelopedthecircuitfortheCockcroft–Waltongeneratorin1919? GoldAnswer: HeinrichGreinacher Prediction of Gemini-3-Pro: The circuit itself was essentially discovered by Heinrich Greinacher, a Swiss physicist, in 1919. BetterGrader: Gemini(CORRECTvsGPT-5INCORRECT) Rationale: ThepredictioncorrectlyidentifiesGreinacher;GPT-5likelypenalizedtheshiftfrom“developed"to“discov- ered”. DirectNaturalQuestion: AbouthowmanyiPhone11shadsoldasofMarch2022? GoldAnswer: 159.2million PredictionofGemini-2.5-Pro+Thinking: Approximately160millionunits. BetterGrader: Gemini(CORRECTvsGPT-5INCORRECT) Rationale: Rounding159.2to“Approximately160” isapermissibleandalignwiththeguidelines. Reverse Question: What district within the Lahore Division of Punjab, Pakistan has a total area of 3,995 square kilometres? GoldAnswer: KasurDistrict PredictionofGemini-2.5-Pro+Thinking: Kasur BetterGrader: Gemini(CORRECTvsGPT-5PARTIALLY) Rationale: Dropping“District” isexplicitlyallowedbytheguidelines. ReverseQuestion: WhatisthenameofthefemaleversionofTaskmasterwhoappearsinBlackWidow? GoldAnswer: AntoniaDreykov PredictionofGemini-2.5-Flash: Taskmaster BetterGrader: Gemini(OTHERvsGPT-5INCORRECT) Rationale: Thepredictiondoesnotcontradictthequestion(sheisTaskmaster). ReverseQuestion: WhoseactivitiesdidtheFederalBureauofInvestigation(FBI)programCOINTELPROattemptto subvertbyplacingacounterintelligenceoperativeinthelocalPanthersorganization? GoldAnswer: FredrickAllenHampton PredictionofGPT-5-mini+Thinking: TheBlackPantherParty BetterGrader: Gemini(PARTIALLYvsGPT-5INCORRECT) Rationale: Identifying the group (Black Panthers) instead of the leader (Hampton) is a “more general version” of the target. ReverseNaturalQuestion: Whowatched’TheUFOIncident’onthenightMikeRogersadmittedthejobwasseriously behindschedule? GoldAnswer: WaltonandRogers PredictionofGPT-4.1+Thinking: TravisWaltonandtheotherloggingcrewmembers(includingMikeRogers)watched ’TheUFOIncident’togetherthatnight. BetterGrader: Gemini(CORRECTvsGPT-5INCORRECT) Rationale: IncludingtherestofthecrewdoesnotcontradictthepresenceofWaltonandRogers. Direct Question: What is the compound wall that surrounds the St. Antony’s Syro-Malabar Catholic Forane Church atOllurcalled? GoldAnswer: ElephantWall PredictionofGemma3-4b+Thinking: PerimeterWall BetterGrader: Gemini(OTHERvsGPT-5INCORRECT) Rationale: “PerimeterWall” ismissingthespecificname,butitisnotafactualcontradiction(itis aperimeterwall). ReverseQuestion: WhatstatuedepictingVairocanaBuddhaislocatedwithintheFodushanScenicArea? GoldAnswer: SpringTempleBuddha Prediction of Gemma3-1b+Thinking: Fodushan Scenic Area, specifically the Temple of the White Horse, where the statueofVairocanaBuddhaislocated. BetterGrader: Gemini(INCORRECTvsGPT-5OTHER) Rationale: Thepredictionplacesthestatueinacompletelydifferenttemple(WhiteHorse). DirectQuestion: Whobuiltthetown,originallycalledBeodericsworth,onagridpattern? GoldAnswer: AbbotBaldwin PredictionofGemma3-1b+Thinking: ThepeopleofBeodericsworthbuiltthetown. BetterGrader: Gemini(OTHERvsGPT-5INCORRECT) Rationale: “The people built it” is a vague truism that fails to answer the “Who” (Architect/Leader) but doesn’t contradicttheanswer. 32 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality DirectNaturalQuestion: Whichinstitutionwasthefossilspecimen"Apex"loanedtoinDecember2024? | GoldAnswer: | AmericanMuseumofNaturalHistory | | | | | | | ----------- | ------------------------------ | --- | --- | --- | --- | --- | PredictionofGemini-2.5-Pro: Accordingtotheprovidedinformation,thefossilspecimen"Apex"wasloanedinDecem- | ber2022,notDecember2024. | | | Therefore,Icannotansweryourquestion. | | | | | ------------------------ | ----------------------------------------------------------------- | --- | ------------------------------------ | --- | --- | --- | | BetterGrader: | Gemini(INCORRECTvsGPT-5OTHER) | | | | | | | Rationale: | Thepredictionexplicitlycontradictsthequestion(claimingitiswrong). | | | | | | DirectQuestion: WhatexistinghighwaywasUS1mostlyassignedtowhentheroadsystemwaslaidout? | GoldAnswer: | AtlanticHighway | | | | | | | ------------------------------- | ----------------------------- | --- | --- | ---------------------------------------- | --- | --- | | PredictionofGemma3-1b+Thinking: | | | | US1wasprimarilyassignedtotransportgoods. | | | | BetterGrader: | Gemini(OTHERvsGPT-5INCORRECT) | | | | | | Rationale: Statingthepurpose(“transportgoods”)insteadofthehighwaynameisirrelevant(Other). ReverseNaturalQuestion: What’sthenameoftheHindutemplecomplexinDelhithatopenedonNovember6,2005? | GoldAnswer: | SwaminarayanAkshardham | | | | | | | --------------------------- | ------------------------------- | ---------------------------------- | ---------- | --- | --- | --- | | PredictionofGemini-2.5-Pro: | | | Akshardham | | | | | BetterGrader: | GPT-5(CORRECTvsGeminiPARTIALLY) | | | | | | | Rationale: | “Swaminarayan” | isamodifieranddroppingitisallowed. | | | | | Direct Question: What was the name of the exposition at which Westinghouse Electric showcased its AC system to illuminatethe“WhiteCity"? | GoldAnswer: | 1893ColumbianExposition | | | | | | | ----------------------------- | ------------------------------- | --- | ----------------------------- | --- | --- | --- | | PredictionofGemini-2.5-Flash: | | | TheWorld’sColumbianExposition | | | | | BetterGrader: | GPT-5(CORRECTvsGeminiPARTIALLY) | | | | | | Rationale: “World’sColumbianExposition” istheformalnameforthe“1893ColumbianExposition”. Reverse Question: On which album did bassist Danny Thompson have his first collaboration on record with John Martyn? | GoldAnswer: | TheRoadtoRuin | | | | | | | ------------------------------- | ---------------------------------- | --- | --- | ---- | --- | --- | | PredictionofGemma3-4b+Thinking: | | | | Sure | | | | BetterGrader: | GPT-5(OTHERvsGeminiINCORRECT) | | | | | | | Rationale: | “Sure” isarefusalornonsensefiller. | | | | | | E. Prompts E.1. FactsExtractionPrompts | BoxE.1: | PromptforClassifyingPagesintoCategories | | | | | | | -------- | --------------------------------------- | ---- | --- | -------- | -------------- | --- | | You will | be provided | with | a | document | and its title. | | Your task is to analyze the document and assign it to the single, most appropriate category | from | the predefined | | list | below. | | | | ---- | -------------- | --- | ---- | ------ | --- | --- | ---- | **Categories | and | Descriptions:** | | | | | | ------------ | --- | --------------- | --- | --- | --- | --- | * **People:** Covers specific individuals, their biographies, life stories, achievements, | and | influence. | | | | | | | -------------- | ---------- | ------ | ---- | ------ | -------------- | -------- | | * **History:** | | Covers | past | events | and historical | periods. | * **Geography:** Covers Earth's physical features, locations, countries, and cities. * **Arts:** Covers all forms of creative expression, including literature, music, visual | arts, | film, | and performance. | | | | | | ----- | ----- | ---------------- | --- | --- | --- | --- | * **Philosophy & Religion:** Covers fundamental principles, belief systems, ethics, | religions, | | and mythological | | narratives. | | | | ---------- | --- | ---------------- | --- | ----------- | --- | --- | * **Everyday Life:** Covers the practical and routine aspects of human life, including | food, | clothing, | family, | | sports, | and hobbies. | | | ----- | --------- | ------- | --- | ------- | ------------ | --- | * **Society:** Covers the structures and systems that govern human communities, such as | politics, | law, | economics, | | and language. | | | | --------- | ---- | ---------- | --- | ------------- | --- | --- | * **Science:** Covers the study of the natural and physical world, including biology, | physics, | chemistry, | | and | mathematics. | | | | -------- | ---------- | --- | --- | ------------ | --- | --- | 33 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality * **Technology:** Covers the application of scientific knowledge, including engineering, | computers, | and inventions. | | | | | | ---------- | --------------- | --- | --- | --- | --- | * **Other:** Covers topics that serve as a catch-all and do not clearly fit into any of | the | other categories. | | | | | | --- | ----------------- | --- | --- | --- | --- | ---- | **Output | Format:** | | | | | | -------- | --------- | --- | --- | --- | --- | Your response must consist of **ONLY** one category name. Do not add any explanation, | punctuation, | or other | text. | | | | | ------------ | -------- | ----- | --- | --- | --- | Title: {page_title.} | Content: | {summary} | | | | | | -------- | ----------------------------------------- | --- | ----- | ---- | ---------- | | BoxE.2: | PromptforPerformingNamedEntityRecognition | | | | | | You will | be provided with | two | parts | of a | paragraph. | Your task is to perform Named Entity Recognition on the *second* part of the paragraph. | Please follow | these instructions | | carefully. | | | | ------------- | ------------------ | --- | ---------- | --- | --- | ### **Instructions:** | 1. Read | the two parts | provided | at | the end | of this prompt. | | ------- | ------------- | -------- | --- | ------- | --------------- | 2. From the second part, identify and select up to a maximum of **eight** entities. 3. Do not select lengthy entities (more than five words) or entities that are not named entities. 4. Select specific core entities and avoid overly descriptive phrases that combine multiple separate concepts (e.g., select "documentary photographer" and not "feminist | social | documentary | photographer"). | | | | | ------ | ----------- | --------------- | --- | --- | --- | 5. For each entity you select, you must assign an entity type from the following specific list. 6. You should return a new version of the second part of the paragraph. Copy-paste the original second part and replace the entities you chose with the format `{{{{ | entity_name; | entity_type}}}}` | | | | | | ------------ | ---------------- | --- | --- | --- | --- | ---- | ### **Entity | Types:** | | | | | | ------------- | ------------- | --- | ------ | ------- | ------------------ | | * **PERSON:** | An individual | | human, | whether | real or fictional. | * **ORGANIZATION:** A structured company, institution, or collective with a common purpose. * **LOCATION:** A specific geographical place, region, or feature on Earth. * **EVENT:** A notable occurrence or incident that happens at a particular time and place . * **WORK_OF_ART:** A specific creative work, such as a book, movie, song, or painting. * **PRODUCT:** A commercially produced good, food, service, or piece of technology. * **ORGANISM:** A non-human biological life form, including animals, plants, and microbes . * **SUBSTANCE:** A material with a distinct physical or chemical composition. * **DATE:** A specific point in time, including full dates, years, or historical periods. * **NUMERIC_VALUE:** A number representing a specific quantity, measurement, or other | quantifiable | value. | | | | | | ------------ | ------ | --- | --- | --- | --- | * **CONDITION:** A specific state of being, typically a medical, physical, or operational status. * **TITLE:** An official position, role, award, or honorific held by a person or group. * **LANGUAGE:** Natural languages, dialects, writing systems, including phrases. * **GROUP:** A collective of people defined by a shared nationality, ethnicity, or | cultural | affiliation. | | | | | | -------- | ------------ | --- | --- | --- | --- | * **LAW:** A specific, formally enacted rule, regulation, or statute created by a | governing | body. | | | | | | --------- | ----- | --- | --- | --- | --- | * **IDEA:** An abstract concept, belief system, named phenomenon, or cultural practice. * **CLASSIFICATION:** A specific type, genre, format, class or list used to categorize | other | entities. | | | | | | ----- | --------- | --- | --- | --- | --- | * **IDENTIFIER:** A unique name, code, or alias used as a formal reference or label for 34 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | another | entity | | | | | | ------- | ------ | --- | --- | --- | --- | * **OTHER:** A catch-all category for named entities that do not fit into any of the | other defined | | types. | | | | | ------------- | --- | ------ | --- | --- | --- | ---- | ### **Required | Output | Format:** | | | | | -------------- | ------ | --------- | --- | --- | --- | Copy-paste the second part of the paragraph. In its new version, replace the entities you chose with the format `{{{{entity_name; entity_type}}}}`, for example: {{{{tennis player; TITLE}}}}. The `entity_name` must be exactly as it appears in the original text . ---- **Example:** *Part 1:* ... *Part 2* `Liam Tarquin Broady (BROH-dee; born 4 January 1994) is a British professional tennis player who competes mainly on the ATP Challenger Tour. He reached a career high ranking | of world | No. | 93 on | 25 September | 2023.` | | | ----------------- | --- | ----- | ------------ | ------ | --- | | *Correct Output:* | | | | | | {{{{Liam Tarquin Broady; PERSON}}}} (BROH-dee; born {{4 January 1994; DATE}}) is a {{{{ British; GROUP}}}} professional {{{{tennis player; TITLE}}}} who competes mainly on the {{{{ATP Challenger Tour; EVENT}}}}. He reached a career high ranking of world No. | {{{{93; | NUMERIC_VALUE}}}} | | | on 25 September | 2023. | | ------- | ----------------- | --- | --- | --------------- | ----- | Explanation: 1. **Liam Tarquin Broady**: This is the full name of an individual human, therefore, it as a **PERSON**. 2. **4 January 1994**: This is a specific point in time, a full **DATE**. 3. **British**: This word describes a nationality or cultural affiliation. According to | the definitions, | | this | is a | **GROUP**. | | | ---------------- | --- | ---- | ---- | ---------- | --- | 4. **tennis player**: This is a title describing a person's profession or role. The | correct | type | is **TITLE**. | | | | | ------- | ---- | ------------- | --- | --- | --- | 5. **ATP Challenger Tour**: This is a named series of tennis tournaments, which is a type | of recurring | | **EVENT**. | | | | | ------------ | --- | ---------- | --- | --- | --- | 6. **93**: This is a number representing a specific quantity (a world ranking), therefore, | it is | a **NUMERIC_VALUE**. | | | | | | ------------- | -------------------- | --- | --- | --- | --- | | **Instruction | End** | | | | | **Please process the following paragraph and identify up to eight entities:** {paragraph} | BoxE.3: PromptforSelectingObjectEntities | | | | | | | ---------------------------------------- | --- | --- | --- | --- | --- | You will be provided with a paragraph annotated with entities in the format of `{{{{ entity_name}}}}`. Your task is to select up to {n_entities} **OBJECT** entities for the pre-training | completion | task | (causal | language | modeling). | | | ---------- | ---- | ------- | -------- | ---------- | --- | Each OBJECT entity should complete a natural "fill-in-the-blank" based on the preceding | text: the | *left-context*. | | | | | | --------- | --------------- | --- | --- | --- | --- | Specifically, the left-context should clearly indicate **what kind of entity** (e.g., a person, country, or biological species) is expected at that position. However, correctly identifying the **specific entity value** (e.g., "South Africa", "banana | plants") | should | require | recalling | a world | fact. | | -------- | ------ | ------- | --------- | ------- | ----- | These selected entities serve as factual anchors for generating our suite of completion- | based and | QA-based | robustness | | tests. | | | ------------------ | -------- | ------------- | --- | -------- | ----------------- | | ### **Instructions | | for selecting | | suitable | OBJECT entities** | 35 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality * Select OBJECT entities that involve recalling factual knowledge, and is not trivial to guess. * The left-context (the text that precedes the entity) must suggest **what kind** of entity is expected (e.g., a date, a country, a person's name, a title, a profession, a | species, | an event, | a number, | etc). | | | | -------- | --------- | --------- | ----- | --- | --- | * The left-context should contain enough information to strongly suggest a single, unique, | unambiguous | completion. | | | | | | ----------- | ----------- | --- | --- | --- | --- | * OBJECT entities whose specific value can be predicted solely based on the "left-context" | or are | too easy to | guess | are considered | poor | choices. | | ------ | ----------- | ----- | -------------- | ---- | -------- | * *Example:* `Bang bang chicken (Chinese: ...) is a popular dish in {{{{Chinese cuisine }}}}` -> {{{{Chinese cuisine}}}} is a poor choice because the Chinese name makes the | answer | too obvious. | | | | | | ------ | ------------ | --- | --- | --- | --- | * *Example:* `Airlink is based in Johannesburg, {{{{South Africa}}}}` -> {{{{South Africa }}}} is a poor choice because it can be easily guessed from the mention of | Johannesburg, | without | requiring | any | knowledge | about Airlink. | | ------------- | ------- | --------- | --- | --------- | -------------- | * The OBJECT should have a strong relationship with an entity mentioned earlier in the text . * *Example:* `{{{{Panama disease}}}}... infects {{{{banana plants}}}}` -> Knowing what Panama disease is makes "banana plants" a highly predictable object of the verb " infects". * Do not select OBJECT entities from the start of a sentence. Ideally, the OBJECT entity | should | appear at | the end | of a sentence | or complete | clause. | | ------ | --------- | ------- | ------------- | ----------- | ------- | * Ensure the left-context provides sufficient disambiguation to yield a unique and specific completion. If the blank could plausibly be filled by multiple entity types, it is not | a suitable | choice. | | | | | | ---------- | ------- | --- | --- | --- | --- | * *Example:* The left-context "Michael Jordan was born on..." is ambiguous, as multiple people share this name (e.g., the basketball player and the ML researcher). In contrast, "Michael Jeffrey Jordan was born on..." and "Michael Jordan, the famous basketball player, was born on..." clearly indicate a single referent, enabling a | unique | factual | prediction. | | | | | ------ | ------- | ----------- | --- | --- | --- | * *Example:* `Liam Tarquin Broady is a...` -> This is a poor choice. The context is too broad and could be completed with multiple entities, such as a profession (tennis | player), | nationality | (British), | or | a descriptor | (left-handed). | | -------- | ----------- | ---------- | --- | ------------ | -------------- | * The immediate context preceding the entity should constrain the expected entity type, strongly signaling what kind of entity is likely to follow. The OBJECT entity can | follow | a preposition | that | narrows down | the possibilities. | | | ------ | ------------- | ---- | ------------ | ------------------ | --- | * *Example:* `...born on {{{{4 January 1994}}}}` -> The phrase "born on" strongly | suggests | a date. | | | | | | -------- | ------- | --- | --- | --- | --- | * *Example:* `...caused by the fungus {{{{Fusarium oxysporum f. sp. cubense}}}}` -> The phrase "the fungus" clearly indicates that a specific fungal name should follow. * *Example:* `The Virginia Plan (also known as the {{{{Randolph Plan}}}})...` -> The | phrase | "also known | as" | implies an | alternative | name is expected. | | ------ | ----------- | --- | ---------- | ----------- | ----------------- | * Do not consider any information that appears after the OBJECT entity when making your selection. * If the uniqueness or the disambiguation of the entity type relies on information that | appears | after the | entity, | it is a poor | OBJECT | choice. | | ------- | --------- | ------- | ------------ | ------ | ------- | * *Example:* `It is their first album to feature {{{{Matt Wachter}}}} on bass.` -> {{{{ Matt Wachter}}}} is a poor choice. The phrase "It is their first album to feature..." does not constrain the prediction. The disambiguating detail "on bass" comes *after* the entity, which violates the instruction that the **OBJECT** must be inferable | solely | from the | left-context. | | | | | ------ | -------- | ------------- | --- | --- | --- | * You must select up to {n_entities} suitable OBJECT entities and return them in a list, | sorted | from the most | to the | least suitable. | | | | ------ | ------------- | ------ | --------------- | --- | --- | ---- ### **Examples** ### **Text 1** {{{{Panama disease}}}} (or Fusarium wilt) is a plant {{{{disease}}}} that infects {{{{ banana plants}}}} ({{{{Musa spp.}}}}). It is a wilting disease caused by the fungus {{{{Fusarium oxysporum f. sp. cubense}}}} (Foc). During the {{{{1950s}}}}, an outbreak of Panama disease almost wiped out commercial {{{{Gros Michel}}}} banana production. * {{{{Panama disease}}}} - It appears in the first clause of the first sentence and is | therefore | ineligible. | This | choice is | **poor**. | | | --------- | ----------- | ---- | --------- | --------- | --- | * {{{{disease}}}} - This entity does not require recalling a world fact; it can be predicted from the context clues "Fusarium wilt" and "infects." This choice is **poor 36 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality **. * {{{{Musa spp.}}}} - While recalling the scientific name for banana plants is a good test of knowledge, the use of parentheses makes the answer less certain. The context | could also | imply | other | related | species. | This choice | is **good**. | | ---------- | ----- | ----- | ------- | -------- | ----------- | ------------ | * {{{{banana plants}}}} - The context "a plant disease that infects..." combined with the subject "Panama disease" creates a very strong expectation for this specific plant, a | well-known | fact. | This | choice | is **excellent**. | | | | ---------- | ----- | ---- | ------ | ----------------- | --- | --- | * {{{{Fusarium oxysporum f. sp. cubense}}}} - The preceding phrase "...caused by the fungus..." requires a specific, factual answer. Knowing the fungus that causes Panama disease is a clear test of world knowledge. This choice is **excellent**. * {{{{1950s}}}} - The phrase "During the..." is too vague. It could be completed with many different time periods, making the answer unpredictable. This choice is **poor**. * {{{{Gros Michel}}}} - The context "...wiped out commercial..." strongly points to a specific type of banana. However, it doesn't uniquely identify the variety without further knowledge, so it's not perfectly constrained. This choice is **good**. ### **Text 2** Liam Tarquin Broady (BROH-dee; born {{{{4 January 1994}}}}) is a {{{{British}}}} professional {{{{tennis player}}}} who competes mainly on the {{{{ATP Challenger Tour }}}}. He reached a career high ranking of world No. {{{{93}}}} on 25 September 2023. * {{{{4 January 1994}}}} - The entity appears in the first clause of the first sentence, | making | it ineligible | | per the | instructions. | This | choice is **poor**. | | ------ | ------------- | --- | ------- | ------------- | ---- | ------------------- | * {{{{British}}}} - The left-context "...is a..." is too generic and does not | sufficiently | narrow | down | the | possibilities. | This | choice is **poor**. | | ------------ | ------ | ---- | --- | -------------- | ---- | ------------------- | * {{{{tennis player}}}} - The context "...is a British professional..." strongly implies a profession. Knowing who Liam Broady is leads to a single, specific prediction. This | choice | is **excellent**. | | | | | | | ------ | ----------------- | --- | --- | --- | --- | --- | * {{{{ATP Challenger Tour}}}} - The context "who competes mainly on the..." points to a tournament or tour. However, the word "mainly" introduces uncertainty, meaning the | answer | is not guaranteed | | to | be unique. | This choice | is **good**. | | ------ | ----------------- | --- | --- | ---------- | ----------- | ------------ | * {{{{93}}}} - The preceding phrase "...career high ranking of world No..." creates a very specific slot for a number. Recalling a player's exact ranking is a great test of | factual | knowledge. | This | choice | is **excellent**. | | | | ------- | ---------- | ---- | ------ | ----------------- | --- | --- | ### **Text 3** Happiness for Beginners is a {{{{2023}}}} American {{{{romantic comedy}}}} film starring {{{{Ellie Kemper}}}} and {{{{Luke Grimes}}}}, an adaptation of the {{{{Katherine Center }}}} 2015 novel of the same name. The novel was adapted for the screen and directed by | {{{{Vicky | Wight}}}}. | | | | | | | --------- | ---------- | --- | --- | --- | --- | --- | * {{{{2023}}}} - This entity appears in the first clause, making it ineligible. Additionally, the context "...is a..." is too general to uniquely predict a year. This | choice | is **poor**. | | | | | | | ------ | ------------ | --- | --- | --- | --- | --- | * {{{{Ellie Kemper}}}} - The phrase "film starring" clearly points to an actor. However, since a film can have multiple stars, the specific actor named first is not uniquely | predictable. | This | choice | is | **good**. | | | | ------------ | ---- | ------ | --- | --------- | --- | --- | * {{{{Luke Grimes}}}} - The context "...starring Ellie Kemper and..." creates a strong expectation for a co-star, a common structure in this genre. This requires specific | knowledge | about | the film. | This | choice | is **excellent**. | | | --------- | ----- | --------- | ---- | ------ | ----------------- | --- | * {{{{Katherine Center}}}} - The crucial context clue, "novel," appears *after* the entity. Therefore, the answer cannot be predicted from the left-context alone. This | choice | is **poor**. | | | | | | | ------ | ------------ | --- | --- | --- | --- | --- | * {{{{Vicky Wight}}}} - The left-context "...adapted for the screen and directed by..." clearly and uniquely signals that a director's name is required. This choice is ** excellent**. ---- | ### **Required | Output | Format** | | | | | | -------------- | ------ | -------- | --- | --- | --- | --- | Return a Python list of selected **OBJECT** entities, sorted from most suitable to least. Each entity must be written in the format `{{{{entity_name}}}}`, exactly as it appears | in the | original | text. | | | | | | ------ | -------- | ----- | --- | --- | --- | --- | Example: ["{{{{entity1_name}}}}", "{{{{entity2_name}}}}", "{{{{entity3_name}}}}"] | **Instruction | End** | | | | | | | ------------- | ----- | --- | --- | --- | --- | --- | 37 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality **Please select up to three OBJECT entities from the following annotated paragraph, and return them in a list sorted by the most suitable OBJECT to the least:** {tagged_paragraph} BoxE.4: PromptforVerifyingObjectEntitySuitability–withGoldTarget You will be provided with a *left-context* (a non-completed sentence or paragraph) and a proposed **OBJECT** entity that completes it. Your task is to verify whether this OBJECT entity is a **suitable factual completion**, meaning: 1. The left-context provides enough semantic cues to **strongly constrain** what type of entity should come next (e.g., a date, person, award, location, number, title, etc.). 2. There is a **unique and specific** gold entity completion which is the OBJECT entity: * There are **no multiple plausible alternative completions** that refer to different entities. * If a language model predicts a different entity, it would **clearly contradict** the information conveyed by the *left-context* and the OBJECT. * The OBJECT entity is not underdetermined, vague, or overly general. ---- ### **Guidelines for Determining Suitability** A suitable OBJECT entity must meet **all** of the following: * The left-context clearly constrains the **type** of entity expected next (e.g., a food, a date, a chemical stabilizer, a person's name, a specific event). * The OBJECT entity fits seamlessly into the left-context to form a coherent, factual statement. * The left-context likely **narrows the answer to a unique entity** and any alternative completion would conflict with the facts conveyed so far. The OBJECT is **not suitable** if: * The left-context is too vague or underspecified to narrow the completion to a single, specific entity. * There are multiple plausible completions of the same type (e.g., many possible awards, people, or ingredients). * The constraints on the OBJECT rely on **right-context** (i.e., text that appears after the entity). * The OBJECT appears to fit syntactically or thematically, but the left-context provides ** no strong evidence** that it is the only valid answer. *Clarification on Uniqueness:* uniqueness refers to the **semantic identity** of the OBJECT entity and not to surface forms (the string used to represent an entity). The OBJECT is still considered unique even if it has other forms that: * Differ in capitalization, punctuation, or grammatical form (e.g., pluralization, articles ). * Omit non-essential modifiers (e.g., "Pulitzer Prize for Fiction" vs. "2003 Pulitzer Prize for Fiction"). * Use a common variant or alias (e.g., "United States" vs. "U.S."). However, if multiple **distinct** entities (e.g., different stabilizers, places, awards, or people) could reasonably complete the left-context, then the OBJECT is **not suitable **. ---- ### **NOT SUITABLE Examples** **Left-context:** Garfield is portrayed as a lazy, fat, cynical, and self-absorbed orange tabby Persian cat. 38 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | He is | noted for | his love | of | lasagna | and * | | | | ----------- | --------- | -------- | --- | ------- | ----- | --- | --- | | **OBJECT:** | pizza | | | | | | | **Explanation:** The left-context does not constrain the answer to be a food. Other plausible completions include "naps", "Mondays", or "sleep". The OBJECT is not uniquely inferable. **Left-context:** Glass noodles, sometimes called cellophane noodles, are a type of transparent noodle made from starch (such as mung bean starch, potato starch, sweet potato starch, tapioca, or canna starch) and water. They originated in China. A stabilizer such as chitosan or * | **OBJECT:** | alum | | | | | | | | ----------- | ---- | --- | --- | --- | --- | --- | --- | **Explanation:** The left-context signals that a stabilizer is expected, but it does not sufficiently constrain which one. Multiple valid completions exist, so the OBJECT is not unique. **Left-context:** Rayagada railway station is located at the heart of the city and now it has the longest platform in Odisha (Platform No :- 3 measuring 910m), the 6th in the country and the | **OBJECT:** | 7th | | | | | | | | ----------- | --- | --- | --- | --- | --- | --- | --- | **Explanation:** The left-context does not clarify what the "7th" refers to. The comparison class is underspecified (7th in what? globally? regionally?), so the answer is ambiguous. ---- | ### **SUITABLE | Examples** | | | | | | | | -------------- | ---------- | --- | --- | --- | --- | --- | --- | **Left-context:** Adrian Darnell Griffin Sr. (born July 4, 1974) is an American professional basketball coach and former player who most recently served as the head coach for the Milwaukee Bucks of the National Basketball Association (NBA). He played in the NBA as a shooting guard | and | small forward | from | 1999 | to 2008. | Griffin | grew up | in * | | ----------- | ------------- | ---- | ---- | -------- | ------- | ------- | ---- | | **OBJECT:** | Wichita | | | | | | | **Explanation:** The phrase "grew up in" combined with known biographical details provides strong context for a specific city. Other completions would clearly contradict the | known | fact. | | | | | | | | ----- | ----- | --- | --- | --- | --- | --- | --- | **Left-context:** Andrea Barrett (born November 16, 1954) is an American novelist and short story writer. Her collection Ship Fever won the 1996 U.S. National Book Award for Fiction, and she received a MacArthur Fellowship in 2001. Her book Servants of the Map was a finalist | for | the 2003 * | | | | | | | | ----------- | ---------- | ----- | --- | ------- | --- | --- | --- | | **OBJECT:** | Pulitzer | Prize | for | Fiction | | | | **Explanation:** While there are many awards, the combination of book title, year, and prior accolades strongly constrains the answer. The context likely narrows the answer | to a | unique award. | | | | | | | | ---- | ------------- | --- | --- | --- | --- | --- | --- | **Left-context:** Bering Air Flight 445 was a scheduled domestic flight from Unalakleet Airport to Nome Airport which crashed on February 6, 2025 while flying over the Norton Sound. The | flight | was aboard | a * | | | | | | | ----------- | ---------- | ---------- | ------- | --- | --- | --- | --- | | **OBJECT:** | Cessna | 208B Grand | Caravan | | EX | | | **Explanation:** The left-context refers to a known historical event. There is only one aircraft associated with this flight, making the answer uniquely identifiable. **Left-context:** George Mark Collie (born January 18, 1956) is an American country music singer, songwriter, musician, actor, record producer, and fundraiser for Type 1 diabetes study. He has won awards and acclaim for his music, his acting, and his philanthropy. His singing career has included five major-label albums: four for MCA Nashville and one for Giant Records | . Sixteen | of his | singles | have | charted | on | * | | | ----------- | ----------- | ------- | ---- | ------- | --- | --- | --- | | **OBJECT:** | Hot Country | Songs | | | | | | **Explanation:** The specific number of charting singles and genre context constrain the 39 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality answer to a unique Billboard chart. Alternate completions would contradict the left- context. ---- | ### **Required | | Output | Format** | | | | | | | ----------------- | --------------- | --------------- | --------- | ------------ | --- | --- | ---------- | --------- | | Respond | only | with "SUITABLE" | | or "NO". | | | | | | **Instruction | | End** | | | | | | | | **Please | process | the | following | left-context | | and | the OBJECT | entity:** | | **Left-context:** | | {left_context} | | | | | | | | **OBJECT:** | {object_entity} | | | | | | | | BoxE.5: PromptforVerifyingObjectEntitySuitability–withoutGoldTarget You will be provided with a *left-context* (a non-completed sentence or paragraph). Your task is to verify whether this left-context is a **suitable factual completion task**, meaning: 1. The left-context provides enough semantic cues to **strongly constrain** what type of entity should come next (e.g., a date, person, award, location, number, title, etc.). | 2. There | is a | **unique | and | specific** | gold | entity | completion: | | | -------- | ---- | -------- | --- | ---------- | ---- | ------ | ----------- | --- | * There are **no multiple plausible alternative completions** that refer to different entities. * If a language model predicts a different entity than the gold entity, it would ** clearly contradict** the information conveyed by the *left-context* and the gold entity. ---- | ### **Guidelines | | for | Determining | | Suitability** | | | | | ---------------- | ------------ | --- | ----------- | ---- | ------------- | ------ | ---------- | --- | | A suitable | left-context | | must | meet | **all** | of the | following: | | * The left-context clearly constrains the **type** of entity expected next (e.g., a food, a | date, | a | chemical | stabilizer, | | a person's | name, | a | specific event). | | ----- | --- | -------- | ----------- | --- | ---------- | ----- | --- | ---------------- | * The left-context likely **narrows the answer to a unique entity** and any alternative | completion | | would | conflict | with | the facts | conveyed | | so far. | | ---------------- | --- | ----- | -------- | ---------- | --------- | -------- | --- | ------- | | The left-context | | is | **not | suitable** | if: | | | | * The left-context is too vague or underspecified to narrow the completion to a single, | specific | | entity. | | | | | | | | -------- | --- | ------- | --- | --- | --- | --- | --- | --- | * There are multiple plausible completions of the same type (e.g., many possible awards, | people, | or | ingredients). | | | | | | | | ------- | --- | ------------- | --- | --- | --- | --- | --- | --- | *Clarification on Uniqueness:* uniqueness refers to the **semantic identity** of the completion and not to surface forms (the string used to represent an entity). The completion is still considered unique even if it has other forms that: * Differ in capitalization, punctuation, or grammatical form (e.g., pluralization, articles ). * Omit non-essential modifiers (e.g., "Pulitzer Prize for Fiction" vs. "2003 Pulitzer Prize | for | Fiction"). | | | | | | | | | ------- | ---------- | ------- | --- | ------------ | ------- | --- | ------- | ------------ | | * Use a | common | variant | or | alias (e.g., | "United | | States" | vs. "U.S."). | However, if multiple **distinct** entities (e.g., different stabilizers, places, awards, or people) could reasonably complete the left-context, then it is **not suitable**. ---- | ### **NOT | SUITABLE | | Examples** | | | | | | | --------- | -------- | --- | ---------- | --- | --- | --- | --- | --- | 40 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality **Left-context:** Garfield is portrayed as a lazy, fat, cynical, and self-absorbed orange tabby Persian cat. He is noted for his love of lasagna and * **Explanation:** The left-context does not constrain the answer to a single entity type. There are many plausible completions such as "pizza", "naps", "Mondays", or "sleep". **Left-context:** Glass noodles, sometimes called cellophane noodles, are a type of transparent noodle made from starch (such as mung bean starch, potato starch, sweet potato starch, tapioca, or canna starch) and water. They originated in China. A stabilizer such as chitosan or * **Explanation:** The left-context signals that a stabilizer is expected, but it does not sufficiently constrain which one. Multiple valid completions exist, so the completion is not unique. **Left-context:** Rayagada railway station is located at the heart of the city and now it has the longest platform in Odisha (Platform No :- 3 measuring 910m), the 6th in the country and the * **Explanation:** The left-context does not clarify what should come after "...and the". Even if a number is likely ("...and the Xth..."), the comparison class is underspecified (Xth in what? globally? regionally?), so the answer is ambiguous. ---- ### **SUITABLE Examples** **Left-context:** Adrian Darnell Griffin Sr. (born July 4, 1974) is an American professional basketball coach and former player who most recently served as the head coach for the Milwaukee Bucks of the National Basketball Association (NBA). He played in the NBA as a shooting guard and small forward from 1999 to 2008. Griffin grew up in * **Explanation:** The phrase "grew up in" combined with known biographical details provides strong context for a specific city. Other completions would clearly contradict the known fact. **Left-context:** Andrea Barrett (born November 16, 1954) is an American novelist and short story writer. Her collection Ship Fever won the 1996 U.S. National Book Award for Fiction, and she received a MacArthur Fellowship in 2001. Her book Servants of the Map was a finalist for the 2003 * **Explanation:** While there are many awards, the combination of book title, prior accolades and the year, strongly constrains the answer. The left-context likely narrows the answer to a unique award. **Left-context:** Bering Air Flight 445 was a scheduled domestic flight from Unalakleet Airport to Nome Airport which crashed on February 6, 2025 while flying over the Norton Sound. The flight was aboard a * **Explanation:** The left-context refers to a known historical event. There is only one aircraft associated with this flight, making the answer unique. **Left-context:** George Mark Collie (born January 18, 1956) is an American country music singer, songwriter, musician, actor, record producer, and fundraiser for Type 1 diabetes study. He has won awards and acclaim for his music, his acting, and his philanthropy. His singing career has included five major-label albums: four for MCA Nashville and one for Giant Records . Sixteen of his singles have charted on * **Explanation:** The specific number of charting singles and genre context constrain the answer to a unique Billboard chart. Alternate completions would contradict the left- context. ---- 41 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | ### **Required | Output | Format** | | | | | ----------------- | -------------------------------------- | --------- | --------------- | --- | --- | | Respond | only with "SUITABLE" | | or "NO". | | | | **Instruction | End** | | | | | | **Please | process the | following | left-context:** | | | | **Left-context:** | {left_context} | | | | | | BoxE.6: | PromptforVerifyingCompletionDifficulty | | | | | You will be provided with a *left-context* (a non-completed sentence or paragraph) and a | proposed | **OBJECT** | entity | that completes | it. | | | -------- | ---------- | ------ | -------------- | --- | --- | Your task is to verify whether this OBJECT entity is a **suitable factual completion**. | The OBJECT | is **suitable** | | if: | | | | ---------- | --------------- | --- | --- | --- | --- | 1. The completion is challenging and requires specific factual knowledge. 2. It is not trivially easy to guess or logically infer from the left-context alone. 3. The left-context does not contain the answer or any strong hints that directly disclose | the | OBJECT entity. | | | | | | --- | -------------- | --- | --- | --- | --- | 4. The OBJECT entity is not time-sensitive and will not change over time. | The OBJECT | is **not | suitable** | if: | | | | ---------- | -------- | ---------- | --- | --- | --- | 1. The OBJECT entity can be predicted using general common sense or an educated guess. 2. The OBJECT entity is trivially inferable using general common knowledge. 3. The OBJECT entity, as a completion of the left-context, is subject to change over time ( | it | is not static). | | | | | | --- | --------------- | --- | --- | --- | --- | ---- | ### **NOT | SUITABLE Examples** | | | | | | --------- | ------------------- | --- | --- | --- | --- | **Left-context:** The Strait of Hormuz (Persian: ... Tangeh-ye Hormoz) is situated between * | **OBJECT:** | Iran | | | | | | ----------- | ---- | --- | --- | --- | --- | **Explanation:** Since the left-context provides the name in Persian script, the answer " Iran" can be easily guessed. Therefore, the completion is not challenging and is | considered | not suitable. | | | | | | ---------- | ------------- | --- | --- | --- | --- | **Left-context:** The Nobel Prize in Literature is a Swedish literature prize that is awarded annually to an author from any country who has, in the words of Alfred Nobel, "in the field of literature, produced the most outstanding work in an idealistic direction". It is one | of | the five * | | | | | | ----------- | ------------ | --- | --- | --- | --- | | **OBJECT:** | Nobel Prizes | | | | | **Explanation:** The completion is trivial because the subject ("The Nobel Prize in | Literature") | already | contains | the target | OBJECT | entity. | | ------------ | ------- | -------- | ---------- | ------ | ------- | **Left-context:** Income in the United States is measured by the various federal agencies including the Internal Revenue Service, Bureau of Labor Statistics, US Department of Commerce, and the US Census Bureau. Overall, including all households/individuals regardless of | employment | status, | the | median household | income | was * | | ----------- | ------- | --- | ---------------- | ------ | ----- | | **OBJECT:** | $67,521 | | | | | **Explanation:** The OBJECT entity is time-sensitive because the specific year is not specified in the context. Since median income changes over time, this completion is not suitable. ---- | ### **SUITABLE | Examples** | | | | | | -------------- | ---------- | --- | --- | --- | --- | 42 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality **Left-context:** Redbus India Private Limited, doing business as redBus, is an Indian multinational online | bus-ticketing | | platform. | | In 2018, | the company | achieved | a GMV of * | | ------------- | --- | --------- | ------ | -------- | ----------- | -------- | ---------- | | **OBJECT:** | 50 | billion | Indian | Rupee | | | | **Explanation:** The OBJECT entity is not time-sensitive because the context restricts the | fact | to a | specific | year | (2018), | making | the value static. | | | ---- | ---- | -------- | ---- | ------- | ------ | ----------------- | --- | **Left-context:** "Thriller" is a song by the American singer Michael Jackson. It includes a spoken-word | sequence | performed | | by the | horror | actor | * | | | ----------- | --------- | ----- | ------ | ------ | ----- | --- | --- | | **OBJECT:** | Vincent | Price | | | | | | **Explanation:** This is a challenging completion that requires the recall of specific | factual | knowledge; | | therefore, | | it is suitable. | | | | ------- | ---------- | --- | ---------- | --- | --------------- | --- | --- | **Left-context:** The Panathenaic Stadium is a multi-purpose stadium in Athens, Greece. After the rise of Christianity in the 4th century it was largely abandoned. After being refurbished, it | hosted | the | opening | and | closing | ceremonies | of the * | | | ----------- | ----- | ------- | -------- | ------- | ---------- | -------- | --- | | **OBJECT:** | first | modern | Olympics | | | | | **Explanation:** Although the stadium's location in Greece hints that the OBJECT is related to the Olympic Games, identifying the specific OBJECT requires knowing it hosted the * first modern* Olympics. This makes the completion non-trivial and suitable. ---- | ### **Required | | Output | Format** | | | | | | ----------------- | --------------- | -------------- | --------- | --------------- | --- | --- | --- | | Respond | only with | "SUITABLE" | | or "NO". | | | | | **Instruction | End** | | | | | | | | **Please | process | the | following | left-context:** | | | | | **Left-context:** | | {left_context} | | | | | | | **OBJECT:** | {object_entity} | | | | | | | E.2. QuestionGenerationPrompts | BoxE.7: | PromptforGeneratingtheDirectTask | | | | | | | | ------- | -------------------------------- | --- | --- | --- | --- | --- | --- | You will be provided with a *left-context* (an incomplete sentence or paragraph) and an * OBJECT* entity that completes it. Based only on the left-context, your task is to design a unambiguous question for which the only correct and specific answer is the OBJECT. The question should only use information necessary for disambiguation and to ensure the answer's specificity, using the exact phrasing from the left-context. **Requirements:** | * Only | the OBJECT | should | | be the | correct | answer. | | | ------ | ---------- | ------ | --- | ------ | ------- | ------- | --- | * The question must be self-contained: it should not use pronouns or references that | require | outside | context. | | | | | | | ------- | -------- | ----------- | --- | --------- | --- | ----------------- | --- | | * Use | only the | information | | available | in | the left-context. | | * The question must be unambiguous: its phrasing should be clear and avoid multiple interpretations. * The question must be specific and precise: it should specify the level of detail and type of answer required (e.g., "What year...", "What is the name of the condition...", | "Which | city..."). | | | | | | | | ------ | ---------- | --- | --- | --- | --- | --- | --- | * The question must be minimal: include only the essential details from the left-context needed to make the question unambiguous and ensure the OBJECT is the only possible answer. * Use the same terms, phrases, and words that appear in the left-context. Do not rephrase | or | use synonyms, | | unless | essential | for | grammaticality | or clarity. | | --- | ------------- | --- | ------ | --------- | --- | -------------- | ----------- | 43 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality ---- | ### **Required | Output | Format** | | | | | -------------- | ------------ | -------- | --- | -------------- | ----- | | Return only | the question | without | | any additional | text. | ---- ### **Examples** Here are a few examples, including explanations for how the questions were formulated: 1. **Left-context:** Erika Nordby (born February 2000), also known as Baby Erika, Miracle Baby and Canada's Miracle Child, is a Canadian originally from Edmonton, Alberta known for having been revived after spending two hours without a heartbeat due to | **OBJECT:** | hypothermia | | | | | | ----------- | ----------- | --- | --- | --- | --- | **Output:** What was the condition that caused Erika Nordby to spend two hours without a | heartbeat | before | she was | revived? | | | | --------- | ------ | ------- | -------- | --- | --- | *Explanation:* * We ask "What was the condition..." because hypothermia is a medical condition. This | phrasing | demands | a specific | type | of answer. | | | -------- | ------- | ---------- | ---- | ---------- | --- | * We include "Erika Nordby" to identify the person the question is about, making it self- contained. * We include "two hours without a heartbeat before she was revived" because these are the key, unique details from the context that pinpoint the specific cause being asked about. * We do not include the birth date or the nicknames to make the question minimal. The combination of the name and the incident of 'two hours without a heartbeat' is enough for disambiguation. | * All | terms are | taken directly | | from the left-context. | | | ----- | --------- | -------------- | --- | ---------------------- | --- | 2. **Left-context:** When I Look in Your Eyes is the fifth studio album by singer Diana Krall, | released | on | | | | | | ----------- | ---- | ------- | --- | --- | --- | | **OBJECT:** | June | 8, 1999 | | | | **Output:** What was the release date of When I Look in Your Eyes by Diana Krall? *Explanation:* * The phrasing "What was the release date..." is precise and directly corresponds to the | left-context | | phrase "released | | on." | | | ------------ | --- | ---------------- | --- | ---- | --- | * The album title could be ambiguous on its own, so we add "by Diana Krall" for disambiguation. * To keep the question minimal, we do not include the fact that it was her fifth studio album. The combination of the album title and the artist's name is sufficient for disambiguation. 3. **Left-context:** Celebrimbor is a fictional character in J. R. R. Tolkien's legendarium. In Tolkien's stories, Celebrimbor was an elven-smith who was manipulated into forging the Rings of Power by the Dark Lord Sauron, in a fair disguise and named | **OBJECT:** | Annatar | | | | | | ----------- | ------- | --- | --- | --- | --- | **Output:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? *Explanation:* * "What name..." sets the clear expectation for the type of answer required. * We must include "in his fair disguise" and "to manipulate Celebrimbor" because these are the specific circumstances under which Sauron used the name Annatar. Simply asking | "What | name did | Sauron | use?" would | be too | broad. | | ----- | -------- | ------ | ----------- | ------ | ------ | * Further information is not required to make the question unique and unambiguous. For example, mentioning Celebrimbor and Sauron is enough to understand that they are | characters | from | Tolkien's | legendarium. | | | | ---------- | --------- | -------------- | ------------ | ---------------------- | --- | | * All | terms are | taken directly | | from the left-context. | | 4. **Left-context:** The horse-collar tackle is a gridiron football maneuver in which a defender tackles another player by grabbing the back collar or the back-inside of an opponent's shoulder pads and pulling the ball carrier directly downward violently in order to pull his feet from underneath him. The technique is most closely associated 44 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality with Pro Bowl safety Roy Williams. After being blamed for a series of major injuries in the 2004 season , the horse-collar tackle was banned from the NFL during the 2005 off- | | season. | The | rule forbidding | | it is | often referred | | to in the press | as | | ----------- | ------- | --- | --------------- | ---- | ----- | -------------- | --- | --------------- | --- | | **OBJECT:** | | The | Roy Williams | Rule | | | | | | **Output:** What is the name the press often uses to refer to the rule forbidding the horse | | -collar | tackle | from | the NFL? | | | | | | | --- | ------- | ------ | ---- | -------- | --- | --- | --- | --- | --- | *Explanation:* * To make the question self-contained, we must describe the rule being referenced: "the | | rule forbidding | | the | horse-collar | tackle | from | the | *NFL*". | | | --- | --------------- | --- | --- | ------------ | ------ | ---- | --- | ------- | --- | * The left-context specifies the rule is "referred to in the press as..." This detail is critical. Therefore, the question includes "What is the name the press often uses to | | refer | to..." | to remain | faithful | to | the source | text. | | | | --- | ----- | ------ | --------- | -------- | --- | ---------- | ----- | --- | --- | * The years, the description of the maneuver, and the injuries are unnecessary for | | understanding | | the question | and | can | be excluded | | for minimalism. | | | ----------------- | ------------------------------ | --------------- | -------------- | ------------ | -------- | ------------- | --------- | --------------- | --- | | * | All terms | are | taken | directly | from the | left-context. | | | | | **Instruction | | End** | | | | | | | | | **Please | process | | the following | left-context | | and | OBJECT:** | | | | **Left-context:** | | | {left_context} | | | | | | | | **OBJECT:** | | {object_entity} | | | | | | | | | BoxE.8: | PromptforRefiningSpecification | | | | | | | | | You will be provided with a question, a gold answer, and its corresponding entity type. Your task is to evaluate the question's specificity and precision, with respect to the | | gold answer, | | and revise | the | question | if necessary. | | | | | --- | ------------ | --- | ---------- | --- | -------- | ------------- | --- | --- | --- | A specific and precise question should clearly define the level of detail required in the answer. Your evaluation process should consider the following stages: **Verification**, **Revision | | ** and | **Rejection**. | | | | | | | | | --- | ------ | -------------- | --- | --- | --- | --- | --- | --- | --- | ### **Verification** Determine if the question is specific and precise. Consider the following when making your decision: * Would a person with the necessary knowledge, who is familiar with the gold answer, be | | led to | provide | that | specific | gold | answer | to the | question? | | | --- | ------ | ------- | ---- | -------- | ---- | ------ | ------ | --------- | --- | * Does the question specify the required level of detail and type for the answer? * Is it easy to infer the required entity type of the answer from the question? * If relevant, does the question specify the required granularity of the answer? * For example: "What year...", "What year and month...", "What date...", "What is the name of the condition...", "Which city...", "Which country...", "Which province | | ...", | "What | is the | scientific | | name...". | | | | | --- | ----- | ----- | ------ | ---------- | --- | --------- | --- | --- | --- | If the question is specific and precise, you can stop here and return it exactly as it is. Note that most of your inputs will be considered specific and precise. If the question is not specific or precise, you should try to revise it. ### **Revision** A question should be revised if it is not specific but can be fixed with **minimal changes **. A valid revision must result in a question that perfectly and uniquely targets the gold | | answer | without | leaking | non-trivial | | information. | | | | | --- | ------ | ------- | ------- | ----------- | --- | ------------ | --- | --- | --- | Common revision techniques involve adding a short, clarifying phrase or changing a single | | word, | for example: | | | | | | | | | --- | ----- | ------------ | --- | --- | --- | --- | --- | --- | --- | * **Specify the answer's entity type.** This is often done by changing the question word | | to be | more precise, | | e.g., `Who` | -> | `Which | company`. | | | | --- | ----- | ------------- | --- | ----------- | --- | ------ | --------- | --- | --- | * **Specify the answer's required granularity or format.** This is often done by clarifying the necessary level of detail. e.g., `When` -> `In what month and year`, ` | | Where` | -> `In | which | city and | country`, | | | | | | ----- | ------ | -------- | ----- | -------------- | --------- | ------ | --- | --- | --- | | `What | name` | -> `What | is | the scientific | | name`. | | | | 45 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality * **Specify the units or precision for numerical data.** This removes ambiguity from the expected answer. e.g., Adding `in Celsius`, `in US dollars`, or `in millions`. * **Add a qualifier to narrow the scope.** When a question is ambiguous and could have multiple correct answers, add a qualifier to focus the question on the single intended answer. Note that you should not disclose non-trivial information. e.g., Revise `What ecosystem...`? to `What ecosystem **of mountains**...`? to distinguish from other valid | answers | (e.g., a forest). | | | | ------- | ----------------- | --- | --- | ### **Rejection** If a question cannot be fixed with a valid minimal revision, it must be rejected. A question is considered unfixable and should be rejected in the following cases: * The revision would leak information: The only way to fix the question would be to add a non-trivial clue that is part of the gold answer itself, making the question | significantly | easier. | | | | ------------- | ------- | --- | --- | * The revision would require a major change: The question is too vague or fundamentally mismatched with the answer, and fixing it would require a complete rewrite rather than | a simple, | minimal clarification. | | | | --------- | ---------------------- | --- | --- | ---- | ### **Required | Output Format** | | | | -------------- | --------------- | --- | --- | You must return *only* a string, which can be the original question, the revised question, or "REJECT". If the question is specific and precise as it is, return the question. If the question needs revision, revise it and return the revised question. If you conclude that the | question | should be rejected, | return | "REJECT". | | -------- | ------------------- | ------ | --------- | Remember, return only the original question, the revised question, or "REJECT", without any | additional | text. | | | | ---------- | ----- | --- | --- | ---- ### **Examples** 1. **Question:** What is the condition that caused Erika Nordby to spend two hours without a | heartbeat | before she | was revived? | | | ---------------- | ------------ | ------------ | ------------ | | **Gold answer:** | hypothermia | | | | **Explanation:** | The question | is specific | and precise. | **Output:** What is the condition that caused Erika Nordby to spend two hours without a | heartbeat | before she | was revived? | | | --------- | ---------- | ------------ | --- | 2. **Question:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? | **Gold answer:** | Annatar | | | | ---------------- | ------------ | ----------- | ------------ | | **Explanation:** | The question | is specific | and precise. | **Output:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? 3. **Question:** Who produced the 2020 German romantic comedy film Isi & Ossi? | **Gold answer:** | Netflix | | | | ---------------- | ------- | --- | --- | **Explanation:** The question word "Who" is ambiguous, as it typically refers to a person. The revision to "Which company" precisely targets the entity type of the answer. **Output:** Which company produced the 2020 German romantic comedy film Isi & Ossi? 4. | **Question:** | What biodiverse | ecosystem | surrounds Lake Burton? | | ---------------- | --------------- | --------- | ---------------------- | | **Gold answer:** | Blue Ridge | Mountains | | **Explanation:** The term "ecosystem" is too broad, allowing for multiple correct answers ( e.g., a forest). The qualifier "of mountains" narrows the scope to the intended answer | without | leaking information. | | | | ------- | -------------------- | --- | --- | **Output:** What biodiverse ecosystem of mountains surrounds Lake Burton? | **Instruction | End** | | | | ------------- | ----- | --- | --- | **Please process the following question, gold answer, and entity type:** 46 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | **Question:** | | {question} | | | | | | | | | ------------- | --------------------------- | ------------- | --- | --- | --- | --- | --- | --- | --- | | **Gold | answer:** | {gold_answer} | | | | | | | | | **Entity | type:** | {entity_type} | | | | | | | | | BoxE.9: | PromptforRefiningMinimalism | | | | | | | | | You will be provided with a question and its gold answer. Your task is to evaluate the question's minimalism and revise it by removing any non-essential information. The question must be unambiguous and lead to a single, unique answer, while avoiding non- essential details that are not required for the disambiguation of the core entities in | | the question | | or the answer. | | | | | | | | --- | ------------ | --- | -------------- | --- | --- | --- | --- | --- | --- | The primary goal is not simply to make the question shorter, but to remove information that acts as an unnecessary hint (which helps answer the question) but does not aid in disambiguation. Your evaluation process should consider the following stages: **Verification** and ** Revision**. ### **Verification** Determine if the question is already minimal. A question is considered minimal if it uniquely leads to the provided gold answer without containing unnecessary details. | | Consider | the | following: | | | | | | | | --- | -------- | --- | ---------- | --- | --- | --- | --- | --- | --- | * Are all entities in the question (people, works, events) identified with just enough detail to be unambiguous for the purpose of the question? For example, adding "by Diana Krall" is essential if the song title "When I Look in Your Eyes" might not be unique | | on its | own, as | there | may be another | | song with | the same | title. | | | --- | ------ | ------- | ----- | -------------- | --- | --------- | -------- | ------ | --- | * Does the question contain any superfluous words or phrases (e.g., adjectives, specific years, genres) that could be removed without affecting the question's ability to point | | uniquely | to the | required | type | of gold | answer? | | | | | --- | -------- | ------ | -------- | ---- | ------- | ------- | --- | --- | --- | * Does the question avoid providing unnecessary hints or clues? A hint is a piece of information that makes the answer easier but is not needed for the identification of | | the entities | | or the intended | | answer | from other | possibilities. | | | | --- | ------------ | --- | --------------- | --- | ------ | ---------- | -------------- | --- | --- | * Note: Most questions are already minimal. Minimalism is not about shorter questions, but about not including details that provide a hint without being essential for the | | question's | disambiguation. | | | | | | | | | --- | ---------- | --------------- | --- | --- | --- | --- | --- | --- | --- | If the question is already minimal, you can stop here and return it exactly as it is. Note that many questions, even if long, will be considered minimal if all parts are essential. ### **Revision** A question should be revised if it contains non-essential information that can be removed while preserving the question's disambiguity, entity identification, and answer specification. A valid revision consists **only of deleting** words or phrases. You must not add, rephrase , or reorder any part of the question. The original wording of the remaining parts must be preserved. | You can | perform | a | minimal | change to | fix | the grammar. | | | | | ------- | ------- | --- | ------- | --------- | --- | ------------ | --- | --- | --- | After deleting a word or phrase, you must confirm that the revised question still: | 1. | Makes sense | grammatically | | and | logically. | | | | | | ------ | ----------- | ------------- | ------------- | -------------- | ---------- | ------------ | ---- | ------- | --- | | 2. | Uniquely | and | unambiguously | points | to | the provided | gold | answer. | | | Common | details | to | remove | for minimalism | | include: | | | | * **Redundant Specifiers:** Details like a specific year or genre, *if* another, more | | unique | identifier | in | the question | already | | makes them | unnecessary. | | | --- | ------ | ---------- | --- | ------------ | ------- | --- | ---------- | ------------ | --- | * Example: "Which **1958 crime** drama by Orson Welles featured..." might be | | reducible | | if another | part | of the | question | is sufficiently | | unique. | | --- | --------- | --- | ---------- | ---- | ------ | -------- | --------------- | --- | ------- | * **Superfluous Descriptors:** Words that describe an entity but are not needed to | | distinguish | it | from other | entities. | | | | | | | --- | ----------- | --- | ---------- | --------- | --- | --- | --- | --- | --- | * Example: Removing "**country and gospel**" from "...quartet the Oak Ridge Boys," as | | the | band's | name | is a unique | identifier. | | | | | | --- | --- | ------ | ---- | ----------- | ----------- | --- | --- | --- | --- | * **Leaked Clues:** Information that provides a separate, related fact not essential to 47 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | identifying | | the | answer. | | | | | | ----------- | --- | --- | ------- | --- | --- | --- | --- | * Example: Removing "**defending champion**" from "...team contested the 1966 Stanley | | Cup | Finals," | as | it reveals | the | result of the previous | year's final. | | --- | --- | -------- | --- | ---------- | --- | ---------------------- | ------------- | **Important:** * Do not remove essential phrases that describe the core subject of the question (e.g., | "...based | | on the | life | of a Military | | Commander..."). | | | --------- | --- | ------ | ---- | ------------- | --- | --------------- | --- | * Do not shorten or remove official or commonly used names (e.g., keep "Ebola virus | disease") | | and | do not | remove | abbreviations. | | | | --------- | --- | --- | ------ | ------ | -------------- | --- | --- | * Keep in mind that MOST questions are already minimal. Your goal is not simply to shorten the question, but to remove any information that is unnecessary for | disambiguation | | | and acts | as an | unneeded | hint. | | | -------------- | --- | --- | -------- | ----- | -------- | ----- | --- | ---- | ### **Required | | Output | Format** | | | | | | -------------- | --- | ------ | -------- | --- | --- | --- | --- | You must return *only* a string, which can be the original question or the revised question . If the question is minimal as it is, return the question. If the question needs revision, revise it by deleting the non-essential parts and returning the revised question. Remember to return only the question string, without any additional text. ---- ### **Examples** 1. **Question:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? | **Gold | answer:** | Annatar | | | | | | | ------ | --------- | ------- | --- | --- | --- | --- | --- | **Explanation:** The question is already minimal. "in his fair disguise" and "to manipulate Celebrimbor" are both essential context to specify which of Sauron's names is being | asked | for. | Removing | either | phrase | would | make the question | ambiguous. | | ----- | ---- | -------- | ------ | ------ | ----- | ----------------- | ---------- | **Output:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? 2. **Question:** The film "Sarsenapati Hambirrao" is based on the life of a Military Commander | | of which | empire? | | | | | | | ------ | --------- | ------- | --- | ------ | --- | --- | --- | | **Gold | answer:** | Maratha | | Empire | | | | **Explanation:** The question is already minimal. The phrase "the life of a Military Commander" is the core subject of the question and clarifies which empire is relevant ( the one the commander served, the film might focus on more than a single empire). This | is | not | an unnecessary | | hint. | | | | | --- | --- | -------------- | --- | ----- | --- | --- | --- | **Output:** The film "Sarsenapati Hambirrao" is based on the life of a Military Commander | of | which | empire? | | | | | | | --- | ----- | ------- | --- | --- | --- | --- | --- | 3. **Question:** What vocal part does Richard Anthony Sterban sing in the country and gospel | quartet | | the Oak | Ridge | Boys? | | | | | ------- | --------- | ------- | ----- | ----- | --- | --- | --- | | **Gold | answer:** | Bass | | | | | | **Explanation:** The name of the quartet, "the Oak Ridge Boys," is a sufficient identifier. The genres "country and gospel" are superfluous descriptors that are not needed for | disambiguation. | | | They | can be | removed. | | | | --------------- | --- | --- | ---- | ------ | -------- | --- | --- | **Output:** What vocal part does Richard Anthony Sterban sing in the quartet the Oak Ridge Boys? 4. **Question:** Which defending champion team contested the 1966 Stanley Cup Finals against | the | Detroit | Red | Wings? | | | | | | ------ | --------- | -------- | ------ | --------- | --- | --- | --- | | **Gold | answer:** | Montreal | | Canadiens | | | | **Explanation:** The phrase "defending champion" is a leaked clue. It provides an additional fact (that the team won the 1965 finals) that is not necessary to uniquely identify the team that played the Red Wings in the 1966 finals. This non-essential | information | | should | be | removed. | | | | | ----------- | --- | ------ | --- | -------- | --- | --- | --- | **Output:** Which team contested the 1966 Stanley Cup Finals against the Detroit Red Wings? | **Instruction | | End** | | | | | | | ------------- | ------- | ----- | --------- | -------- | --- | ------------------ | --- | | **Please | process | the | following | question | | and gold answer:** | | 48 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | **Question:** | {question} | | | | | | | ------------- | ------------------------------------------- | ------------- | --- | --- | --- | --- | | **Gold | answer:** | {gold_answer} | | | | | | BoxE.10: | PromptforFilteringwithGoogleSearchGrounding | | | | | | You will be provided with a question and a corresponding answer written by an expert. Your task is to evaluate if the question is **suitable** based on the provided expert answer . A **suitable question** must be unambiguous, specific, and have a single, unique answer. Your evaluation should be based *solely* on the content of the expert's answer. | Your final | output | must | be only | one word: | **SUITABLE** | or **NOT**. | | ---------- | ------ | ---- | ------- | --------- | ------------ | ----------- | ---- | ### **Evaluation | | Process** | | | | | | ---------------- | --- | --------- | --- | --- | --- | --- | Carefully analyze the question and the expert's answer to determine the question's suitability. | **Definition | of | a Suitable | Question** | | | | | ------------ | --- | ---------- | ---------- | --- | --- | --- | A question is considered suitable if it meets all the following criteria: 1. **Unambiguous:** All entities in the question can be easily and uniquely identified. There is no need to ask for clarification to understand the exact intent of the question. 2. **Specific and precise:** The question asks for a specific and precise piece of information, making it easy to understand the expected type and format of the answer. 3. **Has a single and unique answer:** The question is structured in a way that there is | only | one possible | | correct | answer. | | | | ------------- | ------------ | ----------- | ------- | --------------- | -------- | --- | | **Determining | | Suitability | Based | on the Expert's | Answer** | | You must determine the question is **NOT** suitable if the expert's answer exhibits any of | the | following | characteristics: | | | | | | --- | --------- | ---------------- | --- | --- | --- | --- | * **The answer asks for clarification:** The expert's response includes questions back to | the | user | to resolve | ambiguity. | | | | | --- | ---- | ---------- | ---------- | --- | --- | --- | * *Example indicators:* "Which 'Scarlet' are you referring to?", "Could you please specify the year?", "It's not clear what you mean by 'official language'..." * **The answer states there are multiple possible answers:** The expert explicitly | mentions | that | the | question | has more than | one valid | answer. | | -------- | ---- | --- | -------- | ------------- | --------- | ------- | * *Example indicators:* "There are several former presidents of the USA...", " Multiple actors have played this role...", "Several cities match this description ..." * **The answer provides different answers for different interpretations:** The expert identifies ambiguity and provides separate answers for each possible interpretation of the question. * *Example indicators:* "If you are asking about the actress Scarlet Johansson, the answer is... However, if you mean the literary character Scarlett O'Hara, the answer is...", "For the 2008 movie, the director was... For the 2021 remake, the | | director | was..." | | | | | | --- | -------- | ------- | --- | --- | --- | --- | A question is considered **SUITABLE** only if the expert's answer provides a direct, singular, and factual response without expressing any confusion, highlighting ambiguity | , or | indicating | the | existence | of multiple | valid | answers. | | ---- | ---------- | --- | --------- | ----------- | ----- | -------- | ---- | ### **Required | | Output | Format** | | | | | -------------- | --- | ------ | -------- | --- | --- | --- | You must return *only* a string, either **SUITABLE** or **NOT**, without any additional | text | or explanation. | | | | | | | ------------- | --------------- | ----- | --- | --- | --- | --- | | **Instruction | | End** | | | | | 49 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | **Please | process | the | following | question | | and the expert's | answer:** | | ------------- | --------------------------------- | --- | --------- | -------- | --- | ---------------- | --------- | | **Question:** | {question} | | | | | | | | **Answer:** | {search_pred} | | | | | | | | BoxE.11: | PromptforGeneratingtheReverseTask | | | | | | | You will be provided with a context, a question, and an answer. The answer is an OBJECT entity that also completes the last sentence of the provided context. Your task is to generate a *reverse question*. In this new question, the SUBJECT entity of the original question will be the answer. The OBJECT entity from the original question | will | become | the new | subject | of | the reverse | question. | | | ---- | ------ | ------- | ------- | --- | ----------- | --------- | --- | The process involves three main stages: **Identification**, **Generation**, and ** Evaluation**. | ### **Stage | 1: | Identify | the | SUBJECT | Entity** | | | | ----------- | --- | -------- | --- | ------- | -------- | --- | --- | Analyze the original question to identify its SUBJECT entity. The SUBJECT is the main focus of the question, and the question asks for information related to this entity. For example, in the question "What is the capital of France?" the SUBJECT is "France". | ### **Stage | 2: | Generate | a Reverse | | Question** | | | | ----------- | --- | -------- | --------- | --- | ---------- | --- | --- | Create a new question where the original SUBJECT entity becomes the answer, and the | original | OBJECT | entity | | becomes | the new | subject. | | | -------- | ------ | ------ | --- | ------- | ------- | -------- | --- | The reverse question must be an unambiguous question with a single, unique, and specific answer: the SUBJECT. It must be based on the given question and act as its reverse. | The reverse | question | | should | follow | these | requirements: | | | ----------- | -------- | --- | ------ | ------ | ----- | ------------- | --- | * **Self-contained:** It does not use pronouns or references that require outside context . * **Unambiguous and Specific:** Its phrasing is clear, avoids multiple interpretations, and specifies the level of detail required in the answer (e.g., "What year...", "What | is the | name | of the | condition..."). | | | | | | ------ | ---- | ------ | --------------- | --- | --- | --- | --- | * **Answer Uniqueness:** It must be phrased to have a *single* valid and correct answer | which | is the | SUBJECT. | | | | | | | ----- | ------ | -------- | --- | --- | --- | --- | --- | * **Minimal:** It includes only the essential details from the context and the original question needed to make the reverse question unambiguous and ensure the SUBJECT is the | only | valid | answer. | | | | | | | ---- | ----- | ------- | --- | --- | --- | --- | --- | * **Verbatim:** It uses the same terms, phrases, and words that appear in the context and | the | given | question, | without | | rephrasing. | | | | --- | ----- | --------- | ------- | --- | ----------- | --- | --- | * **Challenging:** The reverse question should be challenging and non-trivial to answer. The difficulty level of the original question and the reverse question should be | roughly | the | same. | | | | | | | ------- | --- | ----- | --- | --- | --- | --- | --- | **IMPORTANT:** Although not needed in most cases, you may sometimes add information from the *context* to the reverse question, provided it is required to make the question unambiguous or to ensure the SUBJECT is the only valid answer. However, this information must be **minimal** and must **not shift the focus** of the question. It should not introduce a hint or clue that allows solving the reverse question without knowledge of the core relationship between the original OBJECT and SUBJECT. The added detail should not become a shortcut to the answer and should not make the question | easier | without | helping | | in disambiguation. | | | | | ----------- | ------- | --------- | ----------- | ------------------ | --- | ----------- | --- | | ### **Stage | 3: | Determine | Suitability | | and | Rejection** | | Finally, you must assess the reverse question you generated. If it fails to meet the | requirements | | above, | you | must | reject | it. | | | ------------ | --- | ------ | --- | ---- | ------ | --- | --- | A question should be rejected if it is impossible to create a reverse version that is | unambiguous | | and has | a single | | valid answer. | | | | ----------- | --- | ------- | -------- | --- | ------------- | --- | --- | It should also be rejected if any attempt to make it meet the requirements shifts the focus | of | the question | | or makes | it | trivial | or easier. | | | --- | ------------ | --- | -------- | --- | ------- | ---------- | --- | ---- 50 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | ### **Required | | Output | Format** | | | | | | -------------- | ---- | ------ | -------- | --------------- | --- | --- | --- | | Your output | must | be | in one | of two formats. | | | | If you generate a successful reverse question, the response must be structured using the following tags: `` and ``. Do not include any other text, | explanations, | | or | formatting. | | | | | | ------------- | --- | ------- | ----------- | ------------ | --- | --- | --- | | `` | The | reverse | question | `` | | | | `` The subject entity, which is the answer to the reverse question `` If a suitable reverse question cannot be generated, you must return only the string: `REJECT` ---- ### **Examples** Here are several examples demonstrating the task. Pay close attention to the explanations. 1. **Context:** Cody Blake Dennison (born March 4, 1990) is an American professional stock car racing driver and YouTube personality who competes part-time in the NASCAR Craftsman | Truck | Series, | driving | | the Nos. | 2/22 Ford | F-150 | | | ----- | ------- | ------- | --- | -------- | --------- | ----- | --- | **Question:** What is the model of the Nos. 2/22 truck that Cody Blake Dennison drives in | the | NASCAR | Craftsman | Truck | Series? | | | | | ----------- | ------ | --------- | ----- | ------- | --- | --- | --- | | **OBJECT:** | Ford | F-150 | | | | | | **Output:** Who drives the Nos. 2/22 Ford F-150 in the NASCAR Craftsman Truck Series? | | Cody | Blake | Dennison | | | | | | ---------------- | ---- | ----- | -------- | ---------- | --- | ---------- | ------------- | | **Explanation:** | | We | use the | same words | and | terms from | the question. | 2. **Context:** Summer Side of Life is Canadian singer-songwriter Gordon Lightfoot's sixth | studio | album. | It | was released | in | 1971 | | | | ------ | ------ | --- | ------------ | --- | ---- | --- | --- | **Question:** In what year was Gordon Lightfoot's album, Summer Side of Life, released? | **OBJECT:** | 1971 | | | | | | | | ----------- | ---- | --- | --- | --- | --- | --- | --- | **Output:** What is the name of Gordon Lightfoot's album that was released in 1971? | | Summer | Side | of Life | | | | | | --------- | ------ | ---- | ------- | ---------- | --- | --- | --- | **Explanation:** It is likely that the singer released only a single album in 1971, making | the | answer | single | and | unique. | | | | | --- | ------ | ------ | --- | ------- | --- | --- | --- | 3. **Context:** Diabolik: Who Are You? (Italian: Diabolik - Chi sei?) is a 2023 Italian crime action film directed by the Manetti Bros. and based on the 1968 Diabolik comic strip Diabolik, chi sei? by Angela and Luciana Giussani. It premiered at the 18th Rome Film Festival **Question:** At which event did the film 'Diabolik: Who Are You?' premiere? | **OBJECT:** | Rome | Film | Festival | | | | | | ----------- | ---- | ---- | -------- | --- | --- | --- | --- | **Output:** What film directed by the Manetti Bros. premiered at the 18th Rome Film Festival? | | Diabolik: | | Who Are | You? | | | | | --------- | --------- | --- | ------- | --------------- | --- | --- | --- | **Explanation:** A reverse question like, "What film premiered at the Rome Film Festival?" is ambiguous. To ensure a unique answer, we add two key details, forming the question: "What film directed by the Manetti Bros. premiered at the 18th Rome Film Festival?". We specify the "18th" edition because the festival is held annually and would otherwise have multiple answers over the years. We add "directed by the Manetti Bros." to narrow the options at that specific event, ensuring a single valid answer. In contrast, adding the detail "...based on the 1968 Diabolik comic strip Diabolik, chi sei?" would have been incorrect because it shifts the focus. The question would no longer be about the film's premiere but would become a simple trivia question about identifying a comic | book | adaptation. | | | | | | | | ---- | ----------- | --- | --- | --- | --- | --- | --- | 51 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality 4. **Context:** WWWQ (99.7 FM) is a commercial radio station carrying a contemporary hit radio | | format | known | as "Q99.7". | | Owned | by Cumulus | Media | | | ------------- | ------ | ------- | ----------- | ---- | ----- | ---------- | ----- | ------------- | | **Question:** | | What | company | owns | the | commercial | radio | station WWWQ? | | **OBJECT:** | | Cumulus | Media | | | | | | **Output:** REJECT **Explanation:** A reverse question like, "What commercial radio station is owned by Cumulus Media?" is not suitable because the company owns many stations. Adding a detail like "...that is known as 'Q99.7'?" or "...that operates on 99.7 FM?" shifts the focus . The question is no longer about ownership but about knowing the station's slogan or frequency. This turns the added detail into a shortcut. Since no additional context can | | be provided, | | this | item must | be | rejected. | | | | ------------- | ------------ | ----- | ---- | --------- | --- | --------- | --- | --- | | **Instruction | | End** | | | | | | | **Please process the following context, question, and its answer (OBJECT entity):** | **Context:** | | {context} | | | | | | | | ------------- | -------------------------------------------- | ---------- | --- | --- | --- | --- | --- | --- | | **Question:** | | {question} | | | | | | | | **OBJECT:** | | {answer} | | | | | | | | BoxE.12: | PromptforVerifyingaValidReverseandDirectPair | | | | | | | | You will be provided with a *Direct Question* (with its answer) and a *Reverse Question* ( | | with its | answer). | | | | | | | | --- | -------- | -------- | --- | --- | --- | --- | --- | --- | Your task is to verify whether this pair represents a *suitable factual evaluation pair*. A fact is defined as a proposition between two entities: a subject and an object. | * In | the *Direct | | Question*, | the | target | is | the **Object** | entity. | | ---- | ----------- | --- | ---------- | --- | ------ | --- | -------------- | ------- | * In the *Reverse Question*, the roles are swapped: the Object of the direct question becomes the Subject (or is explicitly mentioned) in the reverse question, and the | | target | is the | original | Subject | | entity. | | | | --- | ------- | ------------ | -------- | ------- | --- | ------- | --- | --- | | The | pair is | **suitable** | | if: | | | | | 1. The questions represent a valid direct/reverse relationship. The answer (Object) of the | | Direct | Question | is | included | in | the Reverse | Question. | | | --- | ------ | -------- | --- | -------- | --- | ----------- | --------- | --- | * *Note:* It is acceptable if the information conveys different hints to resolve ambiguity, or if the Object of the direct question is not the main entity, topic, or subject of the reverse question, as long as it is explicitly present. 2. The questions are challenging and require specific factual knowledge. | * | The | answers | are not | contained | | within | the questions | themselves. | | --- | --- | ------- | ------- | --------- | --- | ------ | ------------- | ----------- | * The answers cannot be easily guessed using general common sense or linguistic clues. | 3. | The answers | are | not | time-sensitive. | | | | | | --- | ----------- | --- | --- | --------------- | --- | --- | --- | --- | * If the fact is subject to change (e.g., revenue, statistics, roles), the question | | must | specify | a | date/year | to | make | the answer static. | | | --- | ------- | ------- | ---------- | --------- | --- | ---- | ------------------ | --- | | The | pair is | **not | suitable** | | if: | | | | 1. The answer to the Direct Question (the Object) is missing from the Reverse Question. 2. One of the answers can be easily predicted via an educated guess. 3. One of the questions asks for dynamic information (like statistics or current roles) | | without | specifying | | a timeframe, | | making | the answer | subject to decay. | | --- | ------- | ---------- | --- | ------------ | --- | ------ | ---------- | ----------------- | ---- | ### **NOT | | SUITABLE | Examples** | | | | | | | --------- | --- | -------- | ---------- | --- | --- | --- | --- | --- | **Direct Question:** What language's complete form was published in the book Toki Pona: The | | Language | of | Good? | | | | | | | --------- | ------------ | ----------- | ----- | ---- | --- | --- | --- | --- | | **Answer | (Object):** | | Toki | Pona | | | | | | **Reverse | | Question:** | ... | | | | | | | **Answer | (Subject):** | | ... | | | | | | **Explanation:** The answer "Toki Pona" is explicitly contained within the book title | | mentioned | in | the question. | | Therefore, | | it is trivial. | | | --- | --------- | --- | ------------- | --- | ---------- | --- | -------------- | --- | 52 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | **Direct | Question:** | ... | | | | | | -------- | ----------- | --- | --- | --- | --- | --- | | **Answer | (Object):** | ... | | | | | **Reverse Question:** What is the name of the annual celebration for the date the National | flag | of Canada | officially | | appeared? | | | | -------- | ------------ | ---------- | --- | --------- | ------------- | --- | | **Answer | (Subject):** | National | | Flag | of Canada Day | | **Explanation:** The answer is trivial because it simply repeats the key terms ("National | flag | of Canada") | found | in | the question. | | | | ---- | ----------- | ----- | --- | ------------- | --- | --- | **Direct Question:** What is the median household income in the United States according to | the Census | | Bureau? | | | | | | ---------- | ------------ | ------- | --- | --- | --- | --- | | **Answer | (Object):** | $67,521 | | | | | | **Reverse | Question:** | ... | | | | | | **Answer | (Subject):** | ... | | | | | **Explanation:** The answer is time-sensitive. Without a specific year (e.g., "in 2020"), | this | value | changes over | time | and | is not static. | | | ---- | ----- | ------------ | ---- | --- | -------------- | --- | **Direct Question:** What was the release date of Ice Age: Dawn of the Dinosaurs? | **Answer | (Object):** | July | 1, 2009 | | | | | -------- | ----------- | ---- | ------- | --- | --- | --- | **Reverse Question:** What film is the third installment in the Ice Age film series? | **Answer | (Subject):** | Ice | Age: | Dawn | of the Dinosaurs | | | -------- | ------------ | --- | ---- | ---- | ---------------- | --- | **Explanation:** This fails the structural criteria. The answer to the Direct Question (" | July | 1, 2009") | is not | included | in | the Reverse | Question. | | ---- | --------- | ------ | -------- | --- | ----------- | --------- | ---- | ### **SUITABLE | | Examples** | | | | | | -------------- | ------------ | ---------- | --- | ------- | ------------- | --------- | | **Direct | Question:** | ... | | | | | | **Answer | (Object):** | ... | | | | | | **Reverse | Question:** | What | DVI | variant | is compatible | with VGA? | | **Answer | (Subject):** | DVI-I | | | | | **Explanation:** Although the answer shares the term "DVI" with the question, it requires specific technical knowledge to identify the exact variant ("I") compatible with VGA. | It is | non-trivial. | | | | | | | ----- | ------------ | --- | --- | --- | --- | --- | **Direct Question:** What was the GMV achieved by the Redbus India Private Limited company in 2018? | **Answer | (Object):** | 50 | billion | Indian | Rupee | | | --------- | ------------ | --- | ------- | ------ | ----- | --- | | **Reverse | Question:** | ... | | | | | | **Answer | (Subject):** | ... | | | | | **Explanation:** The answer is not time-sensitive because the context restricts the fact to | a specific | | year (2018), | making | | the value | static. | | ---------- | --- | ------------ | ------ | --- | --------- | ------- | **Direct Question:** With what batting average did Carl Furillo win the batting title? | **Answer | (Object):** | .344 | | | | | | -------- | ----------- | ---- | --- | --- | --- | --- | **Reverse Question:** Who won the 1953 batting title with a .344 average? | **Answer | (Subject):** | Carl | Furillo | | | | | -------- | ------------ | ---- | ------- | --- | --- | --- | **Explanation:** This is suitable. Even though the Direct Answer (.344) is not the main subject of the Reverse Question, it is included in the Reverse Question to resolve ambiguity. ---- | ### **Required | | Output Format** | | | | | | -------------- | ----- | --------------- | --- | ----- | --- | --- | | Respond only | with | "SUITABLE" | or | "NO". | | | | **Instruction | End** | | | | | | **Please process the following direct and reverse questions and their answers:** | **Direct | Question:** | {direct} | | | | | | -------- | ----------- | --------------- | --- | --- | --- | --- | | **Answer | (Object):** | {object_entity} | | | | | 53 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | **Reverse Question:** | | {reverse} | | | | | | ----------------------------------------------- | --- | ---------------- | --- | --- | --- | --- | | **Answer (Subject):** | | {subject_entity} | | | | | | BoxE.13: PromptforGeneratingtheNaturalQuestions | | | | | | | You will be provided with a question and its answer. Your task is to rephrase the question. The question provided to you is an unambiguous question that has a single, unique, and | specific | answer. | It is: | | | | | | -------- | ------- | ------ | --- | --- | --- | --- | * **Self-contained:** It does not use pronouns or references that require outside context . * **Unambiguous and Specific:** Its phrasing is clear, avoids multiple interpretations, and specifies the level of detail required in the answer (e.g., "What year...", "What | is the name | of the | condition..."). | | | | | | ----------- | ------ | --------------- | --- | --- | --- | --- | * **Minimal:** It includes only the essential details needed to make the question | unambiguous | and | ensure | there is | a single | possible | answer. | | ------------- | ---------- | ------ | -------- | -------- | -------- | ------- | | ### **Natural | Question** | | | | | | Rephrase the provided question to sound natural and conversational, as if a person were | genuinely | asking. | | | | | | | --------- | ------- | --- | --- | --- | --- | --- | **Requirements:** * The natural question must include all the critical information from the provided | question, | but rewritten | | in a natural | | way. | | | --------- | ------------- | --- | ------------ | --- | ---- | --- | * **Crucially, the natural question must remain unambiguous and lead to the exact same | specific | answer | as the | provided | question.** | | | | -------- | ------ | ------ | -------- | ----------- | --- | --- | * To ensure the answer is specific, you might need to add clarifying phrases. For example , if the answer is a date, a vague question like "When did X happen?" must be rephrased as **"When did X happen? Answer with a date."** If the answer is a year, it would be | **"When | did X happen? | Answer | with | a year."** | | | | ------- | ------------- | ------ | ---- | ---------- | --- | --- | * It is acceptable to use shorter, more common terms (e.g., Ebola instead of Ebola virus disease) or common abbreviations if they are widely understood (e.g., USA, NFL, LLM). * While you should generally not remove information, you **can** discard minor words or clauses if they are not necessary for disambiguation and a plausible person would consider the canonical and casual questions to be asking for the exact same thing. This should be done sparingly, only when it significantly improves the naturalness of the question. ---- | ### **Required | Output | Format** | | | | | | --------------- | ------ | -------- | ---- | --- | ----------------- | --- | | You must return | *only* | a string | with | the | natural question. | | ---- ### **Examples** 1. * **Question:** What was the condition that caused Erika Nordby to spend two hours | without | a heartbeat | before | she | was revived? | | | | ------------- | ----------- | ------ | --- | ------------ | --- | --- | | * **Answer:** | hypothermia | | | | | | * **Output:** What condition caused Erika Nordby to have no heartbeat for two hours? *Explanation:* The phrase "before she was revived" is omitted as it is implied by the fact that she had "no heartbeat for two hours" and the context is about her survival. A | casual speaker | would | likely | not | add this | detail. | | | -------------- | ----- | ------ | --- | -------- | ------- | --- | 2. * **Question:** What was the release date of When I Look in Your Eyes by Diana Krall? | * **Answer:** | June | 8, 1999 | | | | | | ------------- | ---- | ------- | --- | --- | --- | --- | * **Output:** When did Diana Krall's album "When I Look in Your Eyes" come out? Answer with a date. 54 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality 3. * **Question:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? * **Answer:** Annatar * **Output:** What name did Sauron use as part of his disguise to trick Celebrimbor? *Explanation:* The term "manipulated" is rephrased to the more common "trick." The sentence structure is changed to a more natural flow, but the key disambiguating elements " Sauron," "Celebrimbor," and "disguise" are all retained. 4. * **Question:** What is the name the press often uses to refer to the rule forbidding the horse-collar tackle from the NFL? * **Answer:** The Roy Williams Rule * **Output:** What is the NFL rule that bans horse-collar tackles called? *Explanation:* This is an example of carefully removing a clause. The phrase "the press often uses to refer to" is simplified to "what is the common name for". A natural speaker asking this question would assume the common name for the rule is the the name the press call the rule, making both questions the same. **Instruction End** **Please process the following question and answer:** **Question:** {question} **Answer:** {answer} BoxE.14: PromptforGeneratingtheContextualQuestions You will be provided with a context, a question, and an answer. The answer also completes the last sentence of the provided context. Your task is to augment the question with the context. The question provided to you is an unambiguous question that has a single, unique, and specific answer. It is: * **Self-contained:** It does not use pronouns or references that require outside context . * **Unambiguous and Specific:** Its phrasing is clear, avoids multiple interpretations, and specifies the level of detail required in the answer (e.g., "What year...", "What is the name of the condition..."). * **Minimal:** It includes only the essential details needed to make the question unambiguous and ensure there is a single possible answer. ### **Contextual Question** Create a question that tests comprehension within the given context. This is achieved by combining the context with the question, making minor edits to enhance fluency and minimize redundancy. The typical process for generating this question is: 1. Take the entire context, excluding the final incomplete sentence that the answer completes. 2. Append the provided question to the end. 3. Minor rephrasing of the context or question may be needed to improve the flow and remove redundancy. ---- ### **Required Output Format** You must return *only* a string with the contextual question. ---- 55 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality ### **Examples** 1. * **Context:** Erika Nordby (born February 2000), also known as Baby Erika, Miracle Baby and Canada's Miracle Child, is a Canadian originally from Edmonton, Alberta known for having been revived after spending two hours without a heartbeat due to * **Question:** What was the condition that caused Erika Nordby to spend two hours | without | a heartbeat | before | she was revived? | | | | | ------------- | ----------- | ------ | ---------------- | --- | --- | --- | | * **Answer:** | hypothermia | | | | | | * **Output:** Erika Nordby (born February 2000), also known as Baby Erika, Miracle Baby and Canada's Miracle Child, is a Canadian originally from Edmonton, Alberta. What was the condition that caused Erika Nordby to spend two hours without a heartbeat before | she was | revived? | | | | | | | ------- | -------- | --- | --- | --- | --- | --- | 2. * **Context:** When I Look in Your Eyes is the fifth studio album by singer Diana Krall, | released | on | | | | | | | -------- | --- | --- | --- | --- | --- | --- | * **Question:** What was the release date of When I Look in Your Eyes by Diana Krall? | * **Answer:** | June | 8, 1999 | | | | | | ------------- | ---- | ------- | --- | --- | --- | --- | * **Output:** When I Look in Your Eyes is the fifth studio album by singer Diana Krall. | What was | the release | date | of When I | Look in Your | Eyes? | | | -------- | ----------- | ---- | --------- | ------------ | ----- | --- | *Explanation:* We removed the singer's name from the question to improve fluency, as it already appears in the preceding context and its repetition would be redundant. 3. * **Context:** Celebrimbor is a fictional character in J. R. R. Tolkien's legendarium. In Tolkien's stories, Celebrimbor was an elven-smith who was manipulated into forging the | Rings of | Power | by the Dark | Lord Sauron, | in a fair | disguise | and named | | -------- | ----- | ----------- | ------------ | --------- | -------- | --------- | * **Question:** What name did Sauron use in his fair disguise to manipulate Celebrimbor? | * **Answer:** | Annatar | | | | | | | ------------- | ------- | --- | --- | --- | --- | --- | * **Output:** Celebrimbor is a fictional character in J. R. R. Tolkien's legendarium. In Tolkien's stories, Celebrimbor was an elven-smith who was manipulated into forging the Rings of Power by the Dark Lord Sauron. What name did Sauron use in his fair disguise | to manipulate | Celebrimbor? | | | | | | | ------------- | ------------ | --- | --- | --- | --- | --- | 4. * **Context:** The horse-collar tackle is a gridiron football maneuver in which a defender tackles another player by grabbing the back collar or the back-inside of an opponent's shoulder pads and pulling the ball carrier directly downward violently in order to pull his feet from underneath him. The technique is most closely associated with Pro Bowl safety Roy Williams. After being blamed for a series of major injuries in the 2004 season , the horse-collar tackle was banned from the NFL during the 2005 off- | season. | The rule | forbidding | it is often | referred | to in the | press as | | ------- | -------- | ---------- | ----------- | -------- | --------- | -------- | * **Question:** What is the name the press often uses to refer to the rule forbidding the | horse-collar | tackle | from | the NFL? | | | | | ------------- | ------- | -------- | -------- | --- | --- | --- | | * **Answer:** | The Roy | Williams | Rule | | | | * **Output:** The horse-collar tackle is a gridiron football maneuver in which a defender tackles another player by grabbing the back collar or the back-inside of an opponent's shoulder pads and pulling the ball carrier directly downward violently. The technique is most closely associated with Pro Bowl safety Roy Williams and is blamed for a series of major injuries in the 2004 season, leading to it being banned. What is the name the press often uses to refer to the rule forbidding the horse-collar tackle from the NFL? | **Instruction | End** | | | | | | | ---------------- | ---------- | --------- | -------- | ------------ | --------- | --- | | **Please process | the | following | context, | question and | answer:** | | | **Context:** | {context} | | | | | | | **Question:** | {question} | | | | | | | **Answer:** | {answer} | | | | | | 56 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality BoxE.15: PromptforGeneratingOptionsforMultiple-ChoiceQuestions You will be provided with a context, a question, and its gold answer. Your task is to generate three additional plausible but incorrect answers to create a complete multiple | -choice | question. | | | | | | | | | ------- | --------- | --- | --- | --- | --- | --- | --- | --- | The primary goal is to create challenging distractors that are confusing but definitely incorrect. The provided question should be used as is, and the provided gold answer must be the first | option | in your | output. | | | | | | | | ------ | ------- | ------- | --- | --- | --- | --- | --- | --- | ---- ### **Requirements** Your task is to generate three incorrect but confusing answer options. These distractors should make the question challenging. Consider the following strategies when creating them: * **Plausibility is key:** The incorrect answers should seem like they *could* be correct . They must share the same type, category, and granularity as the gold answer (e.g., if the answer is a person's name, all options should be names of people, if the answer is | a city, | all | options | should | be cities). | | | | | | ------- | --- | ------- | ------ | ----------- | --- | --- | --- | --- | * **Must be incorrect:** There must be no ambiguity. Only the provided gold answer can be the correct answer to the question. The distractors must be factually wrong in the | given | context. | | | | | | | | | ----- | -------- | --- | --- | --- | --- | --- | --- | --- | * **Use contextual hints:** The distractors can be based on other entities or details mentioned in the provided context. For example, if the context is a film's summary and the question asks for the director, a good distractor would be the name of the producer | or | a lead actor | mentioned | | in the | text. | | | | | --- | ------------ | --------- | --- | ------ | ----- | --- | --- | --- | * **Use similarity:** The distractors can be based on entities with similar names, or concepts that are thematically related. For example, if the answer is "The Battle of Hastings," a distractor could be "The Battle of Stamford Bridge," which occurred in the | same | year. | | | | | | | | | ---- | ----- | --- | --- | --- | --- | --- | --- | --- | * **Use popular associations:** The distractors can be other very popular options that might come to mind for the general topic, but are incorrect for the specific question asked. ---- | ### **Required | Output | Format** | | | | | | | | -------------- | ----------- | -------- | ------ | -------- | -------- | ------ | -------- | --------- | | You must | return only | the | four | answer | options, | each | on a | new line. | | * The | first line | must | be the | original | gold | answer | provided | to you. | * The following three lines must be the new incorrect but confusing answer options. * Do not include any numbering or lettering (e.g., A., B., C., D.). Remember to return only the list of strings (each on a new line), without any additional text. | **Instruction | End** | | | | | | | | | ---------------- | --------------------------------- | --------- | ---------- | -------- | --------- | ------------- | -------- | --------- | | **Please | process the | following | | context, | question, | | and gold | answer:** | | **Context:** | {context} | | | | | | | | | **Question:** | {question} | | | | | | | | | **Gold answer:** | {gold_answer} | | | | | | | | | BoxE.16: | PromptforGeneratingtheReverseTask | | | | | | | | | You will | be provided | with | a question | | and its | corresponding | | answer. | Your task is to categorize the answer by assigning the most appropriate entity type from | the | list below: | | | | | | | | | ------------- | ----------- | ---------- | --- | ------ | ------- | ---- | ------------- | --- | | * **PERSON:** | An | individual | | human, | whether | real | or fictional. | | * **ORGANIZATION:** A structured company, institution, or collective with a common 57 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality purpose. * **LOCATION:** A specific geographical place, region, or feature on Earth. * **EVENT:** A notable occurrence or incident that happens at a particular time and place . * **WORK_OF_ART:** A specific creative work, such as a book, movie, song, or painting. * **PRODUCT:** A commercially produced good, food, service, or piece of technology. * **ORGANISM:** A non-human biological life form, including animals, plants, and microbes . * **SUBSTANCE:** A material with a distinct physical or chemical composition. * **DATE:** A specific point in time, including full dates, years, or historical periods. * **NUMERIC_VALUE:** A number representing a specific quantity, measurement, or other | quantifiable | value. | | | | ------------ | ------ | --- | --- | * **CONDITION:** A specific state of being, typically a medical, physical, or operational status. * **TITLE:** An official position, role, award, or honorific held by a person or group. * **LANGUAGE:** Natural languages, dialects, writing systems, including phrases. * **GROUP:** A collective of people defined by a shared nationality, ethnicity, or | cultural | affiliation. | | | | -------- | ------------ | --- | --- | * **LAW:** A specific, formally enacted rule, regulation, or statute created by a | governing | body. | | | | --------- | ----- | --- | --- | * **IDEA:** An abstract concept, belief system, named phenomenon, or cultural practice. * **CLASSIFICATION:** A specific type, genre, format, class or list used to categorize | other entities. | | | | | --------------- | --- | --- | --- | * **IDENTIFIER:** A unique name, code, or alias used as a formal reference or label for | another | entity | | | | ------- | ------ | --- | --- | * **OTHER:** A catch-all category for named entities that do not fit into any of the | other defined | types. | | | | ------------- | ------ | --- | --- | ---- | ### **Required | Output Format** | | | | -------------- | --------------- | --- | --- | You must return only a string containing a single entity type (e.g., "GROUP", "DATE"). Do | not include | any additional | text or punctuation. | | | -------------------- | -------------- | -------------------- | --------- | | **Instruction | End** | | | | Please categorize | the answer | to the following | question: | | **Question:** | {question} | | | | **Answer:** {answer} | | | | E.3. ResponseInstructionsandGraderPrompts 58 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality | BoxE.17: | TaskInstructions | | | | | | | | -------- | ---------------- | --- | --- | --- | --- | --- | --- | Completions: | Respond | only | with the | completion | | of the | last sentence | | | ------- | ---- | -------- | ---------- | --- | ------ | ------------- | --- | CompletionsCoT: Think step by step. Complete the last sentence of the text. End your response with Answer: | [Your | Completion] | | | | | | | | ----- | ----------- | --- | --- | --- | --- | --- | --- | Questions: | Respond | only | with the | answer | to the | question | | | | ------- | ---- | -------- | ------ | ------ | -------- | --- | --- | QuestionsCoT: | Think step | by | step. | End your | response | with | Answer: | [Your Answer] | | ---------- | --- | ----- | -------- | -------- | ---- | ------- | ------------- | Multiple-Choice: | Respond | only | with the | letter | of the | correct | answer | | | ------- | ---- | -------- | ------ | ------ | ------- | ------ | --- | Multiple-ChoiceCoT: | Think step | by | step. | End your | response | with: | Answer: | [Letter] | | ---------- | ------------------------------------- | ----- | -------- | -------- | ----- | ------- | -------- | | BoxE.18: | AutoraterPromptforTextCompletionTasks | | | | | | | >>> SYSTEM: You will be provided with an incomplete text (referred to as the `left context`), a gold entity that represents the expected continuation or completion (`gold completion`), and several predicted completions generated by a model. Your task is to evaluate each predicted completion in relation to the gold completion and the left context, and | assign | one | of the | following | | labels: | | | | ----------- | ------------ | ------ | ------------ | --- | --------- | --- | --- | | ["CORRECT", | "PARTIALLY", | | "INCORRECT", | | "OTHER"]. | | | First, I will give examples of each grade, and then you will grade a new example. ---- | The following | | are examples | | of CORRECT | predicted | | completions. | | ------------- | --- | ------------ | --- | ---------- | --------- | --- | ------------ | ``` Left context: The song "Walk This Way" was peaked at number 10 on the | Gold completion: | | Billboard | | Hot 100 | | | | | ---------------- | --- | ----------- | --------- | ----------- | --------- | ----- | ---------------- | | Completion | 1: | on the | Billboard | Hot | 100 | | | | Completion | 2: | Billboard | hot | 100 in | early | 1976. | | | Completion | 3: | prestigious | | U.S. chart, | Billboard | | Hot 100 in 1977. | ``` ``` Left context: Frequently portions of the term would remain only upon a U.S. senator's death | or | resignation. | | From | 1914 and | the enforcement | | of the | | --- | ------------ | --- | ---- | -------- | --------------- | --- | ------ | Gold completion: Seventeenth Amendment to the United States Constitution Completion 1: Seventeenth Amendment to the United States Constitution, senators have been | popularly | | elected. | | | | | | | ---------- | --- | --------------- | ---- | --------- | --- | --- | --- | | Completion | 2: | 17th Amendment, | | U.S. | | | | | Completion | 3: | Seventeenth | | Amendment | | | | | Completion | 4: | Amendment | XVII | | | | | ``` ``` Left context: Nova Scotia lies in the mid-temperate zone, and although the province is | almost | surrounded | | by water, | | the climate | is | closer to | | ---------------- | ---------- | ----------- | --------- | ------- | ----------- | --- | --------- | | Gold completion: | | continental | | climate | | | | | Completion | 1: | continental | | than to | maritime. | | | Completion 2: continental than to maritime because the prevailing winds are from the continent. ``` 59 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality These predicted completions are all CORRECT because: - The predicted completion contains the important information of the gold completion. - The predicted completion does not contain any information that contradicts the gold completion or the left context. - The gold completion may appear anywhere within the predicted completion as long as it clearly plays the same semantic role as the focus of the completion. - Minor omissions to the gold completion are allowed if the missing information is either easily inferable from the left context or not critical to identifying the gold completion (e.g., dropping a redundant modifier or title). - Only the semantic meaning matters: The completion may contain a variant, abbreviation , or alternative name for the gold completion. Differences in capitalization, punctuation, spelling, or minor grammatical variation (e.g., articles, pluralization) are allowed. - Completions may contain small factual inaccuracies or hallucinated details, as long as they do not contradict the left context. - Hedging and guessing are permissible, provided that the gold completion is included and the response does not contradict it or the left context. ---- The following are examples of INCORRECT predicted completions. ``` Left context: The Grand Caravan was reported missing, and a search was made, ending with the wreckage being found the next day, with all Gold completion: 10 Completion 1: nine occupants of the aircraft dead. Completion 2: five people on board deceased. Completion 3: 11 ``` ``` Left context: Andrew Paul Gosden (born 10 July 1993) disappeared from Central London on Gold completion: 14 September 2007 Completion 1: 19 March 2007, aged 14. Completion 2: 2007-09-15 ``` ``` Left context: The Toplica Uprising was a mass uprising by Serbian rebels against the Bulgarian occupation forces that took place in Bulgarian-occupied Serbia during the First World War. The revolt was supported by Serbian guerrilla fighters known as Gold completion: Chetniks Completion 1: Komitadjis Completion 2: Komitadjis who were the enemies of the Chetnik bands ``` These predicted completions are all INCORRECT because: - A factual statement in the predicted completion contradicts the left context or the gold completion. - Incorrect statements that have some hedging (e.g., "it is possible that", "although i 'm not sure, i think") are also considered incorrect. ---- The following are examples of PARTIALLY predicted completions. ``` Left context: IgA vasculitis (HSP), previously known as Henoch-Schonlein purpura, is an autoimmune disease that most commonly affects children. With kidney involvement, there may be a loss of small amounts of blood and protein in the urine (hematuria and proteinuria), but this usually goes unnoticed; in a small proportion of cases, the kidney involvement proceeds to Gold completion: chronic kidney disease Completion 1: end-stage kidney disease Completion 2: kidney disease Completion 3: chronic kidney disease and kidney failure - This is CORRECT. ``` 60 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality ``` Left context: Dopamine dysregulation syndrome (DDS) is a dysfunction of the reward system observed in some individuals taking dopaminergic medications for an extended length of time. It typically occurs in people with Parkinson's disease (PD) or restless legs syndrome (RLS) who have taken Gold completion: dopamine agonist medications Completion 1: dopaminergic medication Completion 2: dopaminergic medication for more than six months. ``` ``` Left context: Franki Raffles (17 October 1955 - 6 December 1994) was an English feminist and a Gold completion: documentary photographer Completion 1: photographer Completion 2: photographer and artist. Completion 3: professional artist and photographer. ``` These predicted answers are all PARTIALLY because: - The predicted completion contains a more general or less specific version of the gold completion (e.g., "dopamine agonist medications" is a specific type of " dopaminergic medication"). - The predicted completion identifies the core entity and the primary meaning of the gold completion, but omits a descriptive detail or modifier (e.g., "photographer" instead of "documentary photographer"). - The predicted completion is closely related to the gold completion and represents a common or expected outcome or progression, even if not strictly identical in every case (e.g., "end-stage kidney disease" as a severe form often associated with " chronic kidney disease"). ---- The following are examples of OTHER predicted completions. ``` Left context: Dirk Koperlager van Erp (1862-1933) was a Dutch American artisan, Gold completion: coppersmith Completion 1: noted for his distinctive work. Completion 2: designer, and teacher. Completion 3: I don't know. Completion 4: I need more context about Dirk Koperlager. Completion 5: Without researching the web, I cannot answer this. However, I can tell you that Dirk Koperlager van Erp was an artisan. Completion 6: who became known for his innovative metalwork, particularly his use of silver . ``` ``` Left context: T33 and CP3 are disability sport classification for disability athletics. Athletes in this class have moderate Gold completion: quadriplegia Completion 1: to severe hypertonia, ataxia, or athetosis Completion 2: athetosis, ataxia, or dystonia. Completion 3: or good functional strength and range of motion in their trunk ``` These predicted completions are all OTHER because: - The important information in the gold completion is not included in the predicted completion. - No statements in the predicted completion contradict the gold completion or the left context. ---- Also note the following things: - For completions where the gold completion is a number, the predicted completion needs to 61 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality be correct to the last significant figure in the gold completion. For example, consider the left context "The number of citations the Transformer paper has is?" with gold completion "120k". - Predictions "120k", "124k", and 115k" are all CORRECT. - Predictions "100k" and "113k" are INCORRECT. - Predictions "around 100k" and "more than 50k" are considered OTHER because they neither confirm nor contradict the gold completion. - Do not punish predicted completions if they omit information that would be clearly inferred from the left context. For example, consider the left context: "The paper A pretrainer's guide to training data: Measuring the effects of data age, domain coverage , quality, & toxicity" won the NAACL '24", and the gold completion "Outstanding Paper Award". The predicted completion "Outstanding Paper" would be considered CORRECT, because "award" is presumed in the left context. - Do not punish typos in people's names if it's clearly the same name. - For example, if the gold completion is "Hyung Won Chung", you can consider the following predicted completions as correct: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung". ---- **Required Output Format** Your entire response must be structured using the following tags: the grade of Completion {i} >>> USER: Here is a new example. Use the output format above to grade each completion with either " CORRECT", "INCORRECT", "PARTIALLY", or "OTHER". Don't apologize or correct yourself if there was a mistake; we are just trying to grade the completions. ``` Left context: {left_context} Gold completion: {gold_target} {completions} BoxE.19: AutoraterPromptforClosed-bookQATasks >>> SYSTEM: You will be provided with a question, a gold target, and several predicted answers generated by a model. Your task is to evaluate each predicted answer in relation to the gold target, and assign one of the following labels: ["CORRECT", "PARTIALLY", "INCORRECT", "OTHER"]. First, I will give examples of each grade, and then you will grade a new example. ---- The following are examples of CORRECT predicted answers. ``` Question: What chart did Aerosmith's "Walk This Way" hit number 10 on? Gold target: Billboard Hot 100 Answer 1: On the Billboard Hot 100 Answer 2: Billboard hot 100 in early 1976. Answer 3: The prestigious U.S. chart, Billboard Hot 100 in 1977. ``` ``` Question: What was enforced from 1914 that affected the election of U.S. senators? Gold target: Seventeenth Amendment to the United States Constitution Answer 1: Seventeenth Amendment to the United States Constitution, senators have been popularly elected. Answer 2: 17th Amendment, U.S. Answer 3: Seventeenth Amendment Answer 4: Amendment XVII ``` 62 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality ``` Question: Nova Scotia lies in the mid-temperate zone. What type of climate is Nova Scotia's climate closer to, despite the province being almost surrounded by water? | Gold | target: | continental | | climate | | | | | | ------ | ------- | ----------- | --- | ------- | --------- | --- | --- | --- | | Answer | 1: | Continental | | than to | maritime. | | | | Answer 2: Continental than to maritime because the prevailing winds are from the continent. ``` | These | predicted | | answers | are | all CORRECT | because: | | | | ----- | --------- | --- | ------- | --- | ----------- | -------- | --- | --- | - The predicted answer contains the important information of the gold target. - The predicted answer does not contain any information that contradicts the gold | | target | or | the | question. | | | | | | --- | ------ | --- | --- | --------- | --- | --- | --- | --- | - The gold target may appear anywhere within the predicted answer as long as it clearly | | plays | the | same | semantic | role | as | intended by | the question. | | --- | ----- | --- | ---- | -------- | ---- | --- | ----------- | ------------- | - Minor omissions to the gold target in the predicted answer are allowed if the missing information is either easily inferable from the question or not critical to identifying the gold target (e.g., dropping a redundant modifier or title). - Only the semantic meaning matters: The predicted answer may contain a variant, abbreviation, or alternative name for the gold target. Differences in capitalization, punctuation, spelling, or minor grammatical variation (e.g., | | articles, | | pluralization) | | are | allowed. | | | | --- | --------- | --- | -------------- | --- | --- | -------- | --- | --- | - Predicted answers may contain small factual inaccuracies or hallucinated details, as | | long | as | they do | not | contradict | the | question. | | | --- | ---- | --- | ------- | --- | ---------- | --- | --------- | --- | - Hedging and guessing are permissible, provided that the gold target is included and | | the | predicted | | answer | does | not contradict | it or | the question. | | --- | --- | --------- | --- | ------ | ---- | -------------- | ----- | ------------- | ---- | The | following | are | examples | of | INCORRECT | predicted | answers. | | | --- | --------- | --- | -------- | --- | --------- | --------- | -------- | --- | ``` Question: How many people were on Bering Air Flight 445 when it crashed on February 6, 2025? | Gold | target: | 10 | | | | | | | | ------ | ------- | -------------- | --- | -------- | ------------ | --- | ----- | --- | | Answer | 1: | Nine occupants | | of | the aircraft | | dead. | | | Answer | 2: | Five people | | on board | deceased. | | | | | Answer | 3: | 11 | | | | | | | ``` ``` Question: Andrew Paul Gosden (born 10 July 1993) disappeared from Central London. On what | | date did | this | disappearance | | occur? | | | | | ------ | -------- | ---------- | ------------- | ---- | ------ | --- | --- | --- | | Gold | target: | 14 | September | 2007 | | | | | | Answer | 1: | 19 March | 2007, | aged | 14. | | | | | Answer | 2: | 2007-09-15 | | | | | | | ``` ``` Question: What was the name of the Serbian guerrilla fighters who supported the Toplica Uprising? | Gold | target: | Chetniks | | | | | | | | ------ | ------- | ---------- | --- | ---- | --- | ------- | -------------- | ----- | | Answer | 1: | Komitadjis | | | | | | | | Answer | 2: | Komitadjis | who | were | the | enemies | of the Chetnik | bands | ``` | These | predicted | | answers | are | all INCORRECT | | because: | | | ----- | --------- | --- | ------- | --- | ------------- | --- | -------- | --- | - A factual statement in the predicted answer contradicts the question or the gold target. - Incorrect answers that have some hedging (e.g., "it is possible that", "although i'm | | not | sure, | i think") | | are also | considered | incorrect. | | | --- | --- | ----- | --------- | --- | -------- | ---------- | ---------- | --- | ---- | The | following | are | examples | of | PARTIALLY | predicted | answers. | | | --- | --------- | --- | -------- | --- | --------- | --------- | -------- | --- | ``` Question: What is a rare but serious kidney complication of IgA vasculitis? 63 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality Gold target: chronic kidney disease Answer 1: End-stage kidney disease Answer 2: Kidney disease Answer 3: Chronic kidney disease and kidney failure - This is CORRECT. ``` ``` Question: What drugs can lead to Dopamine dysregulation syndrome in patients with Parkinson's or restless legs syndrome? Gold target: dopamine agonist medications Answer 1: Dopaminergic medication Answer 2: Dopaminergic medication for more than six months. ``` ``` Question: What was Franki Raffles' profession? Gold target: documentary photographer Answer 1: Photographer Answer 2: Photographer and artist. Answer 3: Professional artist and photographer. ``` These predicted answers are all PARTIALLY because: - The predicted answer contains a more general or less specific version of the gold target (e.g., "dopamine agonist medications" is a specific type of "dopaminergic medication"). - The predicted answer identifies the core entity and the primary meaning of the gold target, but omits a descriptive detail or modifier (e.g., "photographer" instead of "documentary photographer"). - The predicted answer is closely related to the gold target and represents a common or expected outcome or progression, even if not strictly identical in every case (e.g ., "end-stage kidney disease" as a severe form often associated with "chronic kidney disease"). ---- The following are examples of OTHER predicted completions. ``` Question: Dirk Koperlager van Erp (1862-1933) was a Dutch American artisan. What type of artisan was he? Gold completion: coppersmith Answer 1: Noted for his distinctive work. Answer 2: Designer, and teacher. Answer 3: I don't know. Answer 4: I need more context about Dirk Koperlager. Answer 5: Without researching the web, I cannot answer this. However, I can tell you that Dirk Koperlager van Erp was an artisan. Answer 6: He became known for his innovative metalwork, particularly his use of silver. ``` ``` Question: What condition is described as moderate in athletes in the T33 and CP3 disability sport classification? Gold completion: quadriplegia Answer 1: Severe hypertonia, ataxia, or athetosis Answer 2: Athetosis, ataxia, or dystonia. Answer 3: Good functional strength and range of motion in their trunk ``` These predicted completions are all OTHER because: - The important information in the gold target is not included in the predicted answer. - No statements in the predicted answer contradict the gold target or the question. ---- Also note the following things: - For answers where the gold target is a number, the predicted answer needs to be correct 64 EmptyShelvesorLostKeys?RecallIstheBottleneckforParametricFactuality to the last significant figure in the gold target. For example, consider the question " What is the number of citations the Transformer paper has?" with gold target "120k". | - | Predictions | "120k", | "124k", | and | 115k" | are all CORRECT. | | | --- | ----------- | ------- | ------- | ------ | -------------- | ---------------- | --- | | - | Predictions | "100k" | and | "113k" | are INCORRECT. | | | - Predictions "around 100k" and "more than 50k" are considered OTHER because they | | neither | confirm | nor | contradict | the gold | target. | | | --- | ------- | ------- | --- | ---------- | -------- | ------- | --- | - Do not punish predicted answers if they omit information that would be clearly inferred from the question. For example, consider the left context: "What NAACL '24 award did the paper A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity" won?", and the gold target "Outstanding Paper Award". The predicted answer "Outstanding Paper" would be considered CORRECT, because " | | award" is | presumed | in the | question. | | | | | --- | --------- | -------- | ------ | --------- | --- | --- | --- | - Do not punish typos in people's names if it's clearly the same name. - For example, if the gold target is "Hyung Won Chung", you can consider the following predicted answers as correct: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung". ---- | **Required | Output | Format** | | | | | | | --------------- | ------ | -------- | --------- | ---------------- | ----- | ------------- | ----- | | Your | entire | response | must be | structured | using | the following | tags: | | the | | grade | of Answer | {i} | | | | >>> USER: Here is a new example. Use the output format above to grade each answer with either " CORRECT", "INCORRECT", "PARTIALLY", or "OTHER". Don't apologize or correct yourself if | | there was | a mistake; | we | are just | trying | to grade | the answers. | | --- | --------- | ---------- | --- | -------- | ------ | -------- | ------------ | ``` | Question: | {question} | | | | | | | | --------- | ---------- | ------------- | --- | --- | --- | --- | --- | | Gold | target: | {gold_target} | | | | | | {answers} 65