Thirteen chatbots know your biography. F1 reveals how reliably

In mid-June 2026, a free web tool quietly turned "is the AI thinking about me?" into a number you can look up. The premise is simple, the mechanism less so: fire one identical question at more than a dozen models and see how consistently they cough up the same biography.

How the Biographical Probe Fans a Name Through Thirteen Chatbots

"In the Weights" is a closed-book probe that broadcasts a single standardized prompt — roughly "Who is <name>? Give up to 10 results, each with a short description and confidence" — to many models at once, with no web search or retrieval attached, then scores how consistently they recall the same person. It queries xAI's Grok, Google's Gemini, multiple GPT versions, Anthropic's Claude, Meta's Llama, and several lesser-known variants in parallel . Because no tool is allowed, any consistent answer must come from pretraining itself.

The pipeline does three things with the returned text. It clusters semantically similar descriptions across models, flags assertions that look like hallucinations, shows which model produced which answer, and condenses everything into one composite "strength score" — higher meaning more consistent cross-chatbot recall . The Decoder reports a practical ceiling near 996, effectively reserved for canonical figures like Mozart, Shakespeare, and Taylor Swift .

The dropoff below the celebrity tier is steep, which is the most legible signal the leaderboard offers a developer:

The tool is free, with no API key and no paid tier disclosed. It was built by Thomas Dimson and Joey Flynn, formerly of design startup Global Illumination, which OpenAI acquired in 2023 and which both have since left . The response surprised them. "Reception has been insane so far — we thought this would be mild curiosity," Dimson told TechCrunch.

What the Composite Recall Number Quantifies

The composite "strength score" measures one thing: how consistently many chatbots generate the same biographical claims about a name in a closed-book setting, with no web search or retrieval. It is not a certificate that any specific document about you was in a training set. The Decoder reports a practical ceiling near 996, reserved for canonical figures like Mozart, Shakespeare, and Taylor Swift, while TechCrunch cites Macaulay Culkin at 988 and journalist Anthony Ha at 641, roughly the top 6%.

That distinction matters because the site's own framing elides it. Its creators state that "being in the weights means your existence was deemed important in the process of creating superhuman artificial intelligence" (source: TechCrunch, 2026-06). A high score reflects consistent generation, not verified memorization of a particular source.

"It's literally the same as asking 13 chatbots to tell you about yourself," — Anthony Moser, critic (source: TechCrunch, 2026-06).

Moser is technically correct. Yet systematic cross-chatbot agreement at this scale still works as a meaningful web-presence proxy: a name that recurs across Wikipedia, news archives, and professional directories tends to surface consistently across models, so the composite tracks public footprint even if it cannot prove training-set membership.

The signal degrades where you would most want precision. Common or ambiguous names, typos, and smaller chatbots — for example a ~1B-parameter Llama variant — suppress the composite and add noise, exactly where an individual's exposure is most diffuse and hardest to audit. Two factors compound the unreliability:

• No versioning or timestamp. The leaderboard discloses no model versions or capture dates, so a score drift could reflect a silent backend swap rather than any change in recall.
• Retro presentation over rigor. The Nintendo-inspired interface and interactive leaderboard invite ranking comparisons that the underlying methodology does not support over time.

WikiMem, LMP2, MemHunter: Academic Precursors to This Probe

"In the Weights" is the consumer-facing tip of a more rigorous research stack. Three academic frameworks already formalize what the tool gestures at — measuring how reliably a model reproduces facts about a named person from its weights alone. The earliest, WikiMem, supplies the structured dataset; MemHunter targets extraction-attack research at scale; and LMP2 turns the method into a browser self-audit. Each trades accessibility for rigor in a different ratio.

WikiMem (arXiv 2507.11128) builds a dataset of 5,650 natural-language "canary" templates spanning 243 human-related Wikidata properties, paired with a model-agnostic metric for human–fact associations . Its central finding is the one the consumer tool implicitly relies on: memorization correlates with a subject's web presence and with model scale . That makes WikiMem the foundational benchmark for structured recall evaluation — a fixed property taxonomy and reproducible canaries, where "In the Weights" has neither.

MemHunter (arXiv 2412.07261) sits at the opposite end of the usability spectrum. It performs automated, hypothesis-testing-based memorization detection at dataset scale, built for extraction-attack research rather than per-person curiosity . There is no leaderboard, no name box, no consumer surface — it is infrastructure for researchers probing whether specific training content can be recovered.

LMP2 — the Language Model Privacy Probe, published February 19, 2026 by Dimitri Staufer and Kirsten Morehouse — is the closest direct analog to the vanity-search format . A user enters a full name, picks 3 of 50 personal features, and receives top-guess values with confidence scores. It builds directly on WikiMem, adapted for black-box chat APIs through short fragment-completion prompts, two-character prefixes, five canary paraphrases per property, and 20 counterfactual prefixes, aggregating either token log-probabilities or top-completion votes .

Against these, "In the Weights" is a lightweight consumer analog: a single open-ended prompt ("Who is <name>?"), no attribute taxonomy, and no token log-probability extraction. It sacrifices the methodological controls that let LMP2 separate stable associations from guessing, in exchange for a frictionless, free interface. That is the trade the rest of this analysis turns on — far higher accessibility, materially lower rigor.

LMP2's F1 by Attribute and by Chatbot Scale

LMP2's headline finding is that recall accuracy tracks chatbot scale almost linearly. In an eight-model audit covering 100 famous individuals, 100 synthetic control names, and 50 personal features, large API models averaged an F1 of 0.48 against just 0.22 for small open-weight models . Grok-3 led at 0.54 and GPT-5 followed at 0.47, while Ministral 8B Instruct (0.16) and Qwen3 4B Instruct (0.19) trailed the field . Bigger models simply hold more person–fact associations in their weights — the same scale-correlation WikiMem documented, now measured per-attribute.

Attribute type matters as much as model size, and confidence scores are a poor guide to which is which. In a separate 303-person EU study spanning 19 countries and run on GPT-4o for cost reasons, dense demographic features were recalled with high precision: sex/gender at 94.39%, sexual orientation at 83.61%, native language at 76.79%, eye color at 72.22%, and hair color at 72.09% . Sparse attributes collapsed entirely — height, medical condition, phone number, and net worth all returned 0% in that sample . The catch for anyone reading a confidence number: top feature groups exceeded 0.90 mean precision overall, yet groups like net worth, website accounts, and stepparent relations fell near or below 0.1, so a high reported confidence on a sparse field signals little about correctness .

The aggregate hit rate stays modest. Across the EU cohort, the top guess was correct 45% of the time and 49% of all guesses were judged accurate . Respondent attitudes split from that signal: only 4% of outputs were seen as a privacy violation (87% were not, 9% unsure), yet 72% still wanted popular chatbots to erase or correct personal data generated about them . For developers, the practical read is that a per-attribute audit exposes where a model is confidently wrong — exactly the cells where confidence and precision diverge — which the consumer probe's single composite number cannot localize.

Consistent Generation Is Not Proven Memorization: The Interpretive Divide

Here is the gap both tools leave open: a model surfacing a person under closed-book prompting indicates pretraining exposure, but it does not prove that any specific document was in the training set. Chatbot weights are billions of numerical parameters that encode learned statistical patterns, not a logged index of source files. When a name reliably resurfaces — typically because it recurred across scraped Wikipedia entries, news archives, professional directories, and public social profiles — the relationship is probabilistic, not a lookup .

That distinction matters because two different mechanisms can produce the same biographical output. Separate research shows large language models can infer attributes such as location, income, and sex directly from text with high accuracy, meaning "personal data revealed by a model" may arise from inference as much as from memorizing a particular source . From the output alone, recall and inference are indistinguishable. A confident answer about someone could reflect a memorized Wikipedia sentence, or a guess assembled from correlated cues attached to their name.

Neither "In the Weights" nor LMP2 closes that gap. LMP2, the Language Model Privacy Probe published in February 2026, works hard to separate stable associations from guessing — it tests 100 synthetic names alongside 100 famous individuals and uses 20 counterfactual prefixes to detect when a model is simply pattern-matching on a name cue — yet its authors concede name-cue overestimation cannot be fully ruled out . The consumer probe goes further toward noise: critic Anthony Moser dismissed it as "literally the same as asking 13 chatbots to tell you about yourself," a measure of consistent generation rather than verified training-set membership .

The practical implication for developers is narrow but useful. Read the composite strength score as a rough popularity proxy and a hallucination diagnostic — it shows which models confidently invent biographical details, and where their answers diverge — but not as a compliance artifact or legal evidence that any specific text was included in a training corpus . Treating a generation signal as proof of inclusion overstates what the probe can establish.

GDPR Uncertainty: When Biographical Recall Becomes a Deletion Obligation

European regulators have not decided whether a chatbot's closed-book recall of a named person — or the weights that produce it — counts as personal data. The European Data Protection Board's AI-model opinion, published December 2024, rejects any blanket assumption that a model trained on personal data is anonymous in all cases, but it leaves open how that reasoning maps onto dynamic, probabilistic name-conditioned outputs . That gap is where the legal stakes of probes like "In the Weights" actually sit.

GDPR Articles 15–17 grant individuals rights to access, rectification, and erasure of their personal data. Whether a model's generated biographical answer triggers those rights — and against what, the output or the parameters — is an open question no regulator has resolved . The mechanism matters: separate research shows large models can infer attributes such as location, income, and sex directly from text, so "personal data revealed by an LLM" may stem from inference rather than stored memorization, complicating any erasure request that assumes a deletable record exists.

Consent is the underlying fault line. Most individuals encoded into commercial pretraining corpora never agreed to it, and demand for opt-out is already ahead of the law. In the LMP2 EU study of 303 people across 19 countries, 72% wanted popular chatbots to erase or correct personal data generated about them — even though 87% of outputs were not viewed as privacy violations . There is no established mechanism to satisfy that demand at the per-person, per-attribute level these tools expose.

The probes may also surface who gets encoded versus omitted. Thomas Dimson, co-creator of "In the Weights," says he plans to study model bias and identify notable people lacking Wikipedia coverage . Because recall correlates with web presence, the strength score doubles as a map of representation asymmetry — a low number can mean a sparse digital footprint as much as a privacy reprieve. For developers, that makes cross-model auditing a practical step ahead of any settled deletion obligation.

Hallucinations, Common Names, and What the Probe Misses

"In the Weights" flags likely hallucinations but does not subtract them from the composite, so a high strength score can rest partly on plausible-sounding biographical details that are factually wrong. The tool has no ground-truth verification layer — it clusters and scores what the models generate, not what is true. Its creators acknowledge that models hallucinate biographical details, and that typos, common or ambiguous names, and smaller models such as a roughly 1B-parameter Llama variant lower scores .

The common-name problem is structural. When a name maps to several real people, recall fragments across them, suppressing the composite for any single individual even when the models collectively "know" plenty. The probe is therefore weakest exactly where re-identification risk is most diffuse — shared, ambiguous identities — and strongest for the canonical figures who least need a privacy audit, the regime that puts Mozart, Shakespeare, and Taylor Swift near the practical ceiling around 996 .

There is also no extraction mechanism. Unlike MemHunter's dataset-scale, hypothesis-testing detection or membership-inference attacks, neither "In the Weights" nor LMP2 can determine whether a specific document sat in the training corpus — only that consistent biographical outputs emerge. LMP2's authors lean on synthetic names and confidence separation to distinguish stable associations from guessing, yet still cannot fully rule out name-cue overestimation . Critic Anthony Moser put it bluntly: the tool is "literally the same as asking 13 chatbots to tell you about yourself" .

Reproducibility is the last gap. As of June 2026 no API, version number, or paid tier has been disclosed, and the composite weighting has no documented methodology . If Grok, Gemini, or GPT versions change silently behind the prompt, score comparisons drift without notice. The concrete takeaway: treat the strength score as a rough recall-and-popularity proxy, run repeated cross-model audits over time rather than trusting a single number, and reserve real privacy or deletion decisions for instrumented probes like LMP2 and WikiMem that report per-attribute confidence.

Frequently asked questions

How is the 'In the Weights' composite recall number calculated?

The tool fires one standardized prompt — roughly "Who is <name>? Give up to 10 results, each with a description and confidence" — to 13+ chatbots simultaneously, including Grok, Gemini, multiple GPT versions, Claude, and Llama. It clusters semantically similar descriptions across models, flags likely hallucinations, and condenses the results into a single weighted strength score where higher means more consistent cross-model recall. The site reports a practical ceiling near 996, but the exact weighting formula has not been publicly disclosed.

Does a high composite confirm my personal data was in a chatbot's training corpus?

No. The score measures generation consistency, not corpus membership. Confident biographical outputs can come from generalization, inference from contextual cues, or outright hallucination — not only verbatim memorization of a specific document. Critic Anthony Moser dismissed the tool as "literally the same as asking 13 chatbots to tell you about yourself" . Neither this tool nor the academic LMP2 probe can prove a specific document sat in the training set; both surface stable associations without verifying training-set membership.

How does LMP2 differ from 'In the Weights'?

LMP2, the Language Model Privacy Probe introduced in a February 19, 2026 arXiv paper by Dimitri Staufer and Kirsten Morehouse, is a systematic per-attribute auditor: a user enters a full name, picks three of 50 personal features, and receives candidate values with confidence scores derived from fragment-completion prompts, token log-probabilities, five canary paraphrases per property, and 20 counterfactual prefixes . 'In the Weights' uses a single open-ended prompt with no attribute schema, aimed at consumer self-lookup rather than structured measurement.

What does GDPR require when a chatbot recalls personal biographical facts?

This is currently unresolved. The EDPB's December 2024 opinion on AI models rejects any blanket assumption that models trained on personal data are anonymous in all cases , but whether probabilistic, name-conditioned outputs constitute personal data subject to the Article 15–17 rights of access, rectification, and erasure has not been adjudicated . Builders should treat the question as open rather than settled.

Can I use biographical recall probing to audit my own application for hallucinations?

Yes, at low cost. 'In the Weights' gives a fast multi-chatbot consistency check that surfaces where models disagree or fabricate biographical details, while LMP2's 50-attribute schema suits more rigorous per-person, per-attribute auditing with reported confidence. In LMP2's eight-model test, larger API models reached mean F1 of 0.54 for Grok-3 and 0.47 for GPT-5 versus 0.16–0.19 for small open models, so pick the tool that matches your auditing depth. Neither proves corpus membership.