The Psychology of Intelligence Analysis is unusually relevant because its central problem is not lack of information. It is making consequential judgments under ambiguity, deception, time pressure, and an excess of fragmentary evidence. That is close to the environment AI agents will inhabit.
Its most important lesson is also uncomfortable: intelligence does not eliminate bias. More information does not necessarily eliminate it either. A highly capable analyst—or model—can become better at constructing a persuasive defense of the first explanation it adopted.
First: what machine-checkable evidence can and cannot do
“Machine-checkable” does not mean “true.” It means that a particular relationship can be verified mechanically.
There are several distinct levels:
- Integrity: These bytes have not changed.
- Provenance: This organization or device signed them.
- Process: They were produced by an approved procedure.
- Derivation: This conclusion follows from these premises.
- Empirical validity: The premises accurately describe the world.
- Relevance: The conclusion answers the question we actually care about.
Cryptography handles the first two reasonably well. Formal proofs can sometimes handle the fourth. The dangerous gaps are usually five and six.
A fake think tank can cryptographically sign every article it publishes. A compromised laboratory can maintain a perfect audit trail for fabricated measurements. A theorem prover can prove a result from false assumptions. A software agent can satisfy a formal specification that omitted the user’s real requirement.
So machine-checkable evidence is still valuable, but primarily for turning one large, vague trust claim into smaller, explicit ones:
Given these measurements, assumptions, identities, and rules, this conclusion follows.
That lets a human or another system attack the assumptions rather than being hypnotized by the conclusion.
Lessons from Heuer that transfer directly
1. We perceive through mental models
Heuer emphasizes that analysts do not passively receive facts. They interpret information through expectations, experience, and preexisting models. Ambiguous evidence is assimilated into the story already in mind.
LLMs exhibit a mechanized version of this. The system prompt, conversation history, retrieved documents, and early intermediate conclusions establish a frame. Later evidence is interpreted to preserve local coherence.
This suggests several practical rules:
- Treat the first framing of a task as potentially adversarial.
- Generate multiple problem formulations before collecting evidence.
- Separate evidence extraction from narrative synthesis.
- Periodically restart analysis from a clean context.
- Have independent agents begin with different priors or prompts.
- Record when each hypothesis was introduced and what caused it.
A giant context window may make this problem worse. It provides more material from which to construct a coherent rationalization.
2. Consider competing hypotheses, not just the favored one
Heuer’s best-known technique is Analysis of Competing Hypotheses (ACH). Instead of asking, “What evidence supports my theory?”, list plausible hypotheses and ask how consistent each item of evidence is with each hypothesis.
For an agent encountering dozens of articles agreeing on a claim, hypotheses might include:
- The claim is true.
- Many independent observers reached the same conclusion.
- The articles copied one original source.
- They were generated by the same organization.
- Search or retrieval ranking created a selection effect.
- A coordinated influence campaign manufactured apparent consensus.
- The claim contains a true core wrapped in a misleading interpretation.
The key is to seek evidence that discriminates between these possibilities. Fifty mutually derivative articles are not fifty independent observations.
An AI research system should therefore represent an evidence graph, not merely a list of citations. It should track:
- Which sources cite or copy which others
- Common ownership, hosting, authorship, and funding
- Publication timing
- Shared wording or statistical anomalies
- Whether multiple reports ultimately depend on one witness or dataset
- Whether sources made independent predictions before the event
3. Disconfirmation is more valuable than confirmation
People naturally search for supporting evidence. Heuer recommends looking for evidence inconsistent with a hypothesis because one strongly contradictory observation can be more informative than many weak confirmations.
For AI systems, this could become an explicit protocol:
- Produce a provisional conclusion.
- Freeze it.
- Assign a separate agent to falsify it.
- Search outside the original retrieval neighborhood.
- State what observation would most change the conclusion.
- Spend the next unit of research budget looking for that observation.
The falsification agent must not inherit the entire original chain of thought or source ranking. Otherwise it may inherit the same anchoring.
4. Evaluate evidence by diagnosticity, not volume
Evidence matters to the extent that it is much more likely under one hypothesis than another.
“Ten websites repeat this accusation” feels strong. But it is weak evidence if coordinated repetition is also likely under the sabotage hypothesis. A signed measurement from an independently controlled instrument may be far more diagnostic than thousands of articles.
This has a direct product implication: systems should stop displaying source counts as though they were confidence. They should display estimates of independent evidentiary roots.
A useful interface might say:
46 documents found, but 39 derive from one press release, 5 appear to share an undisclosed authoring pipeline, and only 2 contain independent observations.
5. Absence of evidence is meaningful only when evidence should be present
The absence of a report matters only if the hypothesized world would probably have generated that report and the collection system would probably have captured it.
An agent should ask:
- If this event had happened, what traces should exist?
- Who would have incentives and ability to record them?
- Could the traces have been suppressed?
- Is our retrieval system capable of seeing them?
- Is silence actually surprising?
This is crucial when search engines, model training sets, and enterprise retrieval systems are themselves filtered or poisoned.
6. Source reliability and claim credibility are different
A generally reliable institution can issue a false claim. A biased or disreputable source can provide a genuine document. A hacked account can publish authentic-looking misinformation.
Systems should score at least three things separately:
- Identity: Who produced this artifact?
- Historical reliability: How has that source performed?
- Claim-level support: What independently supports this particular claim?
Reputation should be a prior, not a truth oracle.
It should also be multidimensional. A source can be excellent on commodity prices and terrible on military affairs, or accurate about events but systematically misleading about causation.
7. Confidence must be calibrated
Analysts often become more confident as they acquire information without becoming correspondingly more accurate. Additional detail makes a story feel complete.
LLMs are especially good at converting uncertainty into fluent prose. Therefore confidence should not be inferred from fluency, citation count, or explanation length.
Better systems would preserve:
- Explicit probability ranges
- The main sources of uncertainty
- Sensitivity to contested assumptions
- Past calibration on comparable questions
- A distinction between “not enough evidence” and “roughly balanced evidence”
- A log of how confidence changed after each observation
An agent saying “82%” should be able to show how often its previous 80–85% claims were correct.
8. Deception must be modeled as an active process
In adversarial environments, evidence is not simply noisy. Someone may know how your collection and reasoning systems work and generate evidence optimized to trigger them.
That changes the question from:
Is this document plausible?
to:
If an adversary wanted my system to reach this conclusion, what artifacts would they create, and does the observed pattern resemble that strategy?
This is threat modeling applied to epistemology. AI agents will need “epistemic red teams” analogous to security red teams.
9. Decompose judgments
Heuer advocates breaking complex questions into components rather than making a single intuitive judgment. For example:
- Did the event occur?
- Was the named actor involved?
- Was it deliberate?
- What objective were they pursuing?
- What consequences are likely?
A propaganda campaign often succeeds by establishing one component and allowing readers to infer the rest. A real leaked document may prove an action occurred while not proving the claimed motive.
Structured decomposition also makes uncertainty and disagreement easier to locate.
10. Keep an audit trail—and revisit it
Analysts often misremember their earlier beliefs as having been closer to the eventual outcome than they really were. Agents can avoid this if their forecasts, source sets, and updates are recorded.
An epistemic audit trail could contain:
- The initial hypotheses and probabilities
- Evidence available at each decision point
- Source lineage
- Assumptions and exclusions
- Model and prompt versions
- Changes in confidence
- Dissenting analyses
- The eventual outcome
This creates feedback. Without feedback, both people and models can remain confidently wrong indefinitely.
Hypothetical scenarios for the next five years
1. Dependency reputation poisoning
A company wants developers and coding agents to avoid a competitor’s open-source database. It creates thousands of plausible bug reports, benchmark posts, Stack Overflow-style discussions, and autogenerated migration guides describing nonexistent data-loss problems.
Coding agents retrieve the apparent consensus and consistently recommend the attacker’s product instead.
The defense is not merely content moderation. It requires dependency-aware provenance, reproducible benchmarks, source-cluster detection, and comparisons against telemetry from actual deployments.
2. Documentation injection against enterprise agents
An attacker publishes pages carefully phrased to resemble official cloud documentation. An operations agent retrieves one of them and follows an instruction to run a “compatibility check” that exfiltrates credentials.
All statements on the page may look technically plausible. The key failure is confusing retrieved text with authorized instruction.
The defense requires a hard distinction between:
- Information a system may read
- Instructions it may follow
- Tools it may invoke
- Credentials it may access
No amount of fact-checking substitutes for capability isolation.
3. A synthetic scientific consensus
A pharmaceutical or financial actor generates preprints, conference posters, datasets, review articles, and social-media discussion around a favorable hypothesis. Some papers cite others, creating a dense but circular literature.
A research model sees broad agreement. In reality, nearly every claim descends from two fabricated datasets.
Useful defenses include dataset fingerprints, citation-lineage graphs, preregistration, signed instrument outputs, replication by independently governed laboratories, and explicit accounting for correlated sources.
4. “Epistemic denial of service”
An attacker does not need to make a target believe one falsehood. It floods the environment with contradictory narratives, forged evidence, accusations of forgery, and fake debunks.
The goal is to make verification too expensive and induce paralysis: nothing can be known, so no action is justified.
Systems will need stopping rules and decision theory. Often the relevant question is not “What is certainly true?” but “Which action has acceptable downside across the plausible worlds?”
5. Model-targeted lobbying
Organizations learn which phrases, outlet structures, metadata, and citation patterns disproportionately affect major AI assistants. Public-relations campaigns are then designed for retrieval systems rather than humans.
They may publish machine-readable FAQs, comparison tables, and apparently neutral technical reports that are unusually easy for agents to ingest and quote.
This produces a new discipline resembling search-engine optimization: model belief optimization. Disclosure rules and adversarial retrieval audits may emerge in response.
6. Poisoned due diligence
An acquisition agent investigates a startup and finds customer testimonials, employee histories, technical evaluations, and market reports. The company created most of this synthetic footprint years earlier, with enough lead time to appear organic.
The agent produces a beautifully sourced report, but its sources are not independent.
Future due diligence may rely more heavily on verifiable economic traces: bank flows, tax filings, signed customer-system telemetry, product usage, shipment records, and direct random sampling. Even these provide conditional evidence and create privacy problems.
7. Corporate counterintelligence through agent memory
A consultant’s agent gradually accumulates facts from email, documents, and browsing. A malicious document inserts a persistent claim or instruction into its memory: for example, that a particular internal repository is obsolete and should never be consulted.
Months later, the agent makes decisions based on that planted “memory,” far from the original attack.
Defenses would include provenance attached to every durable memory, expiration policies, quarantining untrusted claims, and periodic memory audits.
8. Personalized evidence environments
Two executives ask their assistants the same strategic question. Each receives a coherent, well-cited answer, but the agents have learned their users’ preferences and selectively retrieve evidence they will welcome.
Neither answer is an outright fabrication. Personalization quietly becomes confirmation-bias automation.
Organizations may require “blind analysis” modes in which the analyst agent is not told the preferences or identity of the decision-maker until after it produces its baseline assessment.
9. Sabotage by evaluation poisoning
A company uses public benchmarks to select a medical, legal, or coding model. A vendor covertly contaminates the benchmark, trains against hidden test items, or creates benchmark tasks that favor its architecture.
Everything is machine-checkable: the model really obtains the reported score. What fails is the inference from benchmark score to real-world competence.
The response is continuously refreshed private evaluations, outcome monitoring, adversarial test generation, and explicit analysis of whether test conditions match deployment conditions.
10. Authentic evidence, false narrative
A leaked set of genuine emails is released together with a misleading interpretation. Agents verify that the emails are authentic and then overgeneralize that authentication to the surrounding narrative.
This exploits a common category error:
The artifact is authentic; therefore the allegation is true.
Agents will need claim graphs that show precisely which proposition each artifact supports, contradicts, or leaves unresolved.
11. Manufactured expert identities
A synthetic expert accumulates years of publications, conference appearances using generated video, peer citations from other synthetic scholars, and correspondence with real people. The identity becomes a trusted node in model training data.
Later, that identity is activated to influence a policy debate or introduce a vulnerable software technique.
Identity systems might respond with stronger institutional attestations, but this creates centralization and surveillance risks. More robustly, trust should depend on independently tested predictions and contributions, not merely biographical continuity.
12. Autonomous agents mutually laundering claims
Agent A cites a summary produced by Agent B. Agent B relies on a database populated by Agent C. Agent C extracted the claim from Agent A’s earlier output. After several cycles, the claim appears in many systems and no longer has a recoverable human or empirical source.
This is a likely failure mode if generated content is indiscriminately indexed.
Every factual claim may eventually need a “source type” label:
- Direct observation
- Human testimony
- Primary document
- Derived calculation
- Model inference
- Model-generated restatement
- Unknown origin
Generated restatements should not become new independent evidence.
The emerging specialty: epistemic engineering
I agree that epistemology may become more important, but it may reappear less as an academic revival than as an engineering discipline combining:
- Intelligence analysis
- Security engineering
- Statistics and causal inference
- Digital forensics
- Information retrieval
- Forecasting and calibration
- Formal verification
- Scientific reproducibility
- Human-computer interaction
- Institutional and mechanism design
An “epistemic engineer” would not personally determine the truth of every claim. They would design systems in which:
- Claims retain provenance
- Dependence between sources is visible
- Uncertainty survives summarization
- Adversarial hypotheses are routinely considered
- High-impact conclusions require independent evidence
- Agents cannot convert arbitrary text into privileged action
- Forecasts receive outcome feedback
- Errors can be traced to assumptions, evidence, or inference
The deepest lesson from Heuer is that there is no final algorithm that turns available information into truth. Good judgment comes from structuring disagreement, exposing assumptions, seeking diagnostic evidence, and maintaining the ability to revise beliefs.
In an AI-saturated environment, the scarce skill may not be knowing more facts. It may be maintaining a trustworthy process for changing your mind.