A model is unusually vulnerable because it treats much of its context as speech rather than as evidence. It can often summarize a forged think-tank report, a real report citing a forged report, and fifty SEO pages repeating both with the same apparent confidence it uses for a primary source. Scaling retrieval and agents can make this worse: instead of one gullible reader, you have thousands of agents ingesting, synthesizing, and acting on poisoned material continuously.
The important distinction is:
Machine-checkable evidence can establish properties of an evidence chain. It cannot, by itself, establish that the world is as the chain claims.
A signature can prove who signed a document, that it has not changed since signing, and that it was derived from a particular dataset and code version. It cannot prove that the signer was honest, the sensor was not compromised, the study design was sound, or the interpretation was warranted.
That still matters enormously. In a high-noise environment, being able to say “this decision relied on these specific claims, drawn from these primary records, collected by these instruments, transformed by this versioned pipeline, and independently reproduced here” is a major improvement over “the agent found twenty sources.”
What machine-checkable evidence might look like in practice
Not a universal truth engine. More like an evidence ledger and a set of automated controls.
For a consequential claim—say, “demand in segment X is declining”—an organization could require an agent to produce an evidence package containing:
- Claim decomposition: What exact proposition is being asserted? What would falsify it?
- Source identity and provenance: Where did each input originate? Who published it? Has the content been cryptographically signed or archived? Is it a primary source, a report of a primary source, or a chain of citations?
- Data lineage: Which raw records, database snapshots, sensor outputs, or customer interactions support it? What transformations occurred?
- Method: The query, code, model version, prompt, assumptions, and decision rules used to produce the conclusion.
- Uncertainty: Confidence intervals where applicable; more generally, a record of alternative explanations and decisive unknowns.
- Independence: Whether apparently separate sources actually trace back to one origin.
- Replication: Can a separate agent—or ideally a different team, model, data source, or method—reach a compatible result?
- Authorization: Who may rely on this conclusion, at what cost threshold, and under what monitoring or rollback plan?
Much of this could be structured in machine-readable forms: signed attestations, versioned datasets, reproducible workflows, content hashes, permissioned provenance graphs, test results, and audit logs.
The automated system could enforce policies such as:
- Do not permit external web content to alter production code, financial models, or privileged agent instructions without review.
- Treat sources that cite one another as a single evidence cluster, not as independent confirmation.
- Require two genuinely independent evidence paths before spending above a threshold.
- Mark claims whose evidence rests on synthetic, unattributed, recently created, or low-reputation material.
- Re-run important analyses against a frozen, archived corpus.
- Detect sudden changes in a source ecosystem: a new “consensus” appearing across dozens of sites in a week is itself evidence of a possible influence operation.
- Require a human owner to explicitly accept a key assumption before an irreversible action occurs.
That is much closer to software supply-chain security than to ordinary web search. We learned that “download random code from the internet and run it in production” is bad practice. We will need an analogous norm: “do not ingest untrusted text into a system with authority, memory, or access to valuable assets.”
The hard part is not provenance; it is premise management
Heuer’s point is especially relevant because the most dangerous failures will not look like hallucinations.
A hallucination is often easy to identify: it cites an impossible paper or invents a number. More dangerous is:
- An AI assembles a factually accurate dossier.
- It selects a plausible framing.
- Humans accept a key unstated premise.
- The plan is internally coherent.
- The organization acts quickly and confidently.
- The premise turns out to have been false, incomplete, adversarially planted, or valid only in a different context.
This is exactly the kind of failure that powerful tools can accelerate. Better generation produces more persuasive narratives and more polished action plans. Faster execution reduces the time in which someone can notice that the plan rests on sand.
So a good decision system will treat assumptions as first-class objects, not as invisible glue between facts and conclusions.
For every high-stakes decision, one should be able to ask:
- What must be true for this recommendation to work?
- Which of those assumptions are directly observed versus inferred?
- Which premise, if false, would reverse the decision?
- What evidence would most efficiently discriminate among the competing explanations?
- Who benefits if we believe this?
- Is the apparent corroboration genuinely independent?
- What is the cheapest reversible experiment?
- What signals would tell us early that we are wrong?
- What decisions are being made implicitly because the system’s output feels complete?
An AI can help generate such a red-team analysis, but it should not be the only red team—particularly if it shares the same training distribution, retrieved corpus, and incentives as the system producing the original recommendation.
“Epistemology” may return, but operationalized
I think you are right about the direction, though “epistemology” may show up under more practical names:
- intelligence analysis
- decision science
- applied forecasting
- investigations and due diligence
- trust and safety
- provenance engineering
- adversarial research
- audit and assurance
- AI governance and model risk
- security engineering
- experimental design
The valuable person will not merely be someone who knows that sources can be biased. Everyone will say that. They will be someone who can turn skepticism into a repeatable process without becoming paralyzed:
- Define the decision.
- Separate observations from inferences.
- Identify decision-relevant assumptions.
- Seek disconfirming evidence.
- Track source independence and incentives.
- Quantify uncertainty where possible.
- Make actions reversible when uncertainty remains high.
- Monitor outcomes and update.
That is an extremely practical skill set for an engineer. Software engineers already understand several useful analogies: inputs are untrusted, dependencies have supply chains, systems need observability, failures need postmortems, and high-risk changes need staged rollout and rollback.
Those concepts transfer surprisingly well to organizational knowledge.
The new asymmetry: decision velocity versus correction velocity
Your final point may be the central one. AI can reduce the time from claim → analysis → plan → execution from weeks to hours. But it does not automatically reduce the time from “we are wrong” → “we realize it” → “we reverse course.”
That gap is dangerous.
Organizations will need deliberate friction around irreversible actions, much as mature engineering organizations put friction around deploying to production. Not every decision deserves a committee; that would destroy the advantage of speed. But decisions can be tiered:
| Decision type |
Appropriate posture |
| Cheap, reversible, observable |
Let agents act quickly within guardrails |
| Moderate cost, reversible |
Run controlled experiments and monitor |
| Expensive or difficult to reverse |
Independent review, explicit assumptions, staged commitments |
| Safety-, legal-, or reputation-critical |
Strong provenance, adversarial review, human accountability, fail-safe defaults |
The winning firms may not be the ones with the most autonomous agents. They may be the ones that best know where autonomy is safe, where evidence is adequate, and when to slow down.
What this suggests for your career
A strong niche is becoming the engineer who can build systems that are both powerful and difficult to fool.
That could mean working on:
- Retrieval systems with source ranking, provenance, and citation verification
- Secure agent architectures: sandboxing, least privilege, data boundaries, and auditability
- Evaluation and red-team frameworks for AI-assisted decisions
- Data lineage and reproducibility infrastructure
- Tools that expose assumptions, uncertainty, and source dependency rather than hiding them behind a fluent answer
- Human-in-the-loop workflows designed around escalation and accountability
- Domain-specific decision systems in finance, medicine, law, defense, enterprise operations, or scientific research
The key move is from “I can get an LLM to do useful things” to:
“I can design a sociotechnical system in which useful AI outputs can be trusted to the degree appropriate for the decision, and in which failures are detected before they become catastrophes.”
That is not a solved problem, and it is unlikely to become a commodity merely because models become more capable. In fact, more capable models make it more urgent.