“Machine-understandable” hides several different requirements:
- The information is digitized.
- Its meaning can be represented formally.
- A machine can infer the intended meaning in context.
- It can act correctly despite ambiguity and missing information.
- People will accept responsibility being delegated to it.
Many fields have achieved the first while remaining far from the fifth. A hospital can have every record in a database without the patient’s condition being cleanly legible to software.
Why some domains resist legibility
1. The important knowledge is tacit
Much professional knowledge is not written down because humans acquire it through experience:
- An experienced engineer knows which “temporary” dependency is dangerous.
- A nurse notices that a patient looks wrong despite normal measurements.
- A salesperson realizes that an objection is actually a polite refusal.
- A manager knows that a project’s reported status is politically sanitized.
People often cannot provide an explicit decision procedure for these judgments. They recognize patterns involving posture, timing, history, incentives, and deviations from local norms.
Multimodal models can learn some of this, but collecting representative examples with reliable outcomes is difficult. The rare cases where judgment matters most are often the least represented.
2. Reality does not naturally come with a stable schema
Software works best when the world can be represented as a set of entities, properties, and permitted operations. Many domains do not cooperate.
Is a contractor effectively an employee? Is a medical event a new illness, a complication, a coding artifact, or the continuation of an old condition? Is a customer “active” if they pay but no longer use the product?
These are not merely data-cleaning problems. Categories are created for particular purposes. Clinical, billing, legal, and scientific definitions of the same event may differ without any one of them being simply wrong.
This is why ontology projects often disappoint. They attempt to discover one true schema where there are actually several incompatible but useful perspectives.
3. Meaning depends on context
Consider the sentence “The customer approved the change.” Its operational meaning may depend on:
- Which employee said it
- Whether they had contractual authority
- What version they saw
- Whether approval was conditional
- Local conventions about what “approved” means
- Relevant communications not stored in the system
Humans fill gaps using shared context. Machines either need that context captured or must guess it. LLMs are unusually good guessers, but confident guessing is unacceptable in many high-stakes settings.
4. The world is only partially observable
A machine cannot reason from information that was never sensed or recorded. Organizations contain enormous “dark matter”:
- Hallway conversations
- Undocumented workarounds
- Informal authority
- Broken sensors
- Private motives
- Facts omitted to avoid embarrassment
- Processes that differ from official procedures
Digitizing the official workflow may produce a precise model of how the organization claims to work, not how it works.
5. Feedback is sparse, delayed, and confounded
Machine learning is easiest when actions receive quick, unambiguous feedback. Many important decisions do not.
Was a hiring decision good? You observe only the person hired, perhaps years later, within a particular team. Did a policy prevent a disaster, or was no disaster going to happen? Did a patient recover because of treatment, despite treatment, or for unrelated reasons?
The machine is asked to learn causality from records generated by previous decisions. This creates selection bias and makes counterfactuals unavailable.
6. Some domains are adversarial
Once a system’s decision rule matters, people adapt to it:
- Applicants optimize résumés for screening models.
- Hospitals optimize documentation for reimbursement.
- Students optimize work for graders.
- Criminals probe fraud detectors.
- Employees optimize metrics rather than outcomes.
The data ceases to be a passive description of reality. It becomes part of a game. A model may successfully learn yesterday’s behavior just as everyone changes behavior in response to the model.
This is Goodhart’s law in operational form: when a measurement becomes a target, its relationship to the underlying goal degrades.
7. The objective is disputed
Machines can optimize a specified objective, but many institutions exist to negotiate among incompatible values.
A city’s transportation system might balance speed, safety, accessibility, cost, emissions, neighborhood character, and fairness. There is no technically correct weighting. Similarly, legal questions often concern which values and precedents should control, not merely which facts apply to a settled formula.
Calling this “ambiguity” can obscure the real issue: society has not agreed on the answer. A model cannot eliminate politics by predicting what a person might say.
8. Accountability is a social institution
People sometimes demand a human decision not because humans are more accurate, but because humans can:
- Explain themselves
- Be cross-examined
- Exercise mercy
- Make commitments
- Carry professional duties
- Be punished or removed
- Represent an institution with legitimate authority
A machine can generate an explanation, but that is different from having an accountable party. Decisions involving liberty, employment, medicine, or large financial consequences require a governance structure, not only a capable predictor.
9. Making a field legible is expensive and politically disruptive
A messy workflow can often be formalized, but doing so requires agreement about definitions, ownership, permissions, exceptions, and responsibility. That exposes conflicts organizations may prefer to leave implicit.
For example, everyone might agree that “customer data should be accurate.” But standardization forces decisions about:
- Which department owns each field
- Whose definition wins
- Who must repair historical records
- Who is blamed for errors
- Which team loses autonomy
The technical integration is often easier than the institutional integration. “Human quirks” are therefore not peripheral. Incentives, status, trust, and conflict are part of the domain being modeled.
Are there fundamental limits?
Some obstacles are fundamental, although they do not imply that machines cannot be extremely useful.
- Irreducible uncertainty: The future contains genuinely random or unknowable events.
- Partial observability: Distinct states of the world can produce identical available evidence.
- Non-identifiability: The same observations can support multiple causal explanations.
- Computational complexity: A well-defined answer may be prohibitively expensive to calculate.
- Undecidability: Some general properties of programs cannot be determined algorithmically.
- Reflexivity: Predictions can alter the behavior being predicted.
- Value pluralism: Some disagreements have no factual resolution because they reflect different preferences.
Humans do not overcome these limits either. We often conceal them through convention, authority, improvisation, or unjustified confidence. Automation exposes the absence of a clean answer because a system designer must decide what happens in every represented case.
What LLMs change
Traditional software required the world to be translated into a rigid schema before it could do much with it. LLMs can operate over language, images, examples, and incomplete instructions, so they push the boundary of machine legibility considerably outward.
They are especially good at translating between representations:
- Free text into structured records
- Policy into tentative rules
- User intent into software actions
- Different professional vocabularies into one another
- Historical cases into plausible recommendations
But they often replace explicit formalization with learned approximation. That is enormously useful, yet it does not make the underlying ambiguity disappear. The model may smoothly interpolate across conflicting definitions, missing context, and uncertain facts without indicating where the cracks are.
So the central engineering question changes from:
Can we encode this domain?
to:
Under what conditions can we trust an approximate interpretation, how do we detect when those conditions fail, and who resolves the failure?
Where the resulting work will be
The durable opportunities are likely to be at this boundary between messy reality and machine action:
- Building domain-specific evaluations from real outcomes
- Capturing provenance and uncertainty
- Designing human escalation procedures
- Constructing permissions and limits for agents
- Reconciling incompatible schemas
- Instrumenting physical and organizational workflows
- Auditing for strategic manipulation
- Converting professional norms into testable constraints
- Assigning accountability when automated work causes harm
That suggests a useful career principle: do not specialize solely in producing artifacts that models can cheaply generate. Specialize in making an important, resistant domain sufficiently legible for machines to help—without pretending it is more legible than it really is.
The most valuable generalists may not be people who can instantly impersonate any specialist. They may be people who can enter a domain, learn where its formal descriptions diverge from reality, and build a reliable bridge between experts, institutions, data, and AI systems.