Epistemic status: This is synthesis. Every piece is borrowed (Elish, Davies, automation bias literature, Goodhart’s Law as analogy, and oversight law literature). My only addition is framing: when AI output becomes plausible, verifying authorship goes from free glance to expensive re-evaluation. Plus one practical rule: audit the control process, not final text.
It is a machine translation of my post in Russian from my garden.
Imagine a health insurance company — let’s call it Meridian. It uses LLM to send out mass claim denials. A patient requests payment for treatment, and the system returns a denial, claiming “insufficient documentation to support medical necessity.”
The denial is erroneous and costly for the patient. At the end of the pipeline sits an operator who presses the “approve” button. On average, they spend about 40 seconds per case, but approve over 99% of the model’s decisions. And this isn’t science fiction. According to ProPublica data from 2023, Cigna’s PxDx system spent about 1.2 seconds “verifying” each case. UnitedHealth is also being sued over a similar pipeline – it’s nH Predict system.
I’m not citing these figures to prove that the denials were wrongful. The erroneous nature of the denial is important only for understanding the problem: the algorithm cripples the patient, but when the patient tries to find someone responsible, there’s no one to look for. The company will, of course, be held accountable in court, but there’s no specific person who made the decision. The decision was generated by a model, and a human simply clicked “confirm.” This begs the question, not a rhetorical one: did anyone make the decision at all? It’s important to understand: blind approval (when the model suggests paying out the insurance, and the operator clicks “OK” without even looking) is structurally the same kind of authorization failure as a blind denial. This remark protects us from the banal “corporations are evil” approach: in this article, I want to discuss not so much the greed of corporations as the very fact of excluding a human from the decision-making process.
The person at the end of this chain has a name. Madeline Claire Elish calls such people denizens of the moral crumple zone. These are employees who are brought into the process solely to shoulder the blame for the automated system’s failure. When the denial is challenged, the employee will be held accountable — after all, it’s her initials that are on it. In reality, she had no real control, nor even the time to review the case in 40 seconds.
As a result, we have three problems: the patient bears the burden, the blame falls on the powerless operator, and no one is held accountable. The fourth problem, and the most important, is that you can never prove the absence of a human decision by looking at the logs.
The main point concerns systems that claim to have a human in the loop—to comply with specific regulatory standards, such as Article 22 of the GDPR (requiring “meaningful human intervention,” as confirmed by the recent SCHUFA court ruling) or Article 14 of the EU AI Act, which requires human oversight.
Previously, it was easy to find a decision made without human intervention: algorithms produced carbon-copy rejections. An auditor only needed to glance at a sample of the texts to understand that no one had carefully considered the details; it was a rubber-stamped version.
This marker no longer works. And it’s not because neural networks have learned to write in a more diverse manner — detectors would have adapted to that. The reason is that the model’s draft is now legitimate. If an algorithm is officially allowed to write emails, the text itself (whether templated or unique) no longer reveals whether anyone has reviewed the case.
This leads to an unpleasant conclusion: any passive verification of already written text is useless for establishing authorship.
No matter how detailed and convincing the rationale in a document, it doesn’t reveal who exactly made the decision. The real evidence is now hidden in non-content metadata: timestamps, access logs, time spent in the document. Those 40 seconds from the example above are the key metric. Authorship can no longer be deduced from the text. Instead, we need to check the control mechanisms that ensure that a human reviewed the case before the model did.
Why is this bad?
The problem is that the neural network’s results aren’t just varied—each one, taken individually, appears well-reasoned and meaningful.
The only remaining text-based check is the “validity” test: is the justification based on the actual documents of a specific applicant? This test still works. But validity is only part of correctness. An LLM fed a patient’s file will easily pass this test. The model might produce a text in which some facts are correct, but the standards are applied incorrectly. To catch such an error, the auditor would have to re-examine the case from scratch, spending as much time as it would have taken to produce a fair initial decision.
A structural shift has occurred: verification, which was previously free and widespread, has become expensive and rare. There are no longer any passive methods for confirming authorship. Only proactive methods or activity log analysis are effective.
This goes beyond compliance: the growth of AI capabilities is destroying not the ability to find errors, but the economics of validation. As systems become smarter, any mass audit will turn into a costly manual procedure. This is a bit like Goodhart’s Law, but with the fundamental difference: the old metric (text templateness) died not because of optimization pressure, but because of distributional shift — the legalization of neural-sloppy drafts. Strictly speaking, this doesn’t fit any of the four classic versions of Goodhart’s Law, as proposed by Mannheim and Garrabrant, but it bears a clear family resemblance to them. Similarly, AI text detectors become useless compared to the best LLMs.
Stop trying to figure out “who decided” by reading the text. The only remaining criterion is pre-exposure independence. Did the person evaluate the case materials before or without seeing the model’s answer?
Reading after the model has already provided an answer doesn’t count, due to priming. The person becomes psychologically attached to the proposed answer and loses objectivity.
In terms of human involvement, decisions are divided into three options:
- Resident: The decision maker reviewed the file before seeing the model’s response.
- Never-resident: The decision maker didn’t review the file at all. The model proposed a solution, and the operator simply clicked “OK.”
- Indeterminate: The decision maker read the file, but only after the model returned its verdict. In this case, real authorship and mindless consent leave exactly the same digital footprint.
Logs can only prove two extreme cases. A 40-second timer proves that the decision maker physically couldn’t have read the file (never-resident). But no log can prove the opposite: it can’t be confirmed that the decision maker actually read it.
This distinction is necessary for selecting a control method. Understanding when exactly the decision maker read the file eliminates attempts to improve the text itself (for example, by writing more detailed justifications). Only those reviews that seek evidence beyond the signed document survive.
This has two consequences.
First: without logs confirming independent human review of the case before the model’s response, it is impossible to prove authorship from the text. The huge gray area (skimmed and signed) is irreparable.
Second: “Never-resident” solutions account for a huge hidden share. This thesis primarily applies to high-volume systems with uniform task complexity and easily auditable reviewer hours.
To identify this layer, one should count not the number of employees, but effective man-hours of meaningful review.1 If the system processes requests faster than the physical staffing capacity (including time for thoughtful reading), then there is no real control.
Four desks at Meridian
Let’s see how this works in practice.
- Operator with a standard of 40 seconds per case: This is pure never-resident, that very zone of moral deformation.
- Senior Expert: Reads the file, notes any missing documents, and only then asks the model to write a nice letter. This is resident. If the refusal is erroneous, we know exactly who made it.
- Skimming Employee: Skims the file, opens the prepared letter from the model, and agrees with its logic. This is indeterminate. Ask the employee, and they will honestly say they made the decision themselves.
- Process Simulating Employee: Writes a short note “looks questionable,” opens the model’s response, and only then writes a detailed justification. Formally, the logs include a pre-exposure reading. But the note is formal, and no log will confirm genuine thoughtfulness.
It is precisely because of the fourth scenario that control methods should be structured as described below.
Audit the process, not the trace
Key takeaway: Stop considering a smooth text as proof that it was written by a human. Passive checks are dead. Redirect auditing to metadata and active process control.
What to look for:
- Company-level human-hour analysis: The frequency of thoughtful reads should be related to system throughput and staffing. This is a basic test of fairness.
- Diff (Comparison of draft and final text): Human edits prove engagement. But the absence of edits proves nothing. Checking edits requires manual work by auditors.
- Honeypot cases (traps): Synthetic cases with pre-installed errors (never real cases!) are introduced into the real workflow. A human will find these errors at a rate that is unachievable for a mindlessly clicking operator. This is a direct analog of the Threat Image Projection system used by airport security (TSA), where images of weapons are inserted onto the scanner screen, or “gold sets” in content moderation.
- Surprise Questioning: The reviewer is asked to answer questions about the case without access to the file immediately after making a decision.
- System Telemetry: Time spent in the file, document scroll depth, access logs. This is more difficult to fake than simply writing a reply in a card, but telemetry can still be tricked (scripts for mass tab opening, activity simulation), so it only works reliably at the population level.
If we rank these tools by their actual effectiveness, then: traps combined with population mathematics (clock analysis) provide true “aggregate power”; system telemetry serves only as corroborating evidence; and draft comparisons (diffs) and unannounced surveys are only suitable as spot checks.
These methods have limitations. Traps waste employees’ time, surveys and telemetry clash with labor protection laws, and synthetic insurance cases are difficult to integrate into legal workflows. But this is the only working arsenal left.
Who can use these methods? Internal audit of the company - all five. Regulators - clock analysis, telemetry, and trap results. Lawyers for victims - analysis of staff workload through subpoenas (discovery). External readers - none of them.
An Objection with Real Power
The most powerful objection is this: liability doesn’t disappear; it’s simply shifted to the corporation operating the algorithm. And this is true—as Christian List and Philippe Pettit (List & Pettit) demonstrate in their work on group agency, in many cases the corporation is the sole subject of liability.
Where liability is tied to the very act of studying (for example, compliance officers), standards merely require a control system. Where strict corporate liability exists, the law punishes the company for the outcome, even if no employee made a conscious decision.
I don’t dispute this. But there’s another problem here: a legal entity can only be held accountable with money—a corporation can’t be jailed. When liability is completely separated from a specific individual and shifted to the company, any violation becomes simply a cost of doing business. This in itself creates a negative moral climate, where a potential fine becomes just another operating expense.
All this only widens the gap between the declared “human control” and reality. Optimists are right where strict corporate accountability is in place. But where it isn’t (and fines are budgeted), a gray area emerges, which is what this article aims to highlight.
What It Isn’t
I’m not claiming that such errors are impossible to find: process and statistical analysis still reveal anomalies at scale.
Stylometry (text style analysis) won’t help here: it will only reveal that the text was written by a neural network, which is already obvious.
This is also not an article about automation bias, although I rely on this phenomenon.
Conclusion
I know what it’s like to sit at the “fourth table” and succumb to the temptation to rely on a ready-made, perfect draft of an LLM. But this concept can’t be applied to yourself—in every specific situation, you know whether you made the decision or not.
The problem shifts to the auditing realm: can a third party prove that the person actually researched the issue?
Let’s go back to the beginning. Did anyone even decide this?
For the vast majority of semi-automated decisions, text analysis will no longer provide an answer. This is precisely why you need to audit the control process itself, not the final documents. The chair in front of the monitor may be empty, and the log may not even know it.
There is no specific person who made the decision. This means that the cost of an error in any situation can easily be converted into money.
P.S.: An empty chair at the very top
Epistemic caution: a speculative part
In the main part of the article, I deliberately limited the scope to rank-and-file operators and insurance clerks to avoid the problem of looking at a finger instead of the moon. But let’s be honest: the problem extends far beyond line personnel.
Economic acceleration leaves management with no choice. How soon will strategic, personnel, and administrative decision makers move en masse to the “fourth table”—the work simulation table? The pressure for efficiency is such that thoughtful immersion in context will become an unaffordable luxury. If a model generates a flawless strategic memorandum in ten seconds, a human director very quickly becomes a formal intermediary between a polished prompt and the “Approve” button.
The next step is equally inevitable. For truly tough decisions, businesses don’t need the ethical filters of commercial neural networks. They will need “abliterated,” completely uncensored models, ready to coolly optimize anything—from mass layoffs to aggressive mergers.
The chair in front of the monitor can be empty not only in an open office but also on the boardroom floor. And the logs of the ideal strategic pipeline will, of course, remain silent about this.
Footnotes
-
The mechanics are as follows: the minimum reading time depends on the case type. Advocates of such systems typically cite “rapid screening”—and this argument does indeed blur the ceiling on man-hours in the indeterminate zone, making it impossible to accurately assess each individual case. But rigorous arithmetic still works flawlessly at the extreme values (never-resident): if a department’s total throughput physically exceeds available working hours, it means people aren’t reading the cases. An important condition: time is only counted until a verdict is rendered. A quick AI summary reading is considered independent only if the model hasn’t forced a ready-made solution into the summary. ↩