Introduction — 99% reliability, and a 0 out of 5
Emma is a teacher. She is also a student retraining for a new career. In a written expression exam, she was given 0 out of 5. The reason: Compilatio’s artificial intelligence detector had found 99% AI in her paper.
Her account is public, posted on Trustpilot on 14 June 2023, rated one star.
The same month Emma was writing that, OpenAI — the company that makes ChatGPT — was preparing to shut down its own detector of AI-generated text. The tool had launched in January 2023. It was withdrawn on 20 July that year, with this line added at the top of the original announcement: no longer available, “due to its low rate of accuracy”.
Between the two, there is an industry. Software sold to universities, schools, publishing houses and recruitment departments, promising to say whether a text was written by a machine. With figures attached: 94 to 99% reliability at Compilatio, one false positive in ten thousand at Pangram.
Those figures deserve a close look, because they are not used to separate tools in a product comparison. They are used to put a zero on a paper, to summon a student before a disciplinary board, and in France for the past few days, to publicly accuse a prize-winning novelist.
Summary of the test
The promise: an AI text detector can reliably say whether a text comes from a machine or from a human.
Our verdict: overstated.
The announced figures are not lies. They are true in the easiest case — a text written entirely by an AI, untouched, produced from a simple question — and that case is almost never the one in which the tool is used to sanction somebody.
Three findings carry the whole article:
- On the real case, Compilatio itself refuses to give a figure. Its official page explains that for a mixed document, blending human and generated passages, “a single figure” would be “poorly representative of reality”, and recommends treating the analysis as “a decision support tool rather than an isolated performance score”.
- The promise inverts. According to the most recent study, a human text simply proofread by an AI is flagged in 38 to 80% of cases, while a text written entirely by AI then run through a humaniser goes unnoticed more than 96% of the time. Honest use is more dangerous than cheating.
- French is measured nowhere publicly. Compilatio says it tested 24 languages including French, but publishes only an overall range. Pangram publishes rates by language — Spanish, Japanese, Arabic — and French is not among them.
What we checked
Le Recul did not run this software on a corpus of its own. So we say exactly what this article rests on, and what it does not claim to be.
We read the vendors’ official pages, where the promise is written: Compilatio’s page on its detector’s reliability, updated on 22 September 2026, and Pangram’s article on false positives, published on 15 September 2026. These are their figures, in their words.
We read the only two serious independent studies published to date, and we present both, including where they contradict each other.
We read user reviews on Trustpilot, noting the overall ratings and quoting dated, signed accounts. These are testimony, not measurements: we give them for what they are.
We read the official documentation of a French university that refused to install the detection module, and the coverage of two recent French cases.
What this article does not contain: no false positive rate measured by us, no numerical comparison between tools carried out by us. When a figure appears here, it comes from a vendor, a study or a user, and its source is named.
What the three vendors actually promise
| Vendor | Announced figure | What it was measured on |
|---|---|---|
| Compilatio | 94 to 99% reliability on academic content; under 1% false positives | 7,400 texts in 24 languages (3,700 human, 3,700 AI), AI texts obtained from "simple questions without specific instructions on writing style" |
| Pangram | around 1 false positive in 10,000 | corpus by genre: essays 0.004%, creative writing 0.009%, medical articles 0.000%, poems 0.05%, recipes 0.23% |
| GPTZero | "the no. 1 detector for students", 10 million users | no comparable false positive rate highlighted; Pangram estimates it at around 1 in 100 |
Two details are worth pausing on.
The first is a Compilatio sentence almost nobody reads, because it sits in the middle of a help page: the texts used to measure the engine were generated by asking the AI “simple questions without specific instructions on writing style”. That is the easiest case there is. No student set on cheating proceeds that way: they give a tone, a plan, instructions, they copy and alter. The 94 to 99% figure therefore describes the tool’s performance against a cheat who makes no effort at all.
The second is that Compilatio explicitly separates two things the public conflates: the “detection engine”, which labels homogeneous extracts, and “the analysis”, which has to find AI passages in a mixed document. The 94 to 99% concerns the first. For the second — that is to say, for a real assignment — the vendor writes that a single figure would be “poorly representative of reality”.
And it is the second that is used to sanction people.
OpenAI built a detector for its own texts, then shut it down
On 31 January 2023, OpenAI put online a classifier meant to tell human texts from AI-generated ones. The company was in the best imaginable position: it knows its own models from the inside.
The results OpenAI published on a set of difficult English texts: the classifier correctly identified 26% of AI-written texts, and wrongly labelled as AI 9% of human-written ones.
On 20 July 2023, the tool was withdrawn. The note added to the original announcement has stayed online since: the classifier is no longer available “due to its low rate of accuracy”.
This point is not incidental. It sets the order of magnitude of the problem. If the maker of the model you need to detect cannot do it satisfactorily, you should at minimum pause in front of a competitor announcing 99%.
That episode also explains a persistent legend: OpenAI’s classifier, like others after it, flagged famous and obviously human texts. ActuaLitte recalled at the end of December 2025 that it had classified passages of Shakespeare and the United States Declaration of Independence as probably machine-generated.
The only two serious studies do not say the same thing
This is the point affiliate articles skip over: independent work exists, but it does not agree.
| Study | Corpus | What it concludes |
|---|---|---|
| NBER / Chicago Booth Jabarian & Imas, August 2025 | 1,992 passages, 6 common genres (press, blogs, reviews, novels, CVs) | Pangram does remarkably well: zero false positives on long passages, under 1% on short ones. GPTZero loses most of its capability against humanisers |
| University of Notre Dame ACM AILS '26, 20 August 2026 | 642 scientific abstracts, 4 disciplines, two periods (2013-2015 and 2023-2025) | On pre-AI texts, no false positives. But on text simply proofread by AI, 38 to 80% flagging, and more than 96% evasion after a humaniser |
Both studies are right, on different questions. The first asks: can this detector recognise a machine text in a varied corpus? The second asks: what happens when a human and an AI have both worked on the same text?
The first question interests a software vendor. The second is the question of everyone who is subjected to the tool.
One result reconciles them and deserves stating, because it runs against what you would expect: on texts published between 2013 and 2015, therefore strictly before consumer generative AI, the Notre Dame team records 0% false positives, for Pangram as for GPTZero. On that precise point — an old, long text never touched by a machine — the detectors do not get it wrong.
The trouble starts as soon as the text is recent, short, formal, or edited.
The figure that inverts the promise
Here is the table that should appear in every manual, and appears in none. It comes from the Notre Dame study, at the standard threshold of 0.50.
| Situation | Flagging |
|---|---|
| Human text published before AI (2013-2015) | 0% |
| Recent human text, unedited (2023-2025) | 8.9% (GPTZero) to 14.9% (Pangram) |
| Human text proofread and corrected by an AI | 38% to 80% |
| 100% AI text run through a humaniser | under 4% detected |
Take two students.
The first writes the assignment themselves, then asks an AI to fix their phrasing — which most sets of rules explicitly allow, and which is recommended to every student working in a second language. They have between 38 and 80% chance of being flagged.
The second writes nothing, has everything generated, then runs the text through a humaniser sold for a few euros. They have under 4% chance of being caught.
The researchers’ conclusion is written in black and white in their abstract: a detector score must not serve as standalone proof of cheating.
One detail that matters for students of literature, law or history: the study notes that humanities texts are flagged far more often than hard science ones, with a statistically very solid difference. The more your discipline writes in constructed prose, the more the machine suspects you.
What users report
Online reviews are not measurements. They say something else: what people experience, and how often they come to complain about it. Two overall ratings, recorded on 23 September 2026, and already eloquent.
| Tool | Trustpilot rating | Reviews |
|---|---|---|
| Compilatio | 4.0 / 5 | 74 |
| GPTZero | 2.2 / 5 | 140 |
The gap is partly explained by the audience: Compilatio is rated by teachers, who appreciate the time saved, as much as by students. Its reviews are sharply polarised. GPTZero is mostly rated by people who come to check their own text.
The accounts resemble each other from one tool to the next.
On Compilatio, Matteo Belloli writes on 18 February 2026, one star: “A text written entirely by a human can go as high as 88%” AI detected, while “a text generated entirely by AI can fall to 0%”. He adds: “I have already been accused of writing 47% of an assignment I did entirely myself.” And Emma, quoted at the start, reports a 0 out of 5 after a reading of 99%.
On GPTZero, Patchyca writes on 24 May 2026, one star: “After several tests on texts written entirely by humans, several results showed a score of 100% AI.” Veritas Roi, on 1 April 2026: “Completely human texts are rated 100% ai even though every other detector puts them at 90%+ human.” Lyonel Shearer, on 21 May 2025: “GPTZero was the only detector to say the texts had been written entirely by AI. (which was not the case).”
These accounts do not prove a rate. They show two useful things. First, that users spontaneously run the only experiment that counts: submitting a text they know with certainty is human. Second, that the gaps reported are not a few points but all-or-nothing — 88%, 99%, 100% on their own writing.
A French university refused the module
This is the least known fact in the whole file, and the most telling.
The University of Lorraine makes Compilatio available to its teaching staff for plagiarism detection. But on the official page of its digital services, you read this, about AI detection:
“Cheating detection tools based on identifying AI-generated content are not reliable. They produce many false positives and are easily circumvented.”
And the consequence, written straight after: “The University of Lorraine has chosen not to adopt the Magister+ solution.”
Magister+ is precisely the Compilatio offering that gives access to the AI detector. A French public university, relying on recommendations published in June 2025, examined the product and decided not to buy it, publicly explaining why.
It is not alone in pulling back. ActuaLitte reported at the end of 2025 that the Australian Catholic University had suspended its use of Turnitin after students were wrongly accused on the strength of automated scores alone. The same article relates the case of a Paris sixth-former accused of using AI during a philosophy exam on computer: the administration eventually acknowledged an error.
We had already seen that caution take hold elsewhere in education, when Norway banned AI in primary schools, and when Anthropic launched an offering aimed at teachers. Detection, for its part, is advancing without that debate.
No figure for French
Here is what is missing, and nobody says it.
Compilatio states it measured its engine on texts in 24 languages, and French is indeed on the list, alongside German, Arabic, Croatian, modern Greek and Ukrainian. But the vendor publishes only an overall range — 94 to 99% — while itself specifying that it varies “according to document length, language and the nature of the content”. No per-language figure.
Pangram, by contrast, publishes rates by language. Here they are, as they appear on its own site on 15 September 2026.
| Corpus | Announced false positives |
|---|---|
| Product reviews, Spanish | 0.008% |
| Product reviews, Japanese | 0.015% |
| Wikipedia, Spanish | 0.07% |
| Wikipedia, Japanese | 0.02% |
| Wikipedia, Arabic | 0.08% |
| French | absent from the table |
The vendor takes the trouble to measure Spanish, Japanese and Arabic, and to publish the results. French is not there.
This is no imputation of bad faith: nothing indicates Pangram errs more in French. But it means that when a Pangram score is held against a French text — which happened this week to a novelist — nobody can say what error rate that score corresponds to. Not the accuser, not the accused, not the software vendor.
The two independent studies do not fill the gap: Chicago Booth’s and Notre Dame’s both cover English corpora.
How these tools decide, and why they get it wrong
A detector does not read a text. It knows nothing of its author, its date, its context. It measures statistical regularities: the predictability of vocabulary, the regularity of syntax, sentence length, the density of certain words.
Compilatio puts it plainly on its help page: detection “relies on recognising stylistic characteristics typical of texts written by an AI”, and “it can happen that a human has a style similar to that of an artificial intelligence”.
Everything else follows from that. The profiles that write in a regular, structured way, without rough edges, are flagged more often. In concrete terms, those are:
- people writing in a language that is not their own, and applying the rules they learned scrupulously;
- pupils and students who have been taught precisely that structure — introduction, development, conclusion, logical connectors;
- technical, legal, administrative or academic texts, whose form is constrained;
- and short texts, where there is too little material to decide.
The Notre Dame study confirms it by measurement: high scores track the density of long words and academic vocabulary, not the author’s intent.
In other words, these tools do not detect fraud. They detect a style. And the style they punish is the one pupils are asked to acquire.
The prize-winning novel affair
On 21 September 2026, Thelyson Orelien receives the Fnac novel prize for C’etait ca ou mourir, published by Boreal in Montreal in March 2026 then by Grasset in France on 19 August. The novel is in contention for the Goncourt, the Medicis, the Femina, the Renaudot and the Decembre.
That same evening, an anonymous X account accuses him of having had his book written “almost entirely” by an artificial intelligence. The evidence produced: Pangram scores close to 100% on several extracts.
The author denies it to Liberation and recalls that he began the manuscript in autumn 2017 and finished a first version in 2019 — several years before generative AI reached the public. The publisher Grasset gives him its “absolute confidence” and denounces a smear campaign.
We obviously have no way of saying what this novel does or does not owe to a machine, and that is not the subject of this article. What the affair does show comes down to three lines:
- a detector score is now enough to launch a public accusation against an author;
- the author’s defence rests on exactly the one protocol researchers consider conclusive — the date of the text;
- and the software invoked has published no error rate for the language the book is written in.
The problem is not specific to publishing. We had already met it in another form in the affair of the books destroyed by Anthropic: as soon as AI enters the book chain, the burden of proof shifts, and it is the human who has to demonstrate that they wrote.
What French courts already say
The courts have not waited.
An order of the Paris administrative court handed down on 12 February 2026 concerns a law student at University Paris 1 Pantheon-Sorbonne, suspected of having written her internship dissertation with an AI. The disciplinary panel had declined to sanction her in January. Asked to rule afterwards, the court held that you cannot sanction the use of AI if the rules governing that use were not defined beforehand and brought to students’ attention.
In Toulouse, a student excluded from her faculty for five years over AI cheating in an exam obtained the suspension of her sanction.
The principle recalled is old and owes nothing to computing: you only sanction on the basis of a rule known in advance. A software score does not create that rule, and does not replace the demonstration.
Compilatio in fact says the same thing as the judge, in its own documentation: its tools “provide indications about suspicious passages”, and “it always falls to the marker to interpret this information to confirm or attribute potential fraud”. The vendor even recommends, in case of doubt, “conducting a deeper examination of the student’s knowledge”.
The sales promise says 99%. The manual says: talk to the student.
Humanisers, by contrast, keep their promise
You have to look at the other half of the market, because it is vast: the tools that promise to make an AI text “undetectable”.
This is the most embarrassing part of the file. Their promise is kept. The Notre Dame study measures that after passing through an off-the-shelf humaniser, fewer than 4% of AI-generated texts are still flagged. The Chicago Booth study reaches the same finding for GPTZero, whose failure rate climbs to around 50% and beyond against these tools — while noting that Pangram holds up markedly better.
The University of Lorraine says it in the same terms when justifying its refusal: these detectors “are easily circumvented”.
The result is a system that penalises exactly the wrong people. The one who does not know humanisers exist — often the most honest, sometimes simply the least resourceful — takes the risk. The one who organises their cheating gets through.
We will obviously give no instructions. The only point of this finding is that it ruins the sales argument: a control system that is circumvented in three clicks for a few euros protects nothing. What it mainly produces is summonses.
What it costs
| Tool | Free | Paid |
|---|---|---|
| GPTZero | 10,000 words a month, no time limit | roughly $8 to $25 a month; $23.99 for 300,000 words, $45.99 for 500,000 |
| Compilatio Magister+ | no | subscription sold to institutions; claims more than 1,100 institutions in more than 50 countries |
| Pangram | no | paid, mainly sold to institutions and platforms |
A point on the business model, which illuminates the rest: in Compilatio’s case, it is not the person being checked who pays. The institution buys, the teacher uses, the student bears it. None of the three has the same interest in the error rate being precisely known.
Our Le Recul verdict
Promise overstated.
We are not saying these tools are useless. Faced with a long, old or entirely generated text with no editing, they work, and both independent studies confirm it. Pangram in particular is the only one to have accepted an external evaluation and come out of it well.
We are saying the figure displayed does not describe the situation in which it is used.
It is measured on homogeneous texts, when reality is mixed. It is measured on AI texts produced from simple questions, when a cheat gives instructions. It is published as an average over 24 languages, when the decision concerns a paper in French. And it is held against a person, when the vendor itself writes that it is an “indication” and that “no AI detector can be 100% reliable”.
The strongest argument against the promise is in fact supplied by Compilatio, in this sentence on its own page: “The reliability rates communicated by AI detectors are not comparable!” If different vendors’ figures are not comparable with each other, they are not opposable to anyone either.
And there remains the fact nothing erases: the company that makes ChatGPT tried to detect ChatGPT’s texts, and preferred to shut its tool down.
If a detector accuses you wrongly
What you can put forward, with verifiable sources:
- The date. It is the only argument researchers consider conclusive: on texts predating generative AI, the Notre Dame study records zero false positives. Timestamped drafts, version history, emails, handwritten notes, a dated deposit: anything proving a chronology is worth more than a counter-score.
- The vendor’s manual. Compilatio itself writes that its tools “provide indications about suspicious passages”, that “it always falls to the marker to interpret this information”, and that “no AI detector can be 100% reliable”.
- The prior rule. The Paris administrative court held, on 12 February 2026, that the use of AI cannot be sanctioned without a rule defined and brought to students’ attention before the facts.
- A public university’s position. The University of Lorraine refused Magister+, writing that these tools “are not reliable”, produce “many false positives” and “are easily circumvented”.
- The conversation. It is the vendor’s own recommendation as much as common sense: an examination of the student’s knowledge of the suspect passages settles what no percentage will settle.
Frequently asked questions
Which AI text detector is the most reliable? On the published data, Pangram is the only one to have been evaluated by an outside team — the Chicago Booth researchers — and to come out with false positives close to zero on long passages. But that evaluation covers English texts, and Pangram publishes no rate for French. There is therefore no verifiable answer for a text in French.
Are AI text detectors 100% reliable? No, and the vendor says so itself: Compilatio’s help page states that “no AI detector can be 100% reliable”.
Can a detector flag a text I wrote myself? Yes. The University of Notre Dame study measures between 8.9% and 14.9% flagging on recent unedited human texts, and 38% to 80% as soon as an AI has been used to proofread or correct. Several users report scores of 88%, 99% and 100% on their own writing on Trustpilot.
How does Compilatio’s AI detector work? It rests on a language model trained to recognise, according to Compilatio, “stylistic characteristics typical of texts written by an AI”. The vendor specifies on the same page that “it can happen that a human has a style similar to that of an artificial intelligence”.
Do universities use an AI detector? Many do, but not all. The University of Lorraine officially refused the Magister+ solution, which gives access to the detector, judging it unreliable on its digital services page.
Can you be sanctioned on a score alone? Compilatio’s documentation says no: it is for the marker to interpret. The Paris administrative court ruled on 12 February 2026 that a sanction presupposes a rule defined beforehand, and a Toulouse student excluded for five years obtained the suspension of her sanction.
Does a detector catch a text run through a humaniser? Rarely. The Notre Dame study measures fewer than 4% of texts still flagged after passing through an off-the-shelf humaniser, and the University of Lorraine notes that these tools “are easily circumvented”.
Do detectors work as well in French as in English? Nobody publicly knows. Compilatio says it tested 24 languages including French, but publishes only an overall range. Pangram publishes rates for Spanish, Japanese and Arabic, but not for French. Both independent studies cover English.
Why is my academic text flagged more than others? Because detectors measure the regularity of style. The Notre Dame study observes that high scores track the density of long words and academic vocabulary, and that humanities texts are flagged far more often than hard science ones.
Is there a free AI text detector? Yes, GPTZero offers 10,000 words a month free, and several competitors allow a limited paste. Being free changes nothing about the underlying problem: the score obtained remains an indication, whatever the plan.
Conclusion — a figure is not proof
There is a moment, in this file, when everything turns over.
It is not when you read the user accounts, nor even when you discover that a French public university refused to buy the module. It is when you read the vendor’s own help page and find there, written by them, every reservation you were looking for: no detector is 100% reliable, the rates are not comparable with each other, a human can write like a machine, and for a real document there is no representative figure.
All of it is public. All of it is a few clicks away. And all of it disappears the moment a score appears on screen in front of a marker in a hurry.
The promise is therefore not a lie: it is a true promise moved out of its context. It describes a laboratory and is used in a disciplinary board.
If you teach, the most useful thing is not to pick the best detector, it is to define your rules in advance and to check what the pupil knows. If you study or if you write, keep your dated drafts: to this day, the date remains the one thing no statistical model can contradict.
To know which tool does what, and on which measurements, our AI model comparator gives the figures source by source. And for another numbered promise put through the same test, read what the promise of €10,000 a month thanks to AI is really worth.