69%. That's the AI-generated probability score Pangram reportedly assigned to at least one paper that NeurIPS 2026 desk-rejected for alleged AI-writing policy violations - and according to researcher S. Berezin in a LinkedIn post, the detector was never validated against the actual style of academic machine learning writing before being used to make those decisions.
The Methodological Problem
A text classifier - which is what AI detectors are - needs to be tested on samples that match what it will evaluate in real use. If Pangram was built to distinguish between casual AI-generated text and ordinary human prose, it may not perform accurately on the specific style of academic ML papers: dense technical language, heavy use of passive voice, precise citation patterns, and writing from authors who often use AI tools for editing, translation, or grammar assistance without generating content wholesale.
This is not a hypothetical concern. Studies from 2023 and 2024 consistently found that AI detectors misclassify non-native English speakers' writing as AI-generated at meaningfully higher rates than native speakers. Researchers from countries where English is a second language are disproportionately affected - and NeurIPS draws submissions from around the world.
Who Gets Hurt When Detectors Are Wrong
Conferences, journals, and universities are under real pressure to enforce AI writing policies, and that pressure is legitimate. AI can be used to fabricate citations, generate boilerplate methods sections, or skip the thinking that peer review is supposed to evaluate. Having a policy makes sense.
What doesn't make sense is using an unvalidated commercial tool's probability score as a hard cutoff for desk rejection. Desk-rejecting a paper before any human reviewer reads it - based on a 69% confidence score from a system that wasn't tested on the relevant population of writers - is a methodological failure. The same rigor researchers are expected to apply to their own experiments should apply to the tools used to judge them.
There's also a transparency problem. Pangram is a private company. Using a proprietary, black-box detector without disclosing its methodology means researchers can't audit the system that's evaluating their work.
The smarter approach would be using AI detection as a flag for human review, not as grounds for automatic rejection. Scores above a threshold should trigger closer editorial scrutiny: asking authors to explain their AI use, reviewing flagged sections, or requiring explicit tool-use declarations. That's slower. It's also more defensible.
AI detection tools aren't reliable enough to carry the weight of consequential, career-affecting decisions on their own. That was true two years ago, and it's still true in 2026. The researchers at NeurIPS know this better than anyone - that's what makes this hard to excuse.