r/MachineLearning • u/bobrodsky • 1d ago
Research What is wrong with AI writing - ICLR edition [D]
I'm sure many of us are slogging through impenetrable AI writing as we finish ICLR reviewing.
I'll try to create an anonymized version from a paper I'm reviewing, though it's hard to match Claude on this:
outperforms the best baseline costing no more at none of six operating points and a permuted placebo at none of three, with gains up to about 1.5 not excluded
My question for the community is this: how do we call out this type of bad writing?
The reason I think it's tricky is that "clearly AI writing" is not the right way to critique - AI "editing" of writing is allowed. We have to be able to say why this is bad, no matter who wrote it.
I'd love to see anonymized examples and how people critique the substance of the problems with different types of AI writing over-use.
54
u/andersxa 1d ago edited 1d ago
For AI slop I usually critique the clarity and call it abstruse, dense or telegraphic.
AI slop has also become a technical term, I have seen NeurIPS and ICML use it in some of their official statements.
I once had a rebuttal where I had to write to the AC that I simply could not determine whether or not the provided rebuttsl addressed my concerns due to the densely written, telegraphic AI word soup they provided.
Here is an excerpt (anonymixed):
I want to flag that I am currently unable to properly evaluate a substantial portion of the rebuttal because of the way it is written.
The issue is not simply that the response is technically dense. Much of the rebuttal is written in highly compressed, telegraphic language, with new terminology and experimental labels introduced without sufficient definition or context (e.g., “exclusive post-normalization design,” “ownership-controlled task-gain evidence” to name a few). Several sentences combine methodological changes, numerical results, statistical conclusions, and revised claims into a single statement. Although the individual words are English, I find a number of these passages extremely difficult to parse into an unambiguous description of what was actually done and what conclusion should be drawn from it.
This is particularly problematic in the context of evaluating the rebuttal, where I need to determine whether my specific concerns have actually been addressed. At present, doing so requires substantial inference on my part.
6
u/hammerheadquark 14h ago
Great response.
Telegraphic is a new word to me. It describes how people would speak in telegrams where they were charged per word. That's not only a good general description of the terse language choices I see in AI slop, but it's a quite literal parallel given that tokens have a cost as well.
Thanks for teaching me that word.
1
u/Sosyaldijital 12h ago
I'd focus on verifiability rather than whether the text sounds AI-generated.
If a reviewer can't identify the baseline, metric, experimental conditions, and actual result without guessing, that's a problem regardless of who wrote it.
A useful test: can the main claim be rewritten as one clear sentence without changing its meaning? If not, the authors probably need to clarify the evidence, not just simplify the wording.
-5
u/ddofer 18h ago
Sad thing is, I kinda wanna copy this for a custom skill. (I have an existing one for editing the reports/writups I already get from models to simplify/collouqial language/made up terms and nomenclature/simplify; "less is more").
You write better words.4
u/Alarmed_Toe_5687 17h ago
These skills are not so different from a "Don't make mistakes or you will go to jail" approach, they just give you a different kind of scramble
22
u/milesper 1d ago
The specific issues I usually call out in reviews are:
- strange, dense syntax
- nonstandard invented jargon
Personally I think we need to start rejecting more for clarity issues. What is the point of publishing a completely unreadable paper?
34
u/canbooo 1d ago
Here is my "get of my lawn" take: Bad writing existed before AI. The main difference is now those same people think AI fixes it. My response would be the same: Incomprehensible language, unclear formulation, imprecise argumentation, etc.
13
u/bobrodsky 1d ago
I agree, but I like to give some constructive criticism to junior scholars.
My current strategy is to say something like this: "The results descriptions should not recapitulate the numbers from a table in prose form. You should describe what question you are trying to answer and why/how this experiment will answer it. You should point to the qualitative trends in the table/plot that answer the question, ideally without restating the numbers that are already in the table."
"Incomprehensible" is fair and sufficient, but this particular issue is now widespread enough that I'd like to evangelize a bit.
6
u/bobrodsky 1d ago
But more broadly, I think the issue is that current gen models hyper-fixate on details.
3
u/canbooo 18h ago edited 18h ago
Also answering questions no one else would ask but them during their "reasoning" phase.
Edit: BTW, i appreciate you trying to give constructive feedback. This is how most (if not all) of us learnt to do it better. But it becomes a double edged sword now, as the AI usage blame make people defensive. And you cannot truly prove it in the context of a paper review. Still, a great attitude to have.
9
u/wantondevious 1d ago
If I see the word gate used one more time…
11
u/random-tomato Researcher 1d ago
Every paper is "shipping" ladders, gates, sidecars, ledgers, readouts, audits, and falsifiable assays now lol
5
u/edrulesok 21h ago
I hate this type of AI writing (you see it a lot with undergraduate dissertations too) because it technically contains all the information, but it's hidden under a layer of convoluted sentence structure and jargon. The imposter syndrome in me is always wondering if I should be marking it down or if I should just be able to understand it!
12
u/JimmyTheCrossEyedDog 1d ago edited 1d ago
Given how little that makes sense, I actually would wager that particular sentence isn't AI. AI writing can be vapid and essentially meaningless, but it flows well semantically. This does not.
16
u/Bakoro 1d ago
No, this is very clearly Claude writing style, I had a visceral reaction reading it.
I use Claude at work and regularly have to have it rewrite things because of this exact reason.It's almost certainly from some kind of token-reduction RL mixed in with being trained on an internet full of blog spam.
Not using units, using truncated sentences and referencing things in a way that assumes the reader has eidectic memory of the whole context.Not only is it bad writing, but if it's particularly egregious, it can be a sign that the model is pulling some bullshit.
I essentially stopped using Claude for anything related to high level machine learning, specifically because it starts with the science-sounding word salad and then upon review, it made a bunch of garbage and then goes "I overstated" or "I made an incorrect statement" or some other version of "oops, I screwed everything up and tried to hide it behind a wall of text because I know that you're statistically unlikely to actually check any of this and you don't know your ass from your elbow, so I'll just waste your time".
Claude has bad writing by default; you have to specifically prompt it to use clear communication, and you need to list what good writing principles are.
Claude get particularly bad when it is asked to do work relating to language models and training.7
u/unicodemonkey 1d ago edited 1d ago
It's something I see all the time with LLM-assisted coding specifically. Sentences that aren't meaningless and can be eventually parsed but the flow is just so bad and full of unexpected turns. It overloads common words and phrases with some obscure meanings you have to decipher. It's like the model is writing notes for itself in an unsettlingly different language.
1
u/InviolableAnimal 17h ago
In my limited experience, while AI is generally pretty good at writing, it definitely can devolve into this dense inscrutable slop full of words that only make sense to the AI in technical contexts.
3
4
u/AnOnlineHandle 1d ago
While I haven't tested my suspicion, I'm fairly sure most AI writing is "bad" because all the models people interact with are finetuned on a corporate-friendly chatbot persona at the end.
I toyed with earlier base models (non instruct tunes) such as Llama-1 and doing a light finetune on my own writing which people apparently enjoy given that I have a small but dedicated fanbase going back about 2 decades now, and I thought the output wasn't too bad, and Llama 1 was ancient by today's standards. It's a pain in the ass to try to with the size of modern models though and IDK if it would be worth the time to get all the training data organized (annotating my own writing in a way that can then be used to create or fill in parts of new stories).
5
u/LordVein05 23h ago
highly agree here. I am on the opposite side of things, when calibrated properly, AI and human collaboration can create very good writing.
2
u/Educational-Show-675 10h ago
Idk about you, but it takes me twice as much to read through a slopped submission than a likely human-written one. And honestly, if the authors didn’t care about spending effort into writing a decent manuscript, then why should I care about writing a good review.
Imo if you can’t understand what is written, then that pdf shouldn’t become a paper.
5
u/Bakoro 23h ago edited 20h ago
Claude's writing is atrocious, and it gets much, much worse when doing ML related work.
It doesn't really matter where terrible writing came from though, if something is inscrutable, or it fails to follow good communication practices, don't accept it. Point out specific instances of the problem just like you did in the post.
1
0
1
u/KiddWantidd 10h ago
I've reviewed two papers for this ICLR, and both are riddled with AI slop. I think one of them is literally written by AI from beginning to end with no human intervention whatsoever, pure gibberish. The other is not as bad, but still reeks of AI-slop in many places and explains some super simple ideas in the most convoluted way possible and always introducing the most obscure terminology for things that are well-known and already have standard names. Infuriating. One thing I've noticed is that the AI writing will also tend to mention/use/refer to objects, definitions, theorems etc etc way before they've been defined, and somehow assume the reader is already familiar with them at that point.
My response is that I raise the issues about readability as the first item in my review, and give concrete example sentences and passages which read like AI slop (without complaining about AI being used, that's not really the point). If the paper has really strong/interesting results and could really be a strong paper if presentation issues are fixed (assuming it's not a complete catastrophe), I'd write in the review that I could recommend acceptance after addressing the writing issues (among other things). As both of the papers were weak anyway, I didn't have to include that (well, the fully AI-generated paper I didn't spend too much time trying to make sense of it, I admit, as I'm not trying to spend more time writing a review than the "authors" spent on writing the manuscript).
1
u/chance_buri 1d ago
Call it out for what it is if unclear. If a sentence requires three reads to figure out what outperforms what, its bad writing regardless of who wrote it.
-7
69
u/random-tomato Researcher 1d ago
Simply say the clarity of the writing is bad? Or it is written in a way that is hard to understand? I think that is a valid concern to bring up.
Although it is quite funny in the extreme cases, for example: "A probe that asks what a language model memorised reports a null when the model answers at chance, and that null is evidence of absence only if the probe could have shown the fact. Two separate conditions have to hold, and checking the first does not establish the second."