One more generative AI rant for the pile

(this one's about summarising text)

LLM chatbots – that is AI, in the same sense that we could just start saying "doctors" to refer specifically to orthopaedic wrist surgeons if we collectively decided to – 

LLM chatbots continue to slosh about the world. I used to try them out intermittently to see if they were any good.

My contact with the technology is only incidental these days. To wit:

• If you google old phrases and terminology in English, a LLM chatbot will still confidently weigh in with completely spurious "definitions" because they're not well-represented in the training data.
• If you google modern bits of even slightly less-discussed technical knowledge like "does a Kickstarter project video appear on the prelaunch page", a LLM will still confidently tell you the opposite of the truth.
• If you need customer support or anything that even looks like customer support, there is an extra quarter-hour minimum of wasted bot effort before you can get it.

Nothing I've seen has suggested the technology has fundamentally changed.

 

He can't be wrong, he writes so confidently.

 

In the previous edition of discussing the emperor having no clothes, I mentioned  

[Wikipedia] editors pointed out that the LLM summaries generally ranged from 'bad' to 'worthless' by Wiki standards: they didn't meet the tone requirements, left out key details or included incidental ones, injected "information" that wasn't in the article, and so on

and 

bureaucratic wonks note that genAI can't summarise text. It shortens it and fills in the gaps with median seems-plausible-to-me pablum. The kind you get when you average out everything anyone has ever written on the internet.

I recently saw an AI booster shuffle their position back to "at least it's good for summarising, it's going to completely replace human effort there". With that motivation, let's drill down a bit.

In (a) summary

Let's not bury the lede. LLM chatbots can't produce good summaries. Sometimes by chance yes, but not reliably. Summarising, like everything, is a skill-based task, and of the various capabilities required to do it well, LLMs lack four of the most important.

1. LLMs won't reliably retain important structure or order in which information is presented. They will just haphazardly obliterate implicit linkages. They will even occasionally discard explicit structures, as when the text itself points out that C follows from A and B, and therefore D.

2. LLMs can't identify the most important information in a text (a necessary first step to preserving it in the summary). In a good summary, certain content "should" be retained, certain content compressed, and the remaining content discarded. Vital information generally isn't identified within the text in a way that's detectable without broader context, language skills, and understanding of the world. Even when it is, e.g., in texts where repetition of a word corresponds directly to importance, or phrases like "this is vital information" are always appended, LLMs still aren't guaranteed to retain important details! And the same applies to cutting out unimportant information.

3. LLMs can't stick to the source text, that is, the content they're meant to be summarising. Because they just generate text (by predicting which bits of text should come next, based on an enormous model of which bits tend to come after which bits, hence 'language model'), there's no internal representation of Things 'In' The Language Model versus Things 'In' The Text To Be Summarised, and no impetus to perform computational operations that keep them separate where appropriate. All of which is to say that as well as not including things that should be in a summary, an LLM will readily include things that shouldn't be. Oops

3(corollary). That includes things that aren't true. Oops(corollary)

4. LLMs will sometimes just negate statements for no clear reason. When processing text, e.g. when directed to "summarise", they'll turn a claim into the opposite claim. I think what's going on here is that a statement and its negation are syntactically and semantically similar, even though their meanings are devastatingly dissimilar. Too bad LLM technology doesn't get meanings involved, instead just taking a probabilistic walk through a model of language features like, oh I don't know, syntax and semantics!

Note what these four crucial capabilities have in common. It's the reason why LLMs can't do them. That's right, they require understanding to do properly.

Or if not understanding, then at least computational models of understanding, like formal reasoning over symbolically-encoded domain knowledge including useful axioms. I mention this because classic AI systems (planners, searchers, problem solvers, etc) can do just that, in their various limited ways. They symbolically represent domain information and then perform operations on those symbols which can then give something potentially useful back once related back to domain information.

And those systems are limited, yes, but LLMs don't do 'understanding' at all. As far as I can tell, on the back of a postgrad compsci degree and a few days spent reading and partly understanding the computational basis, this is a fundamental limitation of the technology. One which can't just be fixed, but which would need a whole new (at most LLM-inspired) technology to overcome. For exactly the same reason why AI "hallucinations" can't be fixed.

 

Presummary (a digression)

This technical basis of how LLMs work also explains something else. These chatbots are particularly bad at "summarising" documents which contain surprising content.

By surprising content, I mean...

➡️ Statements seeming to defy common wisdom. Things that are the opposite of statements well-represented in the training data. When X is generally true of a field, but your text describes how ¬X is true of some narrow subfield or specific context, you'll see an LLM "summarise" X into ¬X more frequently.

➡️ Deliberate omissions of things that are usually in correlating training data documents. If your text looks like a text of type blarg, and blarg texts in the training data typically report on X, but you have not reported on X for your own reasons, an LLM is likely to just make something up about X while "summarising".

➡️ Unusual pairings of form and content. Performance degrades the more you ask an LLM to do something novel.

➡️ Context-sensitive language like metonyms and homographs. When X is a big important noun well-represented in the training data and X refers to something else in the text, you'll see an LLM (appear to) get confused by the statements about X its produces for the "summary".

➡️ Nontextual information content. The LM stands for language model. If you have a report that includes and discusses images and diagrams, a chatbot might be able to stop and parse those, and then incorporate its own description of the image as part of the text to be summarised, and maybe even put images back in the summary. But you'll nonetheless end up with a worse output.

 

In summary (but for real)

So LLM chatbots can't be (consistently, reliably, etc) good at summarising.

Of course people who don't know what a good summary is might not notice this; likewise people who possess the skill but don't carefully check the job they told it to do.

(I would argue that in either case, if the task was worth doing to begin with, you should prefer the task not getting done to having no idea whether your document is a good, adequate, or terrible summary)

Anyway this is why you may have seen people who do know what a good summary is point out that LLMs actually "shorten" text rather than "summarise" it. I'm not certain but I think the first time I saw this was in one of Bjarnason's essays.

The sentiment "this technology sure can't do [thing I am skilled at] for shit, but I guess it might be good at [thing I don't know about]" will continue to carry the day as long as people let it

I'll self-indulgently close by quoting myself again:

A lot of people with a lot of money would like you to think that genAI chatbots are going to fundamentally change the world by being brilliant at everything. From the sidelines, it doesn't feel like that's going to work out.

Trying not to be a Gell-Mann Amnesiac

I sometimes wonder how much Gell-Mann Amnesia people experience. Paraphrasing Crichton, when you're a domain expert, you'll sometimes read an article that gets every aspect of your field completely and absurdly wrong, have a little laugh about it... then keep on reading and trusting articles that are about other fields, even from the same publication or writer.

As if they're some pure spring of wisdom which only coughed out a lump of mud when it came to the thing you happen to know about.

It's just an idea from a novelist, not the kind of cognitive bias that's supported by real-world studies that I know of, but you have to admit that it has a kind of... truthiness to it.

Stack this up with Dunning-Kruger and it's easy to become cynical. You might decide that actually, all the loudest voices are talking complete nonsense, all of the time. That might be too far. But I do think it pays to put deliberate hard effort into distinguishing domain experts from overconfident bullshitting pundits.

Now, anyone with their ear to the ground and a weather eye out for Gell-Mann Amnesia should have arrived at the obvious conclusion about generative AI. To wit, that the current state of the technology is that it is an overconfident bullshitter.

On being a piece of software and being confidently wrong

The case studies are easy to find, and the ones from domain experts sound pretty different from the ones from the tech industry and the reporters too busy and/or demoralised to do more than repackage their press releases as articles.

➡️ I am not a historian. The historians I've read say genAI gets softball history questions mostly right and deep ones mostly wrong. Sometimes subtly, sometimes dramatically. It just makes things up when the evidence is scarce. It makes errors of commission and omission as well as having misplaced focus and drawing weird conclusions from premises.

➡️ I am not an artist. The artists I listen to say genAI art looks bland and awful and organic because it doesn't understand composition or anatomy or separate objects (because it doesn't 'understand' anything). It can't make an image that isn't well-represented in the training data, like a camel and a steampunk automaton jousting from the backs of sumo wrestlers. Same in other kinds of media: filmmakers say genAI can't do film because it can't take direction or keep track of characters or have a consistent shot.

➡️ I am not a Wikipedia editor (except incidentally). Earlier this year there was a wretched moment when the Wikipedia editors were going to have genAI article summaries foisted on them, although I think that's turned around now. The skilled editors pointed out that the LLM summaries generally ranged from 'bad' to 'worthless' by Wiki standards: they didn't meet the tone requirements, left out key details or included incidental ones, injected "information" that wasn't in the article, and so on.

➡️ I am not a manager. The managers say genAI can't even collate timesheets reliably.

➡️ I am not a novelist. The novelists say a genAI book reads like a statistical summary of all creative writing anyone has ever done, including all the embarrassing teenage fanfiction. It sucks at originality. And because it doesn't have an internal model or understanding of its outputs, it can't keep track of things and make a coherent satisfying story. Things are vague, tropey, or contradictory.

➡️ I am not a lawyer. The lawyers are, um, well, by the sound of it a lot of them are being sanctioned for using generative AI to cite completely nonexistent caselaw. (☉__☉”)

➡️ I am not a public policy wonk. The bureaucratic wonks note that genAI can't summarise text. It shortens it and fills in the gaps with median seems-plausible-to-me pablum. The kind you get when you average out everything anyone has ever written on the internet. If you try to have an LLM summarise or draw conclusions from a study, it will usually do a bad job, fabricating statements more along the lines of what an average person would guess if they'd only read the study's title.

➡️ I am not a software engineer. The software engineers seem to have mixed opinions. They say that genAI works as code autocomplete (something that has existed for fifty years, but this new kind has pretty sophisticated lookahead, neat). At least some are saying it can't do principled software engineering, it introduces security flaws, its performance drops off for obscure languages, it overconfidently generates bad code, it plagiarises from code repositories that it doesn't have the rights to...

I could go on.

I'm no longer a domain expert in anything, this many years after my stint in academia. I think I'm halfway to being an expert in a few different areas, though. I deliberately concocted some thoughtful questions at the intersection of those areas, just to see.

For example, I asked about the (obvious) mapping of choose-your-path text adventure books onto mathematical graph structures, which the LLM chatbot identified. I followed up with technical questions about the features of those graphs in context: what would the game be like if they weren't digraphs, would you expect cyclic vs acyclic, would a finite state machine be more appropriate and if so why, etc.

And lo, the generative AI output was absurdly, hopelessly, and confidently wrong when given questions that needed expertise.

A lot of people with a lot of money would like you to think that genAI chatbots are going to fundamentally change the world by being brilliant at everything. From the sidelines, it doesn't feel like that's going to work out.

Sometimes I read posts from experts along the lines of

"I've noticed it's almost worthless at [my field], but it sounds like it's pretty useful for [other thing]."

But less so lately, maybe?

So I'm left wondering: are people experiencing massive Gell-Mann Amnesia about these chatbots? Or does everybody know that the emperor has no clothes?

(But oh no, we've invested so, so, so very much money into the emperor's finery, and all the wealthiest people at the imperial court agree: pleeeease could you keep squinting to see this amazing new clothing?)