This essay from @jenniferplusplus is very good, and very important.
It’s good enough and important enough that I’m just going to QFT the heck out of it here on Mastodon until I annoy you into readying the whole thing.
https://jenniferplusplus.com/losing-the-imitation-game/
This essay isn’t the last word on AI in software — but what it says is the ground level for having any sort of coherent discussion about the topic that isn’t all hype and panic.
1/
“Artificial Intelligence is an unhelpful term. It serves as a vehicle for people's invalid assumptions. It hand-waves an enormous amount of complexity regarding what ‘intelligence’ even is or means.“
“Our understanding of intelligence is a moving target. We only have one meaningful fixed point to work from. We assert that humans are intelligent. Whether anything else is, is not certain. What intelligence itself is, is not certain.”
2/
“While the capabilities are fantasy, the dangers are real. These tools have denied people jobs, housing, and welfare. All erroneously. They have denied people bail and parole, in such a racist way it would be comical if it wasn't real.
“And the actual function of AI in all of these situations is to obscure liability for the harm these decisions cause.”
3/
“What [LLM] parameters don't represent is anything like knowledge or understanding. That's just not what LLMs do. The model doesn't know what those tokens mean. I want to say it only knows how they're used, but even that is over stating the case, because it doesn't •know• things. It •models• how those tokens are used.
“…The model doesn't know, or understand, or comprehend anything about that data any more than a spreadsheet containing the same information would understand it.”
4/
“The hard part of programming is building and maintaining a useful mental model of a complex system. The easy part is writing code.”
Do you see where this is going? Have I convinced you to read the whole thing yet?
https://jenniferplusplus.com/losing-the-imitation-game/
5/
Here it is: the One Weird Thing that people who aren’t programmers (or are bad programmings) just don’t understand about writing software. This is it. If you miss this, you’ll miss what LLMs can and can’t do for software development. You’ll be prey to the hype, a mark for the con.
6/
“They're positioning this tool as a universal solution, but it's only capable of doing the easy part. And even then, it's not able to do that part reliably. Human engineers will still have to evaluate and review the code that an AI writes. But they'll now have to do it without the benefit of having anyone who understands it.”
7/
“No one can explain it. No one can explain what they were thinking when they wrote it. No one can explain what they expect it to do.
“Every choice made in writing software is a choice not to do things in a different way. And there will be no one who can explain why they made this choice, and not those others. In part because it wasn't even a decision that was made. It was a probability that was realized.”
8/
You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
Alas, that does not remotely resemble how people are pitching this technology.
9/
I love, for example, this student’s reaction to having ChatGPT trying to write some of her paper:
https://hachyderm.io/@inthehands/109491316523726437
Indignant outrage is a powerful thought-sharpening tool!
Alas, AI vendors are not pitching LLMs as indignant outrage generators.
10/
I’ve heard from several students that LLMs have been really useful to them in that “where the !^%8 do I even start?!” phase of learning a new language, framework, or tool. Documentation frequently fails to share common idioms; discovering the right idiom in the current context is often difficult. And “What’s a pattern that fits here, never mind the correctness of the details?” is a great question for an LLM.
Alas, the AI hype is around LLMs •replacing• thought, not •prompting• it.
11/
The hard part of programming is •thinking about what you’re doing•, because the computer that runs your code isn’t going to do that.
And as Jennifer points out in the essay, we do that by thinking about code. Not just about our abstract mental models, not just about our natural language descriptions of the code, but about the code itself. Where human understanding meets machine interpretation, •that’s• where the real work is, •that’s• what makes software hard:
12/
Code is cost. It costs merely by •existing• in any context where it might run. Code is a burden we bear because (we hope) the cost is worth it.
What happens if we write code with a tool that (1) decreases the cost per line of •generating• code while (2) vastly increasing the cost per line of •maintaining• that code? How do we use such a tool wisely? Can we?
Useful conversation about that starts on this ground floor:
https://jenniferplusplus.com/losing-the-imitation-game/
/end
@inthehands One of the obscure methods I have been using recently when I'm writing (not coding, actual 'write a story' type thing) is when I blank page for a while, I kick one of the LLM bots and send it a semi-related prompt.
Then I get so irked by how it completely missed the point, that it gets my competitive backbrain going and saying 'damnit, I know how to say it better than THAT'. And once I start, that solves the 'getting started' portion.
Does fuckall for plot, characters, ect, but...
@inthehands i found the blog post a mixed bag. The specific points about software development are very good.
But the general critique of LLMs is lacking. Two examples:
1. She claims AI/LLMs are not intelligent, but does specify what that means.
Maybe AI then is intelligent by *some* measure?
2. She claims the model parameters don't represent knowledge or understanding. Same problem here: what *is* knowledge?
She falls into the same fallacy as the AI hype train: vague terms.
@inthehands the big value i see in this blog post is the concretness of the risk descriptions and the factual references to actual AI project failures.
If these are all the objections you can raise, consider the blog post a success – and be mindful to not be part of the problem.
If you are interested, this is a meditation on this fact that I wrote a long time ago now.
https://mutable-states.com/who-are-the-programmers.html
the hard part of programming is *also* thinking about what future you is going to be doing
@psu_13 @inthehands Yeah, or to use an example of a joke about code quality:
"The hard part of programming is ensuring that the next person to read the code who happens to be an axe murderer who knows where you live doesn't have cause to use that knowledge.".
@inthehands this last bit is the only place I've seen value in LLM. When you're introduced with a new thing, with little to no context, you can often get more information associated with the thing by using a LLM, which will then help you figure out where to go next (or at least give you more to search on).
I almost want to say it's like a context generator, but that's too generous - word association seems more appropriate. Still surprisingly useful in certain situations, like (as you noted) when you have no idea where to even start.
@inthehands "Want to exploit Cunningham's Law without subjecting yourself to the internet? Let an A.I. do it for you! No Internet posting required - just read what it gives you as output!"
@inthehands For me the best use for them is for when you don’t know what to search for. Recently I wanted to do parametrised unit testing in Python, but I couldn’t remember how it was called, and regular searches gave me nothing. I described it to Bing chat and I got the name of the technique, and a couple of libraries for it, with links and sample code. Maybe the sample code was wrong, but I don’t care, I just wanted to know what to search for. >>
@inthehands And our local DIY shop (Jumbo, in Switzerland) has a chatbot. You tell it what you want to do and it answers with with tips (I think it’s human-written articles selected by LLM, not LLM-written tips, but I’d have to check,) and product suggestions. You describe a part you don’t know how it’s called, like the cylindrical part with a thread perpendicular to its axis used to assemble some wooden furniture, and it tells you it’s called a cylindrical nut. It’s extremely convenient.
@inthehands this is my thing. nobody can present a coherent story about how will be useful beyond gulling the public with deepfakes.
@hwestiii
I mean, I give some ideas downthread about how having a bullshit generator can be useful. Just people aren’t talking about it in remotely those terms.
I think you overlook that the BS generation is done by ignoring the copyright and consent of creatives.
Whether you think (C) is good or not, under the legal system in place creative people should be getting paid for their work and not have it used without their consent.
No significant number of artists consented to Midjourney etc scraping.
@Homebrewandhacking @hwestiii
Yes, that whole area of law is just a mess right now. We need some sort of legal regime in which creative people share in the value created by their work, whatever that value is, and copyright in its current form isn’t even coming close to doing that job.
Yes sure, and more to the point, this work, my work, shouldn't be taken and fed into a slurrification machine so someone can get over their creative block unless i) I consent, ii) I get paid.
Apparently my work has no value to the creators of generative AI (but they can't actually use their machines without it.)
@inthehands Using AI seems like the software would make it easy to say 'by design' and ignore a bug. And bugs can be anything from annoyances to security flaws...I've tested software and trying to be a pseudo user and advocate for a better product ran up against developer/PM reluctance to fix the code and that's *without* AI...
@servelan
I’ve had people argue to me that it basically doesn’t matter how bad AI-generated code is, because you just fiddle with the prompt / the output until it passes the tests, and we all know how to test, and…I’m just like, have you ever •created• actual software?
@inthehands @servelan@newsie.social The authors of the glibc implementation of the C standard library, presumably very expert in writing C, introduced a buffer overflow that got missed for years: https://www.zdnet.com/article/patch-now-this-serious-linux-vulnerability-affects-nearly-all-distributions/
Until ChatGPT *et al.* are better at programming than an expert-level C programmer (not likely, ever), it's a much better idea to let the humans do most of the work.
@inthehands @servelan Yeah, that's the crowd that really doesn't care about solving problems at all - all they want is mimic solving problems. These days more and more I'm reminded of what R.P. Feynman called "Cargo Cult Science": https://en.wikipedia.org/wiki/Surely_You%27re_Joking,_Mr._Feynman!
@inthehands ahhhhh just YESTERDAY I was trying to make this exact same point during a conversation with a coworker, without the benefit of knowing about this essay. Shoot, I would've saved myself the effort of writing my own blog post ranting about these things if I'd seen this sooner.
@inthehands
With the complicating factor that mental capacity diminishes over time.
@zl2tod @inthehands yours, mine, or the LLM's? Changes rather than diminishes until quite near the end for many humans.
@inthehands
This reminds me exactly of the stuff I would say to people about "automated journalism" back when there was much wailing and gnashing of teeth over basic templated text. "The writing is the easy part!"
(This was almost 10 years ago, and LLMs hadn't hit the mainstream yet. I was at The Associated Press as "news automation editor", which meant I got lots of panicked questions from across the news industry about what this meant for reporters.)
@myersjustinc
Very, very similar, yes. “Good writing is good thinking,” my English prof mom always says.
@inthehands@hachyderm.io Fully agree! Much of the software I encounter in the Library Space is written such that it successfully executes and does it's thing. That's certainly good but seldom enough to build reliable systems. It's creating a mental model of the task and it's environment, thinking about the assumptions and conditions, and dealing with the things that can/will go wrong what qualifies software development for me.
@inthehands staring off into to space when asked about a novel requirement for an established system is a key part of the process (for me at least).
What is this sourced from?
@inthehands nevermind. You'd think I never use this app.
@inthehands @aral @jenniferplusplus
I think it's the iterative design process more than the mental model that's important.
Using something like BPMN as a modern version of the "top-down stepwise refinement" approach allows you to black box stuff (coherent sub-processes/functions) until needed without necessarily trying to keep everything in your noggin at once. A state diagram (ot two...) can also really help ensure the diagrammatic logic and your mental model doesn't miss anything, in my experience. Both allow you to communicate what you're doing to others (including future you).
You still have to think hard though!
@inthehands @hailey it was ported to my attention that Peter Naur did a very similar argument 40y ago in "programming as a Building Theory"
https://fosstodon.org/@arosien/112011681335546370
Both text complement each other quite well, with Naur setting the academic background on the subject, and Jennifer having a more synthetic and actualized text tailored for AI and its specific danger