Recent searches
Search options
OK, trying an experiment with my Programming Languages class!
• Have an AI generate some of your writing assignment.
• Critique its output. Call BS on its BS.
Assignment details in screenshots below. I’ll let you know how it goes.
(Here are the links from the screenshots:)
Raw AI Text:
https://gist.github.com/pcantrell/7b68ce7c5b2e329543e2dadd6853be21
Comments on AI Text:
https://gist.github.com/pcantrell/d51bc2d4257027a6b4c64c9010d42c32
(Better) Human Text
https://gist.github.com/pcantrell/f363734336e6063f61e451e2658b50a6
Early returns are promising. Students got into the activity. Discussion took over half the last day of class.
This is a Programming Languages class, and discussion took an interesting turn: how would •languages• change in a world where we expect AI code generation to be a normal part of the development process, and the problem for humans shifts (even further) away from “How do I generate code?” and toward “Does the code I’ve generated actually do what I think? What I want?”
#proglang #ai
My favorite outcome so far: a student remarked (paraphrasing here) that she didn’t realize how much she had to say in her paper until she saw how wrong the AI was, how much it missed the point.
Observing her own reaction to BS about her topic made her realize she’d underestimated the extent of her own newly-forming knowledge. That…that is the sort of outcome an educator dreams of.
Contextualizing thoughts on the goals of this assignment:
• I want students to think critically about AI, beyond all the hype juice, based on real experience.
• AI will likely become a useful assistive technology (just like web search). Students should learn it to use it _wisely_ (just like web search).
• Examples of bad writing are always useful. Now we can generate them!
• Humans are better at reacting than imagining. “That's not quite right!” is an excellent writing-sharpening thought.
• I want to prepare students for the future they’ll encounter — and help them identify what makes •them• valuable (for the job market, for the good of the world, for their own sense of purpose) in that future.
How will humans still matter in the face of automation?
I find the answer to that question often refocuses pedagogy where it should have been in the first place. Automatability correlates with BS teaching.
Results of the experiment (having students generate and critique AI output as a writing exercise; see upthread):
Success! It seems to me the paper quality leveled up.
Hard to judge: not a controlled experiment, small N, subjective, etc., but gosh…the papers this year are more focused on content, more coherent, and less prone to rambling page-filler text.
Having “What does BS on my topic sound like?” as reference point really seems to have been helpful.
While I imagine AI could be genuinely useful in the future for information discovery in research, it’s not clear that it served that function for any students this time around.
But it didn’t need to in order to be useful. Simply having this touchstone — “here is what mediocrity looks like for your topic” — challenged them to do better, and got them thinking about writing from the reader’s POV. They learned from their own reactions.
The future is uncertain, but in near term, it’s nice to know that the current generation of AI can be good for something in education beyond automated plagiarism.
(Surely a prerequisite for this exercise working is having students who care at all about their topic and their education in the first place. I’m quite fortunate in that respect.)
Will definitely repeat this exercise in the future.
@inthehands excellent. This was going to be my proposed solution also. Hope you’ll tell us how it went. Greetings from Switzerland.
@harphat And to you from Minneapolis!
@inthehands That’s the thing that bothers me most about code generation: I like writing code, the design process, much much more than testing/debugging. Yet that’s the part I’d give up to the machine? No thanks!
@finestructure Indeed. So does AI mean that our lives will mostly be testing and debugging in the future? Or will it lead to more declarative languages, where writing algorithms is more like writing SQL or Prolog: we specify goals, constraints, and verification, and the machine determines the algorithm? 
@inthehands I wonder to what degree these are separable. When/how do we trust the algorithm the machine comes up with?
I guess we’re already doing it to a degree when we let compilers of higher level languages produce the machine code.
I’ve been programming for >20 years yet I’ve never written assembly. I have to trust that translation process. The difference is, I think, that there’s a peer-reviewed machine developed by experts, not a nebulous “somehow” trained AI in the middle.
@inthehands SQL is a great example: not a language I particularly enjoy working with: very easy to make mistakes, hard to test. A perfect example of a language where I get a result and then I don't fully trust it. Too often I've been burned because my test data didn't expose issues.
Not so much a problem of the language as a lack of testability due to missing tooling and poor composability. It's hard to break an SQL query down into its components and test them in isolation.
@inthehands A machine producing an SQL monster query is my personal nightmare 
@finestructure I’m suggesting the opposite: the code we humans write might resemble SQL in that it describes the desired result without specifying exactly how to get it.
@inthehands I see! The problem I have with all of this is that we're essentially talking about turning a spec into a program. Anyone who's ever taken a spec and gone to implement it will probably agree that a precise spec essentially has to be some sort of a program. So then your natural language input eventually morphs into a higher level specification language.
@inthehands The only times I've ever had a good spec to work off was when I was re-implementing an existing program and the old code was the spec.
I can't quite see where the AI wouldn't be more than a fancy transpiler/compiler, which we already have 
@finestructure Have you ever spent any time with a model checker like Alloy? That’s what I’m picturing: not a perfect spec that comes out of thin air, but a dialogue between the programmer and the machine where you gradually build up a spec through iterative feedback about examples and assertions.
@inthehands No, I’d never heard of it. This is https://alloytools.org? Interesting!
If I understand correctly, this goes in the direction I was talking about: a layer above our current programming languages that then (deterministically) generates a program.
@finestructure Yes, that’s it. The thing about Alloy in particular that’s relevant here is that it’s not just a •verifier•; it’s built around the idea of generating •examples•.
You can ask it for counterexamples: “I think X is always true.” “Here’s a case where X is false!” But you can also ask it for examples: “Just show me what I allow.”
Quick example in screenshots:
1. Here’s my model of a bicycle.
2. Generate example: Good!
3. Another example: OOPS
4. Here’s my revised model
@finestructure Alloy would need a whole lot of work to become a general use tool working alongside AI, but this general direction of could hold something promising.
In this world, determinism isn’t important; the AI can propose lots of solutions. The interesting work is in verification: “Does this code do what I thought? Does it do what we •want•? Do we want the right thing?”
…which is already the truly hard part of programming.
@inthehands Very interesting, thanks for the additional details!
@stph this thread may pick your interest
@inthehands having examples of mediocrity is what is often missing in teaching and why I think being amongst other students (who are not perfect by definition) is one of the most valuable parts of learning at a school, imo.
Being able to bounce stuff off an ai when self learning is what I really value, because it teaches me what boring is, and it puts me in the teachers shoes to explain what it did wrong.
I often double down on that concept by having the ai explicitly take on that role.
1/
@inthehands It helps me not only avoid getting fooled by the convincingly wrong phrasing, but makes it more interactive/emotionally intense (and thus better learning retention).
https://hachyderm.io/@mnl/111467764783141450
As an avid self learner who doesn’t often find online cohorts of people studying exactly the same thing at exactly the same time, having a bot that convinces my “social brain” that it is real has been a huge boost and makes me feel less “lonely” while learning.
2/2
@inthehands It very interesting. The knee-jerk reaction I have read about is from one school blocking access to ChatGPT on the school network. But if they instead consider how AI is going to be leveraged in the careers of their students they would start planning on how to leverage it effectively.
@brennansv Knee-jerk indeed. It’s certainly an open question whether and how AI will be useful in people’s careers. What is certain is that ignoring it and uncritically as a kind of benign magic are both very foolish approaches.
@inthehands It's been a while since I read the original Dunning-Kruger paper but I seem to recall they found participants developed a more accurate sense of their own performance by observing other people's performance. So it seems to me that in a sense you replicated their results but with an LLM as the point of comparison. In addition to all your other observations, I think that's an interesting outcome too!
@inthehands you must be one kickass teacher 
@inthehands I like how self-directed this assignment is! Also how it challenges students to identify the qualities that make their work excellent as they asses the work of he AI.
Happened across this, as you do, and realized exactly how old it was before I just blindly hit reply (but i'm hitting it anyway! No gods no masters, etc).
Have you followed up on this? I think it's a great story, curious if you've done more with it (another class, eg), or something similar?