It wrote this blog title.
It also might be the most annoying ‘content producer’ I’ve ever worked with.
So you’ve probably heard, ChatGPT is going to change the world. All writers will be out of a job, replaced by a speedier, cheaper AI version that doesn’t complain about click targets. It can do anything in minutes (and make you millions at the same time).
So writing FOIFriday should be a snap. It’s a straightforward round up of FOI in the news each week – short intro (some news, some tips, some rambling), then a series of summaries of articles that have made good use of the Freedom of Information Act. All of the actual writing work has been done by actual journalists, it just needs pulling together. No problem?
How wrong was I.
So far I’ve learnt:
- ChatGPT can’t be trusted
- ChatGPT can’t follow simple instructions
- ChatGPT has a really short-term memory
- ChatGPT really can’t multitask
- ChatGPT is a lot perkier than me
But I did get there in the end – it managed a respectable version of the FOIFriday post from the week before last.
Some of these trials and tribulations may be on me. There’s an art to prompting ChatGPT and I don’t think I have it. And even after trying quite a bit, I don’t think I’ve got it – I’ve learnt some things but getting consistent results is still beyond me.
ChatGPT can’t be trusted
Any opportunity to dash off a series of well written paragraphs in seconds will be taken, even if you were just asking a question to try and check what it would need to write the thing you actually want it to write. Mostly this output will not be what you wanted.
So the first attempt started with giving ChatGPT a link to an article and asking it to write a summary. And it kinda gets the gist. But it likes to add its own interpretation. So I have a summary about a councillor not paying their council tax but just a random councillor from the area, not the one mentioned in the story, a random amount unpaid, and the request was made by a member of the public.
Asking it to correct the details sort of works. It manages to attribute the FOI request to the Derby Telegraph but just subs in a different councillor and amount instead.
I did manage to solve this particular issue by giving ChatGPT specific paragraphs in the linked article to focus on rewriting. That worked fine – it stopped producing potentially defamatory nonsense and the gaslighting me about it and actually produced a summary of the paragraphs it had been given.
ChatGPT won’t follow simple instructions
But FOIFriday is a round up of several different articles each week. So we need to do this but for seven to nine links. Shouldn’t be an issue?
Apparently, it was not that straightforward.
One of the tips for using ChatGPT is to ask it to help you write prompts that it can follow. So I asked the best way to write a prompt for a series of article summaries and got what seemed like a useful prompt. I then tried combining that with a request to write an intro for the blog post about the latest FOI news (the ICO announcement of how it plans to prioritise complaints).
I asked if it needed any more information about this task. It launched into an explanation of how the ICO’s Deputy Commissioner Elizabeth Denham was announcing a new proactive approach.
It said to give it the prompt again with the information needed. I did, and got back a blog outline.
This happened several times. Either I thought the instructions were clear and they weren’t or I asked ChatGPT for a prompt idea, used that, and then didn’t get back the right thing. I think this might need some trial and error to find the right way of setting out instructions so they really are clear and specific enough for the flaky AI.
Though I thought asking for nine articles to be summarised and giving nine links to articles with paragraphs to focus on was a clear instruction. I got five article summaries, so apparently not.
ChatGPT has a really short-term memory
Trying to fix that brings us to the next problem.
ChatGPT is supposed to be able to remember things from earlier in the same chat but this ability seems to come and go.
When I got five article summaries instead of the requested nine, I asked ChatGPT to summarise the rest of the articles I’d given it. It didn’t do that, apparently unable to connect that request to the previous information. Instead I got summaries of some random articles off the internet.
Asking it to write tweets about the text it had just finished writing resulted in some tweets about other articles about FOI that possibly exist somewhere on the internet. Apart from the time it suddenly managed to write really good tweets, with correct @ mentions and links from an earlier prompt.
ChatGPT really can’t multitask
Well, sometimes it can. But other times, if you give it a series of related tasks to do it’ll skip one (and if you ask it to then do the task, it’ll have forgotten what you previously requested). Or it’ll kind of combine the tasks into one piece of writing.
And that’s how I ended up with article summaries with hashtags (that should have gone with the Twitter thread I asked for).
Breaking down tasks really seems to be key. I got better results when I asked for one thing at a time, even if it meant things took longer (and it also made for a slightly disjointed blog post written in four parts).
ChatGPT is a lot perkier than me
It certainly loves an emoji.
I do not.
Something I think I need to learn is how to get ChatGPT to write in a particular style. I had so much trouble getting it to write the content I wanted, I didn’t even try with this one. Mostly it seems to have two modes – fairly serious or very enthusiastic. I think I’d quite like something in between.
What I’ve learnt
I’d say working with ChatGPT is trial and error but it seems quite inconsistent. Sometimes a prompt works well, other times a similar prompt generates a totally unhelpful response.
I did, eventually, get a version of FOIFriday written.
To get this, I broke down the sections into separate prompts – an intro, some tips, five article summaries, then four more article summaries. I also made sure there was clear background information for the article summaries (I feel like they still need checking – everything ChatGPT writes could do with subbing, just to make sure). I then asked it for a Twitter thread and a Linked In post.
That makes it sound like ChatGPT worked through each of those tasks with no real problems. Oh no, there were several false starts and things needing to be explained several times to get a sensible answer.
It was not quicker than just doing it myself and at times I felt like I was having to explain the entire concept to someone very keen but very clueless…repeatedly.
You can judge whether what ChatGPT wrote was better than the original.
Having more luck with ChatGPT than me? Cracked prompts and getting the output you want every time? Made your first million with AI? Feel free to share any tips I’m missing in the comments.