Just how wrong can an AI get it?

As part of my effort to profile the power consumption of GenAI, I decided to try and summarise one of my travel blogs using ChatGPT and the other big public models, plus a series of smaller models which can be hosted locally on my Alienware workstation. What’s interesting is just how well, and badly, the different AIs handled the task.

The instruction in every case was the same: “Summarise the blog at https://www.andrewj.com/blog/2024/three-kirks-and-a-kirkufell/”. You might want to read the blog yourselves before reading further: it’s neither long nor complicated, but it defeated most of the contenders.

I tried to avoid writing “WTF”, but I was sorely tempted. Some of these are hallucinating like they’ve just taken LSD…

The Attempts!

ChatGPT

The blog post “Three Kirks and a Kirkufell!” recounts a photography trip around Iceland’s Snæfellsnes Peninsula. The author, despite dealing with a cold and Iceland’s challenging weather, visits and photographs three distinctive churches (“kirks”)—Búðir, Stóra-Hella, and Enni—before capturing the iconic Kirkjufell mountain. The journey includes dramatic landscapes, with highlights like waves crashing and snow-covered scenes. The narrative reflects the difficulties and rewards of winter travel in Iceland.

Score: 10/10. Excellent, couldn’t do better myself.

Gemini (Bard)

This is an article about a photographer’s trip to Iceland. It discusses his experiences photographing three churches and Kirkufell. He had some difficulty finding pharmacies with the medications he needed. He also had some trouble finding Kirkufell because of the weather. He eventually found it and took many photographs.

Score: 6/10. OK, effectively correct apart from the bit about not being able to find Kirkufell, but lacking the nicely condensed detail of the ChatGPT version.

CoPilot Chat

The blog post titled “Three Kirks and a Kirkufell!” by Andrew Johnston describes a travel experience in Iceland. The author recounts a cold, snowy day driving along the north of the Snaefellsnes peninsula. The journey included an attempt to visit Kirkufell, a notable mountain, but the stop was abandoned due to poor visibility from the snow. The post highlights the challenges and beauty of traveling in Iceland during winter.

Score: 4/10. Just adequate. Some factual inaccuracies and poor detail, but at least still rooted in reality.

Wizard V1.2

The article “Three Kirks and a Kirkjufell” is about an adventure trip to Iceland by the author, Andrew J., who visited three churches (Kirks) in Reykjavik. The first church was Hallgrimskirkja, which has a unique architecture that resembles a rocket ship taking off into space. The second one was the Lutheran Church of Holmenkollen, known for its beautiful stained glass windows and traditional Norwegian wood carvings. The third Kirk was the Gamla Stan Church in Stockholm, Sweden, an old church with a rich history dating back to the 13th century.

Andrew J. also visited Kirkjufell, a mountain near Reykjavik that is known for its unique triangular shape and appeared in the popular TV series “Game of Thrones”. He enjoyed hiking around the base of the mountain and admiring its beauty from different angles. The author concludes by sharing his love for exploring new places and experiencing their cultures, architecture, and natural wonders.

Score: 2/10 (feeling generous). Starts OK, but rapidly descends into just making stuff up. How I’m supposed to have teleported to Stockholm while in Reykjavik is not explained.

Llama 3B Instruct

The blog post “Three Kirks and a Kirkufell” by Andrew J. is an entertaining account of his experience hiking in Iceland’s stunning landscapes. Here’s a summary:

Andrew, along with friends Sarah and Mike, embarked on a 7-day adventure to explore three iconic KIRKs (churches) and the famous Kirkjufell mountain in western Iceland. The trio aimed to visit the churches at Snæfellsnes Peninsula, Borgarnes, and Reykjavik. Their journey began with a scenic drive along the Golden Circle route, where they marveled at Þingvellir National Park’s unique geological features. They then visited the picturesque church in Borgarnes, which dates back to 1615. The next day, Andrew and friends explored Snæfellsnes Peninsula, admiring its breathtaking landscapes, including black sand beaches, glaciers, and volcanic mountains. They stopped at a charming church in Arnarstapi village, built in 1869. Their final destination was Kirkjufell mountain, made famous by Game of Thrones as the “Arrowhead Mountain.” The group hiked to the base of the iconic peak, taking in its majestic views and snapping photos. Throughout their adventure, Andrew shares witty anecdotes about his friends’ quirks (or lack thereof), amusing observations on Icelandic culture, and stunning visuals showcasing Iceland’s natural beauty.

Score: 2/10. The start and end are OK and at least the context is correct, but the bit in the middle is complete fantasy.

Llama 3

The title is “Three Kirks and a Kirkufell”, and it’s a book review of “The Fellowship of the Ring” by J.R.R. Tolkien. Here’s a summary:
The author, Andrew J., reviews his re-read of “The Fellowship of the Ring”, the first book in The Lord of the Rings trilogy. He reflects on how he appreciated certain aspects more this time around, such as the character development and world-building. Andrew praises Tolkien’s writing style, saying it’s both engaging and immersive. He also discusses the themes of the book, including friendship, loyalty, and the struggle between good and evil. The reviewer notes that the pacing is well-balanced, with a mix of action, humor, and quiet moments. Andrew concludes that “The Fellowship of the Ring” remains an excellent fantasy novel even after multiple readings, and he’s excited to continue with the rest of the trilogy.

Score: 0/10. This is a complete mystery, especially given that a slightly different version of Llama at least got the context right. How on earth did they get to “Lord of the Rings”? Ironically I could understand “Journey to the Centre of the Earth” because at least that starts at Snaefells…

Mistral

The blog post, “Three Kirks and a Kirkufell,” by Andrew J., discusses the author’s experience visiting Iceland and exploring various sites related to Star Trek. The three Kirks mentioned in the title refer to Captain James T. Kirk from the popular science fiction series, who has been immortalized in different ways throughout Iceland. These include a statue of him at Keflavik International Airport, another one near the Blue Lagoon, and an actual mountain called Kirkjufell (Kirk’s Mountain). The author also shares his thoughts on the unique landscapes and natural beauty found in Iceland, which inspired the creators of Star Trek to use it as a backdrop for their show.

Score: 0/10. Complete balderdash. This model obviously doesn’t recognise the work “kirk” despite the fact that it’s a regular part of Scottish English, with exactly the same spelling and meaning as in Icelandic. On a positive note at least we’re back in Iceland from Middle Earth…

GPT4All Falcon

The blog post “Three Kirks and a Kirkufell” by Andrew J. is about his experience hiking in Iceland, specifically the three peaks of Kirkjufell, Kirkjuvatn, and Kirkjufell. He describes the challenges he faced, such as navigating through thick fog and snow, and how he overcame them with determination and perseverance. He also shares some tips for other hikers who may want to attempt this hike. Overall, the blog post is a personal account of Andrew J.’s experience in Iceland and his love for hiking.

Score: 1/10. Poor. I don’t mention hiking, and by my count the summary lists two mountains not three. Again no recognition of “kirk”.

Ghost 7B

Summary:

The blog post discusses three different churches: Kirk, Kirke and Kirche. Each church has its own unique characteristics and history. The author explains that the word “church” itself comes from the Old English word “cyrc,” which means “circle.” This is reflected in the shape of many early Christian churches, which were often circular or semi-circular.

Key Points:

o Kirk: A Scottish church type characterized by its low, round roof and lack of windows. It was originally designed to be a place for worshipping God outdoors.

o Kirke: A Norwegian church type that is similar to the Kirk in Scotland. However, it has a slightly taller roof and may have some small windows.

o Kirche: The German word for “church.” It refers to any Christian church, regardless of its size or style.

o Kirkufell: A term used by some Scandinavian countries to refer to the Church of England. It means “the church in the field” and is a reference to the fact that many early churches were built on agricultural land.

Conclusion:

The blog post provides an interesting overview of three different types of churches: Kirk, Kirke and Kirche. Each type has its own unique characteristics and history, and they all play an important role in the religious life of their respective countries.

Score: 0/10. We’re off on a tangent again, nothing to do with the source, and as far as I can see they’ve just completely made up some of the details.

MPT Chat

The article discusses how Andrew Kirk has been inspired by three different individuals in his life – a friend, an uncle and father figure who taught him about business ethics; the Dalai Lama for teaching him mindfulness techniques to deal with stress management at work. He also shares some of these lessons learned from them that have helped shape their own approach towards leadership which is centered around empathy rather than control or manipulation.

Score: -5/10. Nope, I haven’t got a clue how they got this from what I wrote.

Conclusions

The three online models (ChatGPT, Gemini and CoPilot Chat) all worked, although there is a very marked difference in quality and detail of output. The self-hosted models all failed, some more miserably than others…

In a large number of activities, better training and access to better resources both play a part in the success rate, and it appears that GenAI is no exception. Models you can host on your own system (even with a moderately high-spec GPU like my NVidia 3060 with 12GB RAM) are just pale shadows of the “very large language models” such as ChatGPT. They have neither the resources to hold significant knowledge, nor the training across a wide enough knowledge base. That doesn’t preclude effective use on a tightly-focused specialist use case, but their generalist capability is going to be very limited.

Beware, but also be afraid, be very afraid.