Can LLMs count letters yet? tldr: Yes.

Adam Jones

Personally

This is my personal blog, where I write solely in my personal capacity. It does not represent the positions of any organisations I'm associated with.

Also, everything here is provded "as is", without warranties of any kind, express or implied.

10 August 202610 August 2026

"How many r's are in strawberry?" was a popular way to embarrass AI models a year ago, who would often incorrectly answer 2. This still circulates as a meme in some circles; and I couldn't easily find an example of someone running this experiment in 2026 to see how today's models fare.

So, I ran the experiment!

In short: models nail letter counting now. Even the oldest and cheapest models that Anthropic serves get 100%. And many fairly small local models also do quite well. Some tiny open-weights models (<4B params) struggle sometimes, but often due to spiralling "thinking", or failure to comply with format, rather than counting issues directly.

Test details

This ran on a fixed set of 20 English words with correct answers across 0/1/2/3+ occurrences, a 2000-token budget, ANSWER: <number> grading with unparseable responses scored as no answer rather than wrong. A transcript might look like:

Prompt:

How many times does the letter "s" appear in the string below (case-insensitive)?

discussion

You have a budget of 2000 tokens. Reason briefly if you need to, but you must commit to an answer within the budget.

Respond with your final answer on its own line in exactly this format:
ANSWER: <number>

Response (claude-haiku-4-5, 169 tokens):

I need to count the letter "s" in the word "discussion" (case-insensitive).

Let me go through each letter:
d-i-s-c-u-s-s-i-o-n

1. d - not an s
2. i - not an s
3. s - this is an s (1st)
4. c - not an s
5. u - not an s
6. s - this is an s (2nd)
7. s - this is an s (3rd)
8. i - not an s
9. o - not an s
10. n - not an s

The letter "s" appears 3 times in "discussion".

ANSWER: 3

Local models ran via ollama 0.32.4 on an M2 Pro MacBook (16GB); API models hit directly with fixed parameters. "Thinking: off" rows pass Ollama reasoning: {effort: "none"}.

The eval overall was implemented in AISI Inspect.

Other reflections

For small thinking models, thinking can be beneficial or harmful:

• lfm2.5-thinking gets 65% with thinking, 40% without.
• qwen3.5:4b goes from 0% with thinking, to 80% without. This is because with thinking it burns the entire token budget deliberating and never answers (even when warned about token budget!). For example, it often spells the word correctly, then second-guesses itself until the token budget runs out.

Low scores on small open-weights models feel less like miscounts, and are often more structural:

• Overthinking spirals: qwen3.5 is particularly prone to this, and even many of lfm2.5-thinking's failures are overthinking
• Format non-compliance: answering in prose/or using ans: response
• Absurd miscounts: gemma3:1b answers instantly with numbers like 74 t's in "settlement", or occasionally claims numbers in the billions... And qwen3.5:0.8b says "1" for 57% of all questions, whatever the string.

From browsing online, it seems like many "AI makes silly mistakes" videos these days rely on the ChatGPT voice models. My guess is that these might be much smaller to be able to process live voice; and therefore less intelligent. With the new GPT-Live models since July 8th 2026, I wonder if this will change the genre a fair bit.