Thinking tool: Run the experiment / actually do the thing

Headshot of Adam Jones

Adam Jones

Sometimes it's pretty cheap to try things. Try them!

Two panels. Left, 'argue about it': a chat thread of speech bubbles bouncing back and forth, ending in a dashed circle with a question mark - a mushy guess. Right, 'try it': the same question, one arrow straight down to a tick - the answer

This shows up everywhere:

  • People debate hypotheses for why customers behave a certain way, rather than emailing a bunch of them and asking.
  • People argue about what they think the data would show, when in that time they could go and look it up! Same for arguing about what a stakeholder will say, e.g. whether legal will block something - you can ask legal, that's what they're there for.
  • People debate whether a process change would work, e.g. moving standup to async. Often the best way to find out is trying it for a couple of days. (With behaviour change especially, people hunt for social proof of ideas that would be low-cost to just try!)
  • I often fall into the trap of overanalysing personal purchases - reading reviews, comparing listing pages. It's usually a better use of time to buy the thing while not fully confident, knowing you can return it if it's not right.
  • When hiring, the panel sometimes realises we don't have enough information on an attribute. Six people might spend 10 minutes (60 person-minutes) guessing whether the candidate has it. For the same person-time, an extra 30-minute 1:1 interview probing that attribute (and any other gaps the panel found) is far more accurate.
  • Not sure you'll enjoy a TV show someone's recommended? Rather than spending 20 minutes reading reviews, spend the 20 minutes watching the first episode.
  • You read my argument for thinking in public, but aren't sure anyone will read what you write. Publish a couple of things and check the analytics!

A parable

Shelves packed with dozens of handmade ceramic bowls, mugs and cups in every colour and glaze. Photo: Ellen Tanner, Unsplash

There's a well-known parable about this from Art & Fear:

A pottery teacher grades half the class on the quantity of pots produced, and the other half on producing a single perfect pot.

The best pots all come from the quantity group - they spent the term running experiments and learning from them, while the quality group theorised about perfection.1

Trade-offs

Of course, these experiments aren't free. Sometimes they:

  • add latency, e.g. the extra interview delays the hiring decision
  • carry risk, e.g. the new standup format might reduce productivity for the trial days
  • miss things, e.g. maybe the TV show has a slow start

But directionally, people undervalue their ability to get empirical feedback from the world, and should do it more. My guess is the education system gives little practice at this, so people underweight it.

This is part of my thinking tools series. Also consider:

Footnotes

  1. This is just a fictional story. But I think it illustrates the concept well.

    I would really like to see if this is actually true, i.e. run the experiment to see if this holds up!

    My guess is that there's also an interesting version you could test by yourself with LLMs e.g. in-context learning through many attempts vs pure test-time compute. But it likley wouldn't be very representative of human behaviour and learning, ↩