How to measure “difficult to measure” things.

What lipstick, the Panama Canal and toilet breaks can tell you.

There are some things that are difficult to measure, maybe even impossible to measure directly. The key with 'difficult to measure' things is to look at something that either affects or is affected by the thing you are interested in. Here are three examples of measuring 'difficult to measure' things. The first two focus on the nearly impossible task of finding out what young children really think about a TV show or film...

• The creators of Sesame Street came up with a practical way of seeing how engaged children were with their programme. The children we studied with a TV playing the 'test show' and along side it was a 'distractor' slide show. The researchers would periodically check to see whether the child was watching the show or looking at the distractor slides show. This way they could quantitatively assess the level of engagement in every element of the show.
• The Children's Film Foundation used a tally of the number of times children left the cinema to go to the toilet as an indication of engagement in a film (although there are obvious complications with film length and number of drinks consumed here!).
• Economic growth is another notoriously difficult thing to measure. The Bank of England uses a composite KPI based on things like the amount of traffic though the Panama Canal and the volume of Air Freight to give early reliable indications of economic growth - given the terrible name of 'nowcasting'.

A word of caution

There is a measure called the 'lipstick index', based on the observation that cosmetic sales (in particular lipstick) are inversely correlated to economic growth. When times are hard it is supposed that women reach for the lipstick, although according to the Economist the lipstick index is not supported by the evidence. It is easy to get carried away, so make sure your clever measure is supported by actual evidence.