Randomness

Last night, the weatherman told me that the next two days would be warmer than 'normal', with temperatures in the mid-nineties. By that, he meant that the temps would be above the average for this time of year. But, it does beg the question: If the temperature on any given day is above (or below) the average, is it therefore abnormal? If we were to get a string of above average days (say, 10 or so), would that also be abnormal, or is that something we should expect periodically?

Your favorite team has made it to the World Series (or Stanley Cup). You believe them to be the better team, although probably only slightly. Is 7 or 9 games enough to ensure that the better team wins the series? If one team sweeps the series 4-0, is that significant?

You family is uniformly tall. Your uncles are all above six feet, and even your aunts are close. Your grandparents are tall, as is your parents. However, you've topped out at 5'9”. Are you an aberration?

Each of these is an example of a distribution. In statistics, the standard distribution is an even curve around the mean, or average value. There are other distributions that are lopsided, with long tails to one side or the other. However, distributions are difficult to spot in our day to day lives: We have to keep records, and analyze those records to see the distributions, and they way they influence our lives.

Such analysis is the subject of Leonard Mlodinow's book: “The Drunkard's Walk: How Randomness Rules Our Lives.” In it, Mlodinow traces the development and understanding of probability and statistics from the early attempts to pinpoint the locations of the stars to recent studies of sports, finance, and medicine. Recognizing random distributions, and recognizing what they imply (and just as importantly, what they don't imply!) is a valuable skill.

The last example (height) demonstrates regression towards the mean – that given a sampling, even if there are outliers, most of the values will tend towards the mean (or average) value. It is actually important that tall people don't beget ever taller progeny: The human population would polarize to the very tall and the very short! However, the average height for a human male is in the 5'9”-5'10” range, and although the mean may be increasing slightly, there will be a greater concentration of people close to that height, and those very tall (or very short) are the outliers, and having them in your family tree is no guarantee of height.

The bad news on the World Series: If your team were 5% better than the other team (which might actually be unlikely in real life, the two best teams are likely even close in ability!); It would take something like 293 games to ensure that the better team won the majority. Is it possible for the lesser team to sweep the series 4-0? Not only possible, but likely, given to closely matched teams. (Think of flipping a penny: If you flip it enough times, you expect that half will be heads, and half tails, but if you flip it just 4 times, there is a reasonable chance that you will get all heads or all tails: 4 or 9 is just not enough flips to get the statistically expected outcome.)

And the weather? The weather, too, exhibits an even distribution about the daily means, both above and below. Where I live, the weather is regularly up to 10 degrees above or below the mean on any given day: Taken as a whole (365 days per year, 30 years worth of measurements, 10950 measurements total), the first standard deviation is 7 or 8 degrees of either side of the mean, indicating that 2/3's of the days are between -8 and +8 of the average. So, a day 5 or 6 degrees above average? Normal. 5 degrees below? Normal. 9 degrees below? Well...less common, but in something as variable as the weather, I'd say still normal.

In fact, the weather is one of those things that bedevils our senses. We have such short memories (and lives) that it is impossible from an experience standpoint to determine if the weather is warming, cooling, drying, or changing in a meaningful way. Given that the weather swings on a yearly basis over 50 degrees (and often over 30 degrees in a single day), has measured extremes 133 degrees apart, and yet exhibits a smooth yearly fluctuation of averages makes it the quintessential random distribution. As such, could any given year experience a (to our senses) long string of above or below average temps? Absolutely. In fact, as Mlodinow points out, it would be surprising if it didn't.