I've had cause recently to recall a philosophy course I took in college. I remember one particular day, when, during our daily topic discussion / arguments, one courageous young woman took the unusual position that killing was wrong. I say courageous because, despite strong opposition from a majority of the class, she held to her view.
The opposition didn't really dispute her stance - nobody actually argued that killing was right. But, the argument against her was that she was unwilling to make exceptions, especially exceptions for the state. And, it seemed, many felt that killing wasn't wrong in the same way for the state.
I was mostly an observer that day. My strongest prior influence on the topic had been Justice Van Pelt's assertion that state execution should be reserved for professional assassins - it constituted an occupational hazard. In all other instances, the state should refrain.
But as I listened to the arguments in class that day, it occurred to me how strange it was that something we would abhor, something we would expressly forbid individuals from doing, we would grant authority to the impersonal body of the state to do.
I hear the same argument playing out again today. There are those who take the unusual position that violence, especially violence conducted expressly to advance one's goals or ends, is wrong. They argue that there shouldn't be exceptions - that the state shouldn't retain for itself the right to commit violence in our name without our prior, specific approval.
And then the opposition, yesterday articulated by former Vice President Dick Cheney, arguing that the state application of violence to obtain information is the best means we have, that it saves lives, and that our goal of saving lives justifies the state application of violence in secret. Today, I predict that the violence apologists will spring up across the news spectrum, taking up the call for society's approval of more state violence.
At least Justice Van Pelt's position had a certain logic: To be eligible for state execution, you had to be demonstrably employed in a specific profession, and you had to have been convicted of a very specific crime.
But the calls to allow state sponsored torture lack any of that - there is no requirement for specific actions, no requirement for conviction of a specific crime. Without very specific criteria - criteria that prevent abuse, that prevent it from becoming nothing more than some individuals using the state as cover to explore their personal desire to perpetrate violence, then it becomes just an acceptance of violence in society. It begins to take on overtones of racism and xenophobia - since we have no clear guidelines to its application, it will be applied to individuals based upon other individuals beliefs and fears and preconceptions. Surely a recipe for abuse.
Will we support Attorney General Holder's investigation into torture abuse? Will we pressure our representatives for stronger laws prohibiting and regulating torture?
Or will we be left with the sad conclusion that in America taking the position that violence is wrong is the unusual one?
Monday, August 31, 2009
Friday, August 28, 2009
Market Value
One of our cherished ideas about the market is that it properly (and efficiently) sets the value of items. We even believe that it properly values people: Whether someone is making $10,000, $100,000, or $1,000,000, we often contend that the market has accurately determined their value to society, and they are being paid appropriately.
Leaving aside whether that is true or not, let’s poke a little at some of the ramifications. One observation that we can make is that it is the current value – the market does not attempt (with a couple of possible exceptions) to predict our future value. In other words, our current pay reflects our current value to our current employer – nothing more.
A co-worker made the assertion the other day that people who are poorly valued by the market have no business demanding Health Care. His idea was simply that if they hadn’t done what it takes to be highly valued by the market, and since the market reflects value to society, society is proper in denying them health care – we don’t get a good return on our investment by fixing them up.
But, that only reflects current value. We have no idea (and the market makes no predictions) about their possible future value. Most individuals, upon leaving High School, are relatively unemployable. Gradually, through time, they find a trade, complete college, and raise their wage.
Fortunately, most young people have good health. But, if they were to become sick, or injured: Can we take their current position in society, as determined by their wage, to fairly determine whether they deserve Health Care? Do we know that they will never amount to more, and our judgment that they should have only a bare minimum or face a lifetime attempting to pay their medical debts is fair?
Consider this, too: When CEO’s make mistakes, when those we’ve judged (by their salary only) to have high worth to society over extend their company: It is the rank and file worker who takes the brunt of the punishment, and loses their job, and frequently any Health Insurance / Care.
Did the worker’s value to society actually go to zero at the moment they lost their job?
I think we would say not – and the very act of denying the truth of this statement calls into question the earlier assertion that markets accurately value people.
And if markets don’t accurately value people, the idea that we can use a market valuation to determine eligibility for Health Care is also suspect.
Leaving aside whether that is true or not, let’s poke a little at some of the ramifications. One observation that we can make is that it is the current value – the market does not attempt (with a couple of possible exceptions) to predict our future value. In other words, our current pay reflects our current value to our current employer – nothing more.
A co-worker made the assertion the other day that people who are poorly valued by the market have no business demanding Health Care. His idea was simply that if they hadn’t done what it takes to be highly valued by the market, and since the market reflects value to society, society is proper in denying them health care – we don’t get a good return on our investment by fixing them up.
But, that only reflects current value. We have no idea (and the market makes no predictions) about their possible future value. Most individuals, upon leaving High School, are relatively unemployable. Gradually, through time, they find a trade, complete college, and raise their wage.
Fortunately, most young people have good health. But, if they were to become sick, or injured: Can we take their current position in society, as determined by their wage, to fairly determine whether they deserve Health Care? Do we know that they will never amount to more, and our judgment that they should have only a bare minimum or face a lifetime attempting to pay their medical debts is fair?
Consider this, too: When CEO’s make mistakes, when those we’ve judged (by their salary only) to have high worth to society over extend their company: It is the rank and file worker who takes the brunt of the punishment, and loses their job, and frequently any Health Insurance / Care.
Did the worker’s value to society actually go to zero at the moment they lost their job?
I think we would say not – and the very act of denying the truth of this statement calls into question the earlier assertion that markets accurately value people.
And if markets don’t accurately value people, the idea that we can use a market valuation to determine eligibility for Health Care is also suspect.
Tuesday, August 25, 2009
More Means and Medians
There are a couple of threads that I wanted to follow from my previous entry on Means and Medians. One, of course, relates to the weather. The other is this one, on income.
There are (roughly) 300 million people in America. If you take our Gross Domestic Product (GDP) of $14 trillion dollars, and divide by the population, you arrive at a per-capita income of approximately $45,000. We take that value and compare it with the per-capita income of other countries to get a sense of how well off we are – are we rich, or not?
And, by that measure, Americans are rich. A family of four's share of the output of America is $190,000. We're swimming in money, at least according to the mean.
Of course, that's just a measure of how we have to spread our output around. Unless they are extraordinary, most children don't work, and the majority of our seniors are retired. So a more interesting mean might be the amount of our GDP generated by each worker.
According to the census bureau, there are 138 million employed workers. (So, there are several million more potential workers...). That's a convenient number, because it makes the math easy: The average share of the output (the worker per-capita) is $100,000.
A normal distribution about the average would have 50% of American workers making more than $100,000 a year, and 50% making less. If it were fairly tightly bound to the mean, we would be able to say that American's are quite well off. But is that true?
Again, using the census bureau's numbers, the median household income (the mid-point) is $50,000. And since there are more income earners than households, the median worker income is lower still: $45,000 for men, $35,000 for women, and $41,000 overall.
In fact, the top quintile of all households starts at $91,000. So, less than 20% of American households make the per-worker average. And, since to get into the top quintile 77% of the households had two wage earners, that drops the number of workers who approach the national average income to just under 5%.
95% of all American workers make less than the per-capita value!
Therefore, for determining the actual wealth of the 'average' American, neither the per-capita value nor the per-worker share tells us much useful. Much better is the experience of the median family of 4, with an individual share of just $12,500. That's the experience of most of America. Not $45,000.
Now, economists are a pretty smart lot, and they understand that the difference between the mean and the median income in a country has meaning to the people of that country, and to what you can say about it. To help them, they use the Gini Coefficient (read wikipedia for a fuller account) – but it basically tells them the relative distance between the median and the mean.
Ours is a relatively high 45. Compare that with the bulk of Europe (34 for GB and Switzerland, low 30's for Canada, Ireland and Spain, 28 for France, Belgium, Hungary, Germany and Norway). We're only low compared to Argentina (49), Sri Lanka (50), El Salvador (52), Panama (56), Brazil (57) – and I'll leave it to the reader to consider what those nations have in common.
There's a lot more that could be said and investigated. For instance, what is the typical experience of a worker: Do they start low and progress across the median during their career, or do they remain on one side or the other? Where do our elderly lie? (I think we know: Most are below...)
But to maintain a semblance of brevity, I'll just leave you with this thought. Consider how this large income inequality is the gorilla in the room that silently influences every socio-economic discussion we have: From Education to Health Care, from Taxation to Retirement – the experience of the majority of Americans does not match the experience projected by the statistical mean, nor the experience of the planners and policy makers.
Once again, the mean and the median paint very different pictures. The mean says that we are universally rich.
The median exposes the lie.
There are (roughly) 300 million people in America. If you take our Gross Domestic Product (GDP) of $14 trillion dollars, and divide by the population, you arrive at a per-capita income of approximately $45,000. We take that value and compare it with the per-capita income of other countries to get a sense of how well off we are – are we rich, or not?
And, by that measure, Americans are rich. A family of four's share of the output of America is $190,000. We're swimming in money, at least according to the mean.
Of course, that's just a measure of how we have to spread our output around. Unless they are extraordinary, most children don't work, and the majority of our seniors are retired. So a more interesting mean might be the amount of our GDP generated by each worker.
According to the census bureau, there are 138 million employed workers. (So, there are several million more potential workers...). That's a convenient number, because it makes the math easy: The average share of the output (the worker per-capita) is $100,000.
A normal distribution about the average would have 50% of American workers making more than $100,000 a year, and 50% making less. If it were fairly tightly bound to the mean, we would be able to say that American's are quite well off. But is that true?
Again, using the census bureau's numbers, the median household income (the mid-point) is $50,000. And since there are more income earners than households, the median worker income is lower still: $45,000 for men, $35,000 for women, and $41,000 overall.
In fact, the top quintile of all households starts at $91,000. So, less than 20% of American households make the per-worker average. And, since to get into the top quintile 77% of the households had two wage earners, that drops the number of workers who approach the national average income to just under 5%.
95% of all American workers make less than the per-capita value!
Therefore, for determining the actual wealth of the 'average' American, neither the per-capita value nor the per-worker share tells us much useful. Much better is the experience of the median family of 4, with an individual share of just $12,500. That's the experience of most of America. Not $45,000.
Now, economists are a pretty smart lot, and they understand that the difference between the mean and the median income in a country has meaning to the people of that country, and to what you can say about it. To help them, they use the Gini Coefficient (read wikipedia for a fuller account) – but it basically tells them the relative distance between the median and the mean.
Ours is a relatively high 45. Compare that with the bulk of Europe (34 for GB and Switzerland, low 30's for Canada, Ireland and Spain, 28 for France, Belgium, Hungary, Germany and Norway). We're only low compared to Argentina (49), Sri Lanka (50), El Salvador (52), Panama (56), Brazil (57) – and I'll leave it to the reader to consider what those nations have in common.
There's a lot more that could be said and investigated. For instance, what is the typical experience of a worker: Do they start low and progress across the median during their career, or do they remain on one side or the other? Where do our elderly lie? (I think we know: Most are below...)
But to maintain a semblance of brevity, I'll just leave you with this thought. Consider how this large income inequality is the gorilla in the room that silently influences every socio-economic discussion we have: From Education to Health Care, from Taxation to Retirement – the experience of the majority of Americans does not match the experience projected by the statistical mean, nor the experience of the planners and policy makers.
Once again, the mean and the median paint very different pictures. The mean says that we are universally rich.
The median exposes the lie.
Monday, August 10, 2009
The Market
I am a software engineer. Recently, I've been able to work on the software for autonomous, collaborative, distributed systems.
When a problem grows too large for a single computer to solve because of its complexity, a distributed system often provides a means of gaining the solution: Several to hundreds of computers are pulled into the task. Each computer churns on a portion of the problem, adding its partial solution to the whole, and then the partials are aggregated into a final.
An autonomous system is one where we add rules and logic to the individual computers so that, rather than my choosing as a programmer which parts of the problem will be solved on a computer, the program running there is allowed to survey the problem space, and select where it will search for a solution. We call such systems collaborative when they are allowed to communicate, and base their decisions not just on the problem at hand, but which parts are seen to be worked on by other computers.
We code up the rules and logic, compile, and then place an identical program on several computers, and start them up to see if they can find a solution. Often they do, and often very quickly. However, compared to a single system with its single set of logic and rules, sometimes our distributed system is much, much faster, but has found a local solution, rather than an optimal solution. Whether that is acceptable depends on the nature of the problem we are attempting to solve.
Sometimes, these systems surprise us. We've coded the rules, we've devised the logic, we've predicted just how they'll work together, and how quickly and efficiently they'll search the solution space. They don't always behave that way. Interactions we didn't foresee lead to solutions we didn't anticipate.
And we've coined a term for this: Emergent Behavior.
Sometimes, the emergent behavior opens up new possibilities, new avenues to explore. We see that a system we started developing to solve one problem, with some adjustment, can be made to solve another. Maybe two problems can be combined, and solutions found for both.
And sometimes, the emergent behavior is just bad. Rather than solutions, unanticipated interactions create loops, the same section of the problem is revisited again and again, with no progress. We go back to our cubes, change the rules, re-code the logic, adjust our algorithms, and try again.
We devised these systems out of analogy. Some say that it was a programmer's fascination with ants that lead to the first, others claim bee behavior lead to the insight. It doesn't really matter. We pull from research on not only insects, but animal and even human behavior for our inspiration.
But, it turns out that the analogy works both ways. We are all embedded in an autonomous, distributed, collaborative system in real life. We each work and play, earn money, and consume based upon an individual rule set – need, peer, and culture driven. Collectively, we participate economically, and the emergent behavior of all of our economic decisions and actions has been given a word: The Market.
In a sense, 'The Market' does not exist. It is not an entity to which you can appeal, it doesn't appear in nature. 'The Market' is just a term for the end result of human economic activity.
Like my programs, where the emergent behavior depends upon the rules, algorithms, and logic that I have coded, so too, 'The Market' depends upon the conditions of society in which it emerges: The laws that constrain possibility, the culture that helps determine value, the beliefs that influence individual behavior. Change any one, and the emergent behavior, 'The Market', changes.
Additionally, there is no reason to believe that the market is efficient or optimum. My distributed programs are neither. They use much more resources and energy than a conceivable single system would – the caveat is that we cannot yet build a single computer anywhere near to powerful enough to eliminate the inefficiencies. We see these same economic inefficiencies every day. Does everyone get the precise shoe they want? No. Does everyone get a shoe? No. Are there shoes manufactured that are discarded without a user? Yes.
Am I condemning 'The Market'? No. I'm just pointing out that it is nothing which we should approach on bended knee, that it is nothing 'out there' about which we cannot change. Nothing grates on me more than when an inefficiency (or worse, an inequality) is discovered in our society, and in response to proposals to ameliorate the inefficiency or inequality, someone cries “No! Let the Market Decide.”
'The Market' decides nothing. 'The Market' gives us nothing. 'The Market' is the end result, not a functioning intermediary.
Does the insight that ‘The Market’ is an outcome provide us with anything useful?
I think it does.
First, it ascribes morality to the outcome. There are those who claim that markets are impersonal, unbiased, and amoral; that market outcomes should not be morally judged. But if we recognize that the market is the end result of laws, culture, and beliefs, each of which is infused with our morals, then we can understand that the outcome can be judged against our moral standards, and, if found wanting, should be modified. (Those who appeal to the amoral market are often seeking cover for the furthering of their own self-interest. But, the intersection and balance between self-interest and group interest is exactly what morality is!)
Second, it empowers us to attempt modification. By understanding that ‘The Market’ is an outcome of our making, not some inviolate entity, we realize that we can imagine and then seek a better outcome. We’ll no longer allow ourselves to be bullied into inaction by those who seek to retain the status quo under the guise of some fictional entity.
Thirdly, it removes some of the mythology from ‘The Markets’ entity. When one speaks of rational markets, or efficient markets, we can see from analogy that they are discussing something that exists only in freshman econ textbooks, and nowhere in the real world. For in the real world, markets are made up of inefficient, sometimes irrational agents with incomplete knowledge and conflicting desires. How either rationality or efficiency can be ascribed to such a mess is beyond me. It requires heroic assumptions that just aren’t true. (It’s just like the physics point mass – frictionless surface problem that never occurs: useful for gleaning some understanding, but completely trivial for modeling anything in the real world.)
And, just as my computer programs aren’t useful for attempting every problem, so economics and the emergent ‘Market’ isn’t appropriate for every human problem, either. Our biggest challenge is to determine which problems we can tackle economically, and then collectively determine what attributes the solution should have (justice, equality, fairness, reward), and in what balance, and then work the structure towards such a solution.
When a problem grows too large for a single computer to solve because of its complexity, a distributed system often provides a means of gaining the solution: Several to hundreds of computers are pulled into the task. Each computer churns on a portion of the problem, adding its partial solution to the whole, and then the partials are aggregated into a final.
An autonomous system is one where we add rules and logic to the individual computers so that, rather than my choosing as a programmer which parts of the problem will be solved on a computer, the program running there is allowed to survey the problem space, and select where it will search for a solution. We call such systems collaborative when they are allowed to communicate, and base their decisions not just on the problem at hand, but which parts are seen to be worked on by other computers.
We code up the rules and logic, compile, and then place an identical program on several computers, and start them up to see if they can find a solution. Often they do, and often very quickly. However, compared to a single system with its single set of logic and rules, sometimes our distributed system is much, much faster, but has found a local solution, rather than an optimal solution. Whether that is acceptable depends on the nature of the problem we are attempting to solve.
Sometimes, these systems surprise us. We've coded the rules, we've devised the logic, we've predicted just how they'll work together, and how quickly and efficiently they'll search the solution space. They don't always behave that way. Interactions we didn't foresee lead to solutions we didn't anticipate.
And we've coined a term for this: Emergent Behavior.
Sometimes, the emergent behavior opens up new possibilities, new avenues to explore. We see that a system we started developing to solve one problem, with some adjustment, can be made to solve another. Maybe two problems can be combined, and solutions found for both.
And sometimes, the emergent behavior is just bad. Rather than solutions, unanticipated interactions create loops, the same section of the problem is revisited again and again, with no progress. We go back to our cubes, change the rules, re-code the logic, adjust our algorithms, and try again.
We devised these systems out of analogy. Some say that it was a programmer's fascination with ants that lead to the first, others claim bee behavior lead to the insight. It doesn't really matter. We pull from research on not only insects, but animal and even human behavior for our inspiration.
But, it turns out that the analogy works both ways. We are all embedded in an autonomous, distributed, collaborative system in real life. We each work and play, earn money, and consume based upon an individual rule set – need, peer, and culture driven. Collectively, we participate economically, and the emergent behavior of all of our economic decisions and actions has been given a word: The Market.
In a sense, 'The Market' does not exist. It is not an entity to which you can appeal, it doesn't appear in nature. 'The Market' is just a term for the end result of human economic activity.
Like my programs, where the emergent behavior depends upon the rules, algorithms, and logic that I have coded, so too, 'The Market' depends upon the conditions of society in which it emerges: The laws that constrain possibility, the culture that helps determine value, the beliefs that influence individual behavior. Change any one, and the emergent behavior, 'The Market', changes.
Additionally, there is no reason to believe that the market is efficient or optimum. My distributed programs are neither. They use much more resources and energy than a conceivable single system would – the caveat is that we cannot yet build a single computer anywhere near to powerful enough to eliminate the inefficiencies. We see these same economic inefficiencies every day. Does everyone get the precise shoe they want? No. Does everyone get a shoe? No. Are there shoes manufactured that are discarded without a user? Yes.
Am I condemning 'The Market'? No. I'm just pointing out that it is nothing which we should approach on bended knee, that it is nothing 'out there' about which we cannot change. Nothing grates on me more than when an inefficiency (or worse, an inequality) is discovered in our society, and in response to proposals to ameliorate the inefficiency or inequality, someone cries “No! Let the Market Decide.”
'The Market' decides nothing. 'The Market' gives us nothing. 'The Market' is the end result, not a functioning intermediary.
Does the insight that ‘The Market’ is an outcome provide us with anything useful?
I think it does.
First, it ascribes morality to the outcome. There are those who claim that markets are impersonal, unbiased, and amoral; that market outcomes should not be morally judged. But if we recognize that the market is the end result of laws, culture, and beliefs, each of which is infused with our morals, then we can understand that the outcome can be judged against our moral standards, and, if found wanting, should be modified. (Those who appeal to the amoral market are often seeking cover for the furthering of their own self-interest. But, the intersection and balance between self-interest and group interest is exactly what morality is!)
Second, it empowers us to attempt modification. By understanding that ‘The Market’ is an outcome of our making, not some inviolate entity, we realize that we can imagine and then seek a better outcome. We’ll no longer allow ourselves to be bullied into inaction by those who seek to retain the status quo under the guise of some fictional entity.
Thirdly, it removes some of the mythology from ‘The Markets’ entity. When one speaks of rational markets, or efficient markets, we can see from analogy that they are discussing something that exists only in freshman econ textbooks, and nowhere in the real world. For in the real world, markets are made up of inefficient, sometimes irrational agents with incomplete knowledge and conflicting desires. How either rationality or efficiency can be ascribed to such a mess is beyond me. It requires heroic assumptions that just aren’t true. (It’s just like the physics point mass – frictionless surface problem that never occurs: useful for gleaning some understanding, but completely trivial for modeling anything in the real world.)
And, just as my computer programs aren’t useful for attempting every problem, so economics and the emergent ‘Market’ isn’t appropriate for every human problem, either. Our biggest challenge is to determine which problems we can tackle economically, and then collectively determine what attributes the solution should have (justice, equality, fairness, reward), and in what balance, and then work the structure towards such a solution.
Tuesday, August 4, 2009
Means, Medians, and the Weather
Today, a little math.
Take the sequence of numbers: 11, 12, 13, 14, 15, 16, 17, 18, 19.
The average of these numbers is 15. So is the mid-point, or median. If these numbers were in a bag, and you were to draw one out, there is equal probability that the number you drew would be above or below 15 (and a 1/9 chance that it would be 15 exactly).
If you had a very large bag, and there were equal number of duplicates of each, and you were to draw enough times (say, 400 or so), you would likely end up with a very even distribution, 44 occurrences of each. The average would be unchanged.
Now, in that same bag, let’s remove all of the 11’s, and replace them with 1’s. We’ll do the same 400 draws, but what happens to the average and mid-point?
The average drops to 13.9. However, the median remains unchanged at 15. (The sequence is now: 1, 12, 13, 14, 15, 16, 17, 18, 19). 15 is equidistant from each end.
Which means that the probability of drawing a number greater than or less than 15 remains unchanged – for every number that you draw that turns out to be larger than 15, you will probably draw a counterpart that is less than 15.
But what if, instead of telling you the sequence of numbers in the bag and the median, I just told you that I have a bag of numbers in which the average, over all of the numbers in the bag, is 13.9. Then I invite you to draw 5.
The chance that you will draw numbers greater than 13.9 is very large – 6/9, or 2/3’s, to be exact. And by drawing only 5, there is a chance that you won’t actually draw any less than the average.
So, your perception of the range of numbers in the bag would be skewed. If you were to think about it, you would properly discern that there must be some numbers less than 13.9 in the bag, and if I were to continue to allow you to draw, you might even propose (or draw!) a very low number (the 1).
Now, to an application of exactly this phenomenon.
The average temperature (as published by the NWS) in Denver in late July / early August is 88 degrees.
So, what is the chance that any given day during this period is greater than 88 degrees? Is a 91 degree day ‘above normal’?
You get points if you realized that you can’t answer the question with the data given.
You get extra points if you realized that when our local weather people come on the TV and tell you that ‘it’s going to be a few degrees above normal today, with the expected high of 91 degrees’, they are being foolish, and possibly mis-informing you.
I got to wondering, especially last year, when it seemed that a 90+ degree day in the summer was more common than a 90- degree day. I contacted one of our local weathermen, and asked what the spread of the first standard deviation was on our weather, figuring that would actually tell us much more information about what constituted an abnormally hot (or cold) day. (The first standard deviation encompasses 2/3 of the data – if it were from, say 85 to 95 degrees, you could pretty accurately predict that most of the time, the summer day-time high would be in that range, and an abnormal, by this measure, day would be one above 95 degrees, which would constitute just 1/6 of all summer days.)
His reply astounded me: “We don’t get that information from the NWS – just the daily averages and extremes. BTW, the averages are computed every 10 years over the previous 30, so currently we are using the data collected from 1971 – 2000.”
Well. There is nothing left for an inquisitive person to do but to look at the data directly. Fortunately, the daily recorded highs and lows for the last 30 years can be had from the NWS site. So, I went to work.
And what I found was interesting. Our weather more closely represents the second sequence than it does the first. Rather than a smooth distribution about the average, it has regular, but infrequent, extremely low temps (in the 70’s!). The median temperature, as a result, is about 3.5 degrees above the average. It works out that just shy of 2/3’s of all summer daytime temps are above the published average!
Imagine how this alters your perception of the region’s weather. If you knew that the median temperature was 91.4 degrees, you would know to expect that, given enough days and years, half of the summer day-time highs would be 92 or above, and you wouldn’t be surprised at a string of 93-95 degree days. (Conversely, you also wouldn’t be surprised by an equal string of 87-89 degree days.) And if you knew the full variability of the weather, you also wouldn’t be surprised by a string of 80-82 degree days, or the occasional string of 97-99 degree days.
There is a more important point buried in all of this. The computation and use of means and medians is elementary, in the sense that we all encounter it prior to secondary (9-12) school. Our teachers take the time to create examples like this to illustrate how they differ, although the real meaning (and mismeaning) is not fully explored until statistics, usually in college. But, a rough understanding is vital to our accurate perception of the world around us, and the data presented.
And, we expect people who have a vested interest in molding our perceptions (think politicians and lobbyists) would leave one or the other out.
But, our weather people, and the NWS? What do they have to gain, other than laziness? They are all college educated, they certainly had college statistics as part of their degree program, and yet – they think nothing of going on TV each and every evening and misleading us about the ‘normalness’ of today’s weather.
Which really illustrates just how vigilant we must be whenever numbers and terms like average, median, and deviation are used. As has been said, the worst lies are statistical lies, which we can see are often lies of omission – omission of large parts of the relevant data.
Take the sequence of numbers: 11, 12, 13, 14, 15, 16, 17, 18, 19.
The average of these numbers is 15. So is the mid-point, or median. If these numbers were in a bag, and you were to draw one out, there is equal probability that the number you drew would be above or below 15 (and a 1/9 chance that it would be 15 exactly).
If you had a very large bag, and there were equal number of duplicates of each, and you were to draw enough times (say, 400 or so), you would likely end up with a very even distribution, 44 occurrences of each. The average would be unchanged.
Now, in that same bag, let’s remove all of the 11’s, and replace them with 1’s. We’ll do the same 400 draws, but what happens to the average and mid-point?
The average drops to 13.9. However, the median remains unchanged at 15. (The sequence is now: 1, 12, 13, 14, 15, 16, 17, 18, 19). 15 is equidistant from each end.
Which means that the probability of drawing a number greater than or less than 15 remains unchanged – for every number that you draw that turns out to be larger than 15, you will probably draw a counterpart that is less than 15.
But what if, instead of telling you the sequence of numbers in the bag and the median, I just told you that I have a bag of numbers in which the average, over all of the numbers in the bag, is 13.9. Then I invite you to draw 5.
The chance that you will draw numbers greater than 13.9 is very large – 6/9, or 2/3’s, to be exact. And by drawing only 5, there is a chance that you won’t actually draw any less than the average.
So, your perception of the range of numbers in the bag would be skewed. If you were to think about it, you would properly discern that there must be some numbers less than 13.9 in the bag, and if I were to continue to allow you to draw, you might even propose (or draw!) a very low number (the 1).
Now, to an application of exactly this phenomenon.
The average temperature (as published by the NWS) in Denver in late July / early August is 88 degrees.
So, what is the chance that any given day during this period is greater than 88 degrees? Is a 91 degree day ‘above normal’?
You get points if you realized that you can’t answer the question with the data given.
You get extra points if you realized that when our local weather people come on the TV and tell you that ‘it’s going to be a few degrees above normal today, with the expected high of 91 degrees’, they are being foolish, and possibly mis-informing you.
I got to wondering, especially last year, when it seemed that a 90+ degree day in the summer was more common than a 90- degree day. I contacted one of our local weathermen, and asked what the spread of the first standard deviation was on our weather, figuring that would actually tell us much more information about what constituted an abnormally hot (or cold) day. (The first standard deviation encompasses 2/3 of the data – if it were from, say 85 to 95 degrees, you could pretty accurately predict that most of the time, the summer day-time high would be in that range, and an abnormal, by this measure, day would be one above 95 degrees, which would constitute just 1/6 of all summer days.)
His reply astounded me: “We don’t get that information from the NWS – just the daily averages and extremes. BTW, the averages are computed every 10 years over the previous 30, so currently we are using the data collected from 1971 – 2000.”
Well. There is nothing left for an inquisitive person to do but to look at the data directly. Fortunately, the daily recorded highs and lows for the last 30 years can be had from the NWS site. So, I went to work.
And what I found was interesting. Our weather more closely represents the second sequence than it does the first. Rather than a smooth distribution about the average, it has regular, but infrequent, extremely low temps (in the 70’s!). The median temperature, as a result, is about 3.5 degrees above the average. It works out that just shy of 2/3’s of all summer daytime temps are above the published average!
Imagine how this alters your perception of the region’s weather. If you knew that the median temperature was 91.4 degrees, you would know to expect that, given enough days and years, half of the summer day-time highs would be 92 or above, and you wouldn’t be surprised at a string of 93-95 degree days. (Conversely, you also wouldn’t be surprised by an equal string of 87-89 degree days.) And if you knew the full variability of the weather, you also wouldn’t be surprised by a string of 80-82 degree days, or the occasional string of 97-99 degree days.
There is a more important point buried in all of this. The computation and use of means and medians is elementary, in the sense that we all encounter it prior to secondary (9-12) school. Our teachers take the time to create examples like this to illustrate how they differ, although the real meaning (and mismeaning) is not fully explored until statistics, usually in college. But, a rough understanding is vital to our accurate perception of the world around us, and the data presented.
And, we expect people who have a vested interest in molding our perceptions (think politicians and lobbyists) would leave one or the other out.
But, our weather people, and the NWS? What do they have to gain, other than laziness? They are all college educated, they certainly had college statistics as part of their degree program, and yet – they think nothing of going on TV each and every evening and misleading us about the ‘normalness’ of today’s weather.
Which really illustrates just how vigilant we must be whenever numbers and terms like average, median, and deviation are used. As has been said, the worst lies are statistical lies, which we can see are often lies of omission – omission of large parts of the relevant data.
Subscribe to:
Posts (Atom)