The Simplest Notation Possible
By Blogger
Bill James once wrote that "...baseball statistics, unlike the statistics in any other area, have acquired the power of language." We've reached the point where we can say that James's statement may be fairly applied to the two other major American sports, basketball and football. Whereas in basketball, the statistical movement is still relegated to cult website status, or covert operations at the highest levels, Football Outsiders (FO) has developed a substantial following, supplemented by a relationship with FoxSports.com. Statistically-minded football fans can also point to mainstream breakthroughs with the elevation of Paraag Marathe in the 49ers front office and in Fox's liberal use of Aikman Ratings on television broadcasts as signs of their movement's traction. However, despite those breakthroughs, advanced football statistics are still saddled with the same problems as basketball's advanced statistics, the same problem that plagued baseball's early statistical analysis:
How should numbers be presented so that the average fan can understand what they mean and appreciate them in their proper context?
In a previous phase of my life, I was a 7th grade Language Arts teacher, so I'm particularly sensitive to this issue. I will use football and for my argument, although I suspect it could easily be adjusted to fit basketball.
According the the NFL's official statistics, the "best passing team" is the one that accumulates the most yards through the air in a given season. The fallacy of such a ranking system is apparent when looking at the final results for the 2005 season. Arizona had the most passing yards. Subjective reports regarded them as mediocre at best, and they finished with a 5-11 record. It should be no surprise that Arizona finished last in team rushing yards on the season. If one looks at "total offense", though, Arizona's passing yards pump up the total so that the offense comes out looking pretty impressive, 8th out of 32 teams.
And there lies the problem with using raw yardage as the primary measuring stick. The key insight hit upon and articulated by Aaron Schatz, FO's founder, is that the success of every play in a football game is context-dependent. Three yards on one play might not be equal to three yards on another play. The basic example goes as follows:
A) On 3rd and 15 from his own 10 yard line, a running back rushes for 3 yards. His team then faces 4th and 12 from his own 13.
B) On 3rd and 2 from his own 10 yard line, a running back rushes for 3 yards. His team then gains 1st and 10 from his own 13.
Which run was successful? FO uses a proprietary formula to crunch numbers and determine how successful each play is. They then come up with numbers that they say represents a team's or a player's value and publish them on their site.
There's a big problem, though: their language stinks. I have no reason to believe that they are operating an elaborate hoax, so I don't doubt that they really are trying to spread the word about new football statistics and new ways of watching the game. However, they will not succeed the way baseball's sabermetricians have if they continue to use confusing notation in their stats.
According to FO's numbers and notation, the Indianapolis Colts led the NFL with an Offensive DVOA of 32.5%, and the Chicago Bears led the league with a Defensive DVOA of -30.7%.
First, as with any statistic and statement of fact, the reader must ask what is it actually saying, and what does it mean? DVOA is "Defense-adjusted Value Over Average", and FO gives a pretty good explanation of what Value Over Average is. However, while it might be too late to change the name, they have already set themselves back with a confusing name for the statistic by including the "Defense-adjusted" part in the defensive side of the ball. The issue is even addressed in its explanation! "Confusion alert: Originally, we called the adjusted VOA for defense something else, but finally we decided that it was better to call opponent-adjusted VOA the same thing in every instance, and most people thought 'DVOA"'just sounded better than 'OVOA' or 'AVOA' even though, yes, when we're talking about defenses the 'D' can't stand for defense. Think of it as standing for 'dependent on opponent' or something." So, the statistic's name doesn't actually describe what it is. If you have to explain that much, then you're going to turn off a good number of people who are predisposed to dismissing new ideas. I would have gone with AdVOA, pronounced "Add-Vee-Oh-Ay", for Adjusted Value Over Average.
The second major problem with DVOA is that FO insists on presenting the statistic as a percentage. One of the basic precepts of teaching -- and what FO is doing is teaching their statistics to a broad internet audience -- is that a teacher's job is easiest when the concept being taught is the most difficult part of a lesson. So, in a classroom, if I were to use a worksheet to help me conduct an activity that demonstrates a concept, I would try my damndest to write the simplest instructions possible for the activity so that students don't get hung up on that instead of focusing on the activity. If students get frustrated over instructions and details that should be inconsequential, it's only natural that they assign that frustration to the concept, when it was really that particular process of learning the concept that was frustrating. FO fails on this count by using the % symbol. It is daunting enough to face a percentage and have to make the connection of what the number is a percentage of (in this case, how much better or worse than the average player or team), but it is worse when you add the complication that, with DVOA, negative is bad and positive is good for offense, but the opposite for defense.
So, when FO says that the Chicago Bears defense led the NFL with a -30.7% DVOA, there are four symbols that the reader must work through: the negative, the number, the percentage, and the DVOA. Only two are necessary.
In order to facilitate the mainstreaming of the DVOA statistic, I propose the following change. Instead of using positives and negatives and percentages, DVOA should be converted into a single number following the convention set by baseball's OPS+ and ERA+ (examples of player stats), which sets the baseline at 100 instead of zero and dispenses with the % symbol.
The Bears' defense was 30.7% better than the average NFL defense, so my proposed notation would be 130.7 DVOA. A terrible defense, the Texans', which was listed as being the worst in the NFL with a 27.2% DVOA, would have a 72.8 DVOA.
My goal is to make the numbers friendly and easy to use. With these changes, comparisons and rankings become a matter of simple addition, and we don't have to wade through superflous symbols. Indianapolis had a total team DVOA of 36.6%? Wait, I thought negative was good for defense, so whaaaa?... Well, actually, you disregard the negative symbol in the defensive DVOA, and add offense to defense, but then you do take into account the negative symbol in the special teams number when you add it in, and you end up with
Offense + Defense + Special Teams = Total DVOA
30.5% + 10.5% + (-4.4%) = 36.6%
With my changes, that becomes
130.5 + 110.5 + 95.6 = 136.6 (336.6)
The numbers for the worst team in the NFL, the 49ers, would look like
53.1 + 78.5 + 102.4 = 34 (234)
Note that when adding the numbers together, the new baseline is 300, and I've converted it back into the 100-baseline notation.
The language is more accessible in both written and spoken form. The 49ers, as a team, were 34 DVOA, while the Colts were 136.6 DVOA. That's much better than saying "percent" every time, and mixing in the odd negative. The main problem I see with my revised notation is that, as far as explanation goes, it's yet another step to change the "average" baseline. However, I believe that when the concept has been explained soundly (as FO has done), and there are no confusing elements to the statistics, the extra step is not a problem. They are the exact same numbers, only in a more accessible and friendlier format.
How should numbers be presented so that the average fan can understand what they mean and appreciate them in their proper context?
In a previous phase of my life, I was a 7th grade Language Arts teacher, so I'm particularly sensitive to this issue. I will use football and for my argument, although I suspect it could easily be adjusted to fit basketball.
According the the NFL's official statistics, the "best passing team" is the one that accumulates the most yards through the air in a given season. The fallacy of such a ranking system is apparent when looking at the final results for the 2005 season. Arizona had the most passing yards. Subjective reports regarded them as mediocre at best, and they finished with a 5-11 record. It should be no surprise that Arizona finished last in team rushing yards on the season. If one looks at "total offense", though, Arizona's passing yards pump up the total so that the offense comes out looking pretty impressive, 8th out of 32 teams.
And there lies the problem with using raw yardage as the primary measuring stick. The key insight hit upon and articulated by Aaron Schatz, FO's founder, is that the success of every play in a football game is context-dependent. Three yards on one play might not be equal to three yards on another play. The basic example goes as follows:
A) On 3rd and 15 from his own 10 yard line, a running back rushes for 3 yards. His team then faces 4th and 12 from his own 13.
B) On 3rd and 2 from his own 10 yard line, a running back rushes for 3 yards. His team then gains 1st and 10 from his own 13.
Which run was successful? FO uses a proprietary formula to crunch numbers and determine how successful each play is. They then come up with numbers that they say represents a team's or a player's value and publish them on their site.
There's a big problem, though: their language stinks. I have no reason to believe that they are operating an elaborate hoax, so I don't doubt that they really are trying to spread the word about new football statistics and new ways of watching the game. However, they will not succeed the way baseball's sabermetricians have if they continue to use confusing notation in their stats.
According to FO's numbers and notation, the Indianapolis Colts led the NFL with an Offensive DVOA of 32.5%, and the Chicago Bears led the league with a Defensive DVOA of -30.7%.
First, as with any statistic and statement of fact, the reader must ask what is it actually saying, and what does it mean? DVOA is "Defense-adjusted Value Over Average", and FO gives a pretty good explanation of what Value Over Average is. However, while it might be too late to change the name, they have already set themselves back with a confusing name for the statistic by including the "Defense-adjusted" part in the defensive side of the ball. The issue is even addressed in its explanation! "Confusion alert: Originally, we called the adjusted VOA for defense something else, but finally we decided that it was better to call opponent-adjusted VOA the same thing in every instance, and most people thought 'DVOA"'just sounded better than 'OVOA' or 'AVOA' even though, yes, when we're talking about defenses the 'D' can't stand for defense. Think of it as standing for 'dependent on opponent' or something." So, the statistic's name doesn't actually describe what it is. If you have to explain that much, then you're going to turn off a good number of people who are predisposed to dismissing new ideas. I would have gone with AdVOA, pronounced "Add-Vee-Oh-Ay", for Adjusted Value Over Average.
The second major problem with DVOA is that FO insists on presenting the statistic as a percentage. One of the basic precepts of teaching -- and what FO is doing is teaching their statistics to a broad internet audience -- is that a teacher's job is easiest when the concept being taught is the most difficult part of a lesson. So, in a classroom, if I were to use a worksheet to help me conduct an activity that demonstrates a concept, I would try my damndest to write the simplest instructions possible for the activity so that students don't get hung up on that instead of focusing on the activity. If students get frustrated over instructions and details that should be inconsequential, it's only natural that they assign that frustration to the concept, when it was really that particular process of learning the concept that was frustrating. FO fails on this count by using the % symbol. It is daunting enough to face a percentage and have to make the connection of what the number is a percentage of (in this case, how much better or worse than the average player or team), but it is worse when you add the complication that, with DVOA, negative is bad and positive is good for offense, but the opposite for defense.
So, when FO says that the Chicago Bears defense led the NFL with a -30.7% DVOA, there are four symbols that the reader must work through: the negative, the number, the percentage, and the DVOA. Only two are necessary.
In order to facilitate the mainstreaming of the DVOA statistic, I propose the following change. Instead of using positives and negatives and percentages, DVOA should be converted into a single number following the convention set by baseball's OPS+ and ERA+ (examples of player stats), which sets the baseline at 100 instead of zero and dispenses with the % symbol.
The Bears' defense was 30.7% better than the average NFL defense, so my proposed notation would be 130.7 DVOA. A terrible defense, the Texans', which was listed as being the worst in the NFL with a 27.2% DVOA, would have a 72.8 DVOA.
My goal is to make the numbers friendly and easy to use. With these changes, comparisons and rankings become a matter of simple addition, and we don't have to wade through superflous symbols. Indianapolis had a total team DVOA of 36.6%? Wait, I thought negative was good for defense, so whaaaa?... Well, actually, you disregard the negative symbol in the defensive DVOA, and add offense to defense, but then you do take into account the negative symbol in the special teams number when you add it in, and you end up with
Offense + Defense + Special Teams = Total DVOA
30.5% + 10.5% + (-4.4%) = 36.6%
With my changes, that becomes
130.5 + 110.5 + 95.6 = 136.6 (336.6)
The numbers for the worst team in the NFL, the 49ers, would look like
53.1 + 78.5 + 102.4 = 34 (234)
Note that when adding the numbers together, the new baseline is 300, and I've converted it back into the 100-baseline notation.
The language is more accessible in both written and spoken form. The 49ers, as a team, were 34 DVOA, while the Colts were 136.6 DVOA. That's much better than saying "percent" every time, and mixing in the odd negative. The main problem I see with my revised notation is that, as far as explanation goes, it's yet another step to change the "average" baseline. However, I believe that when the concept has been explained soundly (as FO has done), and there are no confusing elements to the statistics, the extra step is not a problem. They are the exact same numbers, only in a more accessible and friendlier format.
posted by Blogger at 3:06 AM
2 Comments:
what a great post. truly great. thanks.
I enjoyed your line of thinking and look forward to seeing if FO or others take up any of your proposals. I would point out one small fly in the ointment: what if the Niners had sucked as badly on special teams as the average of their offense and defense? With a special teams index of 65.8, they would have ended up with a combined score of 197.4, or -102.6 vs. the 300 baseline for a final team AdVOA of -2.6. And, of course, the negative number is what we're trying to avoid in the statistic (never mind how pathetic a team would have to be to achieve such distinction). Perhaps we'd better leave the combined score as a "vs. 300" relative measure.
Post a Comment
<< Home