Developing talent

Today (Opening Day 2011), an excerpt from Bill James' forthcoming book Solid Fool's Gold: Detours on the Way to Conventional Wisdom appeared on the Slate site. The article is titled "Shakespeare and Verlander: Why are we so good at developing athletes and so lousy at developing writers?", and in it he provides some profound insights into discrimination in sports compared to the rest of society.

But along the way to that point, James takes a shot at the conventional wisdom that expansion dilutes the talent pool. James' contrary view is that expansion creates a short-term dilution, but over the long term more talent develops to fill the increased demand.

The thesis is built on James' assertion that raw talent is abundant, and simply needs the right opportunities -- incentives -- to be developed. In James' thought experiment, an expansion of MLB from 30 teams to 300 would over the long term have no impact on the level of talent, as talent development would expand to ensure the newly available opportunities were filled.

But can we really believe this?  There has been plenty of discussion elsewhere about the distribution of baseball talent (for example, Sabernomics and The Book), all of which would, at first glance, seem to run contrary to Bill James' argument. But those talent curves are drawn based on the current system of incentives, with enough room for 25 roster players on 30 MLB teams and roughly 9,000 players in pro ball in North America and a few more thousand around the world.

Criticisms  of Bill James' essay will no doubt focus on the fact that expanding the number of MLB teams beyond 30 requires some of the non-roster players currently in the minors to move up to The Show ... they aren't good enough to play today, but in an expansion environment they would be.

This might be true in the short term, but as Bill James argues, over the long haul the change in opportunities would shift, and talent would be developed to fill the new opportunities.

Currently around the margins of professional baseball are men who have given up baseball to work as a bartender, and those who have decided to pursue excellence in another sport. Players in both these groups would demonstrate different behaviour when provided a different set of incentives.  The shape of the distribution curve would not change, and the average player's performance would also be unchanged, but the absolute number of players would increase. 

Tom Wilhelmsen, former bartender, now pitching for the Seattle Mariners.

The latter group (the athletically gifted stars in other sports) would provide the increased numbers of players at the top end of the distribution curve, becoming the star players on Teams #31 through #300.  The bartenders of today would become the focus of rigorous development regimes. It's important to remember that not only would there be 10 times more opportunities at very level, but there would also be 10 times more teams trying to succeed, and 10 times more scouts, coaches, and others keen to see their players develop into stars. And this would be repeated around the world, ensuring that the best athletes are active in the sport that provides the greatest opportunities. Given enough time, there would be enough players developed to stock 300 teams with no decrease in overall quality of play.

There are examples of this in the past. One recent example is the growth of information technology occupations -- 40 years ago, very few individuals (both in terms of absolute numbers and as a percentage of the workforce) knew how to write a computer program. But with increased job opportunities and an expansion of training, people who might otherwise chosen other occupations and career paths now can write computer programs. This does not mean the talent pool of computer programmers has been diluted; in fact, an argument could be made that the average talent and the high-end extreme of talent has increased.

Another parallel is the availability of natural resources that lay unused until somebody found a use for it. Petroleum was known to exist for centuries, but wasn't a sought-after resource until the mid-nineteenth century when a method to distill kerosene was developed, making it a cheap alternative to whale oil. In a short period of time opportunities expanded, and as a result there was a rush to develop this previously ignored resource.

-30-

The ERA distribution curve

NOTE: Tango and MGL at The Book, through the "Lemonade" thread, have critiqued the analysis below and pointed out errors in my assumptions. These errors mean that my closing conclusions are wrong -- good math, but bad statistics on my part. Through this post, you will find italicized text describing my errors.
Revised January 5, 2011

---

J.C. Bradbury's recent blog postings (here and here) have included histograms showing the distribution of ERA across major league pitchers for the 2009 season.  For his analysis, Bradbury omitted those pitchers with fewer than 100 batters faced -- in both his blog and book Hot Stove Economics, he justifies this due to the wide variance in ERA scores, much of which will be due to the small number of "samples" for each pitcher.  (As we saw in my earlier post about Bo Hart, it's possible for an average player to do very well over the short term; the inverse applies too.)

But a few comments on Bradbury's blog from readers ask about the impact that those "missing cases", who account for nearly a third (28%) of all individuals who pitched in MLB in 2009, would have on the curve.

Here's the answer: 

Figure 1: MLB Pitching, 2009 -- Number of Pitchers by ERA, by Number of Batters Faced

Incorporating the <100 BFP pitchers (the black chunks of each bar) adds pitchers across the whole range, although they are skewed to the right (i.e. higher ERAs).  While there is a stack on the left with very low ERAs, there's a bigger group of players with an ERA greater than 10. (The highest ERA of this group was 135.00.)

NOTE 1: ERA is a poor measure to use for this type of evaluation -- for pitchers with a low number of batters faced or innings pitched, it's easy for huge numbers to appear. That 135.00 ERA is the equivalent of 15 earned runs with only a single recorded out.  These exaggerated values then lead to an upward distortion of the mean for the group.  A better measure would be wOBA, or other measure that resembles a probability between 0 and 1.

The table below shows the average ERA of this group and three other groups based on the number of batters faced.  What we see is that the <100 BFP pitchers have a higher ERA than those who pitched more frequently.  (This difference is statistically significant.)  In spite of the variation in their ERAs, this group on average are less skilled than the other three groupings of pitchers.

NOTE 2: This is where I went wrong. The math is correct, but there is bias in the sample that I ignored. We can be fairly confident that pitchers who get off to a poor start won't get many opportunities to pitch -- and therefore won't get the opportunity to regress to the mean. Pitchers who do better at the start of their season will continue to pitch, and regress to the mean.  This process may take them some time, which may push them over the arbitrary line of 100 batters faced.  Thus the statistical significance is an artifact of the bias.

Figure 2: MLB Pitching, 2009 -- Average ERA, by Number of Batters Faced

In a thread on The Book blog that covered this same topic, I made a similar statement (reply #8): "What I’m trying to say is that our best estimate of the “true talent” of this group is an ERA of 8.11 [in the current case, 8.72], and that estimate is quite accurate". That statement got a response from Tango (reply #9) of "That is not accurate. If you look at how those pitchers who faced fewer than 100 batters did in the season preceding or the season following, THAT will give you a much better indicator of the true talent level."

So let me clarify.  The average level of skill of the pitchers who faced fewer than 100 batters in 2009, is an average ERA of 8.72. Although Tango is correct in his assertion that the poorest performers would regress upwards, by the same token the best pitchers (some of whom managed a 0.00 ERA in their short stint) would get worse. But if we were to let all 188 of them continue to pitch, we can be 95% certain that the "true" ERA of the group would end up somewhere between 6.92 and 10.52.

Even the lower bound (i.e. the lowest score we would expect with our more rigorous testing) is higher than the highest range from the other groups.

NOTE 3:  My statement above would be correct, if it were not for the bias in the sample.  My belief had been that this group would regress not to the MLB average, but to the average of the <100 BFP pitchers.  But because of the selection bias, this does not hold true.
Here's a simple example to demonstrate how this works. Think of the probability professor's favourite tool, the coin toss. If we have a penny and toss it repeatedly -- say, 10,000 times -- and recorded the result each time, the proportion of heads would very accurately reflect the true probability of the individual penny. And we'd need plenty of tosses to get an accurate measure of the single penny.

But what if instead of one penny we had 188 pennies, and we varied the number of tosses each penny got? Although the average number of tosses would be 50, some pennies might get only one toss, while others would get as many as 100 tosses. Some of those short sequences might come up all heads, while others would heavily favour the tails. On average, though, across the 188 pennies, we would find that the group average was a close reflection of "true average" of the group.

NOTE 4: The error in the initial assumption causes my coin flipping analogy to fall apart.  If “success” is a head, then the coin that comes up heads >0.5 will keep being flipped, possibly with enough flips to no longer be part of the “low flip” group (over that arbitrary threshold).  Meanwhile, a coin that runs tails more often will get pulled from the trials quickly, and end up <0.5 and with few flips.  Thus, as a group, the coins with a smaller number of flips will end up looking worse than those that keep getting flipped.  Selection bias causes an apparent difference, where none really exists.


And so it is with the pitchers in question. If they were like the other pitchers in MLB, we would expect that some of the <100 batters faced pitchers would have ERAs above the league average, while others would fall below. What we see, however, is that while there is a wide variation, the average is substantially higher than the other groups of pitchers.

NOTE 5:  ...because of selection bias!  The lesson:  selection bias can crop up anywhere, even if you are not the one doing the selecting.

-30-

Agreeing with Bill James

In 1988, the Bill James Abstract included "A Bill James Primer", with 15 statements expressing what he deemed to be useful knowledge. On that list was:
2. Talent in baseball is not normally distributed. It is a pyramid. For every player who is 10 percent above the average player, there are probably twenty players who are 10 percent below average.


I agree. (Others don't; for further discussion also see here.)


But what is this thing called "talent"? Talent is a combination of a high level of skill and sustained, consistent performance. Skill in baseball is measured through metrics such as ERA (earned run average) and OPS (on-base average plus slugging percentage) -- measures that turn counting stats into an efficiency or rate measure. While this type of measure is important, they fail to account for the fact that some players have lengthy careers, while other players have a very short MLB career. Teams will sign long-term contracts with aging superstars because the player's skill is still above average, even though they may have diminished with age.


In short, career length becomes a valid proxy for talent.


The charts below plot the number of pitchers over the period 1996-2009, by both the number of games played (which favours the relief pitchers) and innings pitched (which favours the starters). During this period a total of 2,134 individuals pitched in MLB -- but the chart shows that very few of them stuck around for any length of time.


At the head of the "games" list at 898 is the still-active Mariano Rivera, while the pitcher with the most innings over this period was Greg Maddux (2887.67 innings; and Maddux threw more than 2,100 innings before 1996, as well). These two individuals, and other Hall of Fame calibre pitchers, are out at the far right of the long tail. Close to the origin at the left are pitchers whose entire career lasted but 1/3 of an inning -- a single out.
Figure 1: Number of Pitchers, by Career Innings Pitched (1996-2009)




Figure 2: Number of Pitchers, by Career Games (1996-2009)


But what of the average skill level of those pitchers? Pitchers who get a small amount of MLB experience (fewer than 27 innings) have a higher ERA than those who get more opportunities to pitch. This group -- 27% of all MLB pitchers -- recorded an average ERA of 8.08, compared to 5.15 for the 42% who pitched between 27 to 269 innings, and 4.45 for the 27% who threw between 270 and 1349 innings. The elite, those who pitched 1350 innings and above, recorded the lowest ERA of all, 4.17.

In spite of the wide variance in the ERAs of the coffee drinkers, the differences in the mean scores are statistically significant.



Figure 3: MLB Pitchers, average ERA, by number of innings pitched (1996-2009)

In summary: there is an abundance of players who are less talented than the major league average, while at the same time the number of above-average talents is low. The distribution, at the major league level, is not normal. Just like Bill James said 22 years ago.

-30-