A New Approach to Which NHL Teams Draft Best? (1999-2005)

Many people have conducted studies of the NHL Draft (Daoust over at Pension Puppet Plans has a big one), but everyone runs up against some fundamental problems.

• If you use Games Played as your metric of draft success, that metric assumes that a game by Evgeni Malkin counts the same level as a game by Chris Thorburn--clearly one is more valuable than the other.
• Not all draft years are as strong as other years, making comparisons across multiple years more challenging.
• Every team in the draft faces a completely unique situation--their choices are altered by the teams selecting players in front of them.
  
• Not all teams have the same number of picks in a given draft year.
  

At the risk of sounding immodest, I think I've found a solution to all (or most) of these challenges. I call it the Opportunity Cost Model of drafting.

The model begins by asking this simple question, did each NHL team take the best available player when they made their pick in the draft? After all this what most NHL teams claim that they are trying to do (I take them at their word). Every pick is scored on a 0-100% scale. If a team makes the most valuable selection they receive a score of 100%, if they draft a total NHL bust the pick is scored at 0%. Draft picks that fall in between "perfect" and "bust" are scored based on who close they were in value to the "perfect" selection that was still available at that point, (this "imperfect" scores can range from 1%-99%).

The key advantage of this approach is that it adjusts for a) players with similar Game Played by different value and b) the unique set of options available at each individual pick in a draft with nearly 300 selections. Because pick scores can range from 0-100 it is also more nuanced. Most draft studies score picks on 1/0 success/failure basis which obscures some fine grained detail. I hope to capture more of that detail with this approach.

My measurement of NHL career value is Tom Awad's GVT (Goals Versus Threshold) which assigns scores to each NHL player to the degree that they are better than a replacement level player (i.e. talent readily available on waivers or in the minors). No measurement system for NHL players will be perfect, but I have used GVT for several years and I consider Awad's values the best currently available for long periods of NHL history (I also really like Alan Ryder's Player Contribution system, but it only extends back a few seasons). GVT values adjust for differences in scoring levels over time and they pass the "smell test"--star players are about where I think they should be rated. (For more details on my adjustment to GVT please see the technical note at the bottom.) Obviously my conclusions hinge on the accurately of this valuations. To see more of Tom's work you can read him over at Puck Prospectus.

Once we have a good measure of NHL career value it becomes quite easy to do the math to see which team was most efficient in terms of sniffing out the best available player on draft day. I did this for the 7 years between 1999-2005 because those are recent draft years and this is a Thrashers blog.  The 1999-2005 period covers all Thrasher drafts except for the two most recent drafts (too early to get a good read on those in my opinion). I am working on a larger project to extend this over all NHL Drafts.

To illustrate how the Opportunity Cost Model scores each team's draft pick I have created a table (below) showing the first round from the 2001 NHL Draft.

1. Atlanta had the #1 pick and chose the highest rated player (according to Adjusted GVT) in Ilya Kovalchuk, Atlanta's score = 100%.
2. Ottawa went 2nd and also chose the best player available at that slot, Jason Spezza, Ottawa's score =100%.
3. With the 3rd pick we see our first "inefficient" or "sub-optimal" choice when Tampa takes Svitov, who has produced a paltry 13 GVT value so far. The best available player at that point in the draft was Ales Hemsky who has produced a value of 99 so far. Tampa made a big miss, but because Svitov was not a total bust their score is not 0% but 13%. (However, as time passes and Hemsky continues to pile up more GVT value over his career that Svitov pick will become less valuable in a relative sense.)
4. With the 4th pick Florida also failed to make the best selection possible, but their choice of Weiss was much less costly to them as Weiss has amassed a value of 64 Adjusted GVT so far. While technically Weiss was a "miss" in that he was not the "best player available" the miss was not huge and therefore Florida receives a much better score than Tampa, because they received 64% of the value that was still available in that draft year.
5. And so it goes until 13th pick where Edmonton receives a perfect 100% for taking Hemsky the "best player player available" at which point the next "best player becomes D. Roy (with a 95 GVT value) who would fall to Buffalo early in the 2nd round with 32nd overall.

So in the 1st round of the 2001 Draft only three teams get 100% credit for making the perfect choice. Nashville made the best "less-than-perfect choice" receiving 70% of the maximum value available when they picked by taking Dan Hamhuis.

I repeated this process for every round of the 2001 Draft. I then averaged each team's scores across all their picks. I'll illustrate this by showing the LA Kings picks and their scores for the 2001 Draft below.

To summarize the LA Kings 2001 Draft we simply average their Value % across all their 11 picks that year = 20.9%.The Kings had two terrific selections in Cammalleri and Huet and picked up a little extra value with Bednar and Steckel (who continues to add to his GVT value).

Now you might be thinking "Getting 20.9% of the maximum possible value available is not very good performance"--and you would be correct that the Kings left a lot of value on the board in 2001. But here is the truly stunning thing, the LA Kings had one of the highest team scores in 2001. An average team only gets about 8% of the maximum possible value available when they make their pick. The efficiency rate of most NHL teams is exceedingly small. Forecasting draft picks has an amazingly high error rate. Any NHL team that could consistently decrease their error rate by just 10% would realize a major advantage on the rest of the league.

Drafting High is Harder Than It Looks

As a Thrashers blogger I hear people say "anyone could see that Kovalchuk was the number one pick" and "you don't deserve any credit for making slam dunk picks in the 1st round" and so forth. But even in the case of the 2001 Draft there were people who liked Spezza better. I know of one guy who claimed that Kovlachuk wasn't even the best Russian player in the draft--opps. In hindsight, it is VERY easy to say that a team should have taken player X, it is much harder to forecast before the draft.

To illustrate the pitfalls of drafting high let's look at the Draft Efficiency Scores for all teams picking in the #1-10 slots in recent years. Teams picking first or second overall have the highest average score, but even at that spot they typically only get 3/4 of the maximum available value. Once you get past the top two slots, the average NHL team is only getting about half of the maximum value available to them and by the time you reach the #10 spot the number falls to about 1/4. Drafting early is "easier" than drafting later, but even in the top 10 many mistakes are made. Picking high is less of slam dunk than most fans think it is.

Who Has Drafted Best (1999-2005)

Now that I've explained how the scoring system works it is time to reveal the team ranking for recent years. I took the 7 drafts between 1999-2005 and scored every single player for every single team in the fashion illustrated above. I then averaged the scores for every pick taken over 7 years to arrive at a team Draft Efficiency Score. In the table below you can see these scores range from a high of 15% (Buffalo Sabres) to a low of 4% (Tampa Bay Lightning) with middle of the pack teams falling into the 8-10% efficiency area (Atlanta Thrashers). The Draft Pick Efficiency leaders are the Sabres, Canadiens, Wild and Kings. The worst clubs are Tampa, Carolina, Devils and Phoenix.

A few comments. My numbers confirm the conclusion of PPP that between 1999-2005 the Red Wings were not elite on draft day (and my approach gives them perfect scores for Zetterberg and Datsyuk). Detroit's production is just average. Here's something many people miss with respect to Detroit. When you draft Hall of Fame level talents and they play 10-15 years you don't need to have as many draft day "hits" as other teams do. Having a Nick Lidstrom on your roster for his entire career means you don't need to draft another #1 defenseman for a decade and a half. The key to Detroit's dominant position is not drafting, but player retention. Having great players who stick around makes filling out the rest of the roster MUCH easier.

Which Team Has Gotten the Most Value from the Draft (1999-2005)

The approach I have outlined above takes into account the fact that not all teams have high picks. The Opportunity Cost Model expects teams with high picks to find more value and teams with lower picks to find less value. What if we ignore draft pick position for the moment and simply ask which team has drafted the greatest absolute GVT value between 1999-2005? Given the number of very high picks it is not surprising to see Atlanta jump 6th in total value and the LA Kings finish 1st. The top teams are: Kings, Senators and Sabres and the bottom clubs are the Coyotes, Lightning, Hurricanes (some sunbelt scouts are really letting their fans down).

Conclusion

NHL teams fundamentally struggle to pick the best player available. Some teams do better than others but the error rate is very high even for the best drafting clubs. Drafting high is not a guarantee that a team will always realize the maximum value available to them.

Technical Note on Adjusted GVT Values

I have made a small adjustment to the original GVT values for this study. The Awad GVT allow for negative values for players deemed below replacement level (which makes sense). For my purposes, I consider the original "replacement level" threshold to be slightly too high. Why? Several players with long NHL careers (Ken Baumgartner 696 GP, Mark Janssens 711 GP, Jim McKenzie 880 GP) end up with big negative numbers. I assume most NHL GMs try to assemble the best roster that they can therefore if Baumgartner and Janssens played 700 games they must have had a value greater than zero to those GMs. Therefore I adjust downward the threshold by adding a modest games played bonus (GP*.1) to every career GVT Score.

Over on the Hockey Analysis Group (HAG) Tom Awad did persuade me that there are circumstances where a below replacement level goalie might continue to start. Therefore I did not adjust the scores sufficiently to eliminate all goalies with negative GVT scores for their career (Ron Low, Darren Eliot), however for this study all players with negative values were re-coded to zero. Why? Drafting a player who gets a cup of coffee in the NHL but isn't quite good enough is a missed opportunity, but it shouldn't subtract from the value of other good players who were drafted (i.e. worse than a guy who never suits up for a single NHL game).

There is a problem with scoring picks at the very end when there are no impact players left to be taken, but several NHL teams find guys who get cups of coffee and therefore deserve a bit of credit for those minor successes. My crude solution is to simply divide by last impact players that was taken even though he is not longer actually available. The last impact player taken provides the baseline of assigning small slices of credit for finding the cup-of-coffee guys at the tail end of the draft. This is an imperfect solution, but one that smooths out the end of the draft scoring.

While I believe this approach is superior to draft analysis that relies on Game Played or Points there are some flaws to my method. Right after a draft is completed my method will give too much credit to teams that draft over-age Europeans who can step in and play in the NHL immediately (but this bias fades with the passage of time). My method also penalizes teams for career ending injuries that are impossible to forecast (example Dan Blackburn) or players who die prematurely (Alexei Chereponov, Luc Bourdon, Dan Snyder). Another flaw is that my model does not adjust for talented hockey players who choose not to play in the NHL (Alexander Radulov, Bykov, Ilya Nikulin) or who miss NHL time due to contract disputes (Paul Kariya). To be perfectly honest, I don't see any obvious solution to those situations.

UPDATE: Tom Awad has put up a Draft Analysis looking at 15 year period using an expected value frame work along the lines of some old posts from two years ago--only his approach is more elegant with its fitted values. We both use GVT, but both have provide some tweaks to the values--but our conclusions are very similar--the Thrashers are very average (getting almost exactly the value you would expect given their draft position) and the Wings are a bit over-rated by people because they fail to perceive Detroit's strength is in player retention. His comment on the Thrashers:

For a team that gets such a bad rap, the Thrashers have actually done pretty well in the draft. While Ilya Kovalchuk and Dany Heatley may seem like no-brainers in retrospect, maybe Patrik Stefan did as well. Sadly, only one of their picks of the last 5 years, Bryan Little, is now an NHL regular.