Exploratory Draft Data: Comparing pundits mock drafts

Like I have mentioned previously, the NFL Draft is now a major industry that exists within the massive industry that of professional football.  It is a field in which celebrities exist but is accessible to all, people with a lifetime of experience can give their views next to someone who knows little about the subject, and you can even change your ideas as much as you want as the draft season of February to now May mvoes along.  This has both its advantages and disadvantages in predicting where a prospect will be drafted.

What you want to do ultimately do to use the advantages and reduce the disadvantages is aggregate mock draft predictions to reduce the reliance on any one person's judgment.  This new mock ranking should be a combination of both what pick the prospect is predicted to be across the board but also what teams the pundits think are good fits, but for right now is just the draft value points.  This should help quantify the very qualitative process of what a lot of people feel makes sense in terms of fit, which is important as well.

If someone were to expand on this start, I'd suggest they get a much broader array of pundits, I only had time to collect a couple in time to complete this project.  There are literally thousands of people willing to give their opinions of where they think prospects should be drafted.

One thing I do want to note is that in order to numerically compare what pick these prospects should be drafted I used the method that is generally accepted as the standard draft pick value, the Jimmy Johnson Draft Value Chart.  It has been used since the 1990s to come up with a way to numerically compare draft pick trades between teams, so it more accurately describes draft value than just a number slot in my opinion.  I decided to use what the NFL ultimately uses, since I want the prediction to be as accurate as possible.  Incorporating a truer draft value chart would make sense if one were available, so until then the old coach of my Miami Hurricanes will continue to be the way the game is defined.

Comparing mock drafts based on draft pick value:

I collected the final mock predictions of the following draft pundits for any years I could between 2008 and 2013 within my limited timeframe

Some had 2 years worth of data, some had 5 years.  Obviously the more years worth of data, the more accurate the evaluation of the pundit would be but this is what I could collect.  If expanded, I would collect as many years back that I could from many sources.

To compare their accuracy, I fit a linear regression line and am comparing their R-square values.  What this essentially tells me is what percentage of the prospects 1st round draft value you could get right with just each pundit's prediction.  So if Eric Fisher is worth 3000 points as the first pick and Matt Elam is 590 points as the last pick, how close could I come just using the pundit's corresponding draft value prediction as the only variable considered.

Here is one full example, that of the popular Draft godfather himself, Mel Kiper:

This is both a good visual as well as numerical representation of the pundit's accuracy.  The circled blue value Mel guessed around 2200, but the prospect was really "worth" only about 1500.  So Mel overestimated this prospects draft position this particular year. The red line is the linear regression fit line and would go from the bottom left corner to the top right diagonally in an ideal world.  The further this line is off visually indicates how off the accuracy is.  Also the correlation value on the bottom lets me know numerically how closely associated an increase in Mel's predicted draft value is with the prospect's actual draft value.

Here is the full list of pundits, ordered from most accurate to least, along with how many observations I collected of each:

These aren't perfect comparisons because I couldn't find every pundit's mock for each year of every other one (although if I wanted to just compare 2013 I probably could to accurately rank them as of last year).  But it is more for overall accuracy generalizations.  Really what this says is that Todd McShay is better at predicting a prospect's eventual draft value based on JJ's chart better than his contemporary at ESPN, the original draft don, Mel Kiper.