For the second straight year, KPI Sports reviewed the accuracy of computer ranking systems compared to the NCAA Tournament field in 2016. The 2015 analysis can be found here.
The KPI Rankings performed well with regards to seeding in 2016. When comparing KPI with other computer ranking systems (RPI, Sagarin, ESPN BPI, KenPom, and LRMC), KPI was the most accurate in projecting seeds (margin of error of 3.40 spots, the equivalent of 0.85 seed lines per team). After projecting seed lines 35% closer than any other analyzed metric in 2015, KPI projected seed lines 24% closer than any other metric this season.
There was more differentiation in terms of selection this year. After getting 66 teams correctly in the field in 2015, KPI correctly identified 63 in 2016. While the combined accuracy of the NCAA Tournament and NIT was 97/100 teams, getting 63 in the NCAA Tournament is relatively low compared to previous performance. The five teams missed by KPI were Tulsa (56), Temple (57), Michigan (58), Vanderbilt (59), and Syracuse (61). All were on the 10 or 11 seed line. Four of the five teams in the KPI field that didn’t make it were 1 or 2 seeds in the NIT. Four of the five were also No. 1 seeds in their conference tournament who won the regular season title but lost in their tournament. St. Bonaventure was the exception.
It is important to note the intended purpose of each given ranking. There is a substantial difference between results-based metrics and predictive metrics. KPI is intended to “rank resumes” in order to determine the most qualified field of 68 teams. There is no human interference in the math, just a multi-step algorithm ranking performance. Other ranking systems may have different definitions (may focus more on offensive/defensive efficiency for example) and may also have different intents (intended to rank who will win, not how a team should be seeded in the NCAA Tournament). There is nothing wrong with that. There are a lot of people doing some fantastic work to try and evaluate teams.
Teams Accurately Predicted in the NCAA Tournament Field:
- 64, ESPN BPI
- 64, Sagarin
- 63, KPI
- 63, KenPom
- 62, RPI
- 61, LRMC
Margin of Error in Rankings (This is measured as the difference between the given metric and the actual 1-68 List for teams correctly projected in the field):
- 3.40, KPI
- 4.47, RPI
- 5.56, Sagarin
- 5.84, ESPN BPI
- 6.10, KenPom
- 6.11, LRMC
Teams Projected Accurately Within Zero, One, or Two Spots (0.5 seeds) of Actual 1-68 List:
- 31, KPI
- 25, Sagarin
- 24, ESPN BPI
- 23, KenPom
- 23, LRMC
- 22, RPI
Teams Missed by More Than 10 Spots (2.5 seeds) of Actual 1-68 List:
- 2, KPI
- 6, RPI
- 10, Sagarin
- 12, KenPom
- 12, LRMC
- 13, ESPN BPI
The KPI algorithm is intended to rank teams on a game-by-game basis, breaking data down to its finest detail. Each game counts as one and is averaged together for a composite ranking. Factors include (but trust me, is not limited to) the game result, the location of the game (with an evolving adjustment based on year-to-date data), the score (taken as a percentile-based factor based on percentage of total points scored), and the quality of the opponent. Every detail that can create and advantage is offset. A score of -1.0 is approximately the worst loss, a score of +1.0 is approximately the best win, and a zero is a virtual tie on a neutral floor. All teams began the season at zero. Because games are being ranked, a team’s resume can also be ranked from best win to worst loss and queried out based on specified criteria (i.e. home only, conference only, etc.). With the game-by-game resume, it is easier to identify what is causing a team to be ranked higher or lower than one might expect.
Full Data on Accuracy of Each Metric:
