Replay, Challenges, and Making Important What You Can Measure

Harold Koh, my Procedure professor, spent much of today's class telling us all about how the legal system has been thrown into disarray by recent Supreme Court decisions on matters of civil procedure. This tied in with his broader critique of the way civil procedure has been developing over the 75 years since the Federal Rules of Civil Procedure were adopted, with the emphasis shifting to early disposition of cases rather than reaching just outcomes. He quoted some anonymous judge who had said, at a recent conference about the Federal Rules, that, well, we can't measure justice, but we can measure closure of cases, so that's what we focus on. Prof. Koh's point was that this is terrible, and that it reflects the human tendency of, "If you can't measure what's important, you make important what you can measure."

I was reminded of that when I read today's Fangraphs article about the incoming replay system in Major League Baseball. The main point of the article is that, while replay is (in the opinion of the author and most of the Fangraphs-reading community) an obvious and much-needed good thing, the way they're planning on implementing it is all kinds of dumb. That's entirely about this stupid managerial challenge system. I won't get into the details, in part because they haven't been finalized, but the idea is that managers will get a certain number of "challenges" in a game, each of which they can use to request video review of a certain play, and if they use too many challenges on which they lose, they don't get to make any more challenges. Oh, and the challenges will be differently distributed among the various innings. It's really stupid.

But it also, I think, heralds a new era for baseball managers, if it lasts anyway. It was pretty much a throw-away line in the article, but I found interesting the point that "a very obvious outcome [of the challenge system] is the advent of sites like ours beginning to track stats of the success rates of these calls." Yeah. It will be really effing easy to track managers' success rates on challenges. Really easy. And, the internet being a big ol' place, I can guarantee you that it won't take very long before someone has a sortable leaderboard of 2014 managerial challenges, where you can rank managers by total correct challenges or by success rate, or maybe by Successful Challenges Above Average or something. Moreover, this will be the only leaderboard on the entire internet (except for other sites' versions of the same thing) where you'll be able to sort the various MLB managers by some numerical criterion. Currently, we got nothin'. Evaluating managers is pretty much a guessing game at this point, which is why sabermetric types never ever get worked up about the Manager of the Year Award. They think the whole thing is stupid, and that there's no way you even could make it meaningful, or anything other than a proxy for "who was managing the team they did the best, or maybe that most exceeded our (probably irrational in the first place) expectations?"

So what's gonna happen when the only thing about managers we can measure is their skill at using replay challenges?

It's not hard to picture that this will very quickly become one hell of a proxy for overall managerial quality. It is obviously a terrible proxy. There's a ton of stuff that managers do other than make decisions about when to challenge a play. Currently, we call that stuff "managing." The best estimates are that right now, a really good manager can get a team maybe five additional wins over the course of a season, which is the equivalent of adding an All-Star-level player in place of a replacement-level one. No one really doubts that managers are important; indeed, sabermetric types, who love to hate on managers for making idiotic in-game decisions, are among the first to insist that managers are important. But we have almost no way to measure their importance, to really know who are those managers bringing their teams five more wins, and who are the ones preventing their talented roster from winning games. And next year we'll have something to measure, something whose impact will be easy to measure. Hell, I wouldn't be surprised if someone starts up a metric for how many net runs a manager has saved his team through use of challenges. Which means that, 365 days from now, there will be a leaderboard somewhere showing that such-and-such a manager saved his team X runs, while this other manager only saved Y runs.

People are gonna care about that, and they're going to care about it way more than they should because it will be the only thing they can measure with anything resembling that kind of precision. It would be like if the only part of a player's offensive contribution we knew how to measure with any kind of accuracy was their baserunning. It's absurd to say it, since that is in fact one of the tougher parts to quantify, but we'd probably end up saying things about how, well, hitting probably matters, but we're not really sure how much, so we're left with no choice really but to conclude that Jacoby Ellsbury (+11.4 BsR) and Eric Young, Jr. (+9.9 BsR) were their respective League's best players, while Paul Konerko (-8.4 BsR) and Allen Craig (-5.9 BsR) were the worst. Well, okay, the Ellsbury and Konerko parts aren't so far off, but you get the point: it would be absurd. And starting next year, MLB managers are going to produce precisely one set of objective, quantifiable data about their managerial skills, and it will be about a part of that skill-set that has never even existed before. Quite possibly, it won't be long before managers are seen as challenge strategists who happen to do all this other "managing" stuff.



(Of course, since NFL football currently has a challenge system, we don't entirely have to speculate blindly about the impact of this stuff. But, not being a football fan, I honestly just don't have the slightest clue how this stuff works in football. Do people keep track of coaches' success rates? The article suggests that coaches get help from people who were watching on TV, implying that the success rate is probably pretty high, so maybe there just isn't meaningful differentiation among the various coaches. Or maybe there's enough other stuff about coaching strategy and the like that we can measure so that this issue doesn't predominate. Or maybe I'm wrong, and people won't forget that there's more to a manager's life than challenges. Still, I think it's something to worry about, even if it hasn't been a problem in football.)