Is Seth Jones Actually Elite?

The analytics and the eye test disagree. But why?

Blue Jackets limp into season as injury sidelines Seth Jones 4-6 ...

Since 2017-18, Columbus Blue Jackets defenceman Seth Jones has become pretty much universally recognized as one of the NHL’s truly elite blueliners. According to mainstream hockey media outlets like NHL.com and TSN, he’s a top five defenceman in the league. A combination of the eye test and reputation have rocketed Jones into regular Norris contention, and it’s not tough to see why: he has the hallmarks of an elite defenceman, in that he puts up points at a reasonable rate (on pace for between 40 and 57 the past five seasons), he’s big, he plays huge minutes, and he’s said to be a stud defensively. When he’s up for free agency in two years, he’ll probably receive over $9M dollars annually, either from the Jackets or somebody else.

There’s one small dissenting voice amid the coronation though - the analytics community. No, this isn’t a Doughty situation - I’m not about to tell you that he’s actually a bad player now. But it’s almost impossible to seriously engage with Jones’ macro-level metrics and impartially conclude that he’s played better than top-four-calibre hockey since he came to Columbus. If that sounds crazy, please bear with me. There will be plenty of time to call me a lunatic afterwards.

I’m going to approach the question of “Is Jones elite” a few ways. First, I’m going to provide an overview of Jones’ analytical profile, including his individual impact on a variety of offensive and defensive stats, his time on ice and usage, and his microstats. Then I’m going to try to find ways to explain the gap between the analytics and the eye test by watching an average game of his from this season. I think this divide is actually pretty explainable, and gets to the core of why macro-level analytics are such a valuable tool of player evaluation.

Seth Jones’ On-Ice Results Are Average at Best

I’ll concede this right away: Seth Jones is a very productive player. I give him full credit for ranking 16th in points by a defenceman, 9th in even strength points, 19th in EV P/60, 18th in EV G/60, etc. since the 2016-17 season. I don’t particularly value points as a way to evaluate defencemen (especially since they tend to suddenly decline when, say, Artemi Panarin leaves a team), but I would be remiss without mentioning this.

To get a decent idea of how Seth Jones impacts his team (especially defensively), we can’t just use raw stats. That’s true of analyzing any player, but it’s particularly important here considering how much the Jackets’ team defence has improved in the past five seasons (steadily improving from 22nd in the league to 3rd. We need to use stats that account for the all the contextual factors that could screw with the results - that includes quality of competition, quality of teammates, zone starts, game situation, etc.

We also have to remember what it is that these types of analytics value. The stats here are what I would call “macrostats” - they’re built from models that use ridge regression to determine a player’s isolated impact on the core objectives of offence and defence. On offence, a defenceman’s job is to contribute to his team getting strong scoring chances. On defence, their fundamental task is to prevent their goalie from facing high-quality chances against. Any of the actual things they do on the ice - blocking shots, making breakout passes, sending pucks up the boards, etc. - are components serving those two core goals. So if a player is the best in the league at blocking shots but also allows a disproportionate number of shots from the slot, he’s not elite defensively - what exactly is the point of all those blocks then?

We’ll start with Expected Wins Above Replacement, adjusted using a method I describe here to better evaluate defencemen. This stat, created by EvolvingWild, condenses a player’s value on offence, defence, special teams, and penalty drawing & taking into an estimation of the number of wins a player individually contributes to his team. Jones finished with an xWAR of 0.2 this year, a below-average performance. But that’s not remotely enough evidence for me. I’m more interested in his even strength play specifically - his special teams play has been incredibly erratic throughout his career by this stat, and I don’t think anybody’s calling him elite because of his penalty drawing. So let’s see how he ranks against the league simply in terms of even strength offence and defence:

Since being acquired by the Blue Jackets, Seth Jones has turned into a responsible defensive player, largely at the expense of his offensive contributions. But he hasn’t been elite defensively by this measure at even strength either, peaking at the 70th percentile in 2017-18 (meaning he was better than 70% of defencemen who played over 200 minutes). This is especially concerning because these stats are cumulative over the course of a season, meaning they should favour defencemen who play lots of minutes like Jones does. Using a rate-based stat, like RAPM isolates, he looks even less exceptional:

Jones is supposed to be among the top two-way defencemen in the league, and yet he’s never been even above-average at both ends simultaneously. This aligns with the findings of Micah Blake McCurdy’s isolate model as well (which additionally attempts to account for coaching):

Additionally, while it seems unfathomable to say that the Jackets are just as good without Jones on the ice, the numbers bear it out. In the past three seasons, Columbus has had exactly the same percentage of goals for at 5v5 (52.82%) with Jones off the ice as on the ice, and only a slightly smaller xGF%. The same is even true of some of his teammates. Ryan Murray has been significantly better apart from Jones by expected goals for percentage and expected goals against, as has Markus Nutivaara. The only regular Jackets defenceman whose results benefit from playing with Jones is Zach Werenski, but to be fair to him Jones is also much better with him than without him - they make eachother better. With Or Without You stats are flawed, as they leave out valuable contextual information, but as Jones’ ability to elevate his linemates is often cited as a strength of his I thought it was worth mentioning.

And since I know it’s going to come up, no, losing Seth Jones to injury is not why the Blue Jackets collapsed in February. As I’ve pointed out elsewhere, the post-Jones-injury Jackets were still the 8th best defensive team in the league in terms of quality chances against; what really happened is that the Jackets goaltending went from 1st in the NHL to 28th in that time. So unless we should be talking about Seth Jones as a contender for the Vezina Trophy, evidence suggests that it just turned out that Elvis Merzlikins wasn’t the best goalie in the NHL and he regressed rapidly to the mean - as one might expect him to.

Seth Jones’ individual impact on his team’s ability to generate offensive chances and prevent scoring chances against has been average at best during his time in Columbus. But what about his deployment?

Big Minutes + Top Competition = Elite?

That TOI/GP is so often used as a reason why a defenceman is elite is reflective of the lack of mainstream stats that measure defensive play in my mind. Yes, most elite defencemen play big minutes, just like most elite forwards do. It makes sense that they would; if your team has an elite defenceman, you should play them a lot. But playing big minutes against tough competition isn’t a skill in and of itself - it’s a coaching decision. Where the elite skill comes in is excelling in those circumstances, and statistically speaking Jones hasn’t done that. You should be skeptical when someone tells you that a player is good just because they play big minutes, because what actually matters is how they play in those minutes.

But let’s actually dig into Jones’ usage a bit. Seth Jones played the 7th most minutes per game among defencemen this season. But it might surprise you to learn that according to PuckIQ, Jones actually wasn’t deployed against top competition that much, ranking 66th in that category. In terms of time against top competition as a percentage of ice time, Jones ranked 139th out of 233, 5th among Jackets defencemen. Tortorella appears to deploy his top two defensive pairs fairly evenly from a matchup perspective, trusting Savard and Murray almost as much.

So this season at least, while Jones certainly plays a ton of minutes, he doesn’t play as much against top competition as you might think. But have you seen his microstats?

Microstats

When I first tweeted my confusion about Jones’ elite reputation a few months ago, I was told by multiple people that while his macro-level shot and chance impact stats weren’t amazing, I should consider that he’s an elite transition player. While I think microstats like this are an invaluable tool for evaluating the way a player plays, determining what their skills are, and discovering what makes them effective or ineffective, I don’t think that a player who excels in specific areas of the game but has underwhelming overall results should be considered elite. Nonetheless, I would not be doing my due diligience if I didn’t look into this at least.

Most microstats are unavailable to the general public because they have to be tracked manually, which takes a lot of time and effort (and venture capital). Fortunately, Corey Sznajder has made his own data available to the public through his Tableau and visuals by CJ Turtoro. Looking at Seth Jones’ numbers, you can see where his reputation for excellent transition play comes from:

From 2016 to 2019, Jones ranked near the top of defencemen in entering the offensive zone with possession of the puck and very well on the breakout as well. He was extremely active and efficient in the Blue Jackets’ transition game. It’s clear that Seth Jones controls the puck a lot, affirmed by the comments made by the Jackets’ coaching staff in this article by Alison Lukan discussing some internal metrics tracked by the organization (which include puck touches and breakouts). Something worth noting, however, is that Jones’ transition play took a major hit this season, as Werenski and Dean Kukan both ranked decidedly ahead of him in that regard on the Jackets’ blueline.

As I’ve said before, I consider microstats to function as EyeTest+ - they quantify the events we see on the ice so that we can compare players to eachother in certain facets of the game (and so we don’t have to watch all 1,230 games every season). That’s why I think it’s fair to say that the disconnect between the microstats and the macrostats is directly reflective of the eye-test vs. analytics thing here. When you’re watching a game, your eyes are drawn to the things defencemen do with the puck, not what they do without it. Jones’ talent on the breakout leads to eye-popping transition passes, and yet his teams consistently get fewer and lower-quality scoring chances when he’s on the ice. There’s a disconnect here.

Bridging the Gap Between the Macrostats and Eye Test

I’m not a brilliant hockey tactician by any means. I was intramural captain of the year once in university, but that was mostly due to the fact that we improbably didn’t forfeit any games that season because of lack of attendance. In all honesty, my eye test is probably roughly equivalent to most fans’. Nonetheless, I was curious to see what the hell was actually going on with Jones - how do the WAR, RAPM, and isolate stats contradict the eye test and microstats so much? Like most people, I’ve seen him play a bunch of times against my favourite team and in the playoffs and I’ve generally been impressed by him as far as I can remember. But I wasn’t exactly isolating on him the whole time.

I chose to watch a game between the Jackets and the Jets from November, which seemed like the most average and representative performance I could find. His xGF% was 49.9, the closest to his season average of 50.1. This is a single game sample, and while I tried to rid myself of confirmation bias as much as possible there’s no way for me (or anyone) to totally avoid it. This isn’t meant to be a remotely comprehensive solution to the mystery, but rather point to possible explanations. I don’t have the tools or expertise to do a detailed shift-by-shift breakdown, so these are more impressions than anything else.

Jones’ talent when it comes to handling and passing the puck is extremely apparent. He’s noticeable when he’s on the ice and he’s very active in the play outside of the offensive zone - it’s no surprise that he fares very well in the Jackets’ internal “puck touches” stats. His breakout passes in this game were generally pretty short and low-risk, and if that’s how he usually contributes in transition it might help explain his very high success rate. He looked especially confident when he was carrying the puck himself through the neutral zone.

However, there were a few things I noticed that might be indicative of why the rate stats aren’t as kind to him as the eye test. Firstly, as effective as Jones was at driving transition offence, he was far less of a difference-maker in the offensive zone. His playmaking ability was occasionally apparent, but far too often he took very low-percentage shots that were either blocked or easily saved. This made me think of the way that he’s almost always driven shot attempt quantity far more than quality:

Overall, Jones looked far less confident handling the puck in the offensive zone despite getting frequent touches, and often got rid of it hastily when he had time to make a better play.

Secondly, I noticed some things in the defensive end. As you might imagine, Jones was very active in his own zone, getting a lot of touches and winning puck battles using his skating and size; however, in doing so he almost always pushed the puck to an area where the Jets were easily able to recover it and continue putting on pressure. Additionally, when Jones was on the ice, the Jackets almost always allowed the Jets to enter the offensive zone with possession; the one time I remember him really stepping up and defending his blueline he almost got walked. This matches up with the 24th percentile possession entries denied stat from above. Why does this happen? I’ll defer to former Marlies assistant coach Jack Han, who actually is a great hockey tactician, who wrote a brief piece today about Jones' defensive zone play which argues that there are mechanical issues with his skating:

Jones’ way of masking his inferior glide, then, is to play a looser gap across the neutral zone. This allows him to take an extra backward crossover to match speed or to pivot to forward skating while still protecting the middle of the ice. But this also destroys his ability to deny the blue line. 

I recommend that you read the entire piece, because the video illustrates this point very well.

Isolating on Jones at even strength in an average performance helped illuminate where the differences between his statistical profile and reputation come from: excellent transition play marred by underwhelming offensive zone participation and defensive zone issues.

Conclusion

I find Seth Jones to be a really fascinating player. I agree with those who say that he’s a perfect illustration of the disconnect between analytics and the eye test, but while they might mean that it proves that analytics are fundamentally off-base, I think it shows exactly why they’re valuable. I don’t think that the gulf between how Seth Jones looks from the eye test and the analytics is irreconcilable at all; it’s as clear a representation of the way that certain attributes and skills stand out to the eye test even if they don’t lead to measurable results.

If you looked at his statistical and analytical profile (outside of just point totals) without his name attached to it, it would be absolutely impossible to impartially conclude that Seth Jones has played like a top-pairing calibre defenceman, let alone an elite one, in his career. By the best models that we currently have to isolate players from the kinds of context factors that affect their results, Jones has been a pretty unremarkable defenceman since he’s arrived in Columbus. While he’s said to be one of the most dominant defensive players in the league, at the core function of defence (preventing your goalie from having to face quality chances against) he’s been only a bit better than average.

And yet, those who regularly watch him (and those who don’t, but listen to those who regularly watch him) consider him to be a top-five defenceman in the league and an annual Norris contender. I believe that this is in large part because he is so active in the play; his ability to break the puck out consistently without turning it over and carry it in to the offensive zone with so much poise is amazing to watch, and his natural physical talents are often on full display. He uses those gifts in the defensive end, breaking up plays along the boards effectively with his stick and using his body to push forwards out of dangerous areas. His activity in transition is probably a big reason why he gets so many points as well, and he has good shot. It’s impossible not to notice Seth Jones when he’s on the ice.

But it’s the things that the eye isn’t as drawn to that explain why despite all of that he profiles analytically as such a mediocre defender. Players like Jones force us to consider the ways that flashy displays of talent do not necessarily translate to macro-level on-ice results; how a player can be really good at certain very visible things but quietly ineffective in other areas. I do not think Seth Jones is an elite defenceman, because to me an elite defenceman is one who marries their physical skills with the ability to tilt the ice in his team’s favour. An elite defensive defenceman must not only break up passes or win puck battles, but prevent his goalie from facing great chances in other less noticeable ways. An elite offensive defenceman must not only break the puck out and score goals, but use strong judgement and patience in the offensive zone to ensure that his team consistently generates quality scoring chances. An elite two-way defenceman must be strong in both regards. Seth Jones is excellent at many things, but all evidence suggests he does not fit that description.