There's a Solid Argument Against Matthews' Hart Cadidacy. Goals Against Isn't It.
Measuring skaters' defensive play by their goaltending is just bad analysis.
The Hart race, between a couple of players including Connor McDavid, Auston Matthews, and Igor Shesterkin, has gotten especially vicious lately, particularly between advocates of the first two players. If you wade into these arguments (which I recommend against), you’ll almost certainly see at least one and probably several Oilers fans trumpeting this stat:
A big part of Matthews’ Hart case is his defensive results, ranking very highly in scoring chance and expected goals against prevention. But, these fans point out, the Leafs have allowed 3.1 goals against per 60 at 5v5 with him on the ice, so obviously he is actually bad defensively.
I’m lukewarm on Matthews for the Hart, and as a Penguins fan I have zero personal investment in him winning it. But this is a bad argument and bad analysis that goes directly against what we have learned about this sport in the past decade. And as somebody who writes about analytics, I think this should be nipped in the bud before people start widely using on-ice goals against as a defensive stat again.
Auston Matthews’ Defence
The reason Matthews’ defence is getting hyped this season is because the Leafs give up fewer and less dangerous chances against when he’s on the ice, especially in the slot.
This lines up well with the eye test and proprietary micro-stats, which show that he ranks near the top of the league in takeaways, puck management, and puck battle wins.
Based on the EvolvingHockey expected goals model, which estimates the quality of every shot based on past data, the Leafs have given up 35.3 expected goals against with #34 on the ice. But they’ve allowed 56 actual goals in that time.
Now, you could argue that isolate models based on those expected goals are giving Matthews undue credit for those on-ice results, and that’s he’s a product of system or coaching or talented linemates. But the specific argument these fans are making here is that Matthews is personally responsible for his goaltenders allowing those roughly 20 goals above expected, and therefore is actually bad defensively. That is far, far shakier.
Why Goals Against is a Bad Skater Stat
There are things a player can control on the ice, and things they can’t. By playing sound possession hockey, a skater can influence how many shots their team surrenders on the ice. By playing good defence, they can influence the quality of the shots they do surrender. But one thing they can’t control is whether, once the shot is taken, their goalie actually makes the save (unless you believe they have psychic powers).
When analysts talk about measuring defence, they are talking about measuring the first two factors - quantity and quality of shots - and filtering out the insanely random variable of goaltending. This can be really tough to do, and there’s plenty of uncertainty when it comes to assigning credit for on-ice defensive results.
But one thing we can be sure of is that while a forward can’t control whether the guy in net is prime Hasek or a zamboni driver, he can in some way influence the shots the guy faces. If I’m a beer league defenceman, and every time I hop on the ice I pass the puck directly to an opponent for a breakaway, I’ve influenced the quality of shots my goalie is facing. But how often those shots go in will depend on how well the goalie plays. If my team’s goalie is an ex-NHLer and saves 18 out of 20 breakaways, it doesn’t make my defensive performance any better than if the goalie is my buddy Sam who put his mask on backwards and saved 2 out of 20.
This logical point is backed up by data as well, and has been pretty much settled for almost a decade. Carolina Hurricanes AGM Eric Tulsky even wrote a very to-the-point piece about it in 2013.
There have been minor debates since then, but they revolve around the question of whether certain skaters can exert consistent impacts over time, not whether the stat is useful as a way of measure defensive play on its own (let alone in a single season). The way a skater could influence “goaltending” is by causing the chances the goalie faces to be more dangerous than public expected goals models can measure (off odd-man rushes, pre-shot passes, screens, etc.). And as CJ Turturo recently found, there is reason to believe that certain skaters exert a distinguishable if relatively small impact on goals being scored against them above or below expected, based on a consistent and sustained impact over several seasons.
The problem with applying that logic here is that if you look at Matthews’ track record, there’s no observable pattern whatsoever. His on-ice goals saved above expected and save percentage at 5v5 has varied mostly randomly since entering the league; sometimes good, sometimes not-so-good, sometimes average.
If we saw a consistent pattern of Matthews’ goalies conceding more goals than expected then a good argument could be made, but instead we’re looking at a huge outlier in a season in which the Leafs have gotten notoriously poor goaltending, and a pretty small number of actual events. 63 games and 20 goals might seem like a large enough sample size for there to be a strong signal, but it really isn’t. Wild variations in puck luck either way happen all the time at both the team level and the player level in ways that are almost always not repeatable despite happening over an 82 game season. As tempting as it might be to say “but all his linemates play in front of the same goalies!” and “The Oilers have bad goalies too!” that is ignoring that goalies do not always play at a level that is consistent with their talent level all the time, and the sample is not nearly large enough to iron out any variation. Instead of having to compare goalie “skill” from one team to another, the better solution is to simply do our best to remove the goalie from the skater defence equation entirely.
So while there may be some interesting nuances around the margins of expected goals models, evaluating a skater’s defensive play in a single season based on a stat that is heavily if not almost entirely based on something he has no control over, the performance of his goaltenders, is bad analysis.
There is a long list of valid arguments to make against Auston Matthews’ Hart candidacy. You could say Connor McDavid is a more well-rounded offensive player, that he’s a more talented playmaker, that he makes his linemates better, that the defensive difference between them is the result of the system they play in. You could say Shesterkin has had a more direct impact on his team’s success. And look, if you want to argue that you think the Leafs’ (and therefore Matthews’) defence is overrated by expected goals models, you can go ahead. “I think the Leafs are worse defensively than xG can measure so I’m not as sold on Matthews’ defensive numbers” is a perfectly valid opinion as long as you have something to back it up. But the notion that Matthews is singularly or even mostly responsible for the 20 goal gap between his expected and actual goals against - what is implied by relying on Goals Against per 60 - is absurd.
Dusting off a discredited and context-free stat is the kind of bad analysis that makes hockey discourse dumber, and it’s also just not necessary. Let’s not bring back goals against as a skater stat, please.