In Defense of Projection Models

by

On Monday morning the Brewers find themselves on the precipice of missing the postseason, perhaps the most improbable possible outcome from a once-promising season.

The 2022 Brewers started the season 32-18, a franchise record that marked just the second time they’d ever won more than 29 of their first 50 games. They led the NL Central for 95 days, the fifth-most of any team in franchise history and had a margin as big as 4.5 games over second place in late May.

Of course, the Cardinals deserve some credit for their role in the Brewers’ descent from the top: They went 68-46 in the 114 games after the Brewers opened their largest lead of the season on May 26. To hold onto the top spot in the NL Central the Brewers would have needed to go at least 63-50 since that day, a .558 winning percentage. They went 55-58, a .487 mark.

Perhaps no one was more surprised by the Brewers’ decline than the projection models: FanGraphs’ postseason odds calculators were bullish on the Brewers early and showed their chances of reaching the playoffs as being above 95% on three different occasions in late May.

FanGraphs’ projection of the Brewers’ pending success and the results that followed naturally raise some questions about the model (or any model’s) ability to be so certain about a team’s fortunes so early in the season. Certainly, the model sometimes produces puzzling outputs: On April 24, for example, the Brewers beat the Phillies 1-0 on a day when the Cardinals also lost, moving the Crew into a first place tie in the NL Central and improving their record to 10-6 on the season. That morning FanGraphs had their chances of reaching the postseason at 71.6%. That night, when the Brewers still had 146 games to play, the model moved them up nearly nine percentage points to 80.3%. It was a big change in certainty for a team that had played less than 10% of its schedule.

email newsletter iconStay on top of the news of the day
Subscribe to our free, daily e-newsletter to get Milwaukee's latest local news, restaurants, music, arts and entertainment and events delivered right to your inbox every weekday, plus a bonus Week in Review email on Saturdays.

SIGN UP

Even when FanGraphs’ outputs are somewhat counterintuitive, however, it’s tough to argue with their results. The playoff odds data on their site goes back to 2014 and over that time (now nine seasons) there have been 27 occasions before June 1 where they’ve identified a team as having a 95% or better chance to reach the postseason (including six this year). If the Brewers miss the playoffs it will be the fourth time one of those teams have fallen short:

  • The 2015 Washington Nationals, coming off an NL East-winning season, opened the season with the best postseason odds in baseball and peaked above 95% for the first time in mid-May. They were at 98.3% on May 25 but went just 56-61 the rest of the way and finished seven games behind the Mets.
  • In 2019 Cleveland was looking to win the AL Central for the fourth consecutive year. FanGraphs had them at 95.7% to play into October on April 8, when they had played just nine games. Cleveland was a very good team that year and won 93 games but that wasn’t enough to unseat the 101-win Twins and the Wild Card-winning A’s and Rays (97 and 96 wins, respectively).
  • The 2021 Padres were the talk of the spring, having loaded up with talent to chase the Dodgers in the NL West. They had a 96.8% chance to make the postseason on April 15 and got all the way to 98.4% on May 26 before crashing back to earth and going 30-48 after July 1. They finished under .500 and eleven games back of the last postseason berth.

Beyond that group, however, the teams that FanGraphs was relatively certain about have largely lived up to that billing. If the projection models were perfect, teams that FanGraphs gave a 95% chance to reach the postseason would have made it between 25 and 26 out of the 27 instances. If the Brewers come up short this week then those teams will have reached the playoffs 23 times, good for approximately 86%. That’s pretty close, especially in a relatively small sample.

Barring a miracle this week the Brewers are going to leave this season with a sting of disappointment and a persistent question about what might have been. By the numbers, there are only a few teams in recent history that can relate.