Stats That Matter: When Most Positive Test Results Are Wrong

What MLB's top draft picks teach us about disease prevalence and diagnostic reasoning

I'm back after a couple of weeks away — first a family trip to Acadia, then almost immediately after, I took my son to St. Louis to catch a couple of games at Busch for his thirteenth birthday.

We watched the Redbirds take two against the Reds, saw the view from the Arch, and ate toasted ravioli and St. Louis-style pizza. For a first trip ever to St. Louis, it was about as perfect as it could be.

The highlight, though, came before the second game. My son's favorite player is Michael McGreevy. The odds of running into him at exactly the right moment, getting his attention, and walking away with an autograph were not in our favor.

But it happened. Improbable things keep happening at baseball games to a guy who writes about thinking probabilistically. I’m not sure what to make of that.

---

The MLB draft is just a few weeks away, and the speculation over who will be the number one pick always dominates the discussion.

Drafting well is essential to a team’s success. The exclusive opportunity to choose any player from the entire pool of talent in the draft is one that teams rightfully covet.

This year, the consensus top prospect is Roch Cholowsky, and most expect the Chicago White Sox to choose the UCLA shortstop with the first overall pick.

The White Sox have undoubtedly dedicated tremendous resources to determine his worthiness. Their best scouts have spent many hours watching him in person and on film. Their analytics department has meticulously crunched the numbers and analyzed all available metrics. What they ultimately want to know is whether he might become a future franchise player in Chicago.

He’s also ranked as the top overall prospect by the leading media gurus — MLB Pipeline, Baseball America, Keith Law, and Kiley McDaniel.

It’s about as confident an evaluation as the sport produces. But confidence in the evaluation isn’t the same as confidence in the outcome. What’s the probability that their highest-confidence pick actually becomes a franchise player?

To answer this question, I went back and looked at the forty six number one picks from the inaugural draft in 1965 through 2012. I set the definition of a “franchise player” as having accumulated at least 40 career wins above replacement (WAR).

I chose 2012 as a cutoff because most of the number one picks prior to that had completed their careers. Additionally, all the active players included have reached the 40 WAR threshold: Carlos Correa (44.5 bWAR), Gerrit Cole (43.8 bWAR), and Bryce Harper (55.1 bWAR).¹

From 1965-2012, only ten of the forty six (21.7%) number one draft picks reached the 40 WAR threshold.

Most number one draft picks do not become franchise players. It’s not a failure of the evaluation — all of these guys had the raw skills to make it — it’s that a franchise-level player is rare to begin with.

According to Baseball Reference, there have been a total of 23,670 players who have ever played in Major League Baseball. Only 561 have accumulated at least 40 career WAR. Only 2.37% of the players who have ever reached the big leagues were good enough to be called a franchise player.

Even the most confident evaluation can’t overcome that. When the outcome is rare enough, most positive results — even the most justified ones — will be wrong.

🔒 For paid subscribers: where the math gets uncomfortable

The 22% hit rate is the point of the baseball section. The clinical equivalent is more surprising — and more consequential.

Paid subscribers get the full analysis, including why a 95% specific test can still mislead you most of the time.