What Sports Can Teach Us About Evaluating Software Development Talent

Various professional sports take different approaches to evaluate talent, and yet the contributions of athletes in team sports is difficult to predict.  The NFL conducts a combine, yet they make gaffes such as drafting Tom Brady as a low draft pick.

Athletes perform in public and are watched by experienced scouts, yet the scouts did not accurately predict how well Duncan Keith would perform.

Billy Bean famously upended baseball scouting by switching to metrics that predicted success better than the subjective evaluations of scouts.

Is it easier to predict performance in individual sports, such as track and field, golf, or tennis?

Did Dan Marino ever win a superbowl? No. To what extent does this reflect his ability? Is it the quarterback's fault if a team does not have a good running game?

In business, we mostly perform in private and performance is much less amenable to evaluation with metrics. Also, business is largely a team sport.

The larger the team, the more likely that individual performance can be expected to vary from predictions.

Given the extensive advantages that professional sports teams have over traditional businesses in evaluating talent, and the errors that they make in those evaluations, what would lead us to believe that businesses have any hope of properly evaluating software developers?

Is there a way to structure work that avoids the requirement for accurate evaluation?  Is there a way to be robust or even anti-fragile to evaluations? There are articles that discuss how 10x engineers are only 10x in a certain context.

Agile is an attempt to be robust to estimation errors. Estimation should only be used to get in the ballpark. Maybe the same is true of evaluation? Should the goals of software developer evaluation vary depending on factors of the hiring team (size, maturity, experience with the problem domain, maturity of the problem domain pioneers/settlers/town planners).

It is funny to hear the scouting reports for athletes like Tom Brady and Aaron Rogers.  The scouts said many negative things about the people who were to become some of the best quarterbacks ever.