He uses out-of-sample fit (also called cross-validation) to determine the following:

-1) how to discount data from previous years
-2) what the cutoff should be for minimum number of minutes
-3) a reasonable constraint on how noisy the data should be

A brief discussion of each issue:

1)There is a debate among adjusted plus-minus people whether or not you should use prior years in coming up with a number. Proponents of adding previous years say, “the more data, the more reliable the estimate.” Opponents say, “we only really care about this year and players change.” There’s no obvious answer to the question theoretically so he tries to estimate the right balance of new and old information by see how well models predict the future based on different discount rates.

2)There is also a debate about what to do with players that play a small number of minutes. Because the data is noisy, these players can have really wacky estimates for their APM, which throws everything off. On the other hand, if you lump all low minute players together, it is possible that the models incorrectly believe Patty Mills = Jeff Pendegraph. Again, he resolves the issue by looking at how well different minute cut-offs affect the fit of the model.

3) This is the trickiest to explain. Essentially, the idea is that the models that estimate APM assume that any estimate for a player or a unit’s APM is possible. Logically speaking, however, we know that even LeBron James should have a true APM of +50. By giving the model some expectations for how variable different units should be, it might produce better results. He uses the same method to determine what these expectations should look like.

and he’s doing something else.

But Joel played much, much better defense the past two seasons. Oden fouled a ton, didn’t really understand help defense, etc. He was starting to play as well as Joel before getting hurt this season, but there was still a difference.