Process vs. Results: Why It's So Hard to Evaluate General Managers

It's a make or miss league.

This phrase is said so often that it has become one of the league's truisms. Obviously, this stems from the act of shooting a basketball. Whatever the process to get there, the only thing that matters on the scoreboard is whether the ball went in or not. A player can make a beautiful drive and kick to the corner for a wide open three and his team will still get zero points if his teammate misses. A player can chuck a step back three at the end of the shot clock and if it goes in it's still worth three points. We can talk about efficiency and corner threes all we want but, at the end of the day, all that really matters is how many makes and misses there were.

But this phrase goes beyond the scoreboard. It expresses how the culture of the league focuses on results. If a coach wins, he's assumed to be a good coach, regardless of what elite talent he might have had on the floor. If a player wins, he's assumed to have that special "it factor" that allows a player to win. If a great player fails to win a championship, or several, it's a major stain on his legacy. Arguments about process and context are perhaps becoming more frequent but they still lack the cultural capital of winning and losing.

The film never lies

Here's another phrase that's made it's way around the league and then some. Coaches and players will all say it when asked questions at the end of a contest. They're hesitant to make any sort of proclamation about what went right or what went wrong before consulting the tape. There's the implication that the film will illuminate all.

What exactly does the film show?

The film shows the process. It helps coaches and players answer the question of why the team lost, why the team gave up so many points, why the team struggled to score. A good coach can watch the film and see that their pick and roll defense struggled to contain the ball handler, for example. A good coach can watch the film and see that their help defenders weren't disciplined enough and strayed too far from their men. Good players can do this too.

It's always struck me as odd that the act of basketball, the coaching and playing part of it, is so focused on process while the culture and the media coverage of the league is so make or miss. Players spend hours breaking down exactly how they defended a specific play - where they positioned themselves, what their stance was like, how they reacted, where they put their feet - and get praised or criticized regardless of whether the shot went in. If a player executes a coach's scheme perfectly, then he's considered to have done his job and the team will live with the result. Players spend all day in this mindset and then get in front of a microphone and start blabbering about how their new teammate is a winner because he was a bit player on a championship team. And the media seems incapable of evaluating a player's legacy beyond how many rings he accumulated.

Of course, I'm oversimplifying to some extent and there's a number of exceptions, but I think the general point stands. Why is there such a fanatic obsession with the film and the process on one hand and a dogmatic reliance on results on the other?

I think the answer lies in our uneven access to information.

The game film is an incredible tool. Think about all the information it holds. It shows everything that leads up to a decision. It shows where all the players are located on the court, how they all got there, and the player's form or technique. Even as an outsider, I can look at the game film and have a very accurate understanding of both the process and results. If I watch enough of it, then I can make very accurate statements about a player's ability.

There are still gaps in our understanding and we make mistakes. Perhaps a player takes difficult, isolation shots because his coach has asked him to do so in order to limit the opponent's transition offense. Perhaps we don't adequately understand how a player's role in an offense affects his performance (Monta Ellis anybody?).

Most of these limitations do not exist for coaches and players. A player should know exactly what the coach is asking him to do in a certain scheme or play. Every guy should know his role and what they're expected to do. Miscommunication can make this difficult to implement but there's no fundamental limitation. The coach-player relationship and the game film provide them everything they need to fully understand both the process and the result of a play.

That's why there's so much trust and reliance on the game film.

Players usually focus on results when answering questions about players they've never played with. If you've never played with a guy, you can't know how hard he practices or how he approaches the game. But you can know what his win-loss record is and how successful he's been in the playoffs. When knowledge is limited, players focus on these proxies assuming they represent the truth. The make or miss nature of the league is, in many ways, derived from a lack of knowledge.

As fans, we are always struggling with these fundamental limitations and they only get worse as we move from evaluating players, to coaches, to general managers.

With the game film, there's just a few pieces of information we don't have access to and we have plenty information to adequately fill in the gaps. If I start to learn a team's defensive scheme, I can figure out what each guy is being asked to do in each situation, even if I can't talk to the coach. If I start to learn the team's plays, I can tell which guy made the mistake even if it isn't obvious. Through the film, there's very little information hidden from us in terms of player's performance. We can judge a player with a high level of confidence and explain why he struggles or excels.

The same can not be said for coaches. The film gives us some information about the coach's play calling. We can see which plays and systems he favors and what results they produce. But often it's difficult to tell which plays are called by the coach and which are audibled by the point guard or other players. If something isn't working, say the team is miscommunicating on defense, we can't tell if that's because the coach is bad or if the players simply aren't receptive. We don't have access to practices so we can't really say why the coach is a bad coach.

Of course, we can make educated guesses with this as well. We can weave together media reports, compare how different players do under the same coach, and start to create a believable, perhaps even informed, evaluation in our heads. But this is much more difficult to do and we rely on the perspectives of others, many of whom have their own agendas and motivations.

But the hardest to evaluate is the general manager - by far.

As fans, we have no direct observation of a general manager at work. We never get to see what happens during negotiations. We don't see their interactions with players. We don't know what process they use to evaluate talent, how they draw on various perspectives of their staff.

What we do get are the results. We know what moves he actually made because those become official. Perhaps we have some reports about which players he talked to but it's difficult to know which ones are accurate and how many meetings there are that don't get leaked. Even if we think we know a general manager went after a specific player and failed to get him we can't know why. We can't know whether a different GM would have had more success.

Again, we can make educated guesses. We can weave together various sources and try to craft a narrative but we should recognize that it's another level removed. All those sources and reports are framed. Press conferences are spun. We have no idea what information we're missing. We have no direct information about the GM's interactions with players, staff, or how he goes about making decisions. The more indirect our knowledge, the more speculative our conclusions become, and the more we are forced to rely on results as proxies.

I think the Blazers' summer was one of the best examples of this tension. Our lack of knowledge made it extremely difficult, and very contentious, to evaluate Neil Olshey.

Regardless of the numerous reports, we can't know exactly when Neil Olshey knew what. Even if we know when Olshey got a final answer from Aldridge, we can't know what kind of information he was getting from Aldridge's camp before that. We can't evaluate how Olshey managed the risk because we don't know what the extent of the risks were, the likelihoods of each outcome.

On one hand, you might say that since Aldridge left, Olshey was right to avoid long contracts. On the other, it's possible Olshey could have signed older veterans to long contracts and convinced Aldridge to stay. We can't see the process. We couldn't see the options available to Olshey. We can't see the roads he didn't take and what information he had at the time when he made those decisions.

If we don't know all that, then we can't evaluate him in the same way we evaluate players.

I can see what information a point guard has as he dribbles around the screen. I can see the position of all the other players. I have the same information the player had when he makes his decision (for the most part). This allows me to form a strong opinion about that point guard's ability to read a pick and roll.

With general managers, I have no idea what insider information they're privy to. I have no direct access to the reasons a player signs with a team. We read reports and form narratives. We should still have opinions, and sometimes it's painfully obvious when a GM is bad, but we shouldn't hold onto these opinions as strongly or as forcefully as we do with players.

As fans, we always want to have a perspective. Sometimes, the hardest thing is to admit "I don't have enough information to form an opinion." Or at least we should hedge our statements and recognize the lack of information we're struggling with.

I don't know if Neil Olshey is a good GM. I don't know if Neil Olshey is a bad GM. Judging him solely by results is inadequate and judging him by a litany of reports and statements is fraught with bias. But these are our only options.

I will say I think Olshey and his staff have demonstrated the most important skills for the upcoming rebuild. The Trail Blazers will likely be selecting in the mid to late lottery and Olshey has found talent in that position numerous times. Lillard, Leonard, and McCollum were all selected in that range. He's also traded for undervalued players and expertly managed our cap sheet to maintain flexibility. These are all the things you want a GM to do in the first few years of a rebuild.

Anything beyond that and all my opinions have to come with massive caveats.