Monday, February 23, 2015

Shaqtin-a-bias?


All NBA fans know about Shaqtin-a-fool.
Once a week, Shaquille O'Neal hosts this small segment on the NBA on TNT show. Five humorous video clips are shown, with players definitely not at their best. Erratic passes, obvious travels, missed wide-open dunks and layups, lost shoes...
The segment is also available on nba.com, and fans can vote for the best Shaqtin-a-fool moment.


For volume 4 episode 11 (they're referenced just like a TV series, with season and episode), and similarly to over 50% of the voters, I had voted for the last video clip shown which was that week's clear winner. A weird sensation I had been carrying over from week to week suddenly materialized: it seemed to me that the last video clip was winning a disproportionate number of times.

Two explanations came to mind: the video clips were not shown randomly in Shaq's segment, but sorted according to users' preferences. Or the human mind was biased with its short term memory, not exactly remembering the first clips, and finding the last disproportionately funnier.

It was all the more obvious for this episode 11, where the poll results were in the exact reverse order they were shown in:



But before investigating the human brain and mind too deeply, I first had to see if my brain wasn't the one tricking me, and sought statistical confirmation that there was indeed a bias favoring the last video shown.

First things first, data was required. Unable to automatically run a script to pull the survey results from polldaddy.com, I manually went through the last 28 episodes (including some special episodes for the All Star Game, the Playoffs and past eras), noting for each video the order it was shown ("Input Order"), and the position it was in the survey results ("Output Ranking").

A quick first visual exploration of the data, linking Input Order to Output Ranking:


I added some jitter to avoid all the lines overlaying each other and hiding the number of observations. It did seem that the majority of the lines were in the steepest diagonal, indicating that the most common "transition" was from videos being shown in 5th position coming out first in the survey results. At least I wasn't imagining the whole thing!

Because the diagonal has longer length than horizontal lines, there could still be an optical illusion suggesting that indeed there are more lines when we are actually seeing more color from longer lines, not more lines. So I re-generated the same graph but reversing the order of the inputs, so that the last video shown is not labelled 1, and the first video shown is 5.


No visual trick here, definitely looks like the last video shown is the most likely to win the poll (horizontal lines going from 1 to 1).

Now for the statistical confirmation. The most suited test here is a chi-square, comparing observed counts with expected counts under the null hypothesis that video order doesn't matter and all videos are equally likely to end up in any position.

The first test I ran looked at the full data and all the Input Order - Output Ranking counts:

Output: 1 Output: 2 Output: 3 Output: 4 Output: 5
Input: 1 3 3 13 6 3
Input: 2 1 2 3 10 12
Input: 3 4 5 6 9 4
Input: 4 3 8 5 3 9
Input: 5 17 10 1 0 0

The chi-square strongly rejected the null hypothesis: input order and output ranking were strongly linked.

The second test focused uniquely on the winner of the poll. In which position was the winner shown?

The table below summarizes the data:

Input: 1 Input: 2 Input: 3 Input: 4 Input: 5
Count 3 1 4 3 17

That's right, in 60% of cases the survey winner was shown in last position! It's clear from the data that not all positions are created equally and a second chi-square confirmed this.

So back to Shaq. Now that we've confirmed that there is a strong bias, can we try explaining the phenomenon?

My first idea (perhaps having spent too much time doing analyses for marketing teams!) was that the videos were not randomly shown but already sorted according to expected viewers' preference. It's a possibility, but a rather weak one. What would be the rationale? To get people hooked on the show as the clips get funnier and funnier? Sure, but recall that the whole Shaqtin-a-fool lasts 2-3 minutes tops, I'm not not sure if users really need to get hooked. Plus, until they see the last videos, the audience has no way of determining whether the best videos have already been shown.

So, I'm actually leaning towards an unconscious bias. I think the same phenomenon occurs if you were asked to rank your best vacations. There might be some clear "great vacations" (honeymoon), and "bad vacations" (lost wallet, passport, got sick), but I believe that with equally enjoyable vacations, the brain might be tempted to rank the latest one higher. A modality effect has been documented usually to describe the improved recall of the last elements of a list, typically when these are presented visually or auditory. I'd be willing to bet something similar is at play here.

However, even if the survey results are much more predictable now, I'm still going to continue watching Shaqtin-a-fool religiously. For pleasure... and more data.

Thursday, February 12, 2015

Are freethrows game-changers?


"And another missed free throw!"
"That's the story of the game right there, they just can't get those easy points from the charity line."

I've heard very similar discussions to this one over and over throughout the years from basketball commentators. Although not truly meaning it (at least I think), the commentator was heavily implying that the outcome of the game would be extremely different if a given team had made all, or significantly more, of its free throws.

Don't expect a sophisticated analysis here, I was just curious to explore the correlation between difference in final score and number of missed free throws.



So I wanted to investigate the two following questions:
  1. If the losing team had made all its attempted free throws, would the outcome of the game have been different?
  2. If the losing team AND the winning team had made all of their attempted free throws, would the outcome of the game have been different?

To answer those questions, I pulled all boxscores for regular season games from the 1999-2000 season to the last complete one, 2013-2014 and for each game tracked the final score difference, as well as the number of missed free throws for both the winning and losing teams.

Here's a first quick visual of the relationship between the number of missed free throws for each team (losing team on the lefthand graph, winning team on the righthand graph) and the final score difference:



Rather surprisingly, there does not appear to be any link between the number of missed free throws and the outcome of the came in terms of close game or huge blowout. It would have seemed natural to assume that the more free throws the loser team has, the more likely they are of getting blown out, and the opposite argument for the winning team.

How have these numbers evolved over time? Here's the evolution of the average score difference and missed free throws for those 15 seasons:



Not completely obvious trends emerge from the graph, but if anything can somewhat notice that:

  • the lines for the winning and losing team are extremely similar
  • average score difference has stayed flat or perhaps very slightly increased
  • number of missed free throws has decreased (could be due to better shooting and/or less free throws attempted, and it's actually a little of both)

Now of course this analysis has been as naive as they come. You can't just expect free throws to go from missed to made and expect the entire game to follow its original course. Players might get confidence as they rack up easy points, and coaches might change strategies if what would have been a big lead is only a 2/3 point lead. The point of the exercise here was to compare the range of final point differential to number of missed free throws, and it would seem that free throws only account for about half the final gap. This could be a reason why teams, coaches, players don't put in crazy efforts to have all players shoot 99%. Their time is probably better spent on other types of training.

That being said, I just had to mention the other day's game which saw both teams shoot a combined 37% (16 for 43, 8 for 25 for the Clippers, 8 for 18 for the Nets) from the free throw line. To put things in perspective, Shaquille O'Neal who was criticized his entire career for those shots was 53% over his career (despite finding elaborate strategies to boost the percentage). Consider the Clippers lost by a mere two points 100-102, I'm sure they must be kicking themselves for their performance at the line.

Also worth noting, this game from 1999 between Portland and the Lakers. Portland won quite big - by 15 - but missed only one free throw. The Lakers missed 17! Had both teams been perfect, the Lakers could have actually won a game they lost by 15. This was the biggest outcome reversal I observed in the data if both teams had been perfect.



Thursday, February 5, 2015

Are we seeing All-Stars at the All-Star?


The starters for the Western and Eastern teams of the upcoming NBA All-Star game were just announced Jan 22nd. The selection was uniquely based on fan votes.

In the West we have the vote-leading player Steph Curry, along with Marc Gasol, Blake Griffin, Kobe Bryant and Anthony Davis. Their Eastern counterparts will be Pau Gasol (not sure how often two brothers have faced each other in an All Star Game...), LeBron James, Kyle Lowry, John Wall and Carmelo Anthony.

The selection did raise quite a few eyebrows to say the least. Kobe? Sure he's an NBA legend, future hall-of-famer and all, but look at the Lakers record this season, look at his abysmal shooting percentage of 37.3%. Carmelo is also somewhat of a surprise given how the Knicks are performing this year. Sure the All Star is not about the team but the player, but his stats aren't eye-popping either. And then consider all the ones who didn't get in, James Harden, Klay Thompson, the entire Atlanta Hawk roster... Even if not for those reasons but purely on the voting volume, Mark Cuban declared the voting system broken.


fivethirtyeight.com had a very interesting post on the topic, attempting to correlate players' performance with the number of votes received. Performance was measured in terms of Win Above Replacement (WAR), the number of team wins attributable to that player (computed as the difference between the number of wins the team got with that player in the game, versus a hypothetical world where the player is replaced by an average player). It does seem that the above a certain threshold, high-impact players get the votes they deserve, but under that threshold it's all more or less random.

Now I think the real question is: what do we want in an All Star game? Players naturally view it as an honor, a testimony of a great year they're having. But are fans voting for players deserving recognition? Or do they want pure 100% showtime? Imagine a natural born dunker, explosive, athletic and artistic at the rim. Even if that player had below average EFG%, below average WAR, RPM, RAPM or any of the other advanced metrics to measure player performance, wouldn't fans still want to see him in the All Star game?


So while I'm not saying it's fair to the players, I can understand why a Kobe would get voted in, and why a Paul Millsap or Kyle Korver wouldn't. If we really want to understand how fans vote, it would be interesting to see if we could find a metric that better correlates with player votes than WAR. Or perhaps first start including WAR for past seasons as well? I'm sure that if we did that we would have a better understanding as to why Kobe got voted. But how about a combination of team wins + number of dunks in the season? Or number of fast break points?

The debate does seem old and familiar, perhaps because it's so closely related to the one we have every single year about who should be MVP and how MVP is defined? The player with the stellar stats? The player who was most impactful on his team's success?



Thursday, January 29, 2015

One stat to rule them all? It would be a steal


It's been almost a year since Benjamin Morris wrote about The Hidden Value of the NBA Steal on fivethirtyeight.com, and a lot of criticism to say the least followed suit (two examples here and here).

The main criticism stemmed from the comment that "a steal is worth nine points", which caused many to throw their arms up in the air wondering how a player could all of a sudden score nine points in a single try without being in NBA Jam.

My purpose is not to review the original article, the criticisms nor review Morris's four part (!) response (kudos for tackling all the negative comments head-on). However, it is to be noted that since the steal article (Morris' third on fivethirtyeight after two others on basketball), Morris has primarily been tackling other sports than basketball (only 5 of 48, this is an advantage to writing this post so late after the fact).


Trying to take a step back, my attempt was to see how valuable indeed a steal is for measuring the value of a player. If I had to draft/trade for either a player who gets 25 points a game and 1 steal or one who has 16 points and 2 steals (to recycle Morris' example), who should I go for?

There is no perfect gold standard for summarizing a player into a single metric, although their are multiple options that get more and more sophisticated. ESPN reports RPM and WAR, defined on the site as:

  • RPM: Player's estimated on-court impact on team performance, measured in net point differential per 100 offensive and defensive possessions. RPM takes into account teammates, opponents and additional factors
  • WAR: The estimated number of team wins attributable to each player, based on RPM

So are steals are good proxy for a player's "value" assuming RPM and WAR are reliable value metrics?

I generated the following graphs linking steals per game with each of the two variables for the top 30 players in steals for the 2013-2014 season. The two graphs are extremely similar given the strong correlation between RPM and WAR.



I don't know about you, but I'm not seeing a strong correlation with steals.

This doesn't validate or invalidate Morris' analysis, but I thought it would be helpful to get some insight as to whether steals is really as powerful as the original paper would suggest.

I know I said I wouldn't comment on the back-and-forths between Morris and the critics, but one comment I had which I didn't see anywhere was around the fact that Morris seems to focus on steals per game, not my minute, not by possession. It's easier to get more steals if I play more minutes, and I might play more minutes if I'm a good player to start of with, so even if we had found a correlation it wouldn't have allowed us to reach any valuable conclusions.


Saturday, January 24, 2015

Unbe-klay-vable! (apologies, klay-verest pun I could think of)


Exactly nine years and two days ago, Kobe Bryant scored 81 points in a game.
Yesterday, Klay Thompson had a historical feat of his own: 52 points, which in itself is not jaw-dropping, but the way he recorded it was, thanks to 37 points in the third quarter alone (9/9 from 3-point land and 4/4 for 2-pointers).

If there ever was a definition of a player being hot, we witnessed it yesterday! Tracy McGrady did score 13 points in 35 seconds, but what Klay did is on another level.


But for the sake of some fun stat: What was the probability of Klay putting on this insane shooting display?

For the 2014-2015, Klay started the third quarter having attempted 272 3-pointers and made 120 (44.1%). He had also attempted 386 2-pointers and converted 188 of them (48.7%).

Assuming all his third quarter shots are independent of each other (very reasonable assumption, the only thing that could invalidate it is if there was such a thing as a "catching fire" effect), then the probability of Klay scoring a perfect 9-of-9 3-pointers and 4-of-4 2-pointers is 0.441^9 * 0.487^4 = 3.6e-5! Or 1 in almost 30,000. Basically, you would expect such a performance once every 342 seasons! Or Klay would have a higher probability of getting struck by lighting once in his life.

One question that this mind-boggling performance also raises is whether it has cemented Klay Thompson to be as inconsistent as ever? I am sure Nobel prize winner Daniel Kahneman will have some thoughts about this.

Even if you've seen the highlights countless times already, never hurts to review this once-in-342-seasons performance:










Tuesday, January 6, 2015

Fifteen seconds remaining, down by one....Who you gonna foul?


I was following the Mavericks - Kings game earlier this year. Exciting game which went into overtime. With 50 seconds left and the Kings down by two, Rajon Rondo fouled Jason Thompson and in the process sent him to the free throw line. He made the first shot. But before he could attempt the second, the Mavs called a timeout. After the timeout, Jason went back to shoot his second free throw, and missed it.

Did the timeout have any impact on the missed shot? You couldn't make a free throw any more straightforward. Unlike a penalty kick at soccer, there's nothing your opponents can do to alter that shot. Except call a timeout? Legendary coach Phil Jackson was (in)famous for calling timeouts between opponents free throws, but was this ploy effective at all? Thompson could have tied the game on his second free throw attempt, which would have shifted a considerable amount of pressure of his and his teammates' shoulders with less than a minute to go. With the timeout called, Thompson was left there brewing in these thoughts with mounting pressure.


That game made me want to investigate the timeout phenomenon, as well as other external factors that could influence the outcome of a free throw. I also wanted to follow up on an earlier post I made about measuring players' clutch performances via statistical models.

A few words on the data before jumping into the analysis. I focused on the most recent complete NBA season: 2013-2014. I pulled all the play-by-play data from nba.com, and pulled free throw season percentages for each player from espn.
Quite a bit of cleaning up was required, namely around players with same last name and same team the worst example being the Morris twins in Phoenix who also share same initial!
After cleaning everything up, I was left with just under 56K free throws taken in that season, ready to be analyzed!

I was primarily interested in the impact of free throws interrupted by timeouts, but also wanted to capture two additional factors: whether the shooter has homecourt advantage or not, and whether the situation is "clutch". There are countless definitions of "clutch time" available, some sparking heated debates. I have here defined it as "less than 2 minutes to play in the 4th quarter or in overtime, and less than 5 point differential between the teams' scores".

Before jumping into the data and analysis, let's first do some visual explorations.

How many free throws are taken by quarter?

Not surprisingly, significantly more free throws are taken in the fourth quarter than the first. The game is on the line, the defense goes up a notch, and voluntary fouls are committed to regain ball possession and prevent the opponent from running down the clock.

We can even go down one granularity level at look at the number of free throws made by minute played. Rather impressive to visualize the steady increase throughout each quarter, and the giant spike in the final minute of regulation with teams fouling on purpose in tight games.


We've looked at volume, let's know look at efficiency. How well do the home and road teams shoot the ball?


It appears that both teams shoot at very similar rates throughout the contest, with the home team always having an advantage although it is not a significant as one might have expected given the distractions often displayed by the home fans.


Both teams seem to do better in overtime, but we need to caution against the much smaller sample size there.

And now to the more interesting piece, how do teams execute in clutch time?


Quite surprisingly, the home team appears to be performing no differently, whereas the road team gets a nice boost of almost 5%. The fact that we observe a boost might seem counterintuitive for some: under pressure, with fatigue from close to 48 minutes of gameplay, wouldn't it be more difficult to concentrate and sink the shot? However, a reverse argument could be made that especially when games are close, or when a team expects the other one to intentionally foul, the coach might chose to place his best shooters on the floor. So teams aren't necessarily shooting better, just having better shooters take the shots. This however does not fully explain why the road team has a boost and not the home team.

The following graph shows the 1st quantile, median and third quantile for season free throw percentage of the players taking shots in and out of clutch. It is rather apparent that better shooters are on the floor in clutch moments.


Now that we have a better feel for the data, the analysis can begin. The data is extremely rich and offers multiple options from a statistical analysis point of view. We can leave the baseline free throw shooting percentages for each player be determined by the model fitting, or force these to be the players' season averages. But with different players taking a very different number of free throws within a season, and strong dependency in the success of a free throw for all those taken by the same player, a hierarchical structure emerges and a mixed effects model could make sense.

I actually played around with the three options just mentioned, and was satisfied at how close the numerical outputs were to each other.

The conclusions would indicate that:
  • homecourt does have a positive effect on shooters' success, although the effect was only borderline significant
  • calling a timeout before the second (or third) free throw had a negative but insignificant impact
  • clutch time had a negative and significant impact

Regarding timeouts, the fact that the effect was not significant could be due to the low sample size of these events (84 cases in 2013-2014 out of 56K free throws taken), more coaches should test this strategy so I can tell them if it's effective or not!

As for clutch time, the conclusion seems to contradict the visual exploration where percentages were higher in clutch time. But recall that our explanation to this was that the coaches were putting better shooters on the court. The analysis would indicate that even if the best shooters are on the floor in the closing minutes, they are individually performing less well when the game is on the line than in the middle of the second quarter.

Now one might wonder if we could use the data to detect some of the leagues best clutch free throw shooters. Those cold-blooded killers who can step it up an extra notch when all eyes are on them. The Durants, James, Bryants...

I added some interaction terms for players with sufficient (20) in and out of clutch time free throws and see which ones had the potential to elevate their game. And the results are... no one! Out of the 26 players meeting my criteria, none could significantly increase their free throw percentage. This could again be due to small sample size, but even so most players had negative coefficients. While none had significant positive coefficients, two had significant negative coefficients: Chris Paul and Ramon Sessions.

So back to the post's title, if you're playing the Clippers, fifteen seconds to go and down by one, do you foul Chris Paul?





Monday, December 15, 2014

Clutch or not clutch, that is the stat question


There has always been debate on what clutch is, how it is measured, who performs best in these moments, the list goes on...
I'm definitely not going to settle the debate once and for all in one blog post, but wanted to share a few thoughts and ideas instead.

When we talk about clutch in the NBA, a few names immediately come to mind, Jordan, Bryant, Bird, Miller and Horry. Countless lists can be found with the simplest search, all as subjective as the next ("that was an incredible play in that game!").

But can we actually measure it, and rank players by it?

nba.com has a whole section dedicated to clutch stats on its website. A great first step but amidst all the numbers it's hard to compare players.

SBNation also tackled the issue, clustering players into recipients, creators and scorers (not mutually exclusive) during clutch time (Is Kobe Bryant Actually Clutch? Looking At The NBA's Best Performers In Crunch Time). The article stresses the importance of efficiency by placing all performances in perspective using possessions per 48 minutes on the x-axis. Here are the results for the 2010-2011 season:


Efficiency is a trademark at SB Nation, and Kobe is their primary scapegoat given this viewpoint:


SBNation's perspective is interesting and allows swift comparisons across players, but I feel that it lacks some rigor and robustness around these numbers. How large are the sample sizes? Are the effects significant? Which players are shouldering the most pressure and confronting it head-on? The author underlines these issues himself:

But all of that said ... how reliable are these numbers? There's a school of thought that firmly believes that "clutch" is in the eye of the beholder. They contend that as fans, we see things that may not actually be there. We see Kobe hit a step-back 20-footer and credit his clutch ability, when perhaps we simply should have attributed it to the fact that he's amazing at basketball (in the 1st or 4th quarter).
There are rigorous methods of testing for statistical significance. Rather than dive into those, however, a glance at some yearly efficiency trends can be just as telling.


I also came across this very nice post, Measuring Clutch Play in the NBA, on the Inpredictable blog which offers an interesting and elegant alternative. In a nutshell, the idea is to look at how each player's actions impacted his team's probability of winning the game, referred to as Win Probability Added WPA. Made shots, rebounds, steals increase your team's probability, while missed shots and turnovers hurt it. Some adjustments are required to clean up the cumulative WPA for each player (essentially comparing the impact of the same play under normal circumstances), but it does at the end provide an intuitive metric that makes sense and allows quick comparisons.

I do however have some slight concerns with this metric. The first is that, unless I misread, the metric is cumulative, so that players with more minutes in the clutch have more opportunities to modify their team's WPA. The second is best illustrated with a small example: with a few seconds remaining, if a player makes a two-point shot with his team down by 2 or down by 1, it will make a huge impact on the WPA: in the first case they're tied, likely to go to overtime with 50/50% for each team to win the game, in the other case his team leads by 1 and have a good chance of winning the game. But is it fair to credit the player with very different WPA in both cases? What really matters is that, under tremendous pressure, the player made the shot.


This in turn leads to another question: what was the likelihood of that shot going in in the first place? How frequently does that player make that shot under normal circumstances without the game on the line? How frequently do other players make the shot? How much does clutch pressure reduce the average player's chance of making the shot, and was the player able to rise to the occasion and overcome the pressure?

According to Stephen Shea in his book Basketball Analytics, "90% of teams performed worse (in terms of shooting percentages) in the clutch than in non-clutch situations." Can this me modeled? How significant is the effect?

I will try to explore this path further, looking into statistical models that would offer some elements of response to these questions.

But looking at all hat has been said it seems the debate originates from the fact that "being clutch" is never well-defined. Suppose we could at any point during a game give a score from 0 to 100 as to how good a player is. Suppose player A is at 90 throughout non-clutch times, but drops to 80 in clutch situations. Whereas player B is at 60 in non-cutch situations, but steps his game up to 70 when the game is on the line. Which is clutchier? The one with highest absolute value, or the one stepping up his game and taking the pressure head-on. Answering this would already be a giant step in the right direction.

In the meantime, please enjoy this youtube compilation of clutch shots: