Love at first sight? Evolution of a movie's rating

Every day, over 3 million unique visitors go to imdb.com and I am very often one of those. With limited time to watch movies, I heavily rely on IMDB's ratings to determine whether a movie is a rental or theatre go/no-go.

My memory is sufficiently bad that I sometimes need to check a new release's rating a few days apart, but sufficiently good that I can remember rating changes. The most common scenario consists of me checking a whole bunch of ratings for different movies, then trying to talk my wife into going to see the one I like best, using IMDB's rating as an extra argument. She'll systematically - and skeptically - ask for the movie's rating (she's also part of the daily 3 million). And I'll say it's really good, something like 8.3, here let me show you... and a 7.4 appears on my screen and I look like a fool.

Of course, it's completely expected that IMDB ratings would evolve, and even more so when the movies have recently released and have few voters: I fully anticipate Charlie Chaplin's City Lights to still have an 8.6 rating a month from now, but don't think Ted 2 will still have a 7.2 next month, or even when this post gets published! But the question I had was how do movie ratings evolve? How long does it take them to reach their asymptotic value? Are movies over- or under-rated right around their release date? It would seem reasonable that people would have a tendency to overestimate new movies they just saw at the movie theatre. The bias is due to the fact that if they made the effort of going to see the movie shortly after it's release they were probably anticipating it would be worth their money and time. Therefore, they might overrate the movie after seeing it independently of its quality to remain coherent with their prior expectations ('coherency principle' in psychology).
An internet blogger by the name of Gary didn't really phrase it as such, taking the approach of insulting people who bumped 'Up' to the 18th position of best all-time movies in IMDB. A fun read.

To monitor rating evolution, I extracted daily IMDB data up until 2015 data for 22 different movies released in 2012: Branded, Cloud Atlas, Cosmopolis, Dredd 3D, Ice Age: Continental Drift, Killing Them Softly, Lincoln, Paranormal Activity, Paranorman, Rec3: Genesis, Resident Evil Retribution, Rise of the Guardians, Savages, Skyfall, Sparkle, The Big Wedding, The Bourne Legacy, The Dark Knight, The Expendables, The Words, Total Recall and Twilight: Breaking Dawn 2.

For each movie I recorded three maun metrics of interest: IMDB rating, the number of voters and the metascore (from metacritic.com which aggregates reviews to generate a unique rating out of 100 for movies, TV series, music and video games).

Here's an example of the data plotted for The Dark Knight:

Originally rated 9.2, it dropped to 8.8 in the first month, then dropped a little more to 8.6 after 6 months where it appears to have stabilized. I just checked and it seems to have dropped an additional 0.1 point, now at 8.5 three years after release. Let's now look at the number of people who voted for it:

The number of voters rapidly increased right after the release, and although it isn't increasing as fast afterwards, many people continue to vote for it. This curve is quite typical across all movies.

Finally, let's look at the metacritic score:

I'm not sure we can even talk about a curve here. Momentarily rated 85, metascore dropped to 78 at release and hasn't changed to this date. This is perfectly normal as the metascore is based on a small sample of official critics ('Hollywood Reporter', 'Los Angeles Times', 'USA Today'). Reviews are released around the same time the movie is, no critic is going to be reviewing the Dark Knight Rises today which is why metascores are so stable.

Ignoring the y-axes, the shape of the curves are quite similar across movies, though there are some outliers worth showing.

Increasing IMDB rating? Most movies seem to get overrated at first and stabilize to a lower asymptotic rating. But in certain cases we see the rating increase after the release, as seen here with The Big Wedding with Robert De Niro and Diane Keaton:

Still with The Big Wedding, staggered worldwide release dates are clearly highlighted from the shape of the number of voters.

Based on our small sample can we estimate the overestimation of a movie's rating when it is released. After how many months does the rating stabilize?

Combining the data for all the movies together and aligning them based on their release date (thus ignoring and seasonal effects), we obtain the following graph where the x-axis is weeks since release:

We see a steady decline in rating by about 0.6 points over a period of about 7 months. A more surprising phenomenon is the upward trend in ratings that starts about a year after the original release. The trend seems quite strong, however we should keep in mind
that our original sample size of movies was small (22), and we only have data beyond 75 weeks after release for a handful of those movies, so the upward trend on the very right could be completely artificial and a great example of overfitted data!

A similar break in trend occurs for the number of voters:

As for the metascore, it appears remarkably stable right after release for the reasons already mentioned previously.

In a nutshell, movies do appear to be very slightly overestimated at release time (assuming the long-term asymptote is a movie's "true" rating), and the difference in rating (approximately 0.3 between the first month and months 2 through 6) was small yet significant (based on a paired t-test).

So if you do use IMDB to help your movie selection, definitely keep in mind that while the movie is probably good, it most likely isn't as good as 'Up'.