Rendering Rails Partials and Forecasting Soccer Results

A few weeks ago, we pushed a code release to our public market that incorporated a decent number of changes. Just as we always do, we were closely monitoring for any unforeseen errors and performance changes after the deployment. After a few hours, we noticed that the site was running a little slower than normal. Assuming our changes had led to this drop in performance, we set out to determine why.

Pulling up New Relic, the great tool that we use to monitor our apps’ performance, we found that we spent 90% of the time it took to process any of our requests rendering this single file that displays trade reasons for a question. And after watching for a few more minutes, that percentage was increasing. But we hadn’t changed anything relating to that piece of code, or at least that we expected would affect that piece of code.

When we looked at where the most recent trades were being placed, we realized that this was being caused by one question that had an extreme disproportionate amount of activity: “Who will win the 2014/2015 UEFA Champions League?” Turns out that one of the two of the semi-finals matches was being played right then, and a surge of new users, mostly located in Italy, were trading in this market.

It wasn’t just that the market was heavily traded though. The other part about this question was that traders were submitting reasons for their trades along with the trade itself, a really helpful option that gets discussions going in a question. So every time someone reloaded the page, we rendered all of the trade reasons submitted in that market, which ended up being a ton because of all the activity.

So in the end, it wasn’t our new code that was causing the slow down. It was just a bunch of soccer fans. After the game ended, we spent a little time only displaying only a few of the trade reasons initially, and if a user wanted to see more, we load them dynamically. A good lesson in only showing users what they need to know at the beginning, and having the option of getting all the information if they want.

The finals of the Champions League finals are coming up this weekend, and this time, we’ll be ready.
Check out that market here!