Swing for the Fences

Until recently, one of the most remarkable unbeaten records in sport belonged to a football manager, the much reviled Portuguese provocateur and clotheshorse José Mourinho. Before Real Madrid lost 1-0 at home to Sporting Gijón on 2 April, no team managed by Mourinho had lost a home league game for more than nine years, a sequence spanning four different clubs in four different countries (Mourinho has also managed Porto, Chelsea and Inter Milan) and lasting 150 matches. It is true that Porto, Chelsea, Inter and Real are all rich and powerful clubs and you would not expect them to lose at home very often – but still, nine years is a mighty long time. Even if you calculate that on average away teams only ever had a 10 per cent chance of beating one of Mourinho’s sides (for some, like Gijón, it might be a lot less, but for others, like Sporting Lisbon, AC Milan, Manchester United or Barcelona, it would be a lot more), the odds against going unbeaten for 150 matches are more than seven million to one.

How did he do it? The difficulty in answering this question is that there are really two puzzles here. The first is Mourinho. Is he supremely talented, or supremely lucky, or a bit of both? Does he have a secret formula, or is the secret that there is no formula, just enough bravado to make it look like he knows what he is doing? But the other puzzle has nothing to do with Mourinho. It is the mystery of home advantage itself. Why is it so hard to beat a team in its own stadium? Why does every team, no matter how unbeatable at home, lose something of its invincibility when playing away? Chelsea, who did not suffer a single home defeat during Mourinho’s three and a bit years in charge, were beaten ten times away from Stamford Bridge during the same period. If you calculate that the chances of beating one of Mourinho’s teams rise to about 20 per cent when they play away from home, then the odds of his going unbeaten for 150 away matches are nearly 350 trillion to one. It’s never going to happen to him or to anyone else.

It’s not only the big clubs. Take any European football league in which all the teams play each other twice in a season, once at home and once away. Add up the total number of home victories and compare it to the total number of away victories. The ratio will be at least 60:40 in favour of the home sides (often it’s more: in the English Premier League home advantage currently runs at around 63 per cent, in Spain’s La Liga it’s 65 and Italy’s Serie A it’s 67). The advantage holds across almost every major sport, though exactly how big it is tends to vary. Fans are so used to this that they take it for granted their team is much more likely to win on its own turf. They also take it for granted that they know why – it’s because the home crowd is cheering the team on. But there is no evidence for this. In fact, despite a fair amount of research in the top sports science journals, there is no conclusive explanation of what makes teams play better at home. This is the real puzzle about home advantage: everyone knows it exists but no one knows why.

Now here come Tobias Moskowitz and Jon Wertheim to clear up the mystery. Scorecasting is a book in what has become an increasingly familiar genre, a Freakonomics-style investigation into everyday phenomena that we take for granted but can’t explain. The two authors fit the requisite profile: Moskowitz is an economist with an interest in sport; Wertheim is a sports journalist with an interest in numbers; they are old friends. They follow the prescribed method: first take something people think they understand but don’t, then crunch some numbers, strip out the variables, throw in a few one-liners to keep your readers interested and voilà! – whatever you are left with is the truth, however improbable it sounds. Sport has always looked like ripe territory for this sort of approach, because there are lots of numbers to crunch and lots of prejudices to explode. But apart from baseball, which has its own well-established sub-genre of statistical myth-busters (known as sabermetricians), the sportonomics books have all been disappointing. Maybe sport makes it too easy: there are so many stats to play with, and so much nonsense is bandied about as though it were true, that it’s tempting to skip the hard work and simply line up the fish in the barrel. Scorecasting doesn’t do that. It is by far the most engaging book of its kind yet published, crisply written, extensively researched and full of surprises. The biggest surprise of all is home advantage.

So what causes it? First, Moskowitz and Wertheim rule out the conventional explanations, starting with the support of the home fans. How do you isolate the effect of the crowd on a team’s performance? They do it by comparing how well home and away players perform when faced with identical tasks, save only for the presence or absence of a hostile audience. Take basketball: when a player is fouled, he (or she) is awarded free throws at the basket from 15 feet. No one is allowed to interfere, apart from the home fans, who can do what they like to put off the opposition. If you’ve ever seen an NBA game in the States you’ll know this often includes shaking rattles and waving balloons from behind the basket. The result? Nothing. The stats show that away players perform just as well as home players from the free-throw line, despite all the barracking. The same applies to goal-kicking in American football, and penalty shoot-outs in our version. The home side has no better chance of winning at penalties than the away team. Home fans often think they can help the ball into the net with their hushed support, or keep it out with their whistling derision. It seems they might as well save their breath.

If it’s not the fans, maybe it’s the travel. Away teams often have to cross long distances (especially in the US), sleep in unfamiliar beds, and deal with all the discomfort of being far from home. This one is easy to disprove. The record of away teams across all sports is just as bad in local derbies, despite the fact that getting to the ground is no more inconvenient than for the home players. Everton’s and Liverpool’s grounds are less than a mile apart – but Everton are still much more likely to beat Liverpool when they don’t have to make the short journey to Anfield. The historical data back this up. Travelling conditions for top athletes have got immeasurably better over time – where once it might have been as slow and difficult for them to get around as for the rest of us, now it tends to be pampered luxury all the way. But their performances away from home have not got better at all. As Moskowitz and Wertheim put it, ‘the home field advantage is almost eerily constant through time.’ You can do what you like to ensure your players do not suffer all the little inconveniences of being on the road – they are still liable to let you down when they arrive. (The one correlation Moskowitz and Wertheim do find between travel and performance is for those sports where away teams are sometimes forced to cram games together on a single road trip, meaning they have a tighter schedule than their opponents. This is true for US college basketball, where away sides are often disadvantaged by having to rush from town to town while the home teams get a day off – and in college basketball home court advantage runs as high as 69 per cent. But here the test is European soccer, where there are no scheduling anomalies, and where home teams win almost as often.)

What about local knowledge? Every ground is slightly different, so perhaps teams take advantage of their familiarity with their home environment. Even football pitches vary: some are wider, some are narrower, some are blowy, some are sheltered, some are rough, some are smooth. The differences are most noticeable in baseball, where some teams play at stadiums that suit hitters, and others at stadiums that suit pitchers (it’s a question of size, shape and atmospherics). Yet even in baseball, Moskowitz and Wertheim find it makes no difference. Teams that play in hitter-friendly stadiums do not outhit their opponents by any greater margin than teams that play in pitcher-friendly stadiums. This despite the fact that managers can pack the team with sluggers, sure that they will play at least half their games in advantageous conditions. Knowing what you need to do well in your own yard doesn’t help you do it any better. Home advantage seems to be entirely outside anyone’s power to control.

It’s not the crowd, it’s not the travel, it’s not the stadiums, it’s not the players or the managers. So what’s left? Well, there are always the referees (or umpires as they are known in most American sports). And that’s who it is – Moskowitz and Wertheim say home advantage is almost entirely down to the officials. Players aren’t put off by the barracking of the home fans, but the umpires are. It makes sense when you think about it – if tens of thousands of semi-hysterical people were scrutinising your performance, you’d want to try to please them if you could, if only subconsciously. The away players have nothing to gain from the home fans – if they do well they’ll get abuse, if they do badly they’ll get mockery. But the officials can make the home crowd happy and then surreptitiously bask in the warm glow. Away players can’t alleviate the pressure of being in a hostile environment. Referees can.

Moskowitz and Wertheim find plenty of evidence to back this up. In football, it turns out that referees consistently award more injury time when home teams are losing, and less when they are winning (on average, four minutes in the first case and two minutes in the second, enough to make a difference in plenty of matches). Home teams get far fewer players sent off, and receive many more free-kicks. Maybe this is down to the fact that the home side simply plays better and the away players are reduced to desperate measures. But Moskowitz and Wertheim find evidence that crowd effects make a real difference. In the German Bundesliga, for instance, where many of the teams used to play in stadiums incorporating running tracks, putting the crowd much further away from the action, the bias referees normally show to the home side was cut in half. In the British, Spanish and Italian leagues, attendance also has a marked effect on the number of red cards shown to the visitors. The bigger the crowd, the more likely the away team are to end up with fewer players on the pitch at the end.

However, the most compelling evidence for referee bias comes from those sports that have introduced technology to check on the decision-making of the officials. In baseball, a system called QuesTec (similar to Hawk-Eye in cricket and tennis) now shows whether a pitch was in the strike zone or not (the area over the home plate between a batter’s armpits and his knees). Moskowitz and Wertheim have looked at a mass of data and discovered that when a pitch is clearly a strike, baseball umpires do not advantage the home hitters. Equally, when a pitch is way outside the strike zone, they call it against the pitcher. But when it’s on the edges, the home team were getting a large percentage of favourable calls. This shows two things. First, given the choice, umpires prefer to please the locals who are breathing down their necks (in many baseball stadiums almost literally). Second, they know what they are doing – they restrict their bias to areas where it won’t be so obvious (in stadiums that have installed QuesTec umpires have started to eliminate their home bias, now that they realise it’s there for all to see). Moskowitz and Wertheim find the same thing in ice hockey and American football, where the introduction of instant replay reviews showed that for close calls, and in tight games, the officials tend to favour the home team by a significant margin (calls against the away side are more likely to be corrected when impartial technology is called in evidence). Tight games are by definition the ones that can turn on one or two key decisions. And it appears that tight games are also the ones in which the officials go out of their way to help the home team. That’s enough for Wertheim and Moskowitz to finger them as almost entirely responsible for the phenomenon of home advantage.

It’s a lovely theory – simple, elegant and in tune with what most of us believe about human nature (and with what many fans have long suspected but never been able to prove about referees). There’s only one problem – it’s not true. I don’t doubt that referee bias has something to do with home advantage, but the idea that it’s the crucial determining factor is absurd. Just think about it – or rather, think twice about it. The first time you’re told it’s the referees you will probably go ‘aha!’, as I did. But the second time you’ll go ‘huh?’ Look at a football game. Yes, the home side does sometimes seem to get the benefit of the doubt from the referee, and yes, injury time does seem to go on for ever when Manchester United are playing at home – the image of Alex Ferguson consulting his watch as United push forward for a winning goal in the 97th minute at Old Trafford is probably the one that defines the Premier League. But why do the home side always seem more likely to score at the end? Why are they the ones pushing forward? Look, really look. It’s not just because the referee is letting them, it’s because something is making them play better. They believe.

At this point the freakonomists will tell me that I’m the one being absurd. The whole point of trusting to the numbers is that we can’t trust our eyes. We think we know what’s really going on only because we have all sorts of cognitive biases that lead us to misread individual situations. The freakonomics approach is designed to rule out what we assume is happening, forcing us to accept that we have been blinded to the truth by our preconceptions. Didn’t Moskowitz and Wertheim rule out all the plausible-seeming alternatives? Well, no they didn’t. They used the numbers to make it look like that was happening. But really they were just expressing their own bias. This is the trouble with the freakonomics approach. It’s not that the numbers misrepresent human nature by treating us all as twitchy little utility maximisers. Doubtless that’s what most of us are most of the time. The problem is that the freakonomists misrepresent the numbers.

Let’s spool back. Moskowitz and Wertheim claim that the performance of players from the free-throw line or the penalty spot shows the crowd doesn’t have an impact on the performance of the home team. But that’s not what it shows at all: it shows that the crowd doesn’t have an impact on individuals. What if home advantage is a team phenomenon? There is plenty of evidence not considered by Moskowitz and Wertheim to suggest that it is. British tennis players have never seemed to gain much advantage playing at Wimbledon, despite the presence of thousands of people willing balls that are in to be called out (I’m talking pre-Hawk-Eye here). Are phlegmatic British line judges somehow impervious to these pressures in a way that football referees are not? It’s not just us Brits. No Frenchman has won the French Open since 1983; no Australian has won the Australian Open since 1976. Where’s the home advantage? One explanation might be that playing at home really makes a difference only when you’re part of a team. It’s a collective experience, in which case it dissipates for isolated individuals (including the individuals standing at the free-throw line in a basketball game or at the penalty spot in a football match). Somehow, playing at home breeds a sense of solidarity, or what used to be called team spirit, which means that players have more confidence in each other and work better as a unit. I’m not saying that’s definitely what happens. But Moskowitz and Wertheim haven’t proved that it doesn’t.

The key figure that they don’t really discuss is the disparity between home advantage in baseball and football. Baseball has a relatively low home advantage ratio – the lowest for all major sports – at around 54 per cent for the major leagues. This is a huge difference from the 63-67 per cent that holds for the big European soccer leagues. What explains it? Moskowitz and Wertheim spend a lot of time describing how the bias of baseball umpires can account for almost all the home advantage in that sport – if it sways around 3 per cent of games (and they give good reasons for thinking that it does), then that’s practically the whole of it. But what about the extra 10 per cent in football? Their answer is that football is a sport where the referee’s decisions count for more. But they provide no statistics to support this. In fact, they read it backwards: since they are committed to their theory that referee bias accounts for home advantage, and since home advantage is much greater in football, QED referees must have more influence on the games. So who’s suffering from cognitive bias now?

What’s striking about Scorecasting is that there are numbers that could prove Moskowitz and Wertheim’s thesis, but they don’t provide them. Take the German case: if the presence of a running track cuts referee bias in half then it also ought to have cut the home advantage of the teams playing at those stadiums in half. Did it? They don’t say, but somehow I doubt it. Similarly, it should be possible to provide some numbers to decide the question of how much difference refereeing errors make to the outcome of football matches. Moskowitz and Wertheim don’t even try (all they tell us is when a team gets a player sent off, it is considerably more likely to lose – no kidding!). So I’ll have a stab, though I’m only guessing. Let’s say the mistakes of football referees favour the home side by a ratio of 60:40 (corresponding with the basic home advantage ratio). It also seems reasonable to believe that the mistakes of referees decide perhaps 20 per cent of all football matches. But that would still only give a home win advantage of 4 per cent. To get to 20 per cent it would either have to be the case that every refereeing error favoured the home side, or that every football game was decided by a refereeing error. That’s absurd. I am also prepared to believe that the extra 2 per cent of home advantage in Spain, and 4 per cent in Italy, is down to more suggestible referees (though whether it’s the crowds that are influencing them or something more sinister is open to question). But whichever way you spin it, it seems that the bulk of home advantage in football is still unexplained by refereeing bias.

So here’s an alternative explanation: home advantage is lower for baseball because it’s less of a team sport. It’s primarily a series of individual encounters between batters and pitchers. It’s more like tennis than like football. Playing at home makes the biggest difference to passing sports, where the players have to rely on each other. (There is some passing in baseball, from fielder to fielder, but much less than in football or other sports where home advantage is very pronounced, like basketball and ice hockey.) Baseball is also a more disjointed sport (again like tennis): it consists of lots of discrete plays. Team sports where the action flows are the ones in which playing at home really counts. Why? I’m not sure. But Moskowitz and Wertheim have not ruled it out.

I can’t prove my theory, but I can defend it. It chimes with what you can see happening in any team sport – the away players don’t quite believe in themselves in the way the home players do. This is especially true near the end of a close game between two otherwise evenly matched teams, when the home side will usually be the one pressing for a winner. Moskowitz and Wertheim say that’s because the referee is allowing it to happen: what we think is a quality belonging to the players is actually a quality we have misattributed to them because of the indulgence of the officials. But that’s not really convincing – not only does it not tally with the evidence of our own eyes (the home team attacks even during the periods when the referee has no influence on the game) but it doesn’t fit with another claim they make in Scorecasting. As well as identifying a bias in favour of the home side, they also show that officials prefer to avoid making decisions that might make them stand out, especially near the end of a game. This is a widespread phenomenon, and it applies to football as much as any other sport. As Moskowitz and Wertheim report, having studied 15 years of data from the Premier League, La Liga and Serie A: ‘Fouls, offsides and free kicks diminish significantly as close matches draw to a close.’ This is the ‘omission’ bias, and we all tend to suffer from it – we prefer to let bad things happen than to take a chance on doing the right thing and risk carrying the can. If a referee intervenes in a game near the end, it looks like he’s deciding the outcome. That’s going to make some people mad.

Moskowitz and Wertheim describe a classic sporting example of what can happen when an official tries to overcome his or her omission bias. At the 2009 US Open a brave/foolhardy tennis line judge called a foot-fault against Serena Williams at the climax of her semi-final against Kim Clijsters. Subsequent replays showed the call was correct. But it provoked outrage. Line judges rarely call foot-faults, since they don’t want to look conspicuous. What was this one doing interposing herself at such a crucial moment in the match and helping to decide the outcome? Just doing her job? Come on – she was making a spectacle of herself. After she had been foot-faulted, Williams turned on the official and screamed at her: ‘You better be fucking right! You don’t fucking know me! … If I could, I would take this fucking ball and shove it down your fucking throat!’ This outburst meant Williams was docked a point, which cost her the match. The crowd went crazy. John McEnroe commentating on television, agreed: ‘You can’t call that there. Not at that point in the match.’ So it turns out it didn’t matter that the line judge was fucking right; she was still run out of town.

As Moskowitz and Wertheim show, most officials have internalised these sorts of lesson. They don’t like to interpose themselves at crucial moments, when people can say: it was the goddam ump! They don’t even like to make calls at any point in a game when their decision will stand out. So, for instance, in baseball, when an umpire has made three consecutive calls against the pitcher, meaning one more ‘ball’ (a pitch called outside the strike-zone) will give the batter a free walk to first base, he usually shies away from making the call. Better to call a strike, so it doesn’t look like the umpire has dominated that little period of play. This has important implications. It’s a truism of baseball coaching that when hitters are at 3-0 (three balls, no strikes) they shouldn’t swing at the next pitch. Don’t waste a dominant position. Make the pitcher, who is under all the pressure, get it over the plate. But in fact the umpire, who’s really the one under pressure, will probably see the ball as going over the plate regardless of whether it is or not. So the truism is false – you’re better off swinging. This lesson can be applied to many areas of life. Say you’re going to a job interview. You know you’re the outstanding candidate, so you decide to play it safe. But if you really are the outstanding candidate, the umpires are unlikely to want to strike you out on the basis of a single interview. So you might as well swing for the fences.

In any team sport, players and managers suffer from omission bias too: they don’t want to make the dumb mistake or dumb tactical switch that means they can be fingered for a loss. Closely related to omission bias is loss aversion, which is also a widespread human trait. People prefer avoiding losses to acquiring gains, even if the payoffs are the same: given a choice between tossing a coin to win a dollar or tossing a coin to hand back a dollar we have just been given, most people prefer the first over the second. Golfers, when faced with putts of identical length, one for birdie and one for par, will hit the birdie putt more conservatively, even though the penalty of missing in both cases is the same: a dropped shot. It’s just that it feels worse when the shot is dropped to par, an entirely arbitrary measure. Similarly, most coaches would rather avoid contributing to a loss than take a chance on contributing to a win: they play it safe. Across all sports, coaches tend to make the plays that give them the best chance of preserving what they have (including their jobs). But it also means that the best coaches recognise this weakness in their rivals and exploit it. Widespread loss aversion gives the coaches who don’t mind looking stupid a competitive advantage.

Which brings us back to Mourinho. The most plausible explanation for his astonishing home record is that he is a manager who is not afraid to take risks. He has a reputation as a conservative coach, but in fact he is very adventurous, albeit in an unattractive way. He’s not afraid of doing ugly things that draw attention to himself – he actually seems to relish it. The reason he is currently held in such opprobrium is that his Real Madrid side lost to Barcelona after adopting astonishingly negative tactics in their Champions League semi-final. It was hideous to watch. But it said two things about Mourinho: first, he wasn’t frightened of risking a loss that could clearly be pinned on him; second, he thinks he’s worked out the best way to beat Barcelona. And he might just be right: his recent humiliation notwithstanding, he’s done better against them than anyone else (including beating them with Inter Milan in the Champions League last year). Similarly, no one could maintain a 150-game unbeaten record by playing safe. It’s the coaches who try not to lose who get beaten in the end.

One of the many ugly aspects to the way Mourinho’s teams play is that they are not afraid to harass the referees. No doubt Mourinho worked out early on that this is an important part of home advantage. But that’s not all he’s worked out. His teams are so hard to beat at home for reasons that go well beyond the reductive account provided by Moskowitz and Wertheim. Home advantage is much more complicated and much more mysterious. It depends on a range of factors that are effectively impossible to quantify. Maybe even Mourinho’s gorgeous clothes are part of the package: perhaps looking good helps to instil confidence in his team that their boss is the boss, and his territory is theirs to defend. But whatever is really going on, Mourinho must know that the idea home advantage can be reduced to, and blamed on, referees is just the sort of conservative, risk-averse thinking that gets you into trouble. So he knows something the freakonomists don’t: you can’t always trust the numbers.