In my paper on the impact of the recent fracking boom on local economic outcomes, I am estimating models with varying levels of fixed effects. These fixed effects are useful, because they take out, e.g. industry specific heterogeneity at the county level - or state specific time shocks. This is particularly important, since the recession of 2008/ 2009 did not impact all states and industries equally strongly, as is suggested in Mian, A., & Sufi, A. (2011). What Explains High Unemployment ? The Aggregate Demand Channel.
The models can take the form:
Such a specification takes out arbitrary state-specific time shocks and industry specific shocks, which are particularly important in my research context as the recession hit tradable industries more than non-tradable sectors.
How can we estimate such a specification?
Running such a regression straight with the
lm in R or
reg in stata will not make you happy, as you will need to invert a huge matrix. In stata, running such a regression using
areg will not be feasible, as you can only xtset or absorb one fixed-effect, which means, you will still have to evaluate and invert a huge matrix.
However, there is a way around this by applying the Frisch-Waugh Lovell theorem iteratively (remember your Econometrics course?); this basically means you iteratively take out each of the fixed effects in turn by demeaning the data by that fixed effect. The iterative procedure is described in detail in Gaure (2013), but also appears in Guimaraes and Portugal(2010).
Simen Gaure has developed an R-package called
lfe, which performs the demeaning for you and also provides the possibility to run instrumental variables regressions; it theoretically supports any dimensionality of fixed effects. The key benefit of Simen Gaure's implementation is the flexibility, the use of C in the background for some of the computing and its support for multicore processing, which speeds up the demeaning process dramatically.
Stata there is a package called
reg3hdfe which has been developed by Guimaraes and Portugal (2010). As the name indicates, these support only fixed effects up to two or three dimensions.
Lets see how - on the same dataset - the runtimes of
Comparing Performance of Stata and R
I am estimating the following specification
where are county industry fixed effects and are state-time-industry fixed effects. There are about 3000 counties in the dataset and 22 industries. Furthermore, there are 50 states and the time period is also about 50 quarters. This means - in total - there are 3000 x 22 = 66,000 county-industry fixed effects to be estimated and 22 x 50 x 50 = 55,000 time fixed effects to be estimated. The sample I work with has sufficient degrees of freedom to allow the estimation of such a specification - I work with roughly 3.7 million observations.
I have about 10 covariates that are in , i.e. these are control variables that vary within county x industry over state x industry x time.
Performance in Stata
In order to time the lenght of a stata run, you need to run
set rmsg on, which turns on a timer for each command that is run.
The command I run in stata is
You should go get a coffee, because this run is going to take quite a bit of time. In my case, it took t=1575.31, or just about 26 minutes.
Performance in R
In order to make the runs of
lfe, we need to set the tolerance level of the convergence criterion to be the same in both. The standard tolerance in
Stata is set at , while for
lfe package it is set at . In order to make the runs comparable you can set the options in the R package lfe options explicitly:
The second change we need to make is to disallow lfe to use multiple cores, since reg2hdfe uses only a single thread. We can do this by setting:
Now lets run this in R using:
system.time(summary(felm(log(y) ~ x1 + x2 +x3 +x4 + x5 + x6 + x7 +x8 + x9 + x10 + G(id)+G(sitq),
The procedure converges in a lot quicker than Stata...
It took a mere 4 minutes. Now suppose I run this in four separate threads...
Running this on four threads saves about one minute in processing time; not bad, but not too much gained; the gains from multi-threading increase, the more fixed-effects are added and the larger the samples are.