This year has been crazy in terms of data I use for my analysis. Frequently the traditional methods I use in R would fail to allocate enough memory for the task at hand. Luckily, R has great support for such tasks. I will note down a few package names that have served me well lately for future reference.

ffbase – Working with flatfiles, without storing them in memory
biglm – Doing analysis on such ff
foreach – to parallelize loops
parallel – ^
doMC – ^ alternative to parallel
glmmML – simplified random intercepts binary models. Faster than lme4 but still not fast enough for my purposes…

Advertisements