Working with Big Data in R

This year has been crazy in terms of data I use for my analysis. Frequently the traditional methods I use in R would fail to allocate enough memory for the task at hand. Luckily, R has great support for such tasks. I will note down a few package names that have served me well lately for future reference.

ffbase – Working with flatfiles, without storing them in memory
biglm – Doing analysis on such ff
foreach – to parallelize loops
parallel – ^
doMC – ^ alternative to parallel
glmmML – simplified random intercepts binary models. Faster than lme4 but still not fast enough for my purposes…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: