Reproducible Research Data and Project Management in R

Background

Let’s face it…

There are going to be files
LOTS of files
The files will change over time
The files will have relationships to each other

It’ll probably get complicated

Strategy against chaos

File organization and naming is a mighty weapon against chaos

Make a file’s name and location VERY INFORMATIVE about:
- what it is,
- why it exists,
- how it relates to other things
The more things are self-explanatory, the better.

File naming

Names matter

What works, what doesn’t?

NO

myabstract.docx
Joe’s Filenames Use Spaces and Punctuation.xlsx
figure 1.png
fig 2.png
JW7d^(2sl\@deletethisandyourcareerisoverWx2*.txt

YES

2014-06-08_abstract-for-sla.docx
joes-filenames-are-getting-better.xlsx
fig01_scatterplot-talk-length-vs-interest.png
fig02_histogram-talk-attendance.png
1986-01-28_raw-data-from-challenger-o-rings.txt

Three principles for good (file) names

Machine readable
Human readable
Play well with default ordering

Machine readable

Regular expression and globbing friendly
- Avoid spaces, punctuation, accented characters, case sensitivity
Easy to compute on
- Deliberate use of delimiters

Filtering and search through Globbing

Excerpt of complete file listing:

Example of globbing to filter file listing:

Search using Mac OS Finder search facilities

Search using regex in R

Delimit information with punctuation

Deliberate use of "-" and "_" allows recovery of metadata from the filenames:

"_" underscore used to delimit units of metadata I want to access later
"-" hyphen used to delimit words so our eyes don’t bleed

Splitting filenames by delimiters

This happens to be R but also possible in the shell, Python, etc.

Include important metadata

e.g. I’m saving a number of files of temperature data extracted at different resolutions (res) and for a number of months (month). Including these parameters in the filename allows me to use them to target files to read in.

write.csv(df, paste("variable", res, month, sep ="_"))

df <- read.csv(paste("variable", res, month, sep ="_"))

Recap: machine readable

Easy to search for files later
Easy to filter file lists based on names
Easy to extract info from file names, e.g. by splitting

New to regular expressions and globbing? be kind to yourself and avoid + Spaces in file names + Punctuation + Accented characters

Human readable

Name contains info on content
Connects to concept of a slug from semantic URLs

Example

Which set of file(name)s do you want at 3 a.m. before a deadline?

Embrace the slug

Use slugs to link inputs, scripts & outputs

The `R` scripts:

01_marshal-data.r
02_pre-dea-filtering.r
03_dea-with-limma-voom.r
04_explore-dea-results.r
90_limma-model-term-name-fiasco.r

The figures left behind:

02_pre-dea-filtering-preDE-filtering.png
03-dea-with-limma-voom-voom-plot.png
04_explore-dea-results-focus-term-adjusted-p-values1.png
04_explore-dea-results-focus-term-adjusted-p-values2.png
...
90_limma-model-term-name-fiasco-first-voom.png
90_limma-model-term-name-fiasco-second-voom.png

Recap: Human readable

Easy to figure out what the heck something is, based on its name

Play well with default ordering

Put something numeric first
Use the ISO 8601 standard for dates
Left pad other numbers with zeros

Examples

Chronological order:

Use the ISO 8601 standard for dates: YYYY-MM-DD

Logical order: Put something numeric first

Left pad other numbers with zeros

If you don’t left pad, you get this:

10_final-figs-for-publication.R
1_data-cleaning.R
2_fit-model.R

which is just sad :(

Recap: Play well with default ordering

Put something numeric first
Use the ISO 8601 standard for dates
Left pad other numbers with zeros

Recap: Three principles for (file) names

Machine readable
Human readable
Play well with default ordering

Good File Naming

Reproducible Research Data and Project Management in R

https://acce-rrresearch.netlify.app/