From Sheeba Samuel and @EvoMRI
"Computational reproducibility of Jupyter notebooks from biomedical publications"
https://arxiv.org/abs/2308.07333
Found 27271 notebooks in 2660 GitHub repositories associated with 3467 articles
22578 notebooks were written in Python, including 15817 that had their dependencies declared in requirement files
For 10388, all declared dependencies could be installed successfully
1203 notebooks ran through without any errors
879 produced the original results
Interesting.
Didn't read the entire paper but checked "Conclusions":
"The main issues are related to dependencies – both code and
data – which means that reproducibility could likely be improved
considerably if the code – and dependencies in particular – were bet-
ter documented"
Those are not specific to Jupyter notebooks. Could apply pretty much the same way to a repo with plain code.
Should people submit a VM image 😬?
> submit a VM image
...yeah, often that is the only way to make data sourcing like `read("c:\users\sammy\documents\analysis_final\tmp\data_corrected_finalfinal2.xls")` work reproducibly. [sarcastic emoji here]
Anyway, I think that actual tests in CIs that run in some common and easily obtainable environment (such as docker in github CI) would help most.
@exa @carlos @danielskatz I saw this Dokta project "for researchers to create Docker images for their research projects" mentioned at some point in the past. I haven't used it, so I can't vouch for it, but it seems relevant.
https://github.com/stencila/dockta
