-
Notifications
You must be signed in to change notification settings - Fork 0
Github action for atime
atime is an R package that provides functions for asymptotic complexity testing, which means running several pieces of R code for various input data sizes, and recording the time and memory requirements for each piece/size. Different pieces can be code in different packages, or different versions of the same code from one package.
-
atime_versions()can run the same piece of code on different versions of an R package in a github repo. -
atime_pkg()runs user code in pkg/inst/atime/tests.R on different versions of the R package (CRAN, branch, main/master).
Related work in R is limited to non-asymptotic testing
- Rperform supports some similar features, but more focus on metrics for a single data size (not asymptotic/variable data size).
- microbenchmark/bench packages are useful for measuring and comparing time/memory usage of R expressions (and bench is used by atime for that purpose).
The goal of this project would be to write a new GitHub action that can create/update a PR comment showing if there are any changes in performance (time/memory) between a PR and the main/release versions.
The idea is that the package author writes some code like this in their pkg/inst/atime/tests.R file,
- https://github.com/tdhock/aum/blob/73235b9dcf447361389f51d03bf1af7bbde06ca2/inst/atime/tests.R
- https://github.com/tdhock/binsegRcpp/blob/5e5e1349304238b6d20b55dff236989caafec239/inst/atime/tests.R
Then after every push to a github PR, the github action will run the
atime_pkg() function, which computes the benchmark and makes the
plots to inspect for any changes in performance. New code needs to be
written to create/update a PR comment based on these results.
- Creating example PRs where changes are detected.
- Writing blog/tutorial to explain how to setup the new GH action on a new repo.
This project will make atime much more easy to use, and therefore make it much easier for R package developers to see if their PRs introduce performance regressions/improvements.
Contributors, please contact mentors below after completing at least one of the tests below.
- EVALUATING MENTOR: Toby Hocking toby.hocking@r-project.org is the author of numerous R packages, and was the PI on the R consortium RcppDeepState grant.
- Fabrizio Sandri sandri.fabr@gmail.com is an expert in GitHub actions, and has previous experience developing RcppDeepState-action in R-GSOC'22.
Contributors, please do one or more of the following tests before contacting the mentors above.
- Easy: setup a GitHub repo with an R package, and use r-lib/actions to automatically check the package after every commit. Modify your repo's github action so it does something different / non-standard. (maybe print out something that is not usually printed?)
- Medium: create a new github repo for your Easy action, maybe call it custom-check, and use that new github action on your R package github repo.
- Hard: pick an R package which is on github, and use
atime_versions()to make a plot of the performance of several versions of the code.
Contributors, please post a link to your test results here.
- EXAMPLE CONTRIBUTOR 1 NAME, LINK TO GITHUB PROFILE, LINK TO TEST RESULTS.