General Issue: Verify template results

[StefanThoma]

Background Information

It would be ideal to have a way to verify whether the outputs of the templates have changed.

One idea would be to write unit tests, so that running the unit tests would already check that.
I think it could be done like this in a unit test:

1. Create new file using template
2. (Potentially) modify the new file to reduce the dataset size used
3. Source the file
4. Compare the newly created file to the one created last time using diffdf

Definition of Done

Either find a way of verifying the outputs of the templates, or to decide that this is not necessary.

[bms63]

I would be hesitant to make these into unit tests as they are very big files to run.

The current CI check for the templates is a separate workflow. I was thinking we would build off that one - pharmaverse/admiralci/.github/workflows/check-templates.yml@main

[StefanThoma]

I see your point.
The downside with a separate run is that you cannot run it locally, afaik.

[bms63]

What about how Edoardo approached that new check in the blog? He sources a file in the workflow - maybe something to consider? Then you could grab the file and run it

[StefanThoma]

Yeah that would not be a bad idea. Let me experiment a bit.

[bundfussr]

The downside with a separate run is that you cannot run it locally, afaik.

We could implement a function which runs a specified list of templates and compares the results with the references datasets (in pharmaverseadam?).

This function could then be used in the CI/CD workflow and also be called locally.

[StefanThoma]

The downside with a separate run is that you cannot run it locally, afaik.

We could implement a function which runs a specified list of templates and compares the results with the references datasets (in pharmaverseadam?).

This function could then be used in the CI/CD workflow and also be called locally.

That sounds like a good idea.

[bms63]

The downside with a separate run is that you cannot run it locally, afaik.

We could implement a function which runs a specified list of templates and compares the results with the references datasets (in pharmaverseadam?).
This function could then be used in the CI/CD workflow and also be called locally.

That sounds like a good idea.

So - pharmaverseadam got re-deployed last week so things might not line up that well with this method going forward. Feels like it is getting decoupled form admiral.

I think we should compare templates runs via stuff in main and stuff in a branch. Could this action just run if code in the templates is updated in the branch? I feel like that is possible ????

[StefanThoma]

@bms63
Teal gives us keys for the following datasets:

##  [1] "ADAE"     "ADAETTE"  "ADCM"     "ADCSSRS"  "ADDV"     "ADEG"    
##  [7] "ADEQ5D5L" "ADEX"     "ADHY"     "ADLB"     "ADMH"     "ADQLQC"  
## [13] "ADQS"     "ADRS"     "ADSAFTTE" "ADSL"     "ADSUB"    "ADTTE"   
## [19] "ADVS"

Which gives us the following keys:

## A join_keys object containing foreign keys between 19 datasets:
## ADSL: [STUDYID, USUBJID]
##   <-- ADAE: [STUDYID, USUBJID]
##   <-- ADEG: [STUDYID, USUBJID]
##   <-- ADTTE: [STUDYID, USUBJID]
##   <-- ADAETTE: [STUDYID, USUBJID]
##   <-- ADCM: [STUDYID, USUBJID]
##   <-- ADEX: [STUDYID, USUBJID]
##   <-- ADLB: [STUDYID, USUBJID]
##   <-- ADMH: [STUDYID, USUBJID]
##   <-- ADQS: [STUDYID, USUBJID]
##   <-- ADRS: [STUDYID, USUBJID]
##   <-- ADSAFTTE: [STUDYID, USUBJID]
##   <-- ADVS: [STUDYID, USUBJID]
##   <-- ADDV: [STUDYID, USUBJID]
##   <-- ADSUB: [STUDYID, USUBJID]
##   <-- ADHY: [STUDYID, USUBJID]
##   <-- ADQLQC: [STUDYID, USUBJID]
##   <-- ADCSSRS: [STUDYID, USUBJID]
##   <-- ADEQ5D5L: [STUDYID, USUBJID]

## ADAE: [STUDYID, USUBJID, ASTDTM, AETERM, AESEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADEG: [STUDYID, USUBJID, PARAMCD, AVISIT]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADTTE: [STUDYID, USUBJID, PARAMCD]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADAETTE: [STUDYID, USUBJID, PARAMCD]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADCM: [STUDYID, USUBJID, ASTDTM, CMSEQ, ATC1CD, ATC2CD, ATC3CD, ATC4CD]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADEX: [STUDYID, USUBJID, PARCAT1, PARAMCD, AVISITN, ASTDTM, EXSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADLB: [STUDYID, USUBJID, PARAMCD, AVISIT]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADMH: [STUDYID, USUBJID, ASTDTM, MHSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADQS: [STUDYID, USUBJID, PARAMCD, AVISIT]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADRS: [STUDYID, USUBJID, PARAMCD, AVISIT]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADSAFTTE: [STUDYID, USUBJID, PARAMCD]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADVS: [STUDYID, USUBJID, PARAMCD, AVISIT]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADDV: [STUDYID, USUBJID, ASTDT, DVTERM, DVSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADSUB, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADSUB: [STUDYID, USUBJID, PARAMCD, AVISITN, ADTM, SRCSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADHY, ADQLQC, ADCSSRS, ADEQ5D5L

## ADHY: [STUDYID, USUBJID, PARAMCD, AVISITN, ADTM, SRCSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADQLQC, ADCSSRS, ADEQ5D5L

## ADQLQC: [STUDYID, USUBJID, PARCAT1N, PARAMCD, BASETYPE, AVISITN, ATPTN, ADTM, QSSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADCSSRS, ADEQ5D5L

## ADCSSRS: [STUDYID, USUBJID, PARAMCD, BASETYPE, AVISITN, DTYPE, ADTM]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADEQ5D5L

## ADEQ5D5L: [STUDYID, USUBJID, PARCAT1N, PARAMCD, BASETYPE, AVISITN, ATPTN, ADTM, QSSEQ]
##   --> ADSL: [STUDYID, USUBJID]
##   --* (implicit via parent with): ADAE, ADEG, ADTTE, ADAETTE, ADCM, ADEX, ADLB, ADMH, ADQS, ADRS, ADSAFTTE, ADVS, ADDV, ADSUB, ADHY, ADQLQC, ADCSSRS

[bms63]

Thanks! I’ll start making a list of our datasets and potential keys

[StefanThoma]

@bms63 just FYI @jimrothstein has approached me to take this over, with some more input from my side.
Did you manage to make any progress on this yet?

[bms63]

No progress from my end, but eager to set this up as it causes a lot of consternation. Actually looking to do something similar in the Pliot R Submissions as well

[bms63]

Converting this to a discussion and will link in the Roadmap

[StefanThoma]

Trying to collect some thoughts here, please feel free to chime in:

I see two ways of going about:

1. Comparing the template results with an already created dataset.

   • Where would we store the original dataset? We could simply use the one from pharmaverseadam package, they should be correct, right?
   • Doing so would work as long as we don't want to change the result of the template, changing outcomes would not be reflected in the already created datasets in pharmaverseadam until that package is updated.

2. Comparing the template results with the result from the main version of the template.

   • Check will take longer as all scripts have to run twice essentially.

Strategies to speed up:

1. Run local files first, as could throw an error early.
2. Compare against main branch to only check changed templates? (This should be an option at least)

How to compare

1. Use the diffdf package. Specify datasets and keys as provided above.

[StefanThoma]

@jimrothstein FYI

[jimrothstein]

@StefanThoma Many thanks. Let me review the discussion from beginning.

[bundfussr]

I have attached the admiralroche script which runs templates and compares them with reference datasets.
create_synthetic_adams.txt
(It is an R script but I needed to change the extension to .txt to be able to attach it.)
I think we can use a simplified version in admiral.

• We don't need to read the config file or create directories.
• We don't need to write parquet files for the reference datasets.
• We don't need to use snakemake. We can just source the templates.
• For the comparison we should specify keys to make the output more readable. The keys could be defined in the script or in a separate JSON file.

[jimrothstein]

@StefanThoma @bms63 @manciniedoardo

My notes

jim
2025-07-06

*“Drop a goat exactly equidistant between 2 identical stacks of hay. The
goat has no problem selecting one. A human in similar situation would
starve” * -unknown

admiral_verify_templates.R

Today’s Objectives

Understand status of ongoing work
Clarify what is yet to be done
Identify and MVP to merge in and test out with the full team.
Decide whether to do this in one of the original PRs or start fresh.

Useful Code Links

older PR: #2662
newer PR: #2773

latest GHA workflow:
https://github.com/pharmaverse/admiral/blob/feedback_template_verification/.github/workflows/verify_templates_experimental.yml

create_adams_data.R (re-generate and save ADaMs by calling each
template)
https://github.com/pharmaverse/pharmaverseadam/blob/main/data-raw/create_adams_data.R

Status - overall

The script does run and produce output, as does the GHA workflow. But it
remains rough with a number of lingering issues.

Stefan has been extremely helpful but his availability is limited.

I list several types of issues below. But the real problem, IMHO, is I
have trouble with fork in the road (Choose to code this way? or that?
Select default behavior? Re-write this function or live with it?)
A shortcoming , yes, but it slows me down. In other words, possible to
have brief, more frequent discussions re: decisions.

Also, helpful:

Itemize and prioritize todo/specification list, before coding?
Standardize (example: use cli_*, less paste0 ?)

Background

Because of overlap, the R file admiral_verify_templates.R inherited some
code from create_adams_data.R (in pharmaverseadam)

However, I had hoped to “improve” the pharamaverseadam code by
documenting and/or simplifying some of the code, especially where there
is overlap.

This is still a to do.

Script Usage?

Pls clarify - Triggers to run GHA? (branch, PR) - Will developer modify
template and then run interactively?

intermediate files

NOW: Intermediate files are now stored in several temporary tempdir()
R directories and therefore disappear when code ends.
This makes debugging a little more difficult. Also USER running this
interactively clobbers results of previous run.

Proposed: Store intermediate results in permanent directories. Cleanup
after run. Allow user to aluse the cleanup.

Related - Use tibble, not lists

Currently, using several names lists (for each ADaM, list for templates,
another for dir, …) Confusing.

Example:

#' Directories used to find files:

#'  cache_dir
#'  diff_dir
#'  tempdir()
#'  template_dir  location of template R files (inst/templates)
#'  adam_new_dir (aka tempdir() or cache_dir ):  after running, templates place new ADaMs here
#'  adam_old_dir : ADaMs downloaded from pharamverseadam

#' Named vectors/lists, use adam_name to name
#'
#'  adam_names (named character vector)
#'  path  (named list)
#'  keys  (named list)
#'  obj   (named list)
#'  templates (named list)

REF:

admiral/data-raw/admiral_verify_templates.R

Lines 16 to 23 in 55c003d

	#' Directories used to find files:
	#' cache_dir
	#' diff_dir
	#' tempdir()
	#' template_dir location of template R files (inst/templates)
	#' adam_new_dir (aka tempdir() or cache_dir ): after running, templates place new ADaMs here
	#' adam_old_dir : ADaMs downloaded from pharamverseadam

Propose Use tibble as lookup. One row for each ADaM, columns for
location of each template, directories etc. Populate as exact strings,
constructed ONCE?

USE RDS, not RDA?

`load(dataset) puts symbol in environment. Need a few lines of code to
get actual databbase.

non-intuitive example (to me):
https://github.com/pharmaverse/pharmaverseadam/blob/db39a719c3b1ddc10d3529bb76c72afbc402a50e/data-raw/create_adams_data.R#L39-L42

better?

save(mtcars, file="mt.rda")
f  <- function() { get(load("mt.rda"))}
f() |> head(n=3)

.rda files are used in templates and put into user’s cache dir (os
dependent)

admiral/inst/templates/ad_adae.R

Lines 149 to 157 in 9052fc3

	# Save output ----
	# Change to whichever directory you want to save the dataset in
	dir <- tools::R_user_dir("admiral_templates_data", which = "cache")
	if (!file.exists(dir)) {
	# Create the folder
	dir.create(dir, recursive = TRUE, showWarnings = FALSE)
	}
	save(adae, file = file.path(dir, "adae.rda"), compress = "bzip2")

Propose: Can we use RDS?

GHA workflows

• When should GHA workflow should trigger? Branch, event?

• Speed things up? Use of docker image for R? (I use act when I can)

• all .rda files, only changed rda?
  Suggestions for the template verification functions #2773 (comment)

To pin down exact rda file changes, I am working through things like:

git diff HEAD~ HEAD --name-only

# and
git diff --exit-code ...
echo $?    # 0 = no difference   1 = difference


echo "--- Files changed (names only) ---"
git diff --name-only ${{ github.event.before }} ${{ github.sha }}

Unit tests

Needs unit tests. And more feedback from USER.

diffdf

• diffdf uses key fields, if provided, or guesses. Uses quess.
• Hard to insert emoji; Eli’s work embeds diffdf in quarto file

Use of key fields; removed

Minor Issues, code checks?

- <https://github.com/pharmaverse/admiral/pull/2773#discussion_r2159570703>