1.3 on the way to 1.4 Roadmap

[bms63]

Rough notes on what admiral 2.0 or 1.3 might look like

These are just musings for discussion - throw in here so we don't lose track of potential work. @pharmaverse/admiral

Check out some of our musings and ideas for 2024 community team!! Feel free to add or comment @pharmaverse/admiral_comm

[bms63]

Tidyselect Implementation

[bms63]

Is it time that exprs() gets hidden from the user?

[bms63]

%>% to |>

Do we want to move to the base |>?

[manciniedoardo]

As I understand it, it wouldn't be a breaking change. Users who have copied a block that does something like:

data %>% 
  derive_vars_joined(
    dataset = .,
    dataset_add = .,
    ...
)

would experience no change. The new block would instead just be:

data |>
  derive_vars_joined(
    dataset_add = NULL,
    ...
)

and in the backend dataset_add would be set to dataset. Is that correct?

[manciniedoardo]

@rossfarrugia to answer your q, I think users would only need to change anywhere they've used dataset_add = . if they also change to the base pipe.

[rossfarrugia]

Seems like a good option and something I would definitely support for the end of year release

[bundfussr]

The new block would instead just be:

data |>
  derive_vars_joined(
    dataset_add = NULL,
    ...
)

and in the backend dataset_add would be set to dataset. Is that correct?

That's correct.

I wouldn't specify dataset_add in the new block because NULL will be the new default value. Though, it would also work with specifying it.

[manciniedoardo]

@StefanThoma fyi

[manciniedoardo]

Call stores

Call stores as described here

[kaz462]

I wonder if we could connect call store with admiraldiscovery and metacore/metatools to make ADaM programming more metadata-driven?

1. Enhance admiraldiscovry to be a centralized metadata that stores functions for each variable/parameter, similar to the purpose of searchability as @bundfussr mentioned here:

If they want to understand a function, they would look into the function documentation. If they are searching code for a particular variable or parameter, they look into the call stores.

• Focus on variables that can be derived with a single function; these variables can then be automated together using use_call(dataset, variables).
• For variables that involve multiple functions during derivation, it might be clearer for programmers to write them in scripts, with admiraldiscovry serving as a reference.

2. metacore: Enhance the metacore object by adding more information, such as a function column to the metacore$var_spec dataset.
   Users can obtain this information by merging their specification with the 'centralized metadata' from admiraldiscovry.

3. metatools: Add a new function ( use_call) to this package that derives a series variables based on metadata.

Not sure if the above aligns the idea of call store though, what do you all think?

[bundfussr]

I wonder if we could connect call store with admiraldiscovery and metacore/metatools to make ADaM programming more metadata-driven?

1. Enhance admiraldiscovry to be a centralized metadata that stores functions for each variable/parameter, similar to the purpose of searchability as @bundfussr mentioned here:

If they want to understand a function, they would look into the function documentation. If they are searching code for a particular variable or parameter, they look into the call stores.

• Focus on variables that can be derived with a single function; these variables can then be automated together using use_call(dataset, variables).
• For variables that involve multiple functions during derivation, it might be clearer for programmers to write them in scripts, with admiraldiscovry serving as a reference.

Once call stores are implemented admiraldiscovery could provide a call store with the "global" derivations and use it to create the webpage. However, I would expect that the majority of derivations are stored in call stores on company, project, and study level. Users would then generate a page like the admiraldiscovery page which combines the calls from all call stores they want to consider.

2. metacore: Enhance the metacore object by adding more information, such as a function column to the metacore$var_spec dataset.

I agree that for a full automation from specs to ADaM dataset we need an enhanced specification format. I am not sure if metacore can cover this. At the moment it is based on define-XML. But this is intended for reviewer and not for automation. For example the order of the derivations is not included and there is no structure for variables which require more than one derivation like DTYPE.

I think first we would need to define a specs structure which allows automation. This structure should be independent of the programming language. Ideally it should also allow to generate the define-XML and it should cover both study specs and standard specs (in standard specs there can be more than one option for a derivation of a variable/parameter). Thereafter we can extend metacore or create a new package to read the structure and create an R object.

Users can obtain this information by merging their specification with the 'centralized metadata' from admiraldiscovry.
3. metatools: Add a new function ( use_call) to this package that derives a series variables based on metadata.

I think use_call() should be part of the call store package. Once we have the specs structure mentioned above we could add a function to metatools (or in a new package) which generates a script of use_call() calls.

[kaz462]

I have created two issues, feel free to comment and look forward to further discussion on this topic!
atorus-research/metacore#99
pharmaverse/metatools#56

[manciniedoardo]

Behaviour with large datasets

See here, where derive_vars_joined() failed with a large dataset

[bms63]

Discussion in a PR of where we might involve some tests #2599 (comment)

[manciniedoardo]

admiral's behaviour with regards to retaining labels

See here for an example where variable labels are dropped. Should admiral always retain all labels?

[manciniedoardo]

@kaz462 is that an ADaM-compliant derivation? to me it feels a little weird to base a derivation, which affects individual records in the dataset, on the variable labels, which differ across columns not records.

[bundfussr]

Should we clarify it in admiral documentation: labels may not be retained after using admiral functions?

Yes, I would add it to the Input and Output section on the "Get Started" page.

[kaz462]

@kaz462 is that an ADaM-compliant derivation?

it's a Bayer-specific variable derived in ADaM

[bms63]

I;m sort of intrigued by this - when you have time @kaz462 could you make us a reproducible example??

[kaz462]

here is an example of concatenating labels to derive ADAE.AESREAS

[bundfussr]

Improve "See also"

I would propose to change the usage of the @family tag. At the moment it replicates the section of the "Reference" page. Therefore you can't leave the section by the links provided in the "See also" section.

I would use more than one @family tag. For instance, for derive_summary_records() we could use @family derv_obs and @family derv_sum. The first one lists all derivations adding observations and the second one lists all derivations which summarise like derive_summary_records() and derive_var_merged_summary().

[bms63]

The reference page should have small explanations for the advance functions and when you might use the

[bms63]

Tacking on some outstanding items from previous Documentation Discussion - #2085

Most of these have been addressed, but some were not implemented as they needed further discussion.

[bms63]

The vignettes load soooooooooooooooooooooo slooooooooooooooooooooooooooooooowwwwwwwwwwwwww!!!!!!

[bms63]

I want to make this one happen for 1.3.

[bms63]

okay for 1.4!!

[bundfussr]

Make suppress_warning() available for users

Should we move suppress_warning() from admiraldev to admiral and add an option for printing the suppressed warning and add an classes argument?

[bundfussr]

suppress_warning() should be on the user website. At Roche there are a lot of discussions how to handle warnings. suppress_warning() allows to suppress specific warnings while suppressWarnings() suppress all warnings. Thus I would recommend the former.

[bms63]

do you want to bring it over now rather than later?

[bundfussr]

I think it is OK to do it next year. Users could use it even now if they use admiraldev. If we have it in admiral, it would be just easier to find for users.

[manciniedoardo]

Happy to do this move now, regardless of if/when we move admiraldev back into admiral. That way if the latter is delayed, at least suppress_warning() is still moved. What do you think?

[bms63]

haha i forgot we were talking about moving admiraldev back into admiral....i definetely do not want to do that anymore

but

do we want to bring over https://pharmaverse.github.io/admiraldev/dev/reference/suppress_warning.html still?

[zdz2101]

Cyclocomp lintr

We have 3 functions: derive_vars_dt() , derive_vars_dtm() and derive_vars_query() that seem to use a little too much if logic, should re-evaluate how to cleanup the complexity of these functions

[bms63]

Further improving messaging to users

@ddsjoberg solution for calling environments in cards, gtsummary to fix our messaging for users when calling derive but getting compute

[bms63]

Stress testing our functions with empty datasets - #2645

[bms63]

See this for homepage updates: pharmaverse/admiral-adamig-wg#72 (comment)

[bms63]

Progress Bars in our Functions

Is it worth it to allow users to turn on a progress bar for large dataset processing? #2644 (comment)

[bms63]

admipyal

Well...do we do it all again but in python?

[bms63]

Stratification Variables

pharmaverse/admiral-adamig-wg#72 (comment)

[manciniedoardo]

Rough notes and targets from our core team 2024 retro/2025 roadmap

What went well in 2024

• Stability/maturity of core package & involvement of experts made extension package work easier
• Continued conference/workshop work spreading the word about admiral wide. Buy-in from Posit too.
• Release checklist led to smooth 1.3 release
• Increased community contribution through PRs and forks

Challenges in 2024

• Extension package work remains difficult!
• Finding times to meet as a team with time zones and study conflicts (Action: Meetings now moved to Mondays)
• Environment-related issues/bugs (Action: Environments WG to meet again today and trace out work)
• Team size (need a couple more maintenance core team devs) (Action: @manciniedoardo and @bms63 to work on this)

Priorities for 2025

• Continue work of environments WG
• Improving, extending and reformatting our examples in reference pages using {gt} as gold-standard, to make {admiral} more user-friendly and "AI-ready". Perhaps drive this work with a new WG.
• Enhancing CICD template checks to detect changes/bugs
• Release one more extension package
• Start exploring parallel computing/memory optimisation/performance topics
• Grow dev team by a couple of devs

[bms63]

FAQ Vignette

The FAQ is centered around design and why we think admiral is okay to use. Since admiral is accepted now and the industry is moving towards R I was musing on an FAQ being more focused on actual functions and common pitfalls when programming in R.

[bms63]

Revisit/Consolidate CI Checks

• What about just automating the Rd files in the PR if an R file is modified and not requiring the developer to have to regenerate the files and push up.
• Can we consolidate the spelling check into the tests so one less check to look at
• Can we consolidate the styler check somehow into the tests? Or just auto-style the code for the developer in the PR?
• How can we consolidate the R-CMD actions as we have six right now.

Revisit the PR Checklist

• Can we trim down some of the PR checklist
• Can we use the WIKI to list out the steps/advice?
• Just keep a few critical items (e.g. update the news!!) and move the rest to the WIKI

[rossfarrugia]

More roclets expanded function examples

I loved these updates in the latest release, and I'm sure there are some other functions that could benefit from the same.

We also should ask for feedback from any members of the community who are training LLMs on admiral so they can feedback any ideas on how we can further make admiral a true GenAI enabler. Often the biggest challenge to adopt admiral has been the sheer amount of functions and documentation we have, which humans find overwhelming to read through, but human+AI here makes it a game-changer.

[manciniedoardo]

Explore the use of WebR in our documentation

We are finally starting to see efforts to support WebR within {pkgdown} - see here. A recent update on that issue showed a PoC to integrate WebR within the reference pages in a {pkgdown} site. There's also a really helpful README to set up something similar on other {pkgdown} websites.

Looking forward, I think it would be cool to leverage WebR within the {admiral} docs, or elsewhere in the pharmaverse - of course ensuring that we are doing this in order to add value and improve our resources rather than just for the sake of it! Some potential areas are:

• Vignettes: could explore interactive vignettes where users can execute code on the fly for users to try out {admiral} functions directly from the website.
• Reference pages: similar to vignettes, but now using WebR to enhance our already great examples. Or, for the data packages, potentially making the datasets perusable from the website? @Fanny-Gautier @Lina2689.
• Pharmaverse examples website: again, enhancing the examples there.

I followed the instructions in the README linked above to make our own little PoC. Currently it seems to be supported just for reference pages, so I used {pharmaversesdtm} (branch: webr_testing) to just try see if I could get it to work, and it was fairly easy! Feel free to take a look.

@pharmaverse/admiral @pharmaverse/admiral_comm any thoughts on this?

[bms63]

Looks cool!!! But I would push this to Pharmaverse examples site as I would be very hesitant to introduce this as a dependency to admiral

[bundfussr]

@manciniedoardo , do you know what "the package version might differ from this documentation" means.

I think having interactive examples is great but if they behave differently than documented because a different package version is used, it would be very confusing.

[rossfarrugia]

i too feel pharmaverse examples might be a good first place to try this... very cool Edoardo! NEST TLG catalog uses this too as FYI in case anyone wants to see a live example in a similar space: https://insightsengineering.github.io/tlg-catalog/stable/

[manciniedoardo]

Summary of our 1.3 Retro 1.4 Roadmap (powered by Gemini)

Edoardo Mancini started with a reflection on the 1.3 release cycle, highlighting successful documentation updates and a smooth code freeze, while acknowledging developers' perseverance on complex issues. Ben Straub and Fanny Gautier discussed the utility of the PR checklist, and Ross Farrugia noted the value of Admiral's templates and extensions, particularly with Admiral Neuro's development. Key discussions included aligning release schedules, managing reverse dependencies, consolidating CI jobs, improving example documentation, and addressing CRAN feedback on broken links.

Details

1.3 Retrospective

• 1.3 Release Cycle Reflection Edoardo Mancini initiated a reflection on the 1.3 release cycle, focusing on successes, challenges, and areas for improvement. Ben Straub highlighted the successful documentation update, particularly the scaffolding of examples, which they found impactful and have been sharing widely. Edoardo Mancini agreed that the documentation update was enhanced by new team members trying out the improvements, leading to smaller, manageable issues.
• Unexpected Admiral Usage Ben Straub shared their surprise at discovering Merk had trained a custom LLM on Admiral, noting that Admiral is a niche package within an already small community. Ross Farrugia added that Admiral's excellent documentation has led to great results when training internal LLMs, even for complex endpoints, which helps both humans and AI in making predictions.
• Code Freeze Success Edoardo Mancini was pleased that the code base was successfully frozen a week before the 1.3 release, which helped ensure a smoother process compared to previous releases where bugs sometimes necessitated bug fix releases. Ben Straub jokingly suggested their absence during releases might be a contributing factor to the smooth process.
• Developer Perseverance Edoardo Mancini acknowledged the perseverance of developers, citing Fanny Gautier's comment, who tackled issues that initially seemed simple but proved to be much more complex, with some pull requests exceeding 100 comments.
• PR Checklist Utility Fanny Gautier emphasized the value of the pull request (PR) checklist for ensuring all steps are performed correctly, even for experienced programmers, to avoid missing steps. Ben Straub, who created the checklist, appreciated Fanny Gautier's positive feedback, noting that they had faced criticism for its verbosity. Edoardo Mancini suggested creating an issue to review the checklist and potentially streamline it while maintaining its direct accessibility.
• Templates Edoardo Mancini noted that Jeff and their team find value in Admiral's templates, with open issues existing to enhance these open-source templates with additional variables.
• Extension packages Ross Farrugia discussed the challenges of maintaining consistency across package extensions as the "franchise model" grows, but highlighted the significant value of clear documentation and guidance for new companies and those new to disease areas. Ross Farrugia also expressed excitement for Admiralneuro's development, seeing it as a continuation of a success story where multiple co-development partners own package extensions.
• Admiralneuro and Team Involvement Arianna Cascone, a neuroscientist, expressed interest in learning more about Admiral Neuro and potentially getting involved. Edoardo Mancini encouraged their participation, explaining that core team members, including themself, Fanny Gautier, and Lina, are involved in extension packages to ensure alignment with Admiral's standards. Edoardo Mancini also thanked Fanny Gautier and Lina for taking on the data packages, noting their significant positive updates and the high impact of these packages on new users.
• Managing Commitments and Time Stefan Pascal Thoma shared their personal reflection on committing to a single issue for the 1.3 release, finding it helpful for managing their time despite the issue's unexpected complexity. Edoardo Mancini reinforced this by advising everyone to manage their time carefully and set realistic expectations, especially as many contributors are volunteers, to avoid self-imposed pressure or delays.
• Aligning Release Schedules Edoardo Mancini raised the challenge of aligning Admiral's release schedule with extension packages, noting that some extensions release outside the advertised two-to-three-week timeframe. Daniel Sjoberg suggested a looser release schedule for smaller extension packages, allowing them to release based on their needs rather than strictly adhering to Admiral's schedule, provided they update if core Admiral changes cause breaks. Ross Farrugia supported this, suggesting that the reduced need for strict alignment indicates the core Admiral is more stable and allows for independent team management of extensions. Daniel Sjoberg added that external companies managing extensions might have their own release needs, independent of Admiral's schedule. Gordon Miller inquired about how to handle breaking changes in Admiral that affect extension packages if the connection between teams becomes looser. Daniel Sjoberg explained that CRAN checks reverse dependencies and requires notification to affected packages at least two weeks before a release, suggesting that reverse dependency checks should be run well in advance to give extension packages time to adapt.
• CI Job Consolidation Ben Straub noted the growth in the number of CI jobs for pull requests, suggesting some overlap and expressing interest in consolidating them, especially given the upcoming review of Admiral CI. Ross Farrugia mentioned that Dinakar intends to move towards more reusable, less Admiral-specific CI, leveraging available GitHub Actions to reduce maintenance burden and potentially retire some current jobs.
• Guidance for Example Documentation Ben Straub proposed creating a primer for writing good example documentation, noting that even previous "top-notch" examples sometimes lacked depth upon rework. Edoardo Mancini suggested that a well-written primer could potentially be used with an LLM to speed up the process of generating new examples, ensuring they meet expectations for enhanced examples.
• ADaM Working Group Edoardo Mancini acknowledged that while the ADaM working group "fizzled out," significant positive outcomes resulted, including issues around changing how AVALC and AVAL are handled in datasets. Ben Straub added that the group produced many issues and provided a good review, noting that Celine identified numerous mismatches in documentation. Edoardo Mancini suggested restarting these meetings, potentially involving new joiners interested in understanding the ADaM IG, but prioritized other open issues.
• Encouraging Industry Adoption Ross Farrugia discussed the need to proactively communicate Admiral's usefulness to encourage industry adoption, suggesting that sharing real success stories from users would be more impactful than presentations from the core team. Daniel Sjoberg shared that the cards package, which they work on, is experiencing rapid growth, and they have observed similar widespread adoption of Admiral by pharma companies moving into R.
• Acknowledging Contributors Fanny Gautier raised the question of when to acknowledge individuals who have made varied contributions to different packages, noting the difficulty in defining the threshold for including them as contributors versus authors. Ben Straub suggested that engagement, such as reviewing pull requests or helping with complex issues, indicates an active contributor, in contrast to one-off minor fixes. Edoardo Mancini clarified that Admiral defines authors as active maintainers and contributors as those who made significant past contributions, with acknowledgements for minor contributions or bug flagging, and noted that this is at the maintainer's discretion. Daniel Sjoberg suggested using a usethisis::tidy_thanks() function for auto-generating acknowledgements and shared their own rule of thumb: one-off reasonable code contributions lead to being added as a contributor, while active maintenance leads to authorship.
• Vignette Loading Performance Ben Straub brought up the slow loading times of tables in vignettes due to the DT package, suggesting this as an area for improvement in 1.4, even if a solution is not immediately obvious.

1.4 Release Roadmap

• 1.4 Release and Future Challenges Edoardo Mancini outlined plans for the 1.4 release, scheduled for December or January, noting the continued growth and diversity of extension packages and the successful continuation of Monday meetings. They emphasized the goal of committing to maturity and stability, ensuring users can rely on Admiral 1.x versions throughout their study lifecycle, and prioritizing breaking changes only when mandated by users or absolutely necessary.
• Performance and Deprecation Stefan Bundfuss and Ben Straub discussed improving the package's performance, potentially by deprecating older functions for which newer, stable alternatives exist within Admiral, which would also reduce the package's footprint and build time.
• CRAN Feedback and Documentation Links Edoardo Mancini highlighted a critical issue: a notice from CRAN about broken links on help pages, threatening to remove Admiral from CRAN if not fixed by August 25th. Ben Straub noted this is a recurring issue with CRAN, and while it poses challenges due to the multi-version package down setup, Daniel Sjoberg advised keeping the links simple. Stefan Bundfuss suggested using the vignette() function within documentation to automatically link to vignettes on the website.
• Advanced Website Features Edoardo Mancini expressed interest in exploring advanced website features, such as integrating WebR to allow users to run examples live in the browser, acknowledging challenges with package versions and stability. Daniel Sjoberg confirmed WebR's use in other projects and its challenges, including potential crashes and versioning issues, suggesting it might be better suited for a separate site. Ben Straub agreed, recommending the pharmaverse example site for WebR integration due to the Admiral pkgdown site's current slow loading speed and the established interactivity on the example site.
• Ongoing Documentation Enhancements Edoardo Mancini reminded the team about the ongoing master issue for reference page enhancements, encouraging new and existing contributors to improve example pages for complex functions by cataloging their work. Stefan Pascal Thoma also discussed scripting
• Admiral Template Generalization Stefan Bundfuss discussed a draft in Admiralroche to generalize and automate templates, which could help remove functions that are merely calls to more general functions. Edoardo Mancini highlighted the time-intensive and repetitive nature of updating 25 Admiral Rush templates due to changes in the computing environment, indicating a need for generalization. Stefan Bundfuss suggested this would likely be a separate package providing a framework, with calls stored in an Admiral package, an extension package, or a dedicated package for calls.
• Base Pipe Integration Edoardo Mancini raised the possibility of switching entirely to the base pipe from the magrittr pipe, suggesting that updating the dataset_add argument might facilitate this transition. Daniel Sjoberg recommended starting by updating examples to use the more modern approach internally and considered it a lower priority to fully transition due to minimal performance or aesthetic differences. Ross Farrugia noted that users often use the base pipe with other packages, and consistency would be beneficial for Admiral users.
• AI Assistant Integration Edoardo Mancini proposed integrating an AI assistant into the Admiral website, similar to the "Ask AI" feature on the dplyr website, as a potential future avenue. Ben Straub questioned the feasibility of using open-source LLMs given the cost implications of proprietary models and the verbose nature of Admiral's documentation. Ross Farrugia expressed concerns about potential website slowdowns and suggested that such an effort might be better suited as a PhUSE working group initiative, making it feasible for Admiral to link to rather than develop independently. Arianna Cascone noted that the Kapa.ai has a fillable form for open source efforts which may apply to Admiral. Ross Farrugia suggested to contact Phil at Posit to enquire how they did it for dplyr.
• Challenges for Admiral 1.4 Release Edoardo Mancini discussed the main challenges for the 1.4 release, emphasizing the difficulty in maintaining momentum for Admiral, as 90-95% (roughly!) of the package is already developed. They also mentioned the challenge of predicting the release size, which impacts resource allocation and team engagement. However, Edoardo Mancini expressed confidence in the current team's ability to deliver on the 1.4 release.
• IG Group and Working Groups Ben Straub announced they are seeking a new lead for the IG group and encouraged interested individuals to reach out. Ben Straub also mentioned the restart of a working group focused on learning about environments in Admiral and R, inviting new participants to join for informative discussions.

[nbrucken17]

Thanks! It's been added to the agenda for the Aug 27 ADaM full-team meeting. Would any of you like to attend to give a short presentation/background on admiral? I can try, but you're far more familiar with it than I am. Or is there a standard slide deck out there somewhere that I could use?

[bms63]

@nbrucken17 I'm happy to come out and talk about admiral. What time is the meeting?

@rossfarrugia, @manciniedoardo, @bundfussr, @adcascone would any of you like to come along?

[manciniedoardo]

I will be on leave but happy (and keen!) to give someone else a chance to rep {admiral}!

[adcascone]

@bms63 @nbrucken17 I'm happy to be there, too, if the timing works out! Just let me know the attendee information - thanks all

[rossfarrugia]

@bms63 please continue without me - i have quite a few holidays this month so will be struggling for time.

[nbrucken17]

The Aug 27 ADaM full-team meeting runs from 9-10 AM ET. @bms63, @adcascone, @bundfussr, @rossfarrugia, @cpiraux - please send me an email if you'd like to attend (nancy.brucken@iqvia.com- I know, how old-fashioned...), and I'll forward the meeting invitation to you. I'm reluctant to post the Teams link on GitHub since it's under CDISC's Teams account.