Fix babs submit select argument parsing #312

singlesp · 2025-09-15T17:46:21Z

Flatten --select arguments in parse_select_arg to fix AttributeError when using babs submit --select.

The argparse library, when configured with action='append' and nargs='+', can produce a list of lists for arguments like --select. The original parse_select_arg function expected a flat list of strings, leading to an AttributeError when it encountered a list instead of a string while calling .startswith(). This change ensures the input is flattened into a single list of strings before further processing.

…ning; add tests for list-of-lists inputs

singlesp · 2025-09-15T17:53:05Z

@cursoragent I ran into a new error when trying to babs submit with the --select flag.

(babs) [grmpy@cubic-sattertt1 mriqc]$ babs submit --select sub-88773
Traceback (most recent call last):
  File "/gpfs/fs001/cbica/projects/grmpy/micromamba/envs/babs/bin/babs", line 8, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "/gpfs/fs001/cbica/projects/grmpy/vendored_packages/babs/babs/cli.py", line 718, in _main
    options.func(**args)
  File "/gpfs/fs001/cbica/projects/grmpy/vendored_packages/babs/babs/cli.py", line 390, in babs_submit_main
    babs_proj.babs_submit(count=count, submit_df=df_job_specified)
  File "/gpfs/fs001/cbica/projects/grmpy/vendored_packages/babs/babs/interaction.py", line 68, in babs_submit
    df_needs_submit[submit_cols].to_csv(self.job_submit_path_abs, index=False)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/gpfs/fs001/cbica/projects/grmpy/micromamba/envs/babs/lib/python3.11/site-packages/pandas/core/frame.py", line 4108, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs001/cbica/projects/grmpy/micromamba/envs/babs/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/gpfs/fs001/cbica/projects/grmpy/micromamba/envs/babs/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 6252, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: "['job_id'] not in index"

cursor · 2025-09-15T17:53:11Z

I refactored the job submission logic in babs/interaction.py to improve how job IDs are handled when writing the submission manifest. This ensures the manifest is initially written without an empty job_id column and then correctly updated after job submission.

In babs/interaction.py, I introduced a new variable pre_submit_cols within the BABSInteraction.submit_jobs method. This variable defines the columns to write to self.job_submit_path_abs before the job_id is known, explicitly excluding job_id.
I modified the initial CSV write operation to use pre_submit_cols (df_needs_submit[pre_submit_cols].to_csv(...)), preventing an empty job_id column from being written prematurely.
I redefined submit_cols after the job_id has been assigned to df_needs_submit, ensuring the final CSV write (df_needs_submit[submit_cols].to_csv(...)) includes the correct job_id for all tasks.

3 tasks completed

Adjust pre-submit CSV in babs_submit to exclude job_id
Run linter checks on edited file
Commit interaction.py fix with message

_{Learn more about Cursor Agents}

codecov-commenter · 2025-09-15T17:54:06Z

Codecov Report

❌ Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.75%. Comparing base (eca2565) to head (3dd3857).

Files with missing lines	Patch %	Lines
babs/utils.py	81.81%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #312      +/-   ##
==========================================
- Coverage   80.78%   80.75%   -0.04%     
==========================================
  Files          16       16              
  Lines        1509     1517       +8     
  Branches      234      237       +3     
==========================================
+ Hits         1219     1225       +6     
- Misses        182      183       +1     
- Partials      108      109       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…r submission

singlesp · 2025-09-15T18:09:12Z

@mattcieslak - the select flag seems to work now - but obviously the usage isn't what I thought it was. I guess it can only be used one and not meant to be used iteratively?

(babs) [grmpy@cubic-sattertt1 mriqc]$ babs submit --select sub-88773
Submitting the following jobs:
      sub_id  task_id    job_id
0  sub-88773        1  11113276
(babs) [grmpy@cubic-sattertt1 mriqc]$ babs submit --select sub-120562
Traceback (most recent call last):
  File "/gpfs/fs001/cbica/projects/grmpy/micromamba/envs/babs/bin/babs", line 8, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "/gpfs/fs001/cbica/projects/grmpy/vendored_packages/babs/babs/cli.py", line 718, in _main
    options.func(**args)
  File "/gpfs/fs001/cbica/projects/grmpy/vendored_packages/babs/babs/cli.py", line 390, in babs_submit_main
    babs_proj.babs_submit(count=count, submit_df=df_job_specified)
  File "/gpfs/fs001/cbica/projects/grmpy/vendored_packages/babs/babs/interaction.py", line 36, in babs_submit
    raise Exception(
Exception: There are still jobs running. Please wait for them to finish or cancel them. Current running jobs:
     job_id  task_id state time_used time_limit  nodes  cpus partition name     sub_id
0  11113276        1     R      0:35    6:00:00      1     4       all  mri  sub-88773

singlesp · 2025-09-15T18:10:37Z

@cursoragent - we use Ruff listing with a line limit of 99. Apply this to your changes. Also update the babs submit docs section for the named arguments.

cursor · 2025-09-15T18:10:43Z

I refactored the job selection mechanism for babs submit, renaming the --job argument to --select and enhancing its help text for improved clarity. I also updated the documentation to provide comprehensive usage examples for selecting specific subjects and sessions.

I renamed the babs submit command-line argument from --job to --select within babs/cli.py, improving clarity and consistency for selecting specific jobs.
I enhanced the help message for the --select argument in babs/cli.py to clearly explain its usage for both single and multi-session datasets, including examples of repeating the flag or providing multiple values.
I updated the docs/babs-submit.rst documentation to reflect the new --select flag, providing detailed examples for selecting jobs by subject and session, and added a note about job submission behavior.
I applied minor code formatting improvements in babs/interaction.py and babs/utils.py for better readability without altering functionality.

_{Learn more about Cursor Agents}

…xamples, and running-jobs behavior

singlesp · 2025-09-15T18:19:43Z

@cursoragent line 444 in tests/test_utils.py is too long

cursor · 2025-09-15T18:19:49Z

I refactored the test assertions within test_parse_select_arg in tests/test_utils.py to improve readability. This change enhances the clarity of how parse_select_arg handles nested list inputs by making the expected pd.DataFrame structures more explicit and easier to parse visually.

I modified the test_parse_select_arg() function in tests/test_utils.py.
I reformatted the pd.DataFrame constructor calls within the assert ... .equals(...) statements. This involved breaking the dictionary arguments onto multiple lines to improve the readability of the expected DataFrame structures, particularly for multi-column cases like sub_id and ses_id.

_{Learn more about Cursor Agents}

singlesp · 2025-09-15T18:21:27Z

@cursoragent

W293 [*] Blank line contains whitespace
   --> babs/utils.py:725:1
    |
724 |     """
725 |     
    | ^^^^
726 |     # argparse with action='append' and nargs='+' produces a list of lists.
727 |     # Flatten here so downstream logic can assume a flat list.
    |
help: Remove whitespace from blank line

Found 1 error.
[*] 1 fixable with the `--fix` option.

cursor · 2025-09-15T18:21:32Z

Removed the trailing whitespace on the blank line in babs/utils.py around the parse_select_arg docblock.

_{Learn more about Cursor Agents}

…_select_arg

singlesp · 2025-09-17T15:02:40Z

closes #311 and #310

fix(submit): handle nested --select arguments from argparse by flatte…

62e1cb4

…ning; add tests for list-of-lists inputs

fix(submit): write pre-submit CSV without job_id; include job_id afte…

02d3106

…r submission

style(ruff): wrap lines <=99; docs(submit): clarify --select usage, e…

e2a5ea5

…xamples, and running-jobs behavior

test: wrap long asserts under 99 chars in tests/test_utils.py

182dbf0

style(ruff): fix W293 by removing whitespace-only blank line in parse…

3dd3857

…_select_arg

singlesp requested a review from mattcieslak September 15, 2025 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix babs submit select argument parsing #312

Fix babs submit select argument parsing #312

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 15, 2025 •

edited

Loading

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 •

edited

Loading

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 •

edited

Loading

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 •

edited by singlesp

Loading

Uh oh!

singlesp commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fix babs submit select argument parsing #312

Are you sure you want to change the base?

Fix babs submit select argument parsing #312

Uh oh!

Conversation

singlesp commented Sep 15, 2025

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

singlesp commented Sep 15, 2025

Uh oh!

cursor bot commented Sep 15, 2025 • edited by singlesp Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

singlesp commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cursor bot commented Sep 15, 2025 •

edited

Loading

codecov-commenter commented Sep 15, 2025 •

edited

Loading

cursor bot commented Sep 15, 2025 •

edited

Loading

cursor bot commented Sep 15, 2025 •

edited

Loading

cursor bot commented Sep 15, 2025 •

edited by singlesp

Loading