Run error in rpt.py

[sdrewc]

@weiwang108

Traceback (most recent call last):
File "Y:\sdc\simwrapper_topsheet\scripts\rpt.py", line 412, in
df = df.astype(int)
^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\generic.py", line 6534, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\internals\managers.py", line 414, in astype
return self.apply(
^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\internals\managers.py", line 354, in apply
applied = getattr(b, f)(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\internals\blocks.py", line 616, in astype
new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\dtypes\astype.py", line 238, in astype_array_safe
new_values = astype_array(values, dtype, copy=copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\dtypes\astype.py", line 183, in astype_array
values = _astype_nansafe(values, dtype, copy=copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\dtypes\astype.py", line 101, in _astype_nansafe
return _astype_float_to_int_nansafe(arr, dtype, copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda3\envs\topsheet_data\Lib\site-packages\pandas\core\dtypes\astype.py", line 146, in _astype_float_to_int_nansafe
raise IntCastingNaNError(
pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

ERROR conda.cli.main_run:execute(125): conda run python rpt.py failed. (See above for error)

[sdrewc]

@weiwang108 I'm not sure if this is all a consequence of the error above.

I ran a test writing data to: X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet\data
And then copied the dashboard.yaml files to X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet. Many of the visualizations are not loading. It looks like the maps are expecting a "geo" subfolder. Does this currently need to be manually copied in? Some of the vega lite visualizations are also broken

[weiwang108]

Hi Drew, I apologize for not providing clear instructions earlier. The topsheet YAML file is configured to read shape files from the "geo" folder, which should be in the same directory as the YAML files and the "data" folder. For optimal organization, it's recommended to set the output directory of the topsheet run to this same "data" folder, where the Vega configuration files are already located. [image: image.png] *Wei Wang*TD&A Intern Office: 415-593-1664 ***@***.*** sfcta.org <https://www.sfcta.org/> | sign up for our newsletter <https://www.sfcta.org/stay-connected>

…

On Tue, May 21, 2024 at 2:28 PM Drew C ***@***.***> wrote: @weiwang108 <https://github.com/weiwang108> I'm not sure if this is all a consequence of the error above. I ran a test writing data to: X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet\data And then copied the dashboard.yaml files to X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet. Many of the visualizations are not loading. It looks like the maps are expecting a "geo" subfolder. Does this currently need to be manually copied in? Some of the vega lite visualizations are also broken — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDBK5I4UPOQZFG63U3W5MOLZDO4AZAVCNFSM6AAAAABICMU4R6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGQ3TCNZXHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[weiwang108]

Hi Drew, To set up a dashboard in the directory: X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet, please follow these steps: Download the entire GitHub repository from simwrapper_topsheet into this folder. In the control file (ctl), specify the input folder as X:\Projects\CHAMP7\Run25w_Market and set the output folder to the "data" folder located within the GitHub repository. Execute the batch file provided. This will generate all CSV and Markdown files into the "data" folder. Subsequently, the YAML files will access shape files from the "geo" folder and datasets from the "data" folder. *Wei Wang*TD&A Intern Office: 415-593-1664 ***@***.*** sfcta.org <https://www.sfcta.org/> | sign up for our newsletter <https://www.sfcta.org/stay-connected>

…

On Tue, May 21, 2024 at 2:36 PM Wei Wang ***@***.***> wrote: Hi Drew, I apologize for not providing clear instructions earlier. The topsheet YAML file is configured to read shape files from the "geo" folder, which should be in the same directory as the YAML files and the "data" folder. For optimal organization, it's recommended to set the output directory of the topsheet run to this same "data" folder, where the Vega configuration files are already located. [image: image.png] *Wei Wang*TD&A Intern Office: 415-593-1664 ***@***.*** sfcta.org <https://www.sfcta.org/> | sign up for our newsletter <https://www.sfcta.org/stay-connected> On Tue, May 21, 2024 at 2:28 PM Drew C ***@***.***> wrote: > @weiwang108 <https://github.com/weiwang108> I'm not sure if this is all > a consequence of the error above. > > I ran a test writing data to: > X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet\data > And then copied the dashboard.yaml files to > X:\Projects\CHAMP7\Run25w_Market\dc_test\topsheet. Many of the > visualizations are not loading. It looks like the maps are expecting a > "geo" subfolder. Does this currently need to be manually copied in? Some of > the vega lite visualizations are also broken > > — > Reply to this email directly, view it on GitHub > <#7 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BDBK5I4UPOQZFG63U3W5MOLZDO4AZAVCNFSM6AAAAABICMU4R6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGQ3TCNZXHE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[sdrewc]

@weiwang108 thanks for the explanation. We want to be able to set up a topsheet for every model run and have it in a subfolder within the model run directory. So we can't just write out to the topsheet repo because you'll just overwrite the contents and only be able to have a single topsheet at a time.

It makes sense to have a common shared set of shapefiles and vega config files. Maybe the best way to accomplish this is to specify the geo and data paths in the config file, then check whether the needed files are there, and if not copy them in?

[weiwang108]

Hi Drew, I'm not sure if I've understood correctly, but I believe we can manage multiple topsheet dashboards simultaneously. For each model run, we can download the simwrapper_topsheet GitHub repository into the specific model run directory, execute the batch file, and all input data will be written into the "data" folder of that particular topsheet. This way, each topsheet for each model run operates independently, with separate "data" folders (for datasets) and "geo" folders (for shape files), ensuring that they do not interfere with one another. *Wei Wang*TD&A Intern Office: 415-593-1664 ***@***.*** sfcta.org <https://www.sfcta.org/> | sign up for our newsletter <https://www.sfcta.org/stay-connected>

…

On Tue, May 21, 2024 at 2:47 PM Drew C ***@***.***> wrote: @weiwang108 <https://github.com/weiwang108> thanks for the explanation. We want to be able to set up a topsheet for every model run and have it in a subfolder within the model run directory. So we can't just write out to the topsheet repo because you'll just overwrite the contents and only be able to have a single topsheet at a time. It makes sense to have a common shared set of shapefiles and vega config files. Maybe the best way to accomplish this is to specify the geo and data paths in the config file, then check whether the needed files are there, and if not copy them in? — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDBK5I2MEGQ7XE2IYRM252DZDO6GJAVCNFSM6AAAAABICMU4R6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGQ4TIMJYGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>