Timeout when running calc_system_size_and_performance

[amy-wang]

Issue overview

I've followed all of the set up steps. When I try to run the model (dgen_model.py) for New York State, it hangs when calc_system_size_and_performance is run in dgen_model.py (line 230).

It seems like there's an error in Agents.apply_chunk_on_row. When I interrupt this, here's the message I get:

KeyboardInterrupt                         Traceback (most recent call last)
File ~/dgen/dgen_os/python/agents.py:347, in Agents.apply_chunk_on_row(self, func, cores, **kwargs)
    345         futures.append(executor.submit(func, row, **kwargs))
--> 347     results = [future.result() for future in futures]
    348 results_df = pd.concat(results, axis=1, sort=False).T              

File /opt/anaconda3/envs/dg3n/lib/python3.12/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    449     return self.__get_result()
--> 451 self._condition.wait(timeout)
    453 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /opt/anaconda3/envs/dg3n/lib/python3.12/threading.py:355, in Condition.wait(self, timeout)
    354 if timeout is None:
--> 355     waiter.acquire()
    356     gotit = True

KeyboardInterrupt: 

During handling of the above exception, another exception occurred:

KeyboardInterrupt                         Traceback (most recent call last)
File ~/dgen/dgen_os/python/dgen_model.py:334
    331             utilfunc.code_profiler(model_settings.out_dir)
    333 if __name__ == '__main__':
--> 334     main()

File ~/dgen/dgen_os/python/dgen_model.py:230, in main(mode, resume_year, endyear, ReEDS_inputs)
    227 solar_agents.on_frame(agent_mutation.elec.apply_state_incentives, [state_incentives, year, model_settings.start_year, state_capacity_by_year])
    229 # Calculate System Financial Performance
--> 230 solar_agents.chunk_on_row(financial_functions.calc_system_size_and_performance, sectors=scenario_settings.sectors, cores=cores, rate_switch_table=rate_switch_table)
    232 # Calculate the financial performance of the S+S systems
    233 #solar_agents.on_frame(financial_functions.calc_financial_performance)
    234 
    235 # Calculate Maximum Market Share
    236 solar_agents.on_frame(financial_functions.calc_max_market_share, max_market_share)

File ~/dgen/dgen_os/python/agents.py:123, in Agents.chunk_on_row(self, func, cores, in_place, **kwargs)
    120 def chunk_on_row(self, func, cores=1, in_place=True, **kwargs):
    121     """Wrapper function around apply_on_row with runtime tests."""
--> 123     results_df = self.run_with_runtime_tests(how_to_apply='chunk_on_row', func=func, cores=cores, **kwargs)
    125     if in_place:
    126         self.df = results_df

File ~/dgen/dgen_os/python/agents.py:198, in Agents.run_with_runtime_tests(self, how_to_apply, func, func_args, cores, **kwargs)
    196     results_df = pd.merge(self.df, results_df, on='agent_id')
    197 elif how_to_apply == 'chunk_on_row':
--> 198     results_df = self.apply_chunk_on_row(func, cores=cores, **kwargs)
    199     results_df['agent_id'] = results_df['agent_id'].astype(int)
    200     results_df = pd.merge(self.df, results_df.drop_duplicates(subset='agent_id'), on='agent_id')

File ~/dgen/dgen_os/python/agents.py:342, in Agents.apply_chunk_on_row(self, func, cores, **kwargs)
    339 chunk_size = int(self.df.shape[0]/cores)
    340 chunks = [self.df.loc[self.df.index[i:i + chunk_size]] for i in range(0, self.df.shape[0], chunk_size)]
--> 342 with EXECUTOR(max_workers=cores) as executor:
    343     for agent_chunks in chunks:
    344         for _, row in agent_chunks.iterrows():

File /opt/anaconda3/envs/dg3n/lib/python3.12/concurrent/futures/_base.py:647, in Executor.__exit__(self, exc_type, exc_val, exc_tb)
    646 def __exit__(self, exc_type, exc_val, exc_tb):
--> 647     self.shutdown(wait=True)
    648     return False

File /opt/anaconda3/envs/dg3n/lib/python3.12/concurrent/futures/process.py:865, in ProcessPoolExecutor.shutdown(self, wait, cancel_futures)
    862         self._executor_manager_thread_wakeup.wakeup()
    864 if self._executor_manager_thread is not None and wait:
--> 865     self._executor_manager_thread.join()
    866 # To reduce the risk of opening too many files, remove references to
    867 # objects that use file descriptors.
    868 self._executor_manager_thread = None

File /opt/anaconda3/envs/dg3n/lib/python3.12/threading.py:1149, in Thread.join(self, timeout)
   1146     raise RuntimeError("cannot join current thread")
   1148 if timeout is None:
-> 1149     self._wait_for_tstate_lock()
   1150 else:
   1151     # the behavior of a negative timeout isn't documented, but
   1152     # historically .join(timeout=x) for x<0 has acted as if timeout=0
   1153     self._wait_for_tstate_lock(timeout=max(timeout, 0))

File /opt/anaconda3/envs/dg3n/lib/python3.12/threading.py:1169, in Thread._wait_for_tstate_lock(self, block, timeout)
   1166     return
   1168 try:
-> 1169     if lock.acquire(block, timeout):
   1170         lock.release()
   1171         self._stop()

KeyboardInterrupt: 

Has anyone run into this before? Do you have any suggestions on resolving it? Thank you!

Details

Some additional details for this issue (if relevant):

Platform: Mac
Version of dGen: Most recent (forked on 3/15)
Python version: Python 3.12.7
matplotlib: 3.9.2
pandas: 2.2.2
numpy: 1.26.4
scipy: 1.13.1

[michaelavs]

Hi @amy-wang,
This function usually takes the bulk of the time for the model to process and is directly related to the number of agents that are in the input agent file. However, while this behavior may be anticipated, I want to make sure there isn't something else going on that isn't expected. my suggestion to test this out is to subset the input agent file to about 10 agents and then re-run the model using that agent file to double check the model performance.

You can use the following code to subset and save the agent file:

import pandas as pd
df = pd.read_pickle('path/to/your/agent_df_base_res_ny_revised.pkl')

# subset df to the first 10 agents
sub = df[0:10]

# export to pickle file
sub.to_pickle('subset_NY_agents.pkl')

You can create a new python file in the "input_agents" directory, copy and run this code there, and then update the input_sheet to look for the output subset dataset. If the model runs successfully with that file, something that you may be able to do to reduce the time spent on that function is to remove the definition of "res_with_batt" from calc_system_size_and_performance if you are only looking at rooftop PV and not concerned with any battery adoption. The optimization function used for calculating the ideal PV vs PV+battery system takes a while to solve, so removing that call should also help to speed up the model. If you go with this route and have questions about what needs to be removed/changed, feel free to let me know and I can send along some code!