Strategy not found when running interactive diagnostics/backtesting

[pwholzel]

I'm probably not understanding where the strategy name needs to be. Must it be its own folder with the yaml files inside of it?

[tgibson11]

A few things here:

1. The actual error looks to be due to the "." characters in directory names being replaced by "/". That is pysystemtrade trying to support both Python package names and Linux paths. I'll have to leave this one to Rob. My opinion is that this feature results in unnecessary complexity.

2. It looks like you moved your config files to a private/example/ folder, but the config filename in the screenshot doesn't match.

3. You don't actually need to have a folder matching your system name. It's fine if that's what you want...just saying.

[tgibson11]

4. Your private_config and private_control_config should be in private, not private/example.

If you want to use a non-standard location for those files, you can do so by setting the PYSYS_PRIVATE_CONFIG_DIR environment variable. But I think you should probably just use the default location at least until you get the rest of this figured out.

[tgibson11]

I just noticed you are trying to do an interactive backtest. That changes a few things...

First, you need to have a saved result (.pck file). I don't see one of those anywhere. That file is generated when you run a successful production backtest. So you need to do that first, if you haven't.

The .pck file will be stored in the <backtest_store_directory>/<system_name> from your private config. If this isn't defined, I think it goes to pysystemtrade/private/<system_name>/. I personally don't think that's a good place to store them, but in theory it should work. The <system_name> sub-folder will be created automatically if it doesn't exist.

[tgibson11]

I feel like I've thrown a lot of things at you there. To summarize:

1. Make sure your private_config and private_control_config are in the private folder
2. Make sure the path to your system config file (in private_control_config) matches wherever the file actually is
3. Define backtest_store_directory in private_config. I use /home/todd/data/backtests, but it can be anywhere.
4. Install pysystemtrade
5. run_systems
6. Now you should be able to open up the saved backtest with interactive_diagnostics.

[pwholzel]

Thanks, so much Todd. Can you explain why I need to reinstall pysystemtrade?

[tgibson11]

Thanks, so much Todd. Can you explain why I need to reinstall pysystemtrade?

Maybe you don't, but it's what we were discussing in the issue you opened yesterday. The config files being referenced when you run things aren't the ones in the pysystemtrade git repo (at least sometimes they aren't). They are the versions that were installed in the virtual environment the last time you installed pysystemtrade (unless you installed with pip install -e or setup.py develop...ugh.)

Since the main solution to your problem here is the part about adding backtest_store_directory to your private_config, I don't want to take any chances that it's going to use an old version of that file. In other words, I don't want your next post to be "Hey, Todd, your solution didn't work!" 😃

[tgibson11]

I am assuming that you will modify the one in the git repo, then install it in the virtual environment.

If, OTOH, you are modifying things directly in the virtual environment, then you would not want to install over the top of it. Ugh.

[pwholzel]

Thanks, again, Todd.
When you say "system config file", is that the one that specifies which trading rules to use, your targeted volatility, etc?
The .pck file should go in the backtests folder, right?

I followed your above steps, but now seeing this.

[tgibson11]

Thanks, again, Todd. When you say "system config file", is that the one that specifies which trading rules to use, your targeted volatility, etc?

Yes. Just make sure that file is where private_control_config says it should be

The .pck file should go in the backtests folder, right?

It will be saved there after you run the system. It's not something you should need to put there yourself.

I followed your above steps, but now seeing this.

1. Your private_control_config file should have start and end times for each production process. If you are running things manually, you probably want to set those to 00:00 and 23:59.

2. The way you are calling run_systems.py will fail later because it can't find the instrument config file. This is a bug in pysystemtrade. (See PR Fix minor bug in path resolution #1095 if you're interested.) Use the linux run script instead: ~/pysystemtrade/sysproduction/linux/scripts/run_systems.

[pwholzel]

Making progress, but now it says unable to get capital data. Thanks, again.

[tgibson11]

Capital update apparently failed previously. You need to mark it as finished before running again.

~/pysystemtrade/sysproduction/linux/scripts/interactive_controls

Option 4, then option 43.

HOWEVER...run_capital_update attempts to get the current capital from an IB account. Are you in a position to do that?

I think maybe what you really want here is to run a backtest as a simulation, not a production backtest.

If that is the case, see https://github.com/robcarver17/pysystemtrade/blob/master/docs/backtesting.md#how-do-icreate-a-standard-futures-backtest

That doesn't generate something that can be used with interactive_diagnostics...(you could, but you'll have to write some code and figure that out.)

[pwholzel]

Thanks, Todd. That trick to closed the process worked, but I am still getting the same error.

For more context: I have a backtest that I'm happy with pickled into a .pck file, if that helps. I have set up all the preliminaries to start running the system on a paper account at IB (with 250K of paper money). I have a baseline of prices in mongo/arctic, and have created roll calendars and multiple/adjusted price series in mongo/arctic. So, now I'd just like to get the system running live. Or, at the very least, I'd like it to tell me what positions I should take on, so I could manually trade them. I have basically followed all of the steps here (https://github.com/robcarver17/pysystemtrade/blob/master/docs/production.md#quick-start-guide) and am now stuck on the interactive diagnostics step. Thanks, again for any help!

[tgibson11]

Ok, that helps a lot. I was starting to wonder what exactly you were trying to accomplish.

So in that case, I think the next step would indeed be to run the capital update process.

Thanks, Todd. That trick to closed the process worked, but I am still getting the same error.

I don't quite understand this, though. If you're still getting the same error, then it didn't work...which error are you referring to?

[tgibson11]

If you're referring to the error when doing run_systems, you probably have some missing or incorrect config.

In private_config, you should have something like this.

## Strategy configuration
strategy_list:
  futures_system:
    load_backtests:
      object: sysproduction.strategy_code.run_dynamic_optimised_system.runSystemCarryTrendDynamic
      function: system_method
    reporting_code:
      function:  sysproduction.strategy_code.report_system_dynamic_optimised.report_system_dynamic

#
# Allocate capital to strategies
# strategy_weights *must* be defined in private config
strategy_capital_allocation:
  function: sysproduction.strategy_code.strategy_allocation.weighted_strategy_allocation
  strategy_weights:
    futures_system: 100.0

Your values will obviously be different, but you need to add these sections and/or correct the values. Note especially the names of the systems here...it seems like you've changed things from when this discussion first started: peter_system has become peter_system_folder???

The "Allocate capital to strategies" part tells the capital update process what to do. So when you think you have the config correct, you'll need to run_capital_update again.

Then try run_systems and see what happens.

[tgibson11]

And make sure the system names here (private_config) match private_control_config too.

[pwholzel]

Thanks, again, Todd. I was missing those elements in private_config, so I added them back and changed my system name to peter_system to avoid confusion. I have made updates as described (both in my virtual env folders and my normal env folders, to be safe). Here is what I am getting. That value (250692) matches what I have in my paper account, so not sure what the issue is, but then it gets to this point and hangs.

[pwholzel]

Here's my private config yaml, if it helps:

broker_account: DU6891304 #papertrade

email_address: 'email@gmail.com' # add your address here
email_pwd: 'terrible_mash_of_stuff_for_security' # your password
email_server: 'the_outgoing_smtp_server' # the smtp server

ib_ipaddress: 127.0.0.1 # this changes if you set up elsewhere
ib_port: 4001 # the default for the ib Gateway
ib_idoffset: 100 # the default

mongo_host: 127.0.0.1 # can be elsewhere
mongo_db: 'production' # fits with all the documentation

backtest_store_directory: '/private/backtests'

strategy_list:
peter_system:
load_backtests:
object: sysproduction.strategy_code.run_dynamic_optimised_system.runSystemCarryTrendDynamic
function: system_method
reporting_code:
function: sysproduction.strategy_code.report_system_dynamic_optimised.report_system_dynamic

strategy_capital_allocation:
function: sysproduction.strategy_code.strategy_allocation.weighted_strategy_allocation
strategy_weights:
peter_system: 100.0

[pwholzel]

And, here is private control config:

process_configuration_start_time:
default: '00:00'
run_stack_handler: '00:01'
run_capital_update: '01:00'
run_daily_prices_updates: '20:00'
run_daily_fx_and_contract_updates: '07:00'
run_daily_update_multiple_adjusted_prices: '23:00'
run_systems: '17:15'
run_strategy_order_generator: '20:10'
run_backups: '20:15'
run_cleaners: '20:20'
run_reports: '20:25'
process_configuration_stop_time:
default: '23:59'
run_strategy_order_generator: '23:50'
run_stack_handler: '19:45'
run_capital_update: '19:50'
run_daily_prices_updates: '23:50'
run_daily_fx_and_contract_updates: '23:50'
run_daily_update_multiple_adjusted_prices: '23:50'
run_systems: '23:50'
run_backups: '23:50'
run_cleaners: '23:50'
run_reports: '23:50'

process_configuration_methods:
run_capital_update:
update_total_capital:
frequency: 120
max_executions: 10
strategy_allocation:
max_executions: 1
run_daily_prices_updates:
update_historical_prices:
max_executions: 1
run_daily_fx_and_contract_updates:
update_fx_prices:
max_executions: 1
update_sampled_contracts:
max_executions: 1
run_daily_update_multiple_adjusted_prices:
update_multiple_adjusted_prices:
max_executions: 1
run_stack_handler:
refresh_additional_sampling_all_instruments:
frequency: 60
max_executions: -1
check_external_position_break:
frequency: 0
max_executions: -1
spawn_children_from_new_instrument_orders:
frequency: 0
max_executions: -1
generate_force_roll_orders:
frequency: 0
max_executions: 1
create_broker_orders_from_contract_orders:
frequency: 0
max_executions: -1
process_fills_stack:
frequency: 0
max_executions: -1
handle_completed_orders:
frequency: 0
max_executions: -1
safe_stack_removal:
run_on_completion_only: True
run_reports:
costs_report:
max_executions: 1
liquidity_report:
max_executions: 1
status_report:
max_executions: 1
roll_report:
max_executions: 1
daily_pandl_report:
max_executions: 1
reconcile_report:
max_executions: 1
trade_report:
max_executions: 1
strategy_report:
max_executions: 1
risk_report:
max_executions: 1
slippage_report:
max_executions: 1
instrument_risk_report:
max_executions: 1
min_capital:
max_executions: 1
duplicate_market:
max_executions: 1
remove_markets_report:
max_executions: 1
market_monitor_report:
max_executions: 1
account_curve_report:
max_executions: 1
run_backups:
backup_arctic_to_csv:
max_executions: 1
backup_files:
max_executions: 1
backup_mongo_data_as_dump:
max_executions: 1
run_cleaners:
clean_backtest_states:
max_executions: 1
clean_echo_files:
max_executions: 1
clean_log_files:
max_executions: 1

run_systems:
peter_system:
max_executions: 1
object: sysproduction.strategy_code.run_system_classic.runSystemClassic
backtest_config_filename: /private/peter_system.yaml
run_strategy_order_generator:
peter_system:
object: sysexecution.strategies.classic_buffered_positions.orderGeneratorForBufferedPositions
max_executions: 1

[tgibson11]

1. The capital update process runs as a loop by default. Change max_executions to 1 for update_total_capital. It's not hanging, it's waiting 2 hours before doing the next update.

2. You have critical errors happening there...it's trying to send you emails, but you don't have emails configured. I wouldn't worry about that now, but it means you need to pay attention to the process output.

2a) update_total_capital should be able to initialize the capital, but it looks like it can't. This might be a bug. You can try interactive_update_capital_manual, but it looks like that might have the same problem.

2b) Looks like there might also might be a config issue allocating strategy capital, but lets get total capital working before worrying about that.

[tgibson11]

OK, after digging through the code a bit, I see the thing to do is interactive_update_capital_manual. It does have an option for initializing capital.

Do that, then run_capital_update and show me the output.

[pwholzel]

Great, job! It worked. I will try running the system now.

[pwholzel]

It's showing not yet time to run. I guess that is a config somewhere?

[tgibson11]

Scrolling up to your private_control_config, I see you have the run_systems start_time set for 17:15

[pwholzel]

Got it, thanks. Now it can't find peter_system.yaml, although it is right there (both in virtual and normal environments).

[tgibson11]

In private_control_config, you have

backtest_config_filename: /private/peter_system.yaml

You have 3 options:

1. Change to /home/peter/pysystemtrade/private/peter_system.yaml, to refer to the config file in the git repo
2. Remove the leading "/", to refer to the config file in the venv
3. Provide the full absolute path to the config file in the venv

I prefer 1, but for consistency, maybe you should do 2, since other config files are being referenced from the venv.

[pwholzel]

Excellent, it worked! I will plan on running this live (on my paper account) on Monday.

Thanks for all your generous help, Todd.

[pwholzel]

Closing as resolved.