Best way to work with codebase

[dr1ver1]

This question may be better on StackExchange, but asking here, as everyone using pysystemtrade needs to deal with this. I have little prior experience of working with git and I'm wondering how best to work with the codebase. I've created my own fork in github. I want to be able to get all updates to the original codebase.

Rather than change Rob's scripts directly, I'm using import to access them from my own scripts, and then making tweaks, e.g. changing paths, changing the values of variables, and maybe not calling all of Rob's functions as I have preprocessed some of the market data in my own code.
I copy & paste the "if name == main" section, so violating DRY, but (so far) these sections are quite small - I'm currently focused on the "data" section of the implementation.
My reasoning for this approach is I don't know how much the original scripts could change, and I'm worried that if I amend them directly, they could break due to updates to the source, which are pretty frequent.

Is there a best practice? What are other people doing?

[tgibson11]

I like your approach, but I've found that most of the time when I want to do something differently from Rob, unless it's something he's specifically designed to be overridden (like system definition & the config files), it's usually easier to just modify his code.

I don't mean to discourage you from your approach...its just my opinion that it wasn't worth the effort. (I'm lazy.)

OTOH, like you, I also want to keep up to date with his latest changes.

Some ways I manage that:

1. I have a dummy pull request on github to merge Rob's master branch into mine. I never merge or close this PR, but its existence allows me to get an email notification any time Rob makes a commit to master. Maybe github has a simpler way to do this, but I could never find one.

2. On my local git instance, I have added his github repository as an additional remote:

$ git remote add robcarver17 https://github.com/robcarver17/pysystemtrade.git

That makes it easy fetch his latest changes and merge them into my repository.

3. Every now and then I do an audit of all the differences between my code and Rob's master branch. I do this in PyCharm, which integrates nicely with git, but you can also compare branches across forks on github.

Bonus tip: This should go without saying, but if you are running a production system, always test things in a non-production environment first. Especially if you're merging in a bunch of Rob's changes where you may not necessarily understand all the impacts.

[robcarver17]

What I would do is what I do myself - use the private directory for your own stuff, but never change the original source code outside of that (and @tgibson11 has given you a nice method for keeping this up to date). As well as your own config, you might for example inherit from the classes I use (I do this for raw data for example to get extra trading rules).

Then you can just pull the changes that come in automatically (of course being careful to test / understand for production code - I would have a segregated repo, ideally on another machine, for this purpose)

[dr1ver1]

I need to up my git game - @tgibson11 those sound like good suggestions.
Yeah - I am already using a private folder, based on recommendations in the documentation.
In terms of running on separate hardware, I'm going to investigate Dell/HP machines that have been decommissioned by enterprises

[bug-or-feature]

This is a great question. Like @tgibson11, my fork has a lot of custom code, but I also want to keep up to date with Rob's incoming changes. My solution:

1. like Todd, add a remote to Rob's repo, although I call mine upstream
2. keep my master branch as an exact clean copy of upstream
3. make another branch (development) for all my work. I configured this one to be the main branch in my fork's settings
4. occasionally pull down updates onto master, then merge onto development
5. like Todd, occasionally audit differences between master and development. PyCharm/IntelliJ has really good visual tools for this
6. if I want to request something I've done be included in the main repo, the process is:
   1. make sure master is up to date
   2. create a new branch from that
   3. cherry pick the change from development to that new branch
   4. create a PR for that new branch
7. For keeping track of new commits to upstream I use SourceTree. It's a visual windowed app for git, free. Sometimes a nice picture is easier to understand
8. I also want to know about any new Issues, PRs, discussions in the main repo, so my GitHub account is configured to send me an email whenever anything happens. Those messages get filtered to a dedicated folder
9. Have a dedicated machine for production. Code running here is never be edited manually, only pulled from a repo

[dr1ver1]

Thanks for all of the detail on git - that's incredibly helpful.
I'm using VS Code. I'll investigate audit add-ins for that.

In terms of working with the code, what approach do you take? Do you modify Rob's scripts? Create classes that inherit from his? Import his scripts into your scripts? ...etc

[tgibson11]

I don't think those are mutually exclusive. I do all of them, as I feel it is appropriate.

I definitely import his scripts into my own, otherwise I'd be duplicating a lot of code.

The classes that define & run my system inherit from Rob's classes. I might have 1 or 2 others, but not many. In theory it would be nice to do this more, but then I would STILL have to modify Rob's code to use my new class, except where overriding is expected and explicitly supported.

What I mostly do is modify Rob's scripts.

Let me give you an example of what I mean: I use IB's adaptive market orders instead of Rob's execution algo (which uses limit orders). I'd have to override a few of Rob's classes to support that, then I'd have to also modify every script that references the overridden classes to use my own instead. So I'm still modifying Rob's code...unless I also override THOSE classes, and then the classes that reference THOSE, and so on. By the end I would have re-written much of pysystemtrade.

[tgibson11]

One more thing...

If you're already familiar & happy with VS Code, I wouldn't bother trying to switch. I'm sure it can do all the same things as PyCharm (mainly thinking of git integration here, but that is probably generally true also).

Although, PyCharm does seem to be more popular wrt pysystemtrade, and it might make it slightly easier to get help if you need it.

[bug-or-feature]

In terms of working with the code, what approach do you take? Do you modify Rob's scripts? Create classes that inherit from his? Import his scripts into your scripts? ...etc

I'm like @tgibson11 - all of the above. Whatever is needed to get the job done, whilst making minimal changes to the codebase, and improving the code for other users if possible.

The best resource I found for getting to know git was the Atlassian tutorial site:

https://www.atlassian.com/git

[dr1ver1]

Thanks all for the suggestions

[slimalim]

Am considering using a private repo as a submodule to manage my private folder. Does anyone else do it this way or is there a more elegant way to source control things you'd rather not be in the public repo?

[robcarver17]

This is how I do it

[tgibson11]

If your private files are only config, @bug-or-feature has recently implemented a nice feature to allow the config files to reside outside the pysystemtrade directory, using an environment variable to point to them.

If you have Python code that you want to be private, that probably does need to be a submodule.

Regarding the private repo as submodule, you might have problems with a git repo nested inside another git repo. I've never tried that, but it sounds like something I'd avoid just on principle.

[bug-or-feature]

I looked into using git submodules for this, but they were designed to support a different use case, one where the nested project is a public repo. For me, having the private repo outside the pysystemtrade repo made more sense. There's a brief description here

[slimalim]

Thanks @robcarver17, I forgot about that part of production.md 🤦‍♂️. The process of working in pysystemtrade/private in its default location is what I was looking for, so that things look the same as they will in production. Moving it out and in again for source control is neat.
@tgibson11 @bug-or-feature agree - nested repos just feel odd. PYSYS_PRIVATE_CONFIG_DIR is a clean solution. Could open the private repo as an attached project to the main pysystemtrade project in PyCharm to keep the workflow in the same window. I'll give this a go.
Thanks for your thoughts and pointers on this, it is much appreciated.