If I delete a file in a `1-in` directory, re-running `hledger-flow import` does not remove the corresponding files in the `2-preprocessed` and `3-journal` directories

[jfly]

First off, thanks for a wonderful tool!

Describe the bug
If I delete a file in a 1-in directory, re-running hledger-flow import does not remove the corresponding files in the 2-preprocessed and 3-journal directories.

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.14.4 linux x86_64 ghc 9.0 UNKNOWN

I know this is not the latest version of hledger-flow, sorry. I looked at the diff between 0.14.4 and 0.15, and I don't think anything related to this has been changed. Let me know if it's really important to test with the latest version of hledger-flow, and I'll figure out how to update.

Also include the runtime options of the command you ran, e.g:

$ hledger-flow --show-options import

To Reproduce

I can reproduce this with a fresh clone of https://github.com/apauley/hledger-flow-example:

$ rm import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

$ hledger-flow import . --show-options
RuntimeOptions {baseDir = "/home/jeremy/src/github.com/apauley/hledger-flow-example/", importRunDir = "./", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.4 linux x86_64 ghc 9.0 UNKNOWN", hledgerInfo = HledgerInfo {hlPath = "/nix/store/lxv7a7ps6k7girbwm50kr3w42ih90v8w-hledger-1.25/bin/hledger", hlVersion = "hledger 1.25, linux-x86_64"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [9,0], versionTags = []}, cores = 4, availableCores = 4}, verbose = False, showOptions = True, sequential = False, batchSize = 200}
Collecting input files from /home/jeremy/src/github.com/apauley/hledger-flow-example/import/
Found 2 input files in 0.001297681s. Proceeding with import...
Wrote include files for 3 journals in 0.002288837s
Imported 2/2 journals in 0.061123857s

$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

no changes added to commit (use "git add" and/or "git commit -a")

Expected behavior

In the example above, I'd expect import/gawie/bogart/cheque/2-preprocessed/2016/123456789_2016-03-30.csv and import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal to get deleted as well.

For now, I am working around this by first nuking my 2-preprocessed and 3-journal directories before doing an import:

$ rm -r import/*/*/*/2-preprocessed import/*/*/*/3-journal
$ hledger-flow import .

Desktop (please complete the following information):
I'm running Arch Linux. I installed hledger and hledger-flow with the Nix package manager.

[apauley]

Yes, I also run into this from time to time, and then I manually delete those generated files.

I guess we can automatically delete all files in 2-preprocessed and 3-journal during each import - but I don't know if some people have put other files they care about in those directories.

Maybe if we limit our auto-deletion to just .csv and .journal files it should be safe

[jfly]

Just one data point: I'm quite new to this tool, and have read a lot of the documentation over the last few weeks. All the documentation I read left me under the impression that these 2-preprocessed and 3-journal directories are auto-generated, I wouldn't expect anything I put in them to be "safe". (Heck, I probably wouldn't have even version controlled them if not for the comment here about "In a typical software project we don’t add generated files to version control...")

[TC72]

I posted an issue related to this. How do you guys edit your journal files for things you don't catch with the rules file?

Ledger handles this by only importing new items from the csv using a .latest file in the csv directory to track what's already been imported. They also only add to the end of a journal, never overwrite it.

Starting to feel like I should always be running hledger-flow with --new-files-only and use hledger to import csv files that get updated.

[jfly]

@TC72, good question! I left a comment over on your issue: #111 (comment)