Add a way to choose line endings in compiled files

[Jackenmen]

What's the problem this feature will solve?

I'm trying to use the pip-compile hook with my existing pre-commit configuration but one of the hooks I have is a mixed line ending auto-fixer (which I use to ensure all files in the repository use LF for line endings):

repos:
  - repo: https://github.com/jazzband/pip-tools
    rev: 6.2.0
    hooks:
      - id: pip-compile
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.4.0
    hooks:
      - id: mixed-line-ending
        args:
          - "--fix=lf"

This configuration causes issues whenever I try to touch a requirements file.

Describe the solution you'd like

A way to set the line endings used for the generated requirements.txt file. The alternative would be for pip-compile to always use \n but then I'm guessing someone else might complain about it in the future like I'm doing now :)

Alternative Solutions

I can work around this issue by excluding requirement files from the check but this doesn't really solve it as I still end up with requirements files ending with CRLF, it just allows me to continue to use both hooks.

Another solution would be a way for --dry-run to return non-zero exit code when the file would have any dependencies changed (like what #882 and #1215 suggest) but this doesn't fully solve it either as then you lose the benefit of pip-compile auto-fixing the issue during pre-commit run.

Additional context

I'm using on Windows - os.linesep is equal to "\r\n" there

[graingert]

Imho pip-compile should reuse whatever line separator it finds first in requirements.txt, and fall back to the first line separator it finds in requirements.in

[AndydeCleyre]

This configuration causes issues whenever I try to touch a requirements file.

Can you please elaborate on the problem? Do you not have a guaranteed order of hook runs (to ensure the line-ending hook runs last)?

[Jackenmen]

Run this on Windows (due to os.linesep being equal to "\r\n" there):

python -m pip install -U pre-commit
git clone https://github.com/jack1142/pip-tools-issue-1448
cd pip-tools-issue-1448
pre-commit run --all-files

See the output:

pip-compile..............................................................Passed
Mixed line ending........................................................Failed
- hook id: mixed-line-ending
- exit code: 1

requirements.txt: fixed mixed line endings

The problem is that pip-compile reintroduces the issue with line endings whenever it runs therefore causing the mixed-line-ending hook to be triggered whenever requirements.in/txt is modified in the commit or when using pre-commit run --all-files. The guaranteed order of hook runs doesn't matter here, pre-commit will fail the commit if any of the hooks fail and that's what is happening here.

[AndydeCleyre]

I've done a first try at https://github.com/AndydeCleyre/pip-tools/tree/feature/linesep -- no tests yet, no PR yet.

@jack1142 can you test out that branch in your pre-commit setup, and add the arg --newline=lf to the config?

[Jackenmen]

@AndydeCleyre Seems to work as intended.

[AndydeCleyre]

If any Windows users are reading this who haven't already responded to the poll about the best default behavior, please go have a look, thanks!

[AndydeCleyre]

@graingert

Imho pip-compile should reuse whatever line separator it finds first in requirements.txt, and fall back to the first line separator it finds in requirements.in

The PR is now defaulting to preserve behavior, which does intend to reuse what it finds in requirements.txt.

Currently there is no checking of input file(s). I would prefer not to bother with that, as for example we may have multiple input files with differing newlines. Do you think this is good enough, for the preserve behavior?

[AndydeCleyre]

Checking in:

There is a PR in good condition that addresses this in a flexible way, #1491 #1652, but an alternative approach has been proposed to just use LF no matter what.

Either solution Works For Me, but I'm requesting comments from Windows users and those with mixed OS dev teams.

[AndydeCleyre]

@ssbarnea @jack1142

Maybe the move forward is to collpase #1491 #1652 into a single new flag, --preserve-newlines (or similar):

• without it, always use LF
• with it, and existing newlines can be detected, use those
• with it, and no existing newlines are detected, use LF

What do you think?

[graingert]

In the usecase stated in the issue "preserve" is a better choice. The pre-commit hook https://github.com/pre-commit/pre-commit-hooks#mixed-line-ending in combination with "preserve" will handle this

[AndydeCleyre]

So more like an --LF flag is called for?

[Jackenmen]

@ssbarnea @jack1142

Maybe the move forward is to collpase #1491 into a single new flag, --preserve-newlines (or similar):

• without it, always use LF
• with it, and existing newlines can be detected, use those
• with it, and no existing newlines are detected, use LF

What do you think?

Seems fine to me though Thomas commented in the PR that preserve should be default (which I'm not sure I agree with but it doesn't matter to me since it's going to end up as LF in both cases anyway)