[regression][6.10.0] `pip-compile` hangs on named pipes as input due to double-read

[webknjaz]

So I was trying to work out how to organize interaction between tox and pip-tools for my new/future tox-lock plugin where I'd generate constraint files for tox environments.

I figured I'd try making a named pipe in tox's tmp dir and pass that as an input to pip-compile. Then, I'd write whatever's necessary from the tox plugin into the pipe and be done with it. What could go wrong, right?

When I started testing pip-compile even w/o tox in the mix, it was hanging on me.

Eventually, I started running the editable install of pip-tools and stuck breakpoint() into a couple of places. I've found out that it reads from the pipe just fine for the first time. It goes through the dependency resolution and all's good.

It's when it gets to writing the output, the problems begin. _determine_linesep(strategy="preserve", filenames=('-', './named-pipe')) gets called with the default "preserve" strategy. And in this mode, it attempts opening the file by path in binary mode, getting stuck on existing_file.read():

pip-tools/piptools/scripts/compile.py

Line 60 in 78f18e1

existing_text = existing_file.read()

Environment Versions

1. OS Type: Gentoo Linux (shouldn't be relevant)
2. Python version: 3.13.7 (shouldn't be relevant)
3. pip version: 25.2
4. pip-tools version: main (78f18e1)

Steps to replicate

(*NIX instructions)

1. mkfifo -m0600 named-pipe
2. python -m piptools compile -o- ./named-pipe <-- this is where it gets stuck
3. (in a separate terminal tab) echo build > named-pipe <-- this unblocks reading the file contents for the first time and lets pip-compile compute the dep tree, then getting stuck in _determine_linesep()
4. another echo build > named-pipe unblocks pip-compile because it's able to read something from the pipe again

Expected result

It shouldn't hang.

Actual result

It does.

---

Since I'm writing to stdout, the first loop iteration in _determine_linesep() gets a "file not found" and skips to the second iteration that gets stuck. It also occurred to me that if a file called - were to exist, it'd mess up the linesep detection since it just reads from stdin at the beginning but there, it'd read from a file. This is a loosely related bug, I suppose.
I've tested that it's possible to identify a named file programmatically though stat or pathlib, but maybe this needs a more generic "is file" check.

$ python -u -m piptools compile -o- ./named-pipe
> ~/src/github/jazzband/pip-tools/piptools/scripts/compile.py(58)_determine_linesep()
-> breakpoint()
(Pdb) args
strategy = 'preserve'
filenames = ('-', './named-pipe')
(Pdb) pp locals()
{'filenames': ('-', './named-pipe'), 'fname': '-', 'strategy': 'preserve'}
(Pdb) c
> ~/src/github/jazzband/pip-tools/piptools/scripts/compile.py(58)_determine_linesep()
-> breakpoint()
(Pdb) pp locals()
{'filenames': ('-', './named-pipe'),
 'fname': './named-pipe',
 'strategy': 'preserve'}
(Pdb) pp os.path.isfile(fname)
False
(Pdb) pp os.path.isjunction(fname)
False
(Pdb) pp os.path.islink(fname)
False
(Pdb) pp os.path.ismount(fname)
False
(Pdb) pp os.path.isdir(fname)
False
(Pdb) pp os.path.isdevdrive(fname)
False
(Pdb) pp os.path.isabs(fname)
False
(Pdb) import stat
(Pdb) stat.S_ISFIFO(os.stat(fname).st_mode)
True
(Pdb) from pathlib import Path
(Pdb) pp Path(fname).is_fifo()
True

This is probably a regression introduced in @AndydeCleyre's #1652 that first appeared in pip-tools v6.10.0. I bet checked the workaround would be is --newline=[lf|crlf|native].

---

@sirosen I think we need a strategy for dealing with this. Perhaps, have some pathlib-like objects reused across multiple layers. Special-casing strings in multiple places clearly doesn't work well. Can we possibly auto-detect the new lines on first read?

[AndydeCleyre]

If I understand correctly, a (possibly unsatisfactory) fix/work-around could be this in the _determine_linesep loop:

if fname == '-' or Path(fname).is_fifo():
    continue

[sirosen]

I can definitely look at some options for this when I get time to circle back to pip-tools -- I've left a few half finished PRs which I need to come back to first.

My inclination is to say that the hanging behavior is one symptom of a more general problem: doing the same work twice. I think data sizes are small enough that we should ensure that we don't reread files, possibly going so far as to map all of our inputs to the bytes we read.
Does that line of thinking, treating the issue as "pip-tools reads the same file twice", sound good?

My first thought is to introduce a new abstraction and a cached getter to act as a global mapping:

class ParsedFile:
    ... # some pip-tools specific representation

@functools.cache
def parse(filepath: pathlib.Path) -> ParsedFile:
    ...

Maybe this is over-engineered, but it's where my mind runs with the problem.

[webknjaz]

If I understand correctly, a (possibly unsatisfactory) fix/work-around could be this in the _determine_linesep loop:

if fname == '-' or Path(fname).is_fifo():
    continue

Yes — this was my idea of a short-term solution with further fixes afterwards. Maybe even just

-if fname == '-' or Path(fname).is_file():
+if not Path(fname).is_file():
     continue

would do. Or:

-if fname == '-' or Path(fname).is_file():
+if fname == '-' or not Path(fname).is_file():
     continue

My inclination is to say that the hanging behavior is one symptom of a more general problem: doing the same work twice. I think data sizes are small enough that we should ensure that we don't reread files, possibly going so far as to map all of our inputs to the bytes we read. Does that line of thinking, treating the issue as "pip-tools reads the same file twice", sound good?

My first thought is to introduce a new abstraction and a cached getter to act as a global mapping:

class ParsedFile:
    ... # some pip-tools specific representation

@functools.cache
def parse(filepath: pathlib.Path) -> ParsedFile:
    ...

Maybe this is over-engineered, but it's where my mind runs with the problem.

Yes, this sounds like what I've been thinking more or less.