Custom failure handler on Windows causes assert in DEBUG and undefined behavior in Release

[wopss]

I'm writing a custom handler on Windows following the README.md example (the exact same copy) and I'm getting a LIBASSERT_PRIMITIVE_DEBUG_ASSERT in Debug builds. In release mode, the assert is not shown, but it can cause undefined behavior due to LIBASSERT_UNREACHABLE_CALL macro.

The assert is happening because detail::needle(fd).lookup does not find the fd value in the provided ones:

libassert/src/platform.cpp

Lines 43 to 47 in bd33ba1

	DWORD windows_handle = detail::needle(fd).lookup(
	STDIN_FILENO, STD_INPUT_HANDLE,
	STDOUT_FILENO, STD_OUTPUT_HANDLE,
	STDERR_FILENO, STD_ERROR_HANDLE
	);

The README is using libassert::stderr_fileno in the custom handler:

libassert/README.md

Lines 773 to 778 in bd33ba1

	libassert::enable_virtual_terminal_processing_if_needed(); // for terminal colors on windows
	std::string message = info.to_string(
	libassert::terminal_width(libassert::stderr_fileno),
	libassert::isatty(libassert::stderr_fileno)
	? libassert::get_color_scheme()
	: libassert::color_scheme::blank

But the implementation of the default handler is using the STD*_FILENO defines from platform.h:

libassert/src/assert.cpp

Lines 415 to 418 in bd33ba1

	std::string message = info.to_string(
	terminal_width(STDERR_FILENO),
	isatty(STDERR_FILENO) ? get_color_scheme() : color_scheme::blank
	);

My question now is: what is the intended public-facing fd for custom handlers? Should I use the libassert::*_fileno values or the STD*_FILENO macros?

If libassert::*_fileno is the public API, one way to avoid confusion would be to make it a type-safe representation (enum class fileno { stdin_, stdout_, stderr_ }) and accept only that type in the APIs, translating internally to the native values.

[jeremy-rifkin]

Thanks for opening this, it definitely looks like a bug. In theory libassert::*_fileno should match STD*_FILENO, the former is just a convenience that's meant to be cross-platform. I seem to remember some situation on windows where fileno(stderr) might not be 2. Is that what's going on here?

[wopss]

No problem! :)

Yes that is what is happening here. In my case fileno(stderr) is -2.

[jeremy-rifkin]

Gotcha, thanks. It looks like -2 indicates there's no output stream - I guess in such a case we'd just expect no output but the reporting logic should not cause assertion failures. I'll fix the definitions for libassert::std*_fileno on windows.