Proposal: Convert existing WGML documentation set to LaTeX

[pchapin]

Introduction

I propose that we start a project to convert the Open Watcom documentation set, currently in WGML format, to LaTeX. I would be interested in taking the lead on this project. This is a significant undertaking, and it will take time to complete, perhaps years, but I think it's the right move.

To give you an idea of what LaTeX-generated documents can look like, I converted the OWSTL Design Documentation to LaTeX format. Feedback is welcome.

Why?

WGML is an archaic format. As far as I know, Open Watcom is the only active project in the world that uses it. WGML also has a very old-school style with cryptic commands and awkward line-breaking rules, making WGML documents hard to read. I expect these things will prevent new contributors from wanting to maintain the documentation, resulting in documentation that slowly moves toward obsolescence. Personally, I like working with documentation, but I have zero interest in learning WGML. Therefore, if the documentation remains in the WGML format, I won't modify it. I suspect I'm not alone.

In contrast, LaTeX has a large and active user community, and it is continually being developed and improved. Knowledge of LaTeX has value outside of Open Watcom, so new contributors will be more likely to take the time to learn the system. I've used LaTeX extensively in the past, but I welcome the opportunity to delve into it more deeply, precisely because I know the time spent doing so will be helpful in other contexts.

WGML does not support Unicode. It may not support modern font systems, such as OpenType. If we continue to use WGML, we will assume the responsibility of maintaining and updating the document processor, as well as the Open Watcom compilers, libraries, and other tools. Considering the small size of the Open Watcom team, that's unreasonable.

Modern LaTeX processors support Unicode and OpenType fully out of the box. Additionally, the LaTeX community has developed hundreds of packages to address complex formatting needs. If we use LaTeX, we can tap into that ecosystem at once with no additional work on our part.

Why Not?

Aside from the work involved in making the conversion, the main problem is with output formats and build systems. LaTeX naturally speaks in PDF. Creating full-fidelity output to PostScript would also be possible. However, other output formats are problematic. The LaTeX processor is a Turing-complete macro engine. In theory, nothing short of executing all the macros faithfully could be expected to generate the correct output. Conversion tools are available that can convert LaTeX documents to other formats, notably HTML; however, they are imperfect. If we switch to LaTeX, we would have to abandon support for the OS/2 native document viewer and our DOS-based native document viewer.

On the other hand, PDF is nearly ubiquitous.

The LaTeX processor that I'm using is TeX Live, which is available for Windows, Linux, and macOS. However, it is not available for DOS or OS/2. There is an older processor for those systems called emTeX, but it is obsolete and probably can't cope with the modern styles we'd want to use. Switching to LaTeX would mean the documentation could no longer be built on DOS and OS/2 development hosts.

On the other hand, how many people are using those hosts to actually build Open Watcom?

Details of My Proposal

Since this work will take a considerable amount of time to complete, both systems will coexist for a period. Here is my current working plan for completing this project:

• Extract styling information from my OWSTL Design Document so that it can be reused in other documents.
• Manually convert other documents, one at a time, starting with the relatively simple ones that we rarely edit and moving toward the more complex ones. Once a document has been converted, it would become the source of truth for that document; editing the original WGML would no longer be appropriate. To give the community confidence in this process, I would start with documents that we rarely edit so that by the time we reach documents that we edit more often, the viability of this conversion will be clear.
• In parallel with the conversion process, I would develop a conversion assistance tool, likely in Python, that automates as much of the conversion as possible, informed by my manual conversion efforts. The goal would be to create a tool that could automatically convert at least 75% of the longer documents.

This process also provides us with an opportunity to review and refine the format and style of the documentation. For example, some of the documents currently contain large blank areas since they display information about library functions with one function per page. Honestly, I think that is terrible, especially for a document that will likely be searched online rather than printed. My thought was to "fix" that issue during the conversion.

Thoughts?

[Baron-von-Riedesel]

I think the last thing that OW 2.0 needs is another construction area. Also, I cannot see how this switch is going to improve the documentation - the lack of manpower won't be cured by it.

Generally, IMO OW 2.0 rather needs a phase of consolidation. That is, to determine the current status in a list, unbiased:
a) what (real!) features does OW v2.0 have that v1.9 hasn't?
b) what regressions are currently known?
c) what goals has OW v2.0 achieved so far?

[pchapin]

Thanks for your thoughts.

I disagree with your assertion that the conversion won't help with the problem of documentation maintenance. I'm an example of someone who enjoys working with documentation and would like to contribute to maintaining ours; however, I won't touch the documentation if it remains in WGML format. Why? Because I view WGML as a dead end. I believe that time spent building expertise in it is time wasted. That said, I acknowledge that my biases are showing.

I hear you saying that because of the small size of the Open Watcom development team, we need to maintain focus and not get distracted by "the next shiny thing." That's a good point. It comes down to how critical documentation maintenance is (and how much the conversion would help with that) versus other critical needs.

I picked the OWSTL Design Documentation to convert first because it is small and simple. I am also familiar with its content. That document is something of a special case. We could abandon the WGML form of that one document even if we maintain the other WGML documents. We could disconnect the build of the OWSTL Design Documentation from the standard build process, eliminating the need to introduce a LaTeX toolchain for just one document. Since it's a niche document anyway, I doubt anyone would notice. Then, if I restart my work on OWSTL, I'll have a place that I find personally acceptable where I can document that work.

A second alternative would be to abandon both formats of that particular document and move its content to the Wiki. There is already some duplication between it and the Wiki, and obviously, having the same content in two places isn't ideal. Insofar as the OWSTL Design Document goes, this second approach is probably the best one.

[pchapin]

Today, there was a longish conversation on the Open Watcom Discord about this. I'll summarize a few points here for posterity since Discord conversations tend to scroll to oblivion quickly.

The key point I took away from the conversation is that it is essential to continue supporting the various legacy help formats that Open Watcom has traditionally supported. I agree with this. We should support users who are developing applications using Open Watcom on the same legacy systems they are targeting, and that includes viewing the documentation on those systems.

I also learned from some light research that there are more options for generating legacy documentation formats from LaTeX source than I had realized. I will investigate these options further and report back.

Another point was raised about the value of using a documentation system unique to Open Watcom and reflective of its legacy. One virtue of our current system (WGML) is that we have tooling that supports building our documents on the legacy systems we target. This is relevant if we want to support Open Watcom tool development on legacy systems, as opposed to just using Open Watcom on those systems to develop other applications. Even if a path exists that converts LaTeX source to legacy output formats, the required tools may not be available on legacy systems.

My feeling is that it shouldn't be a priority to support building Open Watcom itself on legacy systems. If we hold to that requirement, we will be needlessly restricting the power of the tools available to us. One of Open Watcom's strengths is that it supports application development on legacy systems; however, this doesn't imply that we should be so restrictive when developing the compiler and associated tools.