Concerning the future of this project

[vkbo]

It's been nearly 7 years since I first started working on novelWriter, and by now it is a reasonably feature complete piece of software for writing novels. Sure, there are many features that can and will be added, but as a tool for writing novels using plain text, I think it is more than usable for the task. I use it myself for all my projects.

Where it is lacking is in taking advantage of the meta data it holds to provide analyses of your projects. The recent addition of story structure comments, originally proposed in #1769, adds to this.

In parallel with novelWriter, I've been working on a sister app that is a rich text editor. That project has been abandoned and restarted countless times because I've been experimenting with both similar and very different concepts than novelWriter's. I think now that I've arrived at making it as similar to novelWriter as reasonable, around a rich text editor instead. I will divide my time between the two code bases moving forward.

The rich text app is named Collett (after a Norwegian author). It is actually simpler to develop in the sense that it doesn't need any of the complex formatting features and conversions needed in novelWriter since the formatting is applied directly to the text. The Collett code base is written in C++ for better performance, since I want to customise a lot of the basic features in the Qt library.

One issue this raises is that I don't really want to invest a lot of time implementing graphical story analysis tools in novelWriter. I do plan to revamp the Outline View, but maybe not much beyond that. I'm considering instead to make an analysis tool as a completely standalone companion app to both novelWriter and Collett. That way I don't have to implement this twice, and each app can focus more on the writing process itself.

I haven't landed on the technology for such a tool, but I'm leaning towards QML-based with either a C++ or Python backend. Probably a mix of both with C++ for the active backend and Python for project serialisation/communication since data processing is much simpler in Python than in C++.

I'm making this post both to hear what the usual suspects here think, and to check if there are people who'd want to contribute to any of these projects in one way or another.

[xahodo]

Well, if Collett would be 100% compatible with or is able to import NovelWriter's projects, I'm all for it. I'm willing to test, provide bug reports, and demand 😄 features. Is it on Launchpad?

Python has its limitations - notably performance. C++ might be a way to implement features which you've refused because they would be too much of a performance hit, or were only able to implement in a way which isn't optimal, because of, again, performance.

While you've been able to deal with python's performance issues in such a way I never managed to notice them, I can imagine continuing work with Collett might be a relief and would allow you to implement some features (fully) which would simply cripple NW's performance. I'm all for it.

That being said... would it be an option to rebuild those parts where performance really matters - such as when the user is typing, and widgets you want to rewrite - in C++ and leave the rest in python? Or would this complicate matters unreasonably?

Regarding the shared code: how about putting it in a library instead of making it a separate application? That way both NW and Collett can link it and have the desired features. That way people wouldn't need to change applications for story analysis.

[vkbo]

Running novelWriter in Python is mostly fine because all the heavy rendering parts are handled by the Qt library, not in Python. Processing text in Python is very fast as well as a lot of it is done in the underlying CPython code, which is written i C. It's mostly a matter of knowing the limitations of Python and how to work around them.

The issue with Python is that I cannot alter the functionality of the standard widgets too deeply. That's where I run into performance issues because data has to be passed between Python and C++ a lot. I also run into racing issues between the Python and Qt garbage collectors, sometimes resulting in SegFaults.

As for using a library instead of an external app, I don't see the benefit. The analysis would anyway be done in a separate window, and the user don't really need to know nor care if the window is internal to novelWriter or another app. Integrating it as a library also introduces all the issues of processing data between Python and C++.

[peter88213]

Does “rich text app” mean a move away from markup input and display, or is it more about dialog highlighting and displaying spelling errors? I assume that the peculiar, IDE-like way of working with metadata inserted into the content is preserved, isn't it?

I'm considering instead to make an analysis tool as a completely standalone companion app to both novelWriter and Collett.

I have some experience with this kind of thing with yWriter (which is proprietary and closed source, but has a workable XML file format). All kinds of analysis, editing, and conversion tools can be implemented as Python scripts or applications, also in the form of LibreOffice extensions. In my case, the basis was a self-written Python library with modules for reading and writing the yWriter file format.

Based on this, I was able to realize lightweight applications with a tkinter GUI, for example a relationship grid with editing option

and a timeline viewer

Not for yWriter, but for my own program, I have a word count statistics visualization that is based on a derived data model.

A data model completely with Python data types is essential here, so the file read/write methods can also be included with LibreOffice macros, for example. I'm not showing this because I want to do something similar for novelWriter, but to illustrate the potential of the open source community. Experience has shown that this is not so promising for writing programs because the circle of users does not overlap much with that of programmers. However, novelWriter may have a better chance of attracting computer-savvy writers as it reflects the way programmers work.

If novelWriter is to be open to such contributions from third parties, then a library that is not interwoven with the GUI framework would be helpful. I have already provided one or two Python tools for novelWriter (does anyone use them at all?). In doing so, I have reconstructed the novelWriter read and write operations and adapted them to the yWriter data model. Of course, this is not very future-proof, and I don't plan to keep the Python packages up to date.

As practical as the novelWriter file format may be for users, the mixture of plain text, XML, JSON and markup/YAML files is quite cumbersome for programmers to handle. That's why I think, regarding the community idea, of course, a centrally maintained library would be essential.

[peter88213]

If novelWriter is to be open to such contributions from third parties.

That was not my intention. I simply wanted to have a shared analysis tool between novelWriter and Collett, instead of implementing the same functionality twice with two different languages. I'm looking for collaborators because it's a lot of work. These tools would require more attention to standardising the interchange file format of course.

That is something else, of course. In this case, a lot of code from one of the two application variants could be reused, and the dependency on the qt framework need not be a disadvantage.
I had more of an ecosystem in mind, which is held together by the file format. But who actually wants this?

As practical as the novelWriter file format may be for users, the mixture of plain text, XML, JSON and markup/YAML files is quite cumbersome for programmers to handle.

There is no YAML in the project. The only structured data formats are the single project XML file and the meta data in JSON. The data needed for analysis would be the index.json file. The ProjectXMLReader in the novelWriter source converts the entire project XML into a list of Python dictionaries. Project settings are populated into a data class.

Yes, that is certainly correct: The tag reference inserts in the .nwd files are not true YAML, just as the markup is not true Markdown. Nevertheless, it reminded me of the kind of combination that is currently being made popular e.g. with Obsidian.

The .nwx files are no problem at all - an XML structure can easily be converted into an object model, and it is also quickly defined via a DTD and can therefore be validated.
However, the essential relationships between text content and metadata are contained in the .nwd files. If the file format has not changed significantly since I figured it out, it will not go without a kind of proprietary parser.

[xahodo]

I have already provided one or two Python tools for novelWriter (does anyone use them at all?).

@peter88213 I was not aware of these tools. Could they, perhaps, be provided through the website of novelWriter or integrated in the repository, so everybody gets them?

[peter88213]

Here is a list of links. Note, however, that I am no longer developing these projects, which were more of a proof of concept for me when I evaluated novelWriter.

• nw_metadata Creates a cross reference table with novelWriter metadata.
• mm2nw Generates a novelWriter project from a FreeMind/Freeplane outline.
• yw2nw Converts between yWriter and novelWriter.
• md2nw Generates a novelWriter project from a work in progress written with any text editor or Markdown word processor.
• odt2nw Generates a novelWriter project from a work in progress written with e.g. LibreOffice.

The novelWriter format that the scripts read may be outdated by now, so they no longer work. Therefore, it might not be advisable to link the projects “officially”, especially since the “ecosystem” approach is not actually intended to be followed.

[vkbo]

There is no YAML in the project. The only structured data formats are the single project XML file and the meta data in JSON. The data needed for analysis would be the index.json file. The ProjectXMLReader in the novelWriter source converts the entire project XML into a list of Python dictionaries. Project settings are populated into a data class.

Yes, that is certainly correct: The tag reference inserts in the .nwd files are not true YAML, just as the markup is not true Markdown. Nevertheless, it reminded me of the kind of combination that is currently being made popular e.g. with Obsidian.

Do you mean the meta data at the top of the file, or the markup/meta syntax of the text body? Yeah, the parsing of the latter is handled by the indexer, which caches the content to an index.json file. Of course, I don't provide backwards compatibility for the index file. If it fails validation, it is just regenerated.

Everything needed to handle the core data objects on novelWriter are in the core folder. That includes an XML parser that is compatible all the way to the very first file format. The indexer class is also always up to date with the latest features of course.

[vkbo]

I have already provided one or two Python tools for novelWriter (does anyone use them at all?).

@peter88213 I was not aware of these tools. Could they, perhaps, be provided through the website of novelWriter or integrated in the repository, so everybody gets them?

They are mentioned here: https://novelwriter.io/docs/usage/howto.html#other-tools

[Survivian]

I'm onboard with the idea for a separate program to handle the analysis. My one concern is mainly that with a separate program, it might introduce some friction in the process. Though it doesn't exist yet, my two cents is that it would be cool if there was a path setting in NW and Collett that could be set to the analysis program so it could be automatically brought up with a hotkey or UI button.

The most I'd be able to contribute would be issue reports and feedback as I'm currently in the process of porting a lot of my ongoing fanfic writing into NW and running analysis on that. Maybe that's a unique enough perspective, I dunno.

[peter88213]

My one concern is mainly that with a separate program, it might introduce some friction in the process.

As long as these are read-only analysis tools, this should not be a problem.
Once the files are changed externally, a locking mechanism would have to be introduced.

it would be cool if there was a path setting in NW and Collett that could be set to the analysis program so it could be automatically brought up with a hotkey or UI button.

That sounds like a good idea. On the other hand, novelWriter doesn't claim to be an all-in-one solution, but rather to cover part of the work chain up to the completion of a manuscript. You could also do it the other way around and create an integration environment that embeds novelWriter, as well as backup or version management, and the subsequent steps towards publishing. An analysis tool could fit in seamlessly. An example of such an integration environment would be the commercial Writer's Café.

[vkbo]

My one concern is mainly that with a separate program, it might introduce some friction in the process.

I don't foresee any major issues here. novelWriter would just try to call the external program and pop a dialog if it cannot be called. Either it will need to be installed, or if the user is running an AppImage, set the call path to it in settings. A default could be to look for the AppImage in the same folder as well before asking.

When it is callable, some form of hook can be established, which can be as basic as monitoring the change time stamp on the index.json file (or whatever data file is used). I intend for the visualisation tool to only read from novelWriter/Collett, so sync is not an issue.

[HeyMyian]

I had a hostile encounter with Scrivener's rich text editor, which is why I kept researching other writing software and ended up using novelWriter. If not for novelWriter, I would probably write in VSCode. I personally am in no need of a rich text editor anymore (well, that's my opinion now 😄), but I'll be following the development of Collett, curious about and trusting in a solid rich text implementation.

I could contribute to Collett the same as for novelWriter - German translation, and colors if it needs any.

[vkbo]

Yeah, I also disliked the Scrivener editor window quite a lot.

Regardless, I intend to keep the editor itself fairly simple, with basically the same formatting as in novelWriter, but without the need for the formatting codes. I've already created a good JSON based file format for the storage of the text. It's simple enough that it wouldn't be hard to write a small Python script to extract the text into a plain text file if one wants to.