Brainstorming on design improvements/issues for the initial stable release

[qexat]

I started Ansifmt out of frustration with OCaml's pretty-printing capabilities, and I can't be satisfied with basic string rendering without some nice colored output. Specifically, I wanted to formatting term languages, which are inherently recursive structures, making it difficult to add ANSI escape sequences by string-patching. I happen to have some experience with ANSI escape sequences, especially the Select Graphics Rendition ones ; in fact, I am the author of the Python library Coquille. Thanks to that, I wrote without thinking much what I needed to handle ANSI stuff, and no dependencies involved :)

Originally, my mental model was to essentially do the opposite of parsing, that is transforming my term tree into a stream of tokens. This is the current main branch (as of January 30th). This worked fine, but I noticed that to handle parentheses and such, I tended to rewrite similar code over and over.

Thus, I tried another approach (which corresponds to the experimental-token-tree branch). Instead of a stream of tokens, the structure is a tree, where the leafs are what was formerly the token list, and nodes are lists of leafs or other nodes. When rendering, nodes would be automatically parenthesized. While the idea sounded simple and good on paper, the result was more verbose code for the same amount of control over the output, and with a frustrating handling of parentheses.

This leads us to the current design (found on the experimental-token-term branch). It is similar to the other experiment, but with an additional Parenthesized node which alone makes, in my opinion, writing the tokenize function more pleasing.

Now that I have reached a state for tokenization that satisfies me, I think I want to focus on adding the missing ANSI escape sequences for now.

Pinging @chshersh as you requested :)

[chshersh]

@qexat Ping received 😅

Waiting for the TODO to be filled in!

[qexat]

By the way, I'm also interested in your thoughts license-wise. Ansifmt is currently MIT by default, but if you convince me of a better choice I'm fine with changing :P

[chshersh]

MIT is the most permissive licence, meaning that everyone can copy your code inside their libraries and change it or use it however they want.

In the OSS world, not much you can do about it anyway, no matter the license.

I'm using MPL-2.0 in some projects because it requires people to copy the entire original source code if they want to modify it. Kinda like an extra hassle to motivate people to contribute more instead of just copying the code. Also, people will most likely make it worse because my code is perfect, so at least all the readers will be able to compare.

But recently I enjoyed the freedom of the MIT permissiveness and the ability to just copy the code from libraries. OSS maintainers do their job in free time, and not everyone wants to spend their free time addressing issues and working with contributors. So they might just let everyone do what they want with the code.

[qexat]

Fair enough, I guess I'll stick to MIT then

[chshersh]

Hey @qexat,

I've now seen your update about the design and I checked the experimental-token-term branch. I like the current state of the library and your design decisions. It looks really interesting and helpful even for my use case!

Now that I have reached a state for tokenization that satisfies me, I think I want to focus on adding the missing ANSI escape sequences for now.

Could you elaborate a bit more on this? I checked this module and it seems to be supporting quite a lot of ANSI escape sequences already:

• https://github.com/qexat/ansifmt/blob/experimental/token-tree/lib/Color.ml

[qexat]

Hi, glad you like it!

While I won't implement them all, it's quite a small set compared to Coquille's: https://github.com/qexat/Coquille/blob/main/src/coquille/sequences.pyi

[chshersh]

Ah, I see. Yes, cursor sequences alone can occupy quite a lot of time. Not sure they're required for ANSI formatting and colouring but maybe you had a different idea in mind.

I have a few thoughts to add to this brainstorming discussion but it'll be quite a long comment with my ideas. I'll share my thoughts in detail as soon as I can!

[qexat]

Ah, I see. Yes, cursor sequences alone can occupy quite a lot of time. Not sure they're required for ANSI formatting and colouring but maybe you had a different idea in mind.

Honestly, I'm undecided. You get to choose!

I have a few thoughts to add to this brainstorming discussion but it'll be quite a long comment with my ideas. I'll share my thoughts in detail as soon as I can!

Sure! No pressure :)

[chshersh]

I'll start by writing my use case in GitHub TUI. Programs should solve problems. And if we have vastly different problems that it's impossible to unify them under a single design, making this library too generic can hurt its goals. So let's make sure we align.

In GitHub TUI, I have the following use cases for ANSI escape sequences:

1. Colouring text and box symbols, like here.
   • I think Styling.wrap should be just a drop-in replacement for my case

   • ansifmt/lib/Styling.mli

     Line 31 in 47c610a

     val wrap : contents:string -> t -> string

2. Predefined styles for common cases, like here
   • Again, Styling.create should be a drop-in replacement here

   • ansifmt/lib/Styling.mli

     Lines 13 to 23 in 47c610a

     	(** [create ?foreground ?background ?bold ?dim ?italic ?underlined ()]
     	creates a new style object given the provided configuration. *)
     	val create
     	: ?foreground:Color.t
     	-> ?background:Color.t
     	-> ?bold:bool
     	-> ?dim:bool
     	-> ?italic:bool
     	-> ?underlined:bool
     	-> unit
     	-> t

3. True RGB colours, like here. I'm fetching hex colour codes for GitHub labels and I'd love to display them in terminal using exactly the same codes. I'm currently having my own tiny Color module to parse hex codes and display ANSI sequences.
   • Looks like Ansifmt supports true RGB colours but for now I still need to parse hex codes in my app

   • ansifmt/lib/Color.ml

     Lines 271 to 277 in 47c610a

     	| Rgb (r, g, b) ->
     	Printf.sprintf
     	"%d8;2;%d;%d;%d"
     	(Ground.to_int ground)
     	(Int8.to_int r)
     	(Int8.to_int g)
     	(Int8.to_int b)

4. Luminescence calculation, like here
   • This is needed to display either white or black text for label based on its colour
5. Calculating the length of ANSI-escaped strings. Currently I'm using the OCaml library shape-the-term to do this. This is needed so I can calculate the true displayed width of output characters and calculate the remaining free space. In theory, it's possible to write code this way so I don't need to care about it at all: all formatting will just be applied later. But I have a feeling it'll be too fragile. But let's see.
6. Composing styles, like here.
   • I'm using ANSITerminal, and there styles is just a list of values. So it's pretty easy to compose.
   • But I like Ansifmt design more. It avoids problems with duplicating the same style multiple times.
   • However, some composability utilities would be nice to have

I have other use cases in mind that would be nice to have but I haven't implemented them yet. So I'll describe them separately, since this message is already pretty big.

[qexat]

1. As you mentioned, Ansifmt already supports this.
2. Same as 1.
3. I think it is fair to provide some parsing utilities in the Color module!
4. This would just be a function in the Color module as well.
5. I happen to also have made a super small Python library for that: anstrip. However, it works with regular expressions, meaning bringing at least one dependency to Ansifmt. I'd like your opinion on this.
6. I absolutely agree about adding composability utilities, we have to figure out which ones are the most occurring/useful.

So I'll describe them separately, since this message is already pretty big.
Sure!

[chshersh]

3.I think it is fair to provide some parsing utilities in the Color module!
4.This would just be a function in the Color module as well.
6.I absolutely agree about adding composability utilities, we have to figure out which ones are the most occurring/useful.

If we both agree that these are good ideas to add to the library, would you be okay with me creating three separate issues for each of these three asks and describing my implementation ideas under the corresponding issues so we won't discuss anything in one thread?

[chshersh]

I happen to also have made a super small Python library for that: anstrip. However, it works with regular expressions, meaning bringing at least one dependency to Ansifmt. I'd like your opinion on this.

If you mean bringing the str OCaml dependency then I'm fine with it. It's probably already in one of the transitive dependencies anyway, so I don't see any harm in adding it.

[qexat]

If we both agree that these are good ideas to add to the library, would you be okay with me creating three separate issues for each of these three asks and describing my implementation ideas under the corresponding issues so we won't discuss anything in one thread?

Of course!

[chshersh]

Of course!

Issues are created!

[chshersh]

I'll describe in a separate message my future use cases in GitHub TUI, so we can discuss whether they make sense.

1. I want to fully format Markdown with ANSI sequences. I'm going to use cmarkit for parsing markdown and then some magic to format nodes. This is probably the closest complex use case to your original term-language use case.
2. I'm dumping lots of text on every frame. Ideally, it would be nice if the output ANSI sequences wouldn't be more bloated than necessary. Premature optimisation is the root of all evil, so maybe it won't be a problem at all. Just thinking in advance.
3. This one is probably out of scope for Ansifmt but I'd like to format unicode box drawing characters as bold. Unfortunately, the Bold ANSI sequence doesn't work, I have to use different characters like ┏━┓ instead of ┌─┐. I guess, for this I only need to be able to introspect the formatting and looks like I already can do that:

   • ansifmt/lib/Styling.mli

     Lines 1 to 8 in efcc6ea

     	type t =
     	{ foreground : Color.t option
     	; background : Color.t option
     	; bold : bool
     	; dim : bool
     	; italic : bool
     	; underlined : bool
     	}

[qexat]

1. Neat that you have a use case close to mine :)
2. I understand your concern, and from my experience with dealing with ANSI escape sequences in Python, flickering issues did happen at first. However, OCaml is far more performant, so it's hard to tell. I propose that we deal with that once we have actual, reproducible evidence of said issue (although I think it might happen).
3. It's a nice to have but yeah, I think it's out of scope for this case specifically. I made the decision of leaving the record fields of stylings exposed because I think hiding it brings more cons than pros. That said, perhaps we could have a function that allows the Ansifmt user (so you) to customize some styling behavior. Honestly, I don't have many ideas on how it would concretize.

[qexat]

Addendum to 2: Ansifmt already does some effort to avoid unnecessary escape sequences - for example, it doesn't add resetters for things that aren't set (that is, if bold is false, it's not gonna bother to add bold reset at the end).

[chshersh]

This is all great news to hear!

I think, when issues #2, #3 and #4 are solved, I'll be able to fully switch to Ansifmt without losing any features and with getting lots of benefits!

[qexat]

Ansifmt 0.2.0 is in the process of being released: ocaml/opam-repository#27357

This update brings your efforts from #5, #6 and #7 as a release installable with opam!

[qexat]

Since we reached a milestone, I'll take some time to clean up the repository a bit, adding better issue labels, using the milestone feature etc

[qexat]

Since we reached a milestone, I'll take some time to clean up the repository a bit, adding better issue labels, using the milestone feature etc

I added new issue labels, milestones, CI, funding and a social banner so far.

[chshersh]

Great stuff! Love the labels. They're self-explanatory, and this is great.

I'm in the process of migrating GitHub TUI to Ansifmt. I plan to open a PR soon. I discovered a few potential improvements to Ansifmt in the process, so I'll open issues for them as well once I'm done 🙂

Non-blockers, just nice to have.

[qexat]

I'm in the process of migrating GitHub TUI to Ansifmt. I plan to open a PR soon. I discovered a few potential improvements to Ansifmt in the process, so I'll open issues for them as well once I'm done 🙂

Non-blockers, just nice to have.

Sure! Having feedback from a project like github-tui is really valuable for the library to improve itself!

[chshersh]

@qexat Sounds great!

I'll now try switching GitHub TUI to Ansifmt (it might take some time). I learned a lot about this library and potential design space, so this collaboration experience was valuable for me!

If I notice something I missed during the initial design phase, I'll create new issues in the process, and we can discuss in detail all future developments under corresponding issues if anything comes up.

[qexat]

Sure! Thank you again for your valuable contributions!

[chshersh]

@qexat I finally finished the migration and it was surprisingly easy! Took me only half an hour to do the job. Have a look and leave your comments:

• [#39] Move to Ansifmt chshersh/github-tui#46

[qexat]

Ansifmt Formatting's design was great for the simple cases I had encountered so far.

However, I have started to use it for a more complex project, and while it works nicely for the most part, I am facing a few pain points:

• It is not obvious how to indent a Block.
• There is no simple way to have a multi-line Block in a way that doesn't have a trailing newline that needs to be stripped away when printing (using ~line_end:"" on print_formatted).

For the second point, it used to be easy to do with string-based formatting - it was just about using String.concat "\n". Unfortunately, there is no equivalent in the Ansifmt formatting language.

Ideally, no further changes would need to be made on the formatting tree type as in introducing new variants, but it would be definitely useful to add some information (like indentation and which token to join the subtrees in the list on) to the Block one.

To be honest though, I am not fully satisfied with the formatting language design. Or at least, it feels a bit bare-bones while still being a bit cumbersome and not very intuitive - it is only relatively easy to use for me as I am very familiar with the material, but I don't think it would be the same for someone that would try this part of Ansifmt for the first time.

Seeing Printf.sprintf, I was wondering how costly it would be to have a similar interface that would convert to the more "low-level" formatting language (the one existing today in Formatting.Tree). If too much of Ansifmt's claimed lightweightness had to be sacrificed, I would consider moving that part of Ansifmt to a separate library that would depend on this one.

Sorry for the long text wall. I know this isn't about the part of Ansifmt that interests you the most, but I'd be delighted to hear your opinion on this.

[chshersh]

Apologies for taking some time to reply!

I'm not sure I fully understood your challenge, so I'll try to ask some clarifying questions.

I believe, you're talking about formatting this type:

type t =
  | Simple of Token.t list
  | Parenthesized of t
  | Block of t list

As I see from the Ansifmt code, the Tree.format function is responsible for formatting this type. Looks like currently Ansifmt can only colour individual tokens (correct me if I'm wrong) but now you also want extra abilities to indent a block. So this is something new.

I would love to understand the difference between Simple and Block constructors and what is the purpose of a block. Is block supposed to have list of elements where each list element is on a separate line?

My understanding is that you have a separate Line_break constructor for tracking line breaks, so therefore you don't have direct access to how many lines are in the block or even Simple case without traversing the tree first.

Maybe if you could share more about your use case, I can try to suggest if I had experience with something similar?

---

Generally, I find it easier to follow Data-Oriented Design (DOD) and tracking potential modifications via separate constructors. In GitHub TUI, I don't track line explicitly via my AST, they're implicit via the Vertical constructor.

type t =
  | Str of Style.t * string
  | Horizontal_fill of string
  | Vertical of t list
  | Horizontal of t list
  | With_size of (width:int -> height:int -> t)

Without having a lot of context about your use case and design goals, if I were to come up with a type that should represent "a stream of tokens" or even "a tree of tokens", I would try something like that:

type t =
  | Line of Token.t list (* all tokens within the same line *)
  | Parenthesized of t
  | Vertical of t list (* 't' are supposed to be joined with String.concat "\n" here *)
  | Indent of int * t  (* the indentation applied to a whole subtree *)

Again, this might not entirely work, especially if you have the second requirement about being able to specify line_break for a block. I usually first ask if this problem is worth solving. And if yes, I'd continue with expanding the data type to track this info on the value level inside the Vertical case, for example:

type line_end = No_end | Yes_end

type t =
   | ...
   | Vertical of line_end * t list

[qexat]

Is block supposed to have list of elements where each list element is on a separate line?

Not really. Block is meant as the concatenation of lists of tokens (or other blocks). It is useful e.g. in sequential data structures which elements are themselves tokenizable.

type t =
 | Line of Token.t list (* all tokens within the same line *)
 | Parenthesized of t
 | Vertical of t list (* 't' are supposed to be joined with String.concat "\n" here *)
 | Indent of int * t  (* the indentation applied to a whole subtree *)

This is a very interesting design, I am definitely going with something inspired by that. Thanks!

[qexat]

Made a PR to opam for Ansifmt 0.3.0: ocaml/opam-repository#27487

[qexat]

Ok I did a fresh cold build with Ansifmt 0.3.0 and it all went fine

[chshersh]

Amazing, thanks a lot, @qexat! ❤️

[qexat]

Amazing, thanks a lot, @qexat! ❤️

You are welcome! That was expected, though, I mostly worked on Formatting, I did not touch the modules that github-tui relies on

[chshersh]

You are welcome! That was expected, though, I mostly worked on Formatting, I did not touch the modules that github-tui relies on

Really appreciate your care about backwards compatibility! I know that Ansifmt is still untested and in the early development phase, so this is the time to experiment and break things. But I've been bitten by backwards incompatible changes back in the day too many times, so I became slightly more conservative here.

Anyway, generally I don't mind breaking changes and am happy to do the work. Just didn't have lots of time on my hands recently.

[qexat]

Really appreciate your care about backwards compatibility!

Well to be honest it was not really on purpose - for now I am careless about backwards compatibility in Ansifmt, but this won't hold when we reach the first stable release, and this is why I am taking the time to make sure to minimize the need for breakage once 1.0 is out