Plug-in round 2

[kristopolous]

I talked about the simplistic plugin system and it seems to work but frankly I think I've just been getting lucky.

block versus line

There's two levels this is what HTML 4.0 back in the day called block and line level elements..

Line level is something like text format such as italics or bold. It can wrap around and be in a say, a table.

Block level elements are things like the formatted code blocks, headline tags, and once again tables.

This older paradigm allows for parsing rendering and layout to be handled with the same abstractions.

Modern engines have moved away from that but markdown I think can still use it.

So that's the first one. Plugins can't just take text streams and emit. It needs to have an opinion on these two

partial stream render

To understand this best, let's pretend we are parsing bold. Here's some possibilities

This is **bold text** isn't _it lovely_

This is **bo
ld text** isn't  _it lovely_

And a true nightmare: 

This is *
*bo
ld text
** isn't  _it lovely_

In each of these, the plug-in only has opinion on the bold and it needs to pass thru essentially this every time

Take back "this is" but I'm interested in what comes after

Either 
  Ok nevermind, I actually don't care about it
  Yes but I don't have all of it so I'm keeping it

Finally 
 Alright here's your output to pass down. Also, here's some junk at the end I didn't do anything with

The current plug-in architecture I don't think can accommodate for this. I thought it could last night but I'm thinking about changing my opinion.

So the task here is really to come up with some tests and figure out how much of this needs to be done

[kristopolous]

so anyway what am I going to do here? It really sounds like the complicated solution is the only solution.

[kristopolous]

let's assume we've got block & line ... in state/ not in state and pre/post. I think that's conclusively all the fidelity needed

I mean can we question whether plugins are the right approach at all? Browsers have mostly turned away from extensibility plugins and have gone with monolithic feature design.

I guess ideally I'd do some kind of modular renderer and we are not at that state. It's this brittle sequential fall-through system

One of the issues in this approach is that the renderer sits as a black box. What if you wanted italics and bold to work within your plugin block? I mean it's just not going to. Maybe a hook system is better.

[kristopolous]

so a hook system you'd have something like

block_pre
block_post

line_pre
line_post

with the intention of clarifying these but this still puts the challenge on the implementer. line_pre gets decent markdown and you'll support line_format stuff but line_post may get erroneous parsing that you then have to strip ... it's not great.

[kristopolous]

so really it's more block/line, state/not state, {pretext (content) posttext}, pass-thru/terminate.

line_process( string, state )
block_process( string, state )

And the response should be either

• not interested (pass all through)
• interested and processing (pre)
• {pre: raw, rendered:ansi escaped, post: raw, passthru: true/false}

During processing rendered should be True.

I guess we also need to guarantee that partial words won't be sent. Essentially we'll make sure that we either see white space or a proper EOF so we aren't sending over " ` | ``javas | cript " or whatever bizarre partials.

Ideally I want something that works intuitively without having to read documentation but I don't think that's possible

[kristopolous]

also we want a way to include/exclude plugins if we genuinely want a plugin system and a way to isolate the dependencies to the plugin ... although that may be too ambitious

[kristopolous]

so there's another model I can do, something like this:

entrypoint = [begin match]

The thing is I want to be able to specify plugins dynamically at runtime

[kristopolous]

see https://github.com/day50-dev/llmehelp/blob/main/Talkers/simple-parse.py