Respect flowed format (missing trailing spaces)

[VincentCordobes]

Description

I noticed trailing spaces are removed. They are important as they mean softwrap when writing Content-Type: text/plain; format=flowed, which is the default for plain text in thunderbird.
The consequence is that the text is not flowed.

Note trailing spaces are correctly preserved on quoted lines

Environment

• OS: Linux
• Thunderbird version: 102.0b8
• ExtEditorR extension version: 0.3.0
• ExtEditorR host version: v0.3.0

Configuration

Editor: neovim configured for format=flowed emails
shell: zsh
terminal: kitty

[Frederick888]

Could you share a your TB configs and a few screenshots? An example email can be handy too. The fix should be simple but I'm not sure about the expected outcome since I don't use flowed format usually.

I just reset wrap/flowed settings:

name	value
mail.wrap_long_lines	true
mailnews.wraplength	72
plain_text.wrap_long_lines	true
mailnews.display.disable_format_flowed_support	false
mailnews.send_plaintext_flowed	true

And now my emails also have that format=flowed header. So IIUC this should allow recipients' email clients to rewrap my email visually, despite having hard line wraps, based on their devices' viewport widths?

I tried sending a few and resizing TB's window but failed to see any differences.

[Frederick888]

The fix should be #49. would be nice if you can test it out (and let me know how this actually makes a difference...).

[VincentCordobes]

Thanks. I just tested #49 and unfortunately it does not fix it :(
I feel like TB reformat the text after receiving it from external-editor-revived.

My configuration

name	value
mail.wrap_long_lines	true
mailnews.wraplength	72
plain_text.wrap_long_lines	true
mailnews.display.disable_format_flowed_support	false
mailnews.send_plaintext_flowed	true

Here is an example of what is expected

This very very long line contains a trailing space right after this word 
and it should certainly be wrapped and flowed.

The message contains a trailing space after word, represented by the - character in my editor.
I used mutt and vim, but the behaviour is the same when you type a long line in TB editor. The source mail should contain the trailing space.

After saving it in drafts, it should be reflowed like this:

On small screen :

Also on my phone mail app:

[Frederick888]

I did some test and it seems that you have to physically type into TB editor for it to preserve trailing spaces and make the email 'flow'.

If you just copy the text from e.g. the preformatted block you just shared and paste it into TB, TB'll remove trailing spaces:

And the sent email won't flow:

So I guess the compose API I use works in the same way as how pasting from clipboard works, and I'm afraid this is not something I can fix on my end.

PS: I'll merge #49 anyway since it's still a sensible change.

[Frederick888]

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1780196.

[jobisoft]

Hi,

I would like to discuss how this should be solved. I think I reach more people here, then in the bug.

What currently happens: Every line break in a string used in setComposeDetails(<tabId>, {plainTextBody: <string>}) is interpreted as a hard line break. Even if the line ends with a space.

If the user has enabled flowed output and a certain wrap limit, the composer should be in charge of "flowing" the text. If you use a very long string, and the user has requested flowed output, the editor will correctly flow the text from the following commands:

let [{ id }] = await browser.tabs.query({type:"messageCompose"});
await browser.compose.setComposeDetails(id, {plainTextBody: "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet."})

What should happen if the input data already has flowed content, and maybe even the wrong wrap limit?

I see three options:

1. Require the add-on to provide clean unflowed data (the add-on has to remove \n from the input data)
2. Let setComposeDetails clean up the input data
3. Let setComposeDetails clean up the input data only if the composer is set to use flowed output

Option 2 has the risk of doing something automatically, which may be undesired. Maybe the user wanted to have a space?
Option 3 may have the risk of not being deterministic, because the add-on does not know if the composer is using flowed output.

Note: getComposeDetails always returns unflowed data.

[jobisoft]

With option 3, the composer will correctly reflow any flowed input, and keep the soft line breaks (without the extra space), if the user has disabled flowed output.

This will honor what the user typed into the external editor, something which is not possible with option 1. Except if we expose a property PLAINTEXT_FLOWED, which allows the add-on to check the current flow setting.

To minimize automatic adjustments to input data, which could have unwanted side effects for other add-ons, I currently would opt for option 1 and adding a PLAINTEXT_FLOWED property.

[Frederick888]

@jobisoft Thanks for reaching out! It's getting late here where I live, and as mentioned above I don't usually use flowed format, so I'll need a little time to think it over.

But it seems that all three options involve sanitising the whole email body when flowed output is on, then leaving the flowing work to the composer. May I know, putting Thunderbird composer's behaviour aside, whether flowed format itself can support mixed formats in one email? For example,

This is a long paragraph where flowed format should apply. Pellentesque⌴
pellentesque lobortis ligula nec pellentesque. In placerat dui sit amet⌴
pretium egestas.

Now some code:

fn very_very_very_super_super_super_tremendously_surprisingly_long_long_name() -> u32 {  // don't wrap this line!
    // hello, world!
}

PS: I'll be AFK soon. No rush to get back to me :)

[jobisoft]

Flowed output was invented to be able to write nice looking emails and follow the rules of SMTP, which require line breaks after a certain amount of chars.

So in the early days, where people read the actual mail which came in over the wire (like on the server), they got used to this hard requirement to have line breaks and a width of 72 chars per line. On many modern clients that looks strange, because the available space is not used.

This was fixed by flowed, quoted printable or html. For html, line breaks do not matter at all, but you need an html capable client to read the mail.

With quoted printable, the message basically gets encoded. It can still be read as-is, but it is not fun.

Flowed adds soft line breaks, which are inserted by the client before the message is sent, to comply with the SMTP protocol. The receiving client can remove these soft line breaks again and use the available space to display the message (a.k.a. reflow).

So you need to switch the perspective. Not the user is adding these soft line breaks, but the client. The above message will get soft line breaks in the line which should not be wrapped. And the receiving client will remove them again (when it is showing the message, not in the source of the message).

For your add-on that means: Do not include soft line breaks in the data which you are using in setComposeDetails(). The editor will take care of that.

What could be interpreted as a bug: The editor does not understand soft line breaks when the content is added. It interprets them has hard line breaks. One could assume that the editor should remove them and reflow the text. But I do not think we can change that, as that would have a lot of consequences for other unrelated scenarios. We could clean-up the input data just for setComposeDetails(), but as I said, I do not want to temper with input data. My current proposal is that the add-on should not use soft line breaks, which are supposed to be added by the client just before sending.

What you need to ask your users: What is their goal, when they provide an input text with soft line breaks?

• If Thunderbird is set to use flowed output, they should be ignored and Thunderbird should do its thing.
• If Thunderbird is NOT set to use flowed output, the message will have hard SMTP enforced line breaks. Adding soft line breaks is not sensible, since the receiving client will not remove them, as the mail is not declaring flowed output in its header.

So after re-thinking this after writing that paragraph: Exposing PLAINTEXT_FLOWED will probably not help you in any way. Your add-on should simply ignore soft line breaks with body.replaceAll(" \n", " ");