Feature request : big attachment vs concurrent parsing

[mgarbin]

Hi @mnako ,
good job, i have a question for a big problem that affects lot of mail parser projects, did you try to parse big attachments in a project that analyze message with concurrent jobs ?
I see that you used ioutil.ReadAll in all decoders function, this works well if you are going to parse file with 10Mb to 30Mb but if you try it with 50+ Mb file size this program will eat all the os resource.
I tried this way in a past project, garbage collector offered by golang is good but if you are going to parse concurrently ( 10 to 100 message per second ), it will kill your os.

I suggest you to implement a different way : https://kgrz.io/reading-files-in-go-an-overview.html

[mnako]

Hi @mgarbin , and thank you for the message.

That is a very good suggestion. I still need to fix some known bugs (like the parser crashing if there is no plaintext) but I will get back to that idea as soon as we hit a stable 1.0 release.

[rorycl]

Hi @Manako, @mgarbin

I submitted PR 116 to switch the InlineFile and AttachedFile structs to use io.Readers rather than []bytes. The simple effect for users was intended to be:

@@ -564,11 +565,11 @@ type InlineFile struct {
        ContentID          string
        ContentType        ContentTypeHeader
        ContentDisposition ContentDispositionHeader
-       Data               []byte
+       DataReader         io.Reader
 }
 

However I made a rather foolish error because the as the multipart.NewReader moves between parts of the email, the previous io.Reader becomes nil, rendering the contents inaccessible.

I guess a way around this might be to store file Seek positions for the start and end of each attachment, but that would require two passes: one to determine the seek positions, and another to read out the data. One nice thing about this is the attachment size could be determined before processing.

An alternative possibility is to choose to pass a writer (for example, a func that directly writes emails out to file) to the decodeContent function so that efficient processing could happen without reading the file into an intermediary buffer. That would allow one to pass in a func that does some sort of writing to a stream and/or chooses to abandon processing if the attachment proves too big. I'll give this writer func idea some thought and consider making another PR.

Apologies for taking up your time.

[rorycl]

I've now got a version of using io.Readers instead of []byte allocations working in my nobuffer branch and tests are running ok.

The branch is scruffy since there isn't anywhere in letters at present to attach a default or custom func to process files. As discussed in #118, however, the general idea is that there might be one or two built-ins for processing files (as bytes, or saved to a directory), but users could do their own consumption of the io.Reader for each inline or attached file, possibly in parallel, opening up very efficient ways of streaming data to the network or databases (etc.).

[rorycl]

The feature of custom processing of files is now fully implemented in my fork of letters as you can see here.

An example using this in the mailfinder cli tool is as follows:

// only consider attachments with a "text" content disposition
onlyProcessTextAttachments := func(fe *email.File) error {
    if !strings.HasPrefix(fe.ContentInfo.Type, "text/") {
        return nil 
    }   
    var err error
    fe.Data, err = io.ReadAll(fe.Reader)
    return err 
}

emailParser := letters.NewParser(
    parser.WithCustomFileFunc(
        onlyProcessTextAttachments,
    ),  

You will see also a (crude) Opt to similarly provide to the parser to save files directly to disk, bypassing reading the files first into a buffer by using only io.Readers and io.Copy.

I hope this approach provides some thoughts for the future development of letters.