Is `as-records` streambased?

[TobiasNx]

We are not sure if as-records is stream based when splitting batch input into records. If file is big and no records separator is provided in the batch input, this might cause memory problems.

[TobiasNx]

@dr0i could it be that #691 was the issue and not as-records and that as-records is actually streambased?

[dr0i]

Sure is as-records streambased. We weren't just not sure that the definition of what a record is were set correctly. If not , the whole input would be "cached" first. E.g. open a 1 GB file with open-file without using the separator parameter, which would use the default one ('\u001d') . If this character is not found the stream is loaded to the end.

[TobiasNx]

Okay, but did we confirm that this "caching" is happening?

[dr0i]

Sure, it`s in the code:

public void process(final Reader reader) {
        assert !isClosed();

        try {
            boolean nothingRead = true;
            int size;
            while ((size = reader.read(buffer)) != -1) {
                nothingRead = false;
                int offset = 0;
                for (int i = 0; i < size; ++i) {
                    if (buffer[i] == separator) {
                        builder.append(buffer, offset, i - offset);
                        offset = i + 1;
                        emitRecord();
                    }
                }
                builder.append(buffer, offset, size - offset);
            }
            if (!nothingRead) {
                emitRecord();
            }

        }

[TobiasNx]

Okay, then perhaps there is a way to get rid of the buffering/caching while keeping the function of identifying distinct records?

[dr0i]

I think you have not understood how it works. Let us talk someday about it.