Simple format messing up the encoding

[vmandrilly]

As per screenshot.

I get something like the following, when it was supposed to be Chinese Characters...

The data is OK looking from the mailbox sending the message.
The data also seemed ok (but not certain, would need extra testing, which failed, because resolved or raw data take forever to retrieve just a few forwarded e-mails...) when using resolved or raw data formats.

I am using: "You’re on 1.100.1, which was released 3 weeks ago and is 5 versions behind the latest and greatest n8n"

I need imap node for my project, which is about being able to receive e-mails simple info (and detailed info with attachments, but upon specific request). I have been using n8n for 2 weeks now, betting it would be the best way to make the code accessible for the colleagues, and would help with debugging, and was happy so far, but this single problem makes me wonder if I just went the wrong path.

In case the screenshot doesn't work, I get something like (but on github webage, characters appears yet different again):
[
{
"textHtml":
"
\n
\n
\n
\n
\n <title>����</title>
\n <style>
\n body {
\n font-family: Arial, sans-serif;
\n margin: 0;
\n padding: 0;
\n background-color: #f5f5f5;
\n }
\n p{
\n line-height: 26px;
\n }
\n .container {
\n max-width: 750px;
\n margin: 0 auto;
\n background-color: #fff;
\n font-size: 14px;
\n box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
\n }
\n .header {
\n text-align: center;
\n }
\n .header h1 {
\n margin: 0;
\n }
\n .content {
\n padding: 0 26px;
\n color: #545A6E;
\n
\n }
\n .content h2 {
\n color: #007bff;
\n }
\n .event {
\n margin-bottom: 20px;
\n }
\n .event h3 {
\n color: #ff4d4f;
\n font-size: 14px;
\n line-height: 26px;
\n }
\n .suggestion {
\n /* padding: 10px; /
\n / border-left: 5px solid #007bff; /
\n margin-top: 20px;
\n }
\n .suggestion h4 {
\n margin: 0;
\n color: #007bff;
\n }
\n .qr-code {
\n text-align: center;
\n margin-top: 20px;
\n }
\n .qr-code img {
\n width: 150px;
\n height: 150px;
\n }
\n .footer {
\n text-align: center;
\n padding: 10px;
\n color: #545A6E;
\n padding-top: 0;
\n }
\n .download{
\n padding: 26px;
\n color: #545A6E;
\n padding-top: 0;
\n padding-bottom: 16px;
\n }
\n .c-red{
\n color: #FC2D2D;
\n }
\n .c-black{
\n color: #272E47;
\n }
\n .f-600{
\n font-weight: 600;
\n }
\n a{
\n color: #FC2D2D;
\n text-decoration: none;
\n }
\n .title{
\n font-size: 18px;
\n padding: 0 40px;
\n font-weight: 600;
\n / line-height: 28px; /
\n }
\n .logo{
\n margin-top: 20px;
\n margin-bottom: 10px;
\n }
\n .c-blue{
\n color: #007bff;
\n }
\n </style>
\n <style>
\n .risk-item{
\n margin-top: 6px;
\n }
\n .ring-container {
\n display: inline-block;
\n width: 6px;
\n height: 6px;
\n border-radius: 50%;
\n border: 2px solid #FF4D4F; / ��宽度1px����10px /
\n background: transparent;
\n margin-right: 4px;
\n margin-top: 4px;
\n / transform: translateY(1px); */
\n }
\n </style>
\n
\n
\n

\n
\n

\n

�好�

\n

\n å�ºäº�å®�å�¨è��è��ï¼�é�®ç®±ç®¡ç��å��å·²å¼�å�¯é�®ç®±æ��æ��å¯�ç �å��è�½ã��è�¥ä½¿ç�¨ç¬¬ä¸�æ�¹å®¢æ�·ç«¯ç�»å½�ï¼�å¦�outlookã��foxmailç­�ï¼�ï¼�请ç��æ��æ��æ��å¯�ç �å��å��è¿�è¡�ç�»å½�ï¼�ç��æ��æ�¹å¼�ï¼�
\n

\n

1ã��使ç�¨è´¦å�·+å¯�ç �ç�»å½�ç½�页端ï¼�ç½�æ��ä¼�ä¸�é�®ç®±-ç�»å½�å�¥å�£

\n

2���设置�账�����客�端设置������设置�

\n
\n

3ã��ç�¹å�»ç��æ��æ��æ��å¯�ç �ã��注æ��ï¼�æ��æ��ç �å¯�ç �ä»�å�ºç�°ä¸�次ï¼�请妥å��ä¿�å­�

\n
[...]

[vmandrilly]

For now, I have used AI to make a fix for my use case, but it's not pretty. Hopefully someones figures out how to solve the problem to its roots:

// This node sanitizes the message_content field from the previous step.
// It fixes character encoding corruption AND prepares the HTML for the LLM.

const allItems = $input.all();
const processedItems = [];

// --- NEW FUNCTION TO FIX CORRUPTED UTF-8 TEXT (v2) ---
const fixCorruptedUtf8 = (corruptedString) => {
// This is a more robust regex using hex codes to find the "fingerprint" of
// multi-byte UTF-8 characters that have been misinterpreted as single-byte characters.
// It looks for sequences of 2 or more characters in the high-ASCII range (128-255).
const mojibakePattern = /(?:[\xC0-\xFF][\x80-\xBF])+/;
if (!mojibakePattern.test(corruptedString)) {
// If it doesn't look like our specific problem, don't touch it.
return corruptedString;
}

try {
// 1. Re-encode the misinterpreted string back into raw bytes using 'latin1'.
const bytes = Buffer.from(corruptedString, 'latin1');
// 2. Decode those raw bytes correctly as UTF-8.
return bytes.toString('utf8');
} catch (error) {
// If anything goes wrong, return the original string to avoid crashing.
console.error(Could not fix encoding for a string. Error: ${error.message});
return corruptedString;
}
};

// --- ADVANCED SANITIZATION FUNCTION ---
const sanitizeAndFormatMessageContent = (htmlContent) => {
let content = htmlContent || '';
// 1. REMOVE UNWANTED BLOCKS ENTIRELY (Scripts, Styles)
content = content.replace(/<script[^>]>([\S\s]?)</script>/gmi, '[Script Block Removed]');
content = content.replace(/<style[^>]>([\S\s]?)</style>/gmi, '[Style Block Removed]');

// 2. IDENTIFY AND REPLACE EMBEDDED IMAGES WITH METADATA PLACEHOLDERS
content = content.replace(/src="data:image/(jpeg|png|gif);base64,[^"]+"/gi, 'src="[Embedded Image Data Removed]"');
content = content.replace(/<img[^>]+alt="([^\"]+)"[^>]>/gi, '\n[IMAGE: (alt text: $1)]\n');
content = content.replace(/<img[^>]>/gi, '\n[IMAGE: (no alt text)]\n');

// 3. PRESERVE SEMANTIC MEANING BY CONVERTING HTML TO MARKDOWN
content = content
.replace(/<br\s*/?>/gi, '\n')
.replace(/</p>/gi, '\n\n')
.replace(/

(.?)</li>/gi, '\n $1')
.replace(/<(strong|b)>(.?)</\1>/gi, '$2')
.replace(/<(em|i)>(.?)</\1>/gi, '$2');

// 4. STRIP ALL REMAINING HTML TAGS
content = content.replace(/<[^>]*>/g, '');

// 5. DECODE HTML ENTITIES
content = content
.replace(/ /g, ' ')
.replace(/&/g, '&')
.replace(/</g, '<')
.replace(/>/g, '>')
.trim();

// 6. CLEAN UP EXCESSIVE WHITESPACE
content = content.replace(/(\r\n|\n){3,}/g, '\n\n');

return content;
};

for (const item of allItems) {
const originalData = item.json;

// *** THE COMPLETE FIX ***
// 1. First, fix the character encoding on the raw content.
const fixedEncodingContent = fixCorruptedUtf8(originalData.message_content);
// 2. Then, sanitize the now-correctly-encoded HTML.
const sanitizedContent = sanitizeAndFormatMessageContent(fixedEncodingContent);

// Overwrite the message_content field with the final, clean result.
originalData.message_content = sanitizedContent;

// Recalculate token_count based on the clean content.
originalData.token_count = Math.ceil(sanitizedContent.length / 4);

processedItems.push(item);
}

return processedItems;

[vmandrilly]

I am having ANOTHER issue with this "simple format"

The whole purpose was to rapidly fetch the most important information from the mailbox. Yet it seems to fail to retrieve the NAMES and POSITION of inline images and attachments.

Any e-mail software is able to rapidly fetch the list of e-mails and the inline images/attachements numbers and names WITHOUT having to actually download the actual binary data. Please explain how to set the node to just do that...

Thanks.

[vmandrilly]

Hi! I solved the issue by using a code node, using the imap library to get the structure first, before optionally requesting body/attachments on demand. It's kind of sad that I had to basically completely bypass the imap node and reinvent the code, to get the basic e-mail info quickly, but at least it now works as needed.