bug(robots2policy): Handle multiple User-agents #857
Description
e.g. https://lwn.net/robots.txt
robots2policy -input https://lwn.net/robots.txt
- action: CHALLENGE
expression: path.startsWith("/Search")
name: robots-txt-policy-disallow-1
- action: CHALLENGE
expression: path.startsWith("/ml")
name: robots-txt-policy-disallow-2
- action: DENY
expression: userAgent.contains("ChatGLM-Spider")
name: robots-txt-policy-blacklist-3
should output more than the 3rd expr
cc @jzb