User Agent matching bug

[shivanshuzyte]

Issue:

This incorrectly allows partial/substring matches:

def applies_to(self, robotname: str) -> int:
    if self.user_agent == "*":
        return 1
    if self.user_agent in robotname:  # incorrect substring match
        return len(self.user_agent)
    return 0

Example:

"bot" matches "mybot", but it should not
"google" matches "googlebot-mobile": should match "googlebot"

Possible fix:

def applies_to(self, robotname: str) -> int:
    robotname = robotname.lower()
    ua = self.user_agent.lower()

    if ua == "*":
        return 1
    if robotname.startswith(ua):  # prefix match only
        return len(ua)

    return 0

Additionally, to be safe there should be fallback in _get_matching_rule_set in case no rule matches.