robots.txt rules are ignored when the file starts with a UTF-8 byte order mark #2350

Description

When a robots.txt file starts with a UTF-8 byte order mark (BOM), RobotsTxtFile.load ignores the rules of the first group. is_allowed returns True for paths the file disallows, so a crawler with respect_robots_txt_file=True crawls them.

RobotsTxtFile.load decodes the body with body.decode('utf-8'), which keeps the BOM as \ufeff. Protego then reads the first line as \ufeffUser-agent: *, which is not a User-agent directive, and drops the group. Files saved by some Windows editors start with a BOM, and Google's robots.txt parser skips it.

Steps to reproduce

from protego import Protego

body = b'\xef\xbb\xbfUser-agent: *\nDisallow: /private/\n'

print(Protego.parse(body.decode('utf-8')).can_fetch('https://example.com/private/x', '*'))      # True (current behavior)
print(Protego.parse(body.decode('utf-8-sig')).can_fetch('https://example.com/private/x', '*'))  # False (expected)

Serving the same bytes as /robots.txt and calling await RobotsTxtFile.find(url, http_client) gives is_allowed(url + '/private/page') == True.

Expected behavior

The BOM is ignored and /private/ is disallowed.

Proposed fix

Decode the body with utf-8-sig in RobotsTxtFile.load.