fix: read bundled JS, c4a scripts and CLI config files as UTF-8 - #2418

Summary

A few places read text files with open() / Path.read_text() and no encoding, so they use the platform default. On Windows that is the ANSI code page (cp1252, cp1254, cp936, ...), on Linux it follows LC_ALL. The files themselves are UTF-8, so depending on the code page the read either raises or quietly returns different text.

What I saw:

• load_js_script(): the bundled snippets contain a few em dashes. Under a GBK locale load_js_script("flatten_shadow_dom") raises UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 2648. Under cp1252/cp1254 it returns the file with mojibake in two comments.
• compile_file() / C4ACompiler.compile_file() and USE includes in .c4a scripts: TYPE "你好" read from a file compiles to JS that types 浣犲ソ under a GBK locale (ä½\xa0好 on cp1254), while compile_string() on the same text gives 你好.
• load_config_file() and get_global_config() in the CLI: a --browser-config / --crawler-config / --extraction-config JSON or YAML file with non-ASCII values has the same problem.
• The Google search crawler reads its script.js the same way.

This only touches reads.

List of files changed and why

• crawl4ai/js_snippet/__init__.py: load_js_script opens the snippet as UTF-8.
• crawl4ai/script/c4ai_script.py: compile_file and USE includes read as UTF-8.
• crawl4ai/script/c4a_compile.py: C4ACompiler.compile_file reads as UTF-8.
• crawl4ai/cli.py: get_global_config and load_config_file read as UTF-8.
• crawl4ai/crawlers/google_search/crawler.py: script.js read as UTF-8.
• tests/general/test_utf8_text_files.py: new tests.

How Has This Been Tested?

The new tests run the real function in a child interpreter with a non-UTF-8 default (-X utf8=0, plus LC_ALL=C and PYTHONCOERCECLOCALE=0 on POSIX), so they fail without the change on any platform and not only on a Windows machine.

• Windows 11 (cp1254), Python 3.12: 4 failed on develop, 4 passed with this change.
• ubuntu-latest (Python 3.11) in a GitHub Actions run on my fork: 4 failed on develop, 4 passed with this change.
• Same ubuntu runner with a real GBK locale (localedef -i zh_CN -f GBK zh_CN.GBK, LC_ALL=zh_CN.GBK): on develop, load_js_script("flatten_shadow_dom") raises the UnicodeDecodeError above and compile_file of TYPE "你好" produces 浣犲ソ; with this change the snippet loads (3625 characters) and the compiled JS contains 你好.
• tests/cli/test_cli.py and tests/browser/test_init_script_dedup.py show the same 11 failures (17 pass) with and without this change on my machine, so I left them out of the "pass locally" box below.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes