Skip to content

fix(image): 修复openai图片data URI的MIME识别与编码链路#7017

Open
idiotsj wants to merge 9 commits intoAstrBotDevs:masterfrom
idiotsj:codex/fix-6991-openai-mime-minimal
Open

fix(image): 修复openai图片data URI的MIME识别与编码链路#7017
idiotsj wants to merge 9 commits intoAstrBotDevs:masterfrom
idiotsj:codex/fix-6991-openai-mime-minimal

Conversation

@idiotsj
Copy link
Copy Markdown
Contributor

@idiotsj idiotsj commented Mar 26, 2026

变更说明

修复 OpenAI 兼容链路中图片 data URI MIME 被固定为 image/jpeg
的问题,并统一图片来源到 data URI 的转换逻辑。

主要改动

  • io.py 新增 detect_image_mime_type(data: bytes)
  • io.py 新增 image_source_to_data_uri(image_source: str)
  • ProviderRequest._encode_image_bs64 改为复用统一 helper。
  • ProviderOpenAIOfficial.encode_image_bs64 改为复用统一 helper。
  • 默认 MIME 提取为常量 DEFAULT_IMAGE_MIME_TYPE
  • 明确 helper 约束:不处理 http(s),调用方需先下载。
  • 做了等价简化,减少重复分支与冗余赋值。

行为说明

  • 支持 PNG/JPEG/GIF/WEBP 识别,未知回退 image/jpeg
  • data:image/...;base64,... 透传并返回原 MIME。
  • base64://... 自动识别 MIME,非法 base64 回退 image/jpeg
  • 支持本地路径与 file:// URI。
  • 非图片 data: URI 与不支持 scheme 显式报错。

测试覆盖

  • tests/test_openai_source.py
    • base64:// PNG/GIF/WEBP
    • 本地 PNG/GIF/JPEG/WEBP
    • data:image/png/gif/jpeg 透传
    • 非法 base64 回退 image/jpeg
    • 非图片 data URI / 不支持 scheme 拒绝
    • extra_user_content_parts 的 file:// 图片 MIME 正确
  • tests/unit/test_astr_main_agent.py
    • ProviderRequest.assemble_context 覆盖本地路径、file://、base64://

验证命令

  • python -m pytest tests/test_openai_source.py -q
  • python -m pytest tests/unit/test_astr_main_agent.py -q -k image
  • python -m ruff check astrbot/core/utils/io.py astrbot/core/provider/entities.py astrbot/core/provider/sources/openai_source.py tests/test_openai_source.py tests/unit/test_astr_main_agent.py tests/fixtures/image_samples.py

测试截图

测试输出截图

close #6991

当前文件改动:

  • astrbot/core/utils/io.py
  • astrbot/core/provider/entities.py
  • astrbot/core/provider/sources/openai_source.py
  • tests/fixtures/image_samples.py
  • tests/test_openai_source.py
  • tests/unit/test_astr_main_agent.py

涉及的功能模块:

  • 图片来源标准化与 MIME 识别
  • OpenAI 兼容上下文组装
  • ProviderRequest 图片编码链路

Summary by Sourcery

Unify image source handling by converting local paths, file URIs, base64 sources, and data URIs into data:image/...;base64,... while preserving or inferring the correct MIME type, and update OpenAI-compatible providers to use this shared logic.

New Features:

  • Add a shared helper to detect image MIME types from binary data and convert various image sources into data URIs with best-effort MIME preservation.

Bug Fixes:

  • Fix OpenAI-compatible image handling where data URIs were always treated as image/jpeg, ensuring proper MIME detection for PNG, JPEG, GIF, and WEBP sources.
  • Ensure HTTP(S) image URLs are case-insensitively detected and downloaded before encoding, including uppercase schemes.

Enhancements:

  • Refactor ProviderRequest and OpenAI provider image encoding paths to reuse the common image-to-data-URI helper and reduce duplicated branching logic.

Tests:

  • Extend OpenAI source and main agent tests to cover MIME detection for local files, file:// URIs, base64:// sources, and data URIs, including error handling for invalid or unsupported inputs.
  • Introduce shared image sample fixtures for PNG, JPEG, GIF, and WEBP magic bytes and base64 payloads used across tests.

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Mar 26, 2026
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • Now that both _encode_image_bs64 and encode_image_bs64 always delegate to image_source_to_data_uri (which never returns an empty string), the if not image_data branches in the context assembly logic have effectively become dead code and can be removed or adjusted to only handle exception cases.
  • In image_source_to_data_uri, you explicitly reject http(s) URLs while the callers still use a simple startswith("http") check, which can be brittle (e.g. uppercase schemes or other http-prefixed schemes); consider aligning the scheme detection logic between the helper and its callers, or centralizing the scheme handling in one place to avoid edge-case mismatches.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Now that both `_encode_image_bs64` and `encode_image_bs64` always delegate to `image_source_to_data_uri` (which never returns an empty string), the `if not image_data` branches in the context assembly logic have effectively become dead code and can be removed or adjusted to only handle exception cases.
- In `image_source_to_data_uri`, you explicitly reject `http(s)` URLs while the callers still use a simple `startswith("http")` check, which can be brittle (e.g. uppercase schemes or other `http`-prefixed schemes); consider aligning the scheme detection logic between the helper and its callers, or centralizing the scheme handling in one place to avoid edge-case mismatches.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@dosubot dosubot bot added the area:provider The bug / feature is about AI Provider, Models, LLM Agent, LLM Agent Runner. label Mar 26, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request centralizes image-to-data-URI conversion logic by introducing the image_source_to_data_uri utility, which includes automatic MIME type detection using magic bytes. Redundant encoding logic in ProviderRequest and OpenAI provider sources has been refactored to use this utility, improving support for various image formats like PNG, GIF, and WEBP. New tests were added to ensure correct URI generation and MIME detection. A review comment suggests improving exception handling by catching a more specific error type during base64 decoding.

Comment on lines +256 to +257
except Exception:
pass
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

在处理 base64.b64decode 异常时,捕获更具体的 base64.binascii.Error 而不是通用的 Exception 会更好。这有助于区分不同类型的错误,并使代码更具可读性和可维护性。虽然当前逻辑在功能上没有问题,但更具体的异常处理是良好的编程实践。

Suggested change
except Exception:
pass
image_bytes = base64.b64decode(raw_base64)
mime_type = detect_image_mime_type(image_bytes)
except base64.binascii.Error:
pass

@idiotsj
Copy link
Copy Markdown
Contributor Author

idiotsj commented Mar 26, 2026

@sourcery-ai review

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/utils/io.py" line_range="255-261" />
<code_context>
+            "Remote image URL is not supported in image_source_to_data_uri; download the file before calling this helper.",
+        )
+
+    if image_source.startswith("base64://"):
+        raw_base64 = image_source.removeprefix("base64://")
+        mime_type = DEFAULT_IMAGE_MIME_TYPE
+        try:
+            image_bytes = base64.b64decode(raw_base64)
+            mime_type = detect_image_mime_type(image_bytes)
+        except Exception:
+            pass
+        return f"data:{mime_type};base64,{raw_base64}", mime_type
</code_context>
<issue_to_address>
**suggestion (bug_risk):** base64:// handling swallows all decode errors and still returns the original base64 without any signal.

Decoding errors for `base64://` are caught with a bare `Exception`, and you then return the original `raw_base64` with a default MIME. This makes invalid base64 indistinguishable from a valid non-JPEG image and may hide unrelated failures. Please narrow the exception to the specific decode errors (e.g., `binascii.Error`/`ValueError`), and consider logging at debug level or skipping `detect_image_mime_type` entirely when decode fails so the behavior is explicit.

Suggested implementation:

```python
    if image_source.startswith("base64://"):
        raw_base64 = image_source.removeprefix("base64://")
        mime_type = DEFAULT_IMAGE_MIME_TYPE

        try:
            image_bytes = base64.b64decode(raw_base64)
        except (binascii.Error, ValueError) as exc:
            # Invalid base64 payload: keep the default MIME type and skip detection.
            # This keeps the behavior explicit and does not hide non-base64 related errors.
            try:
                logger.debug("Failed to decode base64 image_source: %s", exc)
            except NameError:
                # logger may not be defined in all usages; fail silently if absent.
                pass
            return f"data:{mime_type};base64,{raw_base64}", mime_type

        try:
            mime_type = detect_image_mime_type(image_bytes)
        except Exception:
            # If MIME detection fails, fall back to the default type but still
            # return the original base64 content.
            try:
                logger.debug("Failed to detect MIME type for base64 image_source", exc_info=True)
            except NameError:
                pass

        return f"data:{mime_type};base64,{raw_base64}", mime_type

```

To fully integrate this change with the rest of the module, you should:

1. Add the missing imports at the top of `astrbot/core/utils/io.py` (if they are not already present):

   ```python
   import binascii
   import logging

   logger = logging.getLogger(__name__)
   ```

2. If your project uses a different logging convention (e.g., a shared logger utility), replace the `logger` usage here with that project-standard logger to keep logging consistent across the codebase.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@idiotsj
Copy link
Copy Markdown
Contributor Author

idiotsj commented Mar 26, 2026

@sourcery-ai review

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • In image_source_to_data_uri, base64:// and file:// handling is case-sensitive while HTTP(S) detection is case-insensitive; consider using lower_source for these checks as well so BASE64:// and FILE:// variants behave consistently.
  • When decoding base64:// sources, you already catch binascii.Error/ValueError; using base64.b64decode(raw_base64, validate=True) would make invalid base64 detection stricter and avoid silently accepting malformed input.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `image_source_to_data_uri`, `base64://` and `file://` handling is case-sensitive while HTTP(S) detection is case-insensitive; consider using `lower_source` for these checks as well so `BASE64://` and `FILE://` variants behave consistently.
- When decoding `base64://` sources, you already catch `binascii.Error`/`ValueError`; using `base64.b64decode(raw_base64, validate=True)` would make invalid base64 detection stricter and avoid silently accepting malformed input.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:provider The bug / feature is about AI Provider, Models, LLM Agent, LLM Agent Runner. size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]图片格式标记固定的问题

1 participant