[codex] Add Gemini batch artifacts and deferred translation#1131
Open
nozuru wants to merge 9 commits into
Open
[codex] Add Gemini batch artifacts and deferred translation#1131nozuru wants to merge 9 commits into
nozuru wants to merge 9 commits into
Conversation
- Add GeminiBatchTranslator for 50% cost reduction via async batch processing - Add batch mode to TranslateConverter: defer translation/typesetting, flush all at once - Output intermediate data (source text, translation, layout) as JSON - Add google-genai dependency - Default batch model: gemini-3-flash-preview
…file support - Split extracted_data_path into source_data_path + translated_data_path for separate source.json / translated.json output - Add --translation-file (-tf) option to reuse translations from external JSON - Extract _collect_source_page_data() to run immediately after Phase A - Merge redundant translation_map/translations into single translations dict - Remove dead extracted_data_path parameter
- Reorganize output into artifacts/{doc_id}/json/{run_id}/ and artifacts/{doc_id}/pdf/
- Generate per-page JSON artifacts (pages/0001.json) and manifest.json per run
- Support pages/ directory input for -tf (translation file) option
- Use atomic writes for all file output to prevent partial files
- Adapt GUI and MCP server to new output file paths and naming
- Write source-only page JSON to artifacts/{doc_id}/source/pages/ (write-once)
- Source pages contain paragraphs and formulas without translations
- Fix manifest relative paths from ../pdf/ to ../../pdf/ (correct from json/{run_name}/)
- Add source_pages_dir reference in manifest artifacts
…nslators - Add thinking_level: low to GeminiTranslator (OpenAI-compatible), GeminiBatchTranslator single and batch requests for better cost/latency balance - Enable batch_mode (deferred Phase B+C) for all translators, not just those with translate_batch - Fall back to ThreadPoolExecutor parallel translation for translators without batch API support
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Impact
This lets PDF translation produce reusable intermediate artifacts and use deferred page processing across translators, including Gemini batch workflows. It also makes output handling safer and keeps generated manifests portable with relative paths.
Validation