googlephotos: fix nil panic in Update when using async batch_mode#9502
Draft
davispw wants to merge 10 commits into
Draft
googlephotos: fix nil panic in Update when using async batch_mode#9502davispw wants to merge 10 commits into
davispw wants to merge 10 commits into
Conversation
The Google Photos Library API does not support permanently deleting media items (https://issuetracker.google.com/issues/109759781). To work around this, rclone now moves deleted and overwritten items into a designated "trash" album (default: "rclone_Trash") and removes them from the active album. Users can then review and permanently delete items from the trash album via the Google Photos web UI. - Move deleted items to trash album in Remove() - Move old item to trash album after Update() re-uploads - Add TrashAlbumName to Options (defaults to "rclone_Trash") - Add api.BatchAddItems and api.BatchRemoveItems request types - Add unit tests: TestRemoveTrashWorkaround, TestUpdateTrashWorkaround
The Google Photos Library API has no delete endpoint for media items (https://issuetracker.google.com/issues/109759781). Add documentation covering the trash album workaround added in the previous commit, including the trash_album_name option and instructions for users to review and permanently delete items from the trash album via the Google Photos web UI.
…he trash workaround
db5fd80 to
544b2da
Compare
When uploading media, rclone can now read description metadata from EXIF/IPTC/XMP tags and pass it to the Google Photos API as the media item description, visible in the Google Photos UI. This is controlled by two new options: - read_exif_description: enable the feature (default: false) - exif_description_fields: ordered list of tag names to try (default: Description,Caption-Abstract,ImageDescription,Title,ObjectName) The first non-empty matching tag value is used. The feature uses the github.com/bep/imagemeta library and reads only the first 512 KiB of the upload stream to extract metadata before uploading. Add unit test TestEXIFDescriptionMapping.
…identical byte deduplication
…ption fields This adds a custom HandleXMP parser to extract Dublin Core nested title and description tags (dc:title and dc:description) which are skipped by the default imagemeta parser or are otherwise ignored since they are nested tags and not attributes. This ensures that Lightroom-exported titles and descriptions successfully map to the Google Photos description on upload.
544b2da to
ef3f14a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
batch_mode=asyncis configured,batcher.Commitreturns immediately with anil*api.MediaItemresult (the actual commit happens later in a background goroutine). The old code unconditionally passed this nil too.setMetaData(info), which dereferenced it and panicked:This made
batch_mode=asynccompletely unusable — anyPutorUpdatecall would crash rclone.Root Cause
In
lib/batcher/batcher.go:In
backend/googlephotos/googlephotos.go(before fix):Fix
setMetaDatais given an explicit nil guard:The
Updatecall site is also updated to use the local source size aso.byteswheninfois nil, so thehasheroverlay (PR #9500) can cache the hash under the correct fingerprint without waiting for the batch to flush:All three
Objectconstructors (newObjectWithInfo,newObject,Put) are also updated to initialiseo.bytes = -1explicitly, removing a latent assumption that the zero value ofint64(0) represents "unknown".Effect on Other Backends
The nil guard in
setMetaDatais defensive programming that has no effect on any backend except Google Photos in async batch mode. Thebytes = src.Size()assignment only executes wheninfo == nil, which only happens frombatcher.Commitin async mode.batch_mode=sync(the default and recommended setting) is completely unaffected.Commit Structure (TDD)
rmsulstwtest:AddsTestAsyncBatchModePanic— assertsNotPanics; fails before fixntmosznrfix:Nil guard insetMetaData; test now passesAutomated Tests
Manual Test Plan
Prerequisites
Step 1: Upload with async batch mode — must not panic
Before the fix this crashed immediately:
Expected after fix: command completes successfully.
(
--gphotos-batch-size=1flushes the async batch after every item so the commit happens before the process exits, making the file immediately visible.)Step 2: Verify the file is visible in the album
Expected:
Step 3: Overwrite with async batch mode — must not panic
Expected: completes without panic.
Step 4: Verify only one file remains
Expected (deduplication means the old version is gone from the album; it may still exist in the library or trash album if
trash_album_nameis configured):Cleanup
Warning
Running
rclone purgeonrclone_Trashdeletes the album container but leaves the files themselves orphaned ("zombie files") in your main Google Photos library. You must delete the files from your library first.rclone_Trashalbum, select all photos, and click "Move to trash" (or delete them).Dependencies (gh-stack)
This PR is part of a stack. It depends on:
gphotos-trash-album: trash workaround + error propagationgphotos-exif-description: EXIF/XMP description mappingIt does not depend on PR #9500 (
fix-equal-hash-modtime-unsupported), which is a standalone hasher fix.