Skip to content

googlephotos: fix nil panic in Update when using async batch_mode#9502

Draft
davispw wants to merge 10 commits into
rclone:masterfrom
davispw:gphotos-async-panic
Draft

googlephotos: fix nil panic in Update when using async batch_mode#9502
davispw wants to merge 10 commits into
rclone:masterfrom
davispw:gphotos-async-panic

Conversation

@davispw

@davispw davispw commented Jun 7, 2026

Copy link
Copy Markdown

Summary

When batch_mode=async is configured, batcher.Commit returns immediately with a nil *api.MediaItem result (the actual commit happens later in a background goroutine). The old code unconditionally passed this nil to o.setMetaData(info), which dereferenced it and panicked:

runtime error: invalid memory address or nil pointer dereference
goroutine ... github.com/rclone/rclone/backend/googlephotos.(*Fs).Put(...)
    backend/googlephotos/googlephotos.go

This made batch_mode=async completely unusable — any Put or Update call would crash rclone.


Root Cause

In lib/batcher/batcher.go:

func (b *Batcher[Item, Result]) Commit(...) (entry Result, err error) {
    ...
    if b.async {
        return entry, nil   // ← returns nil *api.MediaItem
    }
    ...
}

In backend/googlephotos/googlephotos.go (before fix):

info, err = o.fs.batcher.Commit(ctx, o.remote, uploaded)
// ...
o.setMetaData(info)  // ← panics when info == nil

Fix

setMetaData is given an explicit nil guard:

func (o *Object) setMetaData(info *api.MediaItem) {
    if info == nil {
        o.bytes = -1
        return
    }
    // ... fill remaining fields from info
}

The Update call site is also updated to use the local source size as o.bytes when info is nil, so the hasher overlay (PR #9500) can cache the hash under the correct fingerprint without waiting for the batch to flush:

if info == nil {
    o.bytes = src.Size()   // use local size; async batch will resolve later
} else {
    o.setMetaData(info)
}

All three Object constructors (newObjectWithInfo, newObject, Put) are also updated to initialise o.bytes = -1 explicitly, removing a latent assumption that the zero value of int64 (0) represents "unknown".


Effect on Other Backends

The nil guard in setMetaData is defensive programming that has no effect on any backend except Google Photos in async batch mode. The bytes = src.Size() assignment only executes when info == nil, which only happens from batcher.Commit in async mode.

batch_mode=sync (the default and recommended setting) is completely unaffected.


Commit Structure (TDD)

Commit Description
rmsulstw test: Adds TestAsyncBatchModePanic — asserts NotPanics; fails before fix
ntmosznr fix: Nil guard in setMetaData; test now passes

Automated Tests

=== RUN   TestAsyncBatchModePanic
--- PASS: TestAsyncBatchModePanic (0.00s)
ok  github.com/rclone/rclone/backend/googlephotos  0.4s

Manual Test Plan

Prerequisites

go build .

curl -L -o test_photo.jpg  "https://picsum.photos/seed/rclone_pr4_a/800/600"
curl -L -o test_photo2.jpg "https://picsum.photos/seed/rclone_pr4_b/800/600"

Step 1: Upload with async batch mode — must not panic

./rclone copy test_photo.jpg gphotos:album/rclone_PR4_Test \
  --gphotos-batch-mode=async --gphotos-batch-size=1 -v

Before the fix this crashed immediately:

runtime error: invalid memory address or nil pointer dereference

Expected after fix: command completes successfully.

2026/.../INFO  : test_photo.jpg: Copied (new)
Transferred:      1 / 1, 100%

(--gphotos-batch-size=1 flushes the async batch after every item so the commit happens before the process exits, making the file immediately visible.)

Step 2: Verify the file is visible in the album

./rclone lsf gphotos:album/rclone_PR4_Test/

Expected:

test_photo.jpg

Step 3: Overwrite with async batch mode — must not panic

./rclone copyto test_photo2.jpg gphotos:album/rclone_PR4_Test/test_photo.jpg \
  --gphotos-batch-mode=async --gphotos-batch-size=1 --ignore-times -v

Expected: completes without panic.

2026/.../INFO  : test_photo2.jpg: Copied (replaced existing) to: test_photo.jpg
Transferred:      1 / 1, 100%

Step 4: Verify only one file remains

./rclone lsf gphotos:album/rclone_PR4_Test/

Expected (deduplication means the old version is gone from the album; it may still exist in the library or trash album if trash_album_name is configured):

test_photo.jpg

Cleanup

Warning

Running rclone purge on rclone_Trash deletes the album container but leaves the files themselves orphaned ("zombie files") in your main Google Photos library. You must delete the files from your library first.

  1. Open the Google Photos Web UI, navigate to the rclone_Trash album, select all photos, and click "Move to trash" (or delete them).
  2. Purge the empty album containers:
./rclone purge gphotos:album/rclone_PR4_Test
./rclone purge gphotos:album/rclone_Trash
  1. Remove local test files:
rm test_photo.jpg test_photo2.jpg

Dependencies (gh-stack)

This PR is part of a stack. It depends on:

It does not depend on PR #9500 (fix-equal-hash-modtime-unsupported), which is a standalone hasher fix.

davispw added 4 commits June 6, 2026 14:26
The Google Photos Library API does not support permanently deleting
media items (https://issuetracker.google.com/issues/109759781).

To work around this, rclone now moves deleted and overwritten items
into a designated "trash" album (default: "rclone_Trash") and removes
them from the active album. Users can then review and permanently
delete items from the trash album via the Google Photos web UI.

- Move deleted items to trash album in Remove()
- Move old item to trash album after Update() re-uploads
- Add TrashAlbumName to Options (defaults to "rclone_Trash")
- Add api.BatchAddItems and api.BatchRemoveItems request types
- Add unit tests: TestRemoveTrashWorkaround, TestUpdateTrashWorkaround
The Google Photos Library API has no delete endpoint for media items
(https://issuetracker.google.com/issues/109759781). Add documentation
covering the trash album workaround added in the previous commit,
including the trash_album_name option and instructions for users to
review and permanently delete items from the trash album via the
Google Photos web UI.
davispw added 6 commits June 7, 2026 17:48
When uploading media, rclone can now read description metadata from
EXIF/IPTC/XMP tags and pass it to the Google Photos API as the media
item description, visible in the Google Photos UI.

This is controlled by two new options:
- read_exif_description: enable the feature (default: false)
- exif_description_fields: ordered list of tag names to try
  (default: Description,Caption-Abstract,ImageDescription,Title,ObjectName)

The first non-empty matching tag value is used. The feature uses the
github.com/bep/imagemeta library and reads only the first 512 KiB of
the upload stream to extract metadata before uploading.

Add unit test TestEXIFDescriptionMapping.
…ption fields

This adds a custom HandleXMP parser to extract Dublin Core nested title and
description tags (dc:title and dc:description) which are skipped by the default
imagemeta parser or are otherwise ignored since they are nested tags and not attributes.

This ensures that Lightroom-exported titles and descriptions successfully map
to the Google Photos description on upload.
@davispw davispw force-pushed the gphotos-async-panic branch from 544b2da to ef3f14a Compare June 8, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant