Skip to content

iclouddrive: support folders shared by another Apple ID (read + write) - fixes #9477#9487

Open
shadow-enthusiast wants to merge 4 commits into
rclone:masterfrom
shadow-enthusiast:pr-icloud-9477
Open

iclouddrive: support folders shared by another Apple ID (read + write) - fixes #9477#9487
shadow-enthusiast wants to merge 4 commits into
rclone:masterfrom
shadow-enthusiast:pr-icloud-9477

Conversation

@shadow-enthusiast

@shadow-enthusiast shadow-enthusiast commented Jun 2, 2026

Copy link
Copy Markdown

What is the purpose of this change?

Make a folder shared by another Apple ID (current account is a non-owner participant with edit access) usable through the iclouddrive backend. Previously any operation one level below the share root returned HTTP 400, so shared folders were unusable beyond their top level (#9477).

Two commits:

  1. Read / navigate — listing/downloading one or more levels below the share root returned HTTP 400. The CloudKit zone string is com.apple.CloudDocs for every item; the real cause is that items inside a shared folder have a FOLDER_IN_SHARED_FOLDER drivewsid and live in the owner's zone, which the drivews endpoints (caller's own zone only) cannot address. Fixed by routing shared subfolders through the docws item-id endpoints, which resolve the owner's zone cross-zone (enumerate by item_id, thread item_id through the dir cache, download via the enumerate-provided URL).

  2. Write — writing into a shared subfolder failed (HTTP 400 on the drivews move; HTTP 412 PCS-chaining error on a direct create, because a new record has no Protected Cloud Storage key). Rather than reimplement Apple's client-side PCS key-wrapping, this lets the server do the crypto: upload the file into the share root the normal way (the server performs PCS chaining there, in the owner's zone), then re-parent the resulting record into the target subfolder via the ckdatabasews CloudKit shared database — the server re-chains PCS itself because the moved record already carries its own key. Adds a small api/clouddocs.go client (zone discovery, records/lookup, records/modify re-parent + delete). This adds no client-side cryptography, so it is not sensitive to Apple rotating keys or changing the protectionInfo format.

All shared-only code paths are gated on FOLDER_IN_SHARED_FOLDER / FILE_IN_SHARED_FOLDER, so own-zone and share-root behaviour is unchanged.

Verified live against a real shared folder: lsf, cat, and copyto (incl. overwrite) one level inside the share now work where stock rclone returns HTTP 400.

Was the change discussed in an issue or in the forum before?

Yes — #9477.

Known limitations

  • Unit tests are included for the pure ID/zone helpers; full integration tests would need a TestICloud remote backed by a folder shared from another Apple ID — happy to add if you can advise on the fixture.
  • During a write the file briefly lands in the share root before being re-parented; a unique temporary name during that hop would remove a small collision window.
  • Overwrite enumerates the zone to locate the existing record (O(zone)); first writes do not enumerate.
  • Mkdir of new shared subfolders and nesting more than one level below the share root are not implemented here (out of scope).

Checklist

  • I have read the contribution guidelines.
  • I have added unit tests for the new pure helpers (offline; integration tests pending a shared-folder test fixture — see above).
  • I have updated the docs (happy to add an iclouddrive docs note if desired).
  • I have used a parseable commit message in the form iclouddrive: <summary>.

@shadow-enthusiast
iclouddrive: make folders shared by another Apple ID navigable and re…
62f83fa 
…adable

Operations one level below the root of a folder shared by another Apple
ID returned HTTP 400. Contrary to the "wrong zone" theory, the CloudKit
zone string is com.apple.CloudDocs for every item; the real cause is that
items inside a shared folder have a FOLDER_IN_SHARED_FOLDER drivewsid and
physically live in the OWNER's zone, which the drivews endpoints (which
only operate in the caller's own zone) cannot address.

Fix the read/navigation path by routing shared subfolders through the
docws item-id endpoints, which resolve shared zones cross-zone:

- api.IsSharedFolderChildID: detect FOLDER_IN_SHARED_FOLDER /
  FILE_IN_SHARED_FOLDER (and the empty-drivewsid case that deeper shared
  descendants arrive as).
- Thread item_id through the dir cache (drivewsid#etag#itemID) only for
  shared items; own-zone items keep drivewsid#etag unchanged.
- listAll: enumerate shared folders via GET /v1/enumerate/{item_id}
  instead of the drivews retrieveItemDetailsInFolders call that 400s.
- readMetaData: resolve via dirCache.FindPath so the item_id survives
  (Fs.FindPath strips it), fixing cat/NewObject inside shared subfolders.
- Object.Open: download shared files via the enumerate-provided
  url_download, since they have no usable drivewsid.

Listing and downloading one or more levels below the share root now work
where they previously returned 400; normal own-drive behaviour is
unchanged. Writing into a shared subfolder is addressed in a following
commit.

rclone#9477
@shadow-enthusiast
iclouddrive: write files into folders shared by another Apple ID
a950ed0 
Writing one level inside a folder shared by another Apple ID failed: the
drivews/docws upload+move endpoints operate only in the caller's own zone
and return HTTP 400 for any FOLDER_IN_SHARED_FOLDER parent, and creating the
CloudKit records directly is refused with a PCS chaining error because a new
record has no Protected Cloud Storage key.

Avoid client-side PCS crypto entirely by letting the server do it: upload the
file into the share root the normal way (drivews performs PCS chaining there,
in the owner's zone), then re-parent the resulting documentStructure into the
target sub-folder via the ckdatabasews CloudKit shared database. The server
re-chains PCS itself because the moved record already carries its own key.

Adds api/clouddocs.go, a small ckdatabasews client (zone discovery,
records/lookup, records/modify re-parent + delete, changes/zone scan), and
routes Object.Update through it when the destination is a shared sub-folder.
Own-zone and share-root writes and the existing read path are unchanged.

Fixes rclone#9477
@shadow-enthusiast
iclouddrive: add unit tests for shared-folder ID handling
f0492af 
Add offline, table-driven unit tests covering the deterministic helpers
introduced for reading from and writing into folders shared by another
Apple ID:

- api.IsSharedFolderChildID classification (empty / FOLDER_IN_SHARED_FOLDER
  / FILE_IN_SHARED_FOLDER vs own-zone and share-root IDs)
- Construct/Deconstruct/GetDocIDFromDriveID round-trip and short-ID fallback
- DriveItemRaw.SplitName and DriveItemRaw.IntoDriveItem conversion
- Document.DriveID zone defaulting
- the dir-cache ID helpers parseSharedItemID, folderID, IDJoin and
  parseNormalizedID, including the drivewsid#etag#itemID round-trip

All tests run without network access.

rclone#9477
ncw
ncw requested changes Jun 2, 2026
View reviewed changes

@ncw ncw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @shadow-enthusiast

Thanks for your PR :-)

See inline notes and these more general ones:

Please make sure all API calls are wrapped in the pacer for rate-limit, retries etc.

api/clouddocs.go builds requests as nested map[string]any literals throughout, whereas the rest of the api package uses typed request/response structs. Typed structs would be more idiomatic and self-documenting and reusable.

Did you run the integration tests for the backend with your changes?

Thanks

Comment thread backend/iclouddrive/iclouddrive.go Outdated
Comment thread backend/iclouddrive/api/clouddocs.go
Comment thread backend/iclouddrive/iclouddrive.go
Comment thread backend/iclouddrive/api/clouddocs.go Outdated
Comment thread backend/iclouddrive/iclouddrive.go
Comment thread backend/iclouddrive/iclouddrive.go Outdated
Comment thread backend/iclouddrive/iclouddrive.go
@shadow-enthusiast
iclouddrive: address review feedback on shared-folder support
98ca683 
- pacer-wrap all ckdatabasews (clouddocs) calls, with reauth on 401/421/423
- replace map[string]any request/response bodies with typed structs
- refresh the short-lived shared download URL on open failure and retry
- guard changes/zone pagination against an empty or unchanged syncToken
- debug-log a missing existing file on shared overwrite so duplicates are diagnosable
- log when multiple shared zones exist and none matches the expected name
- cache the stable shareRootID per name on the Fs
- use %q error formatting for the shared-folder-root lookup

rclone#9487
@shadow-enthusiast

shadow-enthusiast commented Jun 2, 2026

Copy link
Copy Markdown
Author

Thanks for the review @ncw — all addressed in 98ca683.

General

  • Pacer: every ckdatabasews call now goes through the backend pacer with the same shouldRetry handling as the rest of the package (rate-limit / transient retries), instead of issuing raw requests.
  • Typed structs: api/clouddocs.go no longer builds map[string]any literals — requests and responses (zones/list, records/lookup, records/modify, changes/zone) are now named typed structs with json tags, matching the style in api/drive.go.
  • Integration tests: no — the full backend integration tests need a TestICloud remote, and for this feature specifically a folder shared from a second Apple ID, which I don't have wired into CI. Before opening the PR I did run live integration smoke tests against a real shared folder (create the subfolder, write a file into it, read it back, overwrite, and delete) which confirmed the solution works end-to-end, and it is in local production use. Happy to add proper integration tests if you can advise on the shared-folder fixture.

Inline

  • Short-lived download URL (Object.Open): on an open failure the cached downloadURL is now refreshed (re-resolved) and retried, so long-lived objects (e.g. VFS cache) keep working.
  • changes/zone pagination: now breaks if moreComing is set but the syncToken is empty or unchanged, to avoid an infinite loop.
  • Overwrite duplicate: when the existing file isn't found we now emit an fs.Debugf so a duplicate is diagnosable.
  • Zone discovery: added a guard/log when multiple shared zones exist and none matches the expected com.apple.CloudDocs name, instead of blindly taking index 0.
  • shareRootID caching: the stable share-root id is now cached on the Fs instead of being looked up on every shared write.
  • Error formatting: switched to fmt.Errorf("...: %q", shareName) for consistency.

@shadow-enthusiast shadow-enthusiast requested a review from ncw June 2, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ncw ncw Awaiting requested review from ncw

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shadow-enthusiast @ncw