Skip to content

feat: add upload checksum no-fallback upload-missing include-modtime#9505

Open
Subordinator wants to merge 1 commit into
rclone:fix-9308-hasher-updatefrom
Subordinator:feat-upload-checksum-data-integrity-flags
Open

feat: add upload checksum no-fallback upload-missing include-modtime#9505
Subordinator wants to merge 1 commit into
rclone:fix-9308-hasher-updatefrom
Subordinator:feat-upload-checksum-data-integrity-flags

Conversation

@Subordinator

Copy link
Copy Markdown

What is the purpose of this change?

Added data integrity flags for uploading data with stricter comparison measures.

Was the change discussed in an issue or in the forum before?

Link 1: https://forum.rclone.org/t/hasher-bolt-not-written-after-file-replacement-no-strict-checksum-mode/53623

Link 2: https://forum.rclone.org/t/update-hasher-bolt-not-written-after-file-replacement-no-strict-checksum-mode/53907

Checklist

  • I have read the contribution guidelines.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate.
  • All commit messages are in house style.
  • I'm done, this Pull Request is ready for review :-)

@Subordinator

Copy link
Copy Markdown
Author

FlagTest.bat:

@echo off
set "Executable=%Desktop%\Local Network Folder\rclone.exe"
set "RemoteStorage=FilenHasher"
set "HashType=blake3"

for %%I in ("%Executable%") do for %%J in ("%%~dpI.") do echo Directory: %%~nxJ
echo Remote: %RemoteStorage% (Hash: %HashType%)

:: Prerequisites (Automated):
:: Ensure TestFile.txt exists on the remote without a hash recorded against it.
:: 1. Create local folder and TestFile.txt with content '123' (without apostrophe).
:: 2. Purge the remote folder to start from a clean state.
:: 3. Upload TestFile.txt through %RemoteStorage% so the file lands on the underlying backend in the correct format.
:: 4. Drop the hasher bolt cache (removing the hash recorded) so the destination is left with the file but no hash.
rmdir "%Desktop%\LocalFolder" /s /q >nul 2>&1
mkdir "%Desktop%\LocalFolder"
echo 123 > "%Desktop%\LocalFolder\TestFile.txt"
del /f /q "%Desktop%\FlagTestLog.txt" >nul 2>&1
"%Executable%" purge "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
"%Executable%" copy "%Desktop%\LocalFolder\TestFile.txt" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
"%Executable%" backend drop "%RemoteStorage%:" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"

echo Testing Checksum Upload Missing Flag: %Time%
echo Section 1.1 && echo Section 1.1 >> "%Desktop%\FlagTestLog.txt"
"%Executable%" hashsum %HashType% "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Expecting no hash to be returned because the prerequisite dropped the bolt cache after seeding the file.

echo Section 1.2 && echo Section 1.2 >> "%Desktop%\FlagTestLog.txt"
"%Executable%" copy "%Desktop%\LocalFolder\TestFile.txt" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG" --checksum
"%Executable%" hashsum %HashType% "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Still expecting no hash to be returned because this is the default rclone functionality (will not upload if hash is missing).

echo Section 1.3 && echo Section 1.3 >> "%Desktop%\FlagTestLog.txt"
"%Executable%" copy "%Desktop%\LocalFolder\TestFile.txt" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG" --checksum --checksum-upload-missing
"%Executable%" hashsum %HashType% "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Now expecting a valid hash to be returned because we uploaded the checksum value with --checksum and --checksum-upload-missing.

:: Drop the hasher bolt cache so the destination is back to the no-hash state without needing to re-upload manually.
"%Executable%" backend drop "%RemoteStorage%:" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"

echo Testing Checksum No Fallback Flag: %Time%
echo Section 2.1 && echo Section 2.1 >> "%Desktop%\FlagTestLog.txt"
"%Executable%" hashsum %HashType% "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Expecting no hash to be returned because the file was re-created without uploading its hash.

echo Section 2.2 && echo Section 2.2 >> "%Desktop%\FlagTestLog.txt"
"%Executable%" sync "%Desktop%\LocalFolder" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG" --checksum
"%Executable%" hashsum %HashType% "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Still expecting no hash because --checksum alone falls back to size-only when no hash is available.

echo Section 2.3 && echo Section 2.3 >> "%Desktop%\FlagTestLog.txt"
"%Executable%" sync "%Desktop%\LocalFolder" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG" --checksum --checksum-no-fallback
"%Executable%" hashsum %HashType% "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Now expecting a valid hash because --checksum-no-fallback treated the missing destination hash as a mismatch and re-uploaded.

echo Testing Checksum Include Modtime Flag: %Time%
echo Section 3.1 && echo Section 3.1 >> "%Desktop%\FlagTestLog.txt"
echo     [Remote]: && "%Executable%" lsl "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Note the remote modtime - section 2.3 just re-uploaded the file with the current modtime.

echo Section 3.2 && echo Section 3.2 >> "%Desktop%\FlagTestLog.txt"
powershell -NoLogo -NoProfile -Command "(Get-Item -LiteralPath '%Desktop%\LocalFolder\TestFile.txt').LastWriteTime = Get-Date"
echo     [Local]:  && "%Executable%" lsl "%Desktop%\LocalFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
"%Executable%" sync "%Desktop%\LocalFolder" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG" --checksum
echo     [Remote]: && "%Executable%" lsl "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Expecting the remote modtime to be unchanged because --checksum alone ignores modtime when hashes match.

echo Section 3.3 && echo Section 3.3 >> "%Desktop%\FlagTestLog.txt"
powershell -NoLogo -NoProfile -Command "(Get-Item -LiteralPath '%Desktop%\LocalFolder\TestFile.txt').LastWriteTime = Get-Date"
echo     [Local]:  && "%Executable%" lsl "%Desktop%\LocalFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
"%Executable%" sync "%Desktop%\LocalFolder" "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG" --checksum --checksum-include-modtime
echo     [Remote]: && "%Executable%" lsl "%RemoteStorage%:RemoteFolder" --log-file="%Desktop%\FlagTestLog.txt" --log-level="DEBUG"
:: Now expecting the remote modtime to match the local modtime because --checksum-include-modtime detected the modtime mismatch and re-uploaded.

echo Finished: %Time%
pause >nul

Console Output:

Directory: Local Network Folder
Remote: FilenHasher (Hash: blake3)
Testing Checksum Upload Missing Flag: 21:17:57.36
Section 1.1
                                                                  TestFile.txt
Section 1.2
                                                                  TestFile.txt
Section 1.3
e5f6d77cfcb9eec7e64c043b8e8eedf7308960039de728dea1007f1aedd66c55  TestFile.txt
Testing Checksum No Fallback Flag: 21:18:02.86
Section 2.1
                                                                  TestFile.txt
Section 2.2
                                                                  TestFile.txt
Section 2.3
e5f6d77cfcb9eec7e64c043b8e8eedf7308960039de728dea1007f1aedd66c55  TestFile.txt
Testing Checksum Include Modtime Flag: 21:18:08.35
Section 3.1
    [Remote]:
        6 2026-06-09 21:17:54.123000000 TestFile.txt
Section 3.2
    [Local]:
        6 2026-06-09 21:18:10.231489500 TestFile.txt
    [Remote]:
        6 2026-06-09 21:17:54.123000000 TestFile.txt
Section 3.3
    [Local]:
        6 2026-06-09 21:18:12.722947600 TestFile.txt
    [Remote]:
        6 2026-06-09 21:18:12.722000000 TestFile.txt
Finished: 21:18:16.08

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant