CCM, KCCM, KGCM, EAX, OCB: refuse a second encryption without a new init - #2562

Summary

After an encryption completes, CCMBlockCipher, KCCMBlockCipher, KGCMBlockCipher, EAXBlockCipher and OCBBlockCipher encrypt a further message on the same instance under the same key and nonce if they are used again without init, either directly or after reset().

• For CCM, KCCM, KGCM and EAX this reuses the CTR keystream: C1 xor C2 = P1 xor P2. It also undermines the authentication for that nonce.
• For OCB it is nonce reuse.

The init-time guards ("cannot reuse nonce for ... encryption") don't cover this, because no new init happens. GCMBlockCipher and ChaCha20Poly1305 already refuse it ("... cannot be reused for encryption").

Reproduced on main (a1bb4ad): CCM, KCCM, KGCM and EAX give the same keystream for the second message, and OCB accepts it. bcprov 1.86 behaves the same for CCM.

Change

Each of the five records when an encryption has used the current key and nonce, and only init clears that record. A further encryption throws IllegalStateException("<mode> cipher cannot be reused for encryption"), the same message pattern as GCM.

• CCM, KCCM and KGCM only produce output in processPacket / doFinal. The nonce counts as used once an encryption gets past its output-length check, so a too-small buffer can still be retried. reset() on its own does not use the nonce.
• EAX and OCB stream ciphertext from processByte(s). The nonce is used by an encryption doFinal, or by a reset() after message data has been processed. In OCB's doFinal this happens on entry, since it writes the final block before checking room for the tag. A reset() before any message data still allows encryption.
• Decryption is unaffected.
• The jdk1.1 copy of EAXBlockCipher gets the same change.

Behaviour changes to note

• JCE: Cipher.doFinal() resets the cipher for reuse with the same key and IV. For AES/CCM, AES/EAX, AES/OCB and the DSTU 7624 KCCM/KGCM ciphers, a second encryption without init now throws IllegalStateException. That is what GCM already does in BC and SunJCE.
• KGMac: like GMac over GCM, KGMac now needs a fresh init between messages rather than reset(). Reusing a GMAC nonce for a different message weakens the MAC anyway.
• OpenPGP, TLS and CMS call init with a fresh nonce per chunk, record or message, so they are not affected.

Tests changed

Several existing tests encrypted twice under one key and nonce. They now expect the refusal and use a fresh init or instance for the repeat:

• Core helpers:
  • AEADTestUtil.testReset: the encryption half asserts the refusal; the decryption half is unchanged.
  • AEADTestUtil.testBufferSizeChecks: the encryption half uses init with the existing varyNonce helper instead of reset().
• Core tests:
  • CCMTest: bufferClearingTest and the two processPacket overloads use a fresh instance for the repeat.
  • EAXTest: the repeat runs use fresh encryptors.
  • OCBTest: the "(reused)" run now reuses only the decryptor.
  • DSTU7624Test:
    • doFinalTest now expects the repeat to be refused.
    • KGMacPartialBlockTests re-initialises with a fresh IV, keeping its check that the result doesn't depend on what the instance processed before.
    • KGCMNonceReuseTests and KCCMNonceReuseTests previously asserted that "reset()-based reuse must still work". That is the reuse removed here, so they now assert the refusal. Please check this matches your intent.
• Provider tests:
  • AEADTest expects the refusal.
  • BlockCipherTest already took a fresh Cipher for GCM before repeating an encryption. It now does the same for EAX, OCB and CCM through a small newEncryptor helper that mirrors the original init, including the RC2/RC5 specs.

Release note added to docs/releasenotes.md (1.87).

Tests

• New AEADEncryptionReuseTest (SimpleTest, registered in crypto.test.RegressionTest) covers CCM, KCCM, KGCM, EAX and OCB. It checks that:

  • a second encryption is refused, before and after reset(), and through CCM processPacket;
  • init with a new nonce encrypts and decrypts correctly;
  • reset() before any data still allows encryption;
  • reset() after streamed data uses the nonce (EAX, OCB);
  • decryption stays reusable.

  It fails on main and passes with the change.

• Core tests pass: CCMTest, EAXTest, OCBTest, DSTU7624Test, GCMTest, ChaCha20Poly1305Test, GCMSIVTest.

• Provider tests pass: AEADTest, BlockCipherTest, OCBTest, DSTU7624Test, AESTest, ARIATest, CipherStreamTest2, AlgorithmParametersTest, ECIESTest. Before the test updates, AEADTest and BlockCipherTest failed only on the EAX reuse described above; all of these pass on main too.

• I ran these with javac and the test classes' runners (JDK 25), not through Gradle.

The same issue in bc-csharp is fixed in bcgit/bc-csharp#754.