Evaluate any of the following expressions:

1. ('X' * (2**31 / 3 + 98)).unpack('m').first.length
2. ('X' * (2**31 / 3 + 98)).unpack('u').first.length
3. ('X' * (2**32 / 3 + 99)).unpack('m').first.length
4. ('X' * (2**32 / 3 + 99)).unpack('u').first.length

Expected Behavior

Expressions 1 and 2 evaluate to 536870985 and expressions 3 and 4 to 1073741898 respectively.

Actual Behavior

Expression 1 crashes with:

org.jruby.util.Pack.unpack_m(Pack.java:1258): -536870839 (Java::JavaLang::NegativeArraySizeException)
	from org.jruby.util.Pack.unpackInternal(Pack.java:1083)
	from org.jruby.util.Pack.unpackWithBlock(Pack.java:870)
	from org.jruby.RubyString.unpack(RubyString.java:6605)
	from org.jruby.RubyString$INVOKER$i$unpack.call(RubyString$INVOKER$i$unpack.gen)
	from org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:423)
        ...

Expression 2 crashes with:

org.jruby.util.Pack.unpack_u(Pack.java:1392): -536870839 (Java::JavaLang::NegativeArraySizeException)
	from org.jruby.util.Pack.unpackInternal(Pack.java:1080)
	from org.jruby.util.Pack.unpackWithBlock(Pack.java:870)
	from org.jruby.RubyString.unpack(RubyString.java:6605)
	from org.jruby.RubyString$INVOKER$i$unpack.call(RubyString$INVOKER$i$unpack.gen)
	from org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:423)

Expression 3 crashes with:

org.jruby.util.Pack.unpack_m_nonzeroOccurrences(Pack.java:1320): Index 74 out of bounds for length 74 (Java::JavaLang::ArrayIndexOutOfBoundsException)
	from org.jruby.util.Pack.unpack_m(Pack.java:1266)
	from org.jruby.util.Pack.unpackInternal(Pack.java:1083)
	from org.jruby.util.Pack.unpackWithBlock(Pack.java:870)
	from org.jruby.RubyString.unpack(RubyString.java:6605)
	from org.jruby.RubyString$INVOKER$i$unpack.call(RubyString$INVOKER$i$unpack.gen)

Expression 4 incorrectly evaluates to 74.

Environment Information

Tested on JRuby 9.4.12.0. Inspection of source code suggests 9.4.13.0 and 10.0.1.0 are also affected.

Cause

When the length of the input string exceeds (2³¹ − 1) / 3 bytes, the buffer length calculation in Pack.unpack_m and Pack.unpack_u can overflow. As a result, the methods either try to allocate a buffer with negative length or, if the calculation overflows by enough to end up positive again, will allocate a buffer that is too short for the decoded data.

Notes

Replacing 'm' with 'm0' in expressions 1 and 3 above will trigger the same error (except with Pack.unpack_m_zeroOccurrences instead of Pack.unpack_m_nonzeroOccurrences in the stack trace for expression 3).

Replacing .unpack(...).first with .unpack1(...) in the expression above will also trigger the same behavior, except for the special case of .unpack1('m0'), which takes a special optimized code path that calls Pack.unpackBase46Strict instead of Pack.unpack_m. This method uses a different buffer length formula which does not seem to be vulnerable to overflow.

Suggested fix

A simple fix for the integer overflow would be to cast the remaining buffer length to a long before multiplying it by 3.

Alternatively, the formula currently used in the unpackBase64Strict method (3 * ((length + 3) / 4)) could also be used in unpack_m and unpack_u. This formula can sometimes result in a buffer allocation that is one or two bytes longer than needed, but this seems mostly harmless (and can in any case happen with either formula e.g. if the input contains padding).