Bitwise AND on signed i8/i16 skips sign extension when one operand is wrapped or negative #3061

Bug description

Bitwise & on i8 or i16 omits sign extension (i32.extend8_s / i32.extend16_s) when one operand is already wrapped or is a negative constant, leaving upper bits from an unmasked arithmetic expression intact.

For example, in the reproduction below:

• In f(), a + a wraps to -2 in i16, and -2 & -1 should evaluate to -2, so f() should return -2n, but it returns 65534n.
• In g(100), a + a wraps to -56 in i8, and -56 & -128 should evaluate to -128, so g(100) should return true (1), but it returns false (0).

Flow.canOverflow for BinaryOp.AndI32 treats x & y as non-overflowing if either operand cannot overflow or is a constant <= computeSmallIntegerMask(type), which only holds for non-negative values since negative small integers have 1s in their upper 16/24 bits.

Steps to reproduce

export function f(): i64 {
  let a: i16 = 32767;
  let m: i16 = -1;
  let b: i16 = (a + a) & m; // a + a wraps to -2 in i16; -2 & -1 is -2
  return <i64>b;
}

export function g(a: i8): bool {
  let b: i8 = (a + a) & <i8>-128; // for a = 100: a + a wraps to -56; -56 & -128 is -128
  return b == -128;
}

Compile with asc repro.ts -O0 (or -O3) and call f() and g(100).

Credit: https://ucsbplse.github.io/

AssemblyScript version

v0.28.20

Playground

https://www.assemblyscript.org/editor.html#IyFvcHRpbWl6ZT1zcGVlZCZydW50aW1lPWluY3JlbWVudGFsCi8vIENhc2UgMTogPGJvb2w+IGNhc3Qgb2YgdTggZG9lcyBub3QgdHJ1bmNhdGUKZXhwb3J0IGZ1bmN0aW9uIGYoYTogdTgpOiBib29sIHsKICBsZXQgYzogdTggPSBhICsgMTsKICByZXR1cm4gPGJvb2w+YzsKfQpleHBvcnQgZnVuY3Rpb24gZyhhOiB1OCk6IGJvb2wgewogIGxldCBjOiB1OCA9IGEgKyAxOwogIHJldHVybiBjID8gdHJ1ZSA6IGZhbHNlOwp9Ci8vIENhc2UgMjogaTE2L2k4ICYgc2tpcHMgc2lnbiBleHRlbnNpb24KZXhwb3J0IGZ1bmN0aW9uIGgoKTogaTY0IHsKICBsZXQgYTogaTE2ID0gMzI3Njc7CiAgbGV0IG06IGkxNiA9IC0xOwogIGxldCBiOiBpMTYgPSAoYSArIGEpICYgbTsKICByZXR1cm4gPGk2ND5iOwp9CmV4cG9ydCBmdW5jdGlvbiBrKGE6IGk4KTogYm9vbCB7CiAgbGV0IGI6IGk4ID0gKGEgKyBhKSAmIDxpOD4tMTI4OwogIHJldHVybiBiID09IC0xMjg7Cn0KIyFodG1sCjxwcmUgaWQ9Im91dCI+PC9wcmU+CjxzY3JpcHQgdHlwZT0ibW9kdWxlIj4KY29uc3QgZSA9IGF3YWl0IGluc3RhbnRpYXRlKGF3YWl0IGNvbXBpbGUoKSwge30pOwpkb2N1bWVudC5nZXRFbGVtZW50QnlJZCgib3V0IikudGV4dENvbnRlbnQgPSBbCiAgImYoMjU1KSA9ICIgKyBlLmYoMjU1KSArICIgIChleHBlY3RlZCBmYWxzZSkiLAogICJnKDI1NSkgPSAiICsgZS5nKDI1NSkgKyAiICAoZXhwZWN0ZWQgZmFsc2UpIiwKICAiaCgpID0gIiArIGUuaCgpICsgIiAgKGV4cGVjdGVkIC0yKSIsCiAgImsoMTAwKSA9ICIgKyBlLmsoMTAwKSArICIgIChleHBlY3RlZCB0cnVlKSIKXS5qb2luKCJcbiIpOwo8L3NjcmlwdD4=

Click Run: f(255) prints true (case 1, expected false), h() prints 65534 (expected -2) and k(100) prints false (case 2, expected true).