Fix YJIT backend to account for unsigned int immediates (#6789)
YJIT: x86_64: Fix cmp with number where sign bit is set
Before this commit, we were unconditionally treating unsigned ints as
signed ints when counting the number of bits required for representing
the immediate in machine code. When the size of the immediate matches
the size of the other operand, no sign extension happens, so this was
incorrect. asm.cmp(opnd64, 0x8000_0000) panicked even though it's
encodable as CMP r/m32, imm32. Large shape ids were impacted by this
issue.
Added by jemmai (Jemma Issroff) over 2 years ago
Fix YJIT backend to account for unsigned int immediates (#6789)
YJIT: x86_64: Fix cmp with number where sign bit is set
Before this commit, we were unconditionally treating unsigned ints as
signed ints when counting the number of bits required for representing
the immediate in machine code. When the size of the immediate matches
the size of the other operand, no sign extension happens, so this was
incorrect.
asm.cmp(opnd64, 0x8000_0000)panicked even though it'sencodable as
CMP r/m32, imm32. Large shape ids were impacted by thisissue.
Co-Authored-By: Aaron Patterson [email protected]
Co-Authored-By: Alan Wu [email protected]
Co-authored-by: Aaron Patterson [email protected]
Co-authored-by: Alan Wu [email protected]