Hi LuaVela team,

There are some problems with recording tail-called rindex():

1. Since return type checks are emitted after CALLT (or CALLMT) recording, RETF instruction will be emitted before them:

   0024 rax      p64 CALLL  lj_tab_rawrindex_jit  (0023)
   0025       >  p32 RETF   tailcall.lua:7  [0x7fa5168ad168]
   0026       >  p64 NE     0024  [NULL]
   0027 rax   >  str TVLOAD 0024
   

   Which leads to situation when frame pointer will be unwinded to caller, but then trace exits due to failed return type check and starts executing calle's PC.
   Just to compare, this is IR with regular return (RETF is placed after checks):

   0024 rax      p64 CALLL  lj_tab_rawrindex_jit  (0023)
   0025       >  p64 NE     0024  [NULL]
   0026 rax   >  str TVLOAD 0024
   ....              SNAP   #1   [ ---- true ---- 0026 ]
   0027       >  p32 RETF   tailcall.lua:7  [0x7fb978f55170]
   

   In attached test this is the first example.
   This patch emits return value check before RETF and later fixes guard type. Example above will looks like:

   0024 rax      p64 CALLL  lj_tab_rawrindex_jit  (0023)
   0025       >  p64 NE     0024  [NULL]
   0026 rax   >  str TVLOAD 0024
   0027       >  p32 RETF  tailcall.lua:7  [0x7f6fb1db8168]
   

2. Return type check can get broken and doesn't actually check lj_tab_rawrindex_jit result (since J->base will be pointing to previous frame by the time LJ_POST_FFSPECRET is applied). This is IR for the second test example (instead of rindex its argument "k1" is checked):

   0021          nil TVARG  0019  "k1"
   0022          nil TVARG  0021  "k2"
   0023 rdi      p64 TVARGF 0022  #3  
   0024          p64 CALLL  lj_tab_rawrindex_jit  (0023)
   0025       >  p32 RETF   tailcall.lua:38  [0x7f26b44a4168]
   0026       >  p64 NE     "k1"  [NULL]
   0027       >  tab TVLOAD "k1"
   

   Under O3 lj_tab_rawrindex_jit call will be completely purged out from assembled trace due to it, but NE and TVLOAD still be present and will check random registers.
   This patch fixes this case too:

   0024 rax      p64 CALLL  lj_tab_rawrindex_jit  (0023)
   0025       >  p64 NE     0024  [NULL]
   0026       >  tab TVLOAD 0024
   0027       >  p32 RETF   tailcall.lua:38  [0x7f360fc62170]
   

3. It's possible that J->pc - 1 will point to HOTCNT bytecode instead of CALL(M) during LJ_POST_FFSPECRET fixup. This case covered by 3rd testcase.

PS. Also looks like its safe to remove assert for CALLT and CALLMT, since by the time of LJ_POST_FFSPECRET execution VM is already returned to lower frame and can't hit CALLT or CALLMT.

I'm not absolutely sure about correctness of the patch, please help to figure it out.
Thanks in advance!