Add support for new KV Cache Offloading API #995

Activity

1. Ph0rk0z commented on Dec 14, 2023

   

   Ph0rk0z

   on Dec 14, 2023

   More actions

   I force on the ctx param to offload kv_cache. The logs print that it's using F16 cache. 81 layers offloaded. memory sizes for K and V. assume KV "should" be. Speeds on 70b are now half and prompt processing is less than 1/4 with no context. Womp Womp.
2. Ph0rk0z commented on Dec 14, 2023

   

   Ph0rk0z

   on Dec 14, 2023

   More actions

   Ok, I got it working when I moved the kv to the proper place in the struct.

       _fields_ = [
           ("seed", c_uint32),
           ("n_ctx", c_uint32),
           ("n_batch", c_uint32),
           ("n_threads", c_uint32),
           ("n_threads_batch", c_uint32),
           ("rope_scaling_type", c_int8),
           ("rope_freq_base", c_float),
           ("rope_freq_scale", c_float),
           ("yarn_ext_factor", c_float),
           ("yarn_attn_factor", c_float),
           ("yarn_beta_fast", c_float),
           ("yarn_beta_slow", c_float),
           ("yarn_orig_ctx", c_uint32),
           ("type_k", c_int),
           ("type_v", c_int),
           ("mul_mat_q", c_bool),
           ("logits_all", c_bool),
           ("embedding", c_bool),
           ("offload_kqv", c_bool),
       ]
   
3. brandonrobertz

   added 3 commits that reference this issue on Dec 17, 2023

   Complete removal or f16_kv, add offload_kqv field

   ...

   905e864

   Complete removal or f16_kv, add offload_kqv field

   ...

   9cdfe93

   Complete removal or f16_kv, add offload_kqv field

   ...

   5603782
4. abetlen

   added a commit that references this issue on Dec 18, 2023

   Bugfix: Remove f16_kv, add offload_kqv field (#1019)

   ...

   62944df
5. abetlen

   closed this as completedon Dec 21, 2023