Add min p arg to server

Add min p arg to server
Related to :
ggml-org/llama.cpp#3841

It looks like min p was added to llama-cpp-python recently:

I think it's missing from llama.py sampling

Second this! It looks very promising and I'm excited.

@abetlen

Pinging @abetlen, I finished this. See here: ddh0@e8e05bb

Apologies if this isn't the right way to submit a fix, I'm new to github. But I've tested it and it's working as expected. Let me know if I can help in any way.

Added support for min_p

@ddh0

@ddh0 bad timing, I just made this pr :P #921

Testing would be appreciated.

tested at temperature 1.6 and 3.0 the assistant remains creative and repetition issues are gone even with a lower repetition penalty. at 3.0 seems to be too creative and basically ignore facts passed in the context, but remains coherent and inteligible.

still I think it should be disabled by default for backcompat

Added support for min_p (#921)

tested at temperature 1.6 and 3.0 the assistant remains creative and repetition issues are gone even with a lower repetition penalty. at 3.0 seems to be too creative and basically ignore facts passed in the context, but remains coherent and inteligible.

still I think it should be disabled by default for backcompat

Hey there, I just saw that this is now implemented. How exactly do I activate min_p sampling when calling the model? Is it activated by default?

tested at temperature 1.6 and 3.0 the assistant remains creative and repetition issues are gone even with a lower repetition penalty. at 3.0 seems to be too creative and basically ignore facts passed in the context, but remains coherent and inteligible.
still I think it should be disabled by default for backcompat

Hey there, I just saw that this is now implemented. How exactly do I activate min_p sampling when calling the model? Is it activated by default?

the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

Thank you. So in which order are sampling methods like min_p, top_k and top_p executed? Can I change that order?

the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

Thank you. So in which order are sampling methods like min_p, top_k and top_p executed? Can I change that order?

take a look at these lines here..

llama-cpp-python/llama_cpp/llama.py

Lines 1160 to 1168 in 4026166

    
           self._ctx.sample_top_k(candidates=self._candidates, k=top_k, min_keep=1) 
        
           self._ctx.sample_tail_free(candidates=self._candidates, z=tfs_z, min_keep=1) 
        
           self._ctx.sample_typical( 
        
               candidates=self._candidates, p=typical_p, min_keep=1 
        
           ) 
        
           self._ctx.sample_top_p(candidates=self._candidates, p=top_p, min_keep=1) 
        
           self._ctx.sample_min_p(candidates=self._candidates, p=min_p, min_keep=1) 
        
           self._ctx.sample_temp(candidates=self._candidates, temp=temp) 
        
           id = self._ctx.sample_token(candidates=self._candidates)

The order is hardcoded at the moment.. but in my opinion you don't really want or need mixing min_p with top_p and top_k samplers, it's probably better to just disable top_p (top_p=1) etc and just use min_p and temp.
But if you wanna change the order and play with it, just edit that file.. it's simple.

Added support for min_p (#921)

Navigation Menu

Repository navigation

Add min p arg to server #911

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	self._ctx.sample_top_k(candidates=self._candidates, k=top_k, min_keep=1)
	self._ctx.sample_tail_free(candidates=self._candidates, z=tfs_z, min_keep=1)
	self._ctx.sample_typical(
	candidates=self._candidates, p=typical_p, min_keep=1
	)
	self._ctx.sample_top_p(candidates=self._candidates, p=top_p, min_keep=1)
	self._ctx.sample_min_p(candidates=self._candidates, p=min_p, min_keep=1)
	self._ctx.sample_temp(candidates=self._candidates, temp=temp)
	id = self._ctx.sample_token(candidates=self._candidates)

Add min p arg to server #911

Description

Activity

iandennismiller commented on Nov 15, 2023

LorenzoBoccaccia commented on Nov 15, 2023

ddh0 commented on Nov 15, 2023

ddh0 commented on Nov 16, 2023

tk-master commented on Nov 16, 2023

LorenzoBoccaccia commented on Nov 20, 2023

m-from-space commented on Nov 21, 2023

tk-master commented on Nov 21, 2023

m-from-space commented on Nov 25, 2023

tk-master commented on Nov 25, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions