The temperature parameter is used by the GPT Family of completions model provided by OpenAI. Although it is a handy parameter, it is not very self-explanatory.
Its values are between 0 and 1. This controls the randomness of the LLM, with 0 being deterministic and 1 being more random.
Let’s visualise that by taking an example of a sentence completion model.
Sentence Completion
For this experiment, we have kept our model simple with a prompt:
CREATE PROMPT sentence_completion( system "Your Job is to complete the sentence provided by the user.", human "{{sentence}}"
)
Temperature = 0
Let’s test the lowest temperature for a sentence such as “My favourite animal is “. We ran the model 5 times and got the exact same output. We can expect the same results since the randomness has been eliminated.
The lower temperature is suitable for cases when we need stability and the most probable output (text-to-SQL, financial analysis, etc.)
Temperature = 1
Setting the temperature to 1 can give inconsistent and exciting results. Even for tasks like story generation, which needs creativity, we often see this value kept between 0.7-0.9.
Here are a few outputs of the same query above but keeping the temperature to 1.
The output here is very different from before. In this case, each execution gave a different output along with explaining why that animal is my favourite.
So what value of temperature is best? It depends
The best value for the temperature parameter is not set in stone. As it’s one of the critical parameters that have a significant impact on the model’s expected behaviour, one has to play around to find the sweet spot between randomness and consistency.
Play with the temperature parameter in this notebook: https://app.langdb.ai/share/apps/66fc984e-0fc9-4674-9406-d299d3df9aff