I’m using the Anthropic Claude API and I’m trying to generate multiple completions (n completions) for a given prompt in a single API call. OpenAI’s API provides an n parameter in their sampling settings to achieve this, but I can’t find an equivalent option in the Claude API.
I’m currently using a retry mechanism to handle potential errors during API calls, which looks like this:
from tenacity import retry, stop_after_attempt, wait_exponential def before_sleep(retry_state): print(f"(Tenacity) Retry, error that caused it: {retry_state.outcome.exception()}") def retry_error_callback(retry_state): exception = retry_state.outcome.exception() exception_str = str(exception) if "prompt is too long" in exception_str and "400" in exception_str: raise exception return 'No error that requires us to exit early.' @retry(stop=stop_after_attempt(20), wait=wait_exponential(multiplier=2, max=256), before_sleep=before_sleep, retry_error_callback=retry_error_callback)
def call_to_anthropic_client_api_with_retry(gen: AnthropicGenerator, prompt: str) -> dict: response = gen.llm.messages.create( model=gen.model, max_tokens=gen.sampling_params.max_tokens, system=gen.system_prompt, messages=[ {"role": "user", "content": [{"type": "text", "text": prompt}]} ], temperature=gen.sampling_params.temperature, top_p=gen.sampling_params.top_p, n=gen.sampling_params.n, # Intended to generate multiple completions
stop_sequences=gen.sampling_params.stop[:3], ) return response
Problem:
I can’t find an n parameter in the Anthropic API documentation that allows generating multiple completions in one request.
Questions:
- Does the Claude API support generating multiple completions (n completions) directly within a single API call?
- If not, is there a recommended workaround or best practice to achieve this without resorting to looping multiple requests?
Any guidance or suggestions would be greatly appreciated!
Discover more from Coursity
Subscribe to get the latest posts sent to your email.