Skip to main content

LiteLLM

LiteLLM

Properties used to connect to a LiteLLM proxy server.

liteLLM​

  • Type: true | {
         key?: string,
         model?: string,
         system_prompt?: string,
         temperature?: number,
         max_tokens?: number,
         top_p?: number,
         frequency_penalty?: number,
         presence_penalty?: number,
         stop?: string | string[]
    }
  • Default: {model: "gpt-4o-mini", temperature: 1, max_tokens: 4096}

Connect to a LiteLLM proxy server that acts as a unified gateway to many AI providers.
key is an optional API key used to authenticate against the proxy.
model is the name of the model to route the request to, as configured in your LiteLLM proxy. See the provider list.
system_prompt provides system instructions for the model's behavior.
temperature controls randomness in the response (range depends on the underlying provider).
max_tokens is the maximum number of tokens to generate in the response.
top_p is the nucleus sampling parameter.
frequency_penalty penalizes repeated tokens based on their frequency.
presence_penalty penalizes repeated tokens based on whether they have appeared.
stop is a sequence (or array of sequences) at which the model will stop generating.

info

LiteLLM does not require an API key by default as it is typically self-hosted. If your proxy is configured with a master key or virtual keys, set the key property to authenticate.

Example​

<deep-chat
  directConnection='{
    "liteLLM": {
      "model": "gpt-4o-mini",
      "system_prompt": "You are a helpful assistant.",
      "temperature": 0.7
    }
  }'
></deep-chat>
info

Use stream to stream the AI responses.

Custom URL Example​

By default, LiteLLM connects to http://localhost:4000/v1/chat/completions. You can specify a custom URL using the connect property:

<deep-chat
  directConnection='{"liteLLM": {"key": "placeholder key"}}'
  connect='{"url": "https://your-litellm-proxy.com/v1/chat/completions"}'
></deep-chat>

Prerequisites​

To use LiteLLM with Deep Chat, you need to:

  1. Install LiteLLM: Follow the quick start guide
  2. Configure your providers: Add your provider credentials (OpenAI, Anthropic, etc.) to the proxy config.yaml or as environment variables
  3. Start the proxy: Run litellm --config config.yaml — the proxy will listen on http://localhost:4000 by default