Skip to main content

OpenAI Realtime

OpenAI Realtime

Try this service with custom properties in the Playground.

realtime​

Connect to OpenAI Reatime API. You can either use true or an object:
ephemeralKey is the temporary session key used to connect from the browser safely.
fetchEphemeralKey is a function definition used to retrieve the ephemeralKey.
autoFetchEphemeralKey triggers fetchEphemeralKey on component render.
autoStart begins the conversation on component render.
avatar is used to configure the avatar.
buttons is used to configure the buttons.
config is as object that defines the connection properties.
methods are used to update the conversation context in real time.
events are triggered based on user activity.
loading defines the styling for the loading message.
error defines the styling for the error message. 

<deep-chat
  directConnection='{
    "openAI": {"key": "placeholder key", "realtime": true}
  }'
></deep-chat>
info

If ephemeralKey or fetchEphemeralKey properties are used, the key property is not required. If the key property is used instead, the ephemeralKey will be fetched automatically.

Types​

FetchEphemeralKey​

  • Type: () => string | Promise<string>

This is a function definition that will return the ephemeralKey value programmatically.
Details on how to retrieve the key can be found here.

// There are many ways to define the function depending on your framework.
// The following is a simple VanillaJs example:

chatElementRef.directConnection.openAI.realtime.fetchEphemeralKey = () => 'my-key';
tip

See the OpenAIRealtimeConfig to see available session configuration.

OpenAIRealtimeAvatar​

Configuration for the avatar.
src is the path for the image.
maxScale is the maximum size that the avatar is allowed to expand to when a response is spoken.
styles is an object that is used to style the avatar image and the container which is the area around the avatar. 

<deep-chat
  directConnection='{
    "openAI": {
      "realtime": {
        "avatar": {
          "src": "path-to-file.png",
          "maxScale": 3.5,
          "styles": {
            "image": {"padding": "5px"},
            "container": {"height": "60%"}
          }}}}}'
></deep-chat>

OpenAIRealtimeButtons​

Configuration for the buttons section.
container is styling for the buttons area.
microphone is styling for the various states of the microphone button.
toggle is styling for the various states of the toggle button. 

<deep-chat
  directConnection='{
    "openAI": {
      "realtime": {
        "buttons": {
          "container": {"backgroundColor": "#f7f7f7"},
          "microphone": {
            "default": {
              "svg": {"content": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"800px\" height=\"800px\" viewBox=\"0 0 24 24\" fill=\"none\">\n<path d=\"M3 16.5C2.59 16.5 2.25 16.16 2.25 15.75V8.25C2.25 7.84 2.59 7.5 3 7.5C3.41 7.5 3.75 7.84 3.75 8.25V15.75C3.75 16.16 3.41 16.5 3 16.5Z\" fill=\"#292D32\"/>\n<path d=\"M7.5 19C7.09 19 6.75 18.66 6.75 18.25V5.75C6.75 5.34 7.09 5 7.5 5C7.91 5 8.25 5.34 8.25 5.75V18.25C8.25 18.66 7.91 19 7.5 19Z\" fill=\"#292D32\"/>\n<path d=\"M12 21.5C11.59 21.5 11.25 21.16 11.25 20.75V3.25C11.25 2.84 11.59 2.5 12 2.5C12.41 2.5 12.75 2.84 12.75 3.25V20.75C12.75 21.16 12.41 21.5 12 21.5Z\" fill=\"#292D32\"/>\n<path d=\"M16.5 19C16.09 19 15.75 18.66 15.75 18.25V5.75C15.75 5.34 16.09 5 16.5 5C16.91 5 17.25 5.34 17.25 5.75V18.25C17.25 18.66 16.91 19 16.5 19Z\" fill=\"#292D32\"/>\n<path d=\"M21 16.5C20.59 16.5 20.25 16.16 20.25 15.75V8.25C20.25 7.84 20.59 7.5 21 7.5C21.41 7.5 21.75 7.84 21.75 8.25V15.75C21.75 16.16 21.41 16.5 21 16.5Z\" fill=\"#292D32\"/>\n</svg>"}
            }},
          "toggle": {
            "default": {
              "svg": {
                "styles": {
                  "default": {"filter": "brightness(0) saturate(100%) invert(58%) sepia(77%) saturate(282%) hue-rotate(172deg) brightness(91%) contrast(93%)" }
                }}}}}}}}'
></deep-chat>

OpenAIRealtimeButton​

Configuration for the various sates of a button.

OpenAIRealtimeConfig​

  • Type: {
         model?: string,
         instructions?: string,
         voice?: string,
         temperature?: number,
         max_response_output_tokens?: number | "inf",
         turn_detection?: {
             type?: string,
             threshold?: string,
             prefix_padding_ms?: number,
             silence_duration_ms?: number}
         input_audio_transcription?: true | {
             model?: string,
             language?: string,
             prompt?: string}
         tools?: {
             type?: "function"|"code_interpreter"|"file_search",
             name?: string,
             description?: number,
             parameters?: object}[]
         tool_choice?: string,
         function_handler?: OpenAIRealtimeFunction
    }
  • Default: {model: "gpt-4o-realtime-preview-2025-06-03"}

Session configuration. This object follows the configuration referenced here.
IMPORTANT: If you are providing your own ephemeralKey or using the fetchEphemeralKey function, this configuration will need to be used there. However, this does not apply to the function_handler.
model is the model name used for the session.
instructions allows the model to be guided on desired responses (also known as system message).
voice is the model response's voice. See full list here.
temperature is the sampling temperature for the model.
max_response_output_tokens is the maximum number of output tokens for a single assistant response.
turn_detection is configuration to detect when the model should start speaking. More info can be found here.
input_audio_transcription enables transcription of user audio. Set to true to use default settings with "whisper-1" model or provide an object. This is required to receive user message transcripts via onMessage.
tools is a list of tools available to the model. More info can be found here.
tool_choice is used to define how the model chooses tools. 

<deep-chat
  directConnection='{
    "openAI": {
      "realtime": {
        "config": {
          "instructions": "Answer only with yes or no",
          "voice": "alloy"
    }}}}'
></deep-chat>

OpenAIRealtimeFunction​

  • Type: ({name: string, arguments: string}) => object

The actual function that the component will call if the model wants a response from OpenAIRealtimeConfig tools functions. 

// using JavaScript for a simplified example

chatElementRef.directConnection = {
  openAI: {
    key: 'placeholder-key',
    realtime: {
      config: {
        tools: [
          {
            type: 'function',
            name: 'get_current_weather',
            description: 'Get the current weather in a given location',
            parameters: {
              type: 'object',
              properties: {
                location: {
                  type: 'string',
                  description: 'The city and state, e.g. San Francisco, CA',
                },
                unit: {type: 'string', enum: ['celsius', 'fahrenheit']},
              },
              required: ['location'],
            },
          },
        ],
        function_handler: (functionsDetails) => {
          const {location} = JSON.parse(functionsDetails.arguments);
          return this.getCurrentWeather(location);
        },
      },
    },
  },
};

OpenAIRealtimeMethods​

  • Type: {
         updateConfig?: (config?: OpenAIRealtimeConfig) => void,
         sendMessage?: (text: string, role?: 'user' | 'assistant' | 'system') => void
    }

updateConfig updates the session configuration. Info.
sendMessage sends a message to the session. Info.

info

These methods are automatically assigned and do not need to be defined.
To use these methods, the realtime property must be an object and not a boolean (using realtime: {} is fine).

<deep-chat directConnection='{"openAI": {"key": "key","realtime": {}}}'></deep-chat>;

// retrieve element reference via your framework of choice

chatElementRef.openAI.realtime.methods.updateConfig({instructions: 'Respond with only yes or no'});

chatElementRef.openAI.realtime.methods.sendMessage('Are you a real human?', 'user');

SpeechToSpeechEvents​

  • Functions: { started?: () => void, stopped?: () => void }
  • Events: sts-session-started | sts-session-stopped

started/sts-session-started is triggered when the conversion has started.
stopped/sts-session-stopped is triggered when the conversion has stopped. 

chatElementRef.directConnection.openAI.realtime.events = {
  started: () => console.log('Session started'),
  stopped: () => console.log('Session stopped'),
};

OpenAIRealtimeLoading​

  • Type: {text?: string, html?: string, display?: boolean, style?: CustomStyle}
  • Default: {text: "Loading", "display": true}

Configuration for the loading message.
text is the text that will be displayed when loading.
html can be used to render a special loading element.
display toggles whether the loading message is displayed.
style is the general styling for the loading message. 

<deep-chat
  directConnection='{
    "openAI": {
      "realtime": {
        "loading": {
          "html": "<div class=\"lds-ripple\"><div></div><div></div></div>",
          "style": {"top": "20px"}
      }}}}'
></deep-chat>

OpenAIRealtimeError​

Configuration for the error message.
text is the text that will be displayed for the error message.
style is the general styling for the error message. 

<deep-chat
  directConnection='{
    "openAI": {
      "realtime": {
        "error": {
          "text": "Custom Error!",
          "style": {"color": "orange"}
        }}}}'
></deep-chat>