OpenAI Realtime
OpenAI Realtime
Try this service with custom properties in the Playground.
realtime
- Type: {
ephemeralKey?: string
,
fetchEphemeralKey?: FetchEphemeralKey
,
autoFetchEphemeralKey?: boolean
,
autoStart?: boolean
,
avatar?: OpenAIRealtimeAvatar
,
buttons?: OpenAIRealtimeButtons
,
config?: OpenAIRealtimeConfig
,
loading?: OpenAIRealtimeLoading
error?: OpenAIRealtimeError
} |true
Connect to OpenAI Reatime API. You can either use true
or an object:
ephemeralKey
is the temporary session key used to connect from the browser safely.
fetchEphemeralKey
is a function definition used to retrieve the ephemeralKey
.
autoFetchEphemeralKey
triggers fetchEphemeralKey
on component render.
autoStart
begins the conversation on component render.
avatar
is used to configure the avatar.
buttons
is used to configure the buttons.
config
is as object that defines the connection properties.
loading
defines the styling for the loading message.
error
defines the styling for the error message.
- Sample code
- Full code
<deep-chat
directConnection='{
"openAI": {"key": "placeholder key", "realtime": true}
}'
></deep-chat>
<!-- This example is for Vanilla JS and should be tailored to your framework (see Examples) -->
<deep-chat
directConnection='{
"openAI": {"key": "placeholder key", "realtime": true}
}'
style="border-radius: 8px"
></deep-chat>
If ephemeralKey
or fetchEphemeralKey
properties are used, the key
property is not required. If the key
property is used instead, the ephemeralKey
will be fetched automatically.
Types
FetchEphemeralKey
- Type: () =>
string
|Promise<string>
This is a function definition that will return the ephemeralKey
value programmatically.
Details on how to retrieve the key can be found here.
- Sample function
- OpenAI function
// There are many ways to define the function depending on your framework.
// The following is a simple VanillaJs example:
chatElementRef.directConnection.openAI.realtime.fetchEphemeralKey = () => 'my-key';
// There are many ways to define the function depending on your framework.
// The following is a simple VanillaJs example:
// Calls the OpenAI service to fetch your ephemeral key
chatElementRef.directConnection.openAI.realtime.fetchEphemeralKey = async () => {
try {
const response = await fetch('https://api.openai.com/v1/realtime/sessions', {
method: 'POST',
headers: {
Authorization: 'Bearer OPENAI-API-KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'gpt-4o-realtime-preview-2024-12-17',
voice: 'verse',
}),
});
const data = await response.json();
return data.client_secret.value;
} catch (error) {
console.error('Error fetching ephemeral key:', error);
throw error;
}
};
See the OpenAIRealtimeConfig
to see available session configuration.
OpenAIRealtimeAvatar
- Type: {
src?: string
,
maxScale?: number
,
styles?:
{image?: CustomStyle
,container?: CustomStyle
}
} - Default: {maxScale: 2.5}
Configuration for the avatar.
src
is the path for the image.
maxScale
is the maximum size that the avatar is allowed to expand to when a response is spoken.
styles
is an object that is used to style the avatar image
and the container
which is the area around the avatar.
- Sample code
- Full code
<deep-chat
directConnection='{
"openAI": {
"realtime": {
"avatar": {
"src": "path-to-file.png",
"maxScale": 3.5,
"styles": {
"image": {"padding": "5px"},
"container": {"height": "60%"}
}}}}}'
></deep-chat>
<!-- This example is for Vanilla JS and should be tailored to your framework (see Examples) -->
<deep-chat
directConnection='{
"openAI": {
"key": "placeholder key",
"realtime": {
"avatar": {
"src": "path-to-file.png",
"maxScale": 3.5,
"styles": {
"image": {"padding": "5px"},
"container": {"height": "60%"}
}}}}}'
style="border-radius: 8px"
></deep-chat>
OpenAIRealtimeButtons
Configuration for the buttons section.
container
is styling for the buttons area.
microphone
is styling for the various states of the microphone button.
toggle
is styling for the various states of the toggle button.
- Sample code
- Full code
<deep-chat
directConnection='{
"openAI": {
"realtime": {
"buttons": {
"container": {"backgroundColor": "#f7f7f7"},
"microphone": {
"default": {
"svg": {"content": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"800px\" height=\"800px\" viewBox=\"0 0 24 24\" fill=\"none\">\n<path d=\"M3 16.5C2.59 16.5 2.25 16.16 2.25 15.75V8.25C2.25 7.84 2.59 7.5 3 7.5C3.41 7.5 3.75 7.84 3.75 8.25V15.75C3.75 16.16 3.41 16.5 3 16.5Z\" fill=\"#292D32\"/>\n<path d=\"M7.5 19C7.09 19 6.75 18.66 6.75 18.25V5.75C6.75 5.34 7.09 5 7.5 5C7.91 5 8.25 5.34 8.25 5.75V18.25C8.25 18.66 7.91 19 7.5 19Z\" fill=\"#292D32\"/>\n<path d=\"M12 21.5C11.59 21.5 11.25 21.16 11.25 20.75V3.25C11.25 2.84 11.59 2.5 12 2.5C12.41 2.5 12.75 2.84 12.75 3.25V20.75C12.75 21.16 12.41 21.5 12 21.5Z\" fill=\"#292D32\"/>\n<path d=\"M16.5 19C16.09 19 15.75 18.66 15.75 18.25V5.75C15.75 5.34 16.09 5 16.5 5C16.91 5 17.25 5.34 17.25 5.75V18.25C17.25 18.66 16.91 19 16.5 19Z\" fill=\"#292D32\"/>\n<path d=\"M21 16.5C20.59 16.5 20.25 16.16 20.25 15.75V8.25C20.25 7.84 20.59 7.5 21 7.5C21.41 7.5 21.75 7.84 21.75 8.25V15.75C21.75 16.16 21.41 16.5 21 16.5Z\" fill=\"#292D32\"/>\n</svg>"}
}},
"toggle": {
"default": {
"svg": {
"styles": {
"default": {"filter": "brightness(0) saturate(100%) invert(58%) sepia(77%) saturate(282%) hue-rotate(172deg) brightness(91%) contrast(93%)" }
}}}}}}}}'
></deep-chat>
<!-- This example is for Vanilla JS and should be tailored to your framework (see Examples) -->
<deep-chat
directConnection='{
"openAI": {
"key": "placeholder key",
"realtime": {
"buttons": {
"container": {"backgroundColor": "#f7f7f7"},
"microphone": {
"default": {
"svg": {"content": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"800px\" height=\"800px\" viewBox=\"0 0 24 24\" fill=\"none\">\n<path d=\"M3 16.5C2.59 16.5 2.25 16.16 2.25 15.75V8.25C2.25 7.84 2.59 7.5 3 7.5C3.41 7.5 3.75 7.84 3.75 8.25V15.75C3.75 16.16 3.41 16.5 3 16.5Z\" fill=\"#292D32\"/>\n<path d=\"M7.5 19C7.09 19 6.75 18.66 6.75 18.25V5.75C6.75 5.34 7.09 5 7.5 5C7.91 5 8.25 5.34 8.25 5.75V18.25C8.25 18.66 7.91 19 7.5 19Z\" fill=\"#292D32\"/>\n<path d=\"M12 21.5C11.59 21.5 11.25 21.16 11.25 20.75V3.25C11.25 2.84 11.59 2.5 12 2.5C12.41 2.5 12.75 2.84 12.75 3.25V20.75C12.75 21.16 12.41 21.5 12 21.5Z\" fill=\"#292D32\"/>\n<path d=\"M16.5 19C16.09 19 15.75 18.66 15.75 18.25V5.75C15.75 5.34 16.09 5 16.5 5C16.91 5 17.25 5.34 17.25 5.75V18.25C17.25 18.66 16.91 19 16.5 19Z\" fill=\"#292D32\"/>\n<path d=\"M21 16.5C20.59 16.5 20.25 16.16 20.25 15.75V8.25C20.25 7.84 20.59 7.5 21 7.5C21.41 7.5 21.75 7.84 21.75 8.25V15.75C21.75 16.16 21.41 16.5 21 16.5Z\" fill=\"#292D32\"/>\n</svg>"}
}},
"toggle": {
"default": {
"svg": {
"styles": {
"default": {"filter": "brightness(0) saturate(100%) invert(58%) sepia(77%) saturate(282%) hue-rotate(172deg) brightness(91%) contrast(93%)" }
}}}}}}}}'
style="border-radius: 8px"
></deep-chat>
OpenAIRealtimeButton
Configuration for the various sates of a button.
OpenAIRealtimeConfig
- Type: {
model?: string
,
instructions?: string
,
voice?: string
,
temperature?: number
,
max_response_output_tokens?:
number
|"inf"
,
turn_detection?:
{
type?: string
,
threshold?: string
,
prefix_padding_ms?: number
,
silence_duration_ms?: number
}
tools?:
{
type?:
"function"
|"code_interpreter"
|"file_search"
,
name?: string
,
description?: number
,
parameters?: object
}[]
tool_choice?: string
,
function_handler?: OpenAIRealtimeFunction
} - Default: {model: "gpt-4o-realtime-preview-2024-12-17"}
Session configuration. This object follows the configuration referenced here.
IMPORTANT: If you are providing your own ephemeralKey
or using the fetchEphemeralKey
function, this configuration will need to be used there.
However, this does not apply to the function_handler
.
model
is the model name used for the session.
instructions
allows the model to be guided on desired responses (also known as system message).
voice
is the model response's voice. See full list here.
temperature
is the sampling temperature for the model.
max_response_output_tokens
is the maximum number of output tokens for a single assistant response.
turn_detection
is configuration to detect when the model should start speaking. More info can be found here.
tools
is a list of tools available to the model. More info can be found here.
tool_choice
is used to define how the model chooses tools.
- Sample code
- Full code
<deep-chat
directConnection='{
"openAI": {
"realtime": {
"config": {
"instructions": "Answer only with yes or no",
"voice": "alloy"
}}}}'
></deep-chat>
<!-- This example is for Vanilla JS and should be tailored to your framework (see Examples) -->
<deep-chat
directConnection='{
"openAI": {
"key": "placeholder key",
"realtime": {
"config": {
"instructions": "Answer only with yes or no",
"voice": "alloy"
}}}}'
style="border-radius: 8px"
></deep-chat>
OpenAIRealtimeFunction
- Type: ({
name: string
,arguments: string
}) =>object
The actual function that the component will call if the model wants a response from OpenAIRealtimeConfig tools
functions.
- Sample code
- Full code
// using JavaScript for a simplified example
chatElementRef.directConnection = {
openAI: {
key: 'placeholder-key',
realtime: {
config: {
tools: [
{
type: 'function',
name: 'get_current_weather',
description: 'Get the current weather in a given location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g. San Francisco, CA',
},
unit: {type: 'string', enum: ['celsius', 'fahrenheit']},
},
required: ['location'],
},
},
],
function_handler: (functionsDetails) => {
const {location} = JSON.parse(functionsDetails.arguments);
return this.getCurrentWeather(location);
},
},
},
},
};
// using JavaScript for a simplified example
chatElementRef.directConnection = {
openAI: {
key: 'placeholder-key',
realtime: {
config: {
tools: [
{
type: 'function',
name: 'get_current_weather',
description: 'Get the current weather in a given location',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g. San Francisco, CA',
},
unit: {type: 'string', enum: ['celsius', 'fahrenheit']},
},
required: ['location'],
},
},
],
function_handler: (functionsDetails) => {
const {location} = JSON.parse(functionsDetails.arguments);
return this.getCurrentWeather(location);
},
},
},
},
};
function getCurrentWeather(location) {
location = location.toLowerCase();
if (location.includes('tokyo')) {
return {location, temperature: '10', unit: 'celsius'};
} else if (location.includes('san francisco')) {
return {location, temperature: '72', unit: 'fahrenheit'};
} else {
return {location, temperature: '22', unit: 'celsius'};
}
}
OpenAIRealtimeLoading
- Type: {
text?: string
,html?: string
,display?: boolean
,style?: CustomStyle
} - Default: {text: "Loading", "display": true}
Configuration for the loading message.
text
is the text that will be displayed when loading.
html
can be used to render a special loading element.
display
toggles whether the loading message is displayed.
style
is the general styling for the loading message.
- Sample code
- Full code
<deep-chat
directConnection='{
"openAI": {
"realtime": {
"loading": {
"html": "<div class=\"lds-ripple\"><div></div><div></div></div>",
"style": {"top": "20px"}
}}}}'
></deep-chat>
<!-- This example is for Vanilla JS and should be tailored to your framework (see Examples) -->
<deep-chat
directConnection='{
"openAI": {
"key": "placeholder key",
"realtime": {
"loading": {
"html": "<div class=\"lds-ripple\"><div></div><div></div></div>",
"style": {"top": "20px"}
}}}}'
auxiliaryStyle="
.lds-ripple {
color: #1c4c5b
}
.lds-ripple,
.lds-ripple div {
box-sizing: border-box;
}
.lds-ripple {
display: inline-block;
position: relative;
width: 80px;
height: 80px;
}
.lds-ripple div {
position: absolute;
border: 4px solid currentColor;
opacity: 1;
border-radius: 50%;
animation: lds-ripple 1s cubic-bezier(0, 0.2, 0.8, 1) infinite;
}
.lds-ripple div:nth-child(2) {
animation-delay: -0.5s;
}
@keyframes lds-ripple {
0% {
top: 36px;
left: 36px;
width: 8px;
height: 8px;
opacity: 0;
}
4.9% {
top: 36px;
left: 36px;
width: 8px;
height: 8px;
opacity: 0;
}
5% {
top: 36px;
left: 36px;
width: 8px;
height: 8px;
opacity: 1;
}
100% {
top: 0;
left: 0;
width: 80px;
height: 80px;
opacity: 0;
}
}"
style="border-radius: 8px"
></deep-chat>
OpenAIRealtimeError
- Type: {
text?: string
,style?: CustomStyle
} - Default: {text: "Error"}
Configuration for the error message.
text
is the text that will be displayed for the error message.
style
is the general styling for the error message.
- Sample code
- Full code
<deep-chat
directConnection='{
"openAI": {
"realtime": {
"error": {
"text": "Custom Error!",
"style": {"color": "orange"}
}}}}'
></deep-chat>
<!-- This example is for Vanilla JS and should be tailored to your framework (see Examples) -->
<deep-chat
directConnection='{
"openAI": {
"key": "placeholder key",
"realtime": {
"error": {
"text": "Custom Error!",
"style": {"color": "orange"}
}}}}'
style="border-radius: 8px"
></deep-chat>