Skip to main content

Web Model

Run a chat model entirely on your browser. No need to connect to any service.

Video demo

Setup​

See the External Modules section on how to integrate the deep-chat-web-llm module into your project. Then proceed with using the webModel property:

webModel​

Set this to true or define an object with any of the following properties:
model is the name of the model to be used. See full list.
instruction directs how the model should respond.
urls defines the endpoint to retrieve the web model assets.
load defines how and when the model is loaded.
introMessage is the configuration for the introductory web model message.
worker is a Web Worker that can enhance the rendering performance. Example.

Example​

<deep-chat webModel="true"></deep-chat>

Types​

Object types for webModel:

WebModelUrls​

  • Type: {model?: string, wasm?: string}

Deep Chat uses the webModelConfig.ts file to determine where the model and the wasm files will be downloaded from. You can overwite the links to your prefered locations like your own server.

Example​

<deep-chat
  webModel='{
    "urls": {
      "model": "http://localhost:8080/model-file",
      "wasm": "http://localhost:8080/wasm-file"
  }}'
></deep-chat>

WebModelLoad​

  • Type: {onInit?: boolean, onMessage?: boolean, clearCache?: boolean, skipCache?: boolean}

This is an object that is used to define the web model loading/downloading behaviour:
onInit will start loading the model as soon as the component is rendered.
onMessage will start loading the model when the user submits a message (or clicks the Start button).
clearCache is used to remove all the cached web model files in the browser and replace them with new ones.
skipCache will not use the browser cache. This is useful for trying out multiple models without overfilling the cache.

Example​

<deep-chat webModel='{"load": {"onMessage": true}}'></deep-chat>

WebModelIntro​

  • Type: {
         displayed?: boolean,
         autoScroll?: boolean
         removeAfterLoad?: boolean,
         removeAfterMessage?: boolean,
         initialHtml?: string,
         downloadClass?: string,
         uploadClass?: string,
         fileInputClass?: string,
         afterLoadHtml?: string,
         exportFilesClass?: string
    }
  • Default: {displayed: true, autoScroll: true}

This is an object that controls the introductory web model message behaviour:
displayed is used to toggle whether the message is visible.
autoScroll toggles whether the chat automatically scrolls to the intro message.
removeAfterLoad removes the message after the model files have successfully loaded.
removeAfterMessage removes the message after the model files have loaded and the user types a message.
initialHtml is a string that contains html used for the initial message.
downloadClass is the name of the class used for the button that downloads the model files.
uploadClass is the name of the class used for the button that upload model files.
fileInputClass is the name of the class used for the hidden file input element that toggles the file upload functionality.
afterLoadHtml is a string that contains html used for the message displayed after the model has been uploaded.
exportFilesClass is the name of the class used for the button that exports the model files.

Initial Example​

// using JavaScript for a simplified example

chatElementRef.webModel = {
  introMessage: {
    initialHtml: `<button class="start">Start</button> <button class="files">Files</button> <input type="file" class="input" hidden multiple />`,
    downloadClass: 'start',
    uploadClass: 'files',
    fileInputClass: 'input',
  },
};

After Load Example​

Need to first download/upload the model.

// using JavaScript for a simplified example

chatElementRef.webModel = {
  afterLoadHtml: {
    initialHtml: `Finished loading the model! <br /> <button class="export">Export Files</button>`,
    exportFilesClass: 'export',
  },
};
tip

Element styles can be customized using the htmlClassUtilities property.

Worker​

This is a reference to a Web Worker that is used to process chat responses in another thread and improves browser rendering performance. To note, this does not allow model files to be uploaded or exported.

To define its property value, please use the following:
worker: new Worker(new URL('./worker.js', import.meta.url), {type: 'module'})

Create a worker.js (or worker.ts) file in the same folder - which contains the following code:

import {ChatWorkerHandler, ChatModule} from 'deep-chat-web-llm';

const chat = new ChatModule();
const handler = new ChatWorkerHandler(chat);
self.onmessage = (msg) => {
  handler.onmessage(msg);
};

Example​

chatElementRef.webModel = {
  worker: new Worker(new URL('./worker.js', import.meta.url), {type: 'module'}),
};

Troubleshooting​

If the chat displays an error, we recommend trying any of the following options:

If you are using an SSR related framework such as NextJs or SvelteKit and get the following error: unhandledRejection: Error [ReferenceError]: require is not defined in ES module scope...

if (typeof window !== 'undefined') {
  import('deep-chat-web-llm').then((module) => {
    window.webLLM = module;
  });
}

If you are still experiencing issues, please see github issues or create a new issue ticket and we will look into it as soon as possible.

Thankyou​

The deep-chat-web-llm module is a fork based on the mlc-ai/web-llm repository. Make sure to check them out and give them a star!