Skip to main content

​
Usage

​
Description

Launches an OpenAI-compatible inference server for a quantized model. If the model has not been downloaded, it is downloaded automatically before the server starts. By default, the server runs in the foreground and is stopped and removed when you press Ctrl+C. Use --detach to leave the server running in the background after the command exits.

​
Arguments

​
Options

​
Server options

Additional server options can be appended as --key=value pairs or --key boolean flags. These options are passed directly to the backend server.
Server options are backend-specific. The options below apply to the anaconda-desktop backend and map to llama-server parameters. The ai-catalyst backend might support different options.
Common server options for anaconda-desktop include:

​
Examples

Launch a server in the foreground
Launch in the background (detached)
The command returns to the terminal immediately while the server runs in the background. Launch with custom server options
Launch and open the server in a browser