The operating cost and availability of a model service are primarily determined by resource configuration. This document describes how to select compute resources for a service version, set the number of replicas, and adjust the running status of the service through the console or SDK.
Prerequisites
You have created a model service and completed the basic information and version configuration. For details on how to create a service, see Deploying an Online Inference Service.
The current account has permission to edit model services in the workspace.
Selecting Compute Resources
When creating or editing a service version, select Platform Pay-as-You-Go (Serverless) for Resource Source. In this mode:
The platform bills based on the running duration of replicas, eliminating the need to purchase machine resource groups in advance.
Resource pools are uniformly scheduled by the platform, and replicas can be dynamically scaled up or down based on request volume.
Billing stops immediately after the service is stopped.
Resource specifications must be selected based on the model's compute requirements. Examples of common tiers:
Type | Typical Specifications | Applicable models |
CPU | Multiple CPU Specifications | Traditional machine learning models (tree models, linear models, and lightweight recommendation models). |
GPU | Specifications with GPU Card | Deep learning models (Image Analysis, NLP, and vector search). |
Tip
The specification of a single replica and the number of replicas jointly determine the service capacity. We recommend that you first launch a stress test with one replica and a medium specification, and then scale out based on QPS/latency metrics.
Sets the number of replicas
Enter the target number of replicas on the version configuration or service details page, and the platform immediately scales to the specified number. Multiple replicas improve service availability and are suitable for temporary stress testing, urgent scaling out, and stable production services.
Updating a Service
After the service is running, click Edit on the service details page to go to Update Service and adjust resources and running parameters:
Modifiable: configuration items such as service description, resource specifications, request throttling, number of replicas, replica adjustment, scheduling policy, and automatic stop.
Non-modifiable: service name (grayed out on the page).
Change Model Version:
In the console, the Update Service operation does not modify the deployed model. To change the model version, use Add Version on the service details page.
SDK: update_deployment supports changing the model version through the model_uri parameter.
The Configuration Overview on the right side of the page summarizes this configuration in real time.
Tip
After resource specifications are modified, the specific restart behavior of replicas is subject to the actual platform implementation.
Adjusting Resources and Runtime Status via SDK
In addition to the console, you can also use the MLflow Deployments SDK to adjust the number of replicas (config={"replicas": N}) and start or stop services (service_action set to STOP / RESUME / SCALE) in Notebook / Python tasks. For details about the calling methods and precautions, see Update, Scaling, and Version Switching in Deploying Online Inference Services.
FAQs
Q: Can the service still be called when the number of replicas is 0?
A: No. When the number of replicas is 0, the service enters the stopped state and the calling address returns a failure. To temporarily stop the service and stop billing, click Stop on the service details page, or call update_deployment(config={"service_action": "STOP"}) through the SDK.
References
Deploy an Online Inference Service (REST API)