After a model service is running, its call volume, latency, error rate, and model performance need to be continuously monitored. DataBuddy provides three types of capabilities: \\run metrics, real-time logs, and inference tables\\, covering the full chain from online troubleshooting to offline model quality monitoring.
Prerequisites
A model service has been created and is running. For the basic process, see Deploy an Online Inference Service (REST API).
Before enabling inference tables, you must have the Use Catalog, Use Schema, and Create Table permissions on the target Catalog / Schema.
Viewing Runtime Metrics
The platform provides two monitoring views, located on the service page and the version page respectively:
Call Monitoring (tab under Service Details > Service Version): displays call statistics (number of calls) by time range (last 1 hour, last 24 hours, last 7 days, or a custom time range) and by call method (such as http).
Metric | Description |
Number of Received Requests | Total number of requests received by the service. |
Number of Successful Requests | Number of requests that are returned normally. |
Number of Failed Requests | Number of requests that return errors. |
Number of Throttled Requests | Number of requests rejected due to rate limiting. |
Monitoring (tab under Version Details > Instance List): view traffic information by time range and time granularity (all / 1 minute, and so on) and by instance.
Type | Metric Value | Description |
Traffic information | Network traffic (MBytes) | Downstream (ingress) traffic for network data transmission of a service instance. It reflects the communication load of the Model Services module. Burst traffic may indicate a surge in requests or an abnormal call, and it can be analyzed together with QPS. |
Traffic information | QPS (queries/second) | Number of requests processed per second (queries per second). It is a core metric for measuring service throughput. |
Traffic information | QPS rate limiting (queries/second) | A mechanism that actively limits the maximum requests per second. If the allowed number of requests is exceeded, the additional requests will be rejected. |
Traffic information | Number of concurrent requests | Number of requests being processed at the same time (including requests waiting to be processed and those being calculated). |
Resource information | CPU utilization (%) | CPU usage percentage of a service instance. |
Resource information | MEM utilization (%) | Memory usage percentage of a service instance. |
Resource information | GPU memory utilization (%) | GPU memory usage percentage of a service instance. |
Resource information | GPU utilization (%) | Load percentage of GPU computing units of a service instance. |
Instance information | Number of instances | Total number of configured service instances (including those in all states such as Starting, Running, and Abnormal). |
Instance information | Number of running instances | Number of instances that actually process requests normally. |
Viewing Instance Logs
If Service Log Delivery (Log Service) is enabled when the service is created, you can view instance logs on the Version Details > Instance List > Logs tab.
1. Select the instance name and time range (such as the last 24 hours, or a custom time range).
2. You can enable Auto Refresh to continuously pull the latest logs, and enable Log Wrap to optimize the display of long logs.
3. In the search box, search for keywords (such as error codes or Trace IDs) by whole word.
Instance logs are mainly used to troubleshoot system-level issues (connection timeouts, model loading failures, and memory overflows). For in-depth analysis such as feature drift and performance monitoring based on requests and responses, use the inference table capability.
Tip
CLS stores logs for 15 days by default. For long-term analysis, you are advised to also enable inference tables to persist details to the Catalog.
Querying Instance Logs and Restarting Instances via SDK
In addition to the console, you can also use the MLflow Deployments SDK to pull instance logs by time range with pagination and restart abnormal instances in Notebook / Python tasks. For details about SDK integration, see Deploying Online Inference Services.
Pull instance logs by time range with pagination:
import datetime as dt
end = dt.datetime.now(dt.timezone(dt.timedelta(hours=8)))
start = end - dt.timedelta(hours=1)
ctx = None
while True:
resp = client.get_pod_logs(
service_id=service_id,
pod_name=f"{service_id}-*", # Supports wildcard matching for instances under this service
limit=200, # The recommended number of records per request is less than or equal to 500.
start_time=start.isoformat(), # ISO 8601. Make sure to include the time zone (for example, +08:00).
end_time=end.isoformat(),
context=ctx, # Pagination context from the previous response.
)
for entry in resp["logs"]:
print(entry.get("timestamp"), entry.get("pod_name"), entry.get("message"))
ctx = resp["context"]
if not ctx: # An empty string indicates that the last page has been reached.
break
Restart (rebuild) a specified instance to handle exceptions such as memory overflows and persistent health check failures:
resp = client.restart_pod(service_id=service_id, pod_name="ms-xxxxxx-1abc-xxxx")
print(resp["request_id"])
Note:
Prerequisite for enabling logs: Log delivery is enabled when the service is created (log_enable=True). Otherwise, get_pod_logs returns empty.
The time parameter must be in ISO 8601 format with a time zone (for example, +08:00). Using local time without a time zone can easily cause logs to be missed due to time zone misalignment.
restart_pod does not support wildcards. You must pass a specific instance name, which can be obtained from the pod_name field returned by get_pod_logs.
Viewing Service Events
Go to the Version Details > Instance List > Events tab to view runtime events of the underlying instance resources. These events help you troubleshoot issues such as replica startup failures and health check exceptions. The event list fields are as follows:
Field | Description |
First Occurrence Time / Last Occurrence Time | Time when the event first and most recently occurred. |
Level. | Normal (normal) or Warning (alarm).
|
Resource Type | Underlying resource type that triggers the event. You can filter by type, including Pod, Deployment, ReplicaSet, Service, Ingress, Endpoints, HorizontalPodAutoscaler, DistDeployment, and PyTorchJob. |
Resource Name | Specific resource name that triggers the event, such as instance name ms-xxxxxx-1-xxxx. |
Detailed Description | Event details, such as Readiness probe failed and pod group is ready. |
Occurrences | Total number of times this event has occurred. |
You can enable Auto Refresh to continuously pull the latest events.
Tip
Events are retained only for the last 15 days. Review them as soon as possible.
Events are used to troubleshoot \\sudden-change\\ issues such as abnormal replica exits, persistent health check failures, and ineffective scaling operations. They are the preferred entry point for diagnosing instance-level faults.
Service Log Delivery and Inference Table Collection
You can deliver inference service logs to a specified CLS instance. You can also enable Inference Table collection to write request and response payloads to a Delta table in the Catalog in real time, providing factual data for subsequent quality monitoring.
Enabling When Creating a Service
After enabling the switch under Advanced Configuration > Service Log Delivery when creating a service, configure CLS delivery:
Parameter | Description | Required | Default Value |
Logset | CLS logset, used for delivering service logs. | Yes | - |
Log topic | CLS log topic, used for delivering service logs. | Yes | — |
Then enable Inference Table Collection and configure the following in order:
Parameter | Description | Required | Default Value |
Catalog | Catalog to which the inference table belongs. The drop-down list displays only catalogs for which the current account has the Use Catalog permission. | Yes | — |
Schema | Schema to which the inference table belongs. The drop-down list displays only schemas for which the current account has the Use Schema and Create Table permissions. | Yes | — |
Table Name Prefix | Prefix of the inference table name. If the prefix is specified, the table name is {prefix}; if the prefix is not specified, the table name defaults to {service_name}_payload. | No | — |
The platform automatically creates a Delta table in the specified Catalog/Schema and maintains the field structure and sample data.
Enabling or Disabling on a Running Service
On the service details page, go to Edit Advanced Configuration:
If it is not enabled during creation, you can enable it later when editing.
If it is enabled during creation, you can disable it later when editing. After it is disabled, the historical inference tables remain in the Catalog, and only subsequent writes are stopped.
Common Fields in Inference Tables
Fields included by the platform by default:
Request timestamp.
The unique request ID (Trace ID).
Enter the payload (JSON).
Output the payload (JSON).
Model name, model version, and service version.
Specific fields are automatically adapted based on the model type (classification, regression, or deep learning). Go to the Overview page of the corresponding table in the Catalog to view the complete Schema, sample data, and table properties.
Initiating Quality Monitoring Based on Inference Tables
After inference tables are stored in the Catalog, you can create quality monitoring tasks on the inference table details page or in the data quality module, using the inference table as the fact table for long-term monitoring of data drift, model performance, fairness, and other metrics. For details about specific metrics and configuration methods, see Data Quality.
FAQs
Q: Why is there no data in the inference table?
A: Check the following items in order: ① The inference table switch is enabled in the advanced service configuration. ② The service has real traffic. ③ The current account has read permission on the Catalog/Schema where the inference table resides. If all the above conditions are met but there is still no data, go to the Logs tab on the version details page to check whether there are any errors related to Catalog write failures.
Q: What is the difference between CLS and inference tables?
A: CLS is a real-time troubleshooting tool that stores system-level request and response logs with a short retention period, making it unsuitable for long-term analysis. Inference tables are structured offline fact tables stored in the Catalog. They can be queried by SQL, used for drift monitoring, and serve as a source of retraining samples.
References
Deploy an Online Inference Service (REST API).
Data Quality