Model experiments are a capability in the DataBuddy data science module for organizing and tracking machine learning training processes. When you train models in Studio, you can use the model experiment module to systematically record the parameters, metrics, and artifacts of each run. After training is complete, you can compare the performance of multiple runs in the experiment panel, select the best model, and register it to Catalog for unified management, eliminating the need to manually maintain training logs or compare results one by one.
Key Concepts
Run
A Run is a specific training or hyperparameter tuning process. Each call to mlflow.start_run() in a Notebook starts a new Run. Within a Run, you can record:
Run ID and Run Name: A unique identifier. The run name must be unique within its experiment.
Run Status: Running, Completed, Failed.
Running Duration: Automatically recorded. The unit switches automatically among ms / s / min / h.
Creator, Creation Time, Source Notebook: Automatically collected from the execution environment.
Dataset: The input data used for this training. You can view its Schema in the details.
Parent and Child Runs
Call mlflow.start_run(nested=True) again within the context of a parent Run, and the new Run becomes a child Run of the parent Run. This is commonly used in hyperparameter tuning scenarios. Example:
import mlflow
with mlflow.start_run(run_name="Parent Run"):
mlflow.log_param("model_type", "CNN")
for i in range(3):
with mlflow.start_run(run_name=f"Child Run {i}", nested=True):
mlflow.log_param("learning_rate", 0.01 * (i + 1))
mlflow.log_metric("accuracy", 0.8 + 0.05 * i)
The Run task list on the experiment details page displays parent Runs and their child Runs in a nested manner.
Parameter
The input configuration recorded at the start of training. The value type can be a string, an integer, or a floating-point number. Common examples:
learning_rate: 0.01
batch_size: 64
n_estimators: 100
max_depth: 6
Record parameters by calling mlflow.log_param(key, value). Once recorded, parameters cannot be modified.
Metric
A model performance metric recorded during or at the end of training. Metric values can vary with the training step, making it easy to plot curves. Common examples:
accuracy, AUC, F1, precision, recall (classification)
RMSE, MAE, R2 (regression)
loss, val_loss (deep learning)
Record metrics by calling mlflow.log_metric(key, value, step=None). You can call it multiple times to record the training curve.
Artifact
All auxiliary files generated by this Run. Each Run has its own independent storage space, and files are displayed in a directory tree. Common contents:
Model files obtained from training (.pkl / .pt / .onnx, and so on).
Log files (stdout, stderr, log.txt)
Configuration files (config.yaml, params.json)
Evaluation results (prediction outputs, evaluation metrics, images)
Record by using mlflow.log_artifact(local_path, artifact_path=None) or mlflow.<flavor>.log_model(...).
System Metrics
System resource usage metrics automatically collected by MLflow are used to evaluate the consumption of computing resources during training. The following metrics are recorded by default:
Metric | Description |
cpu_utilization_percentage
| CPU Utilization |
system_memory_usage_megabytes
| System memory usage |
system_memory_usage_percentage
| System memory usage percentage |
network_receive_megabytes
| Inbound network traffic |
network_transmit_megabytes
| Outbound network traffic |
disk_usage_megabytes
| Used disk capacity |
disk_available_megabytes
| Available disk capacity |
Dataset
The input data used by this Run. In the Run details, you can click to expand a side panel with dataset details to view data sources and field information. A Run can be associated with multiple datasets.
Models and Registered Models
A model object recorded in a Run through mlflow.<flavor>.log_model(...) is saved as an artifact of the Run. A model can be in one of the following two states:
Unregistered Model: exists only as an artifact of the Run and is not governed by Catalog.
Registered Model: After you perform the Register Model operation or pass the registered_model_name= parameter in your code, the model is registered to the MLflow Model Registry under Catalog and has a three-part full name in the format of catalog.schema.model_name and an incremented version number.
Best Run
The best Run in an experiment is automatically identified based on evaluation metrics. In the experiment details list, the best run is pinned to the top by default.
Relationship Between Experiments, Runs, and Models
Experiments, Runs, and models have different relationships in different scenarios:
Scenario | Relationship |
A Run trains and saves a model (basic scenario). | A Run corresponds to a model on a one-to-one basis. |
A Run saves multiple model snapshots across multiple epochs (deep learning). | A Run corresponds to multiple model artifacts (such as torch-iris-10, torch-iris-50, and torch-iris-100). |
Training the same model architecture multiple times with different parameters (hyperparameter tuning) | The same registered model corresponds to multiple Runs, each representing a different model version. |
Parent and child Runs save models separately (for example, hyperparameter tuning experiments). | A parent Run is associated with multiple child Runs, and each child Run is associated with one model artifact. |
Permission Model
Experiment permissions reuse the base platform permission system. By default, each experiment can be managed by its creator and Admin accounts. Other members can grant permissions at the following three levels in the Permissions dialog in the upper-right corner of the experiment details page:
Permission Level | Available Operations |
Read (Can Read) | View basic experiment information, parameters/metrics/models/artifacts of all runs, and Run history; export experiment results; view permission settings (read-only). |
Edit (Can Edit) | Have all capabilities of the Can Read role; create Runs and modify parameters/metrics/artifacts in Runs; edit experiment descriptions and Tags. |
Manage (Can Manage) | Have all capabilities of the Can Edit role; delete experiments and assign permissions to other users. |
In the No Permissions state, the experiment is completely inaccessible to users.
References