DataBuddy supports two types of comparison scenarios: comparing different experiments and comparing different runs within the same experiment. This document describes the steps and related features of these two comparison methods.
Prerequisites
The current account has read or higher permissions on the target experiment.
Note:
Experiment comparison is not currently supported for agent experiments.
Comparing Different Experiments
When you perform a horizontal comparison across multiple experiments of the same type, you can compare experiments across the experiment list page.
1. Log in to the DataBuddy console and choose Data Science > Model Experiments in the left sidebar. 2. In the experiment list, select two or more experiments of the same type by using the checkboxes on the left.
3. Click Compare Experiments that appears above the list to go to the cross-experiment comparison window.
Tip
Cross-experiment comparison is mainly used to compare the final results of different solutions at a macro level. To compare the differences between runs at a fine-grained level, go to a single experiment and perform run comparison.
Comparing Different Runs of the Same Experiment
An experiment typically contains multiple runs (with different hyperparameter combinations, training data, and so on). After you go to the experiment details, you can view the performance of each run in table view or chart view, and select multiple runs for item-by-item comparison.
1.Entering Experiment Details
1. Log in to the DataBuddy console and choose Data Science > Model Experiments in the left sidebar. 2. In the experiment list, click the name of the target experiment to go to the experiment details.
2. Viewing Runs and Models
After you go to the experiment details, select a view mode by using the view switcher above the list:
Run view (default): Displays all training records by run.
Table mode: Displays basic information about runs.
You can sort runs by creation time in ascending or descending order.
The search box above the list allows you to search by run name or run id.
Chart mode: Click the view switcher to switch to the chart view, which compares the metrics of each run in a bar chart. You can search for chart metrics, hide or show specific runs, and download metric charts.
Model view: Click the switcher to switch to the model view and view the model versions registered to Catalog in this experiment.
Table mode: Displays basic information about models. The operation column provides two operations: registering the model to Catalog and deleting the model.
Chart mode: Similar to the run chart mode, you can search for metrics and visually compare the evaluation metrics of each model version to easily identify performance differences between versions.
Tip
If multiple versions of the same model are registered in the same run, they are displayed in separate rows. If multiple models are registered in the same run, they are displayed together in a single row.
If a model generated by a run is not bound to an experiment when it is recorded (the source is empty or only Notebook), the model will not appear in the model view.
3. Multi-Select Comparison
In the run table view, select two or more runs by using the checkboxes on the left, and then click Compare runs above to go to the comparison page. The run comparison page provides two comparison modes, which you can switch between by using the tabs at the top:
Value comparison: Displays the selected runs side by side for item-by-item comparison, and provides a Show differences only toggle. When the toggle is enabled, only fields that differ among the selected runs are displayed, helping you focus on key differences.
Chart comparison: You can select different parameter configurations and chart types (such as parallel coordinates, scatter plots, and box plots) to visually compare the results.
4. Selecting the Best Run
The experiment details automatically identify the best run based on evaluation metrics and pin it to the top of the experiment panel.
FAQs
Q: Why Can't I Find My Newly Trained Run in the Table View?
Check the following:
Are you viewing the correct experiment? Check the experiment name in the breadcrumb in the upper-left corner.
Did the run complete successfully? Check the run status in the overview.
Was it filtered out by the filter conditions? Clear the filter conditions and check again.
Q: Why Are Fewer Runs Shown in the Chart View than Actually Exist?
When the number of runs exceeds 10, the chart view displays only the first 10 runs by default, and the other runs are automatically hidden. You can manually expand the hidden items in the metric list to view all of them.
Q: What Should I Do If I Cannot Scroll to the Bottom of the Comparison Window?
When there are too many comparison metrics, the comparison window may have scrollbars that are difficult to reach. We recommend that you first select the difference items to reduce the number of displayed rows before comparing.
Q: Why Is the Model View Empty Even Though the Run View Shows Output Models?
The model view displays only models that are bound to experiments. If the source of a model generated by a run is empty or only Notebook (not bound to an experiment), the model will not appear in the model view. When calling log_model, make sure that experiment_id is set correctly.
References