Note:
Estimated duration: 90 minutes. Learning objective: Using the Ops scenario of a production data pipeline as an example, run through the complete Ops pipeline end to end, from platform preparation → alarm and retry configuration → real-time monitoring → failure rerun and historical backfill → root cause analysis → usage and capacity management.
Scenario Introduction
You are an Ops engineer who has just joined the data platform team. The team is responsible for supporting an e-commerce order ETL workflow that runs daily at 00:00. Your daily work includes:
Grant workspace permissions to new colleagues.
Configure alarm and automatic retry policies once in the production workflow to avoid missed alerts.
Receive alarms immediately when a task fails in the early morning, and locate the failed task within 5 minutes to decide whether to rerun it or backfill data.
Quickly locate root causes through logs and intelligent diagnostics, and create a review that can be shared with upstream and downstream teams.
This tutorial walks you through the complete workflow, with all paths aligned to the Platform Management, Workflow, and Workflow Run modules in the workspace.
Prerequisites
Before starting this tutorial, make sure the following prerequisites are met:
You have activated the DataBuddy service and completed workspace initialization.
The current account has the Workspace Member role in the target workspace.
You have at least one published and daily scheduled workflow. You can use the quickstart_orders_etl workflow produced in Data Engineer Quick Start. This workflow should have execution records for at least 7 days in Workflow Run to demonstrate historical trends and data backfill. You have prepared an email address and a WeCom group bot (or a Slack/Teams Incoming Webhook URL) for receiving test alarms.
Tutorial Scenario
The main Ops scenario in this tutorial:
Scenario | Description |
Workflow | quickstart_orders_etl, runs daily at 00:00 and contains two tasks: sync_mysql2dlc and pyspark_transform.
|
Channels for receiving alarms | Team email data-ops@example.com + WeCom group bot. |
Fault injection test | Intentionally stop the upstream MySQL instance temporarily to make sync_mysql2dlc fail; after recovery, practice rerunning + backfilling data. |
Step 1: Adding Members and Assigning Space Roles
When a new colleague joins, the first step is to add them to the workspace and grant appropriate permissions based on their responsibilities.
1.1 Adding a CAM Sub-Account to the Console
1. Log in to the DataBuddy console, and choose User Management > Users in the left navigation pane. 2. Click Add User and select the sub-account to add.
3. Console Role: Select Console Member for all users (can only access workspaces they are added to and cannot create new workspaces).
4. Click Add to complete the addition.
1.2 Inviting Members to the Target Workspace and Assigning Roles
1. Go to the target workspace, and choose Workspace > User Management > Users in the left navigation pane.
2. Click Add User to add the user from the previous step to the workspace and grant them the workspace member role, which has full read, write, and delete permissions except for workspace management.
3. Click **Add** to complete the addition.
Step 2: Configuring Alarm Notification Channels
Notification channels are workspace-level alarm outlets, and all entities that support alarms (workflows, tasks, data quality rules, and so on) share these channels.
1. Go to Workspace > Notification Management.
2. Click Add Notification Channel and configure the following two channels in sequence:
2.1 Configuring Email Channels
Parameter | Value |
Type | Email |
Name | email_data_ops
|
Address | data-ops@example.com (Separate multiple email addresses with line breaks or half-width commas; line breaks are not supported).
|
Click Add to save.
2.2 Configuring an Enterprise IM Channel (Webhook / Slack / Teams, Choose One)
Take a WeCom group bot as an example. Use the Webhook type:
Parameter | Value |
Type | Webhook |
Name | webhook_wechat_ops
|
URL | Incoming Webhook URL generated by the group bot on WeCom |
Username / Password | Leave empty in general and configure according to bot-side rules. |
If your team uses Slack / Teams, create an Incoming Webhook on the corresponding platform first, and then select the corresponding type in DataBuddy and enter the URL. For details, see Configure Notification Channels. Note:
Notification channels are workspace-level configurations and are not shared across workspaces. To use the same channel in multiple workspaces, configure it separately in each workspace. Before deleting a channel, make sure it is not referenced by any alarm rules. Otherwise, alarms that reference the channel cannot be sent after deletion.
After an email channel is configured in a workspace, it cannot be directly referenced for alarms. You need to manually enter it in the alarm channel.
Step 3: Configuring Alarms and Retries for Workflows
Attach the notification channel configured in the previous step to the workflow, and then add an automatic retry for critical tasks.
3.1 Configuring Workflow-Level Alarms
1. In the left navigation pane, select Workflow, and click quickstart_orders_etl to go to the workflow details page.
2. In the Monitoring Metrics area, click Add monitoring metrics:
Metric: Running Duration.
Warning Threshold: 01h00m00s (The tutorial workflow normally completes within 10 minutes. If it runs for more than 1 hour, it is considered abnormal).
Timeout Threshold: 02h00m00s (If the duration exceeds 2 hours, a retry on failure is triggered or the task is marked as failed).
1. Configure alarms in **Alarm Configuration**:
Receiving Channel: Click Add Channel, select Personal Email, and enter the email address of email_data_ops in the input box. Select Webhook and choose webhook_wechat_ops.
Alarm Conditions: Select Failure (selected by default and required) + Monitoring Metrics (used for timeout alarms).
Do Not Disturb: Select Manual Termination (no alarm is sent when a task is manually terminated, preventing false alarms for on-call personnel).
1. Click Confirm to save.
3.2 Configuring Automatic Retries for Critical Tasks
1. Go to the workflow, select the Task Orchestration tab to enter the Workflow Editor canvas, and click the sync_mysql2dlc task node in the canvas.
2. In the task configuration panel, locate Failure and Timeout Policy:
Parameter | Recommended Value | Description |
Retry after Execution Failure | On | Automatically recover when network jitter occurs or the upstream is temporarily unreachable. |
Behavior after Timeout | Trigger Retry | - |
Maximum Number of Retries | 3
| Mark as failed if retries fail three times. |
Retry Interval | 5 minutes
| Allow sufficient time for the upstream to recover. |
1. Click Confirm.
Note:
Task-level alarms and workflow-level alarms do not interfere with each other: We recommend that you configure overall failure and overall timeout alarms at the workflow level, and add task runtime, wait time, and completion time alarms at the key task level to implement layered alerting. For details, see Workflow and Task Alarms and the Failure and Timeout Policy section in General Task Configuration. 3.3 Triggering an Alarm for Verification
To verify that the alarm chain is working properly, you can manually trigger a failure:
1. In the workflow editor, temporarily switch the data source of sync_mysql2dlc to a non-existent MySQL instance.
2. Click Run at the top of the workflow.
3. You should receive alarm notifications in the email_data_ops mailbox and the WeCom group within 1–2 minutes.
4. After validation is complete, return to the task configuration, restore the original data source, and save.
Step 4: Tracking Execution Status During Workflow Runs
After a workflow goes live, daily Ops is mainly performed in the Workflow Run module.
4.1 Viewing the Run List to Quickly Identify Exceptions
1. In the left navigation pane, choose Workflow > Workflow Run.
2. In the filter bar at the top:
Workflow Name: Enter quickstart_orders_etl.
Start Time: The default is the last 3 days, and it can be adjusted to up to the last 60 days.
Run Account: You can filter by run account.
1. The chart area at the top of the list displays:
Run Quantity Trend: Displays the number of runs in each state grouped by time.
Error Code: Click an error code to filter the run instances with that error code.
4.2 Status Lifecycle (Key Comparisons)
Workflows and tasks have two independent state machines. We recommend that you familiarize yourself with the following high-frequency states:
Workflow Run Status:
Status | Description | Available Operations |
Pending | Workflow queuing | Terminate |
Running | At least one task is running. | Terminate |
Successful | All tasks have reached the final state without any failures. | Rerun, Delete |
Failed | All tasks have reached the final state and at least one has failed. | Rerun, Delete |
Terminating | Terminating | - |
Task Run Status (12 in total, 6 common):
Status | Description | Troubleshooting tips |
Waiting for Upstream | The upstream task has not reached the final state. | Check the upstream status first. |
Waiting for Resources | Waiting for Scheduling or Engine Resources | Check the computing resource load and CU quota. |
Running | Task running | - |
Retry upon Failure | Waiting for automatic retry after failure | Check the interval set in **step 3.1**. |
Failed | Final state, task failed. | Go to root cause analysis **(step 6)**. |
Upstream Failed | Upstream failure causes downstream tasks to be skipped. | Fix the upstream first, and then determine the rerun scope for the downstream. |
4.3 Switching Between Three Views to See Run Details
Click any Run Stream ID to go to the run details page. You can switch between tabs in the upper-left corner:
View | Scenario |
Graph mode (DAG) | View the status color of each task node at a glance: green/red/blue/gray/yellow. |
List mode | Display the run ID, type, duration, compute resources, and source file path of all tasks in a table, and support sorting by column. |
Gantt chart | Arrange the start and end time range of all tasks along a horizontal timeline, best suited for identifying long-running tasks and concurrency bottlenecks. |
Three typical scenarios for using Gantt charts to identify bottlenecks:
A single task bar is too wide: This indicates a long-tail task. Check whether it can be split or run with increased concurrency.
Multiple tasks are connected end-to-end but could run in parallel: The dependencies are redundant. Go back to the workflow editor to adjust them.
Large gaps appear between tasks: They are caused by waiting for resources or upstream tasks. Pay attention to the Wait Duration metric.
Step 5: Rerunning Failed Tasks and Backfilling Historical Data
The core action of troubleshooting is to repair failed instances and backfill their data.
5.1 Rerunning the Current Failed Workflow Run
1. On the workflow run details page or at the end of a run list row, click Rerun.
2. In the pop-up window:
By default, all failed nodes are selected in the DAG diagram. Hold down Shift and click nodes to adjust the selection range.
Parameter: The parameter values used for the current run are filled in by default and can be adjusted.
1. Click Run to trigger a rerun.
Scenario | Recommended Action |
Downstream rerun after upstream fix | Select "Failed Tasks + Their Downstream Tasks" in the DAG. |
Partial rerun after single-task code fix | On the task run details page, directly click Rerun and select only the current task. |
Rerun from scratch due to configuration error | In the workflow run list, click Rerun and select all tasks in the DAG. |
Note:
A rerun uses the latest task code and configuration. When dependencies change, the button is grayed out, and hovering over it displays a message stating that task dependencies in the workflow have changed and rerun is not supported. In this case, create a new workflow run instead of rerunning a historical instance. For more information, see Ops operations for workflows and tasks. 5.2 Backfilling Data for Historical Dates (Advanced Run)
When historical data is missing (for example, when scheduling is newly launched or the last 3 days of data need to be redone), use Advanced Run to trigger runs separately by date:
1. In the workflow details page action bar, click Advanced Run.
2. In the pop-up window:
In the DAG diagram on the left, all nodes are selected by default. If you only need to backfill specific tasks, click nodes to deselect them.
In the Parameter Configuration section on the right, change the business date in the workflow parameters (for example, dp_data_dt) to the target historical date 2025-01-13.
You can also click + Add Parameter to add temporary parameters, which take effect only for the current run.
1. Click Run to submit once.
2. Repeat the previous steps to submit the dates that need to be backfilled in sequence, such as 2025-01-14 and 2025-01-15.
Note:
Workflow scheduling in DataBuddy is triggered at the workflow run level. Backfilling is essentially manually triggering N runs with specified parameters. If the scheduling frequency is once per day, submit 3 runs for 3 days of backfilling. If the frequency is hourly, backfill at an hourly granularity. For more information, see Ops operations for workflows and tasks and system built-in parameters. 5.3 Terminating Abnormal Runs
Operation | Entries | Action |
Terminate a single run | End of the run list row Terminate | Terminate only this run (available for running / waiting / failed retry states) |
Terminate all | Operations bar on the workflow details page | Terminate all runs of the current workflow (including running and waiting), with a second confirmation |
Terminate all waiting runs | Operations bar on the workflow details page | Terminate only runs in waiting state and protect running instances. |
Step 6: Root Cause Analysis (Identifying Causes)
After receiving an alarm, follow the process below to locate the root cause. This section uses the failure of the pyspark_transform task as an example.
6.1 Going to the Task Run Details Page
1. In Workflow > Workflow Runs, find the failed instance and click its Run ID to go to the run details.
2. In Graph Mode, click the red node pyspark_transform to go to the task run details page.
The task run details page provides three tabs and one button:
Tab/Button | Scenario |
Tab-Execution Result | Task code and outputs |
Tab-Running Log | Error codes, error messages, and full logs (most important) |
Tab-Task Insights | Performance metrics such as cumulative CPU time, scanned data volume, Shuffle, and output row quantity |
Button-Spark UI | Access Spark native UI to inspect operator-level details. |
6.2 Reading Logs in Three Stages
Switch to the Run Logs Tab and check the following three types of information in order:
Phase | What to Look For |
Waiting for Upstream | Dependencies and status of upstream tasks |
Waiting to Run | Resource queuing status and scheduling logs |
Task Running | Error code (such as RESOURCE_* / PERMISSION_* / DATA_* / CONFIG_* / TIMEOUT_*) + full error stack + submitted task code |
Tips:
Use the Search box to locate keywords (Error / Exception / error codes).
The Auto Wrap toggle allows long stack traces to be viewed on a single screen.
Use Download Logs to take the complete logs off the platform for use with your internal knowledge base or ticket submission.
6.3 Quickly Identifying Fault Types with Intelligent Diagnosis
6.4 Emergency Handling Reference for Fault Classification
Error Code Prefix | Typical Scenario | Emergency handling |
RESOURCE_*
| Insufficient resources, CU queuing, Driver OOM | Temporarily scale out compute resources; optimize Spark executor.memory and shuffle.partitions |
PERMISSION_*
| Missing Permissions for Data Source/Catalog | Check the ACL in Data Source Management or Data Catalog, and apply for permissions if necessary. |
DATA_*
| Missing Upstream Data / Format Error | First check the upstream data source and confirm the data backfill link. |
CONFIG_*
| Parameter/SQL Syntax Error | View the submitted code and parameters in Run Results on the task details page; collaborate with developers to make modifications. |
TIMEOUT_*
| Task Timeout | Check whether the timeout threshold in Monitoring Metrics is reasonable, and optimize the code or increase the threshold if necessary. |
6.5 Using Data Lineage for Impact Analysis
If the failed task has downstream workflows, go to the details page of the table to which the failed task belongs (DataLakeCatalog.wedata_demo_db.quickstart_amazon_orders) and switch to the Lineage Tab:
List mode: Quickly lists the first-level upstream and downstream dependencies of the current table.
Graph mode: View from a topological perspective who else on the entire chain will be affected.
Verification Result
After completing all steps, you should be able to confirm:
1. Members and Roles: analyst_alice is a workspace member, and ops_bob is a workspace administrator. The user group data-ops-team can be managed in batches.
2. Notification Channels: Both email_data_ops and webhook_wechat_ops appear in the Notification Management list and have received at least one test alarm.
3. Alarms and Retries: Workflow-level failure alarms and monitoring metric alarms have been attached to channels. The failure retry policy for the sync_mysql2dlc task is 3 times × 5 minutes.
4. Rerun and Backfill: You can use Rerun to fix a single failure, or use Advanced Run to backfill data for 3 days by date.
5. Root Cause Analysis: You can locate error codes, read key stack traces, and identify fault categories within 5 minutes on the task run details page.
Summary
In this tutorial, you have completed:
In the workspace, onboard new members and assign permissions based on the workspace role + ACL two-layer model.
In Notification Management, integrate email and IM channels to achieve a unified outlet and cross-scenario reuse.
Configure alarms, automatic retries, and monitoring metric thresholds at both the workflow and task levels.
In Workflow Runs, become proficient in using the three views: DAG / List, and Gantt chart, and understand the 12 task states.
Use Rerun and Advanced Run to fix failures and backfill historical data.
Use the run logs, intelligent diagnostics, and data lineage toolkit to locate root causes and create a review document.
This end-to-end Ops pipeline covers more than 80% of the high-frequency daily operations for Ops/SRE roles. You can further explore advanced capabilities such as real-time data ingestion task Ops, data quality blocking pipelines, CI/CD pipelines, and metering and billing control.