Ray tasks are used to schedule and submit Ray jobs to Ray clusters within Workflows, and are suitable for scenarios such as distributed machine learning training, reinforcement learning, and large-scale data preprocessing.
Overview
Ray is a high-performance distributed computing framework that covers:
Distributed machine learning training: Large-scale model training and hyperparameter tuning.
Reinforcement learning: RL training based on RLlib.
Large-scale data preprocessing: Feature engineering and data cleaning.
Model serving: Online inference based on Ray Serve.
For general configurations (task name, upstream dependencies, run conditions, parameters, monitoring metrics, alarms, and failure and timeout policies), see Task General Configuration. Prerequisites
The workspace is bound to job cluster - Ray cluster compute resources. For more information, see Create Compute Resources. A zip package for the Ray job has been prepared, which contains the entry script and dependencies.
Operation Steps
Step 1: Creating a Job
1. On the workflow canvas, click + Add Task.
2. Select Ray Job as the task type.
Step 2: Configuring Basic Information
Parameter | Description | Required |
Task Name | Unique within the same workflow. | Yes |
Task type | Ray job (not modifiable). | Yes |
Task Sources | Upload a zip file. | Yes |
Step 3: Uploading a zip Package
Parameter | Description | Required |
zip file | A compressed package that contains the Ray job entry script and dependencies. | Yes |
After upload, the zip package is stored in the workspace for scheduling by this task.
Step 4: Configuring Compute Resources
Parameter | Description | Required |
Computing Resource | Select the compute resource of the job cluster - Ray type. | Yes |
Entry command | Command to start a Ray job, for example, python -c 'import ray; ray.init(); ...'. | Yes |
Storage configuration | DLC storage configuration determines the storage location for Ray job reads and writes. | Yes |
Compute environment | DLC compute environment, corresponding to the Ray kernel version. | Yes |
Note:
The specific options for storage configuration and compute environment are consistent with the DLC console, and the detailed fields are subject to the console.
Step 5: Upstream Dependencies and Runtime Conditions
Step 6: Monitoring Metrics, Alarms, and Failure and Timeout Policies
Step 7: Saving
Click Create Task or Save Task.
Parameter Description
Ray Job Parameters
Ray tasks do not support parameter passing in the current phase, which is different from other task types. To pass dynamic values in a Ray job, manage them through built-in configurations in the zip package or environment variables.
Job Run Details
Tabs on the run details page of a Ray task:
Tab | Description |
Execution result | Not available in this release (DLC limitation). |
Running logs | Not available in this release. |
Task Insights | Not available in this release. |
Event Logs | Event logs provided by DLC, displayed in this task. |
Pod Details | DLC Pod list and status. |
Spark UI | Corresponds to Ray Dashboard. Click the button to go to Ray UI. |
Use Limits
The compute resource type must be job cluster - Ray cluster. Other types are not supported.
Ray tasks do not support parameter passing in the current phase.
For task source, only local zip package upload is supported in the current phase. Referencing files or zip packages from Studio workspace, Volume, or remote Git repositories is not supported yet.
Due to current DLC API limitations, the Run Result, Run Log, and Task Insight tabs are not displayed.
FAQs
Q1: How do I prepare a zip package for a Ray job?
Recommended structure:
your_ray_job.zip
├── main.py # Entry script
├── requirements.txt # Python dependencies
└── modules/ # Custom modules
└── trainer.py
The entry command is triggered by python main.py, which initializes a Ray cluster and submits the job.
Q2: Can Ray tasks be used for real-time inference?
This task is used for offline Ray job scheduling. For real-time inference, Ray Serve deployment is recommended.
Q3: Can task execution details be viewed in the Ray UI?
Yes. On the task run details page, click the Ray UI button to go to the Ray Dashboard, where you can view the job topology, task queue, and resource usage.
References