Principles for Adding Preset Resources When a Scale-Out Rule Is Executed
Each cluster can have up to 10 scaling specifications. When a scale-out rule is triggered, scale-out is performed according to the priority of the specifications. When higher-priority specifications are insufficient, resources of lower-priority specifications are mixed with those of higher-priority specifications for scale-out to supplement computing resources (the execution order is the same for pay-as-you-go and spot instances).
When resources are sufficient: 1 > 2 > 3 > 4 > 5
Example
Suppose five specifications are preset and resources are sufficient. When a scale-out rule is triggered and 10 nodes need to be scaled out, 10 nodes of specification 1 are scaled out in order, and other preset specifications are not selected.
When resources are insufficient: 1 + 2 > 1 + 2 + 3 > 1 + 2 + 3 + 4 > 1 + 2 + 3 + 4 + 5
Example
Suppose specification 1 has eight nodes, specification 2 has four nodes, and specification 3 has three nodes. When a scale-out rule is triggered and 13 nodes need to be scaled out, eight nodes of specification 1, four nodes of specification 2, and one node of specification 3 are scaled out in order.
When a resource specification is out of stock (suppose specification 2 is out of stock): 1 + 3 > 1 + 3 + 4 > 1 + 3 + 4 + 5
Example
Suppose specification 1 has eight nodes, specification 2 is out of stock, and specification 3 has three nodes. When a scale-out rule is triggered and 10 nodes need to be scaled out, eight nodes of specification 1 and two nodes of specification 3 are scaled out in order, and specification 2 is not selected.
Suppose specification 1 has eight nodes and all other preset specifications are out of stock. When a scale-out rule is triggered and 10 nodes need to be scaled out, the rule will be triggered and eight nodes of specification 1 will be scaled out. The scale-out is partially successful.
Scale-out method: Three methods, namely node, memory, and number of cores, are supported. Only non-zero integers are accepted as input values for all three methods. When the method is number of cores or memory, the system will calculate the number of nodes to be scaled out based on maximum computing power.
Example
Scale out by number of cores: If 10 cores are set for scale-out, but the system selects the 8-core node specification according to the priority of available specifications, the rule will be triggered to provision two nodes of the 8-core specification.
Scale out by memory: If 20 GB is set for scale-out, but the system selects the 16-GB node specification according to the priority of available specifications, the rule will be triggered to provision two nodes of the 16-GB specification.
Principles for Scaling Out Elastic Nodes When Achieve Targets as Possible Is Enabled
During peak order placement, automatic scale-out may result in the actual number of scale-out machines failing to reach the elastic target quantity due to resource contention. When the Achieve Targets as Possible policy is enabled and the configured scaling specification resources are sufficient, the system will automatically retry resource requests until the target quantity is achieved or approached.
Attention:
The maximum number of retries is 5. After this limit is reached, no more retries will be made regardless of whether the scale-out quantity has achieved the target quantity.
The priority of preset resource specifications is 1 > 2 > 3 > 4 > 5. Suppose specification 1 has eight nodes, specification 2 has two nodes, specification 3 has six nodes, and specification 4 has six nodes.
Example:
Achieve Targets as Possible disabled: When a scale-out rule is triggered and 15 nodes need to be scaled out, eight nodes of specification 1, two nodes of specification 2, and five nodes of specification 3 are scaled out in order. However, due to resource contention during order placement, the actual delivery is: eight nodes of specification 1, zero nodes of specification 2, and five nodes of specification 3, totaling 13 nodes. The scale-out is partially successful.
Achieve Targets as Possible enabled: When a scale-out rule is triggered and 15 nodes need to be scaled out, eight nodes of specification 1, two nodes of specification 2, and five nodes of specification 3 are scaled out in order. However, due to resource contention during order placement, the actual delivery for the initial scale-out is: eight nodes of specification 1, zero nodes of specification 2, and five nodes of specification 3, totaling 13 nodes. A replenishment retry is then triggered, polling specifications 1, 2, 3, 4, and 5, and the final supplementary delivery is: one node of specification 2 and one node of specification 3. The total for both deliveries is: eight nodes of specification 1, one node of specification 2, and six nodes of specification 3, totaling 15 nodes. The scale-out is successful.
Principles for Scaling In Elastic Nodes When a Scale-In Rule Is Triggered
When a scale-in rule is triggered for elastic nodes scaled out by the auto scaling feature: For time-based scale-in, idle nodes are scaled in first following the principle of "nodes scaled out earlier are scaled in later, while nodes scaled out later are scaled in first". When the quantity is insufficient, nodes running containers are selected for scale-in. For load-based scale-in, nodes where the service owning the load metric is deployed are scaled in first, and idle nodes are scaled in first, following the principle of "nodes scaled out earlier are scaled in later, while nodes scaled out later are scaled in first". When the quantity is insufficient, nodes running containers are selected for scale-in. Non-elastic nodes are not affected by scale-in rules and only support manual scale-in.
Attention
Nodes with scheduled termination are not subject to the "nodes scaled out earlier are scaled in later, while nodes scaled out later are scaled in first" principle or the cluster "minimum number of nodes" constraint. Scale-in is executed as soon as the scheduled time arrives, with a default graceful scale-in range of 30 minutes.
A node is identified as idle if no container is running on it for 5 minutes.
For load-based scale-in, assume the node creation time from earliest to latest is A > B > C > D > E.
Example:
A YARN load metric is set for scale-in of five nodes. C, D, and E have the YARN component deployed, and D and E are running containers. When the scale-in rule is triggered, the scale-in order is C > E > D > B > A.
A Trino load metric is set for scale-in of five nodes. C, D, and E have the Trino component deployed, and D and B are running containers. When the scale-in rule is triggered, the scale-in order is E > C > D > A > B.
For time-based scale-in, assume the node creation time from earliest to latest is A > B > C > D > E.
Example:
Scale in by node: Five nodes are set for scale-in, and D and E are running containers. When the scale-in rule is triggered, the scale-in order is C > B > A > E > D.
Scale-in method: Three methods, namely node, memory, and number of cores, are supported. Only non-zero integers are accepted as input values for all three methods. When the method is number of cores or memory, the system will calculate the minimum number of nodes to be removed while ensuring business continuity. Nodes with no running tasks will be scaled in in reverse chronological order.
Example
Scale in by number of cores: Twenty cores are set for scale-in. When the scale-in rule is triggered, the elastic nodes in the cluster are three 8-core 16 GB nodes and two 4-core 8 GB nodes. Nodes are scaled in in reverse chronological order, and two 8-core 16 GB nodes will be successfully scaled in.
Scale in by memory: 30 GB is set for scale-in. When the scale-in rule is triggered, the elastic nodes in the cluster are three 8-core 16 GB nodes and two 4-core 8 GB nodes. Nodes are scaled in in reverse chronological order, and one 8-core 16 GB node will be successfully scaled in.
Scale in by number of cores: Twenty cores are set for scale-in. When the scale-in rule is triggered, the elastic nodes in the cluster are three 32-core 64 GB nodes and two 64-core 128 GB nodes. As no eligible elastic node is available, the scale-in rule will fail to be triggered.
Principles for Executing Scaling Rules in Sequence
Mixed elastic rule settings of time-based scaling and load-based scaling are supported. Rules are triggered in the principle of "first triggered, first executed; simultaneous triggers are executed by rule priority". The rule status marks whether a rule is enabled and defaults to Enabled. If you do not need a rule to run but want to keep its configuration, set the rule status to Disabled.
Load-based scaling only
1.1 The principle of "first triggered, first executed; simultaneous triggers are executed by rule priority" is followed, for example: 1 > 2 > 3 > 4 > 5.
1.2 A single load-based scaling rule supports multiple metrics. You can choose to trigger the rule when all metrics meet the conditions or when any metric meets the conditions.
1.3 You can specify a time period to monitor cluster load changes and make load-based scaling take effect.
Time-based scaling only
1.1 The principle of "first triggered, first executed; simultaneous triggers are executed by rule priority" is followed, for example: 1 > 2 > 3 > 4 > 5.
1.2 For repeated execution rules, if a rule expires, its status becomes invalid and is set to Disabled. An alarm notification will be sent before expiration. For details, see Alarm Configuration. Mixed load-and-time scaling
The principle of "first triggered, first executed; simultaneous triggers are executed by rule priority" is followed, for example: 1 > 2 > 3 > 4 > 5.
Queue Load Metric Mapping
The cluster types, components, and versions supported by load-based scaling are as follows:
Hadoop clusters:
YARN component: Node Label is supported only by EMR-V3.x or later.
Trino component: Supported only by EMR-V2.7.0 and EMR-V3.40 or later.
Ray clusters: Uniffle component.
StarRocks clusters: StarRocks component.
|
YARN | AvailableVCores | root | AvailableVCores#root | Number of available virtual cores in the root queue |
|
| root.default | AvailableVCores#root.default | Number of available virtual cores in the root.default queue |
|
| Custom subqueue | For example: AvailableVCores#root.test | Number of available virtual cores in the root.test queue |
|
| Node Label | For example: AvailableVCores#Node Label.test | Number of available virtual cores under the test label |
| PendingVCores | root | PendingVCores#root | Number of virtual cores pending availability in the root queue |
|
| root.default | PendingVCores#root.default | Number of virtual cores pending availability in the root.default queue |
|
| Custom subqueue | For example: PendingVCores#root.test | Number of virtual cores pending availability in the root.test queue |
|
| Node Label | For example: PendingVCores#Node Label.test | Number of virtual cores pending availability under the test label |
| AvailableMB | root | AvailableMB#root | Available memory in the root queue (MB) |
|
| root.default | AvailableMB#root.default | Available memory in the root.default queue (MB) |
|
| Custom subqueue | For example: AvailableMB#root.test | Available memory in the root.test queue (MB) |
|
| Node Label | For example: AvailableMB#Node Label.test | Available memory under the test label (MB) |
| PendingMB | root | PendingMB#root | Memory pending availability in the root queue (MB) |
|
| root.default | PendingMB#root.default | Memory pending availability in the root.default queue (MB) |
|
| Custom subqueue | For example: PendingMB#root.test | Memory pending availability in the root.test queue (MB) |
|
| Node Label | For example: PendingMB#Node Label.test | Memory pending availability under the test label (MB) |
| AvailableMemPercentage | Cluster | AvailableMemPercentage | Percentage of remaining memory |
| ContainerPendingRatio | Cluster | ContainerPendingRatio | Ratio of the number of containers to be allocated to the number of allocated containers |
| AppsRunning | root | AppsRunning#root | Number of running tasks in the root queue |
|
| root.default | AppsRunning#root.default | Number of running tasks in the root.default queue |
|
| Custom subqueue | For example: AppsRunning#root.test | Number of running tasks in the root.test queue |
| AppsPending | root | AppsPending#root | Number of pending tasks in the root queue |
|
| root.default | AppsPending#root.default | Number of pending tasks in the root.default queue |
|
| Custom subqueue | For example: AppsPending#root.test | Number of pending tasks in the root.test queue |
| PendingContainers | root | PendingContainers#root | Number of containers to be allocated in the root queue |
|
| root.default | PendingContainers#root.default | Number of containers to be allocated in the root.default queue |
|
| Custom subqueue | For example: PendingContainers#root.test | Number of containers to be allocated in the root.test queue |
| AllocatedMB | root | AllocatedMB#root | Amount of allocated memory in the root queue |
|
| root.default | AllocatedMB#root.default | Amount of allocated memory in the root.default queue |
|
| Custom subqueue | For example: AllocatedMB#root.test | Amount of allocated memory in the root.test queue |
|
| Node Label | For example: AllocatedMB#Node Label.test | Amount of allocated memory under the test label |
| AllocatedVCores | root | AllocatedVCores#root | Number of allocated virtual cores in the root queue |
|
| root.default | AllocatedVCores#root.default | Number of allocated virtual cores in the root.default queue |
|
| Custom subqueue | For example: AllocatedVCores#root.test | Number of allocated virtual cores in the root.test queue |
|
| Node Label | For example: AllocatedVCores#Node Label.test | Number of allocated virtual cores under the test label |
| ReservedVCores | root | ReservedVCores#root | Number of reserved virtual cores in the root queue |
|
| root.default | ReservedVCores#root.default | Number of reserved virtual cores in the root.default queue |
|
| Custom subqueue | For example: ReservedVCores#root.test | Number of reserved virtual cores in the root.test queue |
|
| Node Label | For example: ReservedVCores#Node Label.test | Number of reserved virtual cores under the test label |
| AllocatedContainers | root | AllocatedContainers#root | Number of allocated containers in the root queue |
|
| root.default | AllocatedContainers#root.default | Number of allocated containers in the root.default queue |
|
| Custom subqueue | For example: AllocatedContainers#root.test | Number of allocated containers in the root.test queue |
| ReservedMB | root | ReservedMB#root | Amount of reserved memory in the root queue |
|
| root.default | ReservedMB#root.default | Amount of reserved memory in the root.default queue |
|
| Custom subqueue | For example: ReservedMB#root.test | Amount of reserved memory in the root.test queue |
|
| Node Label | For example: ReservedMB#Node Label.test | Amount of reserved memory under the test label |
| AppsKilled | root | AppsKilled#root | Number of killed tasks in the root queue |
|
| root.default | AppsKilled#root.default | Number of killed tasks in the root.default queue |
|
| Custom subqueue | For example: AppsKilled#root.test | Number of killed tasks in the root.test queue |
| AppsFailed | root | AppsFailed#root | Number of failed tasks in the root queue |
|
| root.default | AppsFailed#root.default | Number of failed tasks in the root.default queue |
|
| Custom subqueue | For example: AppsFailed#root.test | Number of failed tasks in the root.test queue |
| AppsCompleted | root | AppsCompleted#root | Number of completed tasks in the root queue |
|
| root.default | AppsCompleted#root.default | Number of completed tasks in the root.default queue |
|
| Custom subqueue | For example: AppsCompleted#root.test | Number of completed tasks in the root.test queue |
| AppsSubmitted | root | AppsSubmitted#root | Number of submitted tasks in the root queue |
|
| root.default | AppsSubmitted#root.default | Number of submitted tasks in the root.default queue |
|
| Custom subqueue | For example: AppsSubmitted#root.test | Number of submitted tasks in the root.test queue |
| AvailableVCoresPercentage | Cluster | AvailableVCoresPercentage | Percentage of available virtual cores in the cluster |
| MemPendingRatio | root | MemPendingRatio#root | Percentage of memory pending availability in the root queue |
|
| root.default | MemPendingRatio#root.default | Percentage of memory pending availability in the root.default queue |
|
| Custom subqueue | For example: MemPendingRatio#root.test | Percentage of memory pending availability in the root.test queue |
| usageRatioVCores | root | usageRatioVCores#root | CPU utilization of the root queue |
|
| root.default | usageRatioVCores#root.default | CPU utilization of the root.default queue |
|
| Custom subqueue | For example: usageRatioVCores#root.test | CPU utilization of the root.test queue |
| usageRatioMem | root | usageRatioMem#root | Memory utilization of the root queue |
|
| root.default | usageRatioMem#root.default | Memory utilization of the root.default queue |
|
| Custom subqueue | For example: usageRatioMem#root.test | Memory utilization of the root.test queue |
Trino | FreeDistributed | Cluster | FreeDistributed | Available distributed memory in the cluster |
| QueuedQueries | Cluster | QueuedQueries | Total number of queries waiting to be executed in the queue |
StarRocks | CNUsedVCores | Cluster | CNUsedVCores | Number of virtual cores used by all CN nodes in the cluster |
|
| Compute group |
| Number of virtual cores used by all CN nodes in the specified compute group |
| CNUsedMB | Cluster | CNUsedMB | Amount of memory used by all CN nodes in the cluster (MB) |
|
| Compute group |
| Amount of memory used by all CN nodes in the specified compute group (MB) |
| CNUsedVCoresPercentage | Cluster | CNUsedVCoresPercentage | CPU utilization of all CN nodes in the cluster |
|
| Compute group |
| CPU utilization of all CN nodes in the specified compute group |
| CNUsedMBPercentage | Cluster | CNUsedMBPercentage | Memory utilization of all CN nodes in the cluster |
|
| Compute group |
| Memory utilization of all CN nodes in the specified compute group |
Uniffle | DataDiskUsedPercentage | Cluster | DataDiskUsedPercentage | Average utilization of the data disk space of shuffle servers |
| DataDiskIOPercentage | Cluster | DataDiskIOPercentage | Average utilization of the data disk I/O devices of shuffle servers |