The DCU rate shown below is an hourly / monthly rate. It is prorated and billed per second, with a 1-minute minimum charge.

The shuffle storage rate shown below is an hourly / monthly rate. It is prorated and billed per second, with a 1-minute minimum charge for standard shuffle storage and a 5-minute minimum charge for Premium shuffle storage. Premium shuffle storage can only be used with Premium Compute Unit.

The accelerator rate shown below is an hourly / monthly rate. It is prorated and billed per second, with a 5-minute minimum charge.

If the Serverless for Apache Spark batch workload runs with 12 DCUs (spark.driver.cores=4,spark.executor.cores=4,spark.executor.instances=2) for 24 hours in the us-central1 region and consumes 25GB of shuffle storage, the price calculation is as follows.

Notes:

1. The example assumes a 30-day month. Since the batch workload duration is one day, the monthly shuffle storage rate is divided by 30.

If the Serverless for Apache Spark batch workload runs with 12 DCUs and 2 L4 GPUs (spark.driver.cores=4,spark.executor.cores=4, spark.executor.instances=2,spark.dataproc.driver.compute.tier=premium,  spark.dataproc.executor.compute.tier=premium, spark.dataproc.executor.disk.tier=premium,  spark.dataproc.executor.resource.accelerator.type=l4) for 24 hours in the us-central1 region and consumes 25GB of shuffle storage, the price calculation is as follows.

Notes:

1. The example assumes a 30-day month. Since the batch workload duration is one day, the monthly shuffle storage rate is divided by 30.

If the Serverless for Apache Spark interactive workload runs with 12 DCUs (spark.driver.cores=4,spark.executor.cores=4,spark.executor.instances=2) for 24 hours in the us-central1 region and consumes 25GB of shuffle storage, the price calculation is as follows:

Notes:

1. The example assumes a 30-day month. Since the batch workload duration is one day, the monthly shuffle storage rate is divided by 30.

When a batch workload completes, Serverless for Apache Spark calculates UsageMetrics, which contain an approximation of the total DCU, accelerator, and shuffle storage resources consumed by the completed workload. After running a workload, you can run the gcloud dataproc batches describe BATCH_ID command to view workload usage metrics to help you estimate the cost of running the workload.

Example:

Serverless for Apache Spark runs a workload on an ephemeral cluster with one master and two workers. Each node consumes 4 DCUs (default is 4 DCUs per core—see spark.dataproc.driver.disk.size) and 400 GB shuffle storage (default is 100GB per core—see spark.driver.cores). Workload run time is 60 seconds. Also, each worker has 1 GPU for a total of 2 across the cluster.

The user runs gcloud dataproc batches describe BATCH_ID --region REGION to obtain usage metrics. The command output includes the following snippet (milliDcuSeconds: 4 DCUs x 3 VMs x 60 seconds x 1000 = 720000, milliAcceleratorSeconds: 1 GPU x 2 VMs x 60 seconds x 1000 = 120000, and shuffleStorageGbSeconds: 400GB x 3 VMs x 60 seconds = 72000):

Your Serverless for Apache Spark workload can optionally utilize the following resources, each billed at its own pricing, including but not limited to:

• Cloud Storage
• Global Networking
• Cloud Monitoring
• BigQuery
• Bigtable