gcloud_beta_dataproc_jobs_submit (1)

NAME

gcloud beta dataproc jobs submit - submit Google Cloud Dataproc jobs to execute on a cluster

SYNOPSIS

gcloud beta dataproc jobs submit COMMAND [--async] [--bucket=BUCKET] [--region=REGION] [GCLOUD_WIDE_FLAG ...]

DESCRIPTION

(BETA) Submit Google Cloud Dataproc jobs to execute on a cluster.

FLAGS

--async

Does not wait for the job to run.

--bucket=BUCKET

The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

--region=REGION

Cloud Dataproc region to use. Each Cloud Dataproc region constitutes an independent resource namespace constrained to deploying instances into Compute Engine zones inside the region. The default value of global is a special multi-region namespace which is capable of deploying instances into all Compute Engine zones globally, and is disjoint from other Cloud Dataproc regions. Overrides the default dataproc/region property value for this command invocation.

GCLOUD WIDE FLAGS

These flags are available to all commands: --account, --configuration, --flags-file, --flatten, --format, --help, --log-http, --project, --quiet, --trace-token, --user-output-enabled, --verbosity. Run $ gcloud help for details.

COMMANDS

COMMAND is one of the following:

hadoop

(BETA) Submit a Hadoop job to a cluster.

hive

(BETA) Submit a Hive job to a cluster.

pig

(BETA) Submit a Pig job to a cluster.

pyspark

(BETA) Submit a PySpark job to a cluster.

spark

(BETA) Submit a Spark job to a cluster.

spark-r

(BETA) Submit a SparkR job to a cluster.

spark-sql

(BETA) Submit a Spark SQL job to a cluster.

EXAMPLES

To submit a Hadoop MapReduce job, run:

$ gcloud beta dataproc jobs submit hadoop --cluster my_cluster \
    --jar my_jar.jar -- arg1 arg2

To submit a Spark Scala or Java job, run:

$ gcloud beta dataproc jobs submit spark --cluster my_cluster \
    --jar my_jar.jar -- arg1 arg2

To submit a PySpark job, run:

$ gcloud beta dataproc jobs submit pyspark --cluster my_cluster \
    my_script.py -- arg1 arg2

To submit a Spark SQL job, run:

$ gcloud beta dataproc jobs submit spark-sql --cluster my_cluster \
    --file my_queries.q

To submit a Pig job, run:

$ gcloud beta dataproc jobs submit pig --cluster my_cluster \
    --file my_script.pig

To submit a Hive job, run:

$ gcloud beta dataproc jobs submit hive --cluster my_cluster \
    --file my_queries.q

NOTES

This command is currently in BETA and may change without notice. These variants are also available:

$ gcloud dataproc jobs submit $ gcloud alpha dataproc jobs submit