emr

class pulumi_aws.emr.Cluster(resource_name, opts=None, additional_info=None, applications=None, autoscaling_role=None, bootstrap_actions=None, configurations=None, configurations_json=None, core_instance_count=None, core_instance_type=None, custom_ami_id=None, ebs_root_volume_size=None, ec2_attributes=None, instance_groups=None, keep_job_flow_alive_when_no_steps=None, kerberos_attributes=None, log_uri=None, master_instance_type=None, name=None, release_label=None, scale_down_behavior=None, security_configuration=None, service_role=None, steps=None, tags=None, termination_protection=None, visible_to_all_users=None, __name__=None, __opts__=None)

Provides an Elastic MapReduce Cluster, a web service that makes it easy to process large amounts of data efficiently. See Amazon Elastic MapReduce Documentation for more information.

Attributes for the Amazon EC2 instances running the job flow

  • key_name - (Optional) Amazon EC2 key pair that can be used to ssh to the master node as the user called hadoop
  • subnet_id - (Optional) VPC subnet id where you want the job flow to launch. Cannot specify the cc1.4xlarge instance type for nodes of a job flow launched in a Amazon VPC
  • additional_master_security_groups - (Optional) String containing a comma separated list of additional Amazon EC2 security group IDs for the master node
  • additional_slave_security_groups - (Optional) String containing a comma separated list of additional Amazon EC2 security group IDs for the slave nodes as a comma separated string
  • emr_managed_master_security_group - (Optional) Identifier of the Amazon EC2 EMR-Managed security group for the master node
  • emr_managed_slave_security_group - (Optional) Identifier of the Amazon EC2 EMR-Managed security group for the slave nodes
  • service_access_security_group - (Optional) Identifier of the Amazon EC2 service-access security group - required when the cluster runs on a private subnet
  • instance_profile - (Required) Instance Profile for EC2 instances of the cluster assume this role
NOTE on EMR-Managed security groups: These security groups will have any missing inbound or outbound access rules added and maintained by AWS, to ensure proper communication between instances in a cluster. The EMR service will maintain these rules for groups provided in emr_managed_master_security_group and emr_managed_slave_security_group; attempts to remove the required rules may succeed, only for the EMR service to re-add them in a matter of minutes. This may cause Terraform to fail to destroy an environment that contains an EMR cluster, because the EMR service does not revoke rules added on deletion, leaving a cyclic dependency between the security groups that prevents their deletion. To avoid this, use the revoke_rules_on_delete optional attribute for any Security Group used in emr_managed_master_security_group and emr_managed_slave_security_group. See Amazon EMR-Managed Security Groups for more information about the EMR-managed security group rules.

Attributes for Kerberos configuration

  • ad_domain_join_password - (Optional) The Active Directory password for ad_domain_join_user
  • ad_domain_join_user - (Optional) Required only when establishing a cross-realm trust with an Active Directory domain. A user with sufficient privileges to join resources to the domain.
  • cross_realm_trust_principal_password - (Optional) Required only when establishing a cross-realm trust with a KDC in a different realm. The cross-realm principal password, which must be identical across realms.
  • kdc_admin_password - (Required) The password used within the cluster for the kadmin service on the cluster-dedicated KDC, which maintains Kerberos principals, password policies, and keytabs for the cluster.
  • realm - (Required) The name of the Kerberos realm to which all nodes in a cluster belong. For example, EC2.INTERNAL

Attributes for each task instance group in the cluster

  • instance_role - (Required) The role of the instance group in the cluster. Valid values are: MASTER, CORE, and TASK.
  • instance_type - (Required) The EC2 instance type for all instances in the instance group
  • instance_count - (Optional) Target number of instances for the instance group
  • name - (Optional) Friendly name given to the instance group
  • bid_price - (Optional) If set, the bid price for each EC2 instance in the instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances. bid_price can not be set for the MASTER instance group, since that group must always be On-Demand
  • ebs_config - (Optional) A list of attributes for the EBS volumes attached to each instance in the instance group. Each ebs_config defined will result in additional EBS volumes being attached to each instance in the instance group. Defined below
  • autoscaling_policy - (Optional) The autoscaling policy document. This is a JSON formatted string. See EMR Auto Scaling

Attributes for the EBS volumes attached to each EC2 instance in the instance_group

  • size - (Required) The volume size, in gibibytes (GiB).
  • type - (Required) The volume type. Valid options are gp2, io1, standard and st1. See EBS Volume Types.
  • iops - (Optional) The number of I/O operations per second (IOPS) that the volume supports
  • volumes_per_instance - (Optional) The number of EBS volumes with this configuration to attach to each EC2 instance in the instance group (default is 1)
  • name - (Required) Name of the bootstrap action
  • path - (Required) Location of the script to run during a bootstrap action. Can be either a location in Amazon S3 or on a local file system
  • args - (Optional) List of command line arguments to pass to the bootstrap action script

Attributes for step configuration

  • action_on_failure - (Required) The action to take if the step fails. Valid values: TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, and CONTINUE
  • hadoop_jar_step - (Required) The JAR file used for the step. Defined below.
  • name - (Required) The name of the step.

Attributes for Hadoop job step configuration

  • args - (Optional) List of command line arguments passed to the JAR file’s main function when executed.
  • jar - (Required) Path to a JAR file run during the step.
  • main_class - (Optional) Name of the main class in the specified Java file. If not specified, the JAR file should specify a Main-Class in its manifest file.
  • properties - (Optional) Key-Value map of Java properties that are set when the step runs. You can use these properties to pass key value pairs to your main function.
Parameters:
  • resource_name (str) – The name of the resource.
  • opts (pulumi.ResourceOptions) – Options for the resource.
  • additional_info (pulumi.Input[str]) – A JSON string for selecting additional features such as adding proxy information. Note: Currently there is no API to retrieve the value of this argument after EMR cluster creation from provider, therefore Terraform cannot detect drift from the actual EMR cluster if its value is changed outside Terraform.
  • applications (pulumi.Input[list]) – A list of applications for the cluster. Valid values are: Flink, Hadoop, Hive, Mahout, Pig, Spark, and JupyterHub (as of EMR 5.14.0). Case insensitive
  • autoscaling_role (pulumi.Input[str]) – An IAM role for automatic scaling policies. The IAM role provides permissions that the automatic scaling feature requires to launch and terminate EC2 instances in an instance group.
  • bootstrap_actions (pulumi.Input[list]) – List of bootstrap actions that will be run before Hadoop is started on the cluster nodes. Defined below
  • configurations (pulumi.Input[str]) – List of configurations supplied for the EMR cluster you are creating
  • configurations_json (pulumi.Input[str]) – A JSON string for supplying list of configurations for the EMR cluster.
  • core_instance_count (pulumi.Input[float]) – Number of Amazon EC2 instances used to execute the job flow. EMR will use one node as the cluster’s master node and use the remainder of the nodes (core_instance_count-1) as core nodes. Cannot be specified if instance_groups is set. Default 1
  • core_instance_type (pulumi.Input[str]) – The EC2 instance type of the slave nodes. Cannot be specified if instance_groups is set
  • custom_ami_id (pulumi.Input[str]) – A custom Amazon Linux AMI for the cluster (instead of an EMR-owned AMI). Available in Amazon EMR version 5.7.0 and later.
  • ebs_root_volume_size (pulumi.Input[float]) – Size in GiB of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Available in Amazon EMR version 4.x and later.
  • ec2_attributes (pulumi.Input[dict]) – Attributes for the EC2 instances running the job flow. Defined below
  • instance_groups (pulumi.Input[list]) – A list of instance_group objects for each instance group in the cluster. Exactly one of master_instance_type and instance_group must be specified. If instance_group is set, then it must contain a configuration block for at least the MASTER instance group type (as well as any additional instance groups). Defined below
  • keep_job_flow_alive_when_no_steps (pulumi.Input[bool]) – Switch on/off run cluster with no steps or when all steps are complete (default is on)
  • kerberos_attributes (pulumi.Input[dict]) – Kerberos configuration for the cluster. Defined below
  • log_uri (pulumi.Input[str]) – S3 bucket to write the log files of the job flow. If a value is not provided, logs are not created
  • master_instance_type (pulumi.Input[str]) – The EC2 instance type of the master node. Exactly one of master_instance_type and instance_group must be specified.
  • name (pulumi.Input[str]) – The name of the job flow
  • release_label (pulumi.Input[str]) – The release label for the Amazon EMR release
  • scale_down_behavior (pulumi.Input[str]) – The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized.
  • security_configuration (pulumi.Input[str]) – The security configuration name to attach to the EMR cluster. Only valid for EMR clusters with release_label 4.8.0 or greater
  • service_role (pulumi.Input[str]) – IAM role that will be assumed by the Amazon EMR service to access AWS resources
  • steps (pulumi.Input[list]) – List of steps to run when creating the cluster. Defined below. It is highly recommended to utilize the lifecycle configuration block with ignore_changes if other steps are being managed outside of Terraform.
  • tags (pulumi.Input[dict]) – list of tags to apply to the EMR Cluster
  • termination_protection (pulumi.Input[bool]) – Switch on/off termination protection (default is off)
  • visible_to_all_users (pulumi.Input[bool]) – Whether the job flow is visible to all IAM users of the AWS account associated with the job flow. Default true
additional_info = None

A JSON string for selecting additional features such as adding proxy information. Note: Currently there is no API to retrieve the value of this argument after EMR cluster creation from provider, therefore Terraform cannot detect drift from the actual EMR cluster if its value is changed outside Terraform.

applications = None

A list of applications for the cluster. Valid values are: Flink, Hadoop, Hive, Mahout, Pig, Spark, and JupyterHub (as of EMR 5.14.0). Case insensitive

autoscaling_role = None

An IAM role for automatic scaling policies. The IAM role provides permissions that the automatic scaling feature requires to launch and terminate EC2 instances in an instance group.

bootstrap_actions = None

List of bootstrap actions that will be run before Hadoop is started on the cluster nodes. Defined below

configurations = None

List of configurations supplied for the EMR cluster you are creating

configurations_json = None

A JSON string for supplying list of configurations for the EMR cluster.

core_instance_count = None

Number of Amazon EC2 instances used to execute the job flow. EMR will use one node as the cluster’s master node and use the remainder of the nodes (core_instance_count-1) as core nodes. Cannot be specified if instance_groups is set. Default 1

core_instance_type = None

The EC2 instance type of the slave nodes. Cannot be specified if instance_groups is set

custom_ami_id = None

A custom Amazon Linux AMI for the cluster (instead of an EMR-owned AMI). Available in Amazon EMR version 5.7.0 and later.

ebs_root_volume_size = None

Size in GiB of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Available in Amazon EMR version 4.x and later.

ec2_attributes = None

Attributes for the EC2 instances running the job flow. Defined below

instance_groups = None

A list of instance_group objects for each instance group in the cluster. Exactly one of master_instance_type and instance_group must be specified. If instance_group is set, then it must contain a configuration block for at least the MASTER instance group type (as well as any additional instance groups). Defined below

keep_job_flow_alive_when_no_steps = None

Switch on/off run cluster with no steps or when all steps are complete (default is on)

kerberos_attributes = None

Kerberos configuration for the cluster. Defined below

log_uri = None

S3 bucket to write the log files of the job flow. If a value is not provided, logs are not created

master_instance_type = None

The EC2 instance type of the master node. Exactly one of master_instance_type and instance_group must be specified.

master_public_dns = None

The public DNS name of the master EC2 instance.

name = None

The name of the job flow

release_label = None

The release label for the Amazon EMR release

scale_down_behavior = None

The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized.

security_configuration = None

The security configuration name to attach to the EMR cluster. Only valid for EMR clusters with release_label 4.8.0 or greater

service_role = None

IAM role that will be assumed by the Amazon EMR service to access AWS resources

steps = None

List of steps to run when creating the cluster. Defined below. It is highly recommended to utilize the lifecycle configuration block with ignore_changes if other steps are being managed outside of Terraform.

tags = None

list of tags to apply to the EMR Cluster

termination_protection = None

Switch on/off termination protection (default is off)

visible_to_all_users = None

Whether the job flow is visible to all IAM users of the AWS account associated with the job flow. Default true

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters:prop (str) – A property name.
Returns:A potentially transformed property name.
Return type:str
translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters:prop (str) – A property name.
Returns:A potentially transformed property name.
Return type:str
class pulumi_aws.emr.InstanceGroup(resource_name, opts=None, cluster_id=None, ebs_configs=None, ebs_optimized=None, instance_count=None, instance_type=None, name=None, __name__=None, __opts__=None)

Provides an Elastic MapReduce Cluster Instance Group configuration. See Amazon Elastic MapReduce Documentation for more information.

NOTE: At this time, Instance Groups cannot be destroyed through the API nor web interface. Instance Groups are destroyed when the EMR Cluster is destroyed. Terraform will resize any Instance Group to zero when destroying the resource.
Parameters:
  • resource_name (str) – The name of the resource.
  • opts (pulumi.ResourceOptions) – Options for the resource.
  • cluster_id (pulumi.Input[str]) – ID of the EMR Cluster to attach to. Changing this forces a new resource to be created.
  • ebs_configs (pulumi.Input[list]) – One or more ebs_config blocks as defined below. Changing this forces a new resource to be created.
  • ebs_optimized (pulumi.Input[bool]) – Indicates whether an Amazon EBS volume is EBS-optimized. Changing this forces a new resource to be created.
  • instance_count (pulumi.Input[float]) – Target number of instances for the instance group. Defaults to 0.
  • instance_type (pulumi.Input[str]) – The EC2 instance type for all instances in the instance group. Changing this forces a new resource to be created.
  • name (pulumi.Input[str]) – Human friendly name given to the instance group. Changing this forces a new resource to be created.
cluster_id = None

ID of the EMR Cluster to attach to. Changing this forces a new resource to be created.

ebs_configs = None

One or more ebs_config blocks as defined below. Changing this forces a new resource to be created.

ebs_optimized = None

Indicates whether an Amazon EBS volume is EBS-optimized. Changing this forces a new resource to be created.

instance_count = None

Target number of instances for the instance group. Defaults to 0.

instance_type = None

The EC2 instance type for all instances in the instance group. Changing this forces a new resource to be created.

name = None

Human friendly name given to the instance group. Changing this forces a new resource to be created.

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters:prop (str) – A property name.
Returns:A potentially transformed property name.
Return type:str
translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters:prop (str) – A property name.
Returns:A potentially transformed property name.
Return type:str
class pulumi_aws.emr.SecurityConfiguration(resource_name, opts=None, configuration=None, name=None, name_prefix=None, __name__=None, __opts__=None)

Provides a resource to manage AWS EMR Security Configurations

Parameters:
  • resource_name (str) – The name of the resource.
  • opts (pulumi.ResourceOptions) – Options for the resource.
  • configuration (pulumi.Input[str]) – A JSON formatted Security Configuration
  • name (pulumi.Input[str]) – The name of the EMR Security Configuration. By default generated by Terraform.
  • name_prefix (pulumi.Input[str]) – Creates a unique name beginning with the specified prefix. Conflicts with name.
configuration = None

A JSON formatted Security Configuration

creation_date = None

Date the Security Configuration was created

name = None

The name of the EMR Security Configuration. By default generated by Terraform.

name_prefix = None

Creates a unique name beginning with the specified prefix. Conflicts with name.

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters:prop (str) – A property name.
Returns:A potentially transformed property name.
Return type:str
translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters:prop (str) – A property name.
Returns:A potentially transformed property name.
Return type:str