Updated to support CDH v5

Author: smithzc

README.md

@@ -1,8 +1,8 @@
 # oci-quickstart-cloudera
 This is a Terraform module that deploys [Cloudera Enterprise Data Hub](https://www.cloudera.com/products/enterprise-data-hub.html) on [Oracle Cloud Infrastructure (OCI)](https://cloud.oracle.com/en_US/cloud-infrastructure).  It is developed jointly by Oracle and Cloudera.
 
-## Alternate Versions
-Future development will include support for EDH v5 clusters.  In the meantime, use the [1.0.0 release](https://github.com/oci-quickstart/oci-cloudera/releases/tag/1.0.0) for v5 deployments.
+## Deployment Information
+The following table shows Recommended and Minimum supported OCI shapes for each cluster role:
 
 |             | Worker Nodes   | Bastion Instance | Utility and Master Instances |
 |-------------|----------------|------------------|------------------------------|
@@ -12,6 +12,15 @@ Future development will include support for EDH v5 clusters.  In the meantime, u
 ## Resource Manager Deployment
 Using [OCI Resource Manager](https://docs.cloud.oracle.com/iaas/Content/ResourceManager/Concepts/resourcemanager.htm) makes deployment quite easy.  Simply [download the .zip](https://github.com/oracle/oci-quickstart-cloudera/zipball/resource-manager) and follow the [Resource Manager instructions](https://docs.cloud.oracle.com/iaas/Content/ResourceManager/Tasks/usingconsole.htm) for how to build a stack.  Prior to building the Stack, you may want to modify some parts of the deployment detailed in the sections below.
 
+Alternatively you can also use a schema file to make setting deployment variables much easier.   In order to leverage this feature, the GitHub zipball must be re-packaged so that it's contents are top-level prior to creating the ORM Stack.  This is a straight forward process:
+```
+unzip oci-quickstart-cloudera*.zip
+cd oci-quickstart-cloudera-<TAB_COMPLETE>
+zip -r oci-quickstart-cloudera.zip *
+```
+
+Use the oci-quickstart-cloudera.zip file created in the last step to create the ORM Stack.
+
 ## Python Deployment using cm_client
 The deployment script "deploy_on_oci.py" uses cm_client against Cloudera Manger API v31.  As such it does require some customization before execution.  Reference the header section in the script, it is highly encouraged you modify the following variables before deployment:
 
@@ -42,58 +51,7 @@ High Availability is also offered as part of the deployment process.  When secur
 You can customize the default root password for MySQL by editing the source script [cms_mysql.sh](https://github.com/oracle/oci-quickstart-cloudera/blob/master/scripts/cms_mysql.sh#L188).  For the various Cloudera databases, random passwords are generated and used.  These are stored in a flat file on the Utility host for use at deployment time.
 
 ## Object Storage Integration
-As of the 2.1.0 release, included with this template is a means to deploy clusters with configuration to allow use of OCI Object Storage using S3 Compatability.  In order to implement, an S3 Access and Secret key must be set up in the OCI Tenancy first.  This process is detailed [here](https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#Working2).  Once that is in place, modify the [deploy_on_oci.py](https://github.com/oracle/oci-quickstart-cloudera/blob/master/scripts/deploy_on_oci.py#L101-L108) script, and set the following values:
-
-	s3_compat_enable = 'False'
-	s3a_secret_key = 'None'
-	s3a_access_key = 'None'
-	s3a_endpoint = 'None'	
-
-The first should be set to 'True', then replace 'None" with each of the required values.   This configuration will then be pushed as part of the cluster deployment.
-
-## Resource Manager Variables
-Step 2 for setting up a stack is Configure Variables.   By default all variables are filled in, with the exception of the SSH Public and Private keypair used for host access.   If you don't have a keypair for use with this deployment, generating one on Linux/Mac is simply:
-
-	ssh-keygen -t rsa
-
-Follow the prompts to generate the key, do not associate a password with it.    Copy the contents of each file and paste into the appropriate variable fields as shown here:
-
-![Resource Manager Variables](https://github.com/oracle/oci-quickstart-cloudera/blob/resource-manager/images/RM_variables.png)
-
-This list also can be modified to suit your specific deployment requirements.   You should review the settings for the following and ensure you have the capacity in your Tenancy prior to deployment:
-
-	worker_instance_shape
-	worker_node_count
-	block_volumes_per_worker
-	utility_instance_shape
-	master_instance_shape
-	bastion_instance_shape
-
-Note that it is not suggested to modify the data_blocksize_in_gbs to lower than the default value of 700GB.   This is because 700GB is the minimum value to achieve maximum throughput per block volume.  Lowering this has a negative impact on HDFS performance.  If you need more HDFS capacity, best practice is to increase the block_volumes_per_worker which adds more DFS volumes for capacity and aggregate throughput.  For even higher density, the data_blocksize_in_gbs can be increased in tandem.   
-
-When using DenseIO shapes, it's also possible to set the block_volumes_per_worker to "0" to leverage only local NVME disk for HDFS.   In the case that you have both local NVME and block, data tiering will automatically be enabled as part of the deployment process.
-
-## Resource Manager Stack Steps
-After building the stack, it only takes 2 actions to deploy:
-
-	Terraform Actions -> Plan
-	Terraform Actions -> Apply
-
-This will create all the required elements in a compartment in the target OCI tenancy.  This includes VCN and Security List parameters.  Security audit of these in the [network module](https://github.com/oracle/oci-quickstart-cloudera/blob/master/terraform/modules/network/main.tf) is suggested.
-
-The output of the Apply command will contain a URL to access Cloudera Manager.   This is the public IP of the Utility Host, which runs the deployment.   
-
-## Monitoring Cluster Build
-Because all tasks are done in CloudInit, there are two ways to monitor the deployment.   Firstly you can login to the Cloudera Manager URL once it is up and running a few minutes after the Apply command finishes.  Alternatively you can SSH into the Utility node, and monitor the log file "/var/log/cloudera-OCI-initialize.log" which contains detailed output from the deployment.
-
-## Destroy the Deployment
-
-When you no longer need the deployment, you can destroy it:
-
-	Terraform Actions -> Destroy
-
-## Deployment Architecture
-
+???LINES MISSING
 Here is a diagram showing what is deployed using this template.   Note that resources are automatically distributed among Fault Domains in an Availability Domain to ensure fault tolerance.   Additional workers deployed will stripe between the 3 fault domains in sequence starting with the Fault Domain 1 and incrementing sequentially.
 
 ![Deployment Architecture Diagram](https://github.com/oracle/oci-quickstart-cloudera/blob/master/images/deployment_architecture.png)

schema.yaml

@@ -67,6 +67,13 @@ variables:
   cdh_version:
     type: enum
     enum:
+    - "5.10.2.5"
+    - "5.11.2.4"
+    - "5.12.2.4"
+    - "5.13.3.2"
+    - "5.14.4.3"
+    - "5.15.2.3"
+    - "5.16.2.8"
     - "6.0.0"
     - "6.0.1"
     - "6.1.0"

scripts/deploy_on_oci.py

@@ -20,11 +20,7 @@
 import subprocess
 
 start_time = time.time()
-
-#
 # Global Parameter Defaults - These are passed to the script, do not modify
-#
-
 disk_count = 'None'
 worker_shape = 'None'
 cm_server = 'None'
@@ -33,53 +29,36 @@
 host_fqdn_list = []
 data_tiering = 'False'
 nvme_disks = 0
-cluster_version = '6.2.0'  # type: str
 deployment_type = 'simple'  # type: str
-
-#
 # Custom Global Parameters - Customize below here
-#
-
-# Enable Debug Output set this to 'True' for detailed API output during execution
 debug = 'False'  # type: str
-
 # Define new admin username and password for Cloudera Manager
 # This replaces the default (insecure) admin/admin account
 admin_user_name = 'cdhadmin'  # type: str
 admin_password = 'somepassword'  # type: str
-
-# Define cluster name
+# Defaults
 cluster_name = 'TestCluster'  # type: str
-
-# Set this to 'True' (default) to enable secure cluster (Kerberos) functionality
-# Set this to 'False" to deploy an insecure cluster - This is automatically off for simple deployment 
+cdh_version = ' '  # type: str
+cluster_primary_version = ' '
+kafka_parcel_url = ' '
 secure_cluster = 'True'  # type: bool
-
-# Set this to 'False' if you do not want HDFS HA - useful for Development or if you want to save some setup time
-# This is automatically off for simple deployment
 hdfs_ha = 'True'  # type: bool
-
-# They should match what is in the Cloudera Manager CloudInit bootstrap file and instance boot files
+# These should match what is in the Cloudera Manager CloudInit bootstrap file and instance boot files
 realm = 'HADOOP.COM'
 kdc_admin = 'cloudera-scm@HADOOP.COM'
 kdc_password = 'somepassword'
-
 # Set port number for Cloudera Manager - used to build API endpoints and check if CM is up/listening
 # Only modify if you customized the Cloudera Manager deployment to use a non-standard port
 cm_port = '7180'
 # Set API version to use with Cloudera Manager
 api_version = 'v31'
-
 # Define DB port number for Cluster Metadata
 # This should match the CloudInit boot file you chose in Terraform
 # For MySQL this is 3306
 # For Postgres this is 5432
 meta_db_port = '3306'
-
-# Define Remote Parcel URL & Distribution Rate if desired
-remote_parcel_url = 'https://archive.cloudera.com/cdh6/' + cluster_version + '/parcels'  # type: str
+# Distribution rate for remote repo
 parcel_distribution_rate = "1024000"  # type: int
-
 # Cluster Services List
 cluster_service_list = ['SOLR', 'HBASE', 'HIVE', 'SPARK_ON_YARN', 'HDFS', 'OOZIE', 'SQOOP_CLIENT', 'ZOOKEEPER',
                         'YARN', 'KAFKA', 'IMPALA']
@@ -366,12 +345,12 @@ def delete_default_admin_user():
         print('Exception when calling UsersResourceApi->delete_user2: {}\n'.format(e))
 
 
-def init_cluster():
+def init_cluster(cdh_version):
     """
     Initialize Cluster
     :return:
     """
-    cluster = [cm_client.ApiCluster(name=cluster_name, display_name=cluster_name, full_version=cluster_version)]
+    cluster = [cm_client.ApiCluster(name=cluster_name, display_name=cluster_name, full_version=cdh_version)]
     body = cm_client.ApiClusterList(cluster)
 
     try:
@@ -574,7 +553,8 @@ def dda_parcel(parcel_product):
     :param parcel_product: Parcel Product Name - e.g. CDH, SPARK_ON_YARN
     :return:
     """
-
+    parcel_version = ' '
+    target_stage = ' '
     def monitor_parcel(parcel_product, parcel_version, target_stage):
         while True:
             parcel = parcel_api.read_parcel(cluster_name, parcel_product, parcel_version)
@@ -2081,9 +2061,11 @@ def options_parser(args=None):
     :return:
     """
     global objects
+    global remote_parcel_url
+    global cdh_version
     parser = argparse.ArgumentParser(prog='python deploy_on_oci.py', description='Deploy a Cloudera EDH %s Cluster on '
                                                                                  'OCI using cm_client with Cloudera '
-                                                                                 'Manager API %s' % (cluster_version,
+                                                                                 'Manager API %s' % (cdh_version,
                                                                                                      api_version))
     parser.add_argument('-D', '--deployment_type', metavar='deployment_type', help='simple, no HA or Kerberos at deployment, or secure to enable both')
     parser.add_argument('-m', '--cm_server', metavar='cm_server', required='True',
@@ -2101,6 +2083,19 @@ def options_parser(args=None):
     parser.add_argument('-ad', '--availability_domain', metavar='availability_domain', help='OCI Availability Domain')
     parser.add_argument('-N', '--cluster_name', metavar='cluster_name', help='CDH Cluster Name')
     options = parser.parse_args(args)
+    cluster_primary_version = options.cdh_version.split('.')
+    cluster_primary_version = cluster_primary_version[0]
+    cdh_version = options.cdh_version
+    if cluster_primary_version == '6':
+	remote_parcel_url = 'https://archive.cloudera.com/cdh6/' + options.cdh_version + '/parcels'  # type: str
+    else:
+	remote_parcel_url = 'https://archive.cloudera.com/cdh5/parcels/' + options.cdh_version  #type: str
+        if options.cdh_version.split('.')[2] >= '13':
+                kafka_version = '4.1.0.4'
+	else:
+                kafka_version = '2.2.0.68'
+
+        kafka_parcel_url = 'https://archive.cloudera.com/kafka/parcels/' + kafka_version  # type: str
     if not options.cm_server:
         print('Cloudera Manager Server IP required.')
         parser.print_help()
@@ -2134,13 +2129,14 @@ def options_parser(args=None):
             sys.exit()
 
     return (options.cm_server, options.input_host_list, options.disk_count, options.license_file, options.worker_shape,
-            options.num_workers, options.deployment_type, options.cdh_version, options.availability_domain, options.cluster_name)
+            options.num_workers, options.deployment_type, options.cdh_version, options.availability_domain, options.cluster_name,
+            cluster_primary_version, kafka_parcel_url)
 
 #
 # MAIN FUNCTION FOR CLUSTER DEPLOYMENT
 #
 
-def build_cloudera_cluster():
+def build_cloudera_cluster(cluster_primary_version):
     """
     Deploy and Configure a Cloudera EDH Cluster
     :return:
@@ -2166,7 +2162,7 @@ def build_cloudera_cluster():
     build_success = 'False'
     while build_success == 'False':
         print('->Initializing Cluster %s' % cluster_name)
-        init_cluster()
+        init_cluster(cdh_version)
         build_host_list(input_host_list)
         if len(host_fqdn_list) < 6:
             print('Error - %d hosts found, Minimum 6 required to build %s!' % (len(host_fqdn_list), cluster_name))
@@ -2191,6 +2187,12 @@ def build_cloudera_cluster():
     update_parcel_repo(remote_parcel_url, parcel_distribution_rate)
     print('->Parcel Setup Running')
     dda_parcel('CDH')
+    if cluster_primary_version == '5':
+        update_parcel_repo(kafka_parcel_url, parcel_distribution_rate)
+        dda_parcel('KAFKA')
+    else:
+        pass
+    time.sleep(10) 
     print('->Mapping Cluster Hostnames and Host IDs')
     cluster_host_id_map()
     print('->Reading DB Passwords')
@@ -2320,7 +2322,7 @@ def enable_kerberos():
 #
 
 if __name__ == '__main__':
-    cm_server, input_host_list, disk_count, license_file, worker_shape, num_workers, deployment_type, cdh_version, cms_version, cluster_name =\
+    cm_server, input_host_list, disk_count, license_file, worker_shape, num_workers, deployment_type, cdh_version, cms_version, cluster_name, cluster_primary_version, kafka_parcel_url =\
         options_parser(sys.argv[1:])
     if debug == 'True':
         print('cm_server = %s' % cm_server)
@@ -2366,7 +2368,7 @@ def enable_kerberos():
     else:
         print('Cluster Deployment options - HA: %s - Kerberos: %s' % (hdfs_ha, secure_cluster))
 
-    build_cloudera_cluster()
+    build_cloudera_cluster(cluster_primary_version)
     if deployment_type == 'simple':
         exit(0)
     else: