Merge branch 'simple'

This is a major change to deployment model.  All deployment is now done leveraging CloudInit, and user_data plus extended_metadata to enable "hands-off" cluster deployment.   This simplifies deployment of Cloudera Enterprise Data Hub on OCI, removing requirements for the end user to install python and dependent libraries, as well as needing the Cloudera Manager host to be publicly accessible.

Author: smithzc

README.md

@@ -9,79 +9,56 @@ Future development will include support for EDH v5 clusters.  In the meantime, u
 | Recommended | BM.DenseIO2.52 | VM.Standard2.4   | VM.Standard2.16              |
 | Minimum     | VM.Standard2.8 | VM.Standard2.1   | VM.Standard2.8               |
 
-Host types can be customized in this template.   Also included with this template is an easy method to customize block volume quantity and size as pertains to HDFS capacity.   See "variables.tf" for more information in-line.
+Host types can be customized in this template.   Also included with this template is an easy method to customize block volume quantity and size as pertains to HDFS capacity.   See [variables.tf](https://github.com/oracle/oci-quickstart-cloudera/blob/master/terraform/variables.tf#L48-L62)  for more information in-line.
 
 ## Prerequisites
 First off you'll need to do some pre deploy setup.  That's all detailed [here](https://github.com/oracle/oci-quickstart-prerequisites).
 
-## Additional Python Dependencies
-This module depends on Python, Paramiko, PIP, and cm_client.   These should be installed on the host you are using to deploy the Terraform module.  
-
-On EL7 hosts, installation can be performed using the following commands:
-
-	sudo yum install python python-pip python-paramiko.noarch -y
-	sudo pip install --upgrade pip
-	sudo pip install cm_client
-
-On Mac, installation can be peformed using the following commands:
-
-	curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
-	sudo python get-pip.py
-	sudo pip install --upgrade pip
-	sudo pip install cm_client paramiko
-
 ### Clone the Module
 Now, you'll want a local copy of this repo.  You can make that with the commands:
 
     git clone https://github.com/oracle/oci-quickstart-cloudera.git
     cd oci-quickstart-cloudera
-    ls
 
 ## Python Deployment using cm_client
-The deployment script "deploy_on_oci.py" uses cm_client against Cloudera Manger API v31.  As such it does require some customization before execution.  Reference the header section in the script, it is highly encouraged you modify the following variables before deployment, ssh_keyfile is required or deployment will fail:
+The deployment script "deploy_on_oci.py" uses cm_client against Cloudera Manger API v31.  As such it does require some customization before execution.  Reference the header section in the script, it is highly encouraged you modify the following variables before deployment:
 
 	admin_user_name
 	admin_password
 	cluster_name
-	ssh_keyfile (REQUIRED)
-	cluster_service_list
 
-Also if you modify the compute.tf in any way to change hostname parameters, you will need to update these variables for pattern matching, otherwise host detection and cluster layout will fail:
+Also if you modify the compute.tf in any way to change hostname parameters, you will need to update these variables for pattern matching, otherwise cluster deployment will fail:
 
-	worker_hosts_contain
-	master_hosts_contain
-	namenode_host_contains
-	secondary_namenode_host_contains
-	cloudera_manager_host_contains
+	worker_hosts_prefix = 'cdh-worker'
+	namenode_host = 'cdh-master-1'
+	secondary_namenode_host = 'cdh-master-2'
+	cloudera_manager_host = 'cdh-utility-1'
 
 In addition, further customization of the cluster deployment can be done by modification of the following functions:
 
 	setup_mgmt_rcg
 	update_cluster_rcg_configuration
 
-This does require some knowledge of Python - modify at your own risk.  These functions contain Cloudera specific tuning parameters as well as host mapping for roles.
+This does require some knowledge of Python and Cloudera - modify at your own risk.  These functions contain Cloudera specific tuning parameters as well as host mapping for roles.
 
-## Kerberos Secure Cluster by Default
+## Kerberos Secure Cluster option
 
-This automation now defaults to using a local KDC deployed on the Cloudera Manager instance for secure cluster operation.  Please read the scripts [README](https://github.com/oci-quickstart/oci-cloudera/blob/master/scripts/README.md) for information regarding how to set these parameters prior to deployment.
+This automation supports using a local KDC deployed on the Cloudera Manager instance for secure cluster operation.  Please read the scripts [README](https://github.com/oracle/oci-quickstart-cloudera/blob/master/scripts/README.md) for information regarding how to set these parameters prior to deployment.
 
 Also - for cluster management, you will need to manually create at a minimum the HDFS Superuser Principal as [detailed here](https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_using_cm_sec_config.html#create-hdfs-superuser) after deployment.
 
-## Cloudera Manager and Cluster Metadata Database
-You are able to customize which database you want to use for Cloudera Manager and Cluster Metadata.   In compute.tf you will see a "user_data" field for the Utility instance:
-
-	user_data           = "${base64encode(file("scripts/cm_boot_mysql.sh"))}"
+Enabling Kerberos is managed using a terraform metadata tag "deployment_type" which is set in [variables.tf](https://github.com/oracle/oci-quickstart-cloudera/blob/master/terraform/variables.tf#L32).   Setting this value to "secure" will enable cluster security as part of the setup process.  Changing this to "simple" will deploy an unsecured cluster.
 
-This is set to use MySQL for the database.  If you want to use Postgres, you would change it:
+## High Availability
 
-	user_data           = "${base64encode(file("scripts/cm_boot_postgres.sh"))}"
+High Availability is also offered as part of the deployment process.  When secure cluster operation is chosen this is enabled by default.  It can be disabled by either changing the deployment_type to "simple", or modifying the [deploy_on_oci.py](https://github.com/oracle/oci-quickstart-cloudera/blob/master/scripts/deploy_on_oci.py#L60) script and changing the value for "hdfs_ha" to False.
 
-You can customize the default root password for MySQL by editing the source script.  For the various Cloudera databases, random passwords are generated and used.  The same is true when using Postgres.
+## Metadata and MySQL
 
-Note that you will also need to change "meta_db_port" in deploy_on_oci.py if you choose to run Postgres.
+You can customize the default root password for MySQL by editing the source script [cms_mysql.sh](https://github.com/oracle/oci-quickstart-cloudera/blob/master/scripts/cms_mysql.sh#L188).  For the various Cloudera databases, random passwords are generated and used.  These are stored in a flat file on the Utility host for use at deployment time.
 
 ## Object Storage Integration
-As of the 2.1.0 release, included with this template is a means to deploy clusters with configuration to allow use of OCI Object Storage using S3 Compatability.  In order to implement, an S3 Access and Secret key must be set up in the OCI Tenancy first.  This process is detailed [here](https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#Working2).  Once that is in place, modify the [deploy_on_oci.py](https://github.com/oci-quickstart/oci-cloudera/blob/master/scripts/deploy_on_oci.py#L133-L141) script, and set the following values:
+As of the 2.1.0 release, included with this template is a means to deploy clusters with configuration to allow use of OCI Object Storage using S3 Compatability.  In order to implement, an S3 Access and Secret key must be set up in the OCI Tenancy first.  This process is detailed [here](https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#Working2).  Once that is in place, modify the [deploy_on_oci.py](https://github.com/oracle/oci-quickstart-cloudera/blob/master/scripts/deploy_on_oci.py#L101-L108) script, and set the following values:
 
 	s3_compat_enable = 'False'
 	s3a_secret_key = 'None'
@@ -97,23 +74,17 @@ Deployment of the module is straight forward using the following Terraform comma
 	terraform plan
 	terraform apply
 
-This will create all the required elements in a compartment in the target OCI tenancy.  This includes VCN and Security List parameters.  Security audit of these in the network.tf is suggested.
-
-After Terraform is finished deploying, the output will show the Python syntax to trigger cluster deployment.  This command can be run immediately following deployment, as it has built-in checks to wait until Cloudera Manager API is up and responding before it executes deployment.  The syntax is as follows:
-
-	python scripts/deploy_on_oci.py -B -m <master_ip> -d <disk_count> -w <worker_shape>
-
-It is also possible to destroy an existing cluster with this script using Cloudera Manager
-
-	python scripts/deploy_on_oci.py -D -m <master_ip>
+This will create all the required elements in a compartment in the target OCI tenancy.  This includes VCN and Security List parameters.  Security audit of these in the [network module](https://github.com/oracle/oci-quickstart-cloudera/blob/master/terraform/modules/network/main.tf) is suggested.
 
 ## Destroy the Deployment
 
 When you no longer need the deployment, you can run this command to destroy it:
 
 	terraform destroy
 
-## Deployment Caveats
-Currently this module requires Cloudera Manager API to be on an edge host with a Public IP address.   This is used to trigger cluster deployment, as well as SSH into the Cloudera Manger host to perform dynamic host discovery to map for Cluster topology.   
+## Deployment Architecture
+
+Here is a diagram showing what is deployed using this template.   Note that resources are automatically distributed among Fault Domains in an Availability Domain to ensure fault tolerance.   Additional workers deployed will stripe between the 3 fault domains in sequence starting with the Fault Domain 1 and incrementing sequentially.
+
+![Deployment Architecture Diagram](https://github.com/oracle/oci-quickstart-cloudera/blob/master/images/deployment_architecture.png)
 
-Future enhancements to this module are planned to support a completely Private (non-Internet exposed) cluster deployment.

images/deployment_architecture.png

@@ -1,11 +1,12 @@
 # scripts
 All scripts in this location are referenced for deployment automation
 
-* boot.sh is invoked by cloudinit on each instance creation via Terraform.  It contains steps which perform inital bootstrapping of the instance prior to provisioning.
-* boot_plus_tmp.sh is an alternate version of boot.sh which demonstrates configuring a RAID0 Block Volume array for use as /tmp.  This is useful for caching data when using Object Storage directly.
-* cm_boot_mysql.sh is invoked by cloudinit on the Utility node to stand up Cloudera Manager and Pre-requisites using MySQL for Metadata.
-* cm_boot_postgres.sh can be used instead of cm_boot_mysql.sh if you want to use Postgres for Cloudera Manager and Cluster Metadata.
-* deploy_on_oci.py is the primary Python script invoked to deploy Cloudera EDH v6 using cm_client python libraries
+* boot.sh is invoked by CloudInit on each instance creation via Terraform.  It contains steps which perform inital bootstrapping of the instance prior to provisioning.
+* boot_plus_tmp.sh is an alternate version of boot.sh which demonstrates configuring a RAID0 Block Volume array for use as /tmp.  This is useful for caching data when using Object Storage.
+* cloudera_manager_boot.sh is a top level boot script for Cloudera Manager (Utility) instance.  This is required because subsequent scripts are too large to fit in metadata without compression.
+* cms_mysql.sh is invoked by cloudinit on the Utility node to stand up Cloudera Manager and Pre-requisites using MySQL for Metadata. It is compressed and loaded into extended metadata.
+* cms_postgres.sh is an older installaltion method using Postgres instead of MySQL for cluster metadata.  This is depracated.
+* deploy_on_oci.py is the primary Python script invoked to deploy Cloudera EDH v6 using cm_client python libraries.  It is compressed and loaded into extended metdata.
 
 # CloudInit boot scripts
 
@@ -15,13 +16,13 @@ With the introduction of local KDC for secure cluster, this requires some setup
   * kdc_server - This is the hostname where KDC is deployed (defaults to Cloudera Manager host)
   * realm - This is set to hadoop.com by default.
   * REALM - This is set to HADOOP.COM by default.
-* cm_boot_mysql.sh
+* cms_mysql.sh
   * KERBEROS_PASSWORD - This is used for the root/admin account.
   * SCM_USER_PASSWORD - By default the cloudera-scm user is given admin control of the KDC.  This is required for Cloudera Manager to setup and manage principals, and the password here is used by that account.
   * kdc_server - Defaults to local hostname.
   * realm - This is set to hadoop.com by default.  
   * REALM - This is set to HADOOP.COM by default.
-* cm_boot_postgres.sh - Same items as cm_boot_mysql.sh
+* cms_postgres.sh - Same items as cm_boot_mysql.sh
 * deploy_on_oci.py
   * realm - This is HADOOP.COM by default.
   * kdc_admin - Set to cloudera-scm@HADOOP.COM by default.
@@ -30,4 +31,4 @@ With the introduction of local KDC for secure cluster, this requires some setup
 It is highly suggested you modify at a minimum the default passwords prior to deployment.
 
 ## CAUTION WHEN MODIFYING BOOT SCRIPTS
-Because boot.sh and cm_boot_mysql.sh/cm_boot_postgres.sh  are invoked as part of user_data in Terraform, if you modify these files and re-run a deployment, default behavior is existing instances will be destroyed and re-deployed because of this change.   
+Because boot.sh and cms_mysql.sh/cms_postgres.sh  are invoked as part of user_data and extended_metadata  in Terraform, if you modify these files and re-run a deployment, default behavior is existing instances will be destroyed and re-deployed because of this change.