Skip to content

Commit 3399b3c

Browse files
MyroslavLevchykMyroslavLevchyk
MyroslavLevchyk
authored and
MyroslavLevchyk
committed
feat: databricks aws runtime
1 parent 2be3dfc commit 3399b3c

File tree

10 files changed

+347
-2
lines changed

10 files changed

+347
-2
lines changed

‎README.md

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,63 @@
1-
# Azure <> Terraform module
2-
Terraform module for creation Azure <>
1+
# Databricks Runtime Terraform module
2+
Terraform module used for management of Databricks Runtime Resources
33

44
## Usage
55

66
<!-- BEGIN_TF_DOCS -->
7+
## Requirements
78

9+
| Name | Version |
10+
|------|---------|
11+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
12+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | ~>5.0 |
13+
| <a name="requirement_databricks"></a> [databricks](#requirement\_databricks) | ~>1.0 |
14+
15+
## Providers
16+
17+
| Name | Version |
18+
|------|---------|
19+
| <a name="provider_databricks"></a> [databricks](#provider\_databricks) | ~>1.0 |
20+
21+
## Modules
22+
23+
No modules.
24+
25+
## Resources
26+
27+
| Name | Type |
28+
|------|------|
29+
| [databricks_cluster.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster) | resource |
30+
| [databricks_entitlements.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/entitlements) | resource |
31+
| [databricks_ip_access_list.allowed_list](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/ip_access_list) | resource |
32+
| [databricks_permissions.clusters](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/permissions) | resource |
33+
| [databricks_permissions.sql_endpoints](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/permissions) | resource |
34+
| [databricks_secret.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret) | resource |
35+
| [databricks_secret_scope.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret_scope) | resource |
36+
| [databricks_sql_endpoint.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/sql_endpoint) | resource |
37+
| [databricks_workspace_conf.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/workspace_conf) | resource |
38+
| [databricks_current_metastore.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/current_metastore) | data source |
39+
| [databricks_group.account_groups](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/group) | data source |
40+
| [databricks_sql_warehouses.all](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/sql_warehouses) | data source |
41+
42+
## Inputs
43+
44+
| Name | Description | Type | Default | Required |
45+
|------|-------------|------|---------|:--------:|
46+
| <a name="input_cloud_name"></a> [cloud\_name](#input\_cloud\_name) | Cloud Name | `string` | n/a | yes |
47+
| <a name="input_clusters"></a> [clusters](#input\_clusters) | Set of objects with parameters to configure Databricks clusters and assign permissions to it for certain custom groups | <pre>set(object({<br/> cluster_name = string<br/> spark_version = optional(string, "14.3.x-scala2.12")<br/> node_type_id = optional(string, "m5d.large")<br/> autotermination_minutes = optional(number, 20)<br/> min_workers = optional(number, 1)<br/> max_workers = optional(number, 2)<br/> availability = optional(string, "ON_DEMAND")<br/> zone_id = optional(string, "auto")<br/> first_on_demand = optional(number, 1)<br/> spot_bid_price_percent = optional(number, 100)<br/> data_security_mode = optional(string, "USER_ISOLATION")<br/> ebs_volume_count = optional(number, 1)<br/> ebs_volume_size = optional(number, 100)<br/> ebs_volume_type = optional(string, "GENERAL_PURPOSE_SSD")<br/> permissions = optional(list(object({<br/> group_name = string,<br/> permission_level = string<br/> })), []),<br/> }))</pre> | `[]` | no |
48+
| <a name="input_custom_config"></a> [custom\_config](#input\_custom\_config) | Map of AD databricks workspace custom config | `map(string)` | <pre>{<br/> "enable-X-Content-Type-Options": "true",<br/> "enable-X-Frame-Options": "true",<br/> "enable-X-XSS-Protection": "true",<br/> "enableDbfsFileBrowser": "false",<br/> "enableExportNotebook": "false",<br/> "enableIpAccessLists": "true",<br/> "enableNotebookTableClipboard": "false",<br/> "enableResultsDownloading": "false",<br/> "enableUploadDataUis": "false",<br/> "enableVerboseAuditLogs": "true",<br/> "enforceUserIsolation": "true",<br/> "storeInteractiveNotebookResultsInCustomerAccount": "true"<br/>}</pre> | no |
49+
| <a name="input_iam_account_groups"></a> [iam\_account\_groups](#input\_iam\_account\_groups) | List of objects with group name and entitlements for this group | <pre>list(object({<br/> group_name = optional(string)<br/> entitlements = optional(list(string))<br/> }))</pre> | `[]` | no |
50+
| <a name="input_ip_addresses"></a> [ip\_addresses](#input\_ip\_addresses) | A map of IP address ranges | `map(string)` | <pre>{<br/> "all": "0.0.0.0/0"<br/>}</pre> | no |
51+
| <a name="input_secret_scopes"></a> [secret\_scopes](#input\_secret\_scopes) | A list of secret scopes to be created | <pre>list(object({<br/> scope_name = string<br/> scope_permissions = optional(set(object({<br/> group_name = string<br/> permission_level = string<br/> })))<br/> secrets = optional(list(object({<br/> key = string<br/> value = string<br/> })), [])<br/> }))</pre> | `[]` | no |
52+
| <a name="input_sql_endpoint"></a> [sql\_endpoint](#input\_sql\_endpoint) | Set of objects with parameters to configure SQL Endpoint and assign permissions to it for certain custom groups | <pre>set(object({<br/> name = string<br/> cluster_size = optional(string, "2X-Small")<br/> auto_stop_mins = optional(number, 15)<br/> max_num_clusters = optional(number, 1)<br/> enable_photon = optional(bool, false)<br/> enable_serverless_compute = optional(bool, true)<br/> spot_instance_policy = optional(string, "COST_OPTIMIZED")<br/> warehouse_type = optional(string, "PRO")<br/> key = optional(string, "user")<br/> value = optional(string, "terraform")<br/> permissions = optional(list(object({<br/> group_name = string,<br/> permission_level = string<br/> })), []),<br/> }))</pre> | `[]` | no |
53+
54+
## Outputs
55+
56+
| Name | Description |
57+
|------|-------------|
58+
| <a name="output_metastore_id"></a> [metastore\_id](#output\_metastore\_id) | The ID of the current metastore in the Databricks workspace. |
59+
| <a name="output_sql_warehouses_list"></a> [sql\_warehouses\_list](#output\_sql\_warehouses\_list) | List of IDs of all SQL warehouses in the Databricks workspace. |
60+
| <a name="output_test"></a> [test](#output\_test) | Full list of IAM account groups for the workspace. |
861
<!-- END_TF_DOCS -->
962

1063
## License

‎cluster.tf

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
resource "databricks_cluster" "this" {
2+
for_each = { for cluster in var.clusters : cluster.cluster_name => cluster }
3+
4+
cluster_name = each.value.cluster_name
5+
spark_version = each.value.spark_version
6+
node_type_id = each.value.node_type_id
7+
autotermination_minutes = each.value.autotermination_minutes
8+
data_security_mode = each.value.data_security_mode
9+
autoscale {
10+
min_workers = each.value.min_workers
11+
max_workers = each.value.max_workers
12+
}
13+
14+
# Dynamic block for AWS
15+
dynamic "aws_attributes" {
16+
for_each = var.cloud_name == "aws" ? [each.value] : []
17+
content {
18+
availability = each.value.availability
19+
zone_id = each.value.zone_id
20+
first_on_demand = each.value.first_on_demand
21+
spot_bid_price_percent = each.value.spot_bid_price_percent
22+
ebs_volume_count = each.value.ebs_volume_count
23+
ebs_volume_size = each.value.ebs_volume_size
24+
ebs_volume_type = each.value.ebs_volume_type
25+
}
26+
}
27+
}
28+
29+
# Cluster permissions
30+
resource "databricks_permissions" "clusters" {
31+
for_each = { for cluster in var.clusters : cluster.cluster_name => cluster.permissions }
32+
33+
cluster_id = databricks_cluster.this[each.key].id
34+
35+
dynamic "access_control" {
36+
for_each = each.value
37+
content {
38+
group_name = access_control.value.group_name
39+
permission_level = access_control.value.permission_level
40+
}
41+
}
42+
}

‎data.tf

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
data "databricks_current_metastore" "this" {}
2+
3+
data "databricks_sql_warehouses" "all" {}

‎iam.tf

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
locals {
2+
iam_account_groups_full_list = distinct(flatten([var.clusters[*].permissions[*].group_name]))
3+
4+
iam_account_map = tomap({
5+
for group in var.iam_account_groups : group.group_name => group.entitlements
6+
if group.group_name != null
7+
})
8+
}
9+
10+
data "databricks_group" "account_groups" {
11+
for_each = local.iam_account_map
12+
13+
display_name = each.key
14+
}
15+
16+
resource "databricks_entitlements" "this" {
17+
for_each = local.iam_account_map
18+
19+
group_id = data.databricks_group.account_groups[each.key].id
20+
allow_cluster_create = contains(coalesce(each.value, ["none"]), "allow_cluster_create")
21+
allow_instance_pool_create = contains(coalesce(each.value, ["none"]), "allow_instance_pool_create")
22+
databricks_sql_access = contains(coalesce(each.value, ["none"]), "databricks_sql_access")
23+
workspace_access = true
24+
}

‎main.tf

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
resource "databricks_workspace_conf" "this" {
2+
custom_config = var.custom_config
3+
}
4+
5+
resource "databricks_ip_access_list" "allowed_list" {
6+
label = "allow_in"
7+
list_type = "ALLOW"
8+
ip_addresses = flatten([for v in values(var.ip_addresses) : v])
9+
10+
depends_on = [databricks_workspace_conf.this]
11+
}

‎outputs.tf

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
output "sql_warehouses_list" {
2+
value = data.databricks_sql_warehouses.all.ids
3+
description = "List of IDs of all SQL warehouses in the Databricks workspace."
4+
}
5+
6+
output "metastore_id" {
7+
value = data.databricks_current_metastore.this.id
8+
description = "The ID of the current metastore in the Databricks workspace."
9+
}
10+
11+
output "test" {
12+
value = local.iam_account_groups_full_list
13+
description = "Full list of IAM account groups for the workspace."
14+
}

‎secrets.tf

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
locals {
2+
secret_scope_config = { for object in var.secret_scopes : object.scope_name => object }
3+
4+
# TODO: Secret ACLs
5+
# secret_scope_config_acl = { for object in flatten([for k, v in local.secret_scope_config : [for permission in v.scope_permissions : {
6+
# scope_name = k,
7+
# acl_group_name = permission.group_name,
8+
# acl_permission_level = permission.permission_level,
9+
# }]]) : "${object.scope_name}:${object.acl_group_name}:${object.acl_pemission_level}" => object }
10+
11+
secret_scope_config_secrets = { for object in flatten([for k, v in local.secret_scope_config : [for secret in v.secrets : {
12+
scope_name = k,
13+
secret_key = secret.key,
14+
secret_value = secret.value,
15+
}]]) : "${object.scope_name}:${object.secret_key}" => object }
16+
}
17+
18+
resource "databricks_secret_scope" "this" {
19+
for_each = local.secret_scope_config
20+
21+
name = each.key
22+
}
23+
24+
# TODO: Secret ACLs
25+
#resource "databricks_secret_acl" "this" {
26+
# for_each = local.secret_scope_config_acl
27+
#
28+
# principal = databricks_group.ds.display_name
29+
# permission = "READ"
30+
# scope = databricks_secret_scope.app.name
31+
#}
32+
33+
resource "databricks_secret" "this" {
34+
for_each = local.secret_scope_config_secrets
35+
36+
key = each.value.secret_key
37+
string_value = each.value.secret_value
38+
scope = databricks_secret_scope.this[each.value.scope_name].name
39+
}

‎variables.tf

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
variable "cloud_name" {
2+
type = string
3+
description = "Cloud Name"
4+
}
5+
6+
variable "sql_endpoint" {
7+
type = set(object({
8+
name = string
9+
cluster_size = optional(string, "2X-Small")
10+
auto_stop_mins = optional(number, 15)
11+
max_num_clusters = optional(number, 1)
12+
enable_photon = optional(bool, false)
13+
enable_serverless_compute = optional(bool, true)
14+
spot_instance_policy = optional(string, "COST_OPTIMIZED")
15+
warehouse_type = optional(string, "PRO")
16+
key = optional(string, "user")
17+
value = optional(string, "terraform")
18+
permissions = optional(list(object({
19+
group_name = string,
20+
permission_level = string
21+
})), []),
22+
}))
23+
description = "Set of objects with parameters to configure SQL Endpoint and assign permissions to it for certain custom groups"
24+
default = []
25+
}
26+
27+
variable "clusters" {
28+
type = set(object({
29+
cluster_name = string
30+
spark_version = optional(string, "14.3.x-scala2.12")
31+
node_type_id = optional(string, "m5d.large")
32+
autotermination_minutes = optional(number, 20)
33+
min_workers = optional(number, 1)
34+
max_workers = optional(number, 2)
35+
availability = optional(string, "ON_DEMAND")
36+
zone_id = optional(string, "auto")
37+
first_on_demand = optional(number, 1)
38+
spot_bid_price_percent = optional(number, 100)
39+
data_security_mode = optional(string, "USER_ISOLATION")
40+
ebs_volume_count = optional(number, 1)
41+
ebs_volume_size = optional(number, 100)
42+
ebs_volume_type = optional(string, "GENERAL_PURPOSE_SSD")
43+
permissions = optional(list(object({
44+
group_name = string,
45+
permission_level = string
46+
})), []),
47+
}))
48+
description = "Set of objects with parameters to configure Databricks clusters and assign permissions to it for certain custom groups"
49+
default = []
50+
}
51+
52+
variable "custom_config" {
53+
type = map(string)
54+
description = "Map of AD databricks workspace custom config"
55+
default = {
56+
"enableResultsDownloading" = "false", # https://docs.databricks.com/en/notebooks/notebook-outputs.html#download-results
57+
"enableNotebookTableClipboard" = "false", # https://docs.databricks.com/en/administration-guide/workspace-settings/notebooks.html#enable-users-to-copy-data-to-the-clipboard-from-notebooks
58+
"enableVerboseAuditLogs" = "true", # https://docs.databricks.com/en/administration-guide/account-settings/verbose-logs.html
59+
"enable-X-Frame-Options" = "true",
60+
"enable-X-Content-Type-Options" = "true",
61+
"enable-X-XSS-Protection" = "true",
62+
"enableDbfsFileBrowser" = "false", # https://docs.databricks.com/en/administration-guide/workspace-settings/dbfs-browser.html
63+
"enableExportNotebook" = "false", # https://docs.databricks.com/en/administration-guide/workspace-settings/notebooks.html#enable-users-to-export-notebooks
64+
"enforceUserIsolation" = "true", # https://docs.databricks.com/en/administration-guide/workspace-settings/enforce-user-isolation.html
65+
"storeInteractiveNotebookResultsInCustomerAccount" = "true", # https://docs.databricks.com/en/administration-guide/workspace-settings/notebooks.html#manage-where-notebook-results-are-stored
66+
"enableUploadDataUis" = "false", # https://docs.databricks.com/en/ingestion/add-data/index.html
67+
"enableIpAccessLists" = "true"
68+
}
69+
}
70+
71+
variable "ip_addresses" {
72+
type = map(string)
73+
description = "A map of IP address ranges"
74+
default = {
75+
"all" = "0.0.0.0/0"
76+
}
77+
}
78+
79+
variable "secret_scopes" {
80+
type = list(object({
81+
scope_name = string
82+
scope_permissions = optional(set(object({
83+
group_name = string
84+
permission_level = string
85+
})))
86+
secrets = optional(list(object({
87+
key = string
88+
value = string
89+
})), [])
90+
}))
91+
description = "A list of secret scopes to be created"
92+
default = []
93+
}
94+
95+
variable "iam_account_groups" {
96+
type = list(object({
97+
group_name = optional(string)
98+
entitlements = optional(list(string))
99+
}))
100+
description = "List of objects with group name and entitlements for this group"
101+
default = []
102+
}

‎versions.tf

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
terraform {
2+
required_version = ">= 1.0"
3+
4+
required_providers {
5+
aws = {
6+
source = "hashicorp/aws"
7+
version = "~>5.0"
8+
}
9+
databricks = {
10+
source = "databricks/databricks"
11+
version = "~>1.0"
12+
}
13+
}
14+
}

‎warehouses.tf

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# SQL Endpoint
2+
resource "databricks_sql_endpoint" "this" {
3+
for_each = { for endpoint in var.sql_endpoint : endpoint.name => endpoint }
4+
5+
name = each.key
6+
cluster_size = each.value.cluster_size
7+
auto_stop_mins = each.value.auto_stop_mins
8+
max_num_clusters = each.value.max_num_clusters
9+
enable_photon = each.value.enable_photon
10+
spot_instance_policy = each.value.spot_instance_policy
11+
enable_serverless_compute = each.value.enable_serverless_compute
12+
warehouse_type = each.value.warehouse_type
13+
14+
tags {
15+
custom_tags {
16+
key = each.value.key
17+
value = each.value.value
18+
}
19+
}
20+
}
21+
22+
# SQL Endpoint permissions
23+
resource "databricks_permissions" "sql_endpoints" {
24+
for_each = { for sql in var.sql_endpoint : sql.name => sql.permissions }
25+
26+
sql_endpoint_id = databricks_sql_endpoint.this[each.key].id
27+
28+
dynamic "access_control" {
29+
for_each = each.value
30+
content {
31+
group_name = access_control.value.group_name
32+
permission_level = access_control.value.permission_level
33+
}
34+
}
35+
}
36+
37+
# NOTE: Waiting for resource to get NCC ID
38+
# resource "databricks_mws_ncc_binding" "ncc_binding" {
39+
# network_connectivity_config_id = databricks_mws_network_connectivity_config.ncc.network_connectivity_config_id
40+
# workspace_id = var.databricks_workspace_id
41+
42+
# provider = databricks.account
43+
# }

0 commit comments

Comments
 (0)