docs: misc updates (#450)

Co-authored-by: Helio Machado <0x2b3bfa0+git@googlemail.com>

Author: casperdcl

.gitignore

@@ -1,17 +1,11 @@
-.DS_Store
-examples/.terraform
-examples
 bin
 
-# Local .terraform directories
-**/.terraform/*
-
-# .tfstate files
-*.tfstate
-*.tfstate.*
-
-# Crash log files
-crash.log
+# Local terraform testing
+/main.tf
+/main.json
+/.terraform*
+/*.tfstate*
+/crash.log
 
 # Ignore any .tfvars files that are generated automatically for each Terraform run. Most
 # .tfvars files are managed as part of configuration and so should be included in

README.md

@@ -76,7 +76,7 @@ terraform init
 ### Run Task
 
 ```
-terraform apply
+TF_LOG_PROVIDER=INFO terraform apply
 ```
 
 This launches a `machine` in the `cloud`, uploads `workdir`, and runs the `script`. Upon completion (or error), the `machine` is terminated.
@@ -88,21 +88,21 @@ With spot/preemptible instances (`spot >= 0`), auto-recovery logic and persisten
 Results and logs are periodically synced to persistent cloud storage. To query this status and view logs:
 
 ```
-terraform refresh
-terraform show
+TF_LOG_PROVIDER=INFO terraform refresh
+TF_LOG_PROVIDER=INFO terraform show
 ```
 
 ### Stop Tasks
 
 ```
-terraform destroy
+TF_LOG_PROVIDER=INFO terraform destroy
 ```
 
 This terminates the `machine` (if still running), downloads `output`, and removes the persistent `disk_size` storage.
 
 ## Help
 
-The [getting started guide](https://registry.terraform.io/providers/iterative/iterative/latest/docs/guides/getting-started) has some more information.
+The [getting started guide](https://registry.terraform.io/providers/iterative/iterative/latest/docs/guides/getting-started) has some more information. In case of errors, extra debugging information is available using `TF_LOG_PROVIDER=DEBUG` instead of `INFO`.
 
 Feature requests and bugs can be [reported via GitHub issues](https://github.com/iterative/terraform-provider-iterative/issues), while general questions and feedback are very welcome on our active [Discord server](https://discord.gg/bzA6uY7).
 

docs/guides/authentication.md

@@ -8,7 +8,7 @@ Environment variables are the only supported authentication method, and should b
 
 ```bash
 export GOOGLE_APPLICATION_CREDENTIALS_DATA="$(cat service_account.json)"
-terraform apply
+TF_LOG_PROVIDER=INFO terraform apply
 ```
 
 ## Amazon Web Services

docs/guides/getting-started.md

@@ -7,16 +7,19 @@ page_title: Getting Started
 ## Requirements
 
 - [Install Terraform 1.0+](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform), e.g.:
+
   - Brew (Homebrew/Mac OS): `brew tap hashicorp/tap && brew install hashicorp/tap/terraform`
   - Choco (Chocolatey/Windows): `choco install terraform`
   - Conda (Anaconda): `conda install -c conda-forge terraform`
   - Debian (Ubuntu/Linux):
-    ```
+
+    ```console
     sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
     curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
     sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
     sudo apt-get update && sudo apt-get install terraform
     ```
+
 - Create an account with any supported cloud vendor and expose its [authentication credentials via environment variables][authentication]
 
 [authentication]: https://registry.terraform.io/providers/iterative/iterative/latest/docs/guides/authentication
@@ -74,7 +77,7 @@ This command will check `main.tf` and download the required TPI plugin.
 ## Run Task
 
 ```console
-$ terraform apply
+$ TF_LOG_PROVIDER=INFO terraform apply
 ```
 
 This command will:
@@ -92,7 +95,8 @@ With spot/preemptible instances (`spot >= 0`), auto-recovery logic and persisten
 ## Query Status
 
 ```console
-$ terraform refresh && terraform show
+$ TF_LOG_PROVIDER=INFO terraform refresh
+$ TF_LOG_PROVIDER=INFO terraform show
 ```
 
 These commands will:
@@ -103,7 +107,7 @@ These commands will:
 ## Stop Task
 
 ```console
-$ terraform destroy
+$ TF_LOG_PROVIDER=INFO terraform destroy
 ```
 
 This command will:
@@ -114,3 +118,26 @@ This command will:
 In this example, after running `terraform destroy`, the `results` directory should contain a file named `greeting.txt` with the text `Hello, World!`
 
 -> **Note:** A large `output` directory may take a long time to download.
+
+## Debugging
+
+Use `TF_LOG_PROVIDER=DEBUG` in lieu of `INFO` to increase verbosity for debugging. See the [logging docs](https://www.terraform.io/plugin/log/managing) for a full list of options.
+
+In case of errors within the `script` itself, both `stdout` and `stderr` are available from the [status](#query-status).
+
+Advanced users may also want to SSH to debug failed scripts. This means preventing TPI from terminating the instance on `script` errors. For example:
+
+```hcl
+timeout     = 60*60*24               # 24h
+environment = { GITHUB_ACTOR = "" }  # optional: GitHub username
+script      = <<-END
+  #!/bin/bash
+  trap 'echo script error: waiting 24h for debugging over SSH. Run \"terraform destroy\" to stop waiting; sleep 24h' ERR
+  # optional: allow GitHub user's ssh keys.
+  # alternatively, use `ssh_private_key` and `addresses` from
+  # https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#attribute-reference
+  curl "https://github.com/$GITHUB_ACTOR.keys" >> "$HOME/.ssh/authorized_keys"
+
+  # ... rest of script ...
+END
+```

docs/index.md

@@ -1,6 +1,6 @@
 # Terraform Provider Iterative (TPI)
 
-![TPI](https://static.iterative.ai/img/cml/banner-terraform.png)
+![TPI](https://static.iterative.ai/img/cml/banner-tpi.svg)
 
 [![tests](https://img.shields.io/github/workflow/status/iterative/terraform-provider-iterative/Test?label=tests&logo=GitHub)](https://github.com/iterative/terraform-provider-iterative/actions/workflows/test.yml)
 [![Apache-2.0](https://img.shields.io/badge/licence-Apache%202.0-blue)](https://github.com/iterative/terraform-provider-iterative/blob/master/LICENSE)

docs/resources/task.md

@@ -18,7 +18,7 @@ resource "iterative_task" "example" {
   disk_size   = 30        # GB
   spot        = 0         # auto-price. Or -1 to disable, or >0 to set a hourly USD limit
   parallelism = 1
-  timeout     = 3600      # max 1h idle
+  timeout     = 60*60     # max 1h before forced termination
 
   environment = { GREETING = "Hello, world!" }
   storage {
@@ -51,8 +51,8 @@ resource "iterative_task" "example" {
 - `storage.workdir` - (Optional) Local working directory to upload and use as the `script` working directory.
 - `storage.output` - (Optional) Results directory (**relative to `workdir`**) to download (default: no download).
 - `environment` - (Optional) Map of environment variable names and values for the task script. Empty string values are replaced with local environment values. Empty values may also be combined with a [glob](<https://en.wikipedia.org/wiki/Glob_(programming)>) name to import all matching variables.
-- `timeout` - (Optional) Maximum number of seconds to run before termination.
-- `name` - (Optional) Discouraged and may be removed in future. Deterministic task name (e.g. `name="Hello, World!"` always produces `id="tpi-hello-world-5kz6ldls-57wo7rsp"`).
+- `timeout` - (Optional) Maximum number of seconds to run before instances are force-terminated. The countdown is reset each time TPI auto-respawns a spot instance.
+- `name` - (Optional) _Discouraged and may be removed in future - change the resource name instead, i.e. `resource "iterative_task" "some_other_example_name"`._ Deterministic task name (e.g. `name="Hello, World!"` always produces `id="tpi-hello-world-5kz6ldls-57wo7rsp"`).
 
 -> **Note:** `output` is relative to `workdir`, so `storage { workdir = "foo", output = "bar" }` means "upload `./foo/`, change working directory to the uploaded folder, run `script`, and download `bar` (i.e. `./foo/bar`)".
 
@@ -70,6 +70,38 @@ In addition to all arguments above, the following attributes are exported:
 
 ~> **Warning:** `events` have different formats across cloud providers and cannot be relied on for programmatic consumption/automation.
 
+After `terraform apply`, these attributes may be obtained using `terraform console`. For example:
+
+```console
+$ echo 'try(join("\n", iterative_task.example.logs), "")' | terraform console
+```
+
+For convenience, this can placed in an `output` block in `main.tf`:
+
+```hcl
+resource "iterative_task" "example" {
+  ...
+}
+output "logs" {
+  value = try(join("\n", iterative_task.example.logs), "")
+}
+```
+
+The above would allow:
+
+```console
+$ terraform output --raw logs
+```
+
+Finally, JSON output can be parsed using `terraform output --json` and `jq` like this:
+
+```console
+$ terraform show --json | jq --raw-output '
+  .values.root_module.resources[] |
+  select(.address == "iterative_task.example") |
+  .values.logs[]'
+```
+
 ## Machine Type
 
 ### Generic