Caching Remote Buckets (User Guide)

This is a user guide for using the namespace cache feature to cache remote buckets using local storage resources.

For more information on caching use cases and design please refer to - TODO.

Step 0 - Prerequisites

1. NooBaa installed in Kubernetes - see https://github.com/noobaa/noobaa-core#deploy-to-kubernetes
2. noobaa cli - get from https://github.com/noobaa/noobaa-operator/releases
3. Account credentials to the Hub

Step 1 - Connect to the Hub

Use check_external_connection to check if the provided hub details are good - that the hub is accessible and the credentials work. Use add_external_connection to actually create the hub connection.

noobaa api account_api check_external_connection '{
  "endpoint_type": "S3_COMPATIBLE",
  "endpoint": "s3.hub.com",
  "identity": "123",
  "secret": "abc"
}'

noobaa api account_api add_external_connection '{
  "name": "hub",
  "endpoint_type": "S3_COMPATIBLE",
  "endpoint": "s3.hub.com",
  "identity": "123",
  "secret": "abc"
}'

Step 3 - Create a Cache Bucket

For every remote bucket create a namespace resource and a bucket with cache policy:

noobaa api pool_api create_namespace_resource '{
  "name": "hub-bucket1",
  "connection": "hub",
  "target_bucket": "bucket1"
}'

noobaa api bucket_api create_bucket '{
  "name": "bucket1-cached",
  "namespace":{
    "write_resource": "hub-bucket1",
    "read_resources": ["hub-bucket1"],
    "caching": { "ttl_ms": 60000 }
  }
}'

Step 4 - Connect an Application

Create account for the application - copy the access_key and secret_key from the response:

noobaa api account_api create_account '{
  "name": "app1",
  "email": "admin@app1.com",
  "has_login": false,
  "s3_access": true,
  "allow_bucket_creation": false,
  "allowed_buckets": {
    "full_permission": false,
    "permission_list": ["bucket1-cached"]
  }
}'

Application S3 config:

AWS_ACCESS_KEY_ID=<from create_account response>
AWS_SECRET_ACCESS_KEY=<from create_account response>
S3_ENDPOINT=s3.noobaa.svc
BUCKET_NAME=bucket1-cached

Step 5 - Monitoring the Cache

TODO - utilization, hit rate, speed, etc.

Troubleshooting

noobaa api system_api read_system
noobaa api bucket_api list_buckets
kubectl logs noobaa-core-0