Skip to content

add dev cloudcasting api module and vars #895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

add dev cloudcasting api module and vars #895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
912ddbd
add dev cloudcasting api module and vars
braddf Aug 20, 2025
7eacdea
add cloudcasting app to readme; fix log links
braddf Aug 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions terraform/nowcasting/development/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,17 +89,23 @@ and [Airflow Dag (GSP)](https://github.com/openclimatefix/ocf-infrastructure/blo
- AWS logs: [aws/ecs/forecast_blend/](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Fecs$252Fforecast_blend$252F)

### 🚀 UK API (Elastic Beanstalk App)
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beantstalk. We use Auth0 to authenticate this API.
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beanstalk. We use Auth0 to authenticate this API.
- Code: [uk-pv-national-gsp-api](https://github.com/openclimatefix/uk-pv-national-gsp-api)
- [Terraform](https://github.com/openclimatefix/ocf-infrastructure/tree/main/terraform/modules/services/api)
- [AWS logs on development](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Felasticbeanstalk$252Fnowcasting-api-development$252Fvar$252Flog$252Feb-docker$252Fcontainers$252Feb-current-app$252Fstdouterr.log)
and [AWS logs on production](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Felasticbeanstalk$252Fnowcasting-api-production$252Fvar$252Flog$252Feb-docker$252Fcontainers$252Feb-current-app$252Fstdouterr.log)
and [AWS logs on production](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Felasticbeanstalk$252Fuk-development-nowcasting-api$252Fvar$252Flog$252Feb-docker$252Fcontainers$252Feb-current-app$252Fstdouterr.log)

### 🚀 Site API (Elastic Beanstalk App)
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beantstalk. We use Auth0 to authenticate this API.
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beanstalk. We use Auth0 to authenticate this API.
- Code: [pv-site-api](https://github.com/openclimatefix/pv-site-api)
- [Terraform](https://github.com/openclimatefix/ocf-infrastructure/tree/main/terraform/modules/services/api_site)
- [AWS logs](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Felasticbeanstalk$252Fpvsite-production-api-sites$252Fvar$252Flog$252Feb-docker$252Fcontainers$252Feb-current-app$252Fstdouterr.log)
- [AWS logs](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Felasticbeanstalk$252Fuk-development-sites-api$252Fvar$252Flog$252Feb-docker$252Fcontainers$252Feb-current-app$252Fstdouterr.log)

### 🚀 Cloudcasting API (Elastic Beanstalk App)
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beanstalk. We use Auth0 to authenticate this API.
- Code: [gsoc-cloudcasting-api](https://github.com/openclimatefix/gsoc-cloudcasting-api)
- [Terraform](https://github.com/openclimatefix/ocf-infrastructure/tree/main/terraform/modules/services/cloudcasting-api)
- [AWS logs](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Felasticbeanstalk$252Fuk-development-cloudcasting-api$252Fvar$252Flog$252Feb-docker$252Fcontainers$252Feb-current-app$252Fstdouterr.log)

### Other components:
- Databases: We have a few postgres database that store the PV and forecast data. These is run on RDS.
Expand Down
29 changes: 29 additions & 0 deletions terraform/nowcasting/development/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,35 @@ module "api" {
max_ec2_count = 2
}

#1.2
module "cloudcasting_api" {
source = "../../modules/services/eb_app"
domain = local.domain
aws-region = var.region
aws-environment = local.environment
aws-subnet_id = module.networking.public_subnet_ids[0]
aws-vpc_id = module.networking.vpc_id
container-command = ["docker", "compose", "up", "--build", "--remove-orphans"]
container-env_vars = [
{ "name" : "SENTRY_DSN", "value" : var.sentry_dsn_api },
{ "name" : "AUTH0_DOMAIN", "value" : var.auth_domain },
{ "name" : "AUTH0_API_AUDIENCE", "value" : var.auth_api_audience },
{ "name" : "AUTH0_CLIENT_ID", "value" : var.auth_dashboard_client_id },
{ "name" : "ENVIRONMENT", "value" : local.environment },
Comment on lines +125 to +128
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the app use any of these variables? If not, I'd remove them from here altogether.

{ "name" : "S3_BUCKET_NAME", "value" : var.s3_cloudcasting_bucket_name },
{ "name" : "S3_REGION_NAME", "value" : var.s3_cloudcasting_region_name },
Comment on lines +129 to +130
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[1] These don't need to be variables - since they're accessing the sat bucket, we can use the output from the s3 module that creates the bucket:

{ "name" : "S3_BUCKET_NAME", "value" : module.s3.s3-sat-bucket.id }
{ "name" : "S3_REGION_NAME", "value" : var.region }

{ "name" : "S3_ACCESS_KEY_ID", "value" : var.s3_access_key_id },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as below, you can remove S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY.
There's a small chance we will have to change the code accordindly

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might need to add the

s3_bucket = [{ bucket_read_policy_arn = module.s3.iam-policy-s3-sat-read.arn }]

when replacing these too.

{ "name" : "S3_SECRET_ACCESS_KEY", "value" : var.s3_secret_access_key },
{ "name" : "S3_DOWNLOAD_INTERVAL", "value" : var.s3_download_interval },
{ "name" : "RELOAD", "value" : var.cloudcasting_reload},
Comment on lines +133 to +134
Copy link
Collaborator

@devsjc devsjc Aug 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[2] Do you see these changing often? If not, they might be alright hard coded in the module declaration here, instead of being a variable. Using a var isn't necessarily wrong, but it's just one fewer transient to keep track of. If we do keep them as vars, I'd prefix the s3_download_interval with cloudcasting as well, just so it's clear what it pertains to in the terraform app - it's a bit vague as-is.

]
container-name = "cloudcasting_api"
container-tag = var.api_version
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means it will use the same tag as the (Quartz)API - which I'm assuming isn't what you want? Unless the packaging version of this app is synced to the (Quartz)API?

container-registry = "openclimatefix"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just worth checking, this will only work if the container is deployed to docker hub. I suspect it's only being deployed to GHCR, so this will in fact want to be ghcr.io/openclimatefix. In fact, most of these probably want to be changed to that, but that's another task for another day!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although from what you're saying it sounds like there isn't a build CI, so maybe it is being manually pushed to docker hub - but in that case I wouldn't expect it to be under the openclimatefix org...

eb-app_name = "cloudcasting-api"
eb-instance_type = "t3.micro"
}

# 2.1
resource "aws_secretsmanager_secret" "nwp_consumer_secret" {
name = "${local.environment}/data/nwp-consumer"
Expand Down
33 changes: 33 additions & 0 deletions terraform/nowcasting/development/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,39 @@ variable "sentry_dsn" {
description = "DNS for Sentry monitoring"
}

variable "s3_cloudcasting_bucket_name" {
description = "The name of the S3 bucket to use for storing data"
type = string
default = "not-set"
}

variable "s3_cloudcasting_region_name" {
description = "The region of the S3 bucket to use for storing data"
type = string
default = "not-set"
}
Comment on lines +56 to +66
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These can be removed as per [1]


variable "cloudcasting_reload" {
type = bool
default = true
}
Comment on lines +68 to +71
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe can be removed as per [2]


variable "s3_access_key_id" {
description = "The access key ID for the S3 bucket"
type = string
default = "not-set"
}
variable "s3_secret_access_key" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you wont need these things, it will use what IAM role and permission from that

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was just trying to get deployed with minimal changes to the code for now, this will be better going forward, but is obv different to how Suvan has had to set it up

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea i see. I would probably still remove them here, and adjust Suvans code a tiny bit. In general the less credientials and secruty things we can do, the better.

Ill leave it up to you on that.

if you do keep this in, please make sure the access key and id are tightly locked down, and only have access to relevant things

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the code uses boto3 it should fallback to the default connection strategy. I think EB might even set these variables in the environment by default so you might not even need to make any code changes at all - not that I've looked at the code.

description = "The secret access key for the S3 bucket"
type = string
default = "not-set"
}
variable "s3_download_interval" {
description = "The interval in seconds at which to download data from the S3 bucket"
type = number
default = 30
}
Comment on lines +83 to +87
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe can be removed as per [2]


variable "pvsite_api_version" {
type = string
description = "This gives the version of the PV Site API"
Expand Down