Skip to content

Update external_airtable_transit_tech_stacks_services.yml #3842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 18, 2025

Conversation

fsalemi
Copy link
Contributor

@fsalemi fsalemi commented Apr 16, 2025

Description

The Services table in the Transit Technology Stacks base (Airtable) is a synced table from the Services table in the California Transit base. During the syncing process, all lookup fields are transformed into plain text fields. This leads to multiple type mismatch errors when creating the external table via the related DAG.
For example:

Error while reading table: cal-itp-data-infra.external_airtable.transit_technology_stacks__services, error message: JSON parsing error in row starting at position 51112: Could not convert value 'string_value: "279, 4"' to integer. Field: itp_id__from_provider_; Value: 279, 4 File: gs://calitp-airtable/transit_technology_stacks__services/dt=2022-08-16/ts=2022-08-16T14:33:57.365242Z/services.jsonl.gz

To address this, I'm explicitly adding the schema_fields and including only the required fields to prevent these parsing errors.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Copy link

github-actions bot commented Apr 16, 2025

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 1 to add, 0 to change, 1 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/create_external_tables/airtable/external_airtable_transit_tech_stacks_services.yml"] must be replaced
-/+ resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      content_type        = "text/plain; charset=utf-8" -> (known after apply)
!~      crc32c              = "1+6Yzg==" -> (known after apply)
!~      detect_md5hash      = "sGzoWDTMx3D+FeBre3jn9A==" -> "different hash" # forces replacement
-       event_based_hold    = false -> null
!~      id                  = "***************************************************************************************************************" -> (known after apply)
+       kms_key_name        = (known after apply)
!~      md5hash             = "sGzoWDTMx3D+FeBre3jn9A==" -> (known after apply)
!~      media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/dags%2Fcreate_external_tables%2Fairtable%2Fexternal_airtable_transit_tech_stacks_services.yml?generation=1744830291306882&alt=media" -> (known after apply)
-       metadata            = {} -> null
        name                = "dags/create_external_tables/airtable/external_airtable_transit_tech_stacks_services.yml"
!~      output_name         = "dags/create_external_tables/airtable/external_airtable_transit_tech_stacks_services.yml" -> (known after apply)
!~      self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/dags%2Fcreate_external_tables%2Fairtable%2Fexternal_airtable_transit_tech_stacks_services.yml" -> (known after apply)
!~      storage_class       = "STANDARD" -> (known after apply)
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

📝 Plan generated in Plan Airflow DAGs #4

@fsalemi
Copy link
Contributor Author

fsalemi commented Apr 16, 2025

@vevetron it is ready now.

@fsalemi fsalemi merged commit 4cb4f5a into main Apr 18, 2025
2 checks passed
@fsalemi fsalemi deleted the fsalemi-patch-2 branch April 18, 2025 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants