Skip to content

1.12

Choose a tag to compare

View all tags
@csun-cpointe csun-cpointe released this 13 Mar 19:26
· 200 commits to dev since this release
aissemble-root-1.12.1
5460ef5

Major Additions

DEPRECATION

The following artifacts are deprecated as of 1.12.1 and will be removed in 1.13.0. If you believe you had a good use case for any of these artifacts, please reach out to the aiSSEMBLE team.

  • aissemble-fastapi Docker image
  • aissemble-fastapi-chart Helm chart
  • extensions-encryption-vault-java java data encryption
  • aissemble-extensions-encryption-vault-python python data encryption

Vault Helm V2 Chart

As we are retiring the vault-deploy profile, we are introducing the vault v2 structure helm chart. For details refer to Available Profiles (Kubernetes)

Breaking Changes

Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

  • All Fermenter profiles marked for deletion in 1.12.1 have been removed. Follow the technical documentation for migration instructions. In addition, the following artifacts were removed:
    • The aissemble-data-access-chart Helm chart
    • The foundation-data-access Java module
  • PySpark will no longer throw an exception when a required field is None but instead filter it out. See Changes in Spark/PySpark Schema Behavior below for more details.
  • Spark/PySpark will no longer filter out records with null/None fields that are not required and have validation. See Changes in Spark/PySpark Schema Behavior below for more details.
  • The aiSSEMBLE-vault docker image will no longer be available. We also remove the related vault-deploy profile.
  • The default preparationGoals for the release plugin had been changed to help:help to skip redundant testing during the release preparation phase, providing for a faster and more reliable release process.

Changes in Spark/PySpark Schema Behavior

  • When creating a data frame from a record schema with required fields using PySpark, creation of the data frame (spark_session.createDataFrame()) will no longer throw an exception if a required field is None but instead filter out the record from the data frame as part of validation (record_schema.validate_dataset()).
  • When validating a data frame from a record schema with non-required fields and dictionary validation using Spark/PySpark, validation (recordSchema.validateDataFrame()/record_schema.validate_dataset()) will no longer mistakenly filter out a record from the data frame if the field value is None/null.

Known Issues

Kafka Python Compatibility

The latest releases of the kafka-python library are not compatible with aiSSEMBLE 1.12 out-of-the-box. You would need to upgrade the Kafka image being deployed, or update the kafka-python version range to >=2.0.4, <2.1.0.

Missing Images

  • The 1.12 version of the following images is not available. If you're using any of these images, specify the (functionally identical) 1.11.1 tag. They will be release normally with 1.13.0.
    • aissemble-quarkus
    • aissemble-jenkins-controller
    • aissemble-jenkins-agent

Docker Module Build Failures

When using a Docker daemon that does not reside in /var/run (e.g. running Rancher Desktop without admin privileges) the docker-maven-plugin will fail to build with the message below. To work around this failure, set the DOCKER_HOST variable to the location of the daemon socket file. For example, to make the docker-maven-plugin work with Rancher Desktop, run export DOCKER_HOST=unix://$HOME/.rd/docker.sock.

[ERROR] Failed to execute goal org.technologybrewery.fabric8:docker-maven-plugin:0.45-tb-0.1.0:build (default-build) on project final-513-spark-worker-docker:
   Execution default-build of goal org.technologybrewery.fabric8:docker-maven-plugin:0.45-tb-0.1.0:build failed: 
   No <dockerHost> given, no DOCKER_HOST environment variable, no read/writable '/var/run/docker.sock' or '//./pipe/docker_engine' and no external provider like Docker machine configured

How to Upgrade

The following steps will upgrade your project to 1.12.1. These instructions consist of multiple phases:

  • Automatic Upgrades - no manual action required
  • Precondition Steps - needed in all situations
  • Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
  • Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name Description
upgrade-tiltfile-aissemble-version-migration Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration Updates the Helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.12.1 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.12.1</version>
</parent>

Conditional Steps

For projects that have customized the Hive service

The hive service base image has been switched from docker.io/eclipse-temurin:17-jre to registry.access.redhat.com/ubi9/openjdk-17-runtime:1.21. Projects should update their Hive service Dockerfile or any related startup configuration script to use microdnf instead apt-get as the package manager if applicable

For projects that uses aissemble-fastapi

The fastapi base image has been switched from docker.io/python:3.11 to registry.access.redhat.com/ubi9/python-311:9.5. There may be adjustments needed if applicable.

Final Steps - Required for All Projects

Finalizing the Upgrade

  1. Run ./mvnw clean install and resolve any manual actions that are suggested
    • NOTE: This will run automatic migrations and update any aiSSEMBLE dependencies in 'pyproject.toml' files
  2. Repeat the previous step until all manual actions are resolved

What's Changed

to be auto-generated when published

What's Changed

  • aiSSEMBLE Release 1.11 - Bump dev to next version by @ewilkins-csi in #576
  • #568 Spark/PySpark schema validation should not fail on non required fields by @carter-cundiff in #580
  • 572-relation-schema-validation by @csun-cpointe in #581
  • #577 the policies location property can be undefined for encryption policies by @carter-cundiff in #582
  • [#578] only generate messaging files if messaging is needed by @ewilkins-csi in #584
  • [#585] fix Nvidia ARM build by @ewilkins-csi in #586
  • #141 enable default s3-local deploy template value by @csun-cpointe in #587
  • #588 - Addressing dependabot alerts for the Antora Docs module. by @habibimoiz in #590
  • #589 update the aissemble-hive-service base image to use RHAT UBI by @csun-cpointe in #592
  • #591 improve archetype test by @ewilkins-csi in #593
  • [#591] fix tty error in CI by @ewilkins-csi in #594
  • #596 update metamodel collection type documentation by @csun-cpointe in #597
  • [#579] remove deprecated Fermenter profiles by @ewilkins-csi in #599
  • #598 update Fastapi base image to use RHAT UBI image for better adapt CVE issue by @csun-cpointe in #600
  • [#602] improve Hadoop download for aissemble-hive-service by @ewilkins-csi in #603
  • [#601] Delete untagged SNAPSHOT images by @chang-annie in #605
  • Stabilize CI build by @csun-cpointe in #606
  • #604 remove aissemble-vault docker image and disable the related unit tests using the image by @csun-cpointe in #607
  • #610 update the release:prepare phase to skip execution completely by @csun-cpointe in #611

Full Changelog: aissemble-root-1.11.1...aissemble-root-1.12.1

Contributors

epwilkins, carter-cundiff, and 3 other contributors
Assets 2
Loading