2.0

Major Additions

aiSSEMBLE is now licensed under the Apache License

The aiSSEMBLE baseline license has been updated from the Booz Allen Public License to the Apache License 2.0.

Helmfile Integration

Given that the local and higher environment deployment methods not aligned can cause hard-to-diagnose bugs and slow development, in an effort to have one tool to deploy to all environments, going forward aiSSEMBLE will support Helmfile instead of Tilt and ArgoCD. New project will be generated with Helmfile and existing projects are encouraged to use it but are not required to. Follow Finalizing the Upgrade section for migration instructions.

Removal of Integration Tests

New projects generated on version 2.0.0 of aiSSEMBLE will no longer include an automatically generated <project-name>-tests module. If you upgrade an existing project to 2.0.0, your <project-name>-tests module will remain intact — but be aware that there will not be any further updates or enhancements to these integration tests going forward.

In addition, the following Fermenter profiles related to the integration tests module have been deprecated and will be removed in version 2.1.0 of aiSSEMBLE:

• integration-test-docker
• integration-test-chart
• integration-test-data-pipeline

Spark BOM and Reduced Spark Pipeline Size

To help ensure data delivery pipelines are compatible with the aissemble-spark image, and to exclude jars provided by the image from shaded pipeline jars, we have created the aissemble-spark-bom. This BOM ensures Spark, Hadoop and Hive dependencies are not included in the shaded pipeline jar significantly reducing the spark worker Docker image size and ensures pipelines are unit tested against the same versions of libraries as are on the aissemble-spark image, including patches for CVE resolution.

Spark upgraded from 3.5.4 to 3.5.5. For upgrade information, see the Spark release notes.

Poetry 2 Support and Minimum Python

New projects generated on version 2.0.0 of aiSSEMBLE will use Poetry 2.x. With this update, the minimum version of Python supported by aiSSEMBLE is updated from 3.8 to 3.9.

Habushu 3.0 upgrade

Habushu has been upgraded to version 3.0. Because Habushu 3 has dropped support for other containerization approaches, aiSSEMBLE will now use the containerize-dependencies feature to package PySpark and Machine Learning pipelines. The Habushu project has examples to help illustrate how containerization works.

Dependency Upgrades

Language	Dependency	from	to	Reason/Note
Platform	Spark	3.5.4	3.5.5	For upgrade information, see the Spark release notes.
Java	cucumber-reporting (net.masterthought)	5.8.1	5.9.0	Resolve vulnerabilities
Java	elasticsearch-spark-30_2.12 (org.elasticsearch)	8.9.0	9.1.2	Resolve vulnerabilities
Java	geotools-wrapper (org.datasyslab)	1.4.0-28.2	1.7.1-28.5	Updates to match new Sedona version
Java	jclouds-* (org.apache.jclouds)	2.6.0	2.7.0	Resolve vulnerabilities
Java	jjwt-* (io.jsonwebtoken)	0.11.2	0.12.7	Resolve vulnerabilities. Breaking changes - see 0.12 release notes
Java	kafka-clients (org.apache.kafka)	3.7.0	3.9.1	Resolve vulnerabilities
Java	keycloak-core (org.keycloak)	24.0.4	26.3.2	Resolve vulnerabilities
Java	kubernetes-client (io.fabric8)	6.13.4	7.1.0	Resolve vulnerabilities
Java	mysql-connector-java (mysql)	8.0.30	9.2.0	Resolve vulnerabilities. Renamed to mysql-connector-j.
Java	netty-handler (io.netty)	4.1.111.Final	4.1.24.Final	Resolve vulnerabilities
Java	openlineage-java (io.openlineage)	1.23.0	1.30.0	Resolve vulnerabilities. To prepare for future breaking changes, we recommend using the builder API, or the aiSSEMBLE classes only.
Java	plexus-archiver (org.codehaus.plexus)	4.8.0	4.10.0	Resolve vulnerabilities
Java	poi-ooxml (org.apache.poi)	5.2.3	5.4.1	Resolve vulnerabilities
Java	postgresql (org.postgresql)	42.5.1	42.5.5	Resolve vulnerabilities
Java	quarkus-* (io.quarkus)	3.8.6	3.15.6	Resolve vulnerabilities
Java	resteasy-* (org.jboss.resteasy)	6.2.8.Final	6.2.12.Final	Resolve vulnerabilities
Java	sedona-* (org.apache.sedona)	1.4.0	1.7.1	Upgraded due to compatibility issues with Spark 3.5. Because of artifact renames, we recommend replacing all sedona-* dependencies with sedona-spark-shaded-3.5_2.12. See the Sedona docs
Java ...

1.12

Major Additions

DEPRECATION

The following artifacts are deprecated as of 1.12.1 and will be removed in 1.13.0. If you believe you had a good use case for any of these artifacts, please reach out to the aiSSEMBLE team.

• aissemble-fastapi Docker image
• aissemble-fastapi-chart Helm chart
• extensions-encryption-vault-java java data encryption
• aissemble-extensions-encryption-vault-python python data encryption

Vault Helm V2 Chart

As we are retiring the vault-deploy profile, we are introducing the vault v2 structure helm chart. For details refer to Available Profiles (Kubernetes)

Breaking Changes

Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

• All Fermenter profiles marked for deletion in 1.12.1 have been removed. Follow the technical documentation for migration instructions. In addition, the following artifacts were removed:
  • The aissemble-data-access-chart Helm chart
  • The foundation-data-access Java module
• PySpark will no longer throw an exception when a required field is None but instead filter it out. See Changes in Spark/PySpark Schema Behavior below for more details.
• Spark/PySpark will no longer filter out records with null/None fields that are not required and have validation. See Changes in Spark/PySpark Schema Behavior below for more details.
• The aiSSEMBLE-vault docker image will no longer be available. We also remove the related vault-deploy profile.
• The default preparationGoals for the release plugin had been changed to help:help to skip redundant testing during the release preparation phase, providing for a faster and more reliable release process.

Changes in Spark/PySpark Schema Behavior

• When creating a data frame from a record schema with required fields using PySpark, creation of the data frame (spark_session.createDataFrame()) will no longer throw an exception if a required field is None but instead filter out the record from the data frame as part of validation (record_schema.validate_dataset()).
• When validating a data frame from a record schema with non-required fields and dictionary validation using Spark/PySpark, validation (recordSchema.validateDataFrame()/record_schema.validate_dataset()) will no longer mistakenly filter out a record from the data frame if the field value is None/null.

Known Issues

Kafka Python Compatibility

The latest releases of the kafka-python library are not compatible with aiSSEMBLE 1.12 out-of-the-box. You would need to upgrade the Kafka image being deployed, or update the kafka-python version range to >=2.0.4, <2.1.0.

Missing Images

• The 1.12 version of the following images is not available. If you're using any of these images, specify the (functionally identical) 1.11.1 tag. They will be release normally with 1.13.0.
  • aissemble-quarkus
  • aissemble-jenkins-controller
  • aissemble-jenkins-agent

Docker Module Build Failures

When using a Docker daemon that does not reside in /var/run (e.g. running Rancher Desktop without admin privileges) the docker-maven-plugin will fail to build with the message below. To work around this failure, set the DOCKER_HOST variable to the location of the daemon socket file. For example, to make the docker-maven-plugin work with Rancher Desktop, run export DOCKER_HOST=unix://$HOME/.rd/docker.sock.

[ERROR] Failed to execute goal org.technologybrewery.fabric8:docker-maven-plugin:0.45-tb-0.1.0:build (default-build) on project final-513-spark-worker-docker:
   Execution default-build of goal org.technologybrewery.fabric8:docker-maven-plugin:0.45-tb-0.1.0:build failed: 
   No <dockerHost> given, no DOCKER_HOST environment variable, no read/writable '/var/run/docker.sock' or '//./pipe/docker_engine' and no external provider like Docker machine configured

How to Upgrade

The following steps will upgrade your project to 1.12.1. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the Helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.12.1 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.12.1</version>
</parent>

Conditional Steps

For projects that have customized the Hive service

The hive service base image has been switched from docker.io/eclipse-temurin:17-jre to registry.access.redhat.com/ubi9/openjdk-17-runtime:1.21. Projects should update their Hive service Dockerfile or any related startup configuration script to use microdnf instead apt-get as the package manager if applicable

For projects that uses aissemble-fastapi

The fastapi base image has been switched from docker.io/python:3.11 to registry.access.redhat.com/ubi9/python-311:9.5. There may be adjustments needed if applicable.

Final Steps - Required for All Projects

Finalizing the Upgrade

1. Run ./mvnw clean install and resolve any manual actions that are suggested
   • NOTE: This will run automatic migrations and update any aiSSEMBLE dependencies in 'pyproject.toml' files
2. Repeat the previous step until all manual actions are resolved

What's Changed

to be auto-generated when published

What's Changed

• aiSSEMBLE Release 1.11 - Bump dev to next version by @ewilkins-csi in #576
• #568 Spark/PySpark schema validation should not fail on non required fields by @carter-cundiff in #580
• 572-relation-schema-validation by @csun-cpointe in #581
• #577 the policies location property can be undefined for encryption policies by @carter-cundiff in #582
• [#578] only generate messaging files if messaging is needed by @ewilkins-csi in #584
• [#585] fix Nvidia ARM build by @ewilkins-csi in #586
• #141 enable default s3-local deploy template value by @csun-cpointe in #587
• #588 - Addressing dependabot alerts for the Antora Docs module. by @habibimoiz in #590
• #589 update the aissemble-hive-service base image to use RHAT UBI by @csun-cpointe in #592
• #591 improve archetype test by @ewilkins-csi in #593
• [#591] fix tty error in CI by @ewilkins-csi in #594
• #596 update metamodel collection type d...

1.11

Major Additions

Service account support for spark-infrastructure

To have a more flexible and secure way to authenticate with AWS services, the spark-infrastructure helm chart has been enhanced to support AWS IRSA (IAM Roles for Service Accounts) authentication. See the How to Upgrade for more information.

Path to Production Alignment

To better align development processes with processes in CI/CD and higher environments, we no longer recommend using Tilt live-reloading. As such, upgrading projects should consider narrowing the scope of their Tiltfile. These changes will also help smooth the transition as further alignment is brought to the path to production. See How to Upgrade for more information.

Data Access Upgrade

Data access through GraphQL has been deprecated and replaced with Trino. Trino is optimized for performing queries against large datasets by leveraging a distributed architecture that processes queries in parallel, enabling fast and scalable data retrieval.

Spark & Hadoop Upgrade

Spark and PySpark have been upgraded from version 3.5.2 to 3.5.4. Hadoop has been upgraded from version 3.3.4 to 3.4.1.

Record Relation

To enable nested data records, we have added a new relation feature to the record metamodel. This allows records to reference other records. For more details, refer to the Record Relation Options in the aiSSEMBLE User Guide.
Several features are still a work in progress:

• PySpark and Spark schema based validation for relations will only validate the record and not its relations. Object based validation for relations is available.

Helm Charts Resource Specification

The following Helm charts have been updated to include the configuration options for specifying container resource requests/limits:

• aissemble-spark-history-chart
• aissemble-quarkus-chart

See the official Kubernetes documentation for more details.

Breaking Changes

Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

• The following Java classes have been renamed:

  Old Java Class	New Java Class
  AIOpsModelInstanceRepostory	AissembleModelInstanceRepository
  AiopsMdaJsonUtils	AissembleMdaJsonUtils

• To improve the development cycle and docker build consistency, we have deprecated the docker_build() and local_resources() functions in the Tilt and enable maven docker build for the docker modules. Follow the instruction in the Finalizing the Upgrade to avoid duplicated docker image build.
• In an attempt to harden the aissemble-hive-service image, several changes were made that may impact projects with Hive customization
• Amazon AWS SDK has been upgraded from v1 to v2, which includes a group and artifact ID change

Known Issues

Errors on Initial Build

During the first build of a new project, or the first build after an upgrade to 1.11, Baton error messages may be output saying that POM files could not be processed for each module. This is a harmless side-effect of the data-access-default-migration and will go away on its own when the migration turns itself off by updating the deactiveMigrations configuration of Baton.

Docker Module Build Failures

When using a Docker daemon that does not reside in /var/run (e.g. running Rancher Desktop without admin privileges) the docker-maven-plugin will fail to build with the message below. To work around this failure, set the DOCKER_HOST variable to the location of the daemon socket file. For example, to make the docker-maven-plugin work with Rancher Desktop, run export DOCKER_HOST=unix://$HOME/.rd/docker.sock.

[ERROR] Failed to execute goal org.technologybrewery.fabric8:docker-maven-plugin:0.45-tb-0.1.0:build (default-build) on project final-513-spark-worker-docker:
   Execution default-build of goal org.technologybrewery.fabric8:docker-maven-plugin:0.45-tb-0.1.0:build failed: 
   No <dockerHost> given, no DOCKER_HOST environment variable, no read/writable '/var/run/docker.sock' or '//./pipe/docker_engine' and no external provider like Docker machine configured

Known Vulnerabilities

Date identified	Vulnerability	Severity	Package	Affected versions	CVE	Fixed in

How to Upgrade

The following steps will upgrade your project to 1.11. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the Helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
pipeline-invocation-service-template-migrtion	Include the helm.valueFiles param to ArgoCD pipeline-invocation-service template
docker-module-pom-dependency-type-migration	Updates the maven pipeline dependency type within your project's sub docker module pom file(<YOUR_PROJECT>-docker/*-docker/pom.xml) to fix the build cache checksum calculation issue
enable-maven-docker-build-migration	Remove the maven fabric8 plugin skip configuration within your project's docker module pom file(<YOUR_PROJECT>-docker/pom.xml) to enable the maven docker build
ml-pipeline-docker-pom-migration	Adds pipeline ML pipeline dependencies to relevant docker POMs to improve the Maven build cache functionality
training-api-image-tag-migration	Update training docker image tags to use project version
inference-docker-image-tag-migration	Update inference docker image tags to use project version
spark-worker-docker-image-tag-migration	Updates Spark docker image tags to use project version
spark-infrastructure-server-side-label-migration	Enables the Server-Side Diff Strategy within ArgoCD for the Spark Infrastructure resource so that changes made by the Universal Config Store mutating webhook are ignored
data-access-default-migration	Migrates Record metamodels that were relying on the default Data Access settings to preserve semantics with the updated default value. This migration also disables itself to ensure it's only run once.

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigrati...

1.10

Major Additions

Java 17 Upgrade

The aiSSEMBLE project is now built with Java 17. All dependencies inherited from aiSSEMBLE have been updated to a Java 17 compatible version and automatic migrations (detailed below) have been created to streamline this transition for downstream projects. It is recommended to review non-inherited dependencies and custom logic within your project, as this upgrade can break existing functionality due to deprecated classes and incompatible dependencies.

Python Version Supported Expanded

All aiSSEMBLE python libraries now support a minimum version of 3.8 (previously 3.11).

Improved Licence Generation

Leveraging a new version of Booz Allen Licenses. Booz Allen projects may find some minor updates to header file text that better conforms to some strict linting standards.

Breaking Changes

Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

• All projects must be built with Java 17. The aiSSEMBLE team recommends SDKMan for managing multiple Java versions.
• The new minimum required Maven version is now 3.9.6 to ensure compatibility with Java 17. The aiSSEMBLE team recommens using the Maven Wrapper to ensure compatibility.
• Deprecated features were removed:
  • All SageMaker modules and corresponding references were removed in commit 8ce393f.
  • The Service Discovery module and corresponding references were removed in commit d10db5d.
  • The data.lineage.namespace legacy property was removed. For more guidance, please refer to the Lineage Metadata documentation.

Known Issues

• There is an issue with package dependencies within a SparkApplication yaml (sparkApp.spec.deps.packages) not being correctly saved to the ivy cache directory resulting in a java.io.FileNotFoundException when running starting your application. To resolve this, move the dependencies from packages to jars within the SparkApplication yaml (sparkApp.spec.deps.packages ➡️ sparkApp.spec.deps.jars).

Known Vulnerabilities

Date identified	Vulnerability	Severity	Package	Affected versions	CVE	Fixed in

How to Upgrade

Warning

The upgrade process is a little different for this release to support the Java 17 updates. Specifically, the finalization
section has a few more steps.

The following steps will upgrade your project to 1.10. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the Helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
spark-version-upgrade-migration	Updates the Spark Application executor failure parameters to their new key name to ensure compatibility with Apache Spark 3.5
spark-pipeline-messaging-pom-migration	Updates a Spark pipeline module pom.xml with the new CDI classes dependency to ensure messaging compatibility with Java 17
spark-pipeline-messaging-cdi-factory-migration	Updates a Spark pipeline module CdiContainerFactory.java with the new CDI classes to ensure messaging compatibility with Java 17
spark-pipeline-servlet-api-migration	Updates a Spark pipeline module pom.xml with the javax.servlet-api dependency to ensure compatibility with Apache Spark 3.5 which has not migrated to Jakarta packages yet
it-infrastructure-java-upgrade-migration	Updates the Java docker image version in the integration test docker module to JDK 17
log4j-maven-shade-plugin-migration	Updates the Maven Shade Plugin with the new Log4j dependency information
quarkus-bom-migration	Updates all references to the quarkus-bom and quarkus-universe-bom to use the new aissemble-quarkus-bom for managing Quarkus dependencies
pom-dependency-version-migration	Updates the pom dependencies previously managed by the aiSSEMBLE bom-component to include their necessary versions
java-package-migration	Updates the affected java classes from their old package name to their new package name to ensure compatibility with the updated Java 17 dependencies. This migration primarily updates javax.* packages to their new jakarta.* packages
alerting-cdi-migration	Adds AlertingCdiContext to CdiContainerFactory.getContexts for data delivery pipelines that depend on foundation-alerting

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.10.0 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.10.0</version>
</parent>

BOM Component Replacement

All references to the com.boozallen.aissemble:bom-component should be replaced with the new com.boozallen.aissemble:aissemble-quarkus-bom:

    <dependency>
        <groupId>com.boozallen.aissemble</groupId>
-        <artifactId>bom-component</artifactId>
+        <artifactId>aissemble-quarkus-bom</artifactId>
        <version>${version.aissemble}</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>

Conditional Steps

For projects created before version 1.7.0

Older projects were not generated with a Maven Wrapper configuration by default. To ease M...

1.9.4

Bug Fixes

• Fix Spark Operator ArgoCD template error

For Known Issues and How to Upgrade refer to the release notes for 1.9

What's Changed

Full Changelog: aissemble-root-1.9.3...aissemble-root-1.9.4

1.9.3

Bug Fixes

• Enable SparkOperator ArgoCD Application to use the ServerSideApply option by default for new projects
• Fix Thrift server connection instability and naming inconsistencies with Spark Infrastructure charts
• Prevent manual ArgoCD migration instructions from being added to files in new projects

For Known Issues and How to Upgrade refer to the release notes for 1.9

What's Changed

Full Changelog: aissemble-root-1.9.2...aissemble-root-1.9.3

1.9.2

Bug Fixes

• Remove incorrect manual action for adding Configuration Store to Tiltfile

For Known Issues and How to Upgrade refer to the release notes for 1.9

What's Changed

Full Changelog: aissemble-root-1.9.1...aissemble-root-1.9.2

1.9

Major Additions

Universal Configuration Store

The Configuration Store is a tool that enables the various configurations for a project to be centrally defined and managed. It then provides a standardized way of accessing them, allowing the environment specific configurations to be dynamically provided to the their respective resources within the project at runtime. See the official documentation for more details on leveraging the configuration store.

aiSSEMBLE Infrastructure Helm Chart

The aiSSEMBLE Infrastructure Helm Chart contains the necessary infrastructure for deploying your project within a single umbrella chart. This chart includes support for Argo CD, Jenkins, and Nginx Ingress. See the chart README for more details.

Spark Infrastructure v2 Helm Chart

The following Helm charts have been migrated to the v2 structure and combined into a single spark-infrastructure chart. To migrate your Helm charts to use the v2 pattern, follow the instructions in the technical documentation.

• Spark Infrastructure
• Hive Metastore Service
• Hive Metastore Database

With this new chart, the aissemble-hive-mysql image is no longer being used. As a result, the image is deprecated and will not be updated or maintained moving forward. If you choose to remain on an older version of the Spark Infrastructure charts, you can continue to use the 1.8 version of the aissemble-hive-mysql image. However, we recommend upgrading to the new v2 spark-infrastructure chart to take full advantage of future fixes and improvements.

Helm Chart Updates

MLFlow Helm Chart parent version upgraded from 0.2.1 to 1.4.22. This includes an update to use the community docker image bitnami/mlflow:2.15.1-debian-12-r0 instead of the deprecated boozallen/aissemble-mlflow:1.7.0 image. This new image updates the MLFlow version from 2.3.1 to 2.15.1.

Airflow Helm Chart parent version upgraded from 1.10.0 to 1.15.0. This includes an update to use the community docker image apache/airflow:2.9.3 instead of the deprecated boozallen/aissemble-airflow:1.7.0 image. This new image updates the Airflow version from 2.6.2 to 2.9.3.

Kafka Helm Chart updated to use the community docker image bitnami/kafka:3.5.1-debian-11-r1 instead of the deprecated boozallen/aissemble-kafka:1.7.0. This new image remains on the same Kafka version 3.5.1.

ArgoCD Deployment Branch

Resolved issue when deploying with ArgoCD where apps would fail to utilize the current deploy job branch parameter. Now ArgoCD deployments will use the correct deploy branch when performing test deployments on branches other than the default dev.

Breaking Changes

• New default values were added to the aissemble-spark-application-chart sparkConf key that may interfere with unit tests

Known Issues

There are no known issues with the 1.9 release.

Known Vulnerabilities

Date identified	Vulnerability	Severity	Package	Affected versions	CVE	Fixed in

Recommended Kubernetes Version

aiSSEMBLE recommends any consumer be on a minimum Kubernetes version of 1.30 due to security findings in 1.29. For more information on Kubernetes current security findings, view their CVE feed. If using AWS EKS, please follow AWS documentation on upgrading your clusters and node groups.

How to Upgrade

The following steps will upgrade your project to 1.9. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the Helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
ml-flow-dockerfile-migration	Updates the MLFlow's Dockerfile to use the bitnami/mlflow image as a base instead of the deprecated boozallen/aissemble-mlflow image
airflow-dockerfile-migration	Updates the Airflow's Dockerfile to use the bitnami/airflow image as a base instead of the deprecated boozallen/aissemble-airflow image
update-data-access-thrift-endpoint-migration	For projects using the default data-access thrift endpoint, updates to the new endpoint associated with v2 spark-infrastructure
argocd-value-file-sync-policy-configuration-migration	Updates the ArgoCD values files (<YOUR_PROJECT>-deploy/src/main/resources/) to include the syncPolicy values to enable the Configuration Store to deploy first on your cluster.
argocd-template-sync-policy-configuration-migration	Updates the ArgoCD template files (<YOUR_PROJECT>-deploy/src/main/resources/templates/) to include the syncPolicy helm function to enable the Configuration Store to deploy first on your cluster.

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.9.4 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.9.4</version>
</parent>

Conditional Steps

For Projects with Data Delivery Pipelines

It is strongly recommended that projects migrate from the old Spark infrastructure charts to the new Spark Infrastructure umbrella chart. The previous charts are now deprecated and will not receive any updates in future releases. To migrate to the new chart follow the upgrade instructions outlined on the Path to Production > Containers page in t...

1.8.2

Bug Fixes

• Fixed bug affecting the Pipeline Invocation service due to permissions issues
• Fixed the PDP migration of properties files to apply more broadly to catch more cases

For Known Issues and How to Upgrade refer to the release notes for 1.8.0

What's Changed

• 61352ba Broaden PDP rename migration to catch more cases
• 44d816c cherry-pick ce08e16

Full Changelog: aissemble-root-1.8.1...aissemble-root-1.8.2

1.8.1

Bug Fixes

• Fixed the group name for the new Docker Maven plugin
• Fixed the CI configuration for Docker modules
• Fixed the <project>-tests-docker module's base image
• Fixed Spark Operator functionality for projects created before 1.2 and provided instructions migrating to the new Spark Operator

For Known Issues and How to Upgrade refer to the release notes for 1.8.0

What's Changed

• #284 Create migration instructions for transitioning from v1 Spark Operator helm chart to v2 by @carter-cundiff in #299
• #284 Update hive metastore service to use 1.7.0 image only due to 1.8 incompatibilities by @carter-cundiff in #303
• #306 Resolve migration issues from Orphedomos to Fabric8 by @carter-cundiff in #315
• [#320] fix docker plugin config to account for change from orphedomos by @ewilkins-csi in #326

Full Changelog: aissemble-root-1.8.0...aissemble-root-1.8.1