Deploy userinfofetcher regorules (#580)

* Add group fetcher container

* Spike group fetcher functionality

* Rename group fetcher to user info fetcher

* Generalize enrichment endpoints to have room for arbitrary user info

* Expose user roles

* Make UIF configurable

* Make UIF credentials configurable too

* OPA 0.45 -> 0.51

* Broke out UIF into its own crate

* UIF error handling

* Ingest custom attributes

* Shut down on SIGTERM

* Split keycloak backend out into separate module

* Fix UIF startup when using none backend

* Rename GroupMembershipRequest to UserInfoRequest

* Cache fetched UserInfo

* Switch UIF to also use workspace dependencies

* Configurable UIF cache ttl

* UIF crate metadata

* Move userInfo rule into helm chart

* Turn UIF custom attributes into a multidict

* UIF smoke test

* Remove rules from Helm chart until we have a better way to deploy them

* Lint-b-gone

* Update CRD

* UIF readme

* SNAFU error for UIF config

* Revert Cmd wrapper enum

* docs

* Prototype of new bundle builder

* Include cm ns in bundle path

* Update bundle builder to Axum 0.7

* Disable unused futures 0.1.x compat layer

* Deploy bundle builder v2

* Fix bundle builder log collection

* Avoid cloning bundle for status requests

* Handle and report errors

* Respect watch namespace

* Fix bundle builder Cargo metadata

* Move dependencies into the workspace level

* Loosen dependency bounds for consistency

* Formatting

* Log bundled files

* Log bundle invalidations

* Ship userinfo rego module in bundle builder

* Fix userinfo/v1 syntax errors

* Change userinfo documentation to refer to regolib

* Enable userinfofetcher docs in nav

* Migrate tests to use regolib

* Changelog

* Upgrade to operator-rs main and Kube-rs 0.92

* Switch bundle builder from multilog to tracing-appender

* Log bundle builder output as JSON

* Update operator-rs

* Regenerate Nix definitions

* Update docs/modules/opa/pages/usage-guide/user-info-fetcher.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Changelog

* Bump operator-rs

* Update operator-rs

* Use released build of operator-rs

* Remove dead bundle builder log bytes constant

* Move bundle builder bugfix to unreleased

* Add comment about bundle builder log rotation

* Consistency

* Consistently mention the /admin group

* Update rust/regorule-library/src/userinfo/v1.rego

Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

---------

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

Author: 3 people

CHANGELOG.md

@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Added
+
+- Added regorule library for accessing user-info-fetcher ([#580]).
+
 ### Changed
 
 - Rewrite of the OPA bundle builder ([#578]).
@@ -13,6 +17,7 @@ All notable changes to this project will be documented in this file.
 - Bundle builder should no longer keep serving deleted rules until it is restarted ([#578]).
 
 [#578]: https://github.com/stackabletech/opa-operator/pull/578
+[#580]: https://github.com/stackabletech/opa-operator/pull/580
 
 ## [24.7.0] - 2024-07-24
 

Cargo.lock

@@ -61,91 +61,36 @@ Fetch groups and extra credentials, but not roles.
 
 NOTE: The OAuth2 Client in Keycloak must be given the `view-users` _Service Account Role_ for the realm that the users are in.
 
-// TODO: Document how to use it in OPA regorules, e.g. to authorize based on group membership
-== Example rego rule
-
-[NOTE]
-.User-facing API & API stability
-====
-Since the 24.07 SDP release we provide an example rego rule in our documentation using an HTTP request.
-However, our plan is that the user-facing API of the SPD is *not* the HTTP API of user-info-fetcher, but instead regorules that will automatically be shipped to the OPA server.
-This enables us to make underlying (for example breaking) changes to the HTTP API while keeping the rego rules API stable.
-
-The documentation will be updated to use the deployed rego rules once available.
-====
-
-[NOTE]
-.About unencrypted HTTP
-====
-The User info fetcher serves endpoints over clear-text HTTP.
-
-It is intended to only be accessed from the OPA Server via _localhost_ and to not be exposed outside of the Pod.
-====
-
-[source,rego]
-----
-package test # <1>
-
-# Define a function to lookup by username
-userInfoByUsername(username) := http.send({
-  "method": "POST",
-  "url": "http://127.0.0.1:9476/user",
-  "body": {"username": username}, <2>
-  "headers": {"Content-Type": "application/json"},
-  "raise_error": true
-}).body
-
-# Define a function to lookup by a stable identifier
-userInfoById(id) := http.send({
-  "method": "POST",
-  "url": "http://127.0.0.1:9476/user",
-  "body": {"id": id}, <3>
-  "headers": {"Content-Type": "application/json"},
-  "raise_error": true
-}).body
-
-currentUserInfoByUsername := userInfoByUsername(input.username)
-currentUserInfoById := userInfoById(input.id)
-----
-
-<1> The package name is important in the OPA URL used by the product.
-<2> Lookup by username
-<3> Lookup by id
-
-For more information on the request and response payloads, see <<_user_info_fetcher_api>>
-
 == User info fetcher API
 
-HTTP Post Requests must be sent to the `/user` endpoint with the following header set: `Content-Type: application/json`.
+User information can be retrieved from regorules using the functions `userInfoByUsername(username)` and `userInfoById(id)` in `data.stackable.opa.userinfo.v1`.
 
-You can either lookup the user info by stable identifier:
+An example of the returned structure:
 
 [source,json]
 ----
 {
   "id": "af07f12c-a2db-40a7-93e0-874537bdf3f5",
+  "username": "alice",
+  "groups": [
+    "/admin"
+  ],
+  "customAttributes": {}
 }
 ----
 
-or by the username:
+For example, the following rule will allow access for users in the `/admin` group:
 
-[source,json]
-----
-{
-  "username": "alice",
-}
+[source,rego]
 ----
+package test
 
-If the user is found, the following response structure will be returned:
+import rego.v1
 
-[source,json]
-----
-{
-  "id": "af07f12c-a2db-40a7-93e0-874537bdf3f5",
-  "username": "alice",
-  "groups": [
-    "/superset-admin"
-  ],
-  "customAttributes": {}
+default allow := false
+
+allow if {
+    user := data.stackable.opa.userinfo.v1.userInfoById(input.userId)
+    "/admin" in user.groups
 }
 ----

Cargo.nix

@@ -11,6 +11,8 @@ publish = false
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
+stackable-opa-regorule-library = { path = "../regorule-library" }
+
 axum.workspace = true
 clap.workspace = true
 flate2.workspace = true

docs/modules/opa/pages/usage-guide/user-info-fetcher.adoc

@@ -181,14 +181,19 @@ enum BundleError {
     #[snafu(display("ConfigMap is missing required metadata"))]
     ConfigMapMetadataMissing,
 
-    #[snafu(display("file {file_name:?} in {config_map} is too large ({file_size} bytes)"))]
+    #[snafu(display("file {file_path:?} is too large ({file_size} bytes)"))]
     FileSizeOverflow {
         source: TryFromIntError,
-        config_map: ObjectRef<ConfigMap>,
-        file_name: String,
+        file_path: String,
         file_size: usize,
     },
 
+    #[snafu(display("failed to add static file {file_path:?} to tarball"))]
+    AddStaticRuleToTarball {
+        source: std::io::Error,
+        file_path: String,
+    },
+
     #[snafu(display("failed to add file {file_name:?} from {config_map} to tarball"))]
     AddFileToTarball {
         source: std::io::Error,
@@ -211,20 +216,15 @@ impl BundleError {
 
 async fn build_bundle(store: Store<ConfigMap>) -> Result<Vec<u8>, BundleError> {
     use bundle_error::*;
-    fn file_header(
-        config_map: &ObjectRef<ConfigMap>,
-        file_name: &str,
-        data: &[u8],
-    ) -> Result<tar::Header, BundleError> {
+    fn file_header(file_path: &str, data: &[u8]) -> Result<tar::Header, BundleError> {
         let mut header = tar::Header::new_gnu();
         header.set_mode(0o644);
         let file_size = data.len();
         header.set_size(
             file_size
                 .try_into()
                 .with_context(|_| FileSizeOverflowSnafu {
-                    config_map: config_map.clone(),
-                    file_name,
+                    file_path,
                     file_size,
                 })?,
         );
@@ -237,6 +237,16 @@ async fn build_bundle(store: Store<ConfigMap>) -> Result<Vec<u8>, BundleError> {
     let mut tar = tar::Builder::new(GzEncoder::new(Vec::new(), flate2::Compression::default()));
     let mut resource_versions = BTreeMap::<String, String>::new();
     let mut bundle_file_paths = BTreeSet::<String>::new();
+
+    for (file_path, data) in stackable_opa_regorule_library::REGORULES {
+        let mut header = file_header(file_path, data.as_bytes())?;
+        tar.append_data(&mut header, file_path, data.as_bytes())
+            .context(AddStaticRuleToTarballSnafu {
+                file_path: *file_path,
+            })?;
+        bundle_file_paths.insert(file_path.to_string());
+    }
+
     for cm in store.state() {
         let ObjectMeta {
             name: Some(cm_ns),
@@ -249,8 +259,8 @@ async fn build_bundle(store: Store<ConfigMap>) -> Result<Vec<u8>, BundleError> {
         };
         let cm_ref = ObjectRef::from_obj(&*cm);
         for (file_name, data) in cm.data.iter().flatten() {
-            let mut header = file_header(&cm_ref, file_name, data.as_bytes())?;
             let file_path = format!("configmap/{cm_ns}/{cm_name}/{file_name}");
+            let mut header = file_header(&file_path, data.as_bytes())?;
             tar.append_data(&mut header, &file_path, data.as_bytes())
                 .with_context(|_| AddFileToTarballSnafu {
                     config_map: cm_ref.clone(),

rust/bundle-builder/Cargo.toml

@@ -0,0 +1,13 @@
+[package]
+name = "stackable-opa-regorule-library"
+description = "Contains Stackable's library of common regorules"
+version.workspace = true
+authors.workspace = true
+license.workspace = true
+edition.workspace = true
+repository.workspace = true
+publish = false
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]

rust/bundle-builder/src/main.rs

@@ -0,0 +1,16 @@
+# Stackable library of shared regorules
+
+This contains regorules that are shipped by the Stackable Data Platform (SDP) as libraries to help simplify writing authorization rules.
+
+## What this is not
+
+This library should *not* contain rules that only concern one SDP product. Those are the responsibility of their individual operators.
+
+## Versioning
+
+All regorules exposed by this library should be versioned, according to Kubernetes conventions.
+
+This version covers *breaking changes to the interface*, not the implementation. If a proposed change breaks existing clients,
+add a new version. Otherwise, change the latest version inline.
+
+Ideally, old versions should be implemented on top of newer versions, rather than carry independent implementations.

rust/regorule-library/Cargo.toml

@@ -0,0 +1,4 @@
+pub const REGORULES: &[(&str, &str)] = &[(
+    "stackable/opa/userinfo/v1.rego",
+    include_str!("userinfo/v1.rego"),
+)];

rust/regorule-library/README.md

@@ -0,0 +1,19 @@
+package stackable.opa.userinfo.v1
+
+# Lookup by (human-readable) username
+userInfoByUsername(username) := http.send({
+  "method": "POST",
+  "url": "http://127.0.0.1:9476/user",
+  "body": {"username": username},
+  "headers": {"Content-Type": "application/json"},
+  "raise_error": true
+}).body
+
+# Lookup by stable user identifier
+userInfoById(id) := http.send({
+  "method": "POST",
+  "url": "http://127.0.0.1:9476/user",
+  "body": {"id": id},
+  "headers": {"Content-Type": "application/json"},
+  "raise_error": true
+}).body

rust/regorule-library/src/lib.rs

@@ -5,22 +5,6 @@ commands:
   - script: |
       kubectl apply -n $NAMESPACE -f - <<EOF
       ---
-      apiVersion: v1
-      kind: ConfigMap
-      metadata:
-        name: test
-        labels:
-          opa.stackable.tech/bundle: "true"
-      data:
-        test.rego: |
-          package test
-
-          userInfoByUsername(username) := http.send({"method": "POST", "url": "http://127.0.0.1:9476/user", "body": {"username": username}, "headers": {"Content-Type": "application/json"}, "raise_error": true}).body
-          userInfoById(id) := http.send({"method": "POST", "url": "http://127.0.0.1:9476/user", "body": {"id": id}, "headers": {"Content-Type": "application/json"}, "raise_error": true}).body
-
-          currentUserInfoByUsername := userInfoByUsername(input.username)
-          currentUserInfoById := userInfoById(input.id)
-      ---
       apiVersion: opa.stackable.tech/v1alpha1
       kind: OpaCluster
       metadata:

rust/regorule-library/src/userinfo/v1.rego

@@ -4,4 +4,4 @@ kind: TestAssert
 metadata:
   name: test-regorule
 commands:
-  - script: kubectl exec -n $NAMESPACE test-regorule-0 -- python /tmp/test-regorule.py -u 'http://test-opa-server-default:8081/v1/data/test'
+  - script: kubectl exec -n $NAMESPACE test-regorule-0 -- python /tmp/test-regorule.py -u 'http://test-opa-server-default:8081/v1/data/stackable/opa/userinfo/v1'

tests/templates/kuttl/aas-user-info/10-install-opa.yaml.j2

@@ -5,22 +5,6 @@ commands:
   - script: |
       kubectl apply -n $NAMESPACE -f - <<EOF
       ---
-      apiVersion: v1
-      kind: ConfigMap
-      metadata:
-        name: test
-        labels:
-          opa.stackable.tech/bundle: "true"
-      data:
-        test.rego: |
-          package test
-
-          userInfoByUsername(username) := http.send({"method": "POST", "url": "http://127.0.0.1:9476/user", "body": {"username": username}, "headers": {"Content-Type": "application/json"}, "raise_error": true}).body
-          userInfoById(id) := http.send({"method": "POST", "url": "http://127.0.0.1:9476/user", "body": {"id": id}, "headers": {"Content-Type": "application/json"}, "raise_error": true}).body
-
-          currentUserInfoByUsername := userInfoByUsername(input.username)
-          currentUserInfoById := userInfoById(input.id)
-      ---
       apiVersion: opa.stackable.tech/v1alpha1
       kind: OpaCluster
       metadata:

tests/templates/kuttl/aas-user-info/30-assert.yaml

@@ -4,4 +4,4 @@ kind: TestAssert
 metadata:
   name: test-regorule
 commands:
-  - script: kubectl exec -n $NAMESPACE test-regorule-0 -- python /tmp/test-regorule.py -u 'http://test-opa-server-default:8081/v1/data/test'
+  - script: kubectl exec -n $NAMESPACE test-regorule-0 -- python /tmp/test-regorule.py -u 'http://test-opa-server-default:8081/v1/data/stackable/opa/userinfo/v1'