Merge pull request #8312 from shirady/nsfs-nc-config-dir-restructure-users-dir

NC | NSFS | Config Dir Restructure - Add `users/` Dir

Author: shirady

docs/NooBaaNonContainerized/Configuration.md

@@ -49,7 +49,7 @@ For Developers - Use `--config_root` flag for specifying a custom configuration
 
 ### Configuration files permissions
 Mode
-* Configuration files generated under the `accounts/` or `buckets/` directories will have 600 permissions, granting read and write access exclusively to the owner of each configuration file.
+* Configuration files generated under the `identities/` or `buckets/` directories will have 600 permissions, granting read and write access exclusively to the owner of each configuration file.
 
 Ownership
 * Configuration file created by the NooBaa CLI tool will be owned by the user who ran the NooBaa CLI command.
@@ -62,8 +62,9 @@ The default config directory structure contains the following files/directories
 > sudo ls /etc/noobaa.conf.d/
 system.json                 // Required
 access_keys/                // Required
-accounts/                   // Required
+accounts_by_name/           // Required
 buckets/                    // Required
+identities/                 // Required
 config.json                 // Optional
 master_keys.json            // Optional
 certificates/               // Optional
@@ -81,8 +82,9 @@ config_dir_redirect                             // Required
 > sudo ls /path/to/custom/config/dir/
 system.json                                     // Required
 access_keys/                                    // Required
-accounts/                                       // Required
+accounts_by_name/                               // Required
 buckets/                                        // Required
+identities/                                     // Required
 config.json                                     // Optional
 master_keys.json                                // Optional
 certificates/                                   // Optional
@@ -113,29 +115,33 @@ certificates/                                   // Optional
         }
     }
     ```
-`accounts/` - 
+
+`accounts_by_name/`
 * <u>Type</u>: Directory.
 * <u>Required</u>: Yes.
-* <u>Description</u>: A directory that contains configuration files for individual accounts, each account configuration file is named {account_name}.json and adheres to the [account schema](../../src/server/system_services/schemas/nsfs_account_schema.js).
+* <u>Description</u>: A directory that contains symlinks to accounts configurations, each symlink named 
+{account_name}.symlink, linking to the account config within `identities/<account-id>` directory,
+configuration file is named identity.json and adheres to the [account schema](../../src/server/system_services/schemas/nsfs_account_schema.js). The account name symlink points to a relative path of the account rather than an absolute path, for example: `../identities/1111/identity.json`.
 * <u>Example</u>:
     ```sh
-    > ls /etc/noobaa.conf.d/accounts/
-    alice.json
-    bob.json
-    charlie.json
+    > ls /etc/noobaa.conf.d/accounts_by_name/
+    alice.symlink -> ../identities/1111/identity.json
+    bob.symlink -> ../identities/2222/identity.json
+    charlie.symlink -> ../identities/333/identity.json
     ```
 
 `access_keys/` 
 * <u>Type</u>: Directory.
 * <u>Required</u>: Yes.
-* <u>Description</u>: A directory that contains symlinks to accounts configurations, each symlink named {access_key}.symlink, linking to an account within `accounts/` directory. The access key symlink points to a relative path of the account rather than an absolute path, for example: `../accounts/alice.json`.
+* <u>Description</u>: A directory that contains symlinks to accounts configurations, each symlink named {access_key}.symlink, linking to an account within `identities/<account-id>/` directory. The access key symlink points to a relative path of the account rather than an absolute path, for example: `../identities/3333/identity.json`.
 * <u>Example</u>:
     ```sh
     > ls -la /etc/noobaa.conf.d/access_keys/
-    0kbUZlNM9k4SCvrw1pftEXAMPLE.symlink -> ../accounts/alice.json
-    1kbUTlNM9k4SCvrw2pfxEXAMPLE.symlink -> ../accounts/bob.json
-    2kbUMlNM9k4SCvrw3pfyEXAMPLE.symlink -> ../accounts/charlie.json
+    0kbUZlNM9k4SCvrw1pftEXAMPLE.symlink -> ../identities/3333/identity.json
+    1kbUTlNM9k4SCvrw2pfxEXAMPLE.symlink -> ../identities/1111/identity.json
+    2kbUMlNM9k4SCvrw3pfyEXAMPLE.symlink -> ../identities/2222/identity.json
     ```
+
 `buckets/`
 * <u>Type</u>: Directory.
 * <u>Required</u>: Yes.
@@ -147,6 +153,23 @@ certificates/                                   // Optional
     bucket2.json
     bucket3.json
     ```
+
+`identities/`
+* <u>Type</u>: Directory.
+* <u>Required</u>: Yes.
+* <u>Description</u>: A directory that contains configuration files for individual identities, each identity configuration file is named {identity}.json. In case the identity is an account it adheres to the [account schema](../../src/server/system_services/schemas/nsfs_account_schema.js).
+* <u>Example</u>:
+    ```sh
+    > tree /etc/noobaa.conf.d/identities/
+    /etc/noobaa.conf.d/identities
+    ├── 1111
+    │   └── identity.json
+    └── 2222
+        └── identity.json
+    └── 3333
+    └── identity.json
+    ```
+
 `config.json`
 * <u>Type</u>: File.
 * <u>Required</u>: No.

docs/NooBaaNonContainerized/GettingStarted.md

@@ -31,7 +31,7 @@ NooBaa Non Containerized solution includes the following components -
 5. [Storage File System](#create-storage-file-system-paths) - A File system for storing objects.
 
 ### NooBaa High Level Diagram
-![NooBaa Non Containerized Components Diagram](https://github.com/noobaa/noobaa-core/assets/35330373/8d6cce71-e9e8-4e6a-ad88-5c65255bacc7)
+![NooBaa Non Containerized Components Diagram](https://github.com/user-attachments/assets/dad9ecc8-7d8f-46cb-a832-4fc7e40a640d)
 
 
 ## Build NooBaa RPM

docs/design/iam.md

@@ -127,6 +127,76 @@ Here attached a diagram with all the accounts that we have in our system:
 - IAM DeleteAccessKey: AccessKeyId, UserName
 - IAM ListAccessKeys: UserName (not supported: Marker, MaxItems)
 
+### Configuration Directory Components With users
+If account creates a user its config file will be created under identities/<user-id>.identity.json and under the account will be created `users/` directory and inside it it will link to the config.
+Example:
+Note: In this example, we didn't use `system.json`, `config.json`, and `certificates/`.
+1. Configuration directory with 1 account (name: alice, ID: 1111):
+
+```sh
+  > tree /etc/noobaa.conf.d/
+├── access_keys
+│   └── Zzto3OwtGflQrqD41h3SEXAMPLE.symlink -> ../identities/1111/identity.json
+├── accounts_by_name
+│   └── alice.symlink -> ../identities/1111/identity.json
+├── buckets
+├── identities
+│   └── 1111
+│       └── identity.json
+└── master_keys.json
+```
+
+2. Configuration directory with 1 account (name: alice, ID: 1111) and 1 user (name: Robert, ID: 9999, without access key) - 
+Notice the `users/` directory with a symlink of the username to its config file
+
+```sh
+├── access_keys
+│   └── Zzto3OwtGflQrqD41h3SEXAMPLE.symlink -> ../identities/1111/identity.json
+├── accounts_by_name
+│   └── alice.symlink -> ../identities/1111/identity.json
+├── buckets
+├── identities
+│   ├── 1111
+│   │   ├── identity.json
+│   │   └── users
+│   │       └── Robert.symlink -> ../../9999/identity.json
+│   └── 9999
+│       └── identity.json
+├── master_keys.json
+└── system.json
+```
+
+#### Naming Scope
+- Account names are unique between the accounts, for example, if we have account with name John, you cannot create a new account with the name John (and also cannot update the name of an existing account to John).
+- Usernames are unique only inside the account, for example: username Robert can be under account-1, and another user with username Robert can be under account-2.
+Note: The username cannot be the same as the account, for example: under account John we cannot create a username John (and also cannot update the name of an existing username to John). The reason for limiting it is that in the IAM API of Access Key (for example ListAccessKeys) it can be done by account on himself or on another user, and it passes the `--user-name` flag.
+
+Example: 2 accounts (alice and bob) both of them have user with username Robert (notice the different ID number).
+```sh
+├── access_keys
+│   ├── Zzto3OwtGflQrqD41h3SEXAMPLE.symlink -> ../identities/66d81ec79eac82ed43cdee73/identity.json
+│   └── Yser45gyHaghebY62wsUEXAMPLE.symlink -> ../identities/66d8351a92b8dd91b550aa71/identity.json
+├── accounts_by_name
+│   ├── alice.symlink -> ../identities/66d81ec79eac82ed43cdee73/identity.json
+│   └── bob.symlink -> ../identities/66d8351a92b8dd91b550aa71/identity.json
+├── buckets
+├── identities
+│   ├── 66d81ec79eac82ed43cdee73
+│   │   ├── identity.json
+│   │   └── users
+│   │       └── Robert.symlink -> ../../66d834df78e973023abd80cb/identity.json
+│   ├── 66d834df78e973023abd80cb
+│   │   └── identity.json
+│   ├── 66d8351a92b8dd91b550aa71
+│   │   ├── identity.json
+│   │   └── users
+│   │       └── Robert.symlink -> ../../66d83529e09267f53e705373/identity.json
+│   └── 66d83529e09267f53e705373
+│       └── identity.json
+├── master_keys.json
+└── system.json
+```
+
 ## Other
 ### Terminology - AWS vs NooBaa
 |   | AWS | NooBaa |

src/endpoint/iam/iam_utils.js

@@ -19,13 +19,23 @@ function format_iam_xml_date(input) {
 }
 
 /**
- * create_arn creates the AWS ARN for user
+ * create_arn_for_root creates the AWS ARN for root account user
+ * see: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns
+ * @param {string} account_id (the root user account id)
+ */
+function create_arn_for_root(account_id) {
+    return `arn:aws:iam::${account_id}:root`;
+
+}
+
+/**
+ * create_arn_for_user creates the AWS ARN for user
  * see: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns
  * @param {string} account_id (the root user account id)
  * @param {string} username
  * @param {string} iam_path
  */
-function create_arn(account_id, username, iam_path) {
+function create_arn_for_user(account_id, username, iam_path) {
     const basic_structure = `arn:aws:iam::${account_id}:user`;
     if (username === undefined) return `${basic_structure}/`;
     if (check_iam_path_was_set(iam_path)) {
@@ -508,7 +518,8 @@ function validate_status(input_status) {
 
 // EXPORTS
 exports.format_iam_xml_date = format_iam_xml_date;
-exports.create_arn = create_arn;
+exports.create_arn_for_user = create_arn_for_user;
+exports.create_arn_for_root = create_arn_for_root;
 exports.get_action_message_title = get_action_message_title;
 exports.check_iam_path_was_set = check_iam_path_was_set;
 exports.parse_max_items = parse_max_items;

src/endpoint/s3/s3_rest.js

@@ -243,9 +243,17 @@ async function authorize_request_policy(req) {
     if (is_system_owner) return;
 
     const is_owner = (function() {
+        // Containerized condition for bucket ownership
+        // 1. by bucket_claim_owner
+        // 2. by email
         if (account.bucket_claim_owner && account.bucket_claim_owner.unwrap() === req.params.bucket) return true;
+        // NC conditions for bucket ownership
+        // 1. by ID (when creating the bucket the owner is always an account) - comparison to ID which is unique
+        // 2. by name - account_identifier can be username which is not unique
+        //    to make sure it is only on accounts (account names are unique) we check there's no account's ownership
         if (owner_account && owner_account.id === account._id) return true;
-        if (account_identifier_name === bucket_owner.unwrap()) return true; // TODO: change it to root accounts after we will have the /users structure
+        // checked last on purpose (NC first checks the ID and then name for backward computability)
+        if (account.owner === undefined && account_identifier_name === bucket_owner.unwrap()) return true; // mutual check
         return false;
     }());
 
@@ -267,7 +275,7 @@ async function authorize_request_policy(req) {
             s3_policy, account_identifier_id, method, arn_path, req);
     }
 
-    if (!account_identifier_id || permission === "IMPLICIT_DENY") {
+    if ((!account_identifier_id || permission === "IMPLICIT_DENY") && account.owner === undefined) {
         permission = await s3_bucket_policy_utils.has_bucket_policy_permission(
             s3_policy, account_identifier_name, method, arn_path, req);
     }