NC | NSFS | Config Dir Restructure - Add users/ Dir

1. Update IAM API Users, Access Keys and additional changes in accountspace_fs:
 - Move the config creation from the function _copy_data_from_requesting_account_to_account_config to the create_user.
 - Fix the ARN account ID for root accounts that were operated by the roots accounts manager (before we copied the requesting_account._id which was true only for root accounts on IAM users).
 - Fix _check_root_account as it has a redundant line that was not relevant (it was there when we thought of additional case, but we never get to it).
 - Add 2 helper functions: _get_account_owner_id_for_arn, _get_owner_account_argument.
 - Improve performance in the function _check_if_root_account_does_not_have_IAM_users_before_deletion after we have the new structure.
2. Update the ConfigFS module to support the new structure and operate on users configs.
3. Update docs:
 - With the config dire restructure (identities/, accounts_by_name/, users/directories).
 - IAM docs - regarding the naming scope (that we have with the new structure) and about the new structure with users/ directory.
4. Update the IAM API tests:
 - Mainly reading the config file in the new structure.
 - Add account validation to accounts created hardcoded (to avoid schema changes without them updated).
 - Refactor `it` names to multiple lines.
5. In rest_s3 change the 'is_owner` part (the gap mentioned in NC | Bucket Owner Removal #8289), where it checks the name, to make sure the account is not a user with the same name.

Signed-off-by: shirady <57721533+shirady@users.noreply.github.com>

Author: shirady

docs/NooBaaNonContainerized/Configuration.md

@@ -49,7 +49,7 @@ For Developers - Use `--config_root` flag for specifying a custom configuration
 
 ### Configuration files permissions
 Mode
-* Configuration files generated under the `accounts/` or `buckets/` directories will have 600 permissions, granting read and write access exclusively to the owner of each configuration file.
+* Configuration files generated under the `identities/` or `buckets/` directories will have 600 permissions, granting read and write access exclusively to the owner of each configuration file.
 
 Ownership
 * Configuration file created by the NooBaa CLI tool will be owned by the user who ran the NooBaa CLI command.
@@ -62,8 +62,9 @@ The default config directory structure contains the following files/directories
 > sudo ls /etc/noobaa.conf.d/
 system.json                 // Required
 access_keys/                // Required
-accounts/                   // Required
+accounts_by_name/           // Required
 buckets/                    // Required
+identities/                 // Required
 config.json                 // Optional
 master_keys.json            // Optional
 certificates/               // Optional
@@ -81,8 +82,9 @@ config_dir_redirect                             // Required
 > sudo ls /path/to/custom/config/dir/
 system.json                                     // Required
 access_keys/                                    // Required
-accounts/                                       // Required
+accounts_by_name/                               // Required
 buckets/                                        // Required
+identities/                                     // Required
 config.json                                     // Optional
 master_keys.json                                // Optional
 certificates/                                   // Optional
@@ -113,29 +115,33 @@ certificates/                                   // Optional
         }
     }
     ```
-`accounts/` - 
+
+`accounts_by_name/`
 * <u>Type</u>: Directory.
 * <u>Required</u>: Yes.
-* <u>Description</u>: A directory that contains configuration files for individual accounts, each account configuration file is named {account_name}.json and adheres to the [account schema](../../src/server/system_services/schemas/nsfs_account_schema.js).
+* <u>Description</u>: A directory that contains symlinks to accounts configurations, each symlink named 
+{account_name}.symlink, linking to the account config within `identities/<account-id>` directory,
+configuration file is named identity.json and adheres to the [account schema](../../src/server/system_services/schemas/nsfs_account_schema.js). The account name symlink points to a relative path of the account rather than an absolute path, for example: `../identities/1111/identity.json`.
 * <u>Example</u>:
     ```sh
-    > ls /etc/noobaa.conf.d/accounts/
-    alice.json
-    bob.json
-    charlie.json
+    > ls /etc/noobaa.conf.d/accounts_by_name/
+    alice.symlink -> ../identities/1111/identity.json
+    bob.symlink -> ../identities/2222/identity.json
+    charlie.symlink -> ../identities/333/identity.json
     ```
 
 `access_keys/` 
 * <u>Type</u>: Directory.
 * <u>Required</u>: Yes.
-* <u>Description</u>: A directory that contains symlinks to accounts configurations, each symlink named {access_key}.symlink, linking to an account within `accounts/` directory. The access key symlink points to a relative path of the account rather than an absolute path, for example: `../accounts/alice.json`.
+* <u>Description</u>: A directory that contains symlinks to accounts configurations, each symlink named {access_key}.symlink, linking to an account within `identities/<account-id>/` directory. The access key symlink points to a relative path of the account rather than an absolute path, for example: `../identities/3333/identity.json`.
 * <u>Example</u>:
     ```sh
     > ls -la /etc/noobaa.conf.d/access_keys/
-    0kbUZlNM9k4SCvrw1pftEXAMPLE.symlink -> ../accounts/alice.json
-    1kbUTlNM9k4SCvrw2pfxEXAMPLE.symlink -> ../accounts/bob.json
-    2kbUMlNM9k4SCvrw3pfyEXAMPLE.symlink -> ../accounts/charlie.json
+    0kbUZlNM9k4SCvrw1pftEXAMPLE.symlink -> ../identities/3333/identity.json
+    1kbUTlNM9k4SCvrw2pfxEXAMPLE.symlink -> ../identities/1111/identity.json
+    2kbUMlNM9k4SCvrw3pfyEXAMPLE.symlink -> ../identities/2222/identity.json
     ```
+
 `buckets/`
 * <u>Type</u>: Directory.
 * <u>Required</u>: Yes.
@@ -147,6 +153,23 @@ certificates/                                   // Optional
     bucket2.json
     bucket3.json
     ```
+
+`identities/`
+* <u>Type</u>: Directory.
+* <u>Required</u>: Yes.
+* <u>Description</u>: A directory that contains configuration files for individual identities, each identity configuration file is named {identity}.json. In case the identity is an account it adheres to the [account schema](../../src/server/system_services/schemas/nsfs_account_schema.js).
+* <u>Example</u>:
+    ```sh
+    > tree /etc/noobaa.conf.d/identities/
+    /etc/noobaa.conf.d/identities
+    ├── 1111
+    │   └── identity.json
+    └── 2222
+        └── identity.json
+    └── 3333
+    └── identity.json
+    ```
+
 `config.json`
 * <u>Type</u>: File.
 * <u>Required</u>: No.

docs/NooBaaNonContainerized/GettingStarted.md

@@ -31,7 +31,7 @@ NooBaa Non Containerized solution includes the following components -
 5. [Storage File System](#create-storage-file-system-paths) - A File system for storing objects.
 
 ### NooBaa High Level Diagram
-![NooBaa Non Containerized Components Diagram](https://github.com/noobaa/noobaa-core/assets/35330373/8d6cce71-e9e8-4e6a-ad88-5c65255bacc7)
+![NooBaa Non Containerized Components Diagram](https://github.com/user-attachments/assets/dad9ecc8-7d8f-46cb-a832-4fc7e40a640d)
 
 
 ## Build NooBaa RPM

docs/design/iam.md

@@ -127,6 +127,76 @@ Here attached a diagram with all the accounts that we have in our system:
 - IAM DeleteAccessKey: AccessKeyId, UserName
 - IAM ListAccessKeys: UserName (not supported: Marker, MaxItems)
 
+### Configuration Directory Components With users
+If account creates a user its config file will be created under identities/<user-id>.identity.json and under the account will be created `users/` directory and inside it it will link to the config.
+Example:
+Note: In this example, we didn't use `system.json`, `config.json`, and `certificates/`.
+1. Configuration directory with 1 account (name: alice, ID: 1111):
+
+```sh
+  > tree /etc/noobaa.conf.d/
+├── access_keys
+│   └── Zzto3OwtGflQrqD41h3SEXAMPLE.symlink -> ../identities/1111/identity.json
+├── accounts_by_name
+│   └── alice.symlink -> ../identities/1111/identity.json
+├── buckets
+├── identities
+│   └── 1111
+│       └── identity.json
+└── master_keys.json
+```
+
+2. Configuration directory with 1 account (name: alice, ID: 1111) and 1 user (name: Robert, ID: 9999, without access key) - 
+Notice the `users/` directory with a symlink of the username to its config file
+
+```sh
+├── access_keys
+│   └── Zzto3OwtGflQrqD41h3SEXAMPLE.symlink -> ../identities/1111/identity.json
+├── accounts_by_name
+│   └── alice.symlink -> ../identities/1111/identity.json
+├── buckets
+├── identities
+│   ├── 1111
+│   │   ├── identity.json
+│   │   └── users
+│   │       └── Robert.symlink -> ../../9999/identity.json
+│   └── 9999
+│       └── identity.json
+├── master_keys.json
+└── system.json
+```
+
+#### Naming Scope
+- Account names are unique between the accounts, for example, if we have account with name John, you cannot create a new account with the name John (and also cannot update the name of an existing account to John).
+- Usernames are unique only inside the account, for example: username Robert can be under account-1, and another user with username Robert can be under account-2.
+Note: The username cannot be the same as the account, for example: under account John we cannot create a username John (and also cannot update the name of an existing username to John). The reason for limiting it is that in the IAM API of Access Key (for example ListAccessKeys) it can be done by account on himself or on another user, and it passes the `--user-name` flag.
+
+Example: 2 accounts (alice and bob) both of them have user with username Robert (notice the different ID number).
+```sh
+├── access_keys
+│   ├── Zzto3OwtGflQrqD41h3SEXAMPLE.symlink -> ../identities/66d81ec79eac82ed43cdee73/identity.json
+│   └── Yser45gyHaghebY62wsUEXAMPLE.symlink -> ../identities/66d8351a92b8dd91b550aa71/identity.json
+├── accounts_by_name
+│   ├── alice.symlink -> ../identities/66d81ec79eac82ed43cdee73/identity.json
+│   └── bob.symlink -> ../identities/66d8351a92b8dd91b550aa71/identity.json
+├── buckets
+├── identities
+│   ├── 66d81ec79eac82ed43cdee73
+│   │   ├── identity.json
+│   │   └── users
+│   │       └── Robert.symlink -> ../../66d834df78e973023abd80cb/identity.json
+│   ├── 66d834df78e973023abd80cb
+│   │   └── identity.json
+│   ├── 66d8351a92b8dd91b550aa71
+│   │   ├── identity.json
+│   │   └── users
+│   │       └── Robert.symlink -> ../../66d83529e09267f53e705373/identity.json
+│   └── 66d83529e09267f53e705373
+│       └── identity.json
+├── master_keys.json
+└── system.json
+```
+
 ## Other
 ### Terminology - AWS vs NooBaa
 |   | AWS | NooBaa |

src/endpoint/iam/iam_utils.js

@@ -19,13 +19,23 @@ function format_iam_xml_date(input) {
 }
 
 /**
- * create_arn creates the AWS ARN for user
+ * create_arn_for_root creates the AWS ARN for root account user
+ * see: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns
+ * @param {string} account_id (the root user account id)
+ */
+function create_arn_for_root(account_id) {
+    return `arn:aws:iam::${account_id}:root`;
+
+}
+
+/**
+ * create_arn_for_user creates the AWS ARN for user
  * see: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns
  * @param {string} account_id (the root user account id)
  * @param {string} username
  * @param {string} iam_path
  */
-function create_arn(account_id, username, iam_path) {
+function create_arn_for_user(account_id, username, iam_path) {
     const basic_structure = `arn:aws:iam::${account_id}:user`;
     if (username === undefined) return `${basic_structure}/`;
     if (check_iam_path_was_set(iam_path)) {
@@ -508,7 +518,8 @@ function validate_status(input_status) {
 
 // EXPORTS
 exports.format_iam_xml_date = format_iam_xml_date;
-exports.create_arn = create_arn;
+exports.create_arn_for_user = create_arn_for_user;
+exports.create_arn_for_root = create_arn_for_root;
 exports.get_action_message_title = get_action_message_title;
 exports.check_iam_path_was_set = check_iam_path_was_set;
 exports.parse_max_items = parse_max_items;

src/endpoint/s3/s3_rest.js

@@ -243,9 +243,17 @@ async function authorize_request_policy(req) {
     if (is_system_owner) return;
 
     const is_owner = (function() {
+        // Containerized condition for bucket ownership
+        // 1. by bucket_claim_owner
+        // 2. by email
         if (account.bucket_claim_owner && account.bucket_claim_owner.unwrap() === req.params.bucket) return true;
+        // NC conditions for bucket ownership
+        // 1. by ID (when creating the bucket the owner is always an account) - comparison to ID which is unique
+        // 2. by name - account_identifier can be username which is not unique
+        //    to make sure it is only on accounts (account names are unique) we check there's no account's ownership
         if (owner_account && owner_account.id === account._id) return true;
-        if (account_identifier_name === bucket_owner.unwrap()) return true; // TODO: change it to root accounts after we will have the /users structure
+        // checked last on purpose (NC first checks the ID and then name for backward computability)
+        if (account.owner === undefined && account_identifier_name === bucket_owner.unwrap()) return true; // mutual check
         return false;
     }());
 
@@ -267,7 +275,7 @@ async function authorize_request_policy(req) {
             s3_policy, account_identifier_id, method, arn_path, req);
     }
 
-    if (!account_identifier_id || permission === "IMPLICIT_DENY") {
+    if ((!account_identifier_id || permission === "IMPLICIT_DENY") && account.owner === undefined) {
         permission = await s3_bucket_policy_utils.has_bucket_policy_permission(
             s3_policy, account_identifier_name, method, arn_path, req);
     }