Merge pull request #8552 from nadavMiz/suplemental-groups

NSFS | NC | add option to set account supplemental groups

Author: nadavMiz

docs/NooBaaNonContainerized/AccountsAndBuckets.md

@@ -29,7 +29,11 @@ See all available account properties - [NC Account Schema](../../src/server/syst
 #### Important properties  
   - `encrypted_secret_key` - Account's secrets will be kept encrypted in the account's configuration file.  
 
-  - `uid/gid/user` - An account's access key is mapped to a file system uid/gid (or user). Before performing any file system operation, NooBaa switches to the account's UID/GID, ensuring that accounts access to buckets and objects is enforced by the file system.
+  - `uid/gid/user` - An account's access key is mapped to a file system uid/gid (or user). Before performing any file system operation, NooBaa switches to the account's UID/GID, ensuring that accounts access to buckets and objects is enforced by the file system.  
+  
+  - `supplemental_groups` - In addition to the account main GID, an account can have supplementary group IDs that are used to determine permissions for accessing files. These GIDs are validated against a files group (GID) permissions.
+  Note: depending on the file system there may be 'sticky bit' enabled somewhere on the files path. 'sticky bit' is a user ownership access right flag that prevents other users than the file owner and root from deleting or moving files.
+  In that case some actions will still get access denied regardless of group permissions enabled. sticky bit is denoted by `t` at the end of the permissions list (example: `drwxrwxrwt`). see https://en.wikipedia.org/wiki/Sticky_bit
 
   - `new_buckets_path` - When an account creates a bucket using the S3 protocol, NooBaa will create the underlying file system directory. This directory will be created under new_buckets_path. Note that the account must have read and write access to its `new_buckets_path`.  Must be an absolute path.  
 

docs/NooBaaNonContainerized/NooBaaCLI.md

@@ -84,6 +84,10 @@ noobaa-cli account add --name <account_name> --uid <uid> --gid <gid> [--user]
     - Type: String
     - Description: Specifies the File system user representing the account. (user can be replaced by --uid and --gid option)
 
+- `supplemental_groups`
+    - Type: String
+    - Description: Specifies additional FS groups (GID) a user can be a part of. Allows access to directories/files having one or more of the provided groups. A String of GIDs separated by commas.
+
 - `new_buckets_path` 
     - Type: String
     - Description: Specifies a file system directory to be used for creating underlying directories that represent buckets created by an account using the S3 API.
@@ -152,6 +156,10 @@ noobaa-cli account update --name <account_name> [--new_name][--uid][--gid][--use
     - Type: Number
     - Description: Specifies the File system user representing the account. (user can be replaced by --uid and --gid option)
 
+- `supplemental_groups`
+    - Type: String
+    - Description: Specifies additional FS groups (GID) a user can be a part of. Allows access to directories/files having one or more of the provided groups. A String of GIDs separated by commas.
+
 - `new_buckets_path` 
     - Type: String
     - Description: Specifies a file system directory to be used for creating underlying directories that represent buckets created by an account using the S3 API.

docs/NooBaaNonContainerized/S3Ops.md

@@ -52,7 +52,7 @@ The following lists describe the bucket and object operations available in NooBa
 - Bucket policies are an access policy option available to grant permission to buckets and objects (see [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html) in AWS documentation). You can use bucket policies to add or deny permissions for the objects in a bucket. Bucket policies can allow or deny requests based on the elements in the policy.
 - Bucket policies use JSON-based policy language (for more information see [basic elements in bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html) in AWS documentation)
 - Bucket policy can be added to a bucket using the S3 API or the noobaa-cli.
-- Bucket policy is an additional layer of permissions to the FS permissions (UID and GID), which means that if two accounts do not have the same permissions (UID, GID) just setting bucket policy on the bucket is not enough.
+- Bucket policy is an additional layer of permissions to the FS permissions (UID, GID, supplemental GIDs), which means that if two accounts do not have the same permissions (UID, GID, supplemental GIDs) just setting bucket policy on the bucket is not enough.
 
 #### Bucket Policy in NooBaa CLI
 1. Adding a bucket policy:

src/api/account_api.js

@@ -312,7 +312,10 @@ module.exports = {
                             uid: { type: 'number' },
                             gid: { type: 'number' },
                             new_buckets_path: { type: 'string' },
-                            nsfs_only: { type: 'boolean' }
+                            nsfs_only: { type: 'boolean' },
+                            supplemental_groups: {
+                                $ref: 'common_api#/definitions/supplemental_groups'
+                            },
                         }
                     },
                 },

src/api/common_api.js

@@ -1346,6 +1346,13 @@ module.exports = {
                 }
             }
         },
+        supplemental_groups: {
+            type: 'array',
+            items: {
+                type: 'integer',
+                'minimum': 0
+            }
+        },
         nsfs_account_config: {
             oneOf: [{
                 type: 'object',
@@ -1354,7 +1361,10 @@ module.exports = {
                     uid: { type: 'number' },
                     gid: { type: 'number' },
                     new_buckets_path: { type: 'string' },
-                    nsfs_only: { type: 'boolean' }
+                    nsfs_only: { type: 'boolean' },
+                    supplemental_groups: {
+                        $ref: '#/definitions/supplemental_groups'
+                    },
                 }
             }, {
                 type: 'object',

src/cmd/manage_nsfs.js

@@ -24,7 +24,7 @@ const { print_usage } = require('../manage_nsfs/manage_nsfs_help_utils');
 const { TYPES, ACTIONS, LIST_ACCOUNT_FILTERS, LIST_BUCKET_FILTERS, GLACIER_ACTIONS } = require('../manage_nsfs/manage_nsfs_constants');
 const { throw_cli_error, get_bucket_owner_account_by_name,
     write_stdout_response, get_boolean_or_string_value, has_access_keys, set_debug_level,
-    is_name_update, is_access_key_update } = require('../manage_nsfs/manage_nsfs_cli_utils');
+    is_name_update, is_access_key_update, parse_comma_delimited_string } = require('../manage_nsfs/manage_nsfs_cli_utils');
 const manage_nsfs_validations = require('../manage_nsfs/manage_nsfs_validations');
 const nc_mkm = require('../manage_nsfs/nc_master_key_manager').get_instance();
 const notifications_util = require('../util/notifications_util');
@@ -359,6 +359,11 @@ async function fetch_account_data(action, user_input) {
         data.access_keys[0].secret_key = data.access_keys[0].secret_key === undefined ? undefined :
             new SensitiveString(String(data.access_keys[0].secret_key));
     }
+    //since supplemental_groups is an array, new list will merge with the old one instead of replacing it in fetch_existing_account_data
+    //so we need to replace this value after merging the data
+    data.nsfs_account_config.supplemental_groups = user_input.supplemental_groups === undefined ?
+        data.nsfs_account_config.supplemental_groups : parse_comma_delimited_string(user_input.supplemental_groups);
+
     if (data.new_access_key) data.new_access_key = new SensitiveString(data.new_access_key);
     // fs_backend deletion specified with empty string '' (but it is not part of the schema)
     data.nsfs_account_config.fs_backend = data.nsfs_account_config.fs_backend || undefined;

src/manage_nsfs/manage_nsfs_cli_errors.js

@@ -346,6 +346,11 @@ ManageCLIError.InvalidGlacierOperation = Object.freeze({
     message: 'only "migrate", "restore" and "expiry" subcommands are supported',
     http_code: 400,
 });
+ManageCLIError.InvalidSupplementalGroupsList = Object.freeze({
+    code: 'InvalidSupplementalGroupsList',
+    message: 'supplemental groups must be a list of group ids (group id is zero or a positive integer)',
+    http_code: 400,
+});
 
 
 ////////////////////////

src/manage_nsfs/manage_nsfs_cli_utils.js

@@ -91,7 +91,25 @@ function get_boolean_or_string_value(value) {
 }
 
 /**
- * get_options_from_file will read a JSON file that include key-value of the options 
+ * This function parse a comma delimited string of numbers ('0,212,111') to an array of numbers.
+ * This function assumes string format was validated before calling the function, wrong string format can
+ * lead to unexpected output (usually array of NaN)
+ * 1. if the value is a number return array with this number (3 => [3])
+ * 2. if the value is a string return an array of numbers ('0,212,111' => [0,212,111])
+ * 3. for all other types (including object and undefined) return the value itself
+ */
+function parse_comma_delimited_string(value) {
+    if (typeof value === 'number') {
+        return [value];
+    }
+    if (typeof value === 'string') {
+        return value.split(',').map(val => Number(val));
+    }
+    return value;
+}
+
+/**_
+ * get_options_fromfile will read a JSON file that include key-value of the options
  * (instead of flags) and return its content
  * @param {string} file_path
  */
@@ -156,6 +174,7 @@ function is_access_key_update(data) {
 exports.throw_cli_error = throw_cli_error;
 exports.write_stdout_response = write_stdout_response;
 exports.get_boolean_or_string_value = get_boolean_or_string_value;
+exports.parse_comma_delimited_string = parse_comma_delimited_string;
 exports.get_bucket_owner_account_by_name = get_bucket_owner_account_by_name;
 exports.get_bucket_owner_account_by_id = get_bucket_owner_account_by_id;
 exports.get_options_from_file = get_options_from_file;

src/manage_nsfs/manage_nsfs_constants.js

@@ -44,16 +44,16 @@ const FROM_FILE = 'from_file';
 const ANONYMOUS = 'anonymous';
 
 const VALID_OPTIONS_ACCOUNT = {
-    'add': new Set(['name', 'uid', 'gid', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', FROM_FILE, ...CLI_MUTUAL_OPTIONS]),
-    'update': new Set(['name', 'uid', 'gid', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'new_name', 'regenerate', ...CLI_MUTUAL_OPTIONS]),
+    'add': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', FROM_FILE, ...CLI_MUTUAL_OPTIONS]),
+    'update': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'new_name', 'regenerate', ...CLI_MUTUAL_OPTIONS]),
     'delete': new Set(['name', ...CLI_MUTUAL_OPTIONS]),
     'list': new Set(['wide', 'show_secrets', 'gid', 'uid', 'user', 'name', 'access_key', ...CLI_MUTUAL_OPTIONS]),
     'status': new Set(['name', 'access_key', 'show_secrets', ...CLI_MUTUAL_OPTIONS]),
 };
 
 const VALID_OPTIONS_ANONYMOUS_ACCOUNT = {
-    'add': new Set(['uid', 'gid', 'user', 'anonymous', ...CLI_MUTUAL_OPTIONS]),
-    'update': new Set(['uid', 'gid', 'user', 'anonymous', ...CLI_MUTUAL_OPTIONS]),
+    'add': new Set(['uid', 'gid', 'user', 'supplemental_groups', 'anonymous', ...CLI_MUTUAL_OPTIONS]),
+    'update': new Set(['uid', 'gid', 'user', 'supplemental_groups', 'anonymous', ...CLI_MUTUAL_OPTIONS]),
     'delete': new Set(['anonymous', ...CLI_MUTUAL_OPTIONS]),
     'status': new Set(['anonymous', ...CLI_MUTUAL_OPTIONS]),
 };
@@ -105,6 +105,7 @@ const OPTION_TYPE = {
     owner: 'string',
     uid: 'number',
     gid: 'number',
+    supplemental_groups: 'string',
     new_buckets_path: 'string',
     user: 'string',
     access_key: 'string',

src/manage_nsfs/manage_nsfs_help_utils.js

@@ -8,7 +8,7 @@ Help:
 
     "NSFS" (Namespace FileSystem) is a NooBaa system that runs a local S3 endpoint on top of a filesystem.
     Each subdirectory of the root filesystem represents an S3 bucket.
-    "noobaa-cli" will provide a command line interface (CLI) to create new accounts and map existing directories 
+    "noobaa-cli" will provide a command line interface (CLI) to create new accounts and map existing directories
     to NooBaa as buckets. For more information refer to the NooBaa docs.
 
 `;
@@ -111,10 +111,10 @@ Usage:
     account add [flags]
 
 Flags:
-
     --name <string>                                                         Set the name for the account
     --uid <number>                                                          Set the User Identifier (UID) (UID and GID can be replaced by --user option)
     --gid <number>                                                          Set the Group Identifier (GID) (UID and GID can be replaced by --user option)
+    --supplemental_groups <string>                        (optional)        Set the supplemental group list (List of GIDs) separated by commas (,) example: '212,211,202'
     --new_buckets_path <string>                           (optional)        Set the filesystem's root path where each subdirectory is a bucket
     --user <string>                                       (optional)        Set the OS user name (instead of UID and GID)
     --access_key <string>                                 (optional)        Set the access key for the account (default is generated)
@@ -124,7 +124,6 @@ Flags:
     --force_md5_etag <true | false>                       (optional)        Set the account to force md5 etag calculation. (unset with '') (will override default config.NSFS_NC_STORAGE_BACKEND)
     --iam_operate_on_root_account <true | false>          (optional)        Set the account to create root accounts instead of IAM users in IAM API requests.
     --from_file <string>                                  (optional)        Use details from the JSON file, there is no need to mention all the properties individually in the CLI
-
 `;
 
 const ACCOUNT_FLAGS_UPDATE = `
@@ -142,6 +141,7 @@ Flags:
     --new_name <string>                                   (optional)        Update the account name
     --uid <number>                                        (optional)        Update the User Identifier (UID)
     --gid <number>                                        (optional)        Update the Group Identifier (GID)
+    --supplemental_groups <number[]>                      (optional)        Update the list of supplemental groups (List of GID) seperated by comma(,) example: 211,202,23 - it will override existing list
     --new_buckets_path <string>                           (optional)        Update the filesystem's root path where each subdirectory is a bucket
     --user <string>                                       (optional)        Update the OS user name (instead of uid and gid)
     --regenerate                                          (optional)        Update automatically generated access key and secret key
@@ -151,7 +151,6 @@ Flags:
     --allow_bucket_creation <true | false>                (optional)        Update the account to explicitly allow or block bucket creation
     --force_md5_etag <true | false>                       (optional)        Update the account to force md5 etag calculation (unset with '') (will override default config.NSFS_NC_STORAGE_BACKEND)
     --iam_operate_on_root_account <true | false>          (optional)        Update the account to create root accounts instead of IAM users in IAM API requests.
-
 `;
 
 const ACCOUNT_FLAGS_DELETE = `
@@ -334,7 +333,7 @@ List of actions supported:
     health
     gather-logs
     metrics
-    
+
 `;
 
 const DIAGNOSE_HEALTH_OPTIONS = `
@@ -400,7 +399,7 @@ List of actions supported:
     start
     status
     history
-    
+
 `;
 
 const UPGRADE_START_OPTIONS = `
@@ -413,13 +412,13 @@ Help:
     'upgrade start' should be executed on one node, the config directory changes will be available for all the nodes of the cluster.
 
 Usage:
-        
+
     noobaa-cli upgrade start [flags]
 
 Flags:
 
     --expected_version                             <string>                             The expected target version of the upgrade
-    --expected_hosts                               <string>                             The expected hosts running NooBaa NC, a string of hosts separated by , 
+    --expected_hosts                               <string>                             The expected hosts running NooBaa NC, a string of hosts separated by ,
     --skip_verification                            <boolean>        (optional)          skip verification of the hosts package version
                                                                                         WARNING: can cause corrupted config dir files created by hosts running old code
     --custom_upgrade_scripts_dir                   <string>         (optional)          custom upgrade scripts dir, use for running custom config dir upgrade scripts
@@ -453,7 +452,7 @@ Usage:
 `;
 
 
-/** 
+/**
  * print_usage would print the help according to the arguments that were passed
  * @param {string} type
  * @param {string} action
@@ -489,7 +488,7 @@ function print_usage(type, action) {
     process.exit(0);
 }
 
-/** 
+/**
  * print_help_account would print the help options for account
  * @param {string} action
  */
@@ -516,7 +515,7 @@ function print_help_account(action) {
     process.exit(0);
 }
 
-/** 
+/**
  * print_help_bucket would print the help options for bucket
  * @param {string} action
  */