Add basic docs

Author: TimCsaky

docs/Architecture-Hosted.md

@@ -0,0 +1,45 @@
+This page outlines the architecture and deployment features of the BC Gov Hosted COMS service. It is mainly intended for a technical audience, and for people who want to have a better understanding of how we have the service deployed.
+
+**Note:** For more details of the COMS application itself and how it works, see the [Architecture](Architecture) overview.
+
+## Table of Contents
+
+- [Infrastructure](#infrastructure)
+- [High Availability](#high-availability)
+- [Network Connectivity](#network-connectivity)
+- [Database connection Pooling](#database-connection-pooling)
+- [Horizontal Autoscaling](#horizontal-autoscaling)
+
+## Infrastructure
+
+The BC Govt. Hosted COMS service runs on the OpenShift container ecosystem. The following diagram provides a general logical overview of main component relations. Main network traffic flows are shown in fat arrows, while secondary network traffic relations are shown with a simple black line.
+
+![Hosted COMS Architecture](images/coms_architecture.png)
+
+**Figure 1 - The general infrastructure and network topology of the BC Govt. hosted COMS**
+
+### High Availability
+
+The COMS API and Database are all designed to be highly available within an OpenShift environment. The Database achieves high availability by leveraging [Patroni](https://patroni.readthedocs.io/en/latest/). COMS is designed to be a scalable and atomic microservice. On the OCP4 platform, there can be between 2 to 16 running replicas of the COMS microservice depending on service load. This allows the service to reliably handle a large variety of request volumes and scale resources appropriately.
+
+### Network Connectivity
+
+In general, all network traffic enters through the BC Govt. API Gateway. A specifically tailored Network Policy rule exists to allow only network traffic we expect to receive from the API Gateway. When a client connects to the COMS API, they will be going through OpenShift's router and load balancer before landing on the API gateway. That connection then gets forwarded to one of the COMS API pod replicas. Figure 1 represents the general network traffic direction with the outlined fat arrows. The direction of those arrows represents which component is initializing the TCP/IP connection.
+
+COMS uses a database network pool to maintain persistent database connections. Pooling allows the service to avoid the overhead of repeated TCP/IP 3-way handshakes to start a connection. By reusing existing connections in a network pool, we can pipeline and improve network efficiency. We pool connections from COMS to Patroni within our architecture. The OpenShift load balancer follows general default Kubernetes scheduling behavior.
+
+### Database connection Pooling
+
+We introduced network pooling for Patroni connections to mitigate network traffic overhead. As our volume of traffic increased, it became expensive to create and destroy network connections for each transaction. While low volumes of traffic are capable of operating without any notable delay to the user, we started encountering issues when scaling up and improving total transaction flow within COMS.
+
+By reusing connections whenever possible, we were able to avoid the TCP/IP 3-way handshake done on every new connection. Instead we could leverage existing connections to pipeline traffic and improve general efficiency. We observed up to an almost 3x performance increase in total transaction volume flow by switching to pooling.
+
+### Horizontal Autoscaling
+
+In order to make sure our application can horizontally scale (run many copies of itself), we had to ensure that all processes in the application are self-contained and atomic. Since we do not have any guarantees of which pod instance would be handling what task at any specific moment, the only thing we can do is to ensure that every unit of work is clearly defined and atomic so that we can prevent situations where there is deadlock, or double executions.
+
+While implementing Horizontal Autoscaling is relatively simple by using a [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) construct in OpenShift, we can only take advantage of it if the application is able to handle the different types of lifecycles. Based on usage metrics such as CPU and memory load, the HPA can increase or decrease the number of replicas on the platform in order to meet the demand.
+
+We found that in our testing, we were able to reliably scale up to around 17 pods before we began to crash out our Patroni database. While we haven't been able to reliably isolate the cause of this, we suspect that the underlying Postgres database can only handle up to 100 concurrent connections (and is thus ignoring Patroni's max connection limit of 500) or that the database containers are simply running out of memory before being able to handle more connections. As such, this is why we decided to cap our HPA to a maximum of 16 pods at this time.
+
+Our current limiting factor for scaling higher is the ability for our database to support more connections for some reason or another. If we get into the situation where we need to scale past 16 pods, we will need to consider more managed solutions for pooling db connections such as [PgBouncer](https://www.pgbouncer.org/).

docs/Architecture.md

@@ -0,0 +1,57 @@
+This page outlines the general architecture and design principles of COMS. It is mainly intended for a technical audience, and for people who want to have a better understanding of how the system works.
+
+## Table of Contents
+
+- [Infrastructure](#infrastructure)
+- [Database Structure](#database-structure)
+- [Code Design](#code-design)
+
+## Infrastructure
+
+![COMS Architecture](images/coms_self_architecture.png)
+
+**Figure 1 - The general infrastructure and network topology of COMS**
+
+## Database Structure
+
+The PostgreSQL database is written and handled via managed, code-first migrations. We generally store tables containing users, objects, buckets, permissions, and how they relate to each other. As COMS is a back-end microservice, lines of business can leverage COMS without being tied to a specific framework or language. The following figures depict the database schema structure as of April 2023 for the v0.4.0 release.
+
+![COMS Public ERD](images/coms_erd_public.png)
+
+**Figure 3 - The public schema for a COMS database**
+
+Database design focuses on simplicity and succinctness. It effectively tracks the user, the object, the bucket, the permissions, and how they relate to each other. We enforce foreign key integrity by invoking onUpdate and onDelete cascades in Postgres. This ensures that we do not have dangling references when entries are removed from the system. Metadata and tags are represented as many-to-many relationships to maximize reverse search speed.
+
+![COMS Audit ERD](images/coms_erd_audit.png)
+
+**Figure 4 - The audit schema for a COMS database**
+
+We use a generic audit schema table to track any update and delete operations done on the database. This table is only modified by database via table triggers, and is not normally accessible by the COMS application itself. This should meet most general security, tracking and auditing requirements.
+
+## Code Design
+
+COMS is a relatively small and compact microservice with a very focused approach to handling and managing objects. However, not all design choices are self-evident just from inspecting the codebase. The following section will cover some of the main reasons why the code was designed the way it is.
+
+### Organization
+
+The code structure in COMS follows a simple, layered structure following best practice recommendations from Express, Node, and ES6 coding styles. The application has the following discrete layers:
+
+| Layer      | Purpose                                                                                      |
+| ---------- | -------------------------------------------------------------------------------------------- |
+| Controller | Contains controller express logic for determining what services to invoke and in what order  |
+| DB         | Contains the direct database table model definitions and typical modification queries        |
+| Middleware | Contains middleware functions for handling authentication, authorization and feature toggles |
+| Routes     | Contains defined Express routes for defining the COMS API shape and invokes controllers      |
+| Services   | Contains logic for interacting with either S3 or the Database for specific tasks             |
+| Validators | Contains logic which examines and enforces incoming request shapes and patterns              |
+
+Each layer is designed to focus on one specific aspect of business logic. Calls between layers are designed to be deliberate, scoped, and contained. This hopefully makes it easier to tell at a glance what each piece of code is doing and what it depends on. For example, the validation layer sits between the routes and controllers. It ensures that incoming network calls are properly formatted before proceeding with execution.
+
+#### Middleware
+
+COMS middleware focuses on ensuring that the appropriate business logic filters are applied as early as possible. Concerns such as feature toggles, authentication and authorization are handled here. Express executes middleware in the order of introduction. It will sequentially execute and then invoke the next callback as a part of its call stack. Because of this, we must ensure that the order we introduce and execute our middleware adhere to the following pattern:
+
+1. Run the `require*` middleware functions first (these generally invole the middleware found in `featureToggle.js`)
+2. Validation and structural cheks
+3. Permission and authorization checks
+4. Any remaining middleware hooks before invoking the controller

docs/Authentication.md

@@ -0,0 +1,42 @@
+This page describes how to authenticate requests to the COMS API. The [Authentication Modes](Configuration#authentication-modes) must be enabled in the COMS configuration.
+
+**Note:** The BC Gov Hosted COMS service only allows OIDC Authentication using JWT's issued by the [Pathfinder SSO `standard` keycloak realm](https://github.com/bcgov/sso-keycloak/wiki#standard-service)).
+
+## OIDC Authentication
+
+With [OIDC mode](Configuration#oidc-keycloak) enabled, requests to the COMS API can be authenticated using a **User ID token** (JWT) issued by an OIDC authentication realm. The JWT should be added in an Authorization header (type `Bearer` token).
+
+COMS will only accept JWT's issued by one OIDC realm (specified in the COMS config). JWT's are typically issued to an application and saved to a user's browser when he/she signs-in to a website through the [Authorization Code Flow](https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth). Both the website (client app) and the instance of COMS must be [configured to use the same OIDC authentication realm](https://github.com/bcgov/common-object-management-service/blob/master/app/README.md#keycloak-variables) in order for the JWT to be valid.
+
+When COMS receives the request, it will validate the JWT (by calling the OIDC realm's token endpoint). The JWT is a reliable way of verifying the the user's identity on which the COMS permission model is based.
+
+The authentication when downloading an object also uses S3 pre-signed URLs:
+
+### Authentication flow for readObject
+
+Reference: [API Specification](https://coms.api.gov.bc.ca/api/v1/docs#tag/Object/operation/readObject) for more details.
+
+A common use case for COMS is to download a specific object from object storage.
+Depending on the `download` mode specified in the request, the COMS `readObject` endpoint will return one of the following:
+
+1. The file directly from S3, by first doing a HTTP 302 redirect to a temporary pre-signed S3 object URL
+2. The file streamed/proxied through COMS
+3. The temporary pre-signed S3 object URL itself
+
+COMS uses the redirect flow by default because it avoids unnecessary network hops. For significantly large object transactions, redirection also has the added benefit of maximizing COMS microservice availability. Since the large transaction does not pass through COMS, it is able to remain capable of handling other client requests.
+
+![COMS Network Flow](images/coms_network_flow.png)
+
+**Figure 2 - The general network flow for a typical COMS object request**
+
+## Basic Auth
+
+If [Basic Auth Mode](Configuration#basic-auth) is enabled in your COMS instance, requests to the COMS API can be authenticated using an HTTP Authorization header (type `Basic`) containing the username and password configured in COMS.
+
+This mode offers more direct access for a 'service account' authorized in the scope of the application rather than for a specific user and by-passes the COMS object/bucket permission model.
+
+Basic Auth mode is not available on the BC Gov hosted COMS service.
+
+## Unauthenticated Mode
+
+[Unauthenticated Mode](Configuration#unauthenticated-auth) configuration is generally recommended when you expect to run COMS in a highly secured network environment and do not have concerns about access control to objects as you have another application handling that already.

docs/Buckets.md

@@ -0,0 +1,18 @@
+
+### Configuring Buckets
+
+- COMS is [configured with a 'default' bucket](Configuration#object-storage). Various object management endpoints will use this bucket if no `bucketId` parameter is provided. (**Note:** the default bucket fall-back behaviour is not available in the BC Gov Hosted COMS service.)
+
+- Additional buckets can be added to the COMS system using the [createBucket](https://coms.api.gov.bc.ca/api/v1/docs#tag/Bucket/operation/createBucket) endpoint.
+
+- When a bucket is created, if the createBucket API request is authenticated with a User ID token (JWT), that user will be granted all [5 permissions](Permissions#permission-codes). Bucket Permissions can be granted to other users ([bucketAddPermissions](https://coms.api.gov.bc.ca/api/v1/docs#tag/Permission/operation/bucketAddPermissions)), if the request is authenticated with a JWT for a user with `MANAGE` permission.
+
+If you are self-hosting COMS you can also manage permissions for any object or bucket by using these endpoints with [basic authentication](Authentication#basic-auth).
+
+### Using the Bucket **Key**
+
+When you create a bucket in COMS, technically you are 'mounting' your  S3 bucket (actual bucket provisioned) at a specified path in the `key` property of the [createBucket](https://coms-dev.api.gov.bc.ca/api/v1/docs#tag/Bucket/operation/createBucket) request body.
+
+COMS will only operate with objects at that 'folder' within the actual bucket. A COMS `bucket` can more accurately be thought of as a 'mount' to a single path within a bucket.
+
+To work with objects in 'sub-folders' (with other prefixes), you can create multiple COMS 'buckets' mounted at different paths by specifying different keys.

docs/Endpoint-Notes.md

@@ -0,0 +1,68 @@
+This page outlines the general usage patterns and organization of the COMS API. This article is intended for a technical audience, and for people who are planning on using the API endpoints.
+
+**The COMS API is documented using the [Open API Specification](https://coms.api.gov.bc.ca/api/v1/docs)**
+
+## Table of Contents
+
+- [Bucket](#bucket)
+- [Object](#object)
+  - [Metadata](#metadata)
+  - [Tag](#tag)
+  - [Versions](#versions)
+- [Permission](#permission)
+- [Sync](#sync)
+- [User](#user)
+
+## Bucket
+
+Bucket operations offer the usual CRUD operations for bucket resource management. For example:
+
+- `CREATE /bucket` and `PATCH /bucket/{bucketId}` will pre-emptively check to see if the proposed credential changes represent a network-accessible bucket. These endpoints will yield an error if it is unable to validate the bucket.
+
+## Object
+
+Object endpoints directly influence and manipulate S3 objects and information inherent to them. These endpoints serve as the main core of COMS, focusing on CRUD operations for the objects themselves.
+
+- Uploading (`POST /object`) or updating an object ( `POST /object/{objectId}`) accepts a file in a multipart/form-data body. You can include metadata (via headers) and tags (using query params) in this request.
+- `GET /object/{objectId}` is the main endpoint for users to directly access and download a single object.
+- `HEAD /object/{objectId}` should be used for situations where you need to get information about the object, but do not want the binary stream of the object itself.
+- `DELETE /object/{objectId}` deletes either the object or a specific version of the object. COMS follows the S3 standard for [deleting versioned objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html)
+  - If versioning is enabled, calling `/object/{objectId}` is a soft-delete, adding a 'delete-marker' version. To restore this object, remove the delete-marker with `/object/{objectId}?versionId={VersionId of delete-marker}`. To hard-delete a versioned object, you must delete the last version `/object/{objectId}?versionId={last version}`.
+  - Calling in the Delete endpoint on a bucket without versioning is a hard-delete.
+- The `GET /object` search and `PATCH /object/{objectId}/public` public toggle require a backing database in order to function.
+
+### Metadata
+
+Metadata operation endpoints directly focus on the manipulation of metadata of S3 Objects. Each endpoint will create a copy of the object with the modified metadata attached.
+
+More details found here: [Metadata and Tags](Metadata-Tag)
+
+### Tag
+
+Tag operation endpoints directly focus on the manipulation of tags of S3 Objects. Unlike Metadata, Tags can be modified without the need to create new versions of the object.
+
+More details found here: [Metadata and Tags](Metadata-Tag)
+
+### Versions
+
+Version specific operations focus on listing and discovering versioning information known by COMS. While the majority of version-specific operations are available as query parameters in the Objects endpoints, the `GET /object/{objectId}/version` endpoint focuses on letting users discover and list what versions are available to work with.
+
+## Permission
+
+Permission operation endpoints directly focus on associating users to objects with specific permissions. All of these endpoints require a database to function. Existing permissions can be searched for using `GET /permission/object` and `GET /permission/bucket`, and standard create, read and delete operations for permissions exist to allow users to modify access control for specific objects they have management permissions over.
+
+More details found here: [Permissions](Permissions)
+
+## Sync
+
+*Available in COMS v0.7+*
+
+Sync endpoints allow synchronizing COMS' internal state with that of the actual S3 bucket/object. This can be useful for setting up a S3 bucket with preexisting files for use with COMS without having to re-upload everything through the COMS API, or for synchronizing changes made through an external S3 client (e.g. S3 Browser, Cyberduck etc) to an object already managed by COMS.
+
+API calls to the sync endpoints do not immediately add all detected changes to COMS' internal database; instead, they are added to a queue where they are eventually processed. The endpoint `GET /sync/status` returns the number of items that are currently sitting in this queue.
+
+At the time of writing, synchronization is not done automatically, so the sync endpoints must be used in order for COMS to know of any changes to the bucket/object.
+
+## User
+
+User operation endpoints focus on exposing known tracked users and identity providers. These endpoints serve as a reference point for finding the right user and identity to manipulate in the Permission endpoints. As COMS is relatively agnostic to how a user logs in (it only cares that you exist), the onus of determining which identity provider a user uses falls onto the line of business to handle, should that be something that needs monitoring.