Merge pull request #8254 from magento/MC38472-remote-storage

MC38472 remote storage

Author: hguthrie

src/_data/toc/configuration-guide.yml

@@ -286,6 +286,20 @@ pages:
         - label: Tutorial—Set up multiple websites with Apache
           url: /config-guide/multi-site/ms_apache.html
 
+    - label: Remote storage
+      url: /config-guide/remote-storage/config-remote-storage.html
+      exclude_versions: ["2.3"]
+      children:
+
+        - label: Configure AWS S3 adapter
+          url: /config-guide/remote-storage/config-remote-storage-aws-s3.html
+
+        - label: Configure caching
+          url: /config-guide/remote-storage/config-remote-storage-caching.html
+
+        - label: Configure image resizing
+          url: /config-guide/remote-storage/config-remote-storage-image-resize.html
+
     - label: Set up a custom cron job and cron group
       url: /config-guide/cron/custom-cron.html
       children:

src/common/images/config-remote-storage-nginx-image-resize.png

@@ -0,0 +1,59 @@
+---
+group: configuration-guide
+title: Configure AWS S3 bucket for remote storage
+functional_areas:
+  - Configuration
+  - System
+  - Setup
+---
+
+The [Amazon Simple Storage Service (Amazon S3)][AWS S3] is an object storage service that offers industry-leading scalability, data availability, security, and performance. The AWS S3 service uses buckets, or containers, for data storage. This configuration requires you to create a _private_ bucket.
+
+{:.bs-callout-warning}
+Magento highly discourages the use of public buckets because it poses a serious security risk.
+
+{:.procedure}
+To enable remote storage with the AWS S3 adapter:
+
+1. Log in to your Amazon S3 dashboard and create a _private_ bucket.
+
+1. Set up [AWS IAM][] roles. Alternatively, generate access and secret keys.
+
+1. Configure Magento to use the private bucket. See [Remote storage options][options] for a full list of parameters.
+
+   ```bash
+   bin/magento setup:config:set --remote-storage-driver="aws-s3" --remote-storage-bucket="<bucket-name>" --remote-storage-region="<region-name>" --remote-storage-prefix="<optional-prefix>" --access-key=<optional-access-key> --secret-key=<optional-secret-key> -n
+   ```
+
+## Configure Nginx
+
+Nginx requires an additional configuration to perform Authentication with the `proxy_pass` directive. Add the following proxy information to the `nginx.conf` file:
+
+>nginx.conf
+
+```conf
+location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
+    # Proxying to AWS S3 storage.
+    resolver 8.8.8.8;
+    set $bucket "<s3-bucket-name>";
+    proxy_pass https://s3.amazonaws.com/$bucket$uri;
+    proxy_pass_request_body off;
+    proxy_pass_request_headers off;
+    proxy_intercept_errors on;
+    proxy_hide_header "x-amz-id-2";
+    proxy_hide_header "x-amz-request-id";
+    proxy_hide_header "x-amz-storage-class";
+    proxy_hide_header "Set-Cookie";
+    proxy_ignore_headers "Set-Cookie";
+}
+```
+
+### Authentication
+
+If you use access and secret keys instead of [AWS IAM][] roles, you must include the [`ngx_aws_auth`][ngx repo] Nginx module.
+
+<!-- link definitions -->
+[AWS S3]: https://aws.amazon.com/s3
+[AWS IAM]: https://aws.amazon.com/iam/
+[options]: {{page.baseurl}}/config-guide/remote-storage/config-remote-storage.html#remote-storage-options
+[ngx repo]: https://github.com/anomalizer/ngx_aws_auth

src/common/images/config-remote-storage-schema.png

@@ -0,0 +1,46 @@
+---
+group: configuration-guide
+title: Configure caching for remote storage
+functional_areas:
+  - Configuration
+  - System
+  - Setup
+---
+
+Remote storage provides a cache layer for external API calls. The caching is based on the [Flysystem caching solution][flysystem], which provides cache adapters. Remote storage includes the following cache types:
+
+-  **Memory cache**—Memory cache is enabled by default and provides per-request cache.
+-  **Redis cache**—Redis cache is a persistent cache that is stored in Redis and requires an active Redis service with additional configuration.
+
+## Set up persistent Redis cache
+
+{:.procedure}
+To use Redis cache with [predis/predis][predis] extension:
+
+1. Add the `predis/predis` extension to the `require` section of your `composer.json` file.
+
+   ```bash
+   composer require predis/predis
+   ```
+
+1. Use the following example to add a configuration to the `app/etc/env.php` file.
+
+   ```php
+   [
+       'remote_storage' => [
+           # Remote storage configuration section
+           'cache' => [
+               'adapter' => 'predis',
+                   'config' => [
+                       'scheme' => 'tcp', # Required scheme
+                       'host' => 'redis.internal', # Required hostname of Redis service
+                       'port' => 6379 # Required port of Redis service
+                   ]
+           ]
+       ]
+   ];
+   ```
+
+<!-- link definitions -->
+[flysystem]: https://flysystem.thephpleague.com/v1/docs/advanced/caching
+[predis]: https://github.com/predis/predis

src/guides/v2.4/config-guide/remote-storage/config-remote-storage-aws-s3.md

@@ -0,0 +1,73 @@
+---
+group: configuration-guide
+title: Configure image resizing for remote storage
+functional_areas:
+  - Configuration
+  - System
+---
+
+By default, Magento supports image resizing on the application side. However, by enabling the Remote Storage module, you can use Nginx to offload image resizing to the server side, where you can save disk resources and optimize disk usage.
+
+The following diagram shows how Nginx retrieves, resizes, and stores images in the cache. Resizing is determined by the parameters included in the URL, such as height and width.
+
+![image resize]
+
+## Configure URL format in Magento
+
+To resize images on the server side, you must configure Magento to provide arguments for the height, width, and location (URL) of the image.
+
+{:.procedure}
+To configure Magento for server-side image resizing:
+
+1. In the _Admin_ panel, click **Stores** > **Settings** > **Configuration** > **General** > **Web**.
+
+1. In the right pane, expand **Url options**.
+
+1. In the _Catalog media URL format_ section, clear **Use system value**.
+
+1. Select the `Image optimization based on query parameters` URL in the **_Catalog media URL format_** field.
+
+1. Click **Save Config**.
+
+1. Continue to the [Nginx configuration](#configure-nginx).
+
+## Configure Nginx
+
+To continue configuring server-side image resizing, you must prepare the `nginx.conf` file and provide a `proxy_pass` value for your chosen adapter.
+
+{:.procedure}
+To enable Nginx to resize images:
+
+1. Install the [Nginx image filter module][nginx-module].
+
+   ```shell
+   load_module /etc/nginx/modules/ngx_http_image_filter_module.so;
+   ```
+
+1. Create an `nginx.conf` file based on the included template `nginx.conf.sample` file. For example:
+
+   ```conf
+   location ~* \.(ico|jpg|jpeg|png|gif|svg)$ {
+       set $width "-";
+       set $height "-";
+       if ($arg_width != '') {
+           set $width $arg_width;
+       }
+       if ($arg_height != '') {
+           set $height $arg_height;
+       }
+       image_filter resize $width $height;
+       image_filter_jpeg_quality 90;
+   }
+   ```
+
+1. [_Optional_] Configure a `proxy_pass` value for your specific adapter.
+
+   -  [Amazon Simple Storage Service (Amazon S3)][AWS S3]
+
+<!-- link definitions -->
+
+[AWS S3]: {{page.baseurl}}/config-guide/remote-storage/config-remote-storage-aws-s3.html
+[image resize]: {{site.baseurl}}/common/images/config-remote-storage-nginx-image-resize.png
+{:width="800px"}
+[nginx-module]: http://nginx.org/en/docs/http/ngx_http_image_filter_module.html

src/guides/v2.4/config-guide/remote-storage/config-remote-storage-caching.md

@@ -0,0 +1,74 @@
+---
+group: configuration-guide
+title: Configure remote storage
+functional_areas:
+  - Configuration
+  - System
+  - Setup
+---
+
+The Remote Storage module provides the option to store media files and schedule imports/exports in a persistent, remote storage container using a storage service, such as AWS S3 or Azure Blob Storage. By default, Magento stores media files in the same filesystem that contains the application. This is inefficient for complex, multi-server configurations, and can result in degraded performance when sharing resources. With the Remote Storage module, you can store media files in the `pub/media` directory and import/export files in the `var` directory of the remote object storage to take advantage of server-side image resizing.
+
+![schema image]
+
+## Remote storage options
+
+You can configure remote storage using the `remote-storage` option with the [`setup` CLI command][setup]. The `remote-storage` option uses the following syntax:
+
+```text
+--remote-storage-<parameter-name>="<parameter-value>"
+```
+
+The `parameter-name` refers to the specific remote storage parameter name. The following table lists the parameters available for configuring remote storage:
+
+| Command line Parameter | Parameter name | Description | Default value |
+|--- |--- |--- |--- |
+| `remote-storage-driver` | driver | Adapter name<br>Possible values:<br>**file**: Disables remote storage and uses the local filesystem<br>**aws-s3**: Use the [Amazon Simple Storage Service (Amazon S3)][AWS S3] | none |
+| `remote-storage-bucket` | bucket | Object storage or container name | none |
+| `remote-storage-prefix` | prefix | Optional prefix (location inside of object storage) | empty |
+| `remote-storage-region` | region | Region name | none |
+| `remote-storage-key` | access key | Optional access key | empty |
+| `remote-storage-secret` | secret key | Optional secret key | empty |
+
+### Storage adapters
+
+The default storage location is in the local filesystem. A _storage adapter_ enables you to connect to a storage service and store your files anywhere. Magento supports configuring the following storage services:
+
+-  [Amazon Simple Storage Service (Amazon S3)][AWS S3]
+
+## Enable remote storage
+
+You can install remote storage during a new Magento installation or add it to an existing Magento instance using `remote-storage` parameter name-and-value pairs with `setup` CLI commands. Minimally, you must supply the storage `driver`, `bucket`, and `region`.
+
+The following examples enable the remote storage with an AWS S3 storage adapter in the US:
+
+-  Install new Magento with remote storage
+
+   ```bash
+   bin/magento setup:install --remote-storage-driver="aws-s3" --remote-storage-bucket="myBucket" --remote-storage-region="us-east-1"
+   ```
+
+-  Enable remote storage on existing Magento
+
+   ```bash
+   bin/magento setup:config:set --remote-storage-driver="aws-s3" --remote-storage-bucket="myBucket" --remote-storage-region="us-east-1"
+   ```
+
+## Migrate content
+
+After you enable remote storage for a specific adapter, you can use the CLI to migrate existing _media_ files to the remote storage.
+
+```bash
+./magento2ce/bin/magento remote-storage:sync
+```
+
+{:.bs-callout-info}
+The sync command only migrates files in the `pub/media` directory, _not_ the import/export files in the `var` directory. See [Scheduled Import/Export][import-export] in the _Magento 2.4 User Guide_.
+
+<!-- link definitions -->
+[AWS S3]: {{site.baseurl}}/guides/v2.4/config-guide/remote-storage/config-remote-storage-aws-s3.html
+[import-export]: https://docs.magento.com/user-guide/system/data-scheduled-import-export.html
+[nginx-module]: http://nginx.org/en/docs/http/ngx_http_image_filter_module.html
+[schema image]: {{site.baseurl}}/common/images/config-remote-storage-schema.png
+{:width="500px"}
+[setup]: {{site.baseurl}}/guides/v2.4/install-gde/install/cli/install-cli-subcommands-deployment.html#instgde-cli-subcommands-configphp

src/guides/v2.4/config-guide/remote-storage/config-remote-storage-image-resize.md

@@ -66,7 +66,7 @@ The install command uses the following format:
 magento setup:install --<option>=<value> ... --<option>=<value>
 ```
 
-The following tables describe the installation option names and values. For example installation commands. see [Sample localhost installations](#install-cli-example).
+The following tables describe the installation option names and values, such as installation commands. See [Sample localhost installations](#install-cli-example).
 
 {:.bs-callout-info}
 Any options that contain spaces or special characters must be enclosed in either single or double quotes.
@@ -113,6 +113,9 @@ In Magento Commerce version 2.2.8 and later, you can create the Magento admin us
 |`--db-init-statements`|Advanced MySQL configuration parameter. Uses database initialization statements to run when connecting to the MySQL database. Consult a reference similar to this one before you set any values.<br><br>Default is `SET NAMES utf8;`.|No|
 |`--sales-order-increment-prefix`|Specify a string value to use as a prefix for sales orders. Typically, this is used to guarantee unique order numbers for payment processors.|No|
 
+{:.bs-callout-tip}
+To enable remote storage services during installation, see [Configure Remote Storage]({{ page.baseurl }}/config-guide/remote-storage/config-remote-storage.html) in the _Configuration Guide_.
+
 **Elasticsearch configuration options:**
 
 |Name|Value|Required?|