Skip to content

Commit a08184b

Browse files
author
Phil Varner
authored
Remove opensearch aws connection support (#511)
* remove elasticsearch support * remove aws connection support * update docs on fine-grained access control * add USGS Astrogeology api * remove aws-os-connection dependency
1 parent 7baa4ed commit a08184b

File tree

7 files changed

+268
-305
lines changed

7 files changed

+268
-305
lines changed

‎.npm-upgrade.json

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,5 @@
11
{
22
"ignore": {
3-
"@acuris/aws-es-connection": {
4-
"versions": "^2",
5-
"reason": "deprecated feature, leave as-is"
6-
},
73
"@types/node": {
84
"versions": "^18",
95
"reason": "using 16"

‎CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
1010
### Removed
1111

1212
- Elasticsearch is no longer supported as a backend. Only OpenSearch is now supported.
13+
- Only fine-grained access control is supported for connecting to OpenSearch.
14+
"AWS Connection" mode is no longer supported.
1315

1416
### Added
1517

@@ -28,6 +30,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
2830
## [1.1.0] - 2023-05-02
2931

3032
### Changed
33+
3134
- Adds API Gateway ID to cloudwatch logs.
3235
- Logs the start of the request in case of Lambda timeout.
3336

‎README.md

Lines changed: 52 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
- [Overview](#overview)
77
- [Architecture](#architecture)
88
- [Migration](#migration)
9+
- [0.x or 1.x -\> 2.x](#0x-or-1x---2x)
10+
- [Fine-grained Access Control](#fine-grained-access-control)
911
- [0.4.x -\> 0.5.x](#04x---05x)
1012
- [Elasticsearch to OpenSearch Migration](#elasticsearch-to-opensearch-migration)
1113
- [Preferred Elasticsearch to OpenSearch Migration Process](#preferred-elasticsearch-to-opensearch-migration-process)
@@ -19,8 +21,7 @@
1921
- [OpenSearch Configuration](#opensearch-configuration)
2022
- [Disable automatic index creation](#disable-automatic-index-creation-1)
2123
- [Create collection index](#create-collection-index)
22-
- [Enable OpenSearch fine-grained access control](#enable-opensearch-fine-grained-access-control)
23-
- [Configure OpenSearch for fine-grained access control](#configure-opensearch-for-fine-grained-access-control)
24+
- [OpenSearch fine-grained access control](#opensearch-fine-grained-access-control)
2425
- [Option 1 - API method](#option-1---api-method)
2526
- [Option 2 - Dashboard method](#option-2---dashboard-method)
2627
- [Populating and accessing credentials](#populating-and-accessing-credentials)
@@ -59,7 +60,7 @@ Stac-server is an implementation of the [STAC API specification](https://github.
5960
| 0.3.x | 1.0.0 | 1.0.0-beta.2 |
6061
| 0.4.x | 1.0.0 | 1.0.0-beta.5 |
6162
| 0.5.x-0.8.x | 1.0.0 | 1.0.0-rc.2 |
62-
| 1.0.0 | 1.0.0 | 1.0.0 |
63+
| >=1.0.0 | 1.0.0 | 1.0.0 |
6364

6465
Currently, stac-server supports the following specifications:
6566

@@ -70,14 +71,16 @@ Currently, stac-server supports the following specifications:
7071
- Query Extension
7172
- Fields Extension
7273
- Sort Extension
73-
- Aggregation Extension (experimental, work-in-progress)
74+
- Aggregation Extension (experimental)
7475

7576
The following APIs are deployed instances of stac-server:
7677

77-
| Name | STAC Version | STAC API Version | Description |
78-
| ---------------------------------------------------------- | ------------ | ---------------- | ----------------------------------- |
79-
| [Earth Search](https://earth-search.aws.element84.com/v0/) | 1.0.0-beta.2 | 0.9.0 | Catalog of some AWS Public Datasets |
80-
| [Landsat Look](https://landsatlook.usgs.gov/stac-server) | 1.0.0 | 0.9.0 | |
78+
| Name | STAC Version | STAC API Version | Description |
79+
| -------------------------------------------------------------------- | ------------ | ---------------- | ---------------------------------------- |
80+
| [Earth Search v1](https://earth-search.aws.element84.com/v1) | 1.0.0 | 1.0.0 | Catalog (v1) of some AWS Public Datasets |
81+
| [USGS Astrogeology STAC API](https://stac.astrogeology.usgs.gov/api) | 1.0.0 | 1.0.0 | A STAC API for planetary data |
82+
| [Earth Search v0](https://earth-search.aws.element84.com/v0) | 1.0.0-beta.2 | 0.9.0 | Catalog (v0) of some AWS Public Datasets |
83+
| [Landsat Look](https://landsatlook.usgs.gov/stac-server) | 1.0.0 | 0.9.0 | |
8184

8285
## Architecture
8386

@@ -124,6 +127,14 @@ apiLambda --> opensearch
124127

125128
## Migration
126129

130+
### 0.x or 1.x -> 2.x
131+
132+
#### Fine-grained Access Control
133+
134+
As of 2.0.0, only OpenSearch is supported and only using fine-grained access control.
135+
It is recommended to follow the migration path to upgrade to fine-grained access control
136+
first and then upgrade to stac-server 2.x.
137+
127138
### 0.4.x -> 0.5.x
128139

129140
#### Elasticsearch to OpenSearch Migration
@@ -419,29 +430,44 @@ There are some settings that should be reviewed and updated as needeed in the se
419430
| PRE_HOOK | The name of a Lambda function to be called as the pre-hook. | none |
420431
| POST_HOOK | The name of a Lambda function to be called as the post-hook. | none |
421432
| ES_COMPAT_MODE | Enable Elasticsearch 7.10 compatibility mdoe within the server. | false |
422-
| OPENSEARCH_USERNAME | The username to authenticate to OpenSearch with if fine-grained access control is enabled. | |
423-
| OPENSEARCH_PASSWORD | The password to authenticate to OpenSearch with if fine-grained access control is enabled. | |
424-
| OPENSEARCH_CREDENTIALS_SECRET_ID | The AWS Secrets Manager secret to retrieve the username and password from, to authenticate to OpenSearch with if fine-grained access control is enabled. | |
433+
| OPENSEARCH_CREDENTIALS_SECRET_ID | The AWS Secrets Manager secret use for the username and password to authenticate to OpenSearch. | |
434+
| OPENSEARCH_USERNAME | The username to authenticate to OpenSearch when AWS Secrets Manager is not used. | |
435+
| OPENSEARCH_PASSWORD | The password to authenticate to OpenSearch when AWS Secrets Manager is not used. | |
425436
| COLLECTION_TO_INDEX_MAPPINGS | A JSON object representing collection id to index name mappings if they do not have the same names. | |
426437

427438
| ITEMS_INDICIES_NUM_OF_SHARDS | Configure the number of shards for the indices that contain Items. | none |
428439
| ITEMS_INDICIES_NUM_OF_REPLICAS | Configure the number of replicas for the indices that contain Items. | none |
429440

441+
The preferred mechanism for populating the OpenSearch credentials to stac-server is to
442+
create a secret in AWS Secret Manager that contains the username and password.
443+
The recommended name for this Secret corresponds
444+
to the stac-server deployment as `{stage}/{service}/opensearch`, e.g.,
445+
`dev/my-stac-server/opensearch`.
446+
447+
The Secret type should be "Other type of secret" and
448+
have two keys, `username` and `password`, with the appropriate
449+
values, e.g., `stac_server` and whatever you set as the password when creating that user.
450+
430451
After reviewing the settings, build and deploy:
431452

432453
```shell
433454
npm install
434455
npm run build
435-
npm run deploy
456+
OPENSEARCH_MASTER_USER_PASSWORD='some-password' npm run deploy
436457
```
437458

438459
This will use the file `serverless.yml` and create a CloudFormation stack in the
439460
`us-west-2` region called `stac-server-dev`.
461+
462+
After the initial deployment, the `MasterUserOptions` option in the serverless.yml file
463+
can be commented out so that OPENSEARCH_MASTER_USER_PASSWORD does not need to be passed
464+
at every deployment.
465+
440466
To change the region or the stage name (from `dev`) provide arguments to the deploy command
441467
(note the additional `--` in the command, required by `npm` to provide arguments):
442468

443469
```shell
444-
npm run deploy -- --stage mystage --region eu-central-1
470+
OPENSEARCH_MASTER_USER_PASSWORD='some-password' npm run deploy -- --stage mystage --region eu-central-1
445471
```
446472

447473
Multiple deployments can be managed with multiple serverless config files and specified
@@ -461,40 +487,19 @@ It is recommended to disable the automatic index creation. This prevents the sit
461487
a group of Items are bulk indexed before the Collection in which they are contained has
462488
been created, and an OpenSearch index is created without the appropriate mappings.
463489

464-
This requires installing the requests, requests_aws4auth, and boto3 python libraries, for example, with:
465-
466-
```shell
467-
pip install requests requests_aws4auth boto3
468-
```
469-
470-
Then putting this code into a python file an running it:
471-
472-
```python
473-
from requests_aws4auth import AWS4Auth
474-
import boto3
475-
import requests
490+
This can either be done by calling the `/_cluster/settings` endpoint directly with the
491+
body:
476492

477-
host = 'https://my-test-domain.us-east-1.es.amazonaws.com'
478-
path = '/_cluster/settings'
479-
region = 'us-west-2'
480-
481-
credentials = boto3.Session().get_credentials()
482-
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, 'es', session_token=credentials.token)
483-
484-
485-
r = requests.put(
486-
f'{host}{path}',
487-
auth=awsauth,
488-
json={
493+
```json
494+
{
489495
"persistent": {
490496
"action.auto_create_index": "false"
491497
}
492-
})
493-
494-
print(r.status_code)
495-
print(r.text)
498+
}
496499
```
497500

501+
or setting that configuration via the OpenSearch Dashboard.
502+
498503
#### Create collection index
499504

500505
The `collection` index must be created, which stores the metadata about each Collection.
@@ -518,60 +523,22 @@ aws lambda invoke \
518523

519524
Stac-server is now ready to ingest data!
520525

521-
#### Enable OpenSearch fine-grained access control
526+
#### OpenSearch fine-grained access control
522527

523-
There are two main reasons to enable fine-grained access control to OpenSearch.
524-
The default stac-server deployment controls access to the OpenSearch cluster using
525-
IAM permissions, which results in access to the cluster with full admin permissions.
526-
Additionally, using cross-cluster search or replication requires fine-grained access
527-
control is configured to connect the clusters.
528+
As of version 2.0.0, stac-server on"ly supports fine-grained access control to
529+
OpenSearch, and no longer supports "AWS Connection" mode.
528530

529531
**Warning**: Unfortunately, fine-grained access control cannot be enabled on an
530532
existing OpenSearch
531533
cluster through the serverless deploy, as this is a restriction of CloudFormation
532534
which serverless uses. A migration process between the clusters must be performed similar
533535
to the Elasticsearch -> OpenSearch migration process.
534536

535-
##### Configure OpenSearch for fine-grained access control
536-
537-
Add this to the `AWS::OpenSearchService::Domain` resource:
538-
539-
```yaml
540-
DomainEndpointOptions:
541-
EnforceHTTPS: true
542-
NodeToNodeEncryptionOptions:
543-
Enabled: true
544-
EncryptionAtRestOptions:
545-
Enabled: true
546-
AdvancedSecurityOptions:
547-
Enabled: true
548-
InternalUserDatabaseEnabled: true
549-
MasterUserOptions:
550-
MasterUserName: admin
551-
MasterUserPassword: ${env:OPENSEARCH_MASTER_USER_PASSWORD}
552-
AccessPolicies:
553-
Version: "2012-10-17"
554-
Statement:
555-
- Effect: "Allow"
556-
Principal: { "AWS": "*" }
557-
Action: "es:ESHttp*"
558-
Resource: "arn:aws:es:${aws:region}:${aws:accountId}:domain/${self:service}-${self:provider.stage}/*"
559-
```
560-
561537
The AccessPolicies Statement will restrict the OpenSearch instance to only being accessible
562538
within AWS. This requires the user creation steps below be either executed from or proxied
563539
through an EC2 instance, or that the Access Policy be changed temporarily through the
564540
console in the domain's Security configuration to be "Only use fine-grained access control".
565541

566-
Deploying now requires the `OPENSEARCH_MASTER_USER_PASSWORD` local shell environment
567-
variable be set so it can be set for the OpenSearch "master" user, e.g.:
568-
569-
```shell
570-
OPENSEARCH_MASTER_USER_PASSWORD='some-password' npm run deploy
571-
```
572-
573-
Redeploy the stack, and this will be updated without re-creating the cluster.
574-
575542
The next step is to create the OpenSearch user and role to use for stac-server. This can
576543
either be done through the OpenSearch API or Dashboard.
577544

@@ -927,7 +894,6 @@ Messages published to the post-ingest SNS topic include the following atributes
927894
| ingestStatus | String | `successful` or `failed` |
928895
| collection | String | |
929896

930-
931897
### Ingesting large items
932898

933899
There is a 256 KB limit on the size of SQS messages. Larger items can by publishing a message to the `stac-server-<stage>-ingest` SNS topic in with the format:
@@ -957,14 +923,14 @@ across the clusters, treating a remote cluster as if it were another group of no
957923
cluster, or configure indicies to be replicated (continuously copied) from from one
958924
cluster to another.
959925

960-
Configuring either cross-cluster behavior requires [enabling fine-grained access control](#enable-opensearch-fine-grained-access-control).
926+
Configuring either cross-cluster behavior requires fine-grained access control.
961927

962928
### Cross-cluster Search
963929

964930
The AWS documentation for cross-cluster search can be found
965931
[here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/cross-cluster-search.html).
966932

967-
1. [Enable fine-grained access control](#enable-opensearch-fine-grained-access-control)
933+
1. Ensure fine-grained access control is enabled.
968934
2. Create a connection between the source and destination OpenSearch domains.
969935
3. Ensure there is a `es:ESCrossClusterGet` action in the destination's access policy.
970936
4. In the source stac-server, create a Collection for each collection to be mapped. This
@@ -979,7 +945,7 @@ The AWS documentation for cross-cluster search can be found
979945
The AWS documentation for cross-cluster replication can be found
980946
[here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html).
981947

982-
1. [Enable fine-grained access control](#enable-opensearch-fine-grained-access-control)
948+
1. Ensure fine-grained access control is enabled (default as of v2.0.0)
983949
2. Create the replication connection in the source to the destination
984950
3. Create the collection in the source's stac-server instance
985951

0 commit comments

Comments
 (0)