Merge pull request #9 from sat-utils/develop

release 0.1.0

Author: matthewhanson

README.md

@@ -1,6 +1,8 @@
 # sat-stac-sentinel
 
-This is a repository used for the creation and maintenance of a [STAC](https://github.com/radiantearth/stac-spec) compliant [Sentinel catalog](https://sentinel-stac.s3.amazonaws.com/catalog.json) for data from the [Sentinel on AWS project](https://registry.opendata.aws/sentinel-2/) (located at s3://sentinel-s2-l1c/).
+This is a repository used for the creation and maintenance of a [STAC](https://github.com/radiantearth/stac-spec) compliant [Sentinel catalog](https://sentinel-stac.s3.amazonaws.com/catalog.json) for data from the [Sentinel on AWS project](https://registry.opendata.aws/sentinel-2/) (located at s3://sentinel-s2-l1c/).  New STAC Items (i.e., Sentinel scenes) are published to an SNS Topic on AWS: 'arn:aws:sns:eu-central-1:552188055668:sentinel-stac'
+
+The catalog can be browsed at https://sentinel-stac.s3.amazonaws.com/index.html.
 
 There are two pieces of this repository:
 
@@ -11,6 +13,7 @@ To create the Sentinel STAC catalog located at https://sentinel-stac.s3.amazonaw
 
 ## Installation
 
+Sat-stac-landsat can be installed from this repository. It is not in PyPi because it is not a library that is going to be of general use. It exists to create a Landsat STAC catalog and keep it up to date, which is currently ongoing.
 
 
 ## Usage
@@ -42,20 +45,47 @@ This will fetch the latest inventory files from s3://sentinel-inventory/sentinel
 
 This will ingest records either from a local inventory file or, if not provided, the latest bucket inventory files
 
+```
+$ sat-stac-sentinel ingest -h
+usage: sat-stac-sentinel ingest [-h] [--version] [--log LOG] [--start START]
+                                [--end END] [--prefix PREFIX] [--s3meta]
+                                [--filename FILENAME] [--publish PUBLISH]
+                                catalog
+
+positional arguments:
+  catalog              Catalog that contains the Collection
+
+optional arguments:
+  -h, --help           show this help message and exit
+  --version            Print version and exit
+  --log LOG            0:all, 1:debug, 2:info, 3:warning, 4:error, 5:critical
+                       (default: 2)
+  --start START        Start date of ingestion (default: None)
+  --end END            End date of ingestion (default: None)
+  --prefix PREFIX      Only ingest scenes with a path starting with prefix
+                       (default: None)
+  --s3meta             Get metadata directly from S3 (requestor pays)
+                       (default: False)
+  --filename FILENAME  Inventory filename to use (default to fetch latest from
+                       bucket Inventory files) (default: None)
+  --publish PUBLISH    ARN to publish new Items to (default: None)
+```
 
 The `catalog` argument is the URL to the root catalog which contains a child collection called 'sentinel-2-l1c'. If the 'sentinel-2-l1c' Collection does not exist in the Catalog it will be added. In the case of the catalog maintained by this repo it is located at https://sentinel-stac.s3.amazonaws.com/catalog.json.
 
 If `start` and/or `end` are provided the records are all scanned and only those meeting the date requirements are ingested.
 
+If `filename` is provided it will read the inventory from a file (generated with `sat-stac-sentinel inventory`) instead of retriving the latst bucket inventory files.
 
-## Transforming Sentinel metadata to STAC
+The `s3meta` switch controls where the metadata comes from. The default is to get it from a proxy address rather than directly from the bucket (which is requestor pays). The actual cost is negligible, but it tunns out there is little difference in speed so this switch can be left alone.
 
-The data that is ingested by the sat-stac-sentinel CLI starts with
-
-In addition to the inventories, an SNS message is published (arn:aws:sns:us-west-2:274514004127:NewSceneHTML) whenever a new `index.html` appears in the bucket. The sat-stac-sentinel Lambda function listens for this message to get the link of the s3 path with the new scene.
+The `publish` switch allows publishing of a new STAC Item to an SNS topic. This should not be used when creating a whole catalog, it will create too many SNS messages.
 
+## Transforming Sentinel metadata to STAC
 
+The data that is ingested by the sat-stac-sentinel CLI starts with bucket inventory files that are retrieved and used to find all of the `tileInfo.json` files (this is the metadata for one Sentinel scene). In addition to the inventories, an SNS message is published (arn:aws:sns:us-west-2:274514004127:NewSceneHTML) whenever a new `index.html` appears in the bucket. The sat-stac-sentinel Lambda function listens for this message to get the link of the s3 path with the new scene.
 
+In either case the tileInfo.json contains all of the data needed to create a STAC Item, however the given geometry is in native coordinates rather than lat/lon. The library reprojects the geometry using EPSG:4326. Additionally, the convex hull is calculated for the geometry. In most cases this doesn't make a difference, however some of the tile geometries can be long and complex. Taking the convex hull is a way to simplify the geometry without impacting search and discovery.
 
 ## Development
 

satstac/sentinel/main.py

@@ -100,7 +100,13 @@ def add_items(catalog, records, start_date=None, end_date=None, s3meta=False, pr
 def read_inventory(filename):
     """ Create generator from inventory file """
     with open(filename) as f:
-        f.readline()
+        line = f.readline()
+        if 'datetime' not in line:
+            parts = line.split(',')
+            yield {
+                'datetime': parse(parts[0]),
+                'path': parts[1].strip('\n')
+            }
         for line in f.readlines():
             parts = line.split(',')
             yield {
@@ -159,19 +165,19 @@ def transform(data):
         'info': {'href': op.join(roda_url, 'tileInfo.json')},
         'metadata': {'href': op.join(roda_url, 'metadata.xml')},
         'tki': {'href': op.join(url, 'TKI.jp2')},
-        'B01': {'href': op.join(url, 'B01.TIF')},
-        'B02': {'href': op.join(url, 'B02.TIF')},
-        'B03': {'href': op.join(url, 'B03.TIF')},
-        'B04': {'href': op.join(url, 'B04.TIF')},
-        'B05': {'href': op.join(url, 'B05.TIF')},
-        'B06': {'href': op.join(url, 'B06.TIF')},
-        'B07': {'href': op.join(url, 'B07.TIF')},
-        'B08': {'href': op.join(url, 'B08.TIF')},
-        'B8A': {'href': op.join(url, 'B08.TIF')},
-        'B09': {'href': op.join(url, 'B09.TIF')},
-        'B10': {'href': op.join(url, 'B10.TIF')},
-        'B11': {'href': op.join(url, 'B11.TIF')},
-        'B12': {'href': op.join(url, 'B11.TIF')}
+        'B01': {'href': op.join(url, 'B01.jp2')},
+        'B02': {'href': op.join(url, 'B02.jp2')},
+        'B03': {'href': op.join(url, 'B03.jp2')},
+        'B04': {'href': op.join(url, 'B04.jp2')},
+        'B05': {'href': op.join(url, 'B05.jp2')},
+        'B06': {'href': op.join(url, 'B06.jp2')},
+        'B07': {'href': op.join(url, 'B07.jp2')},
+        'B08': {'href': op.join(url, 'B08.jp2')},
+        'B8A': {'href': op.join(url, 'B08.jp2')},
+        'B09': {'href': op.join(url, 'B09.jp2')},
+        'B10': {'href': op.join(url, 'B10.jp2')},
+        'B11': {'href': op.join(url, 'B11.jp2')},
+        'B12': {'href': op.join(url, 'B11.jp2')}
     })
     #if dt < datetime(2016,12,6):
     #    del assets['tki']

satstac/sentinel/sentinel-2-l1c.json

@@ -30,7 +30,7 @@
             "url": "https://earth.esa.int/web/guest/home"
         },
         {
-            "name": "Synergise",
+            "name": "Sinergise",
             "roles": [
                 "processor"
             ],
@@ -233,4 +233,4 @@
             "href": "https://sentinel.esa.int/documents/247904/690755/Sentinel_Data_Legal_Notice"
         }
     ]
-}
+}