No data for 2019 for a tile in a MODIS collection

[shijithpk]

So I'm looking at temperature data for a part of India and I'm not getting any results for the whole of 2019. The collection id is modis-11A1-061 and the tile id is 51025007.

I am getting results for other years, and I am getting results for other tiles for 2019.

When I checked with NASA's appeears service, I did get some data for 2019 for the same area (link to netcdf file below).

Should I be doing something differently here? Like querying other metadata?

import pystac_client
import planetary_computer

stac = pystac_client.Client.open(
	"https://planetarycomputer.microsoft.com/api/stac/v1",
	modifier=planetary_computer.sign_inplace,
)

# bbox = [74.8640682,  8.2924566, 77.4119403, 12.7960559] 

def print_num_results(datetime_string, tile_id):
	search = stac.search(
		# bbox=bbox,
		datetime=datetime_string,
		collections=["modis-11A1-061"], 
		query={
				'platform':{'eq':'aqua'},
				'modis:tile-id':{ 'eq':tile_id},
				})
	search_results = search.item_collection()
	all_search_results = list(search_results)
	print(len(all_search_results))

print_num_results('2019-01-01/2019-12-31', '51025007')
# got 0
print_num_results('2020-01-01/2020-12-31', '51025007')
# 250
print_num_results('2019-01-01/2019-12-31', '51012005') # tile covering Washington DC
# 336

## If anyone wants to check the netcdf file I got from NASA's appeears service
# import rioxarray 
# import requests

# response = requests.get('https://drive.google.com/uc?export=download&id=1CthUeQJN0CYAaEpN_b92nUwBnYoDfwU2')
# with open('2019_LST.nc', "wb") as opened_file:
#	opened_file.write(response.content)
# nc = rioxarray.open_rasterio('2019_LST.nc')
# nc['LST_Day_1km'][50].plot()

[TomAugspurger]

Thanks for the clear example. I think I've confirmed the existence of at least one NetCDF file without a corresponding STAC item:

import planetary_computer
import pystac_client

cc = planetary_computer.get_container_client("modiseuwest", "modis-061")

prefix = "MYD11A1/25/07/2019"
blobs = cc.list_blobs(name_starts_with=prefix)
blob = next(blobs)

print(blob.name)  # MYD11A1/25/07/2019001/MYD11A1.A2019001.h25v07.061.2020342151634.hdf

import requests

item_id = blob.name.split("/")[-1].rsplit(".", 1)[0]
r = requests.get(f"https://planetarycomputer.microsoft.com/api/stac/v1/collections/modis-11A1-061/items/{item_id}")
# url: https://planetarycomputer.microsoft.com/api/stac/v1/collections/modis-11A1-061/items/MYD11A1.A2019001.h25v07.061.2020342151634
r.status_code  #  404

I'll look into this more and will post an update here.

[shijithpk]

Cool 👍 This MODIS satellite Aqua has been putting out data since 2002, so I checked how many results I'm getting for this particular tile-id 51025007 for other years. Apart from 2019, three other years -- 2013, 2014 and 2017 -- are giving me no results. The rest seem to be fine.

Edit: Also checked for an adjacent tile 51025008. Four years -- 2010, 2013, 2019, 2020 -- are giving me no results, 2014 is missing a lot of data too (just 80 results when it should be over 300). The other years are fine.

[TomAugspurger]

All the missing items should be present in the STAC API now. My example for the item at https://planetarycomputer.microsoft.com/api/stac/v1/collections/modis-11A1-061/items/MOD11A1.A2007001.h00v10.061.2021054173634 now correctly returns the STAC item.

In [2]: requests.get("https://planetarycomputer.microsoft.com/api/stac/v1/collections/modis-11A1-061/items/MOD11A1.A2007001.h00v10.061.2021054173634").status_code
Out[2]: 200

We're still not sure exactly what happened, but we apparently lost some items from this collection as part of a blue / green deployment switch during a release. We're working on a system that'll make it much easier to track the STAC metadata as it makes its way through our systems, which should prevent this sort of issue from occurring again.

Thanks again for reporting this and let us know if you spot any more issues.

[shijithpk]

Just checked, the data's there now, getting 360+ results for every year, thanks for looking into this!