-
Notifications
You must be signed in to change notification settings - Fork 11
Data sources
The SPARC data portal provides access to SPARC generated resources such as files, meta-data and links to other repositories. Below, we describe how/where data of the various modalities is stored and how it can be accessed through the SPARC portal API.
Files are uploaded to the SPARC consortium organization on the DAT-Core. When data is published to the Open Data Library ("Blackfynn Discover"), the files get a static identifier that can be referenced in the metadata that is accessible to the SPARC data portal. Using the API, the SPARC Portal web-application can get a url that will request the file for download from AWS S3.
Meta data is captured per dataset in the DAT-Core based on the metadata schema that is developed by the data standards committee. When users publish their dataset, these metadata are integrated with all other publicly available metadata from SPARC datasets and made available in the SPARC Portal.
We make a copy of the metadata and store this integrated collection of records in a dedicated database that is associated with the SPARC Data Portal using GrapheneDB. The SPARC Webapplication has a read-only permission to this database and web-app developers can use native Cypher to construct graph queries over this database.