Valid data sources for graph sharing through https://cosmograph.app/run/

[bingbong-sempai]

Hi, are there specific data sources allowed for graph sharing?
I've been having mixed results with different data hosting services.
Github (gist) works, but I've been unable to use files from Google drive or Google cloud storage (https://storage.googleapis.com/).

[rokotyan]

The data source should support CORS since we're not proxying any of the requests.

We don't have a list of sources that work yet, but it would be great to have. So if you find any other sources besides GitHub that work, please don't hesitate to post them here.

[bingbong-sempai]

Thanks for the info! BTW is there a data limit for graph sharing?
I'm testing a dataset of 35k nodes, 80k edges and not all nodes/edges make it to the graph.
But loading the data manually generates the complete graph.

[rokotyan]

@bingbong-sempai There're no limits, so probably it's a bug on Cosmograph's end. If you can share your data with us please go ahead and we'll take a look.

[bingbong-sempai]

Apologies, here's a bigger dataset of 82k nodes, 213k edges.
Only 72k nodes, 155k edges make it to the graph.
Manually uploading the data gets all nodes and edges to show.

https://cosmograph.app/run/?nodeColor=color-node_type&nodeSize=size-n_posts&linkWidth=width-avg-n_posts&data=https://gist.githubusercontent.com/bingbong-sempai/c8d79ca1f8f9c1df5b905c9b995197bb/raw/5f462c8490408137205fa9edc0df1b59b20357c1/data.csv&meta=https://gist.githubusercontent.com/bingbong-sempai/c8d79ca1f8f9c1df5b905c9b995197bb/raw/5f462c8490408137205fa9edc0df1b59b20357c1/meta.csv

[kolmakova]

Hello @bingbong-sempai! Thanks for the data you've provided. The reason for the partial data loading is due to CORS restrictions. To parse a file from a remote URL, the parser needs to receive the size of the file, which is what the Content-Length header is responsible for. However, servers like gist.githubusercontent.com restrict access to this header by default due to their CORS policy. As a result, only the first chunk of data (which in the current version of Cosmograph is set to 5 MB) is loaded. We're planning to increase this limit to 20 MB with the next Cosmograph update.

Currently, for personal usage, the best solution to this problem is to use a CORS unblocking extension like this one. Make sure the option Enable Access-Control-[Allow/Expose]-Headers is checked, and then Cosmograph will be able to receive all the data. In Safari, enabling the Disable Cross-Origin Restrictions option from the Develop menu should work.

Alternatively, you could try finding another hosting service without these CORS restrictions and upload your file there.