Replicating scangov

[blink-twice]

Hi @Narlotl

I am able to run the scraper on a selection of domains and generate the necessary json and CSV files. I have a clone of the website running to recreate the interface, reports, etc.

How are the json/csv files with the domain-level report data and the pages served by Jekyll linked? How I can get the data for my list of domains to show up? I am not familiar with node/webdev, so any pointers welcome.

[Narlotl]

The pages that show the sites make a GET request to the data files that live in the data directory. If the data for your domains isn't showing up, clear the cache for the local site and refresh. If it's still not working, create a repository and I'll take a look.

[afomi]

What is/are the node version/s?

[Narlotl]

I think this is an issue on my part, I'm calling caches.open(), which is only available in a secure context, regardless of whether or not it is available.

@blink-twice is the local URL you're using localhost/127.0.0.1 or is it your IP. If you're using your IP, try from localhost:4000 or whatever jekyll says for now and I'll get the caching issue fixed.

[blink-twice]

@afomi

I'm using node through nvm

nvm use 17
Now using node v17.9.1 (npm v8.11.0)

jekyll -v
jekyll 4.3.4

I did try newer versions but ran into the glibc error on my Ubuntu 18.04.1 LTS instance.

@Narlotl Thank you! I'm partly to blame for my non-standard config. Jekyll is running on an AWS Lightsail instance and rather than RDP into 127.0.0.1:4000, I wanted to access it from my browser and so I ran

bundle exec jekyll serve -H 0.0.0.0 -P 4005

and have been testing it using the instance's static IP.

I set up a desktop environment and VNC'd into the instance and it works!

[Narlotl]

I've fixed the caching issue, pull the updates and you should be able to use it as you were before without needing to VNC in.

[blink-twice]

Thanks @Narlotl

The repo is here. I'm using Github pages to host the site.

The home page is https://blink-twice.github.io/scangov_ind/ but the html and JS seem to pick https://blink-twice.github.io as the root, so I tested hardcoding the full path in a couple of places where relative URLs were failing. I was wondering if you know the cleanest way to set the root for the website so that everything gets handled smoothly without needing to modify all instances of paths? I edited index.html, and then util.js , and now the report and profile pages have the same issue. Thanks!