Ideas for making this project sustainable

[jlnr]

This project cannot really work if I have to patch it for every single site that breaks without its custom icon font.

Here are some ideas for improving this process:

• Maintain the list of exceptions in a YAML file that contains the pattern(s), a comment (which JSON does not allow), and maybe a URL for re-evaluating the exception later.
• A tiny Ruby build tool then checks which rules are outdated.
• And most importantly, the tool should also crawl the top 10.000 domains or so, and see if example.com uses a font called example.ttf, which is then very likely to be the custom icon font for this website - and add an exception for this.

Also, replacing icon fonts with SVG files is a thing now: https://sarasoueidan.com/blog/icon-fonts-to-svg/ 🎉 🎉 🎉

[pocketarc]

I think this would be a terrific idea — the "re-evaluating the exception" would actually be fantastic to do with Travis in this repo; you could have all URLs being re-evaluated on every commit to make sure that nothing we do ever breaks an existing exception.

Speaking of using YAML, have you considered having SansFont's iOS (and later the OS X app, when #38 is done) fetch the updated YAML file from the repo every day or so, so that you don't need to worry about releasing new versions just to update compatibility?

[jlnr]

About downloading new definitions – I think that this will actually make the update process more fragile. Right now, if I push an update, most people will automatically receive it, and the OS will hopefully reload the JSON from inside the app bundle (I should probably verify this with the next update).

In contrast, to load updates from within the app, people would have to open it. I don't think people spend a lot of time in the Sans Fonts apps, and so they'd be stuck with outdated definitions forever.

To automate this, I should really set up Fastlane so I can release new versions with fastlane mac or fastlane ios from the command line.

[pocketarc]

Well, you're right there! If you can automate pushing a new release that'd take care of the issue, absolutely. I didn't even think about that. Definitely worth doing it that way if it's not more hassle for you.

[jlnr]

I've put a basic mechanism for regenerating blockerList.json in place (just a short page of Ruby).

To completely automate this extension, a little macOS app would probably work best. It'd scrape the top 1000 pages off Hacker News/Alexa/whatever, intercepting resource loads, and then adding exceptions for icon fonts based on a few heuristics.

[pocketarc]

Nice.

If you're just doing scraping then you don't really need a macOS app, I imagine. Just a command-line tool, no? So we'd run ./scrape and off it would go, finding any necessary exceptions and either committing them directly or creating them as PRs in GH. Heck, I could even build that remotely and have a cron service run it every other day or so, so you'd just get a bunch of PRs and just have to approve them.

Having said that though, I think the most pressing issue for sustainability might not necessarily be the blockerList.json but the release system - if, whenever you merge a PR, a new release was triggered (you talked about automating this in the past), that would be fantastic, because otherwise the project depends on you having enough time to continuously push releases, which makes it more of a hassle for you than it needs to be.

Also, I did not know you had released the Mac app! That's awesome; goodbye extension! 🎉

[jlnr]

Hah, touché. I've run into certificate trouble while releasing 1.5 just now, so I am indeed the bottleneck. I'll set up fastlane later today, which is the first step towards automating this thing.

As for the scraper: My thinking here was that a "real" WebView would correctly trigger all the delayed loading on modern websites, passing each resource to its delegate (the scraper) as it is requested. That would make it relatively easy to get at the filenames of all referenced web fonts. I am not sure if that works reliably when using a non-visual command-line scraping library. I'd like to avoid parsing CSS files, for example.

Are you thinking of any scraping library in particular?

[pocketarc]

I was just thinking about PhantomJS, which uses the same browser engine and can be controlled with code. It even has documentation on dealing with all network requests (which'd allow seeing which fonts were loaded) http://phantomjs.org/network-monitoring.html

But the WebView idea seems like it'd work as well; I didn't even think about that!

[jlnr]

Ah, I've heard of Phantom but never used it. If you want to give it a try, I'd be happy to review and merge it. I probably won't be able to do much in the next days or weeks :( But here's my actionable To Do for this issue:

• Do not manually edit blockerList.json as it is pretty hard to navigate due to its bloaty structure and lack of comments. Also, git blame never figures out which braces belong to which change.
• Write a tool that guesses which fonts on a website are icon fonts, and adds them to some list that Sans Fonts can use (GitHub PR or direct update of Rakefile)
• Let this tool run over thousands of common websites. This site might be a good starting point: https://hn.algolia.com
• Initial Fastlane setup for iOS releases.
• Improve Fastfile to automatically bump the app's version.
• Improve Fastlane setup to automate Mac releases as much as possible. I think the actual uploads cannot be automated right now.

[sebastianludwig]

https://scrapy.org has been mentioned on HN recently when scraping was discussed. Just to throw in another alternative.

[pocketarc]

@jlnr The curse of open source project maintainers; never enough spare time. 😛 I don't know how much time I'll have either, but if I get a few spare hours I'll work on the scraper tool using PhantomJS so it can run on a normal Linux server on a cron.

@sebastianludwig Scrapy looks good but it doesn't look like it actually loads a web page as a proper browser (running JS and whatnot), which is one of @jlnr's requirements, so we can even pick up on fonts loaded with CSS Font Loading or other JS-based techniques.