Blocklists for bad domains

[Csardelacal]

Hey there!

I am opening this as a separate discussion since it's a different topic. Are there any plans to incorporate something like DNSBL (https://www.spamhaus.org/dbl/) for validating domains that a user has input?

[acelaya]

With what purpose?

[Csardelacal]

Hey there,

first of all, thanks for looking into this. I just feel like you assumed I want to specifically add one vendor's product, which is not the case. But I would like to address some of the issues you mention:

It deviates a bit from Shlink's core domain. Nothing prevents you from implementing this logic on top of Shlink, trying to shorten URLs only if they meet whatever requirements you want.

This reasoning is IMO flawed in a series of ways:

1. Spam and bad actors usually rely on changing domains and shifting URLs. So, at the point of creation, the application may not yet be aware that ameazon.com is a bad website or that they shouldn't enter their credit card data into paypal.com.scamdomain069.xyz and may allow the user to shorten that URL. But if the domain is blocked, users should no longer be redirected there.
2. Applications shortening the URL would have to all implement the logic to filter bad URLs or domains. Which requires this code to be sprinkled into every application shortening links and maintaining it across several applications. On top of that, whenever a domain or URL is blocked the database would have to be queried for existing entries.

I think that, whenever a link shortener is used in conjunction with user generated content, there's the risk that a malicious third party starts using the shortener as an open redirect to bypass spam filters.

There has to be a really good reason to couple Shlink with an external service, as it requires a lot of extra error handling and sensible defaults.

I am not bound to one specific vendor. My understanding is that spamhaus is a reputable one and they seem to be very well regarded among mail servers. That's why I am familiar with them. But I am aware that there's many smaller lists of phishing and scam domains out there that can be downloaded.

For reference, their lists are queried by spamassassin: https://spamassassin.apache.org/old/tests_3_3_x.html

I don't mind if there is an option to manage domains and URLs that the application should not redirect to, whether we get an option to upload a text file with domains or just an interface we can plug custom filters into.

[Csardelacal]

TL;DR: My point was that having the option to filter URL that the server redirects to is an important safety measure. I am not trying to sell anyone on a specific vendor.

[acelaya]

You are approaching it wrong in the first place.

You should prevent malicious URLs from being shortened at all. Creating the short URL just to prevent a redirect later on, would be a really bad experience for users.

Instead, you should error out when trying to create it. If Shlink had support for something like this, that's how it would work.

That's why I said you could add this logic yourself. You could query the service you prefer, and only allow for the URL to be shortened (as in, Shlink to be called) if it matches whatever previous logic you want. If it doesn't, you don't even invoke Shlink endpoints, so you don't need Shlink itself to support it.

[Csardelacal]

Again, this implies that at the time of creating the link I already know that it's malicious. Which is not the case.

1. Bad URL are discovered after they are submitted. Which implies that malicious redirects are kept when they should no longer work.
2. Domains get abandoned and are often used to create malicious / spammy redirects years after they were reputable locations.

The application submitting a link to the shortener would also be required to keep track of and regularly monitor the locations it submitted. Which becomes a gargantuan task if we're talking about user submitted content.

It's simply not true that a link can be verified once and permanently assumed to be safe. Here's an example of how Twitter does this: https://help.twitter.com/en/using-twitter/url-shortener

Twitter does mention that one of the three reasons they have t.co at all is to provide this filtering. Twitter does block you from creating new links that are potentially dangerous (obviously) but will also warn you when you click on a bad link, which is a behavior I would expect.

[acelaya]

Those are actually good points. I didn't think about that.

I have created a ticket to track this feature request. However, there's a couple of other things that have higher priority, so it will still take a while to even start considering it.

Also, it needs to be very carefully planned to prevent it from becoming a maintainability hell and a source of hard to chase bugs.

You can find the ticket here shlinkio/shlink#1305

Feel free to leave comments there.