Add filters similar to katana

[dogancanbakir]

@ehsandeep @dogancanbakir

Please consider adding the below features as well for more flexible output and to control junk results

FILTER:
   -mr, -match-regex string[]       regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[]      regex or list of regex to filter on output url (cli, file)
   -f, -field string                field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -em, -extension-match string[]   match output for given extension (eg, -em php,html,js)
   -ef, -extension-filter string[]  filter output for given extension (eg, -ef png,css)

Originally posted by @zy9ard3 in #17

[dogancanbakir]

@zy9ard3, we have these flags available. Could you review them and let us know if you're interested or if you'd like to see any additions?

FILTER:
   -m, -match string[]       URLs or list to match (file or comma-separated)
   -f, -filter string[]      URLs or list to filter (file or comma-separated)

[zy9ard3]

@dogancanbakir

I checked with these and it worked well 👍 like the above requested flags

But there is a small issue on this ;

I want to filter all the .js files from the output and when I used it on -f, it also filtered all the .json endpoints too

So it would be better if we have separate flag for handling extensions more better

FILTER:
   -em, -extension-match string[]   match output for given extension (eg, -em php,html,js)
   -ef, -extension-filter string[]  filter output for given extension (eg, -ef png,css)

[dogancanbakir]

@zy9ard3, could you provide an example? That would really help me reproduce the issue.

[Spidle]

I would like to add in here that I'm looking for a similar but different feature. The old katana --passive command would only return URLs that were a part of the FQDN. urlfinder will return urls that contain a huge number of subdomains, which is not what I want. I think I can write a filter query to get what I'm looking for, but it would be nice if this was a feature out of the box.

Expected behavior:

urlfinder -d example.com

example.com/page.html
example.com/page2.htm
example.com/index.php

Observed behavior

www.example.com/page.html
www.example.com/index.php
dev.example.com/index.php
www2.example.com/home.html

In the observed behavior its including the www subdomain even though I didnt specify it as well as other subdomains.

EDIT: I did find a filter that does what I want. However, I still wonder if this should be default behavior.

-m "^https://example.com"

[dogancanbakir]

@Spidle Could you checkt the -fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn") option and tell us if this meets your needs?

[zy9ard3]

@dogancanbakir

@zy9ard3, could you provide an example? That would really help me reproduce the issue.

Here is the below example ;

From the above screenshot, you can see that there are some .json endpoints on the output, but when I try to filter out .js files, the .json endpoints are also getting filtered out

You can verify this by ;

• urlfinder -d hackerone.com -m ".json"
• urlfinder -d hackerone.com -f ".js"|grep json

So it would be better and helpful if we have separate flag like katana for handling extensions more better and get efficient results

FILTER:
   -em, -extension-match string[]   match output for given extension (eg, -em php,html,js)
   -ef, -extension-filter string[]  filter output for given extension (eg, -ef png,css)