How to Provide a Regex Pattern to Katana to Avoid Entering Specific Links During Crawling

[shsamiei]

To improve the speed of finding the appropriate links, I want to provide a regex pattern to Katana so it can avoid entering certain links with a specified pattern. Initially, I used filter-regex, but I am not sure if Katana is filtering the links only for display or if it is genuinely avoiding entering those links.

Can someone provide a guide on how to check the log of all requests that Katana sends?

[ehsandeep]

@shsamiei You can use below options.

   -cos, -crawl-out-scope string[]  out of scope url regex to be excluded by crawler

[shsamiei]

Thank you for your response. I want to perform a two-step crawl to find product pages of shops. In the first step, I want to crawl with a depth of 2 and without any out-of-scope crawling. Then, my software will proceed to the second step, where it will identify the pattern of product pages (e.g., https://shop.app/product/2343). It will find these links but will not enter the product pages because they don't contain proper links.

The above explanation describes my goal. Is there an option in Katana to avoid entering links that match a specific pattern while still including them in the results?

[ehsandeep]

You can define you scope which limits the scope of crawl and you can use below option to view links for the case you mentioned.

   -do, -display-out-scope          display external endpoint from scoped crawling