Problem with Google crawling

[KevinHansen23]

Hi @Rayn93,

we have the following problem: Google crawls faulty links, all of which originate from FactFinder templates:

We have already added the following rules to robots.txt:

Disallow: */{{
Disallow: /factfinder/result?
Disallow: /fact-finder/proxy

Nevertheless, the URLs are still being crawled. With the exception of record.deeplink and attributes.deeplink, all URLs originate from Suggest, as far as we can tell. The behaviour is described here in the FAQ (How to avoid crawlers indexing unresolved URLs which results in a 404?). However, the data-redirect attributes in the suggest are not parsed.

Things get even stranger with record.Deeplink and attributes.Deeplink. The former only appears in an Slider template (record-list-slider.html.twig), which we do not use at all, and the latter does not even appear anywhere in the source code.

Can you please add the data-redirect functionality to the Suggest and check where the two deeplink attributes come from?

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23

We have already confirmed the problem with data-redirect in suggest component. We will try to add this functionality to suggest.

When it comes to record.deepplink, I have already located and I will fix it soon. It is a bit worse with attributes.deepplink but I will also try to look for where such a link can come from.

[KevinHansen23]

Hi @Rayn93,

Thank you for your, as always, quick feedback. It would be great if you could also integrate the fixes into the SDK version for Shopware 6.6.

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23

I fixed a problem with data-binding for record-list-slider.html.twig. It is already released in v6.5: https://github.com/FACT-Finder-Web-Components/shopware6-plugin/releases/tag/v6.5.0

The data-redirect doesn't work in ff-suggest. Instead you should use deferred attribute resolving: https://web-components.fact-finder.de/api/4.x/DeferredAttributeResolving

[KevinHansen23]

Hi @Rayn93,

fixed a problem with data-binding for record-list-slider.html.twig. It is already released in v6.5: https://github.com/FACT-Finder-Web-Components/shopware6-plugin/releases/tag/v6.5.0

That is excellent. Do you have any idea where the attributes.deeplink could come from?

The data-redirect doesn't work in ff-suggest. Instead you should use deferred attribute resolving: https://web-components.fact-finder.de/api/4.x/DeferredAttributeResolving

Thank you, we will try that out

Kind regards
Kevin

[Rayn93]

@KevinHansen23

Unfortunately I couldn't find the source of the probleme for attributes.deeplink in the SDK. I spoke with WebComponents developer and he said it came from the ff-suggest component. Did you implement any changes on your site for this component?

[KevinHansen23]

Hi @Rayn93,

yes, we overrode suggest.html.twig because the standard integration did not work. Here is an example of our category container:

<div data-container="category" class="text-truncate">
    <h3 class="h5 container-caption pb-2">{{ 'ff.suggest.Categories'|trans|striptags }}</h3>
    <ff-suggest-item class="p-1" type="category">
        <a href="{{ '{{attributes.Url}}' }}" class="d-flex justify-content-between">
            <span class="ff-category">{{ '{{{name}}}' }}</span>
            <span class="badge text-bg-primary text-white" title="{{ '{{$dec hitCount}}' }} {{ "ff.suggest.categoryHitCount"|trans|sw_sanitize }}">{{ '{{$dec hitCount}}' }}</span>
        </a>
    </ff-suggest-item>
</div>

However, there is no {{attributes.deeplink}} in the entire file.

Kind regards
Kevin

[KevinHansen23]

Hi @Rayn93

we have updated to version 6.5.0 and switched to data-defer-href in suggest.html.twig, but unfortunately without success. Google continues to have loads of 404 errors:

This is how the current integration of the categories in suggest.html.twig looks like:

<div data-container="category" class="text-truncate">
    <h3 class="h5 container-caption pb-2">{{ 'ff.suggest.Categories'|trans|striptags }}</h3>
    <ff-suggest-item class="p-1" type="category" data-ffw-defer="href">
        <a data-defer-href="{{ '{{attributes.Url}}' }}" class="d-flex justify-content-between">
            <span class="ff-category">{{ '{{{name}}}' }}</span>
            <span class="badge text-bg-primary text-white" title="{{ '{{$dec hitCount}}' }} {{ "ff.suggest.categoryHitCount"|trans|sw_sanitize }}">{{ '{{$dec hitCount}}' }}</span>
        </a>
    </ff-suggest-item>
</div>

What also seems very strange is that the URL contains the product ID, everything after that is cut off, and then the unresolved variable appears.

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23
Unfortunately, I have no idea what else could be wrong. The issue is more related to the WebC library, so when the lead WebC developer returns from vacation, I'll ask him to check it again.

[KevinHansen23]

Hi @Rayn93,

thanks for the update. We will wait for feedback from the WebComponents team.

[Rayn93]

Hi @KevinHansen23 Thanks for your patience.
@omikron-fp from Web Components developer team has reviewed your banemo.de website (home page, search result page, and category page) and everything looks correct there.

Is Google crawling still an issue? Perhaps it's due to caching and Google still accessing the page before the changes?