PageId property in context object causing crawling errors and exposing project structure

[yeraydavidrodriguez]

Hello all.

We have developed a corporate website that serves content from a headless CMS using Vike.
However, examining our Google's search console reports, we saw some crawling errors because Google tried to index the URL domain/src/pages/@slug.
We think that the cause for this is that Vike is adding the following attribute to its vike_pageContext object:
"pageId":"/src/pages/@slug"
This makes Google (and other robots) think that is a route and tries to index it. But also, it is exposing an internal of the project we would like to keep hidden (the file structure of the project)
We tried to overwrite that property on the onBeforeRender function, but it generates an error as is required by the Vike router.

Is there some way Vike can be configured so that the PageId property is excluded from the context without affecting the router, or at least having it obfuscated?

[brillout]

That's quite unexpected: I expect bots to only crawl <a> links. I don't see why they would crawl text or script content. Maybe it's some kind of user-land issue?

also, it is exposing an internal of the project we would like to keep hidden (the file structure of the project)

I can't think of a reason why it would be an issue. How is this an issue for you?

[brillout]

I think the best would be to have a more references about the issue.

We did a thorough research

Did you find more blog posts and/or reddit discussions about it?

The thread you referred to earlier (https://www.reddit.com/r/TechSEO/comments/1dpowbk/comment/lakniy8/) isn't a signal enough fo me to believe that it's a general issue. The best would be an official statement from Google about it, but enough trustable references would be enough.

If I'd know that it's indeed an issue for all Vike users (and not something peculiar about your app), then I'd be already working on fixing this as we speak.

The thing is that we don't really know what the root issue is. Google crawling JSON data to find links is really weird and hard to believe.

As for obfuscating your project structure, see brillout/telefunc#159.

[yeraydavidrodriguez]

Hello

Thanks for your answer. We understand that you cannot address all particular concerns of Vike users.

If we get more technical details on this topic we will reach to you again. Meanwhile, we will explore our own ways to overcome the issues we are having.

Thanks

I think the best would be to have a more references about the issue.

We did a thorough research

Did you find more blog posts and/or reddit discussions about it?

The thread you referred to earlier (https://www.reddit.com/r/TechSEO/comments/1dpowbk/comment/lakniy8/) isn't a signal enough fo me to believe that it's a general issue. The best would be an official statement from Google about it, but enough trustable references would be enough.

If I'd know that it's indeed an issue for all Vike users (and not something peculiar about your app), then I'd be already working on fixing this as we speak.

The thing is that we don't really know what the root issue is. Google crawling JSON data to find links is really weird and hard to believe.

As for obfuscating your project structure, see brillout/telefunc#159.

[brillout]

👍 Keep us updated.

we will explore our own ways to overcome the issues we are having.

Also keep us updated on that. The more we know about the issue, the better.

Looking forward to make it work.

[brillout]

Conversation about Google crawling JSON data: #2541 (reply in thread).

[brillout]

Fixed: #2541 (reply in thread).