Help with enabling and running search on mkdocs with 100,000 pages

[lshalon]

Right now, when I enable search on mkdocs, it tries to initialize the search live on the website and ends up crashing the page. I'd like to have the bare minimum - ideally something as simple as string search. Is there a better way to configure the search functionality?

When running, the memory object that continuously increases in size until things crash is: search.f8cc74c7.min.js

site_name: Knowledge
docs_dir: output          # where your Markdown lives
site_dir: site            # build target

theme:
  name: material
  language: en
 
  # Obsidian-like typefaces
  font:
    # macOS, iOS, Windows, Linux & fallback Helvetica
    text: "system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif"

    # Monospace stack that starts with SF Mono on macOS, Menlo otherwise
    code: "SFMono-Regular, Menlo, Consolas, 'Liberation Mono', monospace"

  # Instant navigation + Obsidian-style search UX
  features:
    - navigation.instant   # fetch pages by XHR, no full reload
    - navigation.top       # keep sidebar header visible
    - search.highlight     # highlight matches in page   :contentReference[oaicite:0]{index=0}
    - search.suggest       # type-ahead suggestions
    - header.autohide      # hide top bar on scroll      :contentReference[oaicite:1]{index=1}

plugins:
  - search:
      indexing: sections      # or titles
      prebuild_index: true    # build once, ship ready-made JSON
      worker: true            # search runs in Web-Worker (no UI jank)
  - redirects:
      redirect_maps:
        index.md: organization/home.md     # <── root → /organization/

nav:
  - Organizations:
      - Home: organization/home.md
  - People:
      - Home: person/home.md
  - Domain Groups:
      - Home: domain_groups/home.md

# Extra styling – make sure the path exists inside docs_dir
extra_css:
  - css/obsidian.css

extra_javascript:
  - js/org-header.js
  # - js/external-links.js

use_directory_urls: false

validation:
  absolute_links: ignore       # or 'error' with mkdocs build --strict

[squidfunk]

Thanks for asking. I think you'll be running into other problems trying to build 100.000 pages with MkDocs right now – MkDocs is not built for that. However, we're working on the foundation of Material for MkDocs right now, and search is one of the central topics. This will also include searching corpora with 100.000 pages or more, which will not be possible in the browser anymore given the amount of data that would need to be transmitted. A solution will likely include deploying an index to an edge function or server.

You can watch this issue for updates: #6307

[squidfunk]

Also could you maybe share what you're trying to build? 100.000 pages is quite a lot, so we'd be interested in learning more about your use case, as there's only a very small set of static site generators being able to handle that many pages.

[lshalon]

Thanks for the note. We are using it to host a directory of people. Actually, everything works great so far except for search. It would be amazing to offload some processes if you end up building such a platform. For the time being, what do you recommend we do? We can offload the search functionality to our own API or Agoda. Or, it would be ideal if we could replace it with local search but only on the headers of pages (basic string search is fine). Also, right now, ripping out search and replacing it with our own functionality (I tried doing this yesterday) is super hard - and I was ultimately unsuccessful.

…

On Wed, Apr 30, 2025 at 5:04 AM Martin Donath ***@***.***> wrote: Also could you maybe share what you're trying to build? 100.000 pages is quite a lot, so we'd be interested in learning more about your use case, as there's only a very small set of static site generators being able to handle that many pages. — Reply to this email directly, view it on GitHub <#8181 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJWHFL4SQLFRLHLXBMDGRL24CGZZAVCNFSM6AAAAAB4EMYP2KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOJZGA2TINQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[squidfunk]

I think right now, forking + patching is the only way to go. And yes, we'll definitely build such a service.

[lshalon]

Great, would love to work as a design partner for it and try to help expedite the roadmap.

[lshalon]

Another thought - if I intelligently exclude certain pages using https://squidfunk.github.io/mkdocs-material/blog/2021/09/26/excluding-content-from-search/#excluding-sections-search_indexjson, will that avoid the failure-point?

[kamilkrzyskow]

Depends on what you mean by "intelligently". Since it's used as a directory of people, then I doubt you can exclude that many entries.
But perhaps you could post-process the search_index file and split it into multiple files. Like for example alphabetically.
Then you'd have to inject some logic to handle choosing what search_index file to load at a given moment.

Reference:

• use different search_index.json dependent on the selection on the page #8165 (comment)

[kamilkrzyskow]

What is the size in MB of the search_json file? Have you tried to post-process and trim the final result of any HTML tags to make an ultra minimized version?

Except of forking and modifying the underlying TypeScript of the search I also see an option to simply remove the search modal altogether in the header.html and setup a separate search.md page with custom search backend.

As for what backend would be the fastest idk, maybe FastAPI with SQLite given the amount of records.
Also as stated above MkDocs isn't really made for processing that many pages, as there is absolutely no caching for Markdown content.
So on each rebuild it processes all pages, have you tried to use the pages with Hugo? I have a faint memory that it does cache the contents to avoid wasting cycles