Skip to content

Commit 984d5d4

Browse files
corwintinescorwintines
authored
Merge pull request #10309 from ethereum/docsearch-index-update
DocSearch index update
2 parents 6d2bcf0 + 6b53353 commit 984d5d4

27 files changed

+70
-274
lines changed

‎.github/workflows/docsearch-crawl.yml

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎.github/workflows/docsearchConfig.json

Lines changed: 0 additions & 123 deletions
This file was deleted.

‎.github/workflows/docsearchConfigScript.js

Lines changed: 0 additions & 63 deletions
This file was deleted.

‎docs/site-search.md

Lines changed: 14 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,28 @@
11
# Site search on ethereum.org
22

3-
TL;DR: we use Algolia to implement a site search feature on ethereum.org.
3+
TL;DR: we use Algolia to implement a site search feature on ethereum.org. As an open source project, Algolia has sponsored the crawling and indexing of the entire site.
44

5-
## What do we use Algolia and Docsearch for?
5+
## What do we use Algolia and DocSearch for?
66

7-
Algolia allows us to index the content on ethereum.org and implement a powerful site search tool on ethereum.org. In order to create the index of our content, we use a web crawling tool called Docsearch. Docsearch takes a start_urls of ethereum.org and crawls the site to index the content based on a [docsearchConfig file](https://github.com/ethereum/ethereum-org-website/blob/dev/.github/workflows/docsearchConfig.json).
7+
Algolia allows us to index the content on ethereum.org and implement a powerful site search tool on ethereum.org. In order to create the index of our content, we use a web crawling tool called DocSearch. DocSearch takes a starting URL of ethereum.org and crawls the site to index the content, based on a custom configuration setup held with the service.
88

9-
We kick off the crawling and indexing of ethereum.org through a GitHub Action that triggers on the merge to `master` branch. [View the GitHub Action](https://github.com/ethereum/ethereum-org-website/blob/dev/.github/workflows/docsearch-crawl.yml).
9+
Site crawling and indexing is performed by default on a weekly basis on Friday afternoons. This is performed automatically by Algolia servers, which scrape the entire production site of ethereum.org to build an index. This index is hosted by Algolia for use on the site.
1010

11-
## Docsearch Config
11+
## DocSearch Config
1212

13-
Some important notes about the docsearch config file:
13+
Some important notes about the DocSearch config:
1414

1515
### Configuration
1616

17-
- `index_name` is the name of the algolia index where the generated index will be uploaded to.
18-
- `start_urls` are the urls that the crawler will start from. Some important attributes in the `start_urls` that we use are:
19-
- `lang`: regex path to different languages that the site is translated to that need crawling. Since ethereum.org is translated to 37+ languages, we need to be able to crawl the website in each language for indexing.
20-
- `page_rank`: the rank of pages that breaks ties when multiple query results have the same weight. This weight is derived from the selectors.
21-
- `stop_urls` is used to strip out query parameters in the websites urls. We were running into issues where we were getting duplicate query results due to query parameters making urls unique. Stripping these out solved our deduplication problem.
22-
- selectors are used to specify what the crawler should look for when weighting content for the index.
23-
24-
### Generation
25-
26-
We generate the docsearchConfig.json file using a [script](https://github.com/ethereum/ethereum-org-website/blob/dev/.github/workflows/docsearchConfigScript.js). This allows us to dynamically pull in the languages the websites support from the [translations.json data file](https://github.com/ethereum/ethereum-org-website/blob/dev/src/data/translations.json). Our GitHub action executes this script.
17+
- `indexName` is the name of the Algolia index where the generated index will be uploaded to
18+
- `startUrls` are the urls that the crawler will start from
19+
- Translated pages are automatically faceted for search results based on the `<html lang="">` attribute of each page
20+
- Selectors are used to specify what the crawler should look for when weighting content for the index.
21+
- CheerioAPI can be utilized within the crawler using the `$` selector to manipulate the DOM before indexing each page
22+
- Elements to be ignored are removed before indexing using the CheerioAPI library: `$('selector').remove()`. This includes `aside`, `nav`, `footer` and `style` elements.
23+
- While building pages, semantic naming with the aforementioned elements, i.e. `aside`, will ignore any content contained within. This is beneficial for content that is not directly related to the page content, such as callouts, banners, quiz content, or navigation elements.
2724

2825
## Resources
2926

3027
- [Algolia documentation](https://www.algolia.com/doc/)
31-
- [Docsearch documentation](https://docsearch.algolia.com/docs/what-is-docsearch)
32-
- [Docsearch scraper Docker image](https://hub.docker.com/r/algolia/docsearch-scraper)
28+
- [DocSearch documentation](https://docsearch.algolia.com/docs/what-is-docsearch)

‎src/components/BannerNotification/index.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ const BannerNotification: React.FC<IProps> = ({
1616
<>
1717
{shouldShow && (
1818
<Center
19+
as="aside"
1920
maxW={isLGScreen ? oldTheme.variables.maxPageWidth : "100%"}
2021
w="100%"
2122
py="4"

‎src/components/CallToContribute.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ const CallToContribute: React.FC<IProps> = ({ editPath }) => {
5353

5454
return (
5555
<Flex
56+
as="aside"
5657
bg="ednBackground"
5758
align="center"
5859
mt={8}

‎src/components/Callout.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ const Callout: React.FC<IProps> = ({
3030
...rest
3131
}) => (
3232
<Flex
33+
as="aside"
3334
direction="column"
3435
bgGradient="linear-gradient(
3536
49.21deg,

‎src/components/CalloutBanner.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ const CalloutBanner: React.FC<IProps> = ({
2424
...restProps
2525
}) => (
2626
<Flex
27+
as="aside"
2728
direction={{ base: "column", lg: "row-reverse" }}
2829
bg="layer2Gradient"
2930
p={{ base: 8, sm: 12 }}

‎src/components/EthExchanges/index.tsx

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,11 @@ const EthExchanges = () => {
8686

8787
return (
8888
<Flex flexDir="column" align="center" w="full">
89-
<Heading fontSize={{ base: "2xl", md: "2rem" }} fontWeight={600} lineHeight={1.4}>
89+
<Heading
90+
fontSize={{ base: "2xl", md: "2rem" }}
91+
fontWeight={600}
92+
lineHeight={1.4}
93+
>
9094
<Translation id="page-get-eth-exchanges-header" />
9195
</Heading>
9296
<Text maxW="container.sm" mb={8} lineHeight={1.4} textAlign="center">

‎src/components/Layout.tsx

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ import SideNavMobile from "./SideNavMobile"
1515
import TranslationBanner from "./TranslationBanner"
1616
import TranslationBannerLegal from "./TranslationBannerLegal"
1717
import FeedbackWidget from "./FeedbackWidget"
18-
import { SkipLink, SkipLinkAnchor } from "./SkipLink"
18+
import { SkipLink } from "./SkipLink"
1919

2020
import { ZenModeContext } from "../contexts/ZenModeContext"
2121

@@ -135,8 +135,11 @@ const Layout: React.FC<IProps> = ({
135135
<Nav path={path} />
136136
{shouldShowSideNav && <SideNavMobile path={path} />}
137137
</ZenMode>
138-
<SkipLinkAnchor id="main-content" />
139-
<Flex flexDirection={{ base: "column", lg: "row" }}>
138+
<Flex
139+
flexDirection={{ base: "column", lg: "row" }}
140+
id="main-content"
141+
scrollMarginTop={20}
142+
>
140143
{shouldShowSideNav && (
141144
<ZenMode>
142145
<SideNav path={path} />

0 commit comments

Comments
 (0)