[Github] Allow ingestion of additional documentation file formats such as .adoc and .mdx

[keiransteele-phocas]

Problem Description

We would like to ingest documentation for tooling that we use to then be able to search, chat with and integrate into the AI assistant. Generally the documentation is available in Github but some is not in formats that the connector is able to ingest.

This is two examples of documentation available in Github that the connector can't ingest:
Keycloak documentation in asciidoc (.adoc) format: https://github.com/keycloak/keycloak/tree/a8225655cfc1d4d01d6cbeea70cf45e4958e36e8/docs/guides/getting-started
Octopus Deploy documentation in MDX (.mdx) format: https://github.com/OctopusDeploy/docs/blob/main/src/pages/docs.mdx

Given the ease Elastic has with ingesting and searching raw text from Confluence and Markdown ingestion from Github is a similar blob of text, I'm unable to see why there is such strict limitation on the extensions that are being ingested from Github

Proposed Solution

Extend the list of supported extensions to include additional formats that documentation is commonly written in.

Current configuration:

SUPPORTED_EXTENSION = [".markdown", ".md", ".rst"]

Proposed configuration or similar:

SUPPORTED_EXTENSION = [".markdown", ".md", ".rst", ".adoc", ".mdx"]

A more complex solution would be to expose the list to the user via the advanced sync rules or other configuration.

Alternatives

I have considered using the web crawler but ideally would like to use the connector given that it's available and this seems to be one of it's primary use cases. https://github.com/elastic/crawler

Additional Context

I would be happy to have a go at a PR if the proposal to change the list is good enough.

[seanstory]

hi @keiransteele-phocas , thanks for filing!

I would be happy to have a go at a PR if the proposal to change the list is good enough.

You're absolutely welcome to give it a shot! However...

I'm unable to see why there is such strict limitation on the extensions that are being ingested from Github

For github, we wanted to index "documentation" but not "code". However, most all our connectors still allow only a subset of extensions. See usages of:

connectors/connectors/utils.py

Lines 47 to 79 in 81e7eaf

	TIKA_SUPPORTED_FILETYPES = [
	".txt",
	".py",
	".rst",
	".html",
	".markdown",
	".json",
	".xml",
	".csv",
	".md",
	".ppt",
	".rtf",
	".docx",
	".odt",
	".xls",
	".xlsx",
	".rb",
	".paper",
	".sh",
	".pptx",
	".pdf",
	".doc",
	".aspx",
	".xlsb",
	".xlsm",
	".tsv",
	".svg",
	".msg",
	".potx",
	".vsd",
	".vsdx",
	".vsdm",
	]

This is because our connectors don't actually process these file formats themselves, but send the base64-encoded file contents to either the Data Extraction Service or to the Attachment Processor for Elasticsearch Ingest Pipelines. Both of these are powered by Apache Tika, and Tika does not have support for .mdx or .adoc formats.

Note, I requested support for MDX quite a while ago, but it has had 0 traction.

If you modify these lists to send these file formats regardless, you'll end up with them indexed into Elasticsearch still with all their markup. Which isn't necessarily a bad thing! But traditionally has not been our end-goal. So just be aware.

[keiransteele-phocas]

Hi @seanstory

Thanks for additional info, I also saw that a request for .adoc support was made in Tika back in 2015 unfortunately.

I think I will have to find an alternative in the meantime, even if the additional file types are added there is a limitation on the advanced rules that doesn't allow wildcards in the path so I can't limit the sync to just documentation directories with sub directories.