can anyone provide sample configuration files to enable searching markdown files

[parimalpuranik]

I want to enable khoj to search Obsidian markdown files and I tried to configure the UI using *.md ( I didn't know better) and when I tried to run khoj from command line I ran into following error:

PS C:\Users%USER%> C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe
--- Logging error ---
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\logging_init_.py", line 1101, in emit
stream.write(msg + self.terminator)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f318' in position 0: character maps to
Call stack:
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
sys.exit(run())
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 64, in run
logger.info("🌘 Starting Khoj")
Message: '🌘 Starting Khoj'
Arguments: ()
[08:34:12] INFO 🌘 Starting Khoj main.py:64
--- Logging error ---
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\logging_init.py", line 1101, in emit
stream.write(msg + self.terminator)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f4ac' in position 0: character maps to
Call stack:
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
sys.exit(run())
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 88, in run
configure_server(args, required=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 44, in configure_server
state.processor_config = configure_processor(args.config.processor)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 169, in configure_processor
logger.info("💬 Setting up conversation processor")
Message: '💬 Setting up conversation processor'
Arguments: ()
INFO 💬 Setting up conversation processor configure.py:169
--- Logging error ---
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\logging_init.py", line 1101, in emit
stream.write(msg + self.terminator)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f48e' in position 0: character maps to
Call stack:
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
sys.exit(run())
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 88, in run
configure_server(args, required=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 49, in configure_server
state.model = configure_search(state.model, state.config, args.regenerate)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 112, in configure_search
logger.info("💎 Setting up search for markdown notes")
Message: '💎 Setting up search for markdown notes'
Arguments: ()
INFO 💎 Setting up search for markdown notes configure.py:112
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 88, in run
configure_server(args, required=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 49, in configure_server
state.model = configure_search(state.model, state.config, args.regenerate)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 114, in configure_search
model.markdown_search = text_search.setup(
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\search_type\text_search.py", line 180, in setup
entries_with_indices = text_to_jsonl(config).process(previous_entries)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\processor\markdown\markdown_to_jsonl.py", line 41, in process
MarkdownToJsonl.extract_markdown_entries(markdown_files)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\processor\markdown\markdown_to_jsonl.py", line 110, in extract_markdown_entries
with open(markdown_file, "r", encoding="utf8") as f:
OSError: [Errno 22] Invalid argument: 'C:\Users\%USER%\.md'
PS C:\Users%USER%>

I even tried autoconfigure from obsidian but even that didnt work. Perhaps if I had a sample configuration file to go by I could try using that

[debanjum]

Hey @parimalpuranik, I've just fixed these issues today and released khoj version 0.6.1 with these fixes.

You should:

1. Update your khoj server with pip install --upgrade khoj-assistant and restart it (with khoj --no-gui)
2. Update your khoj obsidian plugin (from within Obsidian Settings > Community plugins > Khoj > Update). This should auto-configure your khoj server
3. Wait for your notes to get indexed by the server and now try Khoj search (and/or Khoj chat) from within Obsidian

Let me know if that fixes the issues you mentioned?

As for a sample configs, see khoj_docker.yml or khoj_sample.yml

[debanjum]

Also if you've messed around with your khoj configuration (and it's in a bad state), delete the khoj.yml file at C:\Users<your-windows-username>.khoj\khoj.yml`. Then restart the khoj server and the re-install/re-enable the khoj obsidian plugin to auto-configure khoj server

[parimalpuranik]

I deleted the khoj.yml file and reinstalled / re-enabled the upgraded 0.6.1 plugin and restarted the khoj server. I had the auto-configure option enabled and indexed the notes and I think I have restarted the server a couple of times but I dont see a khoj.yml file in the directory you mention.

PS C:\Users%USER%> C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe --no-gui
[12:43:49] INFO 🌘 Starting Khoj main.py:64 WARNING Khoj is not configured. configure.py:36 Configure it via khoj GUI, plugins or by editing C:\Users%USER%.khoj\khoj.yml.
INFO 🌖 Khoj is ready to use main.py:135[12:43:50] INFO Started server process [11164] server.py:75 INFO Waiting for application startup. on.py:45 INFO Application startup complete. on.py:59 INFO Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) server.py:206

When I tried to run a search I got the familiar check khoj backend is running message.

[parimalpuranik]

Some more error messages :

Exception in thread Thread-38:
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1009, in bootstrap_inner
self.run()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1358, in run
self.function(*self.args, **self.kwargs)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 147, in poll_task_scheduler
schedule.run_pending()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init.py", line 780, in run_pending
default_scheduler.run_pending()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init_.py", line 100, in run_pending
self.run_job(job)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init.py", line 172, in run_job
ret = job.run()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init.py", line 661, in run
ret = self.job_func()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 68, in update_search_index
state.model = configure_search(state.model, state.config, regenerate=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 87, in configure_search
if (t == state.SearchType.Org or t == None) and config.content_type.org:
AttributeError: 'NoneType' object has no attribute 'content_type'

[parimalpuranik]

Thanks for your help, I have khoj working on my setup now. I will try running the chat part next.

[debanjum]

Ah, that's great! Do you know what you did to fix it?

1. Not sure why you saw the khoj backend not running error.
2. But you'd have to re-enable/re-install khoj obsidian (or just obsidian) to auto-configure khoj backend.
3. You'd have seen the AttributeError: 'NoneType' object has no attribute 'content_type' when you'd have tried to search as the khoj backend wasn't configured or hadn't finished indexing your notes

[parimalpuranik]

I think re-enabling / reinstalling khoj obsidian did the trick but I don't remember the full sequence of actions I have taken.

[debanjum]

Ah, that's unfortunate but not an issue. Great you've got Khoj search finally working. Hopefully getting Khoj chat to work shouldn't be too problematic 🤞🏾