Skip to content

Release 3.3.0

Compare
Choose a tag to compare
@thiswillbeyourgithub thiswillbeyourgithub released this 19 Jul 12:44
· 24 commits to main since this release

What's new

This release focuses on adding DuckDuckGo web search capabilities and introducing binary embeddings support for more efficient vector storage.

✨ New Features

DuckDuckGo Web Search Integration

  • [372fe57] Add DuckDuckGo search support with URL extraction and metadata
  • [273195e] Support wdoc wdb "your query" shorthand for web search
  • [03bfe08] Add DuckDuckGo search tests and documentation

Binary Embeddings Support

  • [c528bad] Add support for binary embeddings with 8x memory reduction
  • [8f65197] Enable FAISS vectorstore compression by default
  • [37ebd97] Create CompressedFAISS subclass with zlib compression

🐛 Bug Fixes

Core Functionality

  • [0d72efd] Fix wrong decorator used for load_one_doc
  • [edcf671] Fix ddg_region type (str not int)
  • [66ab177] Fix type hints for ddg_safesearch and loading_failure
  • [957936c] Use keyword arguments instead of fire when calling wdoc

Testing Environment

  • [d3de58e] Fix piped input/output handling in pytest environment
  • [42ff516] Prevent pipe usage in pytest environment
  • [c78dc0b] Add pytest environment detection

🧪 Testing Improvements

  • [1b09996] Fix the run_all_test script
  • [8ed1d0c] Add comprehensive DuckDuckGo search functionality tests
  • [b184177] Split CLI tests into separate test_cli.py file
  • [9d7fe9c] Split parsing tests into separate test_parsing.py file
  • [12b012d] Move vector store tests to dedicated test file

📚 Documentation

  • [d7d6b04] Explain how to run tests in README
  • [dc15001] Clarify how to disable parallel processing
  • [df4b79f] Document debug mode's effect on loading_failure default
  • [1832299] Add shell examples for DuckDuckGo usage

🔧 Enhancements

CLI/UX Improvements

  • [7e994a6] Rename parse_file function to parse_doc
  • [4aa247e] Re-ask for input when empty query provided in CLI
  • [57d5d5f] Fix Fire's pager issue in CLI

Performance THISISANAMPERSAND Reliability

  • [68d4c75] Bump LiteLLM to latest version for improved startup time
  • [ab9c5e9] Add parallel processing option for Whisper audio splits
  • [6b13044] Add loop counter and crash protection for recursive file processing

🔄 Version Update

  • [6435133] Bump version from 3.2.5 → 3.3.0

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

tests/run_all_tests.sh

README.md

tests/test_cli.py

wdoc/utils/loaders.py

wdoc/docs/help.md

wdoc/utils/batch_file_loader.py

wdoc/utils/misc.py

wdoc/docs/help.md

wdoc/utils/misc.py

wdoc/docs/help.md

tests/test_cli.py

tests/test_cli.py

tests/run_all_tests.sh

wdoc/utils/env.py
wdoc/utils/misc.py

tests/conftest.py

wdoc/utils/misc.py

tests/test_wdoc.py
wdoc/utils/env.py
wdoc/utils/misc.py

tests/test_cli.py

tests/test_cli.py

tests/test_cli.py

wdoc/main.py

wdoc/main.py

wdoc/main.py

wdoc/main.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

README.md
tests/test_cli.py
wdoc/docs/examples.md

tests/run_all_tests.sh

tests/run_all_tests.sh

tests/test_cli.py

tests/test_cli.py

tests/test_cli.py

tests/test_cli.py

wdoc/docs/examples.md

wdoc/docs/examples.md

tests/run_all_tests.sh

README.md

tests/run_all_tests.sh

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

setup.py

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

tests/run_all_tests.sh

tests/run_all_tests.sh

setup.py

wdoc/docs/help.md

wdoc/docs/help.md

wdoc/docs/help.md

README.md

README.md

README.md

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

README.md

tests/run_all_tests.sh

README.md

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

wdoc/main.py

wdoc/utils/misc.py

wdoc/main.py

wdoc/wdoc.py

wdoc/main.py

wdoc/wdoc.py

wdoc/utils/interact.py

ddg_test.py

ddg_test.py

wdoc/utils/loaders.py

wdoc/utils/batch_file_loader.py

wdoc/utils/misc.py

tests/run_all_tests.sh

README.md
setup.py
tests/run_all_tests.sh

setup.py
wdoc/utils/batch_file_loader.py
wdoc/utils/misc.py

tests/run_all_tests.sh

tests/run_all_tests.sh

  • [b184177] by @thiswillbeyourgithub, 4 days ago:
    Based on the changes you've made, I'll generate a concise commit message that captures the essence of the refactoring:
    refactor: split CLI tests into separate test_cli.py file

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) aider@aider.chat

tests/test_cli.py
tests/test_parsing.py
tests/test_wdoc.py

tests/test_parsing.py
tests/test_wdoc.py

tests/test_vectorstores.py
tests/test_wdoc.py

tests/test_wdoc.py

tests/test_wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

tests/test_wdoc.py
wdoc/utils/customs/binary_faiss_vectorstore.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/test_wdoc.py
wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

tests/test_wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/test_wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

tests/test_wdoc.py

tests/test_wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/run_all_tests.sh

setup.py

setup.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/run_all_tests.sh

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

setup.py

wdoc/utils/customs/binary_faiss_vectorstore.py

setup.py

wdoc/utils/customs/binary_faiss_vectorstore.py

setup.py

wdoc/utils/env.py

setup.py
wdoc/utils/llm.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/query.py

setup.py

wdoc/utils/env.py

tests/test_wdoc.py

wdoc/utils/misc.py

wdoc/utils/loaders.py

tests/test_wdoc.py

tests/test_wdoc.py

README.md
setup.py
tests/run_all_tests.sh

setup.py

README.md

wdoc/utils/interact.py

wdoc/wdoc.py

wdoc/utils/interact.py

wdoc/wdoc.py

wdoc/utils/customs/litellm_embeddings.py

wdoc/docs/help.md
wdoc/utils/embeddings.py
wdoc/utils/env.py

wdoc/docs/help.md
wdoc/utils/customs/binary_faiss_vectorstore.py

README.md
wdoc/docs/help.md
wdoc/utils/embeddings.py
wdoc/utils/env.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/docs/help.md
wdoc/utils/env.py

wdoc/wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/docs/help.md

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/embeddings.py

README.md

tests/test_wdoc.py

wdoc/docs/help.md
wdoc/utils/customs/binary_faiss_vectorstore.py
wdoc/utils/embeddings.py
wdoc/utils/env.py

wdoc/utils/embeddings.py

wdoc/docs/help.md

wdoc/utils/embeddings.py

wdoc/docs/help.md

LICENSE

shell_completions/wdoc_completion.bash
shell_completions/wdoc_completion.fish
shell_completions/wdoc_completion.zsh

wdoc/main.py

wdoc/main.py

README.md

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/wdoc.py

wdoc/utils/loaders.py

wdoc/docs/help.md
wdoc/utils/env.py
wdoc/utils/loaders.py

wdoc/utils/env.py

setup.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

MANIFEST.in
README.md
docs/source/index.rst
docs/source/parse_doc_help.md
docs/source/parse_doc_help.rst
docs/source/parse_file_help.md
docs/source/single_page_doc.rst
tests/test_wdoc.py
wdoc/main.py
wdoc/docs/help.md
wdoc/docs/parse_doc_help.md
wdoc/utils/logger.py
wdoc/wdoc.py

setup.py

wdoc/wdoc.py

wdoc/wdoc.py

wdoc/docs/help.md
wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/errors.py
wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/misc.py

wdoc/utils/loaders.py

wdoc/main.py

wdoc/docs/parse_file_help.md
wdoc/wdoc.py

wdoc/docs/help.md

wdoc/main.py
wdoc/docs/parse_file_help.md
wdoc/wdoc.py

wdoc/docs/parse_file_help.md
wdoc/wdoc.py

wdoc/utils/batch_file_loader.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/batch_file_loader.py
wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

  • [7541b97] by @thiswillbeyourgithub (aider), 6 weeks ago:
    refactor: Improve argument handling in load_one_doc with runtime and user arg separation

wdoc/utils/loaders.py

wdoc/utils/misc.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py
wdoc/utils/misc.py

wdoc/utils/misc.py

wdoc/utils/loaders.py

wdoc/wdoc.py

wdoc/utils/errors.py
wdoc/utils/loaders.py

wdoc/docs/help.md
wdoc/utils/env.py
wdoc/utils/loaders.py

wdoc/wdoc.py

wdoc/wdoc.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/misc.py
wdoc/wdoc.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py