v0.0.0: guru-snap, bridges-eligibility-manual, ca-edd-web

Initial release

Chat engines

This release contains three chat engines:

• guru-snap: A prototype that uses content from Guru cards exported as JSON
• bridges-eligibility-manual: A prototype that uses content from Bridges Eligibility Manual PDFs
• ca-edd-web: A chatbot that uses content found on edd.ca.gov

You may wish to deploy this release or run it locally if you need to run guru-snap or bridges-eligibility-manual, as these prototypes will likely be deprecated in a future release.

What's Changed

• feat: Install Application-Flask template by @KevinJBoyer in #1
• DST-257: install infra temp by @ccheng26 in #3
• feat: Add Chainlit by @KevinJBoyer in #2
• Remove Flask and example database models by @KevinJBoyer in #4
• feat: Add MockSentenceTransformer by @KevinJBoyer in #5
• feat: Add LiteLLM by @KevinJBoyer in #6
• DST-263: chainlit healthcheck by @ccheng26 in #7
• DST-260: configure env for pgvector by @ccheng26 in #8
• feat: Add Document and Chunk models by @KevinJBoyer in #9
• DST-258: deploy infra by @ccheng26 in #10
• DST-258: deploy infra- enable tests by @ccheng26 in #11
• feat: Ingest Guru Cards by @KevinJBoyer in #13
• DST-271 feat: generate llm result by @ccheng26 in #12
• fix import by @ccheng26 in #14
• feat: Augment response with retrieved cards by @KevinJBoyer in #15
• feat: Show accordions with cards by @KevinJBoyer in #16
• feat: Require login by @KevinJBoyer in #17
• fix: Pin Terraform version for deploys and migrations by @KevinJBoyer in #18
• perf: Lower database auto-scaling settings by @KevinJBoyer in #19
• feat: add ollama call to get_models by @ccheng26 in #20
• feat: productionize uvicorn by @ccheng26 in #22
• fix: Conditionally import ollama by @yoomlam in #23
• feat: Use URL query 'engine' parameter to set chatbot's configuration by @yoomlam in #25
• fix: set uvicorn worker to 1 by @ccheng26 in #26
• feat: Enable ingest of distinct datasets by @yoomlam in #27
• feat: Set default chat engine to 'guru-snap' by @yoomlam in #28
• feat: Retrieve from database based on filters by @yoomlam in #29
• feat: add similarity score to accordion value by @ccheng26 in #30
• refactor: Use generalized ChunkWithScore by @yoomlam in #32
• refactor: Move db_session and embedding_model args into AppConfig by @yoomlam in #33
• refactor: Remove extraneous MockAppConfig and monkeypatch parameter in tests by @yoomlam in #35
• feat: Add ingest-policy-pdfs command to print out PDF file list by @ccheng26 in #31
• feat: Normalize similarity scores of retrieved Guru cards by @yoomlam in #36
• fix: retrieve build date and service env by @ccheng26 in #34
• feat: Add retrieval and docs_shown thresholds by @yoomlam in #37
• feat: Add LLM selection in Chainlit by @yoomlam in #38
• feat: Use URL query params to set initial chat settings by @yoomlam in #39
• feat: chunk and store BEM pdf by @ccheng26 in #40
• fix: Commit after ingesting Guru cards by @KevinJBoyer in #42
• feat: Enable BEM Chatbot by @KevinJBoyer in #41
• feat:add branding logos by @ccheng26 in #43
• feat: Make ingestion drop dataset if already exists by @KevinJBoyer in #44
• feat: add accordion chunk citations by @ccheng26 in #46
• fix: Rename docs_shown_* variable to chunks_shown_* by @yoomlam in #47
• bug: fix accordion by @ccheng26 in #48
• fix: Read titles from PDF metadata by @KevinJBoyer in #49
• feature: Log metadata for retrieved chunks by @KevinJBoyer in #50
• bug: citation duplicates and accordion overflow by @ccheng26 in #51
• feat: Link to BEM documents by @KevinJBoyer in #52
• feat: Support Jupyter notebooks by @KevinJBoyer in #53
• Jupyter notebook: exploration into pdfminer.six capabilities by @yoomlam in #54
• Jupyter notebook: Investigate unstructured for parsing semantics from PDFs by @KevinJBoyer in #55
• feat: Add extract_outline() PDF utility by @yoomlam in #57
• feat: Group markdown text list items by @KevinJBoyer in #56
• fix: Merge list items only if they have the same heading by @yoomlam in #61
• feat: BEM ingest skeleton by @yoomlam in #59
• feat: Convert list items to markdown_texts by @yoomlam in #60
• DST-401: inline citations by @ccheng26 in #58
• feat: update db schema for chunk by @ccheng26 in #62
• feat: Utility to extract bolded text from PDFs by @yoomlam in #63
• feat: Associate and apply stylings to create bolded markdown by @yoomlam in #64
• feat: Format links as markdown by @yoomlam in #65
• feat: Merges texts that are split across consecutive pages by @yoomlam in #66
• test: Use 707.pdf to test ingest_bem_pdfs.py by @yoomlam in #70
• feat: Save BEM JSON chunks to S3 by @KevinJBoyer in #69
• feat: enrich text using unstructured data by @ccheng26 in #68
• fix: Add heuristics and fixes to improve BEM pdf parsing by @yoomlam in #71
• feat: Update citation UI by @KevinJBoyer in #73
• feat: Split long paragraphs and lists into chunks by @yoomlam in #72
• feat: add dash formatting to list items by @ccheng26 in #74
• feat: add ellipses to start/end of text for chunks by @ccheng26 in #75
• package update: format files and update black by @ccheng26 in #77
• feat: add headings to context by @ccheng26 in #76
• CI: Post test coverage report to PR by @yoomlam in #78
• feat: Group text with the same heading by @ccheng26 in #79
• fix: Address BEM PDF ingestion error cases with bigger chunks by @yoomlam in #80
• feat: Sub-chunk citations by @KevinJBoyer in #81
• fix: Prevent citations from being rendered inline by Chainlit by @KevinJBoyer in #82
• fix ci: Set test coverage thresholds by @yoomlam in #84
• fix: Retry waiting for service to be stable by @KevinJBoyer in #86
• fix: Typo in prompt by @KevinJBoyer in #88
• doc: Add data ingestion section by @ccheng26 in #87
• fix: Format markdown to HTML to avoid Chainlit reformatting it by @yoomlam in #90
• ci: Use python:3.12-slim for Docker image by @yoomlam in #91
• feat: Add Scrapy for web scraping edd.ca.gov by @yoomlam in #89
• feat: Group citations from the same document into one accordion by @ccheng26 in #93
• refactor: Citations, subsections, chunks by @yoomlam in #92
• feat: Add system prompt as a configurable setting by @yoomlam in #95
• fix: Add heading and page link to each citation by @yoomlam in #96
• feat: Add engine for CA EDD web by @KevinJBoyer in #94
• refactor: Rename ChunkWithSubsection to Subsection; add terminology by @yoomlam in #97
• feat: Make temperature configurable and set default to 0.0 by @KevinJBoyer in #99
• fix: Resolve incomplete URLs by @yoomlam in #98
• perf: Set default sources to 50 for EDD bot by @KevinJBoyer in #101
• fix: Exclude EDD news-related webpages by @yoomlam in #100
• fix: Exclude WSIN webpages by @yoomlam in #104
• fix: Set CA EDD Web as default engine by @KevinJBoyer in #106
• fix: Add ingest_util.tokenize() to account for padding tokens by @yoomlam in #102
• refactor: Move add_embeddings() to ingest_utils.py by @yoomlam in #103
• feat: Add chunk.headings and include in embeddings by @yoomlam in #105
• feat: add chat history by @ccheng26 in #108
• feat: Split lists and tables markdown into chunks by @yoomlam in #107
• fix: Rejoin headings with the subsequent first paragraph by @yoomlam in #110
• fix: Render citation headings HTML on same line as div by @KevinJBoyer in #112
• fix: Ignore all EDD social-media-toolkit webpages by @yoomlam in #113
• feat: Update prompt to include safeguards by @KevinJBoyer in #111
• perf: Increase CPU and memory of ECS tasks by @KevinJBoyer in #114
• fix: Citations UI: Show single arrow for missing middle headings by @yoomlam in #115

Full Changelog: https://github.com/navapbc/labs-decision-support-tool/commits/v0.0.0