Report for test run llmrag

[rkclimate20]

System : Windows 10, Virtual env : python 3.12

output for python -m coverage report -m

Name                                                  Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------
llmrag\chunking\__init__.py                               1      0   100%
llmrag\chunking\text_splitter.py                          8      0   100%
llmrag\cli.py                                            52     52     0%   1-85
llmrag\embeddings\__init__.py                             3      0   100%
llmrag\embeddings\base_embedder.py                        5      1    80%   6
llmrag\embeddings\sentence_transformers_embedder.py      12      0   100%
llmrag\generators\__init__.py                             0      0   100%
llmrag\generators\local_generator.py                      7      3    57%   16, 29-35
llmrag\main.py                                           56     56     0%   1-72
llmrag\models\__init__.py                                 3      0   100%
llmrag\models\base_model.py                               5      1    80%   6
llmrag\models\transformers_model.py                       8      0   100%
llmrag\pipelines\__init__.py                              1      0   100%
llmrag\pipelines\rag_pipeline.py                         21      6    71%   19, 21, 61-64
llmrag\retrievers\__init__.py                            12      7    42%   7-14
llmrag\retrievers\base_vector_store.py                    8      2    75%   6, 10
llmrag\retrievers\chroma_store.py                        42      3    93%   48, 67-68
llmrag\retrievers\faiss_store.py                         27     27     0%   1-31
llmrag\streamlit_app.py                                  29     29     0%   1-53
-----------------------------------------------------------------------------------
TOTAL                                                   300    187    38%

[anudev21]

System : Windows 10, Virtual env : python 3.12

output error for: coverage run --source=llmrag -m unittest discover -s tests

(llmenv) C:\Users\ANUDEV\llmrag>python -m coverage run --source=llmrag -m unittest discover -s tests
.F.Device set to use cpu
..Retrieved: [('Paris is the capital of France.', 0.2878604233264923)]
.Retrieved: [('Paris is the capital of France.', 0.3702658712863922)]
.
======================================================================
FAIL: test_split_html_with_ids (test_chunking.TestChunking.test_split_html_with_ids)
customise this to extract chunks out of YOUR chapter
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\ANUDEV\llmrag\tests\test_chunking.py", line 30, in test_split_html_with_ids
    chunks = self.create_chunks(test_dir, wg, chapter, div_id, p_count)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ANUDEV\llmrag\tests\test_chunking.py", line 50, in create_chunks
    assert chap_html.exists(), f"chapter {chap_html} should exist"
AssertionError: chapter tests\ipcc\wg2\chapter15\html_with_ids.html should exist

----------------------------------------------------------------------
Ran 7 tests in 50.000s

FAILED (failures=1)

[petermr]

OK -

• what is your analysis of the problem?
• can you fix it?

[deepika8mandakala]

Windows-11,env python 3.12.5
WG1 CHAPTER-04
Output for :
coverage run --source=llmrag -m unittest discover -s tests

PS C:\Users\manda\llmrag> coverage run --source=llmrag -m unittest discover -s tests
E.Device set to use cpu
..Retrieved: [('Paris is the capital of France.', 0.2878604233264923)]
.Retrieved: [('Paris is the capital of France.', 0.3702658712863922)]
.
======================================================================
ERROR: test_chunking (unittest.loader._FailedTest.test_chunking)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_chunking
Traceback (most recent call last):
  File "c:\python312\Lib\unittest\loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\python312\Lib\unittest\loader.py", line 339, in _get_module_from_name
    __import__(name)
  File "C:\Users\manda\llmrag\tests\test_chunking.py", line 4, in <module>
    from lxml.html import HTMLParser
ModuleNotFoundError: No module named 'lxml'


----------------------------------------------------------------------
Ran 6 tests in 154.772s

FAILED (errors=1)

[deepika8mandakala]

Resolved when manually imported by pip install lxml
or you should pull the latest version.

[petermr]

when you pull the latest version , then the best thing is probably

pip install -r requirements.txt

rather than manual installation

[Shabnam2212]

System : Windows 10, Virtual env : python 3.12

output error

ERROR: test_pipeline (unittest.loader._FailedTest.test_pipeline)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_pipeline

ERROR: test_vector_store (unittest.loader._FailedTest.test_vector_store)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_vector_store

module error was resolved

latest output

..Retrieved: [('Paris is the capital of France.', 0.28786033391952515)]
.Retrieved: [('Paris is the capital of France.', 0.3702659010887146)]
.
----------------------------------------------------------------------
Ran 6 tests in 93.853s

OK

[petermr]

"error was resolved" - what was the problem and how did you resolve it?

[HaarthiV13]

System : windows 11, virtual env:python 3.12

llmrag report for WG2 Chapter 4

wg = "wg2" chapter = "chapter04" div_id = "4.1" p_count = 7
output:

(venv) C:\Users\Dell\llmrag>coverage run --source=llmrag -m unittest discover -s tests
.1268: Water security is defined as ‘ the capacity of a population to safeguard sustainable access to adequate quantities of acceptable quality water for sustaining livelihoods, human well-being, and socio-e
115: There are at least four reasons for the centrality of water security in adapting to, and mitigating climate change.
1958: First, approximately half the world’s population (~4 billion out of ~8 billion people) are assessed as being currently subject to severe water scarcity for at least some part fo the year ( medium conf
2648: Second, while climate change directly affects freshwater availability across space and time, it also affects water requirements for different uses, such as irrigation, potentially adding to existing s
1359: Third, a large majority (~60%) of all adaptation responses documented since 2014 are about adapting to water-related hazards like droughts, floods and rainfall variability ( Berrang-Ford et al., 2021b
700: Finally, while limiting global warming to 1.5°C would minimise the increase in risks in the various water use sectors and keep adaptation effective, many mitigation measures can potentially impact fut
1223: This chapter draws on previous IPCC reports and new methodologies ( Section 4.1.1  and SM4.1, SM4.2) and assesses the impacts of climate change on natural and human dimensions of the water cycle with
..Device set to use cpu
..Retrieved: [('Paris is the capital of France.', 0.28786033391952515)]
.Retrieved: [('Paris is the capital of France.', 0.3702659010887146)]
.
----------------------------------------------------------------------
Ran 7 tests in 127.651s

OK

[deepika8mandakala]

Windows:11, Venv :Python 3.12

llmrag report WG1 Chapter04

wg = "wg1" chapter = "chapter04 div_id = "4.1" p_count = 12

Output:

PS C:\Users\manda\llmrag> coverage run --source=llmrag -m unittest discover -s tests
844: This chapter assesses simulations of future climate change, covering both near-term and long-term global changes. The chapter assesses simulations of physical indicators of global climate change, such
872: Essential input to the simulations assessed here is provided by future scenarios of concentrations or anthropogenic emissions of radiatively active substances; the scenarios represent possible sets of
1423: Every projection assessment is conditioned on a particular forcing scenario. If sufficient evidence is available, a detailed probabilistic assessment of a physical climate outcome can be performed for
1534: A central element of this chapter is a comprehensive assessment of the sources of uncertainty of future projections ( Section 1.4.3 ). Uncertainty can be broken down into scenario uncertainty, model u
927: The uncertainty assessment in this chapter builds on one particularly noteworthy advance since the IPCC Fifth Assessment Report (AR5). Internal variability, which constitutes irreducible uncertainty o
823: The expanded treatment of uncertainty allows this chapter a more comprehensive assessment of the benefits from mitigation than in previous IPCC reports, as well as the climate response to carbon dioxi
530: The question of the potential crossing of thresholds relative to global temperature goals ( Geden and Loeschel, 2017 ) is intimately related to the benefits of mitigation; a prerequisite is an assessm
1259: The chapter is organized as follows (Figure 4.1). After ( Section 4.2  on the methodologies used in the assessment,  Section 4.3  assesses projected changes in key global climate indicators throughout
648: ( Section 4.4  covers near-term climate change, defined here as the period 2021–2040 and taken relative to the period 1995–2014.  Section 4.4  focuses on global and large-scale climate indicators, inc
974: ( Section 4.5  then covers mid-term and long-term climate change, defined here as the periods 2041–2060 and 2081–2100, respectively, again relative to the period 1995–2014. The mid-term period is thus
629: ( Section 4.6  assesses the climate implications of climate policies, as simulated with climate models. First,  Section 4.6  assesses patterns of climate change expected for various levels of GSAT ris
390: ( Section 4.7  assesses very long-term changes in selected global climate indicators, from 2100 to 2300.  Section 4.7  continues with climate-change commitment and the potential for irreversibility an
..Device set to use cpu
..Retrieved: [('Paris is the capital of France.', 0.2878604233264923)]
.Retrieved: [('Paris is the capital of France.', 0.3702658712863922)]
.
----------------------------------------------------------------------
Ran 6 tests in 63.483s

OK        

[MoobasharaJ]

Windows:11, Env :Python 3.12
llmrag report for WG2 Chapter 2

wg = "wg1" chapter = "chapter02 div_id = "2.1" p_count = 4

OUTPUT:

.1124: This chapter assesses the evidence basis for large-scale past changes in selected components of the climate system. As such, it combines much of the assessment performed in Chapters 2 through 5 of the
1073: Use is made of paleoclimate, in situ, ground- and satellite-based remote sensing, and reanalysis data products where applicable ( Section 1.5 ). All observational products used in the chapter are deta
654: Trends, when calculated as part of this assessment, have wherever possible been calculated using a common approach following that adopted in Box 2.2 of Chapter 2 of AR5 ( Hartmann et al., 2013 ). In a
723: There exist a variety of inevitable and, in some cases, irreducible uncertainties in performing an assessment of the observational evidence for climate change. In some instances, a combination of sour
..Device set to use cpu
..Retrieved: [('Paris is the capital of France.', 0.28786033391952515)]
.Retrieved: [('Paris is the capital of France.', 0.3702659010887146)]
.
----------------------------------------------------------------------
Ran 7 tests in 108.153s

OK

[MoobasharaJ]

Windows:11, Env :Python 3.12

Ran pytest

OUTPUT:

ERROR tests/test_chunking.py
ERROR tests/test_embedder.py
ERROR tests/test_ingestion.py
ERROR tests/test_pipeline.py
ERROR tests/test_smoke.py
ERROR tests/test_vector_store.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
6 errors in 0.48s

[Shabnam2212]

window:11,Env:python 3.12,VS code

pytest

output:

ModuleNotFoundError: No module named 'llmrag.embeddings'

ImportError while importing test module 'C:\Users\Shabnam Barbhuiya\llmrag\tests\test_chunking.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\.conda\envs\lltest\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests\test_chunking.py:6: in <module>
    from llmrag.utils.yaml_loader import load_paragraphs_yaml
E   ModuleNotFoundError: No module named 'llmrag.utils'
____________________________________________________________________ ERROR collecting tests/test_embedder.py ____________________________________________________________________ 
ImportError while importing test module 'C:\Users\Shabnam Barbhuiya\llmrag\tests\test_embedder.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\.conda\envs\lltest\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests\test_embedder.py:2: in <module>
    from llmrag.embeddings import load_embedder
E   ModuleNotFoundError: No module named 'llmrag.embeddings'
___________________________________________________________________ ERROR collecting tests/test_ingestion.py ____________________________________________________________________ 
ImportError while importing test module 'C:\Users\Shabnam Barbhuiya\llmrag\tests\test_ingestion.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\.conda\envs\lltest\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests\test_ingestion.py:4: in <module>
    from llmrag.embeddings import SentenceTransformersEmbedder
E   ModuleNotFoundError: No module named 'llmrag.embeddings'
____________________________________________________________________ ERROR collecting tests/test_pipeline.py ____________________________________________________________________ 
ImportError while importing test module 'C:\Users\Shabnam Barbhuiya\llmrag\tests\test_pipeline.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\.conda\envs\lltest\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests\test_pipeline.py:7: in <module>
    from llmrag.embeddings import load_embedder
E   ModuleNotFoundError: No module named 'llmrag.embeddings'
_____________________________________________________________________ ERROR collecting tests/test_smoke.py ______________________________________________________________________ 
ImportError while importing test module 'C:\Users\Shabnam Barbhuiya\llmrag\tests\test_smoke.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\.conda\envs\lltest\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests\test_smoke.py:4: in <module>
    from llmrag.embeddings import SentenceTransformersEmbedder
E   ModuleNotFoundError: No module named 'llmrag.embeddings'
__________________________________________________________________ ERROR collecting tests/test_vector_store.py __________________________________________________________________ 
ImportError while importing test module 'C:\Users\Shabnam Barbhuiya\llmrag\tests\test_vector_store.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\.conda\envs\lltest\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests\test_vector_store.py:4: in <module>
    from llmrag.embeddings import load_embedder, SentenceTransformersEmbedder
E   ModuleNotFoundError: No module named 'llmrag.embeddings'

ERROR tests/test_chunking.py
ERROR tests/test_embedder.py
ERROR tests/test_ingestion.py
ERROR tests/test_pipeline.py
ERROR tests/test_smoke.py
ERROR tests/test_vector_store.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
6 errors in 0.48s

[HaarthiV13]

System : windows 11, virtual env:python 3.12

Ran pytest

output:

ModuleNotFoundError: No module named 'llmrag.embeddings'

ModuleNotFoundError: No module named 'langchain'

intsalled langchain with command pip install langchain , but still getting the error

=============================================== short test summary info ===============================================
ERROR tests/test_chunking.py
ERROR tests/test_embedder.py
ERROR tests/test_ingestion.py
ERROR tests/test_pipeline.py
ERROR tests/test_smoke.py
ERROR tests/test_vector_store.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
6 errors in 0.21s

[petermr]

When I run

pytest tests

I get an output like:

.ss.......                                                                                          [100%]
========================================= short test summary info =========================================
SKIPPED [2] venv/lib/python3.12/site-packages/_pytest/unittest.py:385: infinite loop?
8 passed, 2 skipped in 30.71s

Where are your console outputs?

I need to know which tests fail and where.

[petermr]

This gives almost no useful output for debugging.

…

On Wed, Jun 25, 2025 at 7:05 AM MoobasharaJ ***@***.***> wrote: *Windows:11, Env :Python 3.12* Ran pytest OUTPUT: ERROR tests/test_chunking.py ERROR tests/test_embedder.py ERROR tests/test_ingestion.py ERROR tests/test_pipeline.py ERROR tests/test_smoke.py ERROR tests/test_vector_store.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 6 errors in 0.48s — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCSZTAN2C7EIA5E6LTZL3FI33FAVCNFSM6AAAAAB74TU7OCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNJWHE2TSMA> . You are receiving this because you commented.Message ID: ***@***.***>

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

[petermr]

Which methods throw the errors. Please try to get more information.

…

On Wed, Jun 25, 2025 at 7:11 AM Shabnam Barbhuiya ***@***.***> wrote: window:11,Env:python 3.12 pytest output: ModuleNotFoundError: No module named 'llmrag.embeddings' ERROR tests/test_chunking.py ERROR tests/test_embedder.py ERROR tests/test_ingestion.py ERROR tests/test_pipeline.py ERROR tests/test_smoke.py ERROR tests/test_vector_store.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 6 errors in 0.48s — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCS4Z7XFIY7BSCNZCHX33FI4QHAVCNFSM6AAAAAB74TU7OCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNJWHE3DEOA> . You are receiving this because you commented.Message ID: ***@***.***>

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

[petermr]

Is langchain in requirements.txt? You should use that.

…

On Wed, Jun 25, 2025 at 7:18 AM Vallabhaneni Haarthi < ***@***.***> wrote: System : windows 11, virtual env:python 3.12 Ran pytest output: ModuleNotFoundError: No module named 'llmrag.embeddings' ModuleNotFoundError: No module named 'langchain' intsalled langchain with command pip install langchain , but still getting the error =============================================== short test summary info =============================================== ERROR tests/test_chunking.py ERROR tests/test_embedder.py ERROR tests/test_ingestion.py ERROR tests/test_pipeline.py ERROR tests/test_smoke.py ERROR tests/test_vector_store.py!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 6 errors in 0.21s — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCS77P5URKU7M3EJWAAD3FI5LLAVCNFSM6AAAAAB74TU7OCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNJWHE3DONI> . You are receiving this because you commented.Message ID: ***@***.***>

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

[Shabnam2212]

yes langchain is available in requirements.txt

[deepika8mandakala]

Windows:11 ,Venv: Python 3.12.5

Report for llmrag tests

Output Summary:

============== short test summary info =============== 
ERROR tests/test_chunking.py  [Syntax Error]
ERROR tests/test_embedder.py [->Import error"llmrag.embeddings"]
ERROR tests/test_ingestion.py->[Import error"llmrag.embeddings"]
ERROR tests/test_pipeline.py[->Import error"llmrag.embeddings"]
ERROR tests/test_smoke.py->[Import error"llmrag.embeddings"]
ERROR tests/test_vector_store.py->[Import error"llmrag.embeddings"]
!!!!!! Interrupted: 6 errors during collection !!!!!!! 
6 errors in 1.02s

Trying to solve them will update after

[sarthh7777]

Problem:
When running your test suite, pytest couldn’t locate your llmrag package and failed to collect any tests:
=================== short test summary info ===================
ERROR tests/test_chunking.py
ERROR tests/test_embedder.py
ERROR tests/test_ingestion.py
ERROR tests/test_pipeline.py
ERROR tests/test_smoke.py
ERROR tests/test_vector_store.py
!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!

Solution :
From the project root, run:
pip install -e .

Expected Output:
=================== warnings summary ===================
PytestConfigWarning: Unknown config option: python_paths
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
10 passed, 1 warning in 52.48s