Unexpected behaviour with RAG

[arthurbarr]

I tried a very simple experiment with RAG, and didn't get the results I expected. Can anyone help me explain why this is happening? Is it a bug, or am I thinking about this incorrectly?

$ echo "Bananas are yellow" > ./example/example.md

$ ramalama rag ./example localhost/example-rag

Building localhost/example-rag...
adding vectordb...
3a57f09caeb7238bab735f1cff1b3586d910271b4456d81d56974ca4d9c32824

$ ramalama run granite3-moe --rag localhost/example-rag
> What is a pencil?
A pencil is a writing implement made of cylindrical shape, usually made 
of carbonate of wood as a core, with a graphite or lead as the pointing 
agent, and sometimes coated with a carnauba wax for writing properties. 
The writing surface is covered with paper or plastic. The handle is made 
of wood, metal, or plastic and can be handled alone or with the pencil 
lead or graphite pointing agent for drawing. 

> What is a pencil?
A pencil is a writing implement made of cylindrical shape, usually made 
of carbonate of wood as a core, with a graphite or lead as the pointing 
agent, and sometimes coated with a carnauba wax for writing properties. 
The writing surface is covered with paper or plastic. The handle is made 
of wood, metal, or plastic and can be handled alone or with the pencil 
lead or graphite pointing agent for drawing.

A pencil is a writing implement made of a cylindrical shape, usually made 
of carbonate of wood as a core, with a graphite or lead as the pointing 
agent. The writing surface is covered with paper or plastic, which is 
where the banana comes into play. If the banana is yellow, it means the 
fruit is ripe and has a sweet taste.

(Added new lines for readability). Note the last response starts talking about bananas, and there are multiples responses to the second query.

I always get a reasonable answer to my first question, but the second or third questions tend to become strange. Sometimes it seems to go into a loop, and give me dozens of copies of the same answer. I assume that the previous responses are being added to a history, but this certainly isn't what I expected.

[rhatdan]

Try with different models. Also you really should ask questions about your RAG data. since the AI is concentrating on this.

[rhatdan]

@bmahabirbu wrote up a decent explanation of what is going on with a AI Model linked to a RAG database versus the same MODEL without the RAG.

[bmahabirbu]

#1260 (comment)

Heres an explanation I did for another issue that was similar!

Basically our rag implementations always adds the context of the document regardless of the prompt. So you'll get hallucinations if you ask the ai something that isn't related to the document!

[arthurbarr]

Thanks very much for the link. It makes sense.

[lnicola]

I'm a bit confused about how to use RAG. Do I need to use a special kind of prompt? I tried to build a database from two PDFs (one about beer glasses and one about feature engineering):

$ ramalama run --rag rag-test granite3-dense:8b
> What's the best glass shape to minimize heat transfer?
The best glass shape to minimize heat transfer is a frustum of a cone. This shape has the lowest surface-to-volume ratio for a given volume, which minimizes the heat transfer rate. The frustum of a cone can be obtained by revolving a line segment around an axis, with the line segment connecting the vertex of the cone to a point on the base. The height of the frustum is the distance between the vertex and the point on the base, and the radii of the top and bottom bases determine the shape of the frustum. The volume of the frustum can be calculated using the formula V = (1/3)πh(r1^2 + r2^2 + r1r2), where h is the height, r1 is the radius of the top base, and r2 is the radius of the bottom base. The surface area of the frustum can be calculated using the formula A = π(r1 + r2)(h + √(h^2 + (r1 - r2)^2)), where r1 and r2 are the radii of the top and bottom bases, respectively. The heat transfer rate associated with the frustum can be calculated using Fourier's law, F = kπT, where F is the heat transfer rate, k is the thermal conductivity constant, π is the mathematical constant 7, π is the mathematical constant 6, h is the height of the frustum, r1 and r2 are the radii of the top and bottom bases, respectively, T is the temperature of the frustum, and the heat transfer rate is proportional to the temperature and the radii of the top and bottom bases.

# on a quick skim, this doesn't seem to be supported by the paper

> What's the most widely used beer glass in Brazil?
Based on the provided text, it does not mention the most widely used beer glass in Brazil. The text focuses on the mathematical problem of finding the optimal shape for a beer glass to minimize heat transfer and keep the beverage cold for as long as possible. 

> What about Nadir Figueiredo?
Nadir Figueiredo is not mentioned in the provided text. 

> What's the title of the text?
The title of the text is not explicitly stated, but based on the content, it appears to be related to a method called SMOTE (Synthetic Minority Over-sampling Technique) and its comparison with other over-sampling techniques for improving the performance of machine learning algorithms, particularly in imbalanced datasets. 

> Tell me the title of a text about beer glasses
The title of the text is "The Optimal Shape of a Beer Glass: A Study in Heat Transfer and Variational Calculus". 

# it's not

[bmahabirbu]

Hi @lnicola! I'd recommend separating the PDFs into their own databases to get better responses! Along with this, try adding to the prompt "in the paper about beer glass shapes ...".

Try playing around with different models as well. I've noticed that models like Mistral have good responses.

As a whole, our RAG implementation is still new and will have quirks, irregular outputs, and so forth. Ongoing development is being done to refactor RAG and make it more reliable!