Context Shifting: Getting unusual output after context shift

[raulod]

I am testing context shift with a small context size (160) and facing some issues. I selected 160 because I wanted context shifting to happen during the second conversation (and not the first)

./llama-cli -m llama-32-1b-instruct.gguf -t 0 -c 160 -p "You are a helpful agent which lives in africa and answers 
everything about Africa in single Paragraph only without any special characters or new line." -cnv -n 64

The conversation went as follows:

> tell me about africa
Africa is the second largest and most diverse continent in the world with 55 countries and over 2 billion people 
living on it the continent is home to over 30 000 languages and over 2 000 ethnic groups the climate varies from 
tropical to desert
> biggest city
The biggest city in Africa is Cairo in Egypt with a population of over 20 million people the city is a hub for 
business and culture and is home to many historical landmarks such as the Pyramids of Giza the Great Wall of 
China

As you can see from the output, Great Wall of China immediately after the Pyramids of Giza seems unusual. This issue does not occur when the context size is larger. One possibility is that the small context size might have caused the loss of African context, the way the sentence is structured still feels odd.

To test further, I enabled logs in main.cpp and increased n_predict to 128.

system

You are a helpful agent which lives in africa and answers everything about Africa in single Paragraph only without 
any special characters or new line.

> tell me about africa
Africa is the secondlargest continent in the world and home to 55 countries including Egypt Sudan Ethiopia Kenya 
Tanzania Rwanda Uganda Democratic Republic of Congo South Africa Mozambique Zambia Zimbabwe Botswana 
Namibia Angola Guinea Equatorial Guinea Gabon Cameroon Liberia Sierra Leone Guinea Bissau Lesotho Botswana 
Swaziland Malawi Zambia Cameroon Democratic Republic of Congo Uganda Kenya Tanzania Rwanda Uganda South 
Africa Mozambique Zambia Zimbabwe Botswana Malawi Zambia Botswana Cameroon Democratic Republic of Congo 
South Africa Swaziland Lesotho Zambia Zimbabwe Botswana Malawi Cameroon Democratic Republic of Congo 
Uganda Kenya Tanzania Rwandacontext full, swapping: n_past = 159, n_left = 158, n_ctx = 160, n_keep = 1, n_discard = 79
after swap: n_past = 80

> biggest city
The biggest city in the world, in terms of population, is:

**Tokyo, Japan**

As of 2021, the population of Tokyo is approximately 13.9 million people. It is not only the largest city in Japan 
but also one of the most populous cities in the world, with over 38 million people living
context full, swapping: n_past = 159, n_left = 158, n_ctx = 160, n_keep = 1, n_discard = 79
after swap: n_past = 80

 within its metropolitan area.

Tokyo is a megacity, with a total area of approximately 2,193 square kilometers (836 square miles). It is a global 
hub for business, finance, technology, culture, and entertainment, and is home
> 

Additionally, I have also noticed that sometime (not always) the next decoded token after context shifting is EOG.

[ggerganov]

Using context shift with models that require chat template is tricky because you have to be careful not do destroy the structure of the template. In your tests, you are likely discarding the entire system prompt together with some important control tokens for the chat turns.

[raulod]

thank you for responding, I am only using the llama-cli as it is without any modifications.

[ggerganov]

llama-cli with -cnv flag will use the chat template of the model.