Wanted to ask a question about ollama, but i think i hear only echo here 😅
You must log in or # to comment.
Just go with the question instead :p But I’ll shoot some answers, perhaps one’s for you!
Ollana has a few advantages, yes. - Mainly ease of use!
Llama.cpp is more performant at least on non Macs.
No, wouldn’t recommend that.
Yes, sure, try!
If you round Pi to four you get all kinds of interesting reactions from engineers.
Echo echo echo?
But on a serious note: just ask!
Not even an engineer and the thought of rounding pi to 4 triggered me so hard, good job
Glad to hear you are all here. I noticed that whatever llm i use, they all have prity limited “memory” of how long they remember our conversacion. I mean if chat is wery short, they tend to remember what we were talking wery well. But if chat gets a bit longer they quickly start removing upper parts of the chat and remember only some part abowe our curent stage.
Thats why they easly get lost, and dont know anymore what we were talking about and what they need to answer now…
My pc specs are: Arxh Linux with KDE Plasma Gpu - Rx 6800XT 16GB Cpu - Ryzen 9 5950x 16x Ram - 64GB DDR4 A couple of ssd-s and good Corsair PS 1000W
I noticed that whatever llm i use, they all have prity limited “memory” of how long they remember our conversacion. I mean if chat is wery short, they tend to remember what we were talking wery well. But if chat gets a bit longer they quickly start removing upper parts of the chat and remember only some part abowe our curent stage.
Thats why they easly get lost, and dont know anymore what we were talking about and what they need to answer now…
Yup. The context window only goes so far.
If you have the memory to spend, you can get it pretty far out, depending on the model (which is also a factor). I’m currently mostly using a Llama 3 derived-model (AnubisLemonade) with a 128k context window on a 128GB Framework Desktop. IIRC, there have been some subsequent models that can go substantially further, though I understand that the response quality degrades on them. The problem is that systems with that much memory available have gotten really expensive over the past nine months or so and are probably going to stay expensive at least through the end of 2027.
I use llama.cpp these days, not ollama, but I’d guess that it’s probably possible to configure ollama for a larger context window as well, up to the maximum that your model can handle, though I can’t give specifics as to configuration.
searches
How can I specify the context window size?
By default, Ollama uses a context window size of 4096 tokens. This can be overridden with the OLLAMA_CONTEXT_LENGTH environment variable. For example, to set the default context window to 8K, use:
OLLAMA_CONTEXT_LENGTH=8192 ollama serveIf you don’t have the VRAM for a larger context window with whatever model you’re using:
- I don’t do it, but it’s possible to, with some engines, place some model layers in main memory and have the CPU do them, and place some model layers in VRAM.
checks
It looks like ollama can do this; apparently it will use as much video memory as it can, and fall back to system memory.
The Processor column will show which memory the model was loaded into:
- 100% GPU means the model was loaded entirely into the GPU
- 100% CPU means the model was loaded entirely in system memory
- 48%/52% CPU/GPU means the model was loaded partially onto both the GPU and into system memory
If you’re using something like SillyTavern, from my reading, users on /r/SillyTavernAI have generally tried to occasionally summarize the past conversation and insert that into the context. There are some plugins that try to do this automatically; my impression from past reading is that they weren’t very good.
Ask and find out!
