Following the instructions
Hi,
I took the Arcee-Blitz-Q4_K_M.gguf and tried it in LM Studio on a 16GB Nvidia Quadro P5000, but it wouldn't start at all. Then I put it in to Jan and tried to translate the subtitles for a popular science film, by specifying the system and user promptes. As a result, I received about 4 tokens per second, but instead of a translation, he first gave a description. On the next attempt, he gave a resume. Then I managed to get him to start translating, but he lost the numbering. It is worth saying that the original model does not cope well with this task. What kind of model would you recommend for such work, taking into account your knowledge of scientific terminology and support for European languages?
Try virtuoso-lite or virtuoso-small, both should be in our repo. I can't speak to the GGUFs because the tokenizer might be a bit strange after the surgery we do. I can try replacing with mistrals stock tokenizer and see.
I took this model Q4_K_M 14.33 GB for my 5070Ti and it works pretty good in LM Studio until it reaches context 100%. Then it just jumps back to the first user input! As if nothing else happened yet! I also tried it on OpenWebUI and it was good until it started to repeat itself there too. I did some change in the setting that allowed it to go a bit further but at the end, it started over responding to the first User input!