Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If I understand correctly what you are looking for, Ollama might be a solution (https://ollama.com/)?. I have no affiliation, but I lazily use this solution when I want to run a quick model locally.


Better yet install Open Web GUI and ollama at the same time via docker. Most people will want a familiar GUI rather than the terminal.

https://github.com/open-webui/open-webui

This will install ollama and open web GUI:

For GPU support run:

docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

Use for CPU only support:

docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama


Why do people recommend this instead of the much better oobabooga text-gen-webui?

https://github.com/oobabooga/text-generation-webui

It's like you hate settings, features, and access to many backends!


To each their own, how are you using these extra features? I personally am not looking to spend a bunch on API credits and don't have the hardware to run models larger than 7-8b parameters. I use local llms almost exclusively for formatting notes and as a reading assistant/summarizer and therefor don't need these features.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: