Conducting deep web searches and gathering sources is one of the main things I’ve been using LLMs for. How far away are we from being able to self-host something like Claude’s web search capabilities? Or even just a service where I’d pay with my money instead of my data?
Openwebui+searxng on a AMD strix board.
Pro: works like a charm, low power consumption, fast, “big” , LLM (running qwen3.6 35B A3B + gemma4 E4B for website summaries and other smaller tasks)
Con: strix boards start at 2k€, more in USA because of tarrifs
For those who want to know more, rough setup:
- llama-cpp rocmfp4 fork
- currently custom quantized qwen3.6 35B A3B model, working on publishing
- be3 embedding and reranker, also GPU
- gemma4-e4b via FastFlowLM on NPU!
- OpenWebUI and searxng as docker containers on a Pi currently
We get 70-100tok/s generation. Four slots with 256k context length each.
We use a smaller Board with “only” 64GB of shared LPDDR5X. Bottleneck is memory speed, rocmfp4 quants help a lot.
As soon as I get my imatrix calibration right, I will publish the quantized versions.
Most existing quantized models are broken. The authors did some not supported stuff (like using a already quantized model and requantize it) that you may get issues with coherence or sudden Chinese words in the output.
That is not an issue with rocmfp4 but with vibe coders and agent psychosis.
Do you have a walk through for setup?
I’m on the strix halo 128 gb variant and while I got ollama working fine, i haven’t gotten any of these multi headed setups working
I am on Gentoo for it, but everything with a decent rocm should work.
Have a look for llama-swap, that handles multi head endpoints.
Also, as you are on a big board, you can quantize yourself, as the BF16 version of qwen has only 72gb.
I will try and post a full writeup next days. But feel free to dm me, if you need some guidance on quantize or more.
I am using this fork currently: https://github.com/charlie12345/ROCmFPX
Stuff happens fast currently, so may be worth to wait a week or two ig you need something super stable, but if you are up for experimenting, that’s the way to go
THis is great, thanks. I’m on the z-13 and needed to use it for a work project, which is wrapping up soon. I’m planning on re-building it as a locally hosted agent support machine.
Great man! Gentoo lover and long time addicted here… Keep it the good work!
Thank you so so much for pointing out ROCmFP4. I have been tinkering with my RDNA 3 framework on llama. I was struggling with ROCm llama.cpp and have been using vulcan in the meantime. I know there’s some issues on the llama.cpp github to try and fix my issue (UMA stuff), but haven’t come across this specific project. Gonna try it out
Yup. And if you want to take a small step without major hardware requirements: connect your setup to a paid subscription Mistral or Anthropic API. They allow you to switch off training on your data.
On top of that, the costs are way lower than the normal consumer grade chat subscriptions, and your searches + memory are kept locally (e.g., managed through open webui).
Openrouter is also nice for this. You can use real cheap models for embedding and the bigger ones for the actual research.
What’s an example of something you can do with this? I’m not familiar with Claude
“Hey Claude, research for me current research to Nuclear Fusion. What are the biggest hurdles what are the next steps, and how promising is private research” enabling the research feature will give you a report, Fact checked (not clean but ok ish), and all the sources for it.
Claude will spin up a bunch of workers and search the web, following leads, and so on.
One of the few actual useful features of AI IMHO
Perplexica now called Vane is an open source version of perplexity. It lets you search the web with an llm that cites sources for its claims and can be selfhosted https://github.com/BunsDev/perplexica-search-engine-ai
Probably not far? I’d look into some LLM that can run commands with API, hook it to some web search API, and then work on the results.
Did anyone mention that huggingface will quanticize for you? Its like one button push.
Not imatrix or advanced quants, but yes.
But there are more then enough stock models for this task I would say. For specialized use cases though costum quant can be very very powerful
deleted by creator
