Thank you so so much for pointing out ROCmFP4. I have been tinkering with my RDNA 3 framework on llama. I was struggling with ROCm llama.cpp and have been using vulcan in the meantime. I know there’s some issues on the llama.cpp github to try and fix my issue (UMA stuff), but haven’t come across this specific project. Gonna try it out
Fair enough. Let me quickly go through the one-liner, command-by command
# Joined by `&&`, bash runs these commands in sequence (as if run individually in shell), but exits/stops execution early if any command fails (return nonzero)
TMP_DEB=$(mktemp --suffix=.deb) && curl -sSL "https://support.brother.com/g/b/downloadend.aspx?c=us&lang=en&prod=hll2465dw_us&os=128&dlid=dlf106036_000&flang=4&type3=10283" -o "$TMP_DEB" && sudo apt install -y "$TMP_DEB" && rm -f "$TMP_DEB"
# Going command by command:
# First, we create a local variable in the shell, named `TMP_DEB`
# We assign the value to `$(...)`. This stores the string output (to stdout) of running the command `mktemp ...` to `TMP_DEB`
# `mktemp` creates a temporary file and prints its name, which uses the name template `tmp.XXXXXXXXXX`
# `--suffix=.deb` flag appends `.deb` to the name template
TMP_DEB=$(mktemp --suffix=.deb)
# At this point, we've created a temporary file, and saved the name to a variable in bash
# Next, we download the file using curl. `-s` makes output silent, `-S` shows errors in output, and `-L` follows redirects
# note the url doesn't end in `.deb`, implying that we will be redirected by the web server to the file path. without -`L` curl will download a page that stores the redirection response from the web server, not the .deb package
# `-o "$TMP_FILE"` forces curl to store the downloaded file to the tmp file we created
# note the quotes around the variable expansion. `$TMP_FILE` would also resolve the string stored in the variable, but we use quotes to avoid string globbing (google this)
curl -sSL "https://support.brother.com/..."
# Next, we install the package with apt
# note: we use the string stored in the variable `TMP_DEB`, the filepath to the temp file we created, and downloaded the deb package
# `-y` flag skips the confirmation question "install package [y/n]: `
sudo apt install -y "$TMP_DEB"
# Finally, to clean up we delete the tmp file
rm -f "$TMP_DEB"
I certainly wasn’t trying to “encourage” anything. I agree, blindly trusting commands is dangerous.
In this context I present a specific explanation of how the install works. This adds to the novice’s knowledge, and allows them to begin to understand what my one-liner does.
I think that without the context of instructions on how to do it manually, yes, you could make the case i’m enabling beginners to form/reinforce bad habits.
That printer probably supports AirPrint, which Mint supports without any extra tinkering. Connect the printer to your network, and try going through linux mint and adding the printer through the settings. If it doesn’t show up, then you can try using drivers (install using below command) and then re-adding the printer
Install by pasting this into your terminal. Enter your password when prompted.
TMP_DEB=$(mktemp --suffix=.deb) && curl -sSL "https://support.brother.com/g/b/downloadend.aspx?c=us&lang=en&prod=hll2465dw_us&os=128&dlid=dlf106036_000&flang=4&type3=10283" -o "$TMP_DEB" && sudo apt install -y "$TMP_DEB" && rm -f "$TMP_DEB"
Explanation if you want to learn:
sudo apt install ./Downloads/package_name.deb
i would say this is like if open code and open web ui had a baby it would be this. It’s a web interface for self hosting models but runs them through open code to make it agentic. Helpful for non developers to get into running models, but imo isn’t significant bc using open code tui and connecting it to a llama.cpp or vllm self hosted api is not difficult for devs
license for software whose source code is openly available for anyone to view, use, modify, and share
Honestly, you’re a few months late to the whole buying GPUs for local llms party, so expect exorbitant prices even for older cards
The name of the game is vram. For the most part, more is better. If you can get your hands on multiple matching (same model) 24gb or higher cards (within price range), you’re golden.
Going for more than 2 gpus can become challenging with motherboard pcie slot heights, so make sure either your cards aren’t too tall or you have widely spaced out pcie slots.
For inference, speed (tokens/second) is limited by memory bandwidth. Go for faster bandwidth memory cards if you can afford it (e.g. GDDR6 will be faster than GDDR5).
Also with multi gpus you will need an adequate power supply, and a large enough case.
If you want to be a bit eccentric and load huge models, you can also go the CPU route and fill up a motherboard with 256 GB ram, because then you’re in the several hundred B param model territory, which could, depending on your use case, be better than having faster inference on smaller/quantized models. Even then, DDR5 with high MHz is still way slower than gpus.
lol they already support running local models. wtf is the distro gonna do…? pre-install llama.cpp? this is so silly to me that people are resigning over this, too.
No way, a Guix user in the wild???!! I didn’t think you exist! Any opinion you have as to why guix over NixOS other than GNU philosophy and liking Lisp over Nix?
Honestly it heavily depends on the use case, in terms of making the model better and choosing between RAG/FT. The most important thing to consider is what sort of changes you want to make to the model. FT is still a good choice if you’re looking for: strict output formatting (json/yaml/…) and refining for highly specific, narrow domain tasks. RAG is better for knowledge freshness, having source citations, and greatly lowers hallucinations.
RAG will inflate your context windows (more tokens) at inference time, so slower responses and requiring more energy at compute, whereas fine-tuning takes a ton of gpu compute up front (but retains smaller token counts at inference). If you’re doing 100,000 prompts a day, and only need to train once, FT makes more sense; if you’re doing 100 prompts a day and your knowledge database is constantly changing, RAG makes the most sense.
It’s hard to give a formalized estimate on energy efficiency: fine-tuning and getting to a certain training accuracy can take some undeterminate amount of time (and money on rented GPU compute), but could be a better choice if you think that up-front cost will be paid off over time if you use the model very frequently and only fine-tune once. On the other hand, going the RAG route will have an absolutely free up front compute (energy) cost, but be slightly more at compute time due to more tokens.
What’s your specific task you’re considering for FT or no FT? This is the most important thing to choose.
I do AI research for school. I’m specifically interested in safety alignment. I have studied the original papers for different fine tuning methods: LoRA is typically the baseline and there exist many variants, notably Q-LoRA
In general, fine tuning is not practically beneficial for hobby level foundation models. It in fact comes with many disadvantages. Primarily, it is difficult to maintain the intelligence of the model and avoid overfitting.
If you are trying to adapt a model to a specific task, you are generally going to find more success with using RAG and just adding more context to the model that way. Don’t waste time and compute $$ on training.
Has anyone compiled a list of where projects are moving to? I know many linux desktop applications are self hosting on gitlab, but i’ve also seen gitea and codeberg. If anyone has opinions about a preference, do comment. I have been enjoying self hosting gitea for my simple personal projects and for deploying simple web apps, all on $5 vps.
I suggest looking at llm arena leaderboards filtered by open weight models. It offers benchmarks at a very complete and statistically detailed level for models, and usually is quite up to date when new models come out. The new Gemma that just came out might be the best for 1x GPU, and if you have a bunch of vram check out the larger Chinese models
He was a cyber pimp, exploiting sex workers online by stealing from their earnings. He also is one of the largest donors to the pro israeli lobby in the US
i think you need to look into routing. Specifically, split tunneling.
https://tailscale.com/learn/what-is-split-tunneling-secure-critical-data-vpn
https://protonvpn.com/support/protonvpn-split-tunneling
https://mullvad.net/en/help/split-tunneling-with-the-mullvad-app
I’d recommend installing those python dependencies using apt, so that when you update your system packages, the python libraries get updated, too. Pip, on the other hand, is useful for development but is detatched from apt and you will definitely forget to pip update unlike apt update which you hopefully do frequently. Use the names of the packages the readme provides in the pip install … instruction. For example, for numpy, you can install this.
Then, since that python script has a shebang at the top, you can add it to a directory in your $PATH and mark it executable with chmod, and you can invoke the script in your shell from any directory with just the file name.
i’m so incredibly annoyed by the fact that my previously favorite hosting provider digital ocean decided to fund this ridiculousness… any favorite alternatives anyone can offer?