Abacus.ai:

We recently released Smaug-72B-v0.1 which has taken first place on the Open LLM Leaderboard by HuggingFace. It is the first open-source model to have an average score more than 80.

  • Miss Brainfarts@lemmy.blahaj.zone
    link
    fedilink
    English
    arrow-up
    3
    ·
    8 months ago

    That’s nice and all, but what are some FOSS models I can run on GPU with only 4GB?

    I’ve tried Deepseek Coder, and it’s pretty nice for what I use it for. Then there’s TinyLlama, which… well it’s fast, but I need to be veeeery exact in how I prompt it.

    • Toes♀@ani.social
      link
      fedilink
      English
      arrow-up
      2
      ·
      8 months ago

      4GB is practically nothing in this space. Ideally you want at least 10GB of dedicated vram if you can’t get even more. Keep in mind you’re also probably trying to share that vram with your operating system. So it’s more like ~3GB before you even started.

      Kolboldcpp is capable of using both your GPU and CPU together, you might wanna consider that. (Using a feature called layers) There’s a trade-off that occurs between the memory available and the quality of its output and the speed of the calculation.

      The model mentioned in this post can be run on the CPU with enough system ram or swap.

      If you wanna keep it all on the GPU check out 4bit models. Also there’s been a lot of work into trying to do this with the raspberry Pi. I suspect that their work could help you out here as well.

    • Fisch@lemmy.ml
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      8 months ago

      Unfortunately LLMs need a lot of VRAM. You could try using koboldcpp, it runs on the CPU but let’s you offload layers onto the GPU. That way you might be able to stay withing those 4gb even with larger models.

      Edit: I forgot to mention there’s a fork of koboldcpp with rocm for AMD cards, which is about twice as fast if I remember correctly. Only relevant if you have an AMD card tho.

      Edit 2: This is the model I use btw

  • simple@lemm.ee
    link
    fedilink
    English
    arrow-up
    2
    ·
    8 months ago

    I’m afraid to even ask for the minimum specs on this thing, open source models have gotten so big lately

    • TheChurn@kbin.social
      link
      fedilink
      arrow-up
      2
      ·
      8 months ago

      Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.

      1 billion parameters ~ 2 Billion bytes ~ 2 GB.

      From the name, this model has 72 Billion parameters, so ~144 GB of VRAM

      • FaceDeer@kbin.social
        link
        fedilink
        arrow-up
        3
        ·
        8 months ago

        It’s been discovered that you can reduce the bits per parameter down to 4 or 5 and still get good results. Just saw a paper this morning describing a technique to get down to 2.5 bits per parameter, even, and apparently it 's fine. We’ll see if that works out in practice I guess

        • Corngood@lemmy.ml
          link
          fedilink
          English
          arrow-up
          0
          ·
          edit-2
          8 months ago

          I’m more experienced with graphics than ML, but wouldn’t that cause a significant increase in computation time, since those aren’t native types for arithmetic? Maybe that’s not a big problem?

          If you have a link for the paper I’d like to check it out.

          • FaceDeer@kbin.social
            link
            fedilink
            arrow-up
            1
            ·
            8 months ago

            My understanding is that the bottleneck for the GPU is moving data into and out of it, not the processing of the data once it’s in there. So if you can get the whole model crammed into VRAM it’s still faster even if you have to do some extra work unpacking and repacking it during processing time.

            The paper was posted on /r/localLLaMA.

    • girsaysdoom@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      0
      ·
      8 months ago

      I think I read somewhere that you’ll basically need 130 GB of RAM to load this model. You could probably get some used server hardware for less than $600 to run this.

      • ArchAengelus@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        8 months ago

        Unless you’re getting used datacenter grade hardware for next to free, I doubt this. You need 130 gb of VRAM on your GPUs

      • cm0002@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        8 months ago

        Oh if only it were so simple lmao, you need ~130GB of VRAM, aka the graphics card RAM. So you would need about 9 consumer grade 16GB graphics cards and you’ll probably need Nvidia because of fucking CUDA so we’re talking about thousands of dollars. Probably approaching 10k

        Ofc you can get cards with more VRAM per card, but not in the consumer segment so even more $$$$$$

    • DarkThoughts@fedia.io
      link
      fedilink
      arrow-up
      1
      ·
      8 months ago

      Since I had an okay experience with EasyDiffusion I tried running text gen locally through oobabooga, but no matter which model I load, it just crashes whenever it tries to generate anything, regardless if it runs through the UI’s chat or SillyTavern. No error in the terminal either, it just stops and throws me back into the command line.

      • FaceDeer@kbin.social
        link
        fedilink
        arrow-up
        0
        ·
        8 months ago

        And at 72 billion parameters it’s something you can run on a beefy but not special-purpose graphics card.

        • glimse@lemmy.world
          link
          fedilink
          English
          arrow-up
          0
          ·
          8 months ago

          Based on the other comments, it seems like this needs 4x as much ram than any consumer card has

          • FaceDeer@kbin.social
            link
            fedilink
            arrow-up
            1
            ·
            8 months ago

            It hasn’t been quantized, then. I’ve run 70B models on my consumer graphics card at a reasonably good tokens-per-second rate.