Question about local AI-assisted programming

[u/blockCoder2021]

I’m fairly experienced with Python and researching how to solve issues on my own. However, I have found that AI can help in some cases. For instance, giving me specific information on debugging and helping me research solutions for specific problems.

I was using Copilot in VS Code for a while, until I ran out of usage. Now, I’ve been messing around with locally-hosted AI, trying to find a good model for coding that runs well on my computer. My challenge is that I only have 4 GB of VRAM, though I also have 16GB of system RAM. If someone could help me find either a good, intelligent, and accurate local model that runs on that, or at least a free cloud model that’s good, I’d appreciate it.

[u/HotPersonality8126]

This compute is incredibly expensive and they really don’t give it away; not much of it, anyway.

There aren’t going to be any models of any use that fit into 4gb of VRAM. I have 8 and Qwen 8b is a complete dumbass; with so little room for context it doesn’t remember it’s in a coding harness by the second turn.

[u/blockCoder2021 → u/HotPersonality8126]

I’ve been using either Qwen 2.5-coding:7b or recently trying Qwen 3.5. About half the time (or so it seems), it’s either sending an answer in “native JavaScript” or something similar, or just an extremely short answer that may or may not even be relevant. Sometimes, though, it is able to give me a real answer.

[u/IcyCranberry3297]

16GB RAM helps more than you’d think.

[u/PureWasian]

Curious, how much heavy programming are you doing that you can't just leech off of Gemini or GPT free tiers without exhausting limits which reset frequently?

[u/blockCoder2021 → u/PureWasian]

I might be able to, honestly. My challenge is that I usually use AI in short but heavy bursts. I may look into that, though. Thanks!

[u/Naetharu]

You are likely to have issues here. The system is just very under-powered. Using OLama or LMStudio you can offload the model into system RAM. But the cost of doing so is a performance drop by orders of magnitude. Likely to a level where you would struggle to work with it.

Especially if you are looking to lean on it enough that you're hitting a paywall limit for CoPilot.

Being realistic if you're wanting to run local AI models then you really either need a larger sized GPU (16GB + and ideally more) or a system that uses a unified memory architecture like Apple's modern Macs.

Even then be aware that there is a sizable difference in performance between frontier models. like Claude and CoPilot vs the smaller local models that are open source. Local models do have their use, but go into this knowing that you're not about to get something that performs like a paid API service locally.

[u/BITT0]

Try OpenCode; it has some free tier models that are totally enough for normal projects. Just create a context memory for the project so you can easily switch to other models whenever you want.

[u/sugarw0000kie]

I’ve been running bonzai on 2-bit with 24gb, and it’s a pretty decent model, some of the “actually useful” local models seem to need around 16-32gb, but bonzai is an outlier worth looking at.

There’s a 1-bit model that I haven’t experimented with that’s supposed to run on phones, it won’t be as good as 2-bit but *might* run for you, if it does it cojld give you enough for short directed tasks or at least completions. https://prismml.com/news/bonsai-8b

The 2-bit version does well enough in zeds ide and is fast enough for me having it write small functions inline something I’m working on by hand. It does occasionally fumble tool calls (a lot of small models do) but overall is enough for me to actually find it useful

Otherwise opencode is worth checking out, there’s free models but limited to 200 requests per day, opencode go gives a good amount of usage for $10 atm especially if you stick to mimo 2.5 and muse spark. Codex $20 isn’t bad either if you make ample use of luna to implement things