Refactoring using local AI solutions

edit

Unfortunately it looks like the models we can run locally perform terribly compared with gpt-4o via the OpenAI API. I tried Meta-Llama-3-8B-Instruct.Q4_0.gguf and gpt4all-13b-snoozy-q4_0.gguf via gpt4all. The results were more work than writing the code myself, exhibiting many of the problems I remember encountering with OpenAI models about 2 years ago. So for now we're deferring any further effort towards allowing models to be run locally. We'll revisit this, of course, as things advance.

Have to agree - I tested it with some test task I used half a year ago with Claude having excellent experience, this time trying to use qwen2.5-coder-7b-instruct-Q4_K_M.gguf on LMStudio. It produced mediocre results despite pointing out the obvious problems. I also tried smaller models, but those were not able to produce a single line of python code. Didn't try larger models as my computer does not have required resources to run them.

I tried this on a very capable machine with 12 CPU cores, 19 GPU cores, and 16 neural engines, although I have no idea if gpt4all manages to use any of that hardware. It wasn't terrible fast so perhaps not, but I suspect the lack of results is more the model than the computing hardware. It was often stopping early, saying things like "fill in the rest of the implementation here" which OpenAI did a lot a while back but they explicitly fixed that. I'm hopeful that locally run models will eventually catch up...