Setting up Qwen for Agentic Coding!

coding agents are everywhere. but most of them depend on cloud GPUs and paid APIs. this lead me to wonder if i can use my macbook to run a useful coding model locally for free.

i recently upgraded from an M1 pro (2020) to an M5 pro (2026), so this seemed like a good opportunity to test the specs of this as well. my goals were:

i also eventually want to move towards complete cli coding (not ide) so this is another step i am taking towards this goal.

my stack šŸ¤“
updates

[03.15.2026] - finished v1 + published

choosing a model

if the goal is to run a coding model locally, the first practical question is simple:

which model should i try? many models claim to be good at coding, but most of them are:

i ultimately knew i wanted to try qwen mostly from all the internet hype lol. it also stands out on the terminal-bench. terminal-bench measures how well models perform when interacting with real terminal environments and developer tooling.

ranking of llms from https://github.com/harbor-framework/terminal-bench. qwen dictates🤓

ranking of llms from https://github.com/harbor-framework/terminal-bench. qwen dictates🤓

trying my first setup with mlx

since i have the M5 Pro chip, my memory is capped at 64GB.

i first tried running Qwen 3.5 35B through mlx. mlx is developed by Apple machine learning research for Apple silicon. but, it was a fail.

i kept getting:

error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

i tried to force macOS to ā€œwireā€ more memory to the GPU:

sudo sysctl iogpu.wired_limit_mb=12288

āš ļø for obvious reasons, do not do this! it may break your computer. luckily for me, it only crashed the computer lol

why did it fail, you may ask. to understand why, i will tell you more about m5!

is m5 (pro) really worth it?

let me tell you what is so hype about m5 pro!

1. pre-fill

older chips like M3 Ultra took 10+ mins on pre-fill for large contexts. pre-fill (prompt processing) is the phase where model ā€œreadsā€ your prompt before it starts talking. the m5 pro fixed this with a newly introduced matmul hardware unit (the neural accelerator) in each core. it is designed specifically to boost prompt processing speed.

early benchmarks show that a prompt that took:

when i ask claude-code to scan my entire repo, it actually operates. seeing 43 tokens/sec on prompt evaluation for a 22k token repo is insane for a laptop 😱

2. ssd

besides the chip, the supporting silicon is also dope šŸ¤™:

3. UMA

Apple Silicon runs reasonably large models on a laptop without a discrete GPU. this is due to their unified memory architecture, or UMA. they moved the GPU and CPU onto a single die, this eliminated the slow and power draining PCIe transfer between system memory and VRAM. however, Apple’s original purpose of developing UMA is not for improving inference but for improving battery life and video processing.

this UMA architecture merely accidentally helped the LLM inference. Apple Edge can run massive models without choking. while some consumer GPUs can hit limit at 8GB or 12GB of vram.

Apple, ultimately, built one of the most accessible local inference platforms by trying to optimize battery life.

wow that’s great… now what sucks?

memory bandwidth.

the bad part is that even with the new accelerators, the M5 Pro is still limited by memory bandwidth for actual writing (token generation).

ā€œbanana for scale,ā€ one could say. you get significantly more memory (up to 64GB on the Pro), but you only get about 1/3 of the bandwidth of a dedicated desktop GPU. this means for the actual ā€œchattingā€ part, you aren’t hitting DGX speeds. for most SOTA models that use MoE (Mixture of Experts) , this bandwidth matters.

so in conclusion:

large unified memory āœ…

fast pre-fill āœ…

memory bandwidth for GPU āŒ

and that is what caused the crash on day one is because the MoE architecture of Qwen 3.5 loads many experts into memory. even with 64GB unified memory, the combination of model weights, kv cache, and runtime buffers exceeded what my system could handle.

so the fix is to run smaller models that actually fits in memory and use a runtime optimized for local inference, NOT increase memory lol.

solution to my problem (and maybe yours)!

i found this kind of dirty trick using claude code.

  1. install ollama šŸ¦™Ā using homebrew šŸŗ
# Update brew and install
brew update
brew install llama.cpp
  1. launch the api server.

i pivoted to Qwen2.5-Coder-14B. it is something my 64GB can handle.

llama-server -hf unsloth/Qwen2.5-Coder-14B-Instruct-GGUF:Q4_K_M \
  --jinja \
  --fit on \
  --flash-attn on \
  --ctx-size 32768 \
  --port 8001
  1. launch claude

you can trick claude code into thinking it is talking to Anthropic’s cloud while it is talking to my local qwen using a dummy bridge.

first, point the CLI to my local machine instead of the internet:

export ANTHROPIC_BASE_URL="http://localhost:8001/v1"

then, Claude Code will refuse to run if it doesn’t see an API key.Ā  it thinks localhost is the official server, and my local llama-server doesn’t actually care about keys.

provide a fake key to claude:

export ANTHROPIC_API_KEY="sk-no-key-required"

then launch:

claude --model qwen-local

then i ran /init in my HASHI_Teleop repo. i got a 400 error. the agent was trying to read 22,239 tokens but the server only had 8,192 tokens of space.

after bumping the context size to 32k, it worked šŸ„³šŸŽ‰

minor test of qwen

i ran some common test prompts on qwen.

i asked:

is there a seahorse emoji?

it failed:

if you want to know why it failed you can read this! AI’s Uncanny ā€œSeahorse Problemā€ | by Ignacio de Gregorio - Freedium

my future hopes for Apple

i feel that Apple should build a native inference service engine for text, images, and vision that any app could call. right now we have to stitch together runtimes like mlx, llama.cpp, or ollama. it works, but it still feels like a hack. every project ends up managing its own model server, memory limits, and runtime configuration.

instead, Apple could expose a system-level inference service the same way they expose Metal or CoreML. an app could simply request a model, send a prompt, and let the operating system handle scheduling the GPU, memory allocation, and model loading.

it is expensive now, but by the time we get M7 or M8 with 128GB of unified memory and better memory bandwidth, running local coding agents on laptops may stop feeling experimental. maybe in a few generations, running a coding agent locally will feel as normal as running a compiler!!