Setting up a local coding agent

Deploying Qwen3.8-27B with OpenCode on a single 48 GB GPU

This page is a complete, standalone recipe for standing up an OpenAI-compatible vLLM endpoint serving Qwen/Qwen3.8-27B-FP8 on one 48 GB Ada-generation GPU (L40, L40S, or RTX 6000 Ada), and wiring OpenCode to it as an agentic coding CLI. When you are done you have a local, private coding assistant with a 128k context window that runs entirely on your own hardware.

It is written to work on any single-GPU server. Everything specific to the Wright cluster at CMU is called out in boxes like this one:

On Wright. Wright is a three-node SLURM cluster (n01, n02, n03), each node with one NVIDIA L40 (46,068 MiB), 32 CPU cores, ~1 TB of RAM, and a 1.8 TB node-local NVMe scratch filesystem at /mnt/scratch. All of the work below happens inside an interactive allocation on a compute node, never on the login node.

Skip to Restarting after a new allocation if the software is already installed and you just need the server back up.

1. Key terms and definitions

This guide assumes you have used large language models — through Claude Code or a similar tool — but have never deployed one. That gap is mostly vocabulary. This section defines the terms the rest of the page uses, in the order you will meet them.

The one-sentence mental model. Deploying an LLM means starting a long-running server process that loads ~28 GB of model weights into GPU memory once, then answers HTTP requests against them. Almost every decision in this guide is a consequence of a single budget: the weights, plus the memory needed to track ongoing conversations, must fit in 46 GB of GPU memory at the same time.

The serving stack

Term	What it means here
Inference server	A daemon that holds model weights in GPU memory and serves generation requests over HTTP. It replaces the remote API you normally call. Ours runs for the life of the SLURM allocation.
vLLM	The specific inference server used here. It handles batching, memory management, and quantized-weight kernels. Roughly, vLLM is to a model what a database server is to a data file.
OpenAI-compatible endpoint	vLLM speaks the same HTTP protocol as the OpenAI API (/v1/chat/completions, /v1/models). This is why an off-the-shelf client can talk to a local model with no code changes — you only change the base URL.
Checkpoint / weights	The trained parameters, downloaded as files. Qwen/Qwen3.8-27B-FP8 is a repository name on the Hugging Face Hub; "27B" is 27 billion parameters. Nothing is trained or fine-tuned in this guide — the weights are read-only inputs.
safetensors	The file format the weights ship in. Purely a storage detail; both the downloader and vLLM handle it.
Hugging Face Hub / hf download	The public repository the weights come from, and its command-line client. Think CRAN or PyPI, but for model weights, and with files measured in gigabytes.
Served model name	The string clients use to request this model (qwen38-27b). You choose it at launch. It must match the client configuration exactly, and a mismatch is a common first failure.
OpenCode	The agentic CLI client — the Claude Code analogue in this stack. It manages the conversation, decides when to call tools, and edits files. The model itself does none of that; it only emits text and tool-call requests.
tmux	A terminal multiplexer. The server runs inside a tmux session so it survives your SSH connection dropping. Without it, closing your shell kills the server.
SLURM / allocation / compute node	The HPC job scheduler, a reservation of hardware it grants you, and the machine that reservation runs on. You must hold an allocation to touch a GPU, and when it expires everything running inside it is killed.

Memory and capacity

This group is where deployments actually fail, so it repays reading closely.

Term	What it means here
VRAM	Memory physically on the GPU, entirely separate from the node's ~1 TB of system RAM. It is the scarce resource in this guide: 46 GB total, of which the weights take 27.6 GB.
Quantization / FP8	Storing weights in a lower-precision number format to save memory. FP8 is 8-bit floating point; e4m3 names the exponent/mantissa split. This model at full 16-bit precision would be ~54 GB and would not fit on this card; at FP8 it is 27.6 GB and does. There is a small accuracy cost, generally considered a good trade. Our checkpoint is already quantized — nothing converts it at load time.
KV cache	The most important term on this page. As the model reads a conversation, it computes intermediate quantities (keys and values) for every token and caches them so they need not be recomputed on every subsequent token. It is the model's working memory for the current conversation. Its size grows linearly with the number of tokens in flight, it lives in VRAM, and it consumes whatever the weights leave behind. Statistically, it is closest to a sufficient statistic being carried forward so you never rescan the data.
Context window / --max-model-len	The maximum number of tokens the model can consider at once — prompt plus history plus reply. We configure 131,072 (128k), which costs 8 GB of KV cache. Bigger is better for agentic work, and it is not free.
Concurrency / --max-num-seqs	How many requests the server will decode simultaneously. Each concurrent request needs its own slot with its own memory reservation. Set too high, this deployment fails to start — see §6.
Full vs. linear attention	Two layer types. Full attention compares each token to every prior token: quadratic cost, and it is what fills the KV cache. Linear attention keeps a fixed-size running summary instead: constant memory per sequence regardless of length, at some cost in expressiveness. This model is a hybrid, with 16 full-attention and 48 linear-attention layers, which is why its context is unusually cheap.
Recurrent state / "Mamba cache block"	The fixed-size summary each linear-attention layer maintains, ~72 MiB per concurrent request here. It does not grow with context, but it is reserved per request slot, so it caps concurrency rather than length. vLLM calls these blocks "Mamba" after the architecture family that popularized them.
--gpu-memory-utilization	The fraction of VRAM vLLM is permitted to claim (we use 0.94). The remainder is headroom for transient allocations. Raising it buys KV cache and raises the risk of an out-of-memory failure mid-request.
YaRN	A technique for stretching a model past its trained context length. Not needed here — this model natively supports 262,144 tokens, so our 131,072 is well inside its design range.

How the model behaves

Term	What it means here
Chat template	A Jinja template shipped inside the checkpoint that turns a list of messages into the single token string the model actually sees, inserting special markers for roles. Non-obvious but load-bearing: several behaviours documented on this page are properties of the template, not the weights.
Tool calling	The mechanism by which a model requests an action — read a file, run a command — instead of replying in prose. The client executes it and returns the result. This is the entire basis of agentic coding; if it is misconfigured the model will describe editing your files while changing nothing.
Parser (--tool-call-parser, --reasoning-parser)	The model emits one flat stream of text. A parser is the server-side code that recognizes the model family's particular markup and splits it into structured fields — a tool call, or a separate reasoning field. The parser must match the model family, and picking the wrong one fails silently rather than loudly.
Reasoning tokens / thinking	Tokens the model generates to work through a problem before answering. They are billed against the output budget, count against the context window, and are normally hidden from the user. A one-word visible answer can still cost hundreds of them.
reasoning_effort	A per-request field that steers how much the model thinks. On this model it is a prompt instruction, not a token budget — a nudge, not a cap. Its default is the most expensive setting. See §12; the behaviour is counter-intuitive enough to deserve its own section.
Sampling parameters	How the next token is drawn from the model's predicted distribution. temperature flattens (>1) or sharpens (<1) that distribution; top_p and top_k truncate it to the smallest set covering probability p, or to the k most likely tokens; repetition_penalty and presence_penalty down-weight tokens already used. Do not set temperature to 0 on this model family — greedy decoding degenerates into repetition loops. For reproducibility, fix the server's random seed instead.
Compaction	The client's strategy for staying inside the context window on a long session — summarizing or discarding old turns. It is where small models most often lose the thread, which is the argument for a large context window.
LSP diagnostics	Language Server Protocol output — the type errors and lint warnings your editor underlines. The client feeds them back to the model after each edit, so the model can see and fix its own mistakes. One of the largest available quality wins for a model this size.

Performance

Term	What it means here
Prefill vs. decode	The two phases of a request. Prefill processes the whole prompt in parallel and is compute-bound — fast per token, thousands per second. Decode generates the reply one token at a time and is memory-bandwidth-bound — slow, tens per second. The distinction matters because they respond to completely different optimizations, and real agentic work is overwhelmingly prefill (roughly 50:1 here).
Throughput (tok/s)	Tokens per second. Quoted figures almost always mean decode throughput, which is why a "3× faster" claim rarely makes a whole turn 3× faster.
Speculative decoding	A small, fast draft model guesses several tokens ahead; the real model then verifies them all in one pass, keeping the correct prefix. Because decode is bandwidth-bound, verifying four tokens costs little more than generating one, so accepted guesses are nearly free. It is designed to be distribution-preserving — the output should be statistically identical to not using it.
MTP / draft head	Multi-Token Prediction: this model's built-in drafter, shipped inside the checkpoint, so speculative decoding needs no second model. See §13.
Acceptance rate / acceptance length	How often the draft's guesses survive verification, and how many tokens per pass are kept on average. These are the diagnostic for whether speculative decoding is paying off.
torch.compile / CUDA graph capture	One-time compilation and GPU-kernel recording done at startup to make later requests faster. It is why the first launch takes ~4 minutes and later ones under a minute — and why changing certain flags forces a full recompile, since they are part of the cache key.
Prefix cache	Reuse of KV cache across requests that share an identical prompt prefix — very common in an agentic loop, where each turn re-sends the whole conversation. Enormously helpful in practice; a hazard when benchmarking, because a repeated request is not measuring what you think.

2. What you need before starting

Requirement	Notes
A GPU with ≥ 46 GB of free VRAM	The FP8 weights are 27.6 GiB; the rest goes to KV cache and activations. A 40 GB A100 is not enough at 128k context.
~35 GB of fast disk	29 GB of weights plus a virtual environment. Node-local NVMe is strongly preferred over NFS.
Python 3.12	3.11–3.13 all work. uv is much faster than pip if you have it.
tmux	The server must outlive your shell.
Outbound HTTPS to huggingface.co	For the one-time weight download.

The GPU must be genuinely free. Two copies of this model cannot share one 48 GB card — each needs roughly 41 GiB — so stop any other inference server first.

On Wright. Get an interactive allocation before doing anything:

srun --partition=statds --gres=gpu:1 --cpus-per-task=32 --time=2-00:00:00 --pty bash

Partitions are statds (n01), phil (n02), cmist (n03), and all. The maximum walltime is 2 days; when the allocation ends, the server dies and node-local scratch may be wiped. There is a second, older Qwen3.6 deployment at /mnt/scratch/$USER/qwen-l40s/ — leave it alone, and note that it cannot run at the same time as this one.

3. Survey the environment

Do not assume where scratch is; find it.

hostname
nvidia-smi --query-gpu=index,name,memory.total,memory.used,driver_version --format=csv

# Storage: check each candidate, do not guess.
df -h $HOME | tail -1
ls -ld /scratch/$USER /work/$USER /projects/$USER /mnt/scratch/$USER /mnt/scratch 2>/dev/null
mount | grep -iE "scratch|nvme|lustre|gpfs"

free -g; nproc
python3 --version; which uv tmux
timeout 15 curl -sI https://huggingface.co >/dev/null && echo "NET: yes" || echo "NET: no"

# Nothing else may be holding the card.
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Two failure modes to catch here rather than later:

• memory.used is not ~0 MiB. Something already owns the GPU. Find out what before continuing.
• NET: no. The node is air-gapped. You must download the weights from a login node or another host and copy them in.

Prefer node-local NVMe for the work directory even though it is wiped between jobs. It is both the fastest and the largest option, and re-downloading costs far less than fighting NFS for every small file the compile cache writes.

On Wright. The scratch path is /mnt/scratch — none of /scratch, /work, or /projects exist. Reference values observed on n01: 1 × NVIDIA L40, 46,068 MiB, driver 595.84, 32 cores, 1007 GB RAM, 860 GB free on /mnt/scratch. In practice scratch has survived across allocations when the new job landed on the same node, but that is luck, not a guarantee.

4. Create the work directory and environment file

Everything — weights, virtual environment, compile cache, temporary files — lives under a single $WORK directory, so a second deployment can never collide with this one.

export WORK=/mnt/scratch/$USER/qwen38-l40s      # adapt the parent path to your cluster
mkdir -p $WORK/hf $WORK/tmp && cd $WORK

cat > $WORK/env.sh <<'EOF'
export WORK=/mnt/scratch/$USER/qwen38-l40s
export HF_HOME=$WORK/hf
export VLLM_CACHE_ROOT=$WORK/vllm-cache
export TMPDIR=$WORK/tmp
export CUDA_VISIBLE_DEVICES=0
export HF_XET_HIGH_PERFORMANCE=1
EOF

Notes on each variable:

• HF_HOME keeps the 29 GB download off your home directory, which is usually quota-limited.
• VLLM_CACHE_ROOT isolates this deployment's torch.compile artifacts. If you run more than one model or more than one flag configuration, sharing a cache root causes confusing recompiles.
• CUDA_VISIBLE_DEVICES=0 pins to one GPU. No tensor or pipeline parallelism is used anywhere in this guide.
• HF_XET_HIGH_PERFORMANCE=1 is the current fast-download path. Do not set HF_HUB_ENABLE_HF_TRANSFER — it is deprecated and only produces a warning on every invocation.

If you have a Hugging Face token, export HF_TOKEN as well. Unauthenticated downloads are rate-limited, and the download is the single longest step in this guide.

5. Install vLLM and start the download — concurrently

The pip install and the weight download are both slow, independent, and neither saturates the CPU. Run them at the same time; done sequentially they roughly double the setup time.

Start the download first, since it has the longer tail:

source $WORK/env.sh
# If you already have a huggingface_hub install anywhere, borrow its `hf` binary so you are
# not blocked on the new venv. Otherwise create the venv first and use its `hf`.
nohup hf download Qwen/Qwen3.8-27B-FP8 --max-workers 16 > $WORK/download.log 2>&1 &

Then build the virtual environment in parallel:

cd $WORK
uv venv --python 3.12 .venv || python3 -m venv .venv
nohup bash -c "source $WORK/env.sh; source $WORK/.venv/bin/activate; \
  pip install --upgrade pip && pip install --upgrade 'vllm>=0.28.0' huggingface_hub" \
  > $WORK/pipinstall.log 2>&1 &

Version floor: vLLM 0.28.0. The model card states a lower floor (0.17.0), but 0.28.0 is what this configuration is proven against, and the optional speculative-decoding path in §13 requires it. Do not go below it.

Verify when both finish:

source $WORK/.venv/bin/activate
python -c "import vllm, torch; print('vllm', vllm.__version__); print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
du -sh $HF_HOME     # expect ~29 GB

On Wright. uv is not installed, so the plain python3 -m venv path applies and the install takes about 8 minutes. Verified stack: Python 3.12.11, vLLM 0.28.0, torch 2.13.0+cu130, transformers 5.16.1. The unauthenticated download of 29 GB took roughly 35 minutes.

Optional: pre-warm the page cache

If the node has plenty of RAM, read the weights once while the install is still running. This cuts vLLM's weight-load step from minutes to seconds:

SNAP=$(ls -d $HF_HOME/hub/models--Qwen--Qwen3.8-27B-FP8/snapshots/*/ | head -1)
cat $SNAP/*.safetensors > /dev/null

Note that this checkpoint ships as 64 per-layer files (layers-0.safetensors … layers-63.safetensors) plus outside.safetensors and mtp.safetensors, rather than the usual handful of shards. Both hf download and vLLM handle this transparently.

6. Understanding the capacity constraints

This section explains why the flags in §7 are what they are. If you only want a working server, skip ahead — but come back here the first time startup fails.

Qwen3.8-27B is not a dense transformer. It is Qwen3_5ForConditionalGeneration, a hybrid-attention multimodal model. The relevant fields from config.json:

Field	Value
num_hidden_layers	64
layer_types	16 full_attention + 48 linear_attention (every 4th layer is full)
num_key_value_heads	4
head_dim	256
max_position_embeddings	262,144

This has two consequences that drive the whole configuration.

First: the KV cache is four times cheaper than a dense model of this size. Only the 16 full-attention layers hold a growing per-token cache:

per-token KV = 2 × 16 layers × 4 kv_heads × 256 head_dim × 2 bytes = 64 KiB/token (bf16)
131,072 tokens → 8.0 GB

Because 128k of context costs only 8 GB at bf16, do not use --kv-cache-dtype fp8. You do not need the savings, and fp8 KV is the riskier setting on linear-attention layers. (Some published vLLM recipes for this model do pass it — they are targeting 262k context on larger cards.)

Note also that 131,072 is well inside the native 262,144 position embeddings, so no YaRN scaling is required for a 128k window.

Second — and this is the constraint that will actually bite you — the 48 linear-attention layers hold a constant-size recurrent state of roughly 72 MiB per concurrent sequence, independent of context length. vLLM allocates these as "Mamba cache blocks", only about 180 of them fit on a 48 GB card, and vLLM's default --max-num-seqs is 256. So the default configuration fails at CUDA graph capture, after several minutes of compilation:

ERROR EngineCore failed to start.
ValueError: max_num_seqs (256) exceeds available Mamba cache blocks (180).
  Each decode sequence requires one Mamba cache block, so CUDA graph capture cannot
  proceed. Please lower max_num_seqs to at most 180 or increase gpu_memory_utilization.

The fix is to lower --max-num-seqs, not --max-model-len. The constraint is sequence count, not context length; reducing context wastes the model's best feature and does not address the error. This is by far the most likely way the deployment fails to start.

Be aware that changing --max-num-seqs invalidates the torch.compile cache — it feeds the CUDA graph capture sizes and therefore the cache key — so every guess costs a full recompile. Compute the value you want rather than bisecting toward it.

7. Launch the server

cat > $WORK/serve.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source $WORK/env.sh
source $WORK/.venv/bin/activate
exec vllm serve Qwen/Qwen3.8-27B-FP8 \
  --served-model-name qwen38-27b \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.94 \
  --language-model-only \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
EOF
chmod +x $WORK/serve.sh

tmux new -d -s vllm38 "$WORK/serve.sh 2>&1 | tee $WORK/vllm.log"
tail -f $WORK/vllm.log

Why each flag:

Flag	Rationale
--served-model-name qwen38-27b	The id clients must use. It must match the OpenCode configuration in §10 exactly.
--max-model-len 131072	Native (no YaRN) and costs only 8 GB of KV per §6. Context headroom is the dominant quality lever for a 27B model in an agentic loop — compaction is where small models lose the thread.
--max-num-seqs 8	Mandatory — see §6. Anything above ~180 fails outright. 8 is the right choice for a single user: it covers the main loop, one subagent, and title generation, at ~72 MiB each.
--max-num-batched-tokens 2048	Set explicitly to the current default. Measured against 8192 it makes no difference on this hardware (0.03% on total wall time), and pinning it stops a future vLLM default change from silently altering the configuration.
--gpu-memory-utilization 0.94	Leaves ~2.7 GiB of headroom on a 46 GiB card.
--language-model-only	Skips building the vision tower and its multimodal profiling, freeing memory for KV. Correct for a text-only coding client.
--tool-call-parser qwen3_coder	Prescribed by the model card. Do not substitute hermes; it produces an agent that talks about editing files without ever calling a tool.
--enable-auto-tool-choice	Required alongside the tool-call parser for tool_choice: "auto".
--reasoning-parser qwen3	Splits chain-of-thought into a separate reasoning field instead of leaking it into content.

Both parser names are validated at startup, so a typo fails loudly rather than silently.

Do not pass --quantization fp8. The checkpoint is already quantized (quant_method: fp8, fmt: e4m3) and vLLM auto-detects it. Passing the flag attempts a second, on-the-fly quantization.

Expected startup

Stage	Cold cache	Warm cache
Loading weights	7–9 s (page cache warm)	7 s
Model loading total	27.57 GiB	27.57 GiB, ~8 s
torch.compile / Dynamo	~60 s	skipped
Profiling and warmup	~69 s	—
Graph capture	~5 s, 0.10–0.20 GiB	~2 s
Total to serving	~3–4 min	< 1 min

You are looking for three lines in the log: GPU KV cache size: <N> tokens, Graph capturing finished, and Application startup complete.

These warnings are benign and can be ignored: Failed to import from vllm._rocm_C (this is a CUDA build), tl.make_block_ptr is deprecated, and Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! — vLLM ships no tuned FP8 kernel configuration for the L40, which costs real but modest throughput and has no user-space fix.

On Wright. A warm restart on n01 reaches Application startup complete in about 40 seconds, reporting 13.59 GiB available for KV, 213,180 tokens of KV cache (1.63× concurrency at full 131,072 context), and 42,877 / 46,068 MiB of GPU memory in use.

Be aware that the KV pool is not reproducible between restarts of identical flags — observed values for this exact configuration range from 178,529 to 213,180 tokens, a spread of about 22%, and the cause has not been isolated. If you are tuning, average several restarts per arm and wait for nvidia-smi to report ~0 MiB before relaunching. A single before/after pair will mislead you.

8. Verify the endpoint

Save this as $WORK/verify.sh and do not proceed past a failure.

#!/usr/bin/env bash
M=qwen38-27b

echo "=== a: /v1/models ==="
curl -s http://localhost:8000/v1/models | python3 -m json.tool | head -20

echo "=== b: basic completion ==="
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d "{\"model\":\"$M\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":512}" \
  | python3 -c "import json,sys;d=json.load(sys.stdin);m=d['choices'][0]['message'];print('content:',repr(m.get('content')));print('reasoning len:',len(m.get('reasoning') or ''));print('usage:',d['usage'])"

echo "=== c: TOOL CALLING (critical) ==="
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d "{
    \"model\":\"$M\",
    \"messages\":[{\"role\":\"user\",\"content\":\"List the files in the current directory.\"}],
    \"tools\":[{\"type\":\"function\",\"function\":{
      \"name\":\"run_shell\",
      \"description\":\"Run a shell command and return stdout\",
      \"parameters\":{\"type\":\"object\",\"properties\":{\"command\":{\"type\":\"string\"}},\"required\":[\"command\"]}}}],
    \"tool_choice\":\"auto\",
    \"max_tokens\":1024
  }" > /tmp/toolcall38.json
python3 - <<'PY'
import json
d = json.load(open('/tmp/toolcall38.json'))
c = d['choices'][0]; m = c['message']; tc = m.get('tool_calls') or []
if tc and tc[0]['function']['name'] == 'run_shell':
    print("c PASS:", tc[0]['function']['name'], json.loads(tc[0]['function']['arguments']))
    print("  finish_reason:", c['finish_reason'], "| content:", repr(m.get('content')))
else:
    print("c FAIL: no tool_calls. content was:", (m.get('content') or '')[:400])
PY

echo "=== d: reasoning_effort plumbing ==="
for e in low medium xhigh; do
  printf "%-7s " $e
  curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
    -d "{\"model\":\"$M\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":1,\"reasoning_effort\":\"$e\"}" \
    | python3 -c "import json,sys;print('prompt_tokens',json.load(sys.stdin)['usage']['prompt_tokens'])"
done

Pass criteria:

• (a) the model id you chose appears, with max_model_len: 131072.
• (b) content is "OK", and the chain-of-thought arrives in a separate reasoning field. Reasoning text leaking into content means the reasoning parser is wrong.
• (c) — the important one — tool_calls is populated, function.name is run_shell, arguments parses as JSON, finish_reason is tool_calls, and content is null. Text that merely describes calling the tool is a failure: go back to §7 and check the parser names.
• (d) returns 41 / 11 / 53 prompt tokens for low / medium / xhigh. See §12 for what those numbers mean.

Keep max_tokens generous on test (c). This model emits reasoning tokens before the tool call, and a tight limit truncates the response before the call is ever produced — a false failure. Use at least 512.

On Wright. All four tests pass as of 2026-09-04. Test (c) returned run_shell{"command": "ls -la"} with finish_reason: tool_calls and content: null; test (d) returned exactly 41 / 11 / 53, and an invalid reasoning_effort value is rejected with HTTP 400.

9. Install OpenCode

curl -fsSL https://opencode.ai/install | bash
mkdir -p $HOME/.local/bin
ln -sf $HOME/.opencode/bin/opencode $HOME/.local/bin/opencode
export PATH="$HOME/.local/bin:$PATH"
opencode --version

Two quirks of the installer worth knowing: it exits non-zero even on success, and it installs to $HOME/.opencode/bin, not $HOME/.local/bin. The binary is about 185 MB, small enough for a quota-limited home directory.

10. Configure OpenCode

Write ~/.config/opencode/opencode.json. If the file already exists, merge into it rather than overwriting, and back it up first.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "vLLM (local L40)",
      "options": {
        "baseURL": "http://localhost:8000/v1",
        "timeout": false,
        "headerTimeout": 300000,
        "chunkTimeout": 300000
      },
      "models": {
        "qwen38-27b": {
          "name": "Qwen3.8-27B FP8 (local)",
          "tool_call": true,
          "reasoning": true,
          "temperature": true,
          "limit": { "context": 131072, "output": 32768 },
          "options": {
            "top_k": 20,
            "repetition_penalty": 1.0,
            "reasoningEffort": "low"
          }
        }
      }
    }
  },
  "model": "vllm/qwen38-27b",
  "small_model": "vllm/qwen38-27b",
  "agent": {
    "build": { "temperature": 1.0, "top_p": 0.95, "steps": 250 },
    "plan":  { "temperature": 1.0, "top_p": 0.95, "steps": 150 }
  },
  "tool_output": { "max_lines": 400, "max_bytes": 16384 },
  "compaction": { "auto": true, "prune": true, "tail_turns": 4 },
  "autoupdate": false,
  "share": "disabled",
  "subagent_depth": 1
}

And ~/.config/opencode/auth.json — vLLM does not validate the key, but the client requires one to be present:

mkdir -p ~/.config/opencode
echo '{ "vllm": { "type": "api", "key": "sk-local" } }' > ~/.config/opencode/auth.json
chmod 600 ~/.config/opencode/auth.json

The one gotcha that silently breaks sampling

In OpenCode's schema the model-level temperature field is a boolean capability flag meaning "this model accepts a temperature", not a value. Writing "temperature": 1.0 inside the model block does nothing at all. Sampling values belong on agent.<name>.temperature and .top_p, which is why they appear under agent.build and agent.plan above.

Why the other settings are there

Setting	Reason
temperature 1.0, top_p 0.95, top_k 20, repetition_penalty 1.0	The model card's recommended thinking-mode sampling. Greedy decoding on this family degenerates into repetition loops, so do not set temperature to 0 — get determinism from vLLM's seed instead. If you do see loops, the card's designated knob is presence_penalty, raised toward 2.
tool_output: {max_lines: 400, max_bytes: 16384}	The defaults (2000 / 51,200) let a single tool call consume ~13k tokens of context. This is the main way a long session's window silently disappears.
compaction: {prune: true}	prune defaults to false, which carries stale tool output forward indefinitely.
limit: {context: 131072, output: 32768}	The output budget is subtracted from context, and reasoning tokens are billed to output on this model. A small output cap returns content: null with the entire budget spent thinking.
Relaxed timeouts	At ~17 tok/s a long reasoning turn easily outlasts stock HTTP timeouts.
reasoningEffort: "low"	See §12. OpenCode forwards this as the OpenAI reasoning_effort field.

A note on context growth. Qwen3.8's chat template defaults preserve_thinking to on, so it retains thinking blocks from all historical messages rather than only the most recent turn. This is better for agentic continuity and KV-cache reuse, but it consumes the 128k window noticeably faster. If long sessions start compacting early, the lever is "chat_template_kwargs": {"preserve_thinking": false} in the provider options.

Optional: Python language-server feedback

OpenCode feeds LSP diagnostics back to the model after every edit. For a 27B model this is one of the largest available quality wins — it is the difference between the model catching its own bad call signature and shipping it into a long experiment run.

"lsp": {
  "python": {
    "command": ["/path/to/your/env/bin/basedpyright-langserver", "--stdio"],
    "extensions": [".py", ".pyi"]
  }
},
"formatter": {
  "ruff": {
    "command": ["/path/to/your/env/bin/ruff", "format", "$FILE"],
    "extensions": [".py"]
  }
}

Install ruff and basedpyright into the project's Python environment, not the vLLM virtual environment, which is for serving only. Add a pyrightconfig.json in the project root pinning venvPath and venv to that environment — without it the language server falls back to system Python and reports every third-party import as missing.

On Wright. OpenCode 1.18.25 is already installed and symlinked at ~/.local/bin/opencode. Both the qwen36-27b and qwen38-27b model blocks are present in opencode.json, so switching back to 3.6 is a one-line edit of model / small_model. The LSP and formatter point at the qwen conda environment (/home/export/ajshen/.conda/envs/qwen/bin/), which carries ruff 0.16.5 and basedpyright 1.39.10.

Verifying what actually reaches the server

Do not assume a configuration key reaches the wire — the boolean temperature trap above is the standing example of one that does not. A roughly 30-line logging proxy settles it in one run: capture the POST body, return HTTP 400 immediately, and read the capture. Short-circuiting is essential — OpenCode requests stream: true, and a proxy that buffers the response instead of streaming will simply hang.

Point baseURL at the proxy, run any prompt, then restore baseURL. A correctly configured deployment puts this on the wire:

{ "model": "qwen38-27b", "reasoning_effort": "low", "temperature": 1,
  "top_p": 0.95, "top_k": 20, "repetition_penalty": 1, "max_tokens": 32000,
  "stream": true }

11. End-to-end test

Run the first session in a scratch sandbox, never a live repository — OpenCode executes shell commands.

mkdir -p $WORK/sandbox && cd $WORK/sandbox
opencode run "create hello.py that prints the first 10 Fibonacci numbers, then run it"

opencode run is non-interactive, so this is scriptable and needs no TUI.

Pass criterion: hello.py exists on disk afterward and produced correct output. If the model describes creating the file but no file appears, tool plumbing is broken — return to §7 and test (c) in §8.

12. Controlling reasoning effort

Qwen3.8 introduces a reasoning_effort request field with three valid values, and understanding it correctly matters more than it first appears.

It is a prompt-level nudge, not a token budget. The chat template injects a different system instruction for each value, and that is the entirety of the mechanism. Measured by sending a fixed one-word message with max_tokens: 1 and reading prompt_tokens:

reasoning_effort	prompt_tokens	What the template emits
(omitted)	53	identical to xhigh — confirming xhigh is the default
xhigh	53	"think carefully through the task, validate assumptions…"
low	41	"keep your thinking brief and focused…"
medium	11	nothing at all — no system block is emitted
bogus	HTTP 400	the template raises; values are validated

Three things follow, and each one surprises people:

• Omitting the field is exactly xhigh, not a neutral middle. The template reads reasoning_effort|default('xhigh'). A deployment that never sets this field is running the most expensive setting.
• medium is not a midpoint — it is the absence of an instruction. The template branches only on xhigh and low; medium leaves the instruction empty, which suppresses the surrounding system block entirely. That is why it costs 42 fewer prompt tokens than xhigh rather than sitting between the two.
• Do not evaluate this knob by counting reasoning tokens. At temperature 1.0 the run-to-run variance swamps the effect, and measured results are non-monotonic — on one prompt xhigh produced the fewest reasoning tokens of the three. The prompt_tokens table above is the deterministic check; use it to confirm plumbing.

The effect that is operationally real is the tail, not the mean. On one hard prompt, xhigh ran to the full 8,192-token cap with every token spent in reasoning and returned zero-length content after roughly 475 seconds. The same prompt at low finished in 1,761 reasoning tokens with an answer. A runaway that produces no answer is far more costly than a slower one that does — and xhigh is the default.

Recommendation: set reasoning_effort: "low" for agentic use, and raise it deliberately for a hard one-off. Keep max_tokens generous regardless, so a runaway is truncated rather than silently returning empty content. Lower effort can cause retries, but a capped-out empty response is the worse failure.

13. Optional: MTP speculative decoding

The checkpoint ships with a multi-token-prediction draft head (mtp.safetensors, mtp_num_hidden_layers: 1), so speculative decoding needs no second model and has no version skew — vLLM shares the target model's embedding and lm_head weights with the drafter. Enable it by appending one flag to serve.sh:

  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The measured decode speedup is large: 2.93× (17.29 → 50.63 tok/s aggregate), from a paired benchmark with identical prompts and seeds, the server restarted between arms and the first request after each restart discarded. vLLM's own SpecDecoding metrics corroborate it independently — mean acceptance length 3.2–4.0 out of 4, average acceptance rate typically ~83%, and on verbatim code re-emission acceptance length 4.00 at a 100% rate. The L40's untuned FP8 kernels leave this model firmly memory-bound at batch size 1, which is exactly the regime where speculative decoding converts spare compute into wall-clock. The gain would be smaller on a better-fed card.

Three caveats before you enable it.

1. It is a decode-only speedup, and real agentic workloads are prefill-dominated. Measured across 82 assistant turns of actual use: 2,858,148 input tokens against 57,322 output tokens — a ratio of about 50:1, averaging 34,855 in / 699 out per turn. On a mean turn MTP moves generation from ~40 s to ~14 s, which is worth having, but the turn does not get 2.93× faster because the ~35k-token prefill is a fixed cost in both arms.
2. It costs KV cache. The draft head takes memory that would otherwise hold context, reducing concurrency at full context from roughly 1.6× to 1.15–1.40×. Given the restart-to-restart variance noted in §7, treat the magnitude as approximate.
3. There is an unresolved correctness question at long context. A normal agentic turn at 13k context once ended with a reasoning part, no text part at all, and finish=stop — the model produced degenerate output (malformed tags, a verbatim three-times repetition) and no answer. Server crash, OOM, and client-side abort were all ruled out; MTP is the leading suspect but was not confirmed. Speculative decoding is distribution-preserving by construction, so if disabling it changes output quality that is a vLLM bug rather than a tradeoff — but the preservation property was only ever verified on short prompts.

Because of (3), MTP is currently disabled by default in this deployment. If you want the throughput, enable it deliberately and watch for empty turns in long sessions.

Two further practical notes: min_p and logit_bias are silently disabled under speculative decoding (vLLM warns once at startup), and toggling MTP forces a torch.compile recompile of about 60 seconds, so do not flip back and forth casually.

14. Restarting after a new allocation

Compile cache, weights, and virtual environment all survive a restart as long as scratch survives, so this is fast. Save this as $WORK/restart.sh — it locates $WORK from its own path, so it works from any directory with no environment setup, and it is idempotent.

#!/usr/bin/env bash
# Restart the Qwen3.8-27B vLLM server. Safe to re-run: no-op if already up.
set -uo pipefail

export WORK="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$WORK/env.sh"
SESSION=vllm38

if curl -s -m 5 -o /dev/null http://localhost:8000/v1/models; then
  echo "Already up: $(curl -s http://localhost:8000/v1/models \
    | python3 -c 'import json,sys;d=json.load(sys.stdin)["data"][0];print(d["id"],d["max_model_len"])')"
  exit 0
fi

# Weights and venv live on node-local scratch, wiped at job end. Fail loudly, not weirdly.
[ -x "$WORK/.venv/bin/vllm" ] || { echo "FATAL: venv gone ($WORK/.venv). Re-run sections 4, 5."; exit 1; }
[ -d "$WORK/hf/hub" ]         || { echo "FATAL: weights gone ($WORK/hf). Re-run sections 4, 5."; exit 1; }

used=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1)
if [ "$used" -gt 1000 ]; then
  echo "WARNING: GPU already holds ${used} MiB (another server?). Needs ~41 GiB free."
  nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
fi

tmux kill-session -t "$SESSION" 2>/dev/null
tmux new -d -s "$SESSION" "$WORK/serve.sh 2>&1 | tee $WORK/vllm.log"
echo "Launched tmux session '$SESSION'. Waiting for readiness (warm cache: ~1 min)..."

for _ in $(seq 1 90); do
  if curl -s -m 5 -o /dev/null http://localhost:8000/v1/models; then
    echo "UP"
    grep -E "GPU KV cache size" "$WORK/vllm.log" | tail -1
    exit 0
  fi
  # Liveness, not just readiness: a dead process must not look like a slow one.
  tmux has-session -t "$SESSION" 2>/dev/null || { echo "FAILED - server exited. Last lines:"; tail -20 "$WORK/vllm.log"; exit 1; }
  sleep 10
done
echo "TIMED OUT after 15 min - see $WORK/vllm.log"; exit 1

The liveness check inside the wait loop is the part that earns its keep: a fixed sleep 300 cannot tell "still loading" from "died four minutes ago", and this script reports a crash in seconds with the last 20 log lines instead of looking like a slow load for fifteen minutes.

If scratch was wiped, the script tells you which piece is missing. Re-run §4 and §5; the download is the long pole, so start it first and set HF_TOKEN if you have one. If you would rather not re-download every allocation, keep the weights and virtual environment on persistent project space and only the compile cache on local NVMe.

On Wright. One command after a new allocation:

/mnt/scratch/$USER/qwen38-l40s/restart.sh

Verified across a real allocation change on 2026-09-04: /mnt/scratch survived intact — 29 GB of weights, the virtual environment, and the warm torch.compile cache — and only the tmux session and server process were gone. Recovery took about 40 seconds. Scratch survives because the new job landed on the same node; that is not guaranteed.

15. Connecting from a laptop (optional)

# On the compute node:
echo "$(hostname):8000" > $WORK/endpoint.txt

# On the laptop, tunnelling through the login node:
ssh -N -L 8000:<compute-node>:8000 -J <user>@<login-node> <user>@<login-node>

Then point the laptop's opencode.json baseURL at http://localhost:8000/v1. Some clusters block direct SSH to compute nodes; if so, the tunnel must terminate on the login node.

vLLM has no authentication by default. If you expose the port beyond a private SSH tunnel — a VS Code forwarded port set to "Public", for instance — add --api-key to serve.sh first. Public means internet-reachable.

16. Troubleshooting

Symptom	Cause	Fix
ValueError: max_num_seqs (256) exceeds available Mamba cache blocks (N)	The most likely startup failure. Linear-attention state needs one block per concurrent sequence	Set --max-num-seqs 8. Do not lower --max-model-len — wrong axis (§6)
Server fails to allocate memory at startup	Another inference server is holding the card	Two copies of this model cannot coexist on 48 GB. Stop one; confirm nvidia-smi reads ~0 MiB
Other ValueError about KV cache size	--max-model-len genuinely too large	Halve it — but check the Mamba-block error first, it is far more likely
Startup takes ~4 minutes the first time	Inductor compilation, normal	Cached afterward. --enforce-eager skips it at a throughput cost
Recompiles on every launch	You changed a flag in the compile cache key (--max-num-seqs, --max-model-len, speculative config)	Expected. Settle the flags and the cache sticks
Model responds but never calls tools	Wrong --tool-call-parser, or max_tokens too low on the test	Use qwen3_coder (§7); raise max_tokens to 512+
Reasoning text leaking into content	Missing or wrong --reasoning-parser	Set qwen3
Very slow short responses	reasoning_effort defaults to xhigh	Set low per request (§12). It is a request field, not a server flag
A request returns empty content after a very long wait	Reasoning ran to the max_tokens cap; the whole budget was spent thinking. finish_reason is length	Use reasoning_effort: "low" and keep max_tokens generous (§12)
A turn ends with no output at all, no error, session healthy	Model emitted a reasoning part and no text part, then finish=stop. Root cause open	Distinct from the runaway above (which ends finish=length). See §13 caveat 3; try disabling MTP
Context fills faster than expected in long sessions	preserve_thinking defaults on, retaining all historical thinking blocks	Keep compaction.prune: true, or disable via chat_template_kwargs (§10)
KV pool differs between launches of identical flags	Observed, up to ~22%, cause not isolated	Average several restarts before concluding anything (§7)
min_p / logit_bias silently ignored	Disabled under speculative decoding	Expected with MTP; drop them or drop MTP (§13)
JSONDecodeError reading a downloaded config.json	tee f.json \| head truncates the file at the pipe buffer — head exits, tee takes SIGPIPE. This file is ~51 KB	Download with curl -o, then inspect
GET /v1 returns {"detail":"Not Found"}	Not a bug. /v1 is a path prefix, not a route	Test /v1/models. Client baseURL should still end in /v1
Remote client gets a 401 or an HTML login page	A forwarded port is set to private	Make it public — but add --api-key first (§15)
opencode run hangs in a directory that worked before	Per-directory session state poisoned by an earlier aborted run	Run in a fresh directory
OpenCode hangs against a debugging proxy	The proxy buffers instead of streaming; OpenCode requests stream: true	Short-circuit with a 400 after capturing (§10)
Your own shell dies during cleanup	pkill -f "vllm serve Qwen..." matches your own command line, which contains that string	Kill by tmux session (tmux kill-session -t vllm38), or use pkill -f "[v]llm serve"
Server dies when the shell closes	Not running under tmux	§7
Everything dies at a fixed time	SLURM walltime expired	Request a longer allocation
Failed to import from vllm._rocm_C	CUDA build, ROCm absent	Ignore
Using default W8A8 Block FP8 kernel config	vLLM ships no tuned FP8 kernel config for the L40	Real but modest throughput cost; no user-space fix

Where the logs are

What	Where
vLLM stdout and stderr	$WORK/vllm.log, also live in tmux (tmux attach -t vllm38)
OpenCode client	~/.local/share/opencode/log/opencode.log
The model's actual output	~/.local/share/opencode/opencode.db (SQLite)

vllm.log tells you whether the server misbehaved — request status, throughput, speculative-decoding metrics, KV usage. opencode.log tells you what the client did with the response: look for exiting loop (a clean end), error=Aborted (preceded by message=cancel if the user cancelled), and step=N to count agent turns.

Neither log contains the generated text. For that, query the database:

db=~/.local/share/opencode/opencode.db

# Recent assistant turns: finish reason and token split
sqlite3 "$db" "SELECT id, json_extract(data,'\$.finish'),
  json_extract(data,'\$.tokens.output'), json_extract(data,'\$.tokens.reasoning')
  FROM message WHERE json_extract(data,'\$.role')='assistant'
  ORDER BY time_created DESC LIMIT 10;"

# The reasoning and text parts of one message
sqlite3 "$db" "SELECT data FROM part WHERE message_id='msg_...' ORDER BY time_created;"

Trap: messages live in the message table. There is also a session_message table with an identical-looking schema, and it is empty — querying it returns nothing and looks like a missing session rather than a wrong table.

A turn that produced no answer shows up as a message row whose part rows include a reasoning part but no text part. That distinction is invisible from either log file and is the entire diagnosis for the empty-turn failure in §13.

17. Performance expectations

Metric	Value
Decode throughput, no speculative decoding	~17.3 tok/s
Decode throughput with MTP	~50.6 tok/s (2.93×, decode only — see §13)
Prefill throughput	~2,400–3,200 tok/s, falling as context grows
Real agentic workload shape	~50:1 prefill-to-decode; ~34,855 in / 699 out per turn

Set expectations accordingly: short interactions feel disproportionately slow because reasoning-token overhead is a fixed cost, and long ones amortize much better. Prefill is compute-bound here rather than weight-bound, which is why --max-num-batched-tokens makes no measurable difference (§7) — a 2048-token chunk against a 27B model is already a large enough GEMM to saturate the card.

Do not describe this as a 27B dense model. 48 of its 64 layers use linear attention, so long-context scaling is considerably better than a dense model of this size would suggest.

18. Reference deployment

Alex Shen (ajshen) has a successful working deployment. Feel free to contact him with any questions.

Node / GPU	n01 (Wright), 1 × NVIDIA L40, 46,068 MiB, driver 595.84
Work directory	/mnt/scratch/ajshen/qwen38-l40s
Software	Python 3.12.11, vLLM 0.28.0, torch 2.13.0+cu130, transformers 5.16.1
Checkpoint	Qwen/Qwen3.8-27B-FP8, 29 GB, no on-the-fly quantization
Parsers	qwen3_coder (tool), qwen3 (reasoning)
Context / concurrency	131,072 / --max-num-seqs 8
KV cache	13.59 GiB, 213,180 tokens, 1.63× at full context
GPU memory in use	42,877 / 46,068 MiB
Speculative decoding	Disabled (see §13)
Client	OpenCode 1.18.25, vllm/qwen38-27b
tmux session	vllm38

19. Known-untested

Honest gaps in what has been verified, so nobody re-derives them or trusts them too far:

• Whether MTP causes the degenerate empty turn in §13. The replay with MTP disabled has not been run. This is the most important open question here.
• --max-num-batched-tokens 8192 vs 2048 under MTP. The measurement showing no difference was made without speculative decoding, and vLLM explicitly warns at startup that the interaction matters.
• num_speculative_tokens values other than 3.
• Long-context behaviour at the full 131k, and MTP acceptance rates there. All throughput measurements used short prompts, and the first contact with this gap was a failure (§13).
• Whether the 22% KV-pool variance across restarts has a cause worth controlling.