Step-by-step guide based on the real installation of KAT-Coder-V2.5-Dev-APEX-MTP in ollama
Before installing anything: evaluate if the model is worth it (https://huggingface.co/gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF)
Not all Hugging Face (HF) repos are created equal. Before downloading several GB, it’s worth checking:
| Good sign | Alarm sign |
|---|---|
| Author recognized in the community (bartowski, unsloth, etc.) or base model with a track record | New account, README with marketing-style banners/emojis |
| Decent sample size benchmarks and clear methodology | “Research preview” + benchmarks with n=15 without independent comparison |
| SHA256 checksums, build manifest, tensor source explained | No verification, no clear credits |
| Clear license (Apache-2.0, etc.) | |
| Other quantizers recognized have already quantized the same base model | This is the only repo for that specific model |
Tips: If a model has a Bartowski (or similar) quantization, it’s a strong indication that the base model is legitimate and of real interest — Bartowski quantizes almost all relevant content. Quickly validating a new model by searching for bartowski/<model-base-name>-GGUF on HF is one way to do so…
If the HF repository uses standard quantization names (Q4_K_M, IQ3_M, etc.), this is enough:
bash
docker exec -it ollama ollama run hf.co/usuario/repositorio-GGUF:Q4_K_M
Ollama downloads the file, detects metadata and starts. End.
If the repo uses non-standard quantization names (for example I-Compact-v2D-lite, typical of custom methods like APEX), Ollama returns:
Error: pull model manifest: 400: {"error":"The specified tag is not a valid quantization scheme. Please use another tag or \"latest\""}
This happens because Ollama tries to map the tag against its closed list of known schemes and does not recognize the name.
Solution: use the full file name .gguf as a tag, not just the short name:
docker exec -it ollama ollama pull hf.co/usuario/repositorio-GGUF:Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact-v2D-lite.gguf
With the full filename, Ollama treats it as a direct reference to the file instead of trying to interpret a scheme, and resolves without issues.
docker exec -it ollama ollama list
Confirm that it appears with the full name:
NAME ID SIZE hf.co/usuario/repositorio-GGUF:Kwaipilot_KAT-Coder-...I-Compact...gguf ec643c3fa4c7 17 GB
3. The step that often fails: the chat template
Many GGUF —especially those coming from merges, custom injections (MTP grafting, etc.) or manual conversions— do not correctly embed the chat_template metadata. Always check it:
bash
docker exec -it ollama ollama show hf.co/usuario/repositorio-GGUF:archivo.gguf --modelfile
Symptoms of the problem
If you see this:
TEMPLATE {{ .Prompt }}
…means that Ollama has no chat template applied. The model will receive raw messages, without system/user/assistant tags, and incorrect stop tokens. It may work partially because the model infers the format from its training data, but you won’t have real control over turns or clean terminations.
Create a new Modelfile by pointing to the local blob download (so you won’t download anything else — take the path from the FROM given by ollama show --modelfile earlier):
bash
cat > Modelfile.miModelo << 'EOF' FROM /root/.ollama/models/blobs/sha256-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant <think> """ PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>" PARAMETER num_ctx 32768 PARAMETER temperature 0.85 PARAMETER top_p 0.9 EOF
Copy it to the container and create with a clean and short name:
bash
docker cp Modelfile.miModelo ollama:/root/.ollama/Modelfile.miModelo docker exec -it ollama ollama create nombre-corto -f /root/.ollama/Modelfile.miModelo
Verify that it was applied:
bash
docker exec -it ollama ollama show nombre-corto --modelfile
5. Test it
bash
docker exec -it ollama ollama run nombre-corto --verbose "your test prompt"
--verbose gives you tokens/second at the end, useful for comparing performance between models.
Things to watch out for in the first test:
