Ollama Local Model Setup & Performance Troubleshooting
A targeted diagnostic prompt to pinpoint and resolve slow inference, CUDA/ROCm acceleration issues, Modelfile parameters, and context window limits in Ollama.
Customize & Generate Your Prompt
Adjust the variables below. Your customized prompt updates live in real-time ready to copy with zero setup.
Please help me diagnose and fix an issue with my local Ollama model deployment. ### Environment & Hardware Setup - **Model Name & Quantization**: llama3.3:70b-instruct-q4_K_M - **Hardware & VRAM**: NVIDIA RTX 4080 (16GB VRAM) + 64GB DDR5 System RAM - **Operating System**: Windows 11 (Ollama Windows Native) - **Observed Problem or Error Message**: ```text Inference is extremely slow (around 1.5 tokens per second). When I ask questions with long code files, the model stops responding or crashes with "error: out of memory (CUDA error: out of memory)". ``` ### Diagnostic Requirements 1. **Root-Cause Analysis**: - Explain why this error or performance degradation occurred on this specific hardware combination. - Confirm whether the model fits entirely into VRAM or is partially offloading layers to system RAM. 2. **Terminal Remediation Steps**: - Provide the exact shell commands (PowerShell, Bash, or Zsh) to verify GPU detection, check Ollama server status, and restart the daemon. 3. **Optimized Modelfile Configuration**: - Provide a customized `Modelfile` with optimal parameters (`num_ctx`, `num_gpu`, `num_thread`, `temperature`) tuned for this hardware. 4. **Benchmarking & Validation**: - Provide a quick `curl` command to test API response latency and calculate tokens-per-second output.
How to Use & Customize This Prompt
Fill in your target framework, environment rules, or source metrics in the interactive builder above. The assembled template updates instantly.
Click "Copy Prompt to Clipboard". You can copy the full assembled text or copy system instructions separately.
Refer to the Model Guidance below to choose the highest-performing frontier model or local runtime for this specific task.
Paste into your AI coding tool, chat interface, or local Ollama runtime. Verify syntax, types, and schema compliance against expected output.
Workflow Guidance
- Enter your exact model name and your hardware specifications (GPU model and VRAM).
- Paste your error output or describe the slow token generation rate.
- Follow the step-by-step diagnostic commands to configure GPU offloading and optimize your Modelfile.
Representative AI Output
Sample output produced when using frontier AI models with the default parameters:
### Root Cause A 70B model quantized at Q4_K_M requires approximately 43GB of memory. On a 16GB RTX 4080, only ~20-25 of the 80 layers fit into VRAM. The remaining 55+ layers are evaluated in system RAM over the PCIe bus, resulting in 1-2 tokens/sec. When context size (`num_ctx`) expands, the KV cache pushes the GPU past its 16GB threshold, triggering CUDA OOM. ### Recommended Fix: Switch to Qwen2.5-Coder:32b or 14b For full GPU acceleration, use a model that fits completely in your 16GB VRAM: ```bash ollama run qwen2.5-coder:14b ``` ### Custom Modelfile with Safe Context ```dockerfile FROM llama3.3:70b-instruct-q4_K_M PARAMETER num_ctx 4096 PARAMETER num_gpu 22 PARAMETER temperature 0.6 ```
Pro Optimization & Edge-Case Tips
- Check how many layers fit into VRAM in real-time by running `ollama ps` while a query is processing.
- Always leave at least 1.5GB to 2GB of VRAM headroom for Windows OS desktop compositing and browser hardware acceleration.
- If you need long context (32K+), look into flash attention and Q4_0 KV cache compression flags.
Recommended Runtimes & Models for Local AI Hardware
Self-hosted inference engines and quantized open-weight models optimized for consumer GPUs.
Ollama Runtime
Local terminal setup, one-click model pulling, and background API serving on macOS, Linux, and Windows.
- Seamless Modelfile configuration for custom prompt templates
- Automatic GPU layer offloading based on available VRAM
- Native OpenAI-compatible HTTP endpoint for local web UIs
vLLM / llama.cpp
Production high-throughput serving, PagedAttention, and custom GGUF/EXL2 quantization.
- PagedAttention eliminates KV cache memory fragmentation
- Continuous batching supports concurrent API requests smoothly
- Unlocks custom flash-attention kernels for fast token generation
LM Studio
Visual model discovery from Hugging Face, side-by-side token benchmarks, and local GPU monitoring.
- Real-time VRAM allocation bar and token generation counter
- Zero-setup Hugging Face model search and direct GGUF downloading
- Preset temperature and top-p profiles for code vs chat
Copy your assembled prompt and test immediately with approved frontier models or your own API key.
Also Checkout
Verify AI-generated media, inspect deep image/video provenance, check C2PA cryptographic manifests, and strip sensitive EXIF tags with our 100% private, client-side tools.
AI Media Detector
Verify whether an image or video is AI-generated, check C2PA cryptographic manifests, and detect synthetic provenance markers.
Photo Metadata Viewer
Inspect deep EXIF, IPTC, and XMP metadata, view camera optics and GPS coordinates, and strip metadata to protect your privacy.
Related Tools on SPAUL Hub
More in Local AI & Hardware
View CategoryGPU VRAM & Local LLM Quantization Planner Prompt
Plan your local AI hardware footprint with precision. Calculate weights, KV cache consumption across context sizes, and determine the exact quantization level required for full GPU offload.
Hugging Face Model Benchmarking & Evaluation Plan Prompt
Create an empirical evaluation pipeline to compare candidate models from Hugging Face. Measure Time to First Token (TTFT), token throughput, reasoning accuracy, and JSON schema adherence.