1. Establish the memory budget
An 8B model has roughly 16GB of FP16 weights before KV cache and runtime overhead. A 24GB RTX 4090 can be a starting point for inference, not a guarantee for every context length or batch. Start with a short context and one request; measure peak memory.
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv2. Install in an isolated environment
Confirm the vLLM release supports your driver and CUDA runtime. Use a virtual environment, then record and pin the tested versions. Download a licensed model to /srv/models/llama-3-8b; gated models may require provider access approval.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install vllm
python -m pip freeze > requirements-tested.txt3. Start a local inference endpoint
Keep the endpoint on localhost and place an authenticated TLS reverse proxy in front of it. Adjust the model, dtype and maximum context for your tested memory budget. Do not expose port 8000 directly to the internet.
vllm serve /srv/models/llama-3-8b --host 127.0.0.1 --port 8000 --dtype half --max-model-len 2048 --gpu-memory-utilization 0.854. Measure performance and cost
Record prompt length, output tokens, concurrency, time to first token, tokens/second, peak VRAM and wall time. At a reference $1.20/hour, a 45-minute run costs $0.90 for GPU time only; add storage, transfer, idle time and billing-rounding charges. This is a calculation, not a measured benchmark or a live quote.