Cloud VPS / 01Size a cloud server
Estimate CPU, memory, storage and transfer from the workload, then budget headroom.
Procedure
- Measure peak concurrency, request duration, CPU utilization and resident memory. Budget separately for the database, cache and application runtime rather than counting websites.
- Include databases, media, logs, snapshots and growth in storage capacity. A 10Mbps port has a theoretical 1.25MB/s ceiling before protocol and path overhead.
- Load-test representative requests and inspect p95 latency, swapping and disk wait. Address slow queries and caching before choosing a CPU, RAM or bandwidth upgrade.
- Create a resource budget with measured peaks, expected growth, latency targets and upgrade triggers. Image sites emphasize storage and transfer; transactional databases emphasize cache hit rates and synchronous writes. Equal traffic does not imply equal sizing.
Inspection commands
free -h
df -h
vmstat 1 5
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Cloud VPS / 02Verify routing in both directions
Compare CN2 GIA, CMI, PCCW or international BGP using consistent endpoints and time windows.
Procedure
- Request a test address for the exact product. Record client location, ISP, timestamp and protocol. A public regional reference IP does not establish the route used by your instance.
- Capture an outbound MTR from the client and a reverse trace from the server or its Looking Glass. AS4809 and 59.43 hops are clues, not proof of complete bidirectional GIA routing.
- Compare peak and off-peak end-to-end loss, jitter and HTTPS timings. Intermediate routers may rate-limit ICMP; loss at an intermediate hop alone does not establish application packet loss.
- Store measurements by ISP, timestamp and direction alongside application timings. Routing can change with interconnection policy or failover; one pre-purchase traceroute cannot establish long-term experience for every user.
Inspection commands
mtr --report --report-cycles 20 YOUR_TEST_IP
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Cloud VPS / 03Launch WordPress with HTTPS
Take a site from DNS setup to verified certificate renewal.
Procedure
- Point A/AAAA records to the actual addresses; publish AAAA only when working IPv6 is configured. Install supported web-server, PHP and database versions and create a dedicated database user.
- Download WordPress from its official source and configure site and administrator settings. Grant the web process write access only where needed; keep database ports private.
- Configure an ACME client and HTTPS, checking the required ports or DNS permissions for validation. Test automatic renewal, then redirect HTTP once HTTPS works.
- Test login, uploads, email and checkout. Back up the database and uploads off-host and complete a restoration drill before moving production traffic.
- After launch, verify permalinks, upload limits, scheduled tasks and cache rules. Do not cache carts, authenticated sessions or admin pages as anonymous content. Exercise both signed-in and anonymous flows after enabling a CDN or page cache.
Inspection commands
curl -I https://YOUR_DOMAIN
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Dedicated servers / 04Accept dedicated hardware and RAID
Verify delivered specifications, disk health and management access.
Procedure
- Compare CPU model and count, usable RAM, and disk type and count with the order. Distinguish physical cores from threads and raw capacity from usable array capacity.
- Confirm RAID level, hot spares and protected controller cache. RAID 0 has no redundancy; RAID 1/10 and RAID 5/6 have different capacity and failure tolerances. RAID is not a backup.
- Check health alerts, degraded arrays and rebuild status; record serial numbers and acceptance results. Perform stress tests before production and protect data before destructive disk tests or rebuilds.
- Keep an acceptance record separating confirmed and untested items. Port speed does not guarantee throughput to every destination, and disk models alone do not establish array performance. Test representative workloads and record competing jobs.
Inspection commands
lscpu
free -h
lsblk -o NAME,SIZE,TYPE,MODEL
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Dedicated servers / 05IPMI and out-of-band recovery
Check remote console access, boot media and recovery paths.
Procedure
- Confirm the management address, credential delivery and allowed source networks. Use an isolated management network or VPN, restrict public access and replace default credentials.
- During a maintenance window, test video, keyboard and virtual media. Power operations and boot-order changes interrupt service; drain traffic and confirm backups first.
- Document disk recovery and escalation contacts. When networking fails, inspect interfaces, routes and firewall rules from the console before deciding whether a reinstall is necessary.
- Identify the physical host and service owner before recovery. Distinguish graceful shutdown, hard power-off and reinstall consequences. Console access does not prove network recovery; verify services, mounts and databases, then detach temporary installation media.
Virtualization hosts / 07Plan hypervisor capacity
Budget RAM, storage IOPS, CPU and networking rather than counting VMs.
Procedure
- Reserve RAM for the host, filesystem cache and management services. Sum guest peak working sets instead of allocating all physical memory to tenants.
- Choose CPU overcommit based on workload and observe host load, guest steal time and scheduling latency. Quiet websites and latency-sensitive databases require different assumptions.
- Budget random IOPS, synchronous writes and backup windows. Alert on capacity exhaustion, thin-provisioning growth and NIC congestion; prepare migration and rollback before upgrades.
- Include maintenance and failure headroom in capacity planning. Check destination CPU compatibility, RAM, storage access and networking before planning guest moves. Having a backup does not provide spare migration capacity.
GPU compute / 08Size GPU memory for a model
Model weights are only part of the memory budget.
Procedure
- Distinguish inference, LoRA and full training. A 7B model’s FP16 weights alone require about 14GB in decimal units, before framework and runtime overhead; this does not establish that a 16GB card will fit.
- Inference also needs context, concurrency and KV cache. Training adds gradients, optimizer states and activations. Quantization reduces weights but requires quality and operator-compatibility checks.
- Multiple GPUs do not automatically become one memory pool. Confirm model parallelism, PCIe/NVLink topology and framework support, then measure peak usage and throughput at the target batch size.
- Record input length, concurrency, precision and peak memory for each run. Quantized weights can still have temporary loading peaks. For training, test checkpoint saving and resumption before committing to a long run.
Inspection commands
nvidia-smi
nvidia-smi topo -m
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
GPU compute / 09Validate CUDA and PyTorch
Check the driver before installing a compatible runtime and testing GPU execution.
Procedure
- Run nvidia-smi to inspect the device, driver and active processes. Its CUDA version describes driver support and does not necessarily identify an installed CUDA Toolkit.
- Select a compatible build using official PyTorch instructions and install in an isolated environment. Containers also need NVIDIA Container Toolkit and device access. Avoid mixing unverified driver packages.
- Verify torch.cuda.is_available(), the device name and a small tensor operation before model dependencies. Record driver, PyTorch, runtime and model versions for reproducibility and rollback.
- Device discovery is only the first check: run a small real inference or training job and retain its output. For operator or shared-library failures, inspect environment and dependency versions instead of repeatedly overwriting packages. Export a lockfile and pin working images.
Inspection commands
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'No CUDA device')"
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Operations & security / 10Backups, restoration and launch checks
Aim for recoverable data and a measured recovery time, not merely a successful backup job.
Procedure
- Define acceptable data loss (RPO) and recovery time (RTO). Use application-consistent database backups or supported snapshots rather than copying a live data directory.
- Keep encrypted backups on separate infrastructure or in another region with retention and least-privilege access. Document key custody and alert on failed jobs, capacity and expiry.
- Restore a recent backup in isolation, verify records, files, permissions and application flows, and record elapsed time. Confirm monitoring, certificate renewal, escalation contacts and rollback before launch.
- Compare backup intervals with the RPO and total download, decryption, import and validation time with the RTO. Verify database-to-upload consistency. Successfully extracting an archive does not demonstrate application recovery.
Operations & security / 11SSH keys and connection troubleshooting
Set up reliable administrative access and distinguish transport failures from authentication errors.
Procedure
- Generate a passphrase-protected SSH key locally and install only its public key in the account’s authorized_keys. Keep the private key off tickets and shared directories; issue separate keys to each administrator.
- Verify the host fingerprint through the console or another trusted delivery channel before the first connection. Investigate fingerprint changes after a rebuild rather than disabling host verification.
- Keep the current session open while testing the new account, key and required sudo access in a second terminal. Restrict password access only after testing and confirming a recovery console is available.
- For timeouts, check instance state, IP, port, security groups and the client network. Connection refused calls for checking the service listener. Authentication failures require checking the user, key, permissions and server logs.
- Validate changes with sshd -t before reloading through the system service manager. Retain the old session and test a fresh connection; restore through the console if validation fails.
Inspection commands
ssh -vv YOUR_USER@YOUR_SERVER_IP
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Cloud VPS / 12Troubleshoot DNS and HTTPS
Separate DNS, connectivity, certificate and application failures.
Procedure
- Record the full hostname, time, client network and browser error. The apex and www can have different records; a DNS control panel alone does not prove that recursive resolvers have updated.
- Query A, AAAA and CNAME separately. If IPv4 works but some clients fail, verify the IPv6 service behind AAAA. Cached answers can remain until their TTL expires after a change.
- Confirm whether the expected destination is an origin or a CDN. Proxied DNS normally returns edge addresses. Preserve the correct Host and SNI when testing an origin instead of judging its certificate by an IP-only request.
- Check hostname coverage, validity, system time and intermediate certificates. Align application and reverse-proxy HTTPS redirects to prevent loops between layers.
- For 502/504, inspect upstream listeners, proxy logs and application timing; for 403, inspect access rules and permissions. Retest the home page, login, assets and renewal jobs from multiple networks.
Inspection commands
dig YOUR_DOMAIN A
dig YOUR_DOMAIN AAAA
curl -I https://YOUR_DOMAIN
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Cloud VPS / 13Deploy and roll back Docker services
Manage images, persistent data, network exposure and rollback separately.
Procedure
- Use supported operating systems and official package sources. Record Compose and image architecture; ARM and x86 images are not automatically interchangeable. Pin a version or digest instead of relying on latest.
- Store databases and uploads in explicit persistent volumes, documenting paths, ownership and backup procedures. Rebuilding a container does not rebuild its data; avoid keeping unique data only in its writable layer.
- Publish only required ports and keep databases on internal networks. Docker port publication can affect firewall behavior, so verify exposure from an external network rather than trusting a firewall UI alone.
- Configure health checks, resource limits and log rotation. A running container may not be ready; verify dependencies, migrations, permissions and data persistence across restarts.
- Retain the previous image and configuration and rehearse rollback. Database schema changes may be incompatible with old code, so reverting an image is not sufficient; back up and plan recovery before irreversible migrations.
Inspection commands
docker version
docker compose ps
docker stats --no-stream
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Operations & security / 15Migrate websites and plan DNS cutover
Synchronize data in a controlled window and avoid conflicting writes.
Procedure
- Inventory domains, databases, uploads, scheduled tasks, email, callbacks and allowlists. Check runtime compatibility before migrating and avoid combining the move with several major-version upgrades.
- Lower DNS TTL in advance and allow the previous TTL period to pass. A lower setting cannot evict existing cached answers immediately; include CDN, proxy and application caches in the plan.
- Restore an initial backup and test using a staging hostname or local hosts override. Disable live payments, email delivery, queue consumers and duplicate jobs in staging to prevent real side effects.
- Pause writes during the maintenance window, take an application-consistent database backup and perform the final file sync. Verify data, permissions and certificates before switching traffic; changing DNS alone can split orders and uploads.
- Retain the old environment while observing logs, errors and critical workflows. If the new site has accepted writes, reconcile those before rollback rather than simply pointing DNS back. Retire old resources only after acceptance.
GPU compute / 16Troubleshoot GPU OOM and inference latency
Reproduce failures with a small workload and isolate memory, concurrency and input bottlenecks.
Procedure
- Record model, precision, context length, batch size, concurrency and the full error. Reproduce with one short request, then increase one variable at a time; model file size is not the runtime memory requirement.
- Use nvidia-smi to find competing processes, then compare framework allocated, reserved and peak statistics. Releasing cached blocks cannot shrink live tensors that exceed available memory.
- For inference, reduce concurrency, batch or context before evaluating quantization and KV-cache policies. Training may benefit from gradient accumulation or activation checkpointing. Recheck quality, throughput and latency after each change.
- For low utilization, inspect CPU preprocessing, storage, data loading and host-to-device transfers. Separate cold loading or compilation from warmed-up time to first token and steady generation speed.
- Run representative inputs while measuring p95 latency, peak memory and errors. On multiple GPUs, verify actual placement, topology and communication waits. Save the stable configuration and software versions as a baseline.
Inspection commands
nvidia-smi
nvidia-smi topo -m
Run these checks on the relevant host. Replace YOUR_TEST_IP, YOUR_DOMAIN, YOUR_USER and YOUR_SERVER_IP with actual values and ensure the required tools are installed.
Cloud VPS / 17Choose a server region
Map users, ISPs, databases and external APIs
Procedure
- Map users, ISPs, databases and external APIs. Cross-region calls between application and database can matter more than home-page latency.
- Compare equivalent plans with the same requests and time windows. Record p50/p95 latency and failures, then check residency requirements, backup locations and availability.
- Pilot noncritical services or a small traffic share before expanding. Plan database consistency and rollback separately; changing the website DNS is not a complete migration.
Cloud VPS / 18Budget bandwidth and transfer
Distinguish port speed, committed bandwidth and monthly transfer allowances
Procedure
- Distinguish port speed, committed bandwidth and monthly transfer allowances. Divide Mbps by eight for theoretical MB/s before overhead; check inbound and outbound billing rules.
- Estimate transfer from response size times request count and account for downloads, backups and crawlers. A saturated 100Mbps link theoretically transfers 1.08 decimal TB per day; do not assume sustained line rate.
- Alert on daily and monthly totals and confirm overage behavior. Diagnose slow downloads across connection limits, disk throughput, routing and CDN cache hits.
Cloud VPS / 19Choose an operating system
List application runtime, kernel and architecture requirements before choosing an OS
Procedure
- List application runtime, kernel and architecture requirements before choosing an OS. Prefer a release receiving security maintenance rather than following an outdated tutorial onto an unsupported version.
- Document repositories, package management, firewall and networking tools. Service names differ across distributions; validate boot, dependencies, monitoring and backup agents on a test instance.
- Test restoration before a major upgrade and schedule maintenance. Reinstallation can destroy system-disk data; confirm retention behavior instead of treating a rebuild as a routine update.
Cloud VPS / 20Nginx reverse proxy and 502 errors
Verify the application listener before configuring the proxy
Procedure
- Verify the application listener before configuring the proxy. Bind private services to loopback or a private interface and forward the original host and protocol appropriately after TLS termination.
- Validate with nginx -t before reloading. For 502 inspect refused connections, socket permissions and crashed upstreams; for 504 correlate application timings instead of endlessly raising timeouts.
- Exercise uploads, redirects, client-IP handling and WebSockets. Trust forwarded headers only from known proxies so arbitrary client input cannot spoof an identity used by access controls.
Cloud VPS / 21MySQL deployment and slow queries
Use a dedicated application account with restricted sources instead of an administrator account
Procedure
- Use a dedicated application account with restricted sources instead of an administrator account. Limit listening interfaces and firewall exposure and protect remote connections when required.
- Correlate request latency with slow-query records and inspect plans with EXPLAIN. Evaluate indexes and scanned rows; every extra index also adds write and storage cost.
- Record a baseline before changing pools, caches or queries and retest comparable loads. Verify consistent backups and monitor connection exhaustion, storage and replication delay, not merely process uptime.
Cloud VPS / 22Redis memory and eviction
Decide whether Redis holds disposable cache, sessions or critical queues
Procedure
- Decide whether Redis holds disposable cache, sessions or critical queues. These need different durability and eviction policies; restrict networking and configure supported authentication and authorization.
- Leave headroom for persistence, replication buffers and the OS when setting maxmemory. Match eviction to TTL usage and define application behavior when no eligible keys can be evicted.
- Track hit rate, evictions, slow commands and large keys before adding memory. Test database load after a cold cache and restore procedures for data that must persist.
Cloud VPS / 23Connect a CDN to an origin
Validate the origin hostname, certificate and application before enabling the proxy
Procedure
- Validate the origin hostname, certificate and application before enabling the proxy. Define origin TLS and certificate validation rather than disabling verification to suppress errors.
- Separate cache rules for assets, HTML, login and APIs. Explicitly handle cookies, authorization headers and personalized responses to prevent serving private content to other users.
- Compare edge and origin logs, cache hits and errors. If origin access is restricted to CDN addresses, maintain the official IP list and a controlled recovery path.
Cloud VPS / 24Validate IPv6 dual-stack access
Confirm the assigned IPv6 address, prefix and gateway rather than deriving them from IPv4
Procedure
- Confirm the assigned IPv6 address, prefix and gateway rather than deriving them from IPv4. Save network settings and verify console access before changes.
- Check routing, DNS and application listeners; an OS address does not imply the web server listens on IPv6. Review IPv6 firewall rules and preserve required ICMPv6 traffic.
- Test HTTPS and critical flows from an IPv6-capable external network before publishing AAAA. Test both families explicitly so intermittent success does not hide a broken path.
Cloud VPS / 25Schedule reliable recurring jobs
Choose the system or business timezone and account for daylight saving
Procedure
- Choose the system or business timezone and account for daylight saving. Set executable paths and working directories explicitly because scheduled jobs have a different environment from interactive shells.
- Prevent overlapping runs with a lock or lease and assign ownership across multiple hosts. Make retries idempotent so repeated execution cannot duplicate payments or updates.
- Log start, completion, exit status and processed counts, and alert on missed or overdue runs. Test restarts and retries; the presence of a log file does not prove success.
Dedicated servers / 26Plan RAID capacity and resilience
Choose a layout based on disk count and workload
Procedure
- Choose a layout based on disk count and workload. With equal-sized disks RAID 5 reserves one disk of parity and RAID 6 two; account separately for formatting and hot spares.
- Check protected controller cache, write policy and power-loss behavior. RAID 10 tolerance depends on which mirror members fail; arbitrary two-disk failure is not always survivable.
- Budget degraded rebuild performance and additional failure exposure. Keep independent backups because deletion, ransomware and logical corruption affect the array as a whole.
Dedicated servers / 27Monitor disk health and replacement
Map model, serial number, slot and array membership before reading health data
Procedure
- Map model, serial number, slot and array membership before reading health data. Vendor raw attributes differ; interpret them using documentation and trends.
- Correlate media errors, temperature, endurance and kernel I/O reports. A passing health flag is not a guarantee; rising errors and workload latency matter more than a single snapshot.
- Before replacement verify backups, slot identity, redundancy and hot-swap support. Monitor rebuilding to completion and validate services; do not casually remove another member during recovery.
Dedicated servers / 28Understand CPU topology and NUMA
Inspect sockets, cores, threads and NUMA nodes
Procedure
- Inspect sockets, cores, threads and NUMA nodes. A logical thread is not a full physical core; frequency and cache affect single-thread work beyond the total thread count.
- Measure scheduling, memory bandwidth and remote-node access under representative load. Affinity can improve locality but also constrain capacity; measure before pinning.
- Keep the default configuration and compare throughput and tail latency at equal concurrency. Review vNUMA and guest CPU layout so one guest does not unnecessarily span too many nodes.
Dedicated servers / 29Dedicated-server launch checklist
Reconcile the order, detected hardware and management access
Procedure
- Reconcile the order, detected hardware and management access. Record serials, contacts and replacement procedures before an incident forces you to locate credentials.
- Schedule CPU, memory, network and storage checks before importing data and identify destructive tests. Observe temperature, throttling and errors rather than treating a short benchmark as proof of stability.
- Verify mounts, service startup, monitoring and restoration after a reboot. Preserve a baseline for later comparisons that distinguish hardware changes from workload growth.
Dedicated servers / 30Recover with a rescue system
Capture the last change and boot error and distinguish bootloader, filesystem and networking faults
Procedure
- Capture the last change and boot error and distinguish bootloader, filesystem and networking faults. Device names in rescue mode can differ, so identify disks and partitions carefully.
- Prefer read-only inspection and copying critical data. Avoid formatting or forced repairs without verified backups; encrypted, RAID and LVM volumes require the correct keys and membership.
- Repair the identified cause, restore boot order and detach rescue media. Verify networking, mounts, databases and scheduled jobs after restart and document the recovery.
Virtualization hosts / 31Choose Proxmox storage
Separate ISO, backup and guest-disk storage and verify supported content types for directory, LVM-thin, ZFS or shared storage
Procedure
- Separate ISO, backup and guest-disk storage and verify supported content types for directory, LVM-thin, ZFS or shared storage. Total capacity alone does not establish suitability.
- Monitor thin-pool data and metadata separately; promised guest capacity may exceed physical storage. Reserve room for snapshots, growth and migration to prevent pool exhaustion affecting many guests.
- Test provisioning, snapshots, backup and restoration on a disposable guest. Shared visibility does not imply availability; verify network redundancy, permissions and failure behavior.
Virtualization hosts / 32Isolate tenants with VLANs
Document management, storage and tenant boundaries with VLANs, subnets and ports
Procedure
- Document management, storage and tenant boundaries with VLANs, subnets and ports. VLANs separate layer two; routed communication still requires firewall policy.
- Match tagged and untagged settings across switch, bridge and guest. Keep console access before altering management networking and validate with a test VLAN first.
- Test both allowed and blocked traffic between tenants, including IPv6, DHCP and spoofed sources. Keep a policy matrix and repeat it when provisioning new tenants.
Virtualization hosts / 33Check VM migration compatibility
Compare CPU features, hypervisor versions, storage names and bridge mappings
Procedure
- Compare CPU features, hypervisor versions, storage names and bridge mappings. Passed-through GPUs, USB and local devices can prevent live migration and require planned downtime.
- Budget memory and disk transfer plus destination capacity. A high dirty-page rate can prevent convergence, so live migration is not automatically interruption-free for every workload.
- Pilot a low-risk guest and verify networking, time, disks and services. Confirm stability and backups before removing the source, and avoid running two copies with the same identity.
Virtualization hosts / 34Distinguish snapshots from backups
Snapshots often depend on the original storage chain and suit short-term rollback
Procedure
- Snapshots often depend on the original storage chain and suit short-term rollback. Backups must restore independently; a storage failure can destroy both data and local snapshots.
- Use supported guest agents or database-aware procedures for consistency. Freezing disk writes alone may not coordinate external dependencies or distributed transactions.
- Expire snapshots and monitor growth while rehearsing isolated backup restoration. Before reverting, identify lost writes and any external systems requiring coordinated rollback.
Virtualization hosts / 35Protect the hypervisor management plane
Separate management UI and SSH from tenant traffic
Procedure
- Separate management UI and SSH from tenant traffic. Use individual least-privilege accounts and MFA rather than sharing host administrator credentials with tenants.
- Scope and expire API tokens and protect secrets in configuration backups. Review dormant accounts, stale tokens and recent permission changes.
- Validate configuration backups and console recovery before updates, then retest isolation and audit logs. Recovery access must not depend on a guest on the failed host.
GPU compute / 36Validate GPU containers
Verify the host driver before configuring NVIDIA Container Toolkit for the engine
Procedure
- Verify the host driver before configuring NVIDIA Container Toolkit for the engine. Containers use the host driver interface and should not independently install a kernel driver as if they were full hosts.
- Test device visibility and a small computation with a pinned trusted image and explicit GPU assignment. Working nvidia-smi does not establish full model and operator compatibility.
- Restrict mounts and secrets and configure CPU, memory and logs. Revalidate models after image upgrades and retain a rollback image; GPU access does not require unrestricted host-device access.
GPU compute / 37Launch a model inference service
Record model license, version, precision and input limits and verify weight provenance
Procedure
- Record model license, version, precision and input limits and verify weight provenance. Establish outputs for a fixed sample set before validating the API; downloaded weights are not a working service.
- Set authentication, input and context limits, queues and timeouts. Bound concurrency and rate, and ensure cancellation releases resources instead of allowing unbounded queues.
- Measure cold start, first-token latency, generation rate and p95, including invalid inputs and overload. Roll out only after quality and performance gates pass and retain the previous model and configuration.
GPU compute / 38Tune batching and concurrency
Distinguish per-request batch size, dynamic batching and connection concurrency
Procedure
- Distinguish per-request batch size, dynamic batching and connection concurrency. Larger batches can improve throughput while adding wait time; interactive and offline workloads need different latency targets.
- Test realistic input-length distributions and record peak memory, queue delay, first-token and completion time. Short uniform inputs can produce misleadingly favorable throughput.
- Find the highest stable throughput meeting the p95 target and cap queue depth. Keep headroom for long requests and model changes and retest after quantization or multi-GPU changes.
GPU compute / 39Training checkpoints and resumption
Save optimizer, scheduler, step count and random state as required alongside weights
Procedure
- Save optimizer, scheduler, step count and random state as required alongside weights. Inference-only weights generally cannot resume the original training trajectory.
- Choose an interval based on acceptable recomputation and budget checkpoint size, write time and retention. Use supported consistent writes so an interrupted save cannot overwrite the only valid checkpoint.
- Stop and resume a short run and verify step count, loss and data sampling. Copy valid checkpoints off-host and confirm complete, version-matched shards for distributed training.
GPU compute / 40Plan a LoRA fine-tuning job
Confirm model and data licensing and separate training from validation
Procedure
- Confirm model and data licensing and separate training from validation. Clean duplicate and sensitive samples and version the dataset; falling training loss alone does not establish improvement.
- LoRA reduces trainable parameters but still requires the base model and activations. Profile a small run at the target context and batch before estimating memory, duration and cost.
- Save the adapter, base-model identifier and configuration and evaluate both improvements and regressions. Test loading and rollback; an adapter is not necessarily a standalone model.
GPU compute / 41Inspect multi-GPU topology
Inspect PCIe links, NUMA placement and NVLink support and identify the parallelism strategy
Procedure
- Inspect PCIe links, NUMA placement and NVLink support and identify the parallelism strategy. More cards do not automatically pool memory or scale throughput linearly.
- Establish a single-card baseline before scaling and separate compute, loading and synchronization time. For multiple hosts measure collective communication rather than relying on advertised port speed.
- Verify device assignment and logs for every rank so processes do not accidentally share one card. Keep reproducible launch settings and diagnose networking and versions before bypassing communication checks.
Operations & security / 42Coordinate security groups and firewalls
Document source, destination, protocol and port for each service
Procedure
- Document source, destination, protocol and port for each service. Security groups, host firewalls and application permissions are separate layers and any can block a connection.
- Keep a live session and console recovery before changing remote-access rules. Add the required allow rule before tightening access and cover both address families.
- Test from allowed and denied sources and verify persistence after reboot. Record ownership, purpose and expiry and remove temporary openings after troubleshooting.
Operations & security / 43Design actionable monitoring
Start with availability, error rate, latency and critical workflow success, then add resource metrics
Procedure
- Start with availability, error rate, latency and critical workflow success, then add resource metrics. Healthy CPU does not prove that checkout or login works.
- Separate urgent incidents from capacity trends and configure duration, recovery and deduplication. Base thresholds on normal behavior and objectives instead of paging on every brief spike.
- Give each critical alert an owner, investigation path and escalation route and test delivery. Monitor missing telemetry and the monitoring system itself; no data is not proof of health.
Operations & security / 44Rotate logs and protect sensitive data
Inventory application, system, proxy and container logs and their growth rates
Procedure
- Inventory application, system, proxy and container logs and their growth rates. Redact passwords, tokens and sensitive payloads even during debugging.
- Configure rotation, compression and retention and confirm the process reopens its log. Deleting an open file may not reclaim space; use the service’s supported rotation mechanism.
- Verify writing, permissions and search after rotation and alert on storage or collection failures. Protect centralized log access and transport and enforce the required retention period.
Operations & security / 45Plan system patching
Inventory OS, application, database and driver versions
Procedure
- Inventory OS, application, database and driver versions. Separate security fixes from feature upgrades and check reboot, migration and client-compatibility requirements.
- Test on a representative environment and validate backups and rollback. Define expected impact, ownership and acceptance criteria for the maintenance window, not just a start time.
- Verify versions, startup, monitoring and errors after updating, including a reboot where required. Keep change records; a successful installer exit is not complete service acceptance.
Operations & security / 46Diagnose time synchronization
Distinguish clock accuracy, timezone and display formatting
Procedure
- Distinguish clock accuracy, timezone and display formatting. A timezone issue may be cosmetic, while clock drift can break TLS, signatures, queues and distributed jobs.
- Inspect the synchronization service, upstream reachability and host/guest policy. Avoid competing synchronization agents and diagnose drift instead of repeatedly setting the clock manually.
- Assess database and job impact before a large production clock step. Retest cross-node timing, certificates and schedules, then monitor drift and standardize log timestamps.
Operations & security / 47Store backups in object storage
Choose storage by recovery location and access needs and account for requests, retrieval and egress
Procedure
- Choose storage by recovery location and access needs and account for requests, retrieval and egress. Archive delays and minimum retention must fit the RTO.
- Upload with scoped credentials and configure encryption, versioning and lifecycle rules. Verify delete permissions and rule scope because lifecycle policies can remove older backup versions.
- Retrieve and validate a backup from an independent recovery environment and record time and cost. Confirm key availability; a successful upload is not an application restore test.
Operations & security / 48Investigate incidents and prepare support details
Determine whether the impact is one user, region, instance or the whole service and record timezone-aware timestamps
Procedure
- Determine whether the impact is one user, region, instance or the whole service and record timezone-aware timestamps. Preserve errors and changes before restarting multiple components.
- Work through DNS, networking, host, application and database layers. Prioritize critical service recovery and limit changes; assess data consistency before rollback or degradation.
- Provide instance ID, destination, timestamps, reproduction and redacted logs without passwords or private keys. After recovery document cause and prevention and distinguish mitigation from permanent resolution.
Operations & security / 49Automate certificate renewal
Inventory domains, deployment points and ACME accounts and distinguish edge from origin certificates
Procedure
- Inventory domains, deployment points and ACME accounts and distinguish edge from origin certificates. Updated files do not prove the service loaded them; inspect the served certificate.
- Test renewal using the client’s supported method and verify HTTP challenge access or scoped DNS API rights. Keep logs because propagation and firewall changes can break validation.
- Configure safe reload hooks and failure notifications plus independent external expiry checks. Monitor the real hostname and SNI rather than only local files so virtual-host mistakes are detected.
Operations & security / 50Manage application secrets
Separate public configuration from credentials and signing keys and keep real secrets out of repositories and images
Procedure
- Separate public configuration from credentials and signing keys and keep real secrets out of repositories and images. Environment variables are not an encrypted vault and still require process and administrative access controls.
- Issue separate least-privilege credentials per service and document owners and rotation. Redact and protect logs, dumps, backups and diagnostic output that may expose secrets.
- Plan dual-key overlap or an explicit switchover order and validate new credentials before revoking old ones. Suspected exposure requires revocation and access-log review, not merely removing a public file.