Technical documentation

Estimate CPU, memory, storage and transfer from the workload, then budget headroom.

Steps and verification

1. Measure peak concurrency, request duration, CPU utilization and resident memory. Budget separately for the database, cache and application runtime rather than counting websites.
2. Include databases, media, logs, snapshots and growth in storage capacity. A 10Mbps port has a theoretical 1.25MB/s ceiling before protocol and path overhead.
3. Load-test representative requests and inspect p95 latency, swapping and disk wait. Address slow queries and caching before choosing a CPU, RAM or bandwidth upgrade.
4. Create a resource budget with measured peaks, expected growth, latency targets and upgrade triggers. Image sites emphasize storage and transfer; transactional databases emphasize cache hit rates and synchronous writes. Equal traffic does not imply equal sizing.

free -h
df -h
vmstat 1 5

Compare CN2 GIA, CMI, PCCW or international BGP using consistent endpoints and time windows.

Steps and verification

1. Request a test address for the exact product. Record client location, ISP, timestamp and protocol. A public regional reference IP does not establish the route used by your instance.
2. Capture an outbound MTR from the client and a reverse trace from the server or its Looking Glass. AS4809 and 59.43 hops are clues, not proof of complete bidirectional GIA routing.
3. Compare peak and off-peak end-to-end loss, jitter and HTTPS timings. Intermediate routers may rate-limit ICMP; loss at an intermediate hop alone does not establish application packet loss.
4. Store measurements by ISP, timestamp and direction alongside application timings. Routing can change with interconnection policy or failover; one pre-purchase traceroute cannot establish long-term experience for every user.

mtr --report --report-cycles 20 YOUR_TEST_IP

Take a site from DNS setup to verified certificate renewal.

Steps and verification

1. Point A/AAAA records to the actual addresses; publish AAAA only when working IPv6 is configured. Install supported web-server, PHP and database versions and create a dedicated database user.
2. Download WordPress from its official source and configure site and administrator settings. Grant the web process write access only where needed; keep database ports private.
3. Configure an ACME client and HTTPS, checking the required ports or DNS permissions for validation. Test automatic renewal, then redirect HTTP once HTTPS works.
4. Test login, uploads, email and checkout. Back up the database and uploads off-host and complete a restoration drill before moving production traffic.
5. After launch, verify permalinks, upload limits, scheduled tasks and cache rules. Do not cache carts, authenticated sessions or admin pages as anonymous content. Exercise both signed-in and anonymous flows after enabling a CDN or page cache.

curl -I https://YOUR_DOMAIN

Verify delivered specifications, disk health and management access.

Steps and verification

1. Compare CPU model and count, usable RAM, and disk type and count with the order. Distinguish physical cores from threads and raw capacity from usable array capacity.
2. Confirm RAID level, hot spares and protected controller cache. RAID 0 has no redundancy; RAID 1/10 and RAID 5/6 have different capacity and failure tolerances. RAID is not a backup.
3. Check health alerts, degraded arrays and rebuild status; record serial numbers and acceptance results. Perform stress tests before production and protect data before destructive disk tests or rebuilds.
4. Keep an acceptance record separating confirmed and untested items. Port speed does not guarantee throughput to every destination, and disk models alone do not establish array performance. Test representative workloads and record competing jobs.

lscpu
free -h
lsblk -o NAME,SIZE,TYPE,MODEL

Check remote console access, boot media and recovery paths.

Steps and verification

1. Confirm the management address, credential delivery and allowed source networks. Use an isolated management network or VPN, restrict public access and replace default credentials.
2. During a maintenance window, test video, keyboard and virtual media. Power operations and boot-order changes interrupt service; drain traffic and confirm backups first.
3. Document disk recovery and escalation contacts. When networking fails, inspect interfaces, routes and firewall rules from the console before deciding whether a reinstall is necessary.
4. Identify the physical host and service owner before recovery. Distinguish graceful shutdown, hard power-off and reinstall consequences. Console access does not prove network recovery; verify services, mounts and databases, then detach temporary installation media.

Identify the subnet delivery model before configuring guest networking.

Steps and verification

1. Record the assigned gateway, mask, usable addresses and MAC restrictions. Bridged networks, routed subnets and single-IP NAT require different configurations; do not copy another facility’s example blindly.
2. Keep out-of-band access and back up network configuration. Test one guest for bridge connectivity, DNS, return routes and reboot persistence before provisioning more tenants.
3. Separate management and tenant networks. Apply firewalls, anti-spoofing and access controls. Use only assigned addresses; a dedicated IP is not tenant isolation.
4. Store backups in a separate failure domain and document recovery order. Test restored disks, networking, applications and permissions.
5. Before production handover, confirm extra-MAC and nested-virtualization restrictions with the provider. Save the tested network configuration and rollback procedure for future tenant provisioning.

ip -br address
ip route
bridge link

Budget RAM, storage IOPS, CPU and networking rather than counting VMs.

Steps and verification

1. Reserve RAM for the host, filesystem cache and management services. Sum guest peak working sets instead of allocating all physical memory to tenants.
2. Choose CPU overcommit based on workload and observe host load, guest steal time and scheduling latency. Quiet websites and latency-sensitive databases require different assumptions.
3. Budget random IOPS, synchronous writes and backup windows. Alert on capacity exhaustion, thin-provisioning growth and NIC congestion; prepare migration and rollback before upgrades.
4. Include maintenance and failure headroom in capacity planning. Check destination CPU compatibility, RAM, storage access and networking before planning guest moves. Having a backup does not provide spare migration capacity.

Model weights are only part of the memory budget.

Steps and verification

1. Distinguish inference, LoRA and full training. A 7B model’s FP16 weights alone require about 14GB in decimal units, before framework and runtime overhead; this does not establish that a 16GB card will fit.
2. Inference also needs context, concurrency and KV cache. Training adds gradients, optimizer states and activations. Quantization reduces weights but requires quality and operator-compatibility checks.
3. Multiple GPUs do not automatically become one memory pool. Confirm model parallelism, PCIe/NVLink topology and framework support, then measure peak usage and throughput at the target batch size.
4. Record input length, concurrency, precision and peak memory for each run. Quantized weights can still have temporary loading peaks. For training, test checkpoint saving and resumption before committing to a long run.

nvidia-smi
nvidia-smi topo -m

Check the driver before installing a compatible runtime and testing GPU execution.

Steps and verification

1. Run nvidia-smi to inspect the device, driver and active processes. Its CUDA version describes driver support and does not necessarily identify an installed CUDA Toolkit.
2. Select a compatible build using official PyTorch instructions and install in an isolated environment. Containers also need NVIDIA Container Toolkit and device access. Avoid mixing unverified driver packages.
3. Verify torch.cuda.is_available(), the device name and a small tensor operation before model dependencies. Record driver, PyTorch, runtime and model versions for reproducibility and rollback.
4. Device discovery is only the first check: run a small real inference or training job and retain its output. For operator or shared-library failures, inspect environment and dependency versions instead of repeatedly overwriting packages. Export a lockfile and pin working images.

python -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'No CUDA device')"

Aim for recoverable data and a measured recovery time, not merely a successful backup job.

Steps and verification

1. Define the maximum acceptable period of data loss (RPO) and service recovery time (RTO). Use application-consistent database backups or supported snapshots rather than copying a live data directory.
2. Keep encrypted backups on separate infrastructure or in another region with retention and least-privilege access. Document key custody and alert on failed jobs, capacity and expiry.
3. Restore a recent backup in isolation, verify records, files, permissions and application flows, and record elapsed time. Confirm monitoring, certificate renewal, escalation contacts and rollback before launch.
4. Compare backup intervals with the RPO and total download, decryption, import and validation time with the RTO. Verify database-to-upload consistency. Successfully extracting an archive does not demonstrate application recovery.

Set up reliable administrative access and distinguish transport failures from authentication errors.

Steps and verification

1. Generate a passphrase-protected SSH key locally and install only its public key in the account’s authorized_keys. Keep the private key off tickets and shared directories; issue separate keys to each administrator.
2. Verify the host fingerprint through the console or another trusted delivery channel before the first connection. Investigate fingerprint changes after a rebuild rather than disabling host verification.
3. Keep the current session open while testing the new account, key and required sudo access in a second terminal. Restrict password access only after testing and confirming a recovery console is available.
4. For timeouts, check instance state, IP, port, security groups and the client network. Connection refused calls for checking the service listener. Authentication failures require checking the user, key, permissions and server logs.
5. Validate changes with sshd -t before reloading through the system service manager. Retain the old session and test a fresh connection; restore through the console if validation fails.

ssh -vv YOUR_USER@YOUR_SERVER_IP

Separate DNS, connectivity, certificate and application failures.

Steps and verification

1. Record the full hostname, time, client network and browser error. The apex and www can have different records; a DNS control panel alone does not prove that recursive resolvers have updated.
2. Query A, AAAA and CNAME separately. If IPv4 works but some clients fail, verify the IPv6 service behind AAAA. Cached answers can remain until their TTL expires after a change.
3. Confirm whether the expected destination is an origin or a CDN. Proxied DNS normally returns edge addresses. Preserve the correct Host and SNI when testing an origin instead of judging its certificate by an IP-only request.
4. Check hostname coverage, validity, system time and intermediate certificates. Align application and reverse-proxy HTTPS redirects to prevent loops between layers.
5. For 502/504, inspect upstream listeners, proxy logs and application timing; for 403, inspect access rules and permissions. Retest the home page, login, assets and renewal jobs from multiple networks.

dig YOUR_DOMAIN A
dig YOUR_DOMAIN AAAA
curl -I https://YOUR_DOMAIN

Manage images, persistent data, network exposure and rollback separately.

Steps and verification

1. Use supported operating systems and official package sources. Record Compose and image architecture; ARM and x86 images are not automatically interchangeable. Pin a version or digest instead of relying on latest.
2. Store databases and uploads in explicit persistent volumes, documenting paths, ownership and backup procedures. Rebuilding a container does not rebuild its data; avoid keeping unique data only in its writable layer.
3. Publish only required ports and keep databases on internal networks. Docker port publication can affect firewall behavior, so verify exposure from an external network rather than trusting a firewall UI alone.
4. Configure health checks, resource limits and log rotation. A running container may not be ready; verify dependencies, migrations, permissions and data persistence across restarts.
5. Retain the previous image and configuration and rehearse rollback. Database schema changes may be incompatible with old code, so reverting an image is not sufficient; back up and plan recovery before irreversible migrations.

docker version
docker compose ps
docker stats --no-stream

Identify CPU, memory, storage, inode and application bottlenecks before upgrading.

Steps and verification

1. Capture traffic, latency, load and recent releases during the incident. Load average is not CPU utilization; interpret it alongside core count, runnable tasks and I/O waits.
2. Use available memory in free rather than treating filesystem cache as a leak. Sustained swap activity, OOM logs and application heap growth are stronger evidence than a single usage sample.
3. Check both df -h and df -i; exhausted inodes can prevent writes despite free bytes. Investigate deleted-open files, mount points and snapshots when filesystem and directory totals disagree.
4. Correlate disk waits with slow queries, backup jobs and random writes. Sample with low impact and avoid intensive write benchmarks on production volumes. High sequential throughput does not establish sufficient random IOPS.
5. Address rotation, queries or capacity according to evidence, changing one major variable at a time and comparing equivalent loads. Do not remove database files or drop caches to hide the actual memory requirement.

uptime
free -h
df -h
df -i
vmstat 1 5

Synchronize data in a controlled window and avoid conflicting writes.

Steps and verification

1. Inventory domains, databases, uploads, scheduled tasks, email, callbacks and allowlists. Check runtime compatibility before migrating and avoid combining the move with several major-version upgrades.
2. Lower DNS TTL in advance and allow the previous TTL period to pass. A lower setting cannot evict existing cached answers immediately; include CDN, proxy and application caches in the plan.
3. Restore an initial backup and test using a staging hostname or local hosts override. Disable live payments, email delivery, queue consumers and duplicate jobs in staging to prevent real side effects.
4. Pause writes during the maintenance window, take an application-consistent database backup and perform the final file sync. Verify data, permissions and certificates before switching traffic; changing DNS alone can split orders and uploads.
5. Retain the old environment while observing logs, errors and critical workflows. If the new site has accepted writes, reconcile those before rollback rather than simply pointing DNS back. Retire old resources only after acceptance.

Reproduce failures with a small workload and isolate memory, concurrency and input bottlenecks.

Steps and verification

1. Record model, precision, context length, batch size, concurrency and the full error. Reproduce with one short request, then increase one variable at a time; model file size is not the runtime memory requirement.
2. Use nvidia-smi to find competing processes, then compare framework allocated, reserved and peak statistics. Releasing cached blocks cannot shrink live tensors that exceed available memory.
3. For inference, reduce concurrency, batch or context before evaluating quantization and KV-cache policies. Training may benefit from gradient accumulation or activation checkpointing. Recheck quality, throughput and latency after each change.
4. For low utilization, inspect CPU preprocessing, storage, data loading and host-to-device transfers. Separate cold loading or compilation from warmed-up time to first token and steady generation speed.
5. Run representative inputs while measuring p95 latency, peak memory and errors. On multiple GPUs, verify actual placement, topology and communication waits. Save the stable configuration and software versions as a baseline.

nvidia-smi
nvidia-smi topo -m

Map users, ISPs, databases and external APIs

Steps and verification

1. Map users, ISPs, databases and external APIs. Cross-region calls between application and database can matter more than home-page latency.
2. Compare equivalent plans with the same requests and time windows. Record p50/p95 latency and failures, then check residency requirements, backup locations and availability.
3. Pilot noncritical services or a small traffic share before expanding. Plan database consistency and rollback separately; changing the website DNS is not a complete migration.

Distinguish port speed, committed bandwidth and monthly transfer allowances

Steps and verification

1. Distinguish port speed, committed bandwidth and monthly transfer allowances. Divide Mbps by eight for theoretical MB/s before overhead; check inbound and outbound billing rules.
2. Estimate transfer from response size times request count and account for downloads, backups and crawlers. A saturated 100Mbps link theoretically transfers 1.08 decimal TB per day; do not assume sustained line rate.
3. Alert on daily and monthly totals and confirm overage behavior. Diagnose slow downloads across connection limits, disk throughput, routing and CDN cache hits.

List application runtime, kernel and architecture requirements before choosing an OS

Steps and verification

1. List application runtime, kernel and architecture requirements before choosing an OS. Prefer a release receiving security maintenance rather than following an outdated tutorial onto an unsupported version.
2. Document repositories, package management, firewall and networking tools. Service names differ across distributions; validate boot, dependencies, monitoring and backup agents on a test instance.
3. Test restoration before a major upgrade and schedule maintenance. Reinstallation can destroy system-disk data; confirm retention behavior instead of treating a rebuild as a routine update.

Verify the application listener before configuring the proxy

Steps and verification

1. Verify the application listener before configuring the proxy. Bind private services to loopback or a private interface and forward the original host and protocol appropriately after TLS termination.
2. Validate with nginx -t before reloading. For 502 inspect refused connections, socket permissions and crashed upstreams; for 504 correlate application timings instead of endlessly raising timeouts.
3. Exercise uploads, redirects, client-IP handling and WebSockets. Trust forwarded headers only from known proxies so arbitrary client input cannot spoof an identity used by access controls.

Use a dedicated application account with restricted sources instead of an administrator account

Steps and verification

1. Use a dedicated application account with restricted sources instead of an administrator account. Limit listening interfaces and firewall exposure and protect remote connections when required.
2. Correlate request latency with slow-query records and inspect plans with EXPLAIN. Evaluate indexes and scanned rows; every extra index also adds write and storage cost.
3. Record a baseline before changing pools, caches or queries and retest comparable loads. Verify consistent backups and monitor connection exhaustion, storage and replication delay, not merely process uptime.

Decide whether Redis holds disposable cache, sessions or critical queues

Steps and verification

1. Decide whether Redis holds disposable cache, sessions or critical queues. These need different durability and eviction policies; restrict networking and configure supported authentication and authorization.
2. Leave headroom for persistence, replication buffers and the OS when setting maxmemory. Match eviction to TTL usage and define application behavior when no eligible keys can be evicted.
3. Track hit rate, evictions, slow commands and large keys before adding memory. Test database load after a cold cache and restore procedures for data that must persist.

Validate the origin hostname, certificate and application before enabling the proxy

Steps and verification

1. Validate the origin hostname, certificate and application before enabling the proxy. Define origin TLS and certificate validation rather than disabling verification to suppress errors.
2. Separate cache rules for assets, HTML, login and APIs. Explicitly handle cookies, authorization headers and personalized responses to prevent serving private content to other users.
3. Compare edge and origin logs, cache hits and errors. If origin access is restricted to CDN addresses, maintain the official IP list and a controlled recovery path.

Confirm the assigned IPv6 address, prefix and gateway rather than deriving them from IPv4

Steps and verification

1. Confirm the assigned IPv6 address, prefix and gateway rather than deriving them from IPv4. Save network settings and verify console access before changes.
2. Check routing, DNS and application listeners; an OS address does not imply the web server listens on IPv6. Review IPv6 firewall rules and preserve required ICMPv6 traffic.
3. Test HTTPS and critical flows from an IPv6-capable external network before publishing AAAA. Test both families explicitly so intermittent success does not hide a broken path.

Choose the system or business timezone and account for daylight saving

Steps and verification

1. Choose the system or business timezone and account for daylight saving. Set executable paths and working directories explicitly because scheduled jobs have a different environment from interactive shells.
2. Prevent overlapping runs with a lock or lease and assign ownership across multiple hosts. Make retries idempotent so repeated execution cannot duplicate payments or updates.
3. Log start, completion, exit status and processed counts, and alert on missed or overdue runs. Test restarts and retries; the presence of a log file does not prove success.

Choose a layout based on disk count and workload

Steps and verification

1. Choose a layout based on disk count and workload. With equal-sized disks RAID 5 reserves one disk of parity and RAID 6 two; account separately for formatting and hot spares.
2. Check protected controller cache, write policy and power-loss behavior. RAID 10 tolerance depends on which mirror members fail; arbitrary two-disk failure is not always survivable.
3. Budget degraded rebuild performance and additional failure exposure. Keep independent backups because deletion, ransomware and logical corruption affect the array as a whole.

Map model, serial number, slot and array membership before reading health data

Steps and verification

1. Map model, serial number, slot and array membership before reading health data. Vendor raw attributes differ; interpret them using documentation and trends.
2. Correlate media errors, temperature, endurance and kernel I/O reports. A passing health flag is not a guarantee; rising errors and workload latency matter more than a single snapshot.
3. Before replacement verify backups, slot identity, redundancy and hot-swap support. Monitor rebuilding to completion and validate services; do not casually remove another member during recovery.

Inspect sockets, cores, threads and NUMA nodes

Steps and verification

1. Inspect sockets, cores, threads and NUMA nodes. A logical thread is not a full physical core; frequency and cache affect single-thread work beyond the total thread count.
2. Measure scheduling, memory bandwidth and remote-node access under representative load. Affinity can improve locality but also constrain capacity; measure before pinning.
3. Keep the default configuration and compare throughput and tail latency at equal concurrency. Review vNUMA and guest CPU layout so one guest does not unnecessarily span too many nodes.

Reconcile the order, detected hardware and management access

Steps and verification

1. Reconcile the order, detected hardware and management access. Record serials, contacts and replacement procedures before an incident forces you to locate credentials.
2. Schedule CPU, memory, network and storage checks before importing data and identify destructive tests. Observe temperature, throttling and errors rather than treating a short benchmark as proof of stability.
3. Verify mounts, service startup, monitoring and restoration after a reboot. Preserve a baseline for later comparisons that distinguish hardware changes from workload growth.

Capture the last change and boot error and distinguish bootloader, filesystem and networking faults

Steps and verification

1. Capture the last change and boot error and distinguish bootloader, filesystem and networking faults. Device names in rescue mode can differ, so identify disks and partitions carefully.
2. Prefer read-only inspection and copying critical data. Avoid formatting or forced repairs without verified backups; encrypted, RAID and LVM volumes require the correct keys and membership.
3. Repair the identified cause, restore boot order and detach rescue media. Verify networking, mounts, databases and scheduled jobs after restart and document the recovery.

Separate ISO, backup and guest-disk storage and verify supported content types for directory, LVM-thin, ZFS or shared storage

Steps and verification

1. Separate ISO, backup and guest-disk storage and verify supported content types for directory, LVM-thin, ZFS or shared storage. Total capacity alone does not establish suitability.
2. Monitor thin-pool data and metadata separately; promised guest capacity may exceed physical storage. Reserve room for snapshots, growth and migration to prevent pool exhaustion affecting many guests.
3. Test provisioning, snapshots, backup and restoration on a disposable guest. Shared visibility does not imply availability; verify network redundancy, permissions and failure behavior.

Document management, storage and tenant boundaries with VLANs, subnets and ports

Steps and verification

1. Document management, storage and tenant boundaries with VLANs, subnets and ports. VLANs separate layer two; routed communication still requires firewall policy.
2. Match tagged and untagged settings across switch, bridge and guest. Keep console access before altering management networking and validate with a test VLAN first.
3. Test both allowed and blocked traffic between tenants, including IPv6, DHCP and spoofed sources. Keep a policy matrix and repeat it when provisioning new tenants.

Compare CPU features, hypervisor versions, storage names and bridge mappings

Steps and verification

1. Compare CPU features, hypervisor versions, storage names and bridge mappings. Passed-through GPUs, USB and local devices can prevent live migration and require planned downtime.
2. Estimate memory and disk transfers and confirm destination capacity and bandwidth. If the guest changes memory faster than it can be copied, live migration may not converge; not every workload can move without interruption.
3. Pilot a low-risk guest and verify networking, time, disks and services. Confirm stability and backups before removing the source, and avoid running two copies with the same identity.

Snapshots often depend on the original storage chain and suit short-term rollback

Steps and verification

1. Snapshots often depend on the original storage chain and suit short-term rollback. Backups must restore independently; a storage failure can destroy both data and local snapshots.
2. Use supported guest agents or database-aware procedures for consistency. Freezing disk writes alone may not coordinate external dependencies or distributed transactions.
3. Expire snapshots and monitor growth while rehearsing isolated backup restoration. Before reverting, identify lost writes and any external systems requiring coordinated rollback.

Separate management UI and SSH from tenant traffic

Steps and verification

1. Separate management UI and SSH from tenant traffic. Use individual least-privilege accounts and MFA rather than sharing host administrator credentials with tenants.
2. Scope and expire API tokens and protect secrets in configuration backups. Review dormant accounts, stale tokens and recent permission changes.
3. Validate configuration backups and console recovery before updates, then retest isolation and audit logs. Recovery access must not depend on a guest on the failed host.

Verify the host driver before configuring NVIDIA Container Toolkit for the engine

Steps and verification

1. Verify the host driver before configuring NVIDIA Container Toolkit for the engine. Containers use the host driver interface and should not independently install a kernel driver as if they were full hosts.
2. Test device visibility and a small computation with a pinned trusted image and explicit GPU assignment. Working nvidia-smi does not establish full model and operator compatibility.
3. Restrict mounts and secrets and configure CPU, memory and logs. Revalidate models after image upgrades and retain a rollback image; GPU access does not require unrestricted host-device access.

Record model license, version, precision and input limits and verify weight provenance

Steps and verification

1. Record model license, version, precision and input limits and verify weight provenance. Establish outputs for a fixed sample set before validating the API; downloaded weights are not a working service.
2. Set authentication, input and context limits, queues and timeouts. Bound concurrency and rate, and ensure cancellation releases resources instead of allowing unbounded queues.
3. Measure cold-start time, time to first token, token generation rate and p95 request latency. Include invalid inputs and overload tests. Shift traffic gradually after quality and performance checks pass, retaining the previous model and configuration for rollback.

Distinguish per-request batch size, dynamic batching and connection concurrency

Steps and verification

1. Distinguish per-request batch size, dynamic batching and connection concurrency. Larger batches can improve throughput while adding wait time; interactive and offline workloads need different latency targets.
2. Test realistic input-length distributions and record peak memory, queue delay, first-token and completion time. Short uniform inputs can produce misleadingly favorable throughput.
3. Find the highest stable throughput meeting the p95 target and cap queue depth. Keep headroom for long requests and model changes and retest after quantization or multi-GPU changes.

Save optimizer, scheduler, step count and random state as required alongside weights

Steps and verification

1. Alongside model weights, save the optimizer, learning-rate scheduler, training step and random-number state required by the framework. Inference-only weights generally cannot restore the full training state.
2. Choose an interval based on acceptable recomputation and budget checkpoint size, write time and retention. Use supported consistent writes so an interrupted save cannot overwrite the only valid checkpoint.
3. Stop and resume a short run and verify step count, loss and data sampling. Copy valid checkpoints off-host and confirm complete, version-matched shards for distributed training.

Confirm model and data licensing and separate training from validation

Steps and verification

1. Confirm model and data licensing and separate training from validation. Clean duplicate and sensitive samples and version the dataset; falling training loss alone does not establish improvement.
2. LoRA reduces trainable parameters but still requires the base model and activations. Profile a small run at the target context and batch before estimating memory, duration and cost.
3. Save the adapter, base-model identifier and configuration and evaluate both improvements and regressions. Test loading and rollback; an adapter is not necessarily a standalone model.

Inspect PCIe links, NUMA placement and NVLink support and identify the parallelism strategy

Steps and verification

1. Inspect PCIe links, NUMA placement and NVLink support and identify the parallelism strategy. More cards do not automatically pool memory or scale throughput linearly.
2. Establish a single-card baseline before scaling and separate compute, loading and synchronization time. For multiple hosts measure collective communication rather than relying on advertised port speed.
3. Check the GPU assignment, logs and error handling for each distributed process (rank). Prevent accidental device sharing and retain reproducible launch settings. Diagnose networking and version mismatches before disabling communication checks.

Document source, destination, protocol and port for each service

Steps and verification

1. Document source, destination, protocol and port for each service. Security groups, host firewalls and application permissions are separate layers and any can block a connection.
2. Keep a live session and console recovery before changing remote-access rules. Add the required allow rule before tightening access and cover both address families.
3. Test from allowed and denied sources and verify persistence after reboot. Record ownership, purpose and expiry and remove temporary openings after troubleshooting.

Start with availability, error rate, latency and critical workflow success, then add resource metrics

Steps and verification

1. Start with availability, error rate, latency and critical workflow success, then add resource metrics. Healthy CPU does not prove that checkout or login works.
2. Separate urgent service incidents from capacity trends. Configure how long a condition must persist, when to send a recovery notification and how to group duplicate alerts. Base thresholds on normal behavior and service objectives.
3. Give each critical alert an owner, investigation path and escalation route and test delivery. Monitor missing telemetry and the monitoring system itself; no data is not proof of health.

Inventory application, system, proxy and container logs and their growth rates

Steps and verification

1. Inventory application, system, proxy and container logs and their growth rates. Redact passwords, tokens and sensitive payloads even during debugging.
2. Configure rotation, compression and retention and confirm the process reopens its log. Deleting an open file may not reclaim space; use the service’s supported rotation mechanism.
3. Verify writing, permissions and search after rotation and alert on storage or collection failures. Protect centralized log access and transport and enforce the required retention period.

Inventory OS, application, database and driver versions

Steps and verification

1. Inventory OS, application, database and driver versions. Separate security fixes from feature upgrades and check reboot, migration and client-compatibility requirements.
2. Test on a representative environment and validate backups and rollback. Define expected impact, ownership and acceptance criteria for the maintenance window, not just a start time.
3. Verify versions, startup, monitoring and errors after updating, including a reboot where required. Keep change records; a successful installer exit is not complete service acceptance.

Distinguish clock accuracy, timezone and display formatting

Steps and verification

1. Distinguish clock accuracy, timezone and display formatting. A timezone issue may be cosmetic, while clock drift can break TLS, signatures, queues and distributed jobs.
2. Inspect the synchronization service, upstream reachability and host/guest policy. Avoid competing synchronization agents and diagnose drift instead of repeatedly setting the clock manually.
3. Assess database and job impact before a large production clock step. Retest cross-node timing, certificates and schedules, then monitor drift and standardize log timestamps.

Choose storage by recovery location and access needs and account for requests, retrieval and egress

Steps and verification

1. Choose storage by recovery location and access requirements, including storage, request, retrieval and egress charges. Retrieval delays must fit the RTO; minimum storage periods affect retention policy and cost.
2. Upload with scoped credentials and configure encryption, versioning and lifecycle rules. Verify delete permissions and rule scope because lifecycle policies can remove older backup versions.
3. Retrieve and validate a backup from an independent recovery environment and record time and cost. Confirm key availability; a successful upload is not an application restore test.

Determine whether the impact is one user, region, instance or the whole service and record timezone-aware timestamps

Steps and verification

1. Determine whether the impact is one user, region, instance or the whole service and record timezone-aware timestamps. Preserve errors and changes before restarting multiple components.
2. Work through DNS, networking, host, application and database layers. Prioritize critical service recovery and limit changes; assess data consistency before rollback or degradation.
3. Provide instance ID, destination, timestamps, reproduction and redacted logs without passwords or private keys. After recovery document cause and prevention and distinguish mitigation from permanent resolution.

Inventory domains, deployment points and ACME accounts and distinguish edge from origin certificates

Steps and verification

1. Inventory domains, deployment points and ACME accounts and distinguish edge from origin certificates. Updated files do not prove the service loaded them; inspect the served certificate.
2. Test renewal using the client’s supported method and verify HTTP challenge access or scoped DNS API rights. Keep logs because propagation and firewall changes can break validation.
3. Configure safe reload hooks and failure notifications plus independent external expiry checks. Monitor the real hostname and SNI rather than only local files so virtual-host mistakes are detected.

Separate public configuration from credentials and signing keys and keep real secrets out of repositories and images

Steps and verification

1. Separate public configuration from credentials and signing keys and keep real secrets out of repositories and images. Environment variables are not an encrypted vault and still require process and administrative access controls.
2. Issue separate least-privilege credentials per service and document owners and rotation. Redact and protect logs, dumps, backups and diagnostic output that may expose secrets.
3. Plan dual-key overlap or an explicit switchover order and validate new credentials before revoking old ones. Suspected exposure requires revocation and access-log review, not merely removing a public file.

Steps and verification

1. Record the Linux distribution and release, kernel version, CPU architecture and virtualization type. Match installation packages to the detected architecture rather than the product name.
2. Compare hostname, addresses, mounts and system time with the order and configuration record.
3. Keep a secret-free baseline and repeat the inventory after updates or migrations.

cat /etc/os-release
uname -r
uname -m
lscpu

Steps and verification

1. Create a separate account per administrator and verify its shell, home and groups rather than sharing root credentials.
2. Grant required privileges through supported sudo configuration and validate sudoers using visudo.
3. Keep an existing session while testing login and required commands in a new session before restricting old access.

id
sudo -l

Steps and verification

1. Identify the process user and inspect ownership and permissions on the file and every parent directory.
2. Distinguish file reads from directory traversal and grant write access only where required; avoid blanket 777 permissions.
3. Record the original state, test application access and inspect ACL, SELinux or AppArmor restrictions separately.

Steps and verification

1. Check the service status and journal for the current boot. Record the exit code, failure time and executable specified by ExecStart.
2. Check working directory, environment files, service user and dependencies; an interactive shell has a different environment.
3. Validate the changed configuration and run the required manager reload if you edited a unit file. Test startup, restart and application behavior before enabling the service at boot.

systemctl --failed

Steps and verification

1. Limit queries by service, time range and boot before exporting logs to reduce noise and sensitive-data exposure.
2. Correlate failures with updates, restarts and disk events and distinguish the first failure from retry noise.
3. Keep redacted excerpts and query bounds and check retention; missing historical logs do not prove an event never occurred.

journalctl --list-boots
journalctl -p err -b --no-pager -n 50

Steps and verification

1. Check for active update processes; a package lock may belong to legitimate automatic updates and should not simply be deleted.
2. Inspect process state and logs, then follow distribution guidance to repair interrupted transactions and review proposed changes.
3. Verify package state and service versions and schedule a reboot test if core system components were affected.

Steps and verification

1. Identify disks by UUID, capacity and filesystem instead of relying only on changeable device names.
2. Back up fstab and review mount points, options and boot dependencies, including network readiness for remote storage.
3. Verify mounts with console recovery available, then test reboot; do not format disks containing application data.

lsblk -f
findmnt

Steps and verification

1. Confirm the virtual disk was resized and inspect capacity at partition, LVM, encryption and filesystem layers.
2. Back up and use the filesystem-specific growth procedure; XFS and ext4 differ and shrink commands are not interchangeable.
3. Expand supported layers in order, verify free space and writes and stop if the layout is unsupported.

lsblk
df -Th

Steps and verification

1. Check both bytes and inode usage; free storage bytes do not prevent file-creation failure when inodes are exhausted.
2. Locate small-file concentrations and identify caches, sessions, mail or business data before choosing retention rules.
3. Clean in controlled batches using supported procedures, verify services and adjust expiration and monitoring.

df -h
df -i

Steps and verification

1. Correlate available memory, sustained swap traffic, latency and OOM events; some occupied swap alone is not a failure.
2. Identify growing processes and compare container or service limits with total host memory.
3. Address abnormal usage or capacity first; disabling swap can trigger earlier kills and requires workload testing.

free -h
vmstat 1 5

Steps and verification

1. Confirm OOM or cgroup-limit evidence rather than assuming every process exit is memory related.
2. Capture the affected process, peak usage, container limits and concurrent jobs, including exceptional inputs.
3. Bound concurrency and resources and retest after fixing leaks or resizing instead of hiding failures behind restarts.

Steps and verification

1. For too many open files, distinguish process and system limits and measure files, sockets and connection growth.
2. Inspect whether the application releases connections and handles; persistent growth usually needs a lifecycle fix.
3. After changing service limits verify the effective values and test memory use and reclamation under peak concurrency.

ulimit -n
cat /proc/sys/fs/file-nr

Steps and verification

1. Identify process ownership and supervision and use the service manager instead of fighting automatic restarts.
2. Prefer graceful shutdown with time for connection draining, queue acknowledgments and data flushes.
3. After restart verify listeners, dependencies, backlog and errors and record the cause and recovery duration.

Steps and verification

1. Review kernel, driver and boot-entry changes and retain a known-working version and console access.
2. Schedule maintenance, pause or move workloads and verify mount and service-start configuration.
3. After reboot check the running kernel, networking, storage and services, including GPU or third-party module compatibility.

uname -r
systemctl --failed

Steps and verification

1. Identify terminal, file, database and application encodings before treating garbled display as corrupted data.
2. Check UTF-8 locale and import settings and test conversions on a copy to avoid double conversion.
3. Round-trip samples containing non-Latin text, symbols and line breaks and document migration encoding settings.

locale
locale -a

Steps and verification

1. Record edition, version, patch level and architecture and confirm application and driver support.
2. Check the product terms for licensing responsibility and activation; successful boot does not establish entitlement.
3. Keep a record without license secrets and resolve activation through authorized support channels.

Get-ComputerInfo -Property WindowsProductName,WindowsVersion,OsBuildNumber,OsArchitecture

Steps and verification

1. Verify Remote Desktop is enabled and the account has logon rights, separating connectivity from authentication errors.
2. Check client compatibility, clock, lockouts and domain reachability and retain console access before changing NLA.
3. Retest from authorized sources and restore protections instead of leaving authentication weakened.

Steps and verification

1. Check the active network profile and listener because rules may apply only to domain, private or public networks.
2. Scope rules by application, port and source instead of disabling the firewall across profiles.
3. Test allowed and denied sources and verify behavior after reboot or profile changes.

Steps and verification

1. Filter system and application logs around the incident and capture source, event ID and full message.
2. Correlate service, disk, login and update events while accounting for timezone differences.
3. Export relevant redacted records and preserve originals rather than clearing logs before investigation.

Steps and verification

1. Inspect startup type, dependencies and logon identity and verify access to executables and data.
2. Account password rotation can break startup; update affected services and record the change scope.
3. Test both a service restart and a full system reboot. Confirm that the application starts and remains available without anyone signing in interactively.

Get-Service | Where-Object Status -eq Running

Steps and verification

1. Review pending updates, restart requirements and maintenance policy against permitted downtime.
2. Back up data and configuration and test dependencies before combining changes with a restart.
3. Verify update state, RDP, services and requests afterward and preserve failures before choosing rollback.

Steps and verification

1. Confirm platform resizing and identify the correct disk and unallocated space in Disk Management.
2. Review filesystem and partition layout because adjacency and partition type affect expansion.
3. Use a supported procedure after backup and verify drive letters, permissions and writes without initializing existing data disks.

Get-Disk
Get-Volume

Steps and verification

1. Specify the account and working directory for the task, and whether it must run when no user is signed in. Choose a credential-storage method that complies with your account policy.
2. Configure triggers, retries, the maximum run time and what happens if a previous run is still active. Prevent overlapping runs when the task cannot safely run twice.
3. Inspect history, exit codes and outputs and test after reboot without an interactive session.

Steps and verification

1. Review hostname, IP, port and application pool; multiple HTTPS sites require correct SNI and certificate selection.
2. Verify pool identity, content permissions, private-key access and the certificate chain.
3. Test the served certificate through the actual hostname and validate restart and renewal behavior.

Steps and verification

1. Select CPU, memory, disk latency, queue and network counters relevant to the incident.
2. Capture normal and peak periods and correlate processes, scheduled jobs and updates.
3. Compare consistent sampling intervals before and after changes and resize only after identifying the bottleneck.

Steps and verification

1. Verify delegated nameservers match the service where records were changed; registrar and authoritative DNS may differ.
2. Compare authoritative and recursive responses, including TTL, response code and full hostname.
3. Allow the old TTL to expire and investigate delegation, DNSSEC and conflicting records if inconsistency persists.

Steps and verification

1. Identify whether the name is an apex or subdomain; ordinary CNAME records cannot freely coexist with other data at the same name.
2. Apex aliasing depends on provider-specific flattening or ALIAS behavior rather than universal DNS support.
3. Verify final A/AAAA answers and effects on mail and verification records and retain rollback settings.

Steps and verification

1. For resolver SERVFAIL errors compare validating and diagnostic responses before changing application services.
2. Check parent DS records against authoritative DNSKEY data; stale DS records after migration can break validation.
3. Follow supported rollover or removal procedures and verify multiple resolvers after cache updates.

Steps and verification

1. Record destination, port and source and test the actual application protocol instead of relying only on ping.
2. Timeouts can indicate filtering or unresponsive paths; refusal calls for checking listeners and explicit reject rules.
3. Correlate client and server evidence and inspect service state through the console before changing access policy.

Steps and verification

1. Prioritize the destination and application behavior because routers can rate-limit ICMP without dropping forwarded traffic.
2. Repeat at comparable times and capture both directions; outbound and return paths can differ.
3. Correlate persistent downstream and destination loss with requests and ISP details before escalating a routing incident.

Steps and verification

1. If small requests work but uploads or TLS stall, inspect tunnel encapsulation and interface MTU along the path.
2. Use supported path-MTU diagnostics and distinguish blocked ICMP from real packet-size limits.
3. Test changes on a controlled path with large transfers and HTTPS and keep original settings and console recovery.

Steps and verification

1. Confirm assigned subnets, gateways and routing scope rather than placing arbitrary private addresses on public interfaces.
2. Check the most specific matching route, the default route and any policy routing for multiple interfaces. An unexpected return path can cause connection failures.
3. Test local, routed and public access separately and retain route backups and console access.

ip address show
ip route show
ip -6 route show

Steps and verification

1. Inventory listeners and bindings and separate application, database and management ports.
2. Test authorized external reachability over IPv4 and IPv6; local listening alone does not prove public exposure.
3. Retest after closing unnecessary access and include container publishing and proxy configuration.

ss -lntup

Steps and verification

1. Map the CDN, load balancer and reverse-proxy chain and identify trusted sources at each layer.
2. Trust forwarded headers only from configured proxies rather than accepting arbitrary X-Forwarded-For for access control.
3. Test direct, proxied, IPv6 and multi-hop requests and verify logging and rate-limit identity.

Steps and verification

1. Separate liveness from readiness and make the endpoint represent the ability to serve traffic.
2. Set timeouts, intervals and consecutive-result thresholds to avoid flapping every backend during brief disturbances.
3. Test backend shutdown, dependency failures and recovery. Confirm that existing connections finish as intended and that a recovered backend rejoins only when it is ready.

Steps and verification

1. Identify engines, transaction behavior and server version before choosing a supported logical or physical backup.
2. Record backup time, binlog coordinates and write state; copying live files does not guarantee consistency.
3. Restore independently and verify tables, grants and application queries rather than relying only on backup exit status.

Steps and verification

1. Sum pool sizes across application instances and include workers and administrative connections.
2. Investigate leaks, long transactions and slow queries before raising limits with additional memory cost.
3. Set acquisition timeouts and backpressure and validate wait time, errors and reclamation under peak load.

Steps and verification

1. Collect real slow queries, parameter distributions and frequency instead of optimizing an unrepresentative sample.
2. Review scans, joins and sorts alongside index selectivity and write cost.
3. Test at representative scale and plan index creation for its locking, storage and operational impact.

Steps and verification

1. Check the replication topology, server versions and receiver/applier status. Distinguish a lost connection from delays receiving events or applying transactions.
2. Correlate large transactions, write bursts, storage and replica query load; lag seconds alone are not a complete health signal.
3. Validate consistency and application read routing after recovery rather than blindly skipping replication errors.

Steps and verification

1. Choose logical or physical backup based on recovery target, compatibility and downtime.
2. Physical recovery needs compatible versions and a complete WAL chain; copying live data directories is insufficient.
3. Restore roles, extensions, encodings and queries in isolation and measure duration and storage needs.

Steps and verification

1. Inspect update patterns, dead tuples and long transactions that can prevent cleanup.
2. Review autovacuum activity and distinguish ordinary vacuum from VACUUM FULL locking and rewriting behavior.
3. Assess workload impact before making changes. Monitor transaction-ID age, storage growth and query latency, and schedule any table rewrite rather than starting it unexpectedly on a large production table.

Steps and verification

1. Choose persistence according to acceptable loss and understand different RDB and AOF durability boundaries.
2. Reserve memory and disk for background work and inspect persistence failures and paths.
3. Restore a copy and verify keys and expirations; persistence does not replace off-host backups.

Steps and verification

1. Use latency, command statistics and access distribution without unplanned full scans on busy instances.
2. Large keys affect transfer, deletion and replication; evaluate splitting and expiration with application compatibility.
3. Test during a low-impact window and observe latency and reclamation, checking support before asynchronous deletion.

Steps and verification

1. Set daily, weekly and monthly retention from recovery goals, business cycles and cost instead of keeping only the latest copy.
2. Track incremental dependencies because deleting a base can invalidate later backups.
3. Validate deletion rules and recovery samples before automation and alert on failures, capacity and stale backups.

Steps and verification

1. Document transport and at-rest protection and the keys, permissions and tools needed for restoration.
2. Store keys separately with restricted access and a recovery path that survives host failure.
3. Test independent decryption and preserve access to historical backups during key rotation.

Steps and verification

1. Record registry, version and architecture; mutable tags can resolve to different content over time.
2. Deploy validated versions or digests and retain configuration and the previous release instead of relying on latest.
3. Test pull, startup, data compatibility and rollback rather than treating an image download as acceptance.

docker image ls --digests

Steps and verification

1. Identify data that must survive replacement and distinguish named volumes, bind mounts and writable layers.
2. Check host paths, UID/GID mappings and permissions without mounting the entire host filesystem.
3. Recreate a test container and verify persistence, using database-aware procedures for volume backups.

Steps and verification

1. Distinguish variable substitution in Compose files from variables passed to the running container. Check how shell variables, environment files and Compose settings interact before relying on a value.
2. Inspect resolved configuration without leaking secrets and record working directory and file selection.
3. Verify the application receives the intended values and recreate or restart according to the change.

Steps and verification

1. Probe readiness with a command available inside the container and reasonable timeouts.
2. An unhealthy status does not necessarily restart a container; engine and orchestration policies determine behavior.
3. Test exits, dependency failures and host reboot and prevent restart loops or repeated non-idempotent initialization.

docker ps -a

Steps and verification

1. Inspect both logging drivers and application files because duplicate logging increases storage use.
2. Set size and retention and check whether existing containers need recreation for new settings.
3. Verify queries and collection after rotation and alert on host capacity and ingestion failures.

Steps and verification

1. Inside a container localhost refers to that container, not the host or another database container.
2. Check shared networks, service-name resolution and listeners and distinguish internal from published ports.
3. Publish only required services and test internally and externally to prevent accidental database exposure.

Steps and verification

1. Do not bake repository tokens into COPY layers or persistent environment values; deleting files may leave older layers intact.
2. Use temporary secret mounts supported by the build tool. Limit the build context—the files available to the build—and maintain ignore rules to keep credentials out.
3. Inspect history and logs and rotate exposed credentials and affected images rather than only deleting source files.

Steps and verification

1. Verify old and new versions can coexist and share the schema; a single instance does not automatically support zero downtime.
2. Start and validate replacements before stopping new traffic to old instances and draining existing work.
3. Increase traffic against error and latency gates and account for data changes when rolling back.

Steps and verification

1. Use the required Python version in a dedicated environment instead of overwriting system packages.
2. Pin dependencies and record sources and architecture, including native-extension library requirements.
3. Test imports, startup and workflows as the service user and retain reproducible dependency records.

python --version
python -m pip --version

Steps and verification

1. Select a supported compatible runtime and record package-manager and lockfile versions.
2. Run as an unprivileged service identity with explicit working directory, environment and proxy access.
3. Test boot startup, graceful shutdown and log rotation through the service manager and verify APIs after updates.

node --version
npm --version

Steps and verification

1. Map management interfaces, physical NICs and bridges and retain configuration and console access.
2. Follow the provider’s bridge or routed-subnet model rather than placing gateways or unassigned addresses arbitrarily.
3. Test both traffic directions with a guest before production and restore through the console if management access fails.

Steps and verification

1. Install a supported guest-agent version for the platform and operating system.
2. Verify state reporting and supported operations and evaluate filesystem freeze alongside application consistency.
3. Retest shutdown, address reporting and backup integration after updates; agent availability is not complete service health.

Steps and verification

1. Remove temporary credentials, sensitive logs and nonshareable keys while retaining updates and required agents.
2. Use the operating system’s supported preparation process for machine identifiers, SSH host keys and network settings. Clones must not accidentally reuse identities or addresses that should be unique.
3. Validate first boot, accounts and initialization on a test clone before bulk provisioning.

Steps and verification

1. Check the image datasource and platform injection method and distinguish first boot from restart and identity changes.
2. Inspect stage logs, YAML syntax, networking dependencies and command exit status.
3. Test on a copy because resetting initialization can rerun user creation and scripts on production data.

Steps and verification

1. Monitor thin-pool data and metadata independently because provisioned capacity can exceed physical storage.
2. Reserve for snapshots, backup staging and guest growth before the pool fills.
3. Verify the storage layout and backups before expanding the pool. Test alert thresholds and guest writes, and track whether provisioned capacity exceeds what the storage can actually supply.

Steps and verification

1. Record pool state, device errors and redundancy; a scrub validates stored data but is not a backup.
2. Schedule around workload needs and correlate I/O impact and error trends.
3. Review results and verify device identity, backups and remaining redundancy before replacement.

Steps and verification

1. Compare host features and the cluster compatibility floor; host passthrough can expose unsupported destination instructions.
2. Balance performance with migration requirements and record software needing particular instruction sets.
3. Test boot and migration on a guest and schedule downtime if changing the CPU model requires shutdown.

Steps and verification

1. Compare host and guest clocks and synchronization services, including pause, resume and high-load events.
2. Avoid competing agents and use platform-recommended clock sources and guest tools.
3. Measure drift after migration and reboot and verify certificates, authentication and scheduled jobs.

Steps and verification

1. Inventory networking, storage, permissions, guest definitions and cluster state separately from guest data.
2. Export through supported procedures and encrypt access-controlled copies because configurations can contain credentials.
3. Document required versions and rebuild steps and rehearse recovery on a test host.

Steps and verification

1. Observe guest and host metrics together because host contention can cause CPU waits and storage latency.
2. Review CPU weights, memory, I/O and network limits before adding application threads inside the guest.
3. Validate fairness and tail latency under representative concurrency and preserve maintenance headroom.

Steps and verification

1. The CUDA version shown by nvidia-smi indicates driver support, not necessarily an installed matching toolkit.
2. Determine whether the application uses bundled runtimes or compiles extensions before choosing compatible components.
3. Record driver, runtime, framework and compiler versions and verify a small computation before reinstalling components.

nvidia-smi

Steps and verification

1. Record host indices and UUIDs; filtering devices for a process or container can renumber them from zero.
2. Assign visible devices and process counts explicitly to avoid accidental sharing.
3. Compare logs, actual memory use and device identity after startup and recheck mappings after scheduling changes.

nvidia-smi --query-gpu=index,uuid,name --format=csv

Steps and verification

1. Check hardware and framework support and distinguish the types used for weights, activations and optimizer state.
2. Lower precision can reduce memory requirements and computation, but numeric range, stability and operator support must be tested on the target hardware.
3. Use the same evaluation inputs to compare output quality, loss, throughput and peak GPU memory. If a run produces NaN values, retain the triggering inputs, logs and runtime configuration.

Steps and verification

1. Account for model architecture, context, concurrency and cache precision; weights do not include KV-cache memory.
2. Test realistic length distributions and maximum requests because prompts and generation affect peaks.
3. Bound context, concurrency and queues and measure cache-policy effects before scaling traffic.

Steps and verification

1. Fix model, software, precision and input/output lengths and separate cold-start from warmed results.
2. Report throughput, first-token and tail latency, errors and peak memory rather than a single score.
3. Record power, temperature and competing workloads and repeat identical settings instead of treating a peak as a sustained guarantee.

Steps and verification

1. Verify quantization format, framework kernels and GPU support rather than assuming matching suffixes are interchangeable.
2. Compare original and quantized outputs on fixed long-context, special-character and critical-task samples.
3. Measure loading peaks, throughput and errors; smaller weights do not proportionally shrink every runtime allocation.

Steps and verification

1. Separate reading, preprocessing, transfer and compute time; low utilization does not necessarily mean a GPU fault.
2. Inspect storage, small files, CPU workers and RAM because extra workers can create contention.
3. Use the same training samples while adjusting prefetching and the number of data-loader workers. Compare end-to-end training throughput and memory use, not just one stage’s speed.

Steps and verification

1. Inventory downloaded caches, expanded copies, outputs and checkpoints because downloads can temporarily require multiple copies.
2. Set cache paths and retention and check running-process dependencies before removing files.
3. Test saving and restoration and monitor bytes and inodes so long runs do not fail at the final write.

Steps and verification

1. Capture temperature, power, clocks and throttle reasons alongside workload and environmental changes.
2. Distinguish normal power limits from cooling problems and avoid unvalidated power or fan changes.
3. Retain trends and reproducible load for hardware support and retest stable throughput after remediation.

nvidia-smi --query-gpu=name,temperature.gpu,power.draw,clocks.gr --format=csv

Steps and verification

1. Use trusted model sources and record versions, licenses and hashes; weights and dependency code have different risks.
2. Review custom remote code before enabling it and test with restricted secrets, filesystem and networking.
3. Pin reproducible versions, review updates and avoid exposing sensitive prompts or tokens in logs and examples.