A practical cloud log retention policy should preserve enough evidence for incident response and audits without keeping every record in expensive, searchable storage. This guide shows how to classify logs, estimate hot and archive capacity, set access and deletion controls, and test that older records can still be retrieved when needed.
Overview
Cloud logs are not one uniform data set. Infrastructure logs may show instance, network, or control-plane activity; application logs explain user-facing failures; security logs record authentication, authorization, and threat signals; and audit logs may document administrative changes. Each category has a different operational value, search requirement, and retention risk.
The right answer to “how long should security logs be retained?” is therefore a documented decision rather than a universal number. Start by identifying the business purpose of each log source:
- Hot logs: retained in a searchable logging platform for fast investigation, alerting, and operational troubleshooting.
- Warm logs: kept in a lower-cost searchable or semi-searchable tier when investigations are less frequent but retrieval speed still matters.
- Cold archives: stored in object storage or another archive destination for longer-term evidence, trend analysis, or recovery from delayed investigations.
A retention policy should define the minimum period, the storage tier, the owner, the permitted users, and the deletion rule for every important source. It should also distinguish ordinary expiration from a legal hold or active investigation. A scheduled deletion process must never remove records that have been placed on hold.
For a broader backup policy, compare this approach with the Cloud Backup Retention & Restore Playbook. Logs support investigation and accountability; backups support restoration. They should be governed separately even when both use object storage.
How to estimate
Estimate log storage in two layers: the volume that must remain readily searchable and the volume that can move to a lower-cost archive. Use measured data where possible. If measurement is not available, label each input as an assumption and replace it after collecting at least one representative billing or ingestion period.
A simple monthly volume calculation is:
Monthly raw volume = daily ingest × days in the month
Then account for copies, indexes, compression, and metadata:
Billable working volume = monthly raw volume × replica factor × index or overhead factor × compression factor
Use the provider’s actual billing model for the final estimate. Some platforms charge separately for ingestion, searchable storage, queries, exports, and archive retrieval. Do not treat an archive storage estimate as a complete logging budget until those related charges have been checked.
For a retention period, divide the total policy into tiers. For example:
- Keep recent operational and security logs in the hot tier for the period your responders normally investigate.
- Transition older records to archive storage using a lifecycle rule.
- Delete records at the end of the approved retention period unless a hold applies.
Estimate retrieval cost separately. A low-cost archive is only useful if your team can afford and authorize a restore during an incident. Record expected retrieval volume, restore time, export format, and the person responsible for approving access.
Inputs and assumptions
Build a source-by-source retention matrix rather than assigning one number to the entire environment. The following template is a useful starting point:
| Log source | Primary use | Hot or searchable period | Archive period | Owner |
|---|---|---|---|---|
| Identity and access | Investigate sign-ins, privilege changes, and access anomalies | Set according to response needs | Set according to risk and approved requirements | Security or platform team |
| Cloud control plane | Review resource and configuration changes | Set according to change and incident workflows | Set according to audit and investigation needs | Cloud operations |
| Application | Debug errors, transactions, and performance issues | Usually aligned with release and support cycles | Keep only what has ongoing analytical or evidentiary value | Application owner |
| Network and edge | Analyze access patterns, abuse, and availability events | Align with incident response requirements | Archive selected fields or summaries when full detail is unnecessary | Network or security team |
Document these assumptions before calculating cost:
- Average daily ingest and expected growth from new services, regions, or higher traffic.
- Peak ingest during incidents, deployments, or attack traffic.
- Whether logs are duplicated across regions, accounts, clusters, or backup destinations.
- Compression and indexing behavior supplied by the logging or object-storage platform.
- Required search latency and the maximum acceptable archive retrieval time.
- Encryption, immutability, access logging, and separation-of-duties requirements.
- Rules for redaction, personal data minimization, legal holds, and secure deletion.
Limit access to the smallest group that needs it. Use separate roles for viewing, exporting, changing retention, and deleting data. Protect the archive account from ordinary production credentials, monitor policy changes, and test that the log destination cannot be silently disabled. Infrastructure-as-code teams can also review retention and destination settings for drift using the Infrastructure as Code Drift Detection Guide.
Worked examples
Consider an illustrative environment that produces 20 GB of raw logs per day. Assume the team keeps 14 days in a hot tier, moves older data to archive storage, and applies a 1.2 overhead factor for indexing and metadata. These figures are examples for demonstrating the method, not a price or provider benchmark.
Hot raw volume: 20 GB × 14 days = 280 GB.
Hot working estimate: 280 GB × 1.2 = 336 GB before provider-specific compression, replicas, and billing rules.
If the approved archive period is 180 days, the archive population after the first 14 days is approximately 20 GB × 166 days, or 3,320 GB raw. Applying the same illustrative overhead factor produces 3,984 GB before compression and any archive replication. In practice, calculate each source separately if application logs compress differently from audit records or if only selected fields are archived.
Now consider a second case: a small web application generates 2 GB per day, but an incident causes a temporary increase to 10 GB per day. A policy based only on the normal rate may under-size ingestion limits and archive capacity. Add a peak-volume scenario to the worksheet and decide whether the system should retain full-fidelity logs, sample noisy records, or route high-volume events to a separate destination. Sampling should never remove the fields needed for security investigations without an explicit risk decision.
For each example, perform a restore test. Export a defined time window, retrieve it from the archive, verify timestamps and fields, confirm that the files are readable by the approved analysis tool, and record elapsed time and cost. A retention policy that cannot produce usable evidence is incomplete.
When to recalculate
Review the policy at least annually and after any material change to the environment. Recalculate sooner when logging prices, query charges, archive retrieval terms, lifecycle features, or provider limits change. Also review it after a security incident, audit finding, major application release, migration, new region deployment, Kubernetes expansion, or change in data classification.
Use a quarterly checklist:
- Compare actual daily and peak ingest with the assumptions in the policy.
- Identify sources that are missing, duplicated, unexpectedly verbose, or no longer useful.
- Recalculate hot, warm, and archive capacity using current provider inputs.
- Confirm lifecycle transitions and deletion jobs completed without exceptions.
- Review privileged access, exports, policy changes, and legal holds.
- Restore a representative sample from each archive tier and document the result.
Keep the matrix, cost worksheet, access review, and restore-test record together with the policy. If you are also reviewing broader infrastructure spend, the Cloud Cost Optimization Checklist can help connect log storage decisions with other cloud cost controls. Revisit the policy whenever the inputs change; retention is not a set-and-forget setting, but a balance between usable evidence, operational resilience, compliance obligations, and storage cost.