How long should you keep audit logs? A practical retention guide for small teams
Keep audit logs long enough to investigate, meet your obligations, and prove what happened — but separate fast searchable data from long-term archive data.
The short answer: keep audit logs as long as their purpose requires — and don’t use one retention period for every kind of log. A week of searchable application logs can be enough for day-to-day troubleshooting. Authentication, administrative, and audit-relevant events often need a longer, documented retention period. The right number comes from your incident-response needs, contracts, sector rules, and privacy obligations — not from the default setting in your logging tool.
This is a technical decision guide, not legal advice. If a law, regulator, customer contract, or internal policy applies to your organisation, have the responsible legal, privacy, and compliance stakeholders confirm the final retention schedule.
Start with the question the log must answer
“Keep logs for a year” sounds simple, but it hides several different jobs:
- Troubleshooting: What changed after yesterday’s deployment?
- Security investigation: When did an attacker first authenticate, and what did they do afterwards?
- Audit evidence: Who changed a privileged setting, and can we show the sequence of events?
- Capacity and service analysis: Which system or endpoint produced the failures?
Each job has a different time horizon, access pattern, and cost. Teams normally investigate production errors within hours or days. A security incident may be discovered weeks later. An audit request can arrive long after the systems involved have changed.
NIST describes its current log-management work as a planning guide for improvements that support both recommended practices and regulatory requirements.[1] That is the useful mindset: retention is not an isolated storage setting. It is a documented operating decision tied to the risks you need to manage.
Separate hot search from long-term retention
The most expensive version of retention is keeping every event instantly searchable forever. It is rarely necessary.
Instead, separate data into at least two layers:
- Hot, searchable data for active monitoring, incident triage, and investigations. This is the data analysts need to query quickly.
- Archive data for longer-term evidence and exceptional investigations. It can be slower and cheaper, as long as it stays protected, retrievable, and understandable when needed.
For LogScale users, this distinction matters because repository retention controls what is available for search. In the cloud product, age-based retention uses the event’s @timestamp; LogScale deletes data in segments and restricts searches to the configured interval.[3] Do not assume a backup, an archive, and a searchable LogScale repository are interchangeable. Decide which one fulfils which purpose.
Archive the evidence you need, not every raw event
An archive does not need to be a second copy of the whole repository. For a long-lived investigation record, I prefer to define the small set of LogScale queries that answer the questions I expect to revisit: authentication activity, privilege changes, configuration changes, and the alerts or cases that explain a response.
LogScale Archive follows that model. It runs a scheduled, repository-scoped LogScale query and stores its results locally; the web UI searches stored results, not a live LogScale repository.[4] Retention is configured per query version, so a short-lived operational query does not force the same policy on a high-value audit query.[5]
That boundary matters. A query-result archive is evidence selected for a purpose. It is not a complete backup of the original repository, and it cannot recover data a query never selected. Keep a separate backup and restore plan for the Archive database itself.
A practical starting matrix
Use this as a conversation starter, not as a compliance schedule:
| Data type | Primary purpose | Searchable period | Longer-term archive? |
|---|---|---|---|
| Application and debug logs | Troubleshooting | Days to a few weeks | Usually no, unless needed for a specific investigation |
| Infrastructure and service logs | Operations and incident response | Weeks to a few months | Sometimes |
| Authentication and privilege events | Security investigation and audit trail | Months, based on risk | Often |
| Administrative changes and access records | Audit evidence and investigation | Months, based on obligations | Often |
| Security alerts and investigation cases | Evidence of response and decisions | Long enough to reconstruct the case | Often |
| Selected query results | Durable evidence for one defined question | Only while operationally useful | Yes, when that question outlives hot search |
The value of the matrix is not the labels. It forces the questions that a single retention value avoids: What would we lose if the data vanished tomorrow? How late are incidents usually detected? Who needs the data? Can we still read and trust it after the original system is gone?
Privacy is part of the retention design
Logs frequently contain personal data: usernames, IP addresses, device identifiers, email addresses, and sometimes application payloads. Retaining logs “just in case” is therefore not a neutral option.
The GDPR’s storage-limitation principle says that personal data should be kept in a form that identifies people for no longer than necessary for the purpose of processing.[2] That does not mean security logs must be deleted immediately. It means the purpose, necessity, access controls, and retention period should be explicit and defensible.
A practical response is to document, for every important log class:
- the purpose of collection;
- the system owner;
- who can search it;
- the searchable-retention period;
- the archive-retention period, if any; and
- how deletion or expiry is verified.
This also makes future changes easier. When someone asks why a repository holds 90 days instead of 30, or why an archive query keeps 12 months of privilege changes, the answer should not be “because that was the default.”
Design for an investigation that starts late
The retention period should reflect your detection reality, not your ideal one. If a suspicious account is discovered after 45 days but authentication events disappear after 30, the investigation begins with a blind spot.
That does not automatically mean every raw event needs to remain in hot storage for a year. You can retain high-value events longer than low-value volume. Examples include successful and failed authentication, privilege changes, configuration changes, audit events, and security alerts. In contrast, highly verbose debug output may only be valuable for a short troubleshooting window.
The key is to preserve enough context to reconstruct a meaningful timeline. An archived authentication event without host, source address, user identity, timestamp, and relevant action may be cheaper to keep but less useful to investigate.
Make retention enforceable, not aspirational
A policy that says “keep audit logs for 12 months” is incomplete if nobody knows where the data lives or whether deletion actually happens.
For each repository or archive, test these four things:
- Expiry: Does data disappear when the retention period ends?
- Search boundary: Can users query only the intended hot window?
- Archive retrieval: Can the team retrieve a specific period when an investigation requires it?
- Access control: Can only authorised people access sensitive historical events?
For a query-result archive, add two checks: the query must select the fields needed to reconstruct the timeline, and the worker must have completed the scheduled run. A green connection test alone proves neither. If an investigation needs a period outside the hot window, retrieve one known archived result before declaring the policy operational.
Document the test and repeat it after major platform or policy changes. Retention is a control only when the configured behaviour matches the documented intention.
A sensible next step for a small team
Do not begin by choosing a universal number of days. Pick your three highest-value data classes — for example authentication events, privileged changes, and application errors — and write down their purpose and investigation window.
Then set a modest searchable period that your team can afford and operate. If a class needs to outlive that window, define an archive path before the data ages out. Start with one query for authentication or privileged changes, test its results, activate its schedule, and document why that query earns the longer retention. This creates a real retention strategy without turning a small logging project into a compliance programme.
Get the queries
Retention only helps if you can use the data while it is available. Follow the LogScale Archive installation walkthrough to create, test, and activate the first archive query. Browse mylogscale.com for LogScale queries you can adapt for authentication activity, errors, and other investigation starting points. Updates land there, not in this article.