Sebastian Selig · Field Notes

How to reduce alert noise in LogScale

Sun, 13 Sep 2026 00:00:00 GMT

The short answer

I recommend separating early information from alerts that deserve additional attention: use email for the first notification and a separate Teams alert with a longer validation period. Define both conditions around the service, because neither one failed check nor a fixed waiting period means the same thing in every environment. Keep Teams selective so that it does not become a second inbox full of messages people stop reading.

My own email inbox is the starting point for this recommendation. Repeated notifications and brief server interruptions produce messages that eventually receive less attention. A service failure that matters can then disappear among notifications that did not need an immediate response.

The question is how to decide which condition deserves a message, when to send it, and which channel should carry it.

The factors

1. What the check actually confirms

A failed check provides information, but its meaning depends on what it tests. A server briefly becoming unreachable and a service remaining unavailable are different conditions. Treating both as equally urgent makes the messages harder to distinguish.

Validation must fit the environment. For one service, that might mean requiring a failure to persist for a defined period. For another, the check itself may already combine enough evidence to justify an immediate notification.

Before choosing a delay, describe what the result proves. Does it show one unsuccessful connection attempt, several failed checks, or a service that users cannot access? That distinction is the basis for the notification condition.

2. How much delay the service can tolerate

A longer validation period gives a brief interruption time to clear before generating an additional message. It also delays that message when the failure continues.

The responsible person must decide whether that trade-off is acceptable. For example, a temporary interruption to a service needed later in the day may allow more validation time than an unavailable service required for work happening now. These are decision examples, not default severity assignments.

I would choose the acceptable delay for each service before choosing the alert settings. A convenient value in the interface should not determine how long a significant failure goes without attention.

3. What each channel means

In this strategy, email carries the earlier information. Teams carries the additional notification after the stricter condition is met.

That distinction only helps if it remains consistent. Sending every email notification to Teams immediately would give recipients the same messages in two places. It would not address the original problem.

The message still provides information rather than taking action on the recipient’s behalf. It should make the affected service and the detected condition clear enough for the recipient to decide what to do.

4. Whether another message adds information

Repeated notifications about the same unchanged failure deserve their own decision. A reminder may be useful, but receiving the same message again does not necessarily tell the recipient anything new.

LogScale supports throttling to limit repeated actions. Its documentation distinguishes the query time window from the throttle period; these settings serve different purposes. See the LogScale trigger documentation.

Use validation to decide whether a condition deserves a notification. Use repetition controls to decide when another notification about that condition is useful.

Concrete recommendation by situation

For a service where brief interruptions are acceptable, start with two independent alerts. Give the email alert the earlier notification condition. Give the Teams alert a condition that requires evidence over a longer period.

For illustration, a brief interruption might meet the email condition and recover before it meets the Teams condition. A continuing failure would meet both conditions, so the recipient would receive the additional Teams message. This describes the intended behavior, not a tested configuration or a universal timing recommendation.

For a service where waiting is unacceptable, do not add a delay merely to preserve this two-stage pattern. Use the validation that the service requires, which may already be built into the check, and choose the channel according to its importance.

For information that only needs attention during defined working hours, consider restricting notifications to those times. Keep that decision separate from failure validation: business hours determine when information is relevant to the recipient, while validation determines whether the failure condition is sufficiently established.

Get the query: Restrict Alerts to Weekdays and Business Hours.

Apply such a restriction only where the service can tolerate it. An outage that requires attention outside business hours needs a notification path that remains active then.

Edge cases and objections

A longer search window does not prove a continuing outage

The Teams alert is independent. It is not simply a delayed copy of the first email.

LogScale triggers use query results to initiate actions, as described in the official trigger overview. The second query must therefore express the stronger condition you intend to detect.

If it only finds an earlier failure somewhere in a larger window, it may still find that event after recovery. The condition must distinguish the evidence for a continuing problem from an old failure. The right way to do that depends on the available checks and events.

Too much suppression can hide a separate failure

Reducing repetition should not prevent a different service failure from becoming visible. Consider two services failing close together: a rule intended to silence reminders for the first should not accidentally silence the first notification for the second.

Before relying on the setup, check three scenarios in your own environment: a brief interruption, a continuing failure, and a separate failure while the first remains active. Confirm which messages arrive through each channel.

Start with one noisy alert. Define what it proves, when a second message becomes useful, and what should reach Teams. Then verify those three scenarios before applying the same strategy more broadly.

How long should you keep audit logs? A practical retention guide for small teams

Sun, 16 Aug 2026 00:00:00 GMT

The short answer: keep audit logs as long as their purpose requires — and don't use one retention period for every kind of log. A week of searchable application logs can be enough for day-to-day troubleshooting. Authentication, administrative, and audit-relevant events often need a longer, documented retention period. The right number comes from your incident-response needs, contracts, sector rules, and privacy obligations — not from the default setting in your logging tool.

This is a technical decision guide, not legal advice. If a law, regulator, customer contract, or internal policy applies to your organisation, have the responsible legal, privacy, and compliance stakeholders confirm the final retention schedule.

Start with the question the log must answer

“Keep logs for a year” sounds simple, but it hides several different jobs:

Troubleshooting: What changed after yesterday's deployment?
Security investigation: When did an attacker first authenticate, and what did they do afterwards?
Audit evidence: Who changed a privileged setting, and can we show the sequence of events?
Capacity and service analysis: Which system or endpoint produced the failures?

Each job has a different time horizon, access pattern, and cost. Teams normally investigate production errors within hours or days. A security incident may be discovered weeks later. An audit request can arrive long after the systems involved have changed.

NIST describes its current log-management work as a planning guide for improvements that support both recommended practices and regulatory requirements.[1] That is the useful mindset: retention is not an isolated storage setting. It is a documented operating decision tied to the risks you need to manage.

Separate hot search from long-term retention

The most expensive version of retention is keeping every event instantly searchable forever. It is rarely necessary.

Instead, separate data into at least two layers:

Hot, searchable data for active monitoring, incident triage, and investigations. This is the data analysts need to query quickly.
Archive data for longer-term evidence and exceptional investigations. It can be slower and cheaper, as long as it stays protected, retrievable, and understandable when needed.

For LogScale users, this distinction matters because repository retention controls what is available for search. In the cloud product, age-based retention uses the event's @timestamp; LogScale deletes data in segments and restricts searches to the configured interval.[3] Do not assume a backup, an archive, and a searchable LogScale repository are interchangeable. Decide which one fulfils which purpose.

Archive the evidence you need, not every raw event

An archive does not need to be a second copy of the whole repository. For a long-lived investigation record, I prefer to define the small set of LogScale queries that answer the questions I expect to revisit: authentication activity, privilege changes, configuration changes, and the alerts or cases that explain a response.

LogScale Archive follows that model. It runs a scheduled, repository-scoped LogScale query and stores its results locally; the web UI searches stored results, not a live LogScale repository.[4] Retention is configured per query version, so a short-lived operational query does not force the same policy on a high-value audit query.[5]

That boundary matters. A query-result archive is evidence selected for a purpose. It is not a complete backup of the original repository, and it cannot recover data a query never selected. Keep a separate backup and restore plan for the Archive database itself.

A practical starting matrix

Use this as a conversation starter, not as a compliance schedule:

Data type	Primary purpose	Searchable period	Longer-term archive?
Application and debug logs	Troubleshooting	Days to a few weeks	Usually no, unless needed for a specific investigation
Infrastructure and service logs	Operations and incident response	Weeks to a few months	Sometimes
Authentication and privilege events	Security investigation and audit trail	Months, based on risk	Often
Administrative changes and access records	Audit evidence and investigation	Months, based on obligations	Often
Security alerts and investigation cases	Evidence of response and decisions	Long enough to reconstruct the case	Often
Selected query results	Durable evidence for one defined question	Only while operationally useful	Yes, when that question outlives hot search

The value of the matrix is not the labels. It forces the questions that a single retention value avoids: What would we lose if the data vanished tomorrow? How late are incidents usually detected? Who needs the data? Can we still read and trust it after the original system is gone?

Privacy is part of the retention design

Logs frequently contain personal data: usernames, IP addresses, device identifiers, email addresses, and sometimes application payloads. Retaining logs “just in case” is therefore not a neutral option.

The GDPR's storage-limitation principle says that personal data should be kept in a form that identifies people for no longer than necessary for the purpose of processing.[2] That does not mean security logs must be deleted immediately. It means the purpose, necessity, access controls, and retention period should be explicit and defensible.

A practical response is to document, for every important log class:

the purpose of collection;
the system owner;
who can search it;
the searchable-retention period;
the archive-retention period, if any; and
how deletion or expiry is verified.

This also makes future changes easier. When someone asks why a repository holds 90 days instead of 30, or why an archive query keeps 12 months of privilege changes, the answer should not be “because that was the default.”

Design for an investigation that starts late

The retention period should reflect your detection reality, not your ideal one. If a suspicious account is discovered after 45 days but authentication events disappear after 30, the investigation begins with a blind spot.

That does not automatically mean every raw event needs to remain in hot storage for a year. You can retain high-value events longer than low-value volume. Examples include successful and failed authentication, privilege changes, configuration changes, audit events, and security alerts. In contrast, highly verbose debug output may only be valuable for a short troubleshooting window.

The key is to preserve enough context to reconstruct a meaningful timeline. An archived authentication event without host, source address, user identity, timestamp, and relevant action may be cheaper to keep but less useful to investigate.

Make retention enforceable, not aspirational

A policy that says “keep audit logs for 12 months” is incomplete if nobody knows where the data lives or whether deletion actually happens.

For each repository or archive, test these four things:

Expiry: Does data disappear when the retention period ends?
Search boundary: Can users query only the intended hot window?
Archive retrieval: Can the team retrieve a specific period when an investigation requires it?
Access control: Can only authorised people access sensitive historical events?

For a query-result archive, add two checks: the query must select the fields needed to reconstruct the timeline, and the worker must have completed the scheduled run. A green connection test alone proves neither. If an investigation needs a period outside the hot window, retrieve one known archived result before declaring the policy operational.

Document the test and repeat it after major platform or policy changes. Retention is a control only when the configured behaviour matches the documented intention.

A sensible next step for a small team

Do not begin by choosing a universal number of days. Pick your three highest-value data classes — for example authentication events, privileged changes, and application errors — and write down their purpose and investigation window.

Then set a modest searchable period that your team can afford and operate. If a class needs to outlive that window, define an archive path before the data ages out. Start with one query for authentication or privileged changes, test its results, activate its schedule, and document why that query earns the longer retention. This creates a real retention strategy without turning a small logging project into a compliance programme.

Get the queries

Retention only helps if you can use the data while it is available. Follow the LogScale Archive installation walkthrough to create, test, and activate the first archive query. Browse mylogscale.com for LogScale queries you can adapt for authentication activity, errors, and other investigation starting points. Updates land there, not in this article.

Sources

5 LogScale queries to audit Active Directory group changes

Fri, 31 Jul 2026 00:00:00 GMT

Changes to membership in privileged Active Directory groups like Domain Admins, Enterprise Admins, and Backup Operators are usually among the first things an attacker targets after gaining access --- a foothold on a low-privilege account is far less useful than a foothold with domain-wide rights, and group membership is often the fastest path from one to the other. At the same time, these changes are some of the easiest things to miss: they're a handful of specific event IDs buried among the much larger volume of everyday Windows Security logs, and unless you're specifically looking for them, they tend to scroll past unnoticed until an audit or an incident forces the question.

The five queries below start broad and get progressively more targeted, moving from "show me everything" to specific patterns worth alerting on to a rollup suitable for reporting. They assume Windows Security events are sent to LogScale using the standard #windows.EventID field --- if you're ingesting through a custom parser or a different field mapping, adjust the field names accordingly before running them.

One thing worth knowing before you start: MemberName is not reliably populated in these events. Depending on the source and the parser, it arrives as a full distinguished name, as a bare account name, or as nothing at all --- while MemberSid is always present. Every table below therefore carries both, so an empty member column is a parser detail rather than a missing result.

1. Track every group membership change, no matter which group it is

Event IDs 4728 and 4729 (global groups), 4732 and 4733 (local groups), and 4756 and 4757 (universal groups) fire whenever a member is added to or removed from a group, regardless of which group it is or who made the change. Combining all six into a single view gives you the full stream of membership activity across the domain, which is the right starting point before you narrow down to anything specific.

This one is deliberately broad. It's not meant to be alerted on directly --- with any reasonably active AD environment, group membership changes happen constantly for mundane reasons --- but it's the query you reach for when you're investigating an incident and need to see everything that touched group membership around a given time window, or when you want a quick sanity check that the event forwarding pipeline is actually capturing these IDs at all.

in(field="#windows.EventID", values=[4732,4728,4756,4729,4733,4757])
| case {
	in(field="#windows.EventID", values=[4732,4728,4756]) | action:= "added";
	in(field="#windows.EventID", values=[4733,4729,4757]) | action:= "removed";
}
| rename(field="windows.EventData.TargetUserName", as="group")
| rename(field="windows.EventData.MemberName", as="member")
| rename(field="windows.EventData.MemberSid", as="member_sid")
| rename(field="windows.EventData.SubjectUserName", as="administrator")
| table([@timestamp, action, group, member, member_sid, administrator])
| sort(@timestamp, order=desc)

Track every Active Directory group membership change on mylogscale.

The result is a plain event table, one row per membership change:

@timestamp	action	group	member	member_sid	administrator
2026-07-29 14:02:11	added	Helpdesk Tier 1	CN=m.klein,OU=Users,DC=corp,DC=local	S-1-5-21-...-4821	a.weber
2026-07-29 03:47:52	added	Domain Admins		S-1-5-21-...-1174	svc_provision

The second row is the kind of thing the rest of this list exists to isolate: an off-hours addition to a privileged group, made by a service account, with no resolvable member name.

2. Changes to Domain Admins specifically

The broad view is useful for investigation, but day-to-day you care much more about a short list of high-privilege groups than about every group in the domain. This query narrows the same event set down to Domain Admins, Enterprise Admins, Schema Admins, DnsAdmins, Group Policy Creator Owners, and a couple of infrastructure-specific groups (VMware Admins, vCenter Administrators), and adds a filter for changes made outside normal business hours.

The after-hours filter does much of the work here. A membership change to Domain Admins at 2 pm on a Tuesday might be entirely legitimate change management; the same change at 3 am is a different story. Adjust the hour range and the group list to match your own change-management windows and your own list of privileged groups --- the ones above are just a starting point.

Set the timezone argument on formatTime() to your own zone, and do not skip it. Without it, LogScale formats the timestamp in UTC, so a 9-to-17 filter silently becomes 10-to-18 or 11-to-19 for anyone in Central Europe, depending on daylight saving time. The first and last hour of the working day then either escape the filter or get flagged as after-hours, which is the sort of error that only surfaces once someone questions an alert. The parseInt() step is there for the same reason: formatTime() returns a string like "03", and converting it explicitly keeps the comparison numeric instead of relying on LogScale to coerce it.

in(field="#windows.EventID", values=[4732,4728,4756,4729,4733,4757])
| case {
in(field="#windows.EventID", values=[4732,4728,4756]) | action:= "added";
in(field="#windows.EventID", values=[4733,4729,4757]) | action:= "removed";
}

// look for these groups
| in(field="windows.EventData.TargetUserName", values=["Domain Admins",
"Enterprise Admins",
"Schema Admins",
"DnsAdmins",
"Group Policy Creator Owners",
"VMware Admins",
"vCenter Administrators"])

// only show outside office hours, in local time
| hour := formatTime(format="%H", field=@timestamp, timezone="Europe/Berlin")
| parseInt(hour)
| hour < 9 or hour > 17

// format the table
| rename(field="windows.EventData.TargetUserName", as="group")
| rename(field="windows.EventData.MemberName", as="member")
| rename(field="windows.EventData.MemberSid", as="member_sid")
| rename(field="windows.EventData.SubjectUserName", as="administrator")
| table([@timestamp, action, group, member, member_sid, administrator])
| sort(@timestamp, order=desc)

Detect after-hours changes to privileged Active Directory groups on mylogscale.

3. Self-service additions: when an account adds itself to a group

Legitimate administrators rarely add their own accounts to a group --- normal workflows involve managing other people's access, not your own. When SubjectUserSid matches MemberSid in the event data, the same account both requested and received the change, which strongly indicates that something other than routine administration is going on.

In practice, this pattern shows up in a handful of scenarios: an attacker who has compromised a low-privilege account using a token or session with just enough rights to escalate itself, a misconfigured automation script running under a personal account instead of a dedicated service account, or --- less alarmingly --- an admin taking a shortcut during a rushed change. None of these should happen regularly, and none of them should happen silently. That's what makes this one of the higher-signal, lower-noise queries in this list: the baseline rate of legitimate self-additions in most environments is close to zero, so any hit deserves a look.

in(field="#windows.EventID", values=[4732,4728,4756])
| test(windows.EventData.MemberSid == windows.EventData.SubjectUserSid)

// format the table
| rename(field="windows.EventData.TargetUserName", as="group")
| rename(field="windows.EventData.MemberName", as="member")
| rename(field="windows.EventData.MemberSid", as="member_sid")
| rename(field="windows.EventData.SubjectUserName", as="administrator")
| table([@timestamp, group, member, member_sid, administrator])
| sort(@timestamp, order=desc)

Detect Active Directory group self-additions on mylogscale.

4. Group membership changes performed by service accounts

Service accounts typically handle automation, not manual management. Provisioning tools, sync jobs, and identity platforms often use a service account to add or remove users from groups as part of their normal tasks. This is expected. What isn't expected is a service account making changes outside its usual automation window. That usually means someone is using the service account's credentials directly, instead of letting automation handle it.

This query filters for SubjectUserName values matching common service-account naming prefixes (svc_, sa_, srv_) --- adjust the regex to match your own naming convention. The prefix filter on its own would surface every legitimate sync job as well, so it carries the same time-of-day filter as query 2, inverted: automation is expected overnight, and a change from the same account in the middle of the working day is the interesting case. If your sync job runs at 2 am, anything from that account at 2 pm is worth a second look. Set the hour range to the inverse of your own automation window, and use the same timezone you set in query 2.

in(field="#windows.EventID", values=[4732,4728,4756,4729,4733,4757])
| windows.EventData.SubjectUserName=/^(svc_|sa_|srv_)/i

// only show during office hours, when automation should be idle
| hour := formatTime(format="%H", field=@timestamp, timezone="Europe/Berlin")
| parseInt(hour)
| hour >= 9 and hour <= 17

// format the table
| rename(field="windows.EventData.TargetUserName", as="group")
| rename(field="windows.EventData.MemberName", as="member")
| rename(field="windows.EventData.MemberSid", as="member_sid")
| rename(field="windows.EventData.SubjectUserName", as="administrator")
| table([@timestamp, group, member, member_sid, administrator])
| sort(@timestamp, order=desc)

Detect Active Directory group changes by service accounts during office hours on mylogscale.

5. Daily rollup for a compliance report

Auditors checking access controls usually don't want a raw event feed. They want a clear number to reference and a trend they can compare over time. This query reduces the events to a daily count for each group, making months of membership changes easy to show on a single slide or spreadsheet row.

It's not a detection query in the same sense as the others; it won't tell you that something suspicious happened, only how much membership activity a group saw on a given day. But it pairs well with the others --- if the daily rollup for Domain Admins suddenly jumps from its usual near-zero baseline, that's a good prompt to go back to query 2 and look at what actually changed. Keep the same timezone you used there, otherwise a late-evening change lands on the following day in the report and the two queries stop agreeing with each other.

in(field="#windows.EventID", values=[4732,4728,4756,4729,4733,4757])
| day := formatTime(format="%Y-%m-%d", field=@timestamp, timezone="Europe/Berlin")
| rename(field="windows.EventData.TargetUserName", as="group")
| groupBy([day, group], function=count())
| sort(day, order=desc)

Daily Active Directory group membership changes on mylogscale.

Which gives you exactly the shape an auditor asks for:

day	group	_count
2026-07-29	Helpdesk Tier 1	14
2026-07-29	Domain Admins	2
2026-07-28	Helpdesk Tier 1	9

Wiring these into alerts

Queries 3 and 4 are the best ones to set up as scheduled searches with a notifier. Both describe behaviour that should almost never happen in a well-managed environment. A self-addition, or a service account making changes during the hours it is supposed to be idle, is rare enough that a false positive only takes a couple of minutes to check, while a real issue is important to catch right away. Queries 1, 2, and 5 work better as saved dashboards you review regularly. Running them as alerts would mostly create noise, since group membership changes and daily activity can shift for many routine reasons.

For the lookback window, 5 to 15 minutes is a good place to start. However, check how your Windows event forwarding works before deciding. If events come in batches instead of a steady stream, a 5-minute window can leave gaps where an event falls outside both the previous and current search. It's usually safer to make the window a bit wider and let the scheduled search handle deduplication, rather than trying to keep the interval as short as possible.

Make sure the notifier payload actually includes member, member_sid, group, and administrator rather than just a generic "match found" message --- whoever receives the alert should be able to act on it without opening LogScale first. And if query 4 turns out to be noisier than expected in your environment (some shops have more interactive service-account usage than they'd like to admit), consider adding a minimum hit count or a short suppression window before the notifier fires, so a single automation blip doesn't trigger an unnecessary alert.

All Active Directory queries

Browse all Active Directory queries on mylogscale for additional detection, investigation, and reporting use cases.