How to reduce alert noise in LogScale

The short answer

I recommend separating early information from alerts that deserve additional attention: use email for the first notification and a separate Teams alert with a longer validation period. Define both conditions around the service, because neither one failed check nor a fixed waiting period means the same thing in every environment. Keep Teams selective so that it does not become a second inbox full of messages people stop reading.

My own email inbox is the starting point for this recommendation. Repeated notifications and brief server interruptions produce messages that eventually receive less attention. A service failure that matters can then disappear among notifications that did not need an immediate response.

The question is how to decide which condition deserves a message, when to send it, and which channel should carry it.

The factors

1. What the check actually confirms

A failed check provides information, but its meaning depends on what it tests. A server briefly becoming unreachable and a service remaining unavailable are different conditions. Treating both as equally urgent makes the messages harder to distinguish.

Validation must fit the environment. For one service, that might mean requiring a failure to persist for a defined period. For another, the check itself may already combine enough evidence to justify an immediate notification.

Before choosing a delay, describe what the result proves. Does it show one unsuccessful connection attempt, several failed checks, or a service that users cannot access? That distinction is the basis for the notification condition.

2. How much delay the service can tolerate

A longer validation period gives a brief interruption time to clear before generating an additional message. It also delays that message when the failure continues.

The responsible person must decide whether that trade-off is acceptable. For example, a temporary interruption to a service needed later in the day may allow more validation time than an unavailable service required for work happening now. These are decision examples, not default severity assignments.

I would choose the acceptable delay for each service before choosing the alert settings. A convenient value in the interface should not determine how long a significant failure goes without attention.

3. What each channel means

In this strategy, email carries the earlier information. Teams carries the additional notification after the stricter condition is met.

That distinction only helps if it remains consistent. Sending every email notification to Teams immediately would give recipients the same messages in two places. It would not address the original problem.

The message still provides information rather than taking action on the recipient’s behalf. It should make the affected service and the detected condition clear enough for the recipient to decide what to do.

4. Whether another message adds information

Repeated notifications about the same unchanged failure deserve their own decision. A reminder may be useful, but receiving the same message again does not necessarily tell the recipient anything new.

LogScale supports throttling to limit repeated actions. Its documentation distinguishes the query time window from the throttle period; these settings serve different purposes. See the LogScale trigger documentation.

Use validation to decide whether a condition deserves a notification. Use repetition controls to decide when another notification about that condition is useful.

Concrete recommendation by situation

For a service where brief interruptions are acceptable, start with two independent alerts. Give the email alert the earlier notification condition. Give the Teams alert a condition that requires evidence over a longer period.

For illustration, a brief interruption might meet the email condition and recover before it meets the Teams condition. A continuing failure would meet both conditions, so the recipient would receive the additional Teams message. This describes the intended behavior, not a tested configuration or a universal timing recommendation.

For a service where waiting is unacceptable, do not add a delay merely to preserve this two-stage pattern. Use the validation that the service requires, which may already be built into the check, and choose the channel according to its importance.

For information that only needs attention during defined working hours, consider restricting notifications to those times. Keep that decision separate from failure validation: business hours determine when information is relevant to the recipient, while validation determines whether the failure condition is sufficiently established.

Get the query: Restrict Alerts to Weekdays and Business Hours.

Apply such a restriction only where the service can tolerate it. An outage that requires attention outside business hours needs a notification path that remains active then.

Edge cases and objections

A longer search window does not prove a continuing outage

The Teams alert is independent. It is not simply a delayed copy of the first email.

LogScale triggers use query results to initiate actions, as described in the official trigger overview. The second query must therefore express the stronger condition you intend to detect.

If it only finds an earlier failure somewhere in a larger window, it may still find that event after recovery. The condition must distinguish the evidence for a continuing problem from an old failure. The right way to do that depends on the available checks and events.

Too much suppression can hide a separate failure

Reducing repetition should not prevent a different service failure from becoming visible. Consider two services failing close together: a rule intended to silence reminders for the first should not accidentally silence the first notification for the second.

Before relying on the setup, check three scenarios in your own environment: a brief interruption, a continuing failure, and a separate failure while the first remains active. Confirm which messages arrive through each channel.

Start with one noisy alert. Define what it proves, when a second message becomes useful, and what should reach Teams. Then verify those three scenarios before applying the same strategy more broadly.