Observability

Splunk Observability, modules in view

Splunk Observability: which modules are included and what do they actually do

“We have Splunk Observability” is a sentence that says surprisingly little. After all, it is not a single product but a collection of modules, and which of these you use completely determines what you see. One organization means by it that infrastructure metrics are coming in. Another has distributed tracing running down to the last microservice. Both mean the same thing.

That makes it difficult to assess what you are missing. Below is a list of the Splunk Observability modules: what each component does, who it is intended for, and more importantly, when you do not need it.

First the basics: metrics, traces, and logs

The backbone of Splunk Observability consists of two modules that are rarely questioned. Infrastructure Monitoring collects metrics from servers, containers, Kubernetes, and cloud services. APM, application performance monitoring, tracks requests across your application landscape using distributed tracing, so that in the event of a slow transaction, you can see in which service the time is being wasted.

Beneath that lies OpenTelemetry, the open standard for collecting and transmitting telemetry. Splunk uses this as an instrumentation layer, which practically means that your data instrumentation is not exclusively tied to Splunk. That is a relevant consideration when choosing a platform that will stay with you for years.

The modules below build upon this. Without this foundation, they offer little.

Real User Monitoring: what your users actually experience

Splunk real user monitoring measures the performance of your application as it occurs in the browser or app of your real users. Page load time, time to display, JavaScript errors, slow API calls—measured on the user's device, not on your server.

That difference is the whole reason RUM exists. Your backend can respond in twenty milliseconds, while a user on a poor mobile connection stares at a blank screen for four seconds. Server-side monitoring doesn't see that; RUM does. And because it is linked to tracing, you can click through from a slow page view to the backend call that caused it.

Where it yields less: for internal applications with a small, homogeneous user group on the same network. In that case, you are primarily measuring your own office network.

Synthetic Monitoring: testing without waiting for users

Splunk synthetic monitoring turns this around. Instead of measuring what users experience, you run automated tests at fixed intervals from various locations: is the site accessible, does the login flow still work, and does the API provide the correct response within the agreed time?

The difference with RUM is not academic. RUM tells you that something went wrong *after* it went wrong, and only on the paths users took. Synthetic tests also run at three in the morning, even on the checkout page that had little traffic today, and even from a country where you normally get few visitors. You catch out on outages with it before your customers call, and you measure availability in a way that you can use in an SLA report.

In practice, you use them together: synthetic for the critical paths and the baseline, RUM for the reality beyond. Using both alone is only half the truth.

Incident Intelligence: from report to someone who takes action

Detection is one thing; ensuring the right person wakes up at three in the morning is another. Splunk Incident Intelligence focuses on the latter: grouping alerts into a single incident, managing on-call rosters and escalation paths, and sending the context needed to get started.

The gain lies in consolidation. In an average environment, a single database failure generates dozens of separate alerts from different modules. If these all land with an engineer individually, it takes unnecessary time to realize that it is a single problem. For teams working with on-call, that is the difference between one alert and a night of downtime.

However, without an established on-call organization, this module is of little value. The tool does not solve any missing process. 

The AI ​​layer: anomaly detection without fixed thresholds

Splunk AI observability is the newest layer on top of the whole, and at the same time the layer about which there is the most confusion. At its core, it concerns two things that are separate from each other.

The first is anomaly detection: instead of setting threshold values ​​itself, the system learns the normal pattern of a metric and flags deviations from it. This is particularly valuable for metrics with a clear daily or weekly rhythm, where a fixed threshold is always set incorrectly—too high during the night batch, too low during the morning peak.

The second is support for investigation: an assistant who helps determine what has changed and where you should look. Useful as an accelerator, but not a replacement for someone who knows the environment.

Tracking business processes, and what happened to Splunk Business Flow

Technical monitoring answers the question of whether systems are running. Process observability answers a different question: is the process running across those systems actually completed? How many orders were started today but not completed, where do requests drop out, which step takes the most time? Technically, everything may appear green, while a quarter of the customers drop out halfway through.

Splunk once had a separate product for that: Splunk Business Flow. If you search for it today, the short answer is that it no longer exists; you can no longer purchase this functionality as a standalone module. If you come across an older presentation or an outdated comparison article that still lists it, you now know why you won't find it in the portfolio.

That doesn't mean the question has disappeared, only the ready-made box. What replaces it is built on the tracing you already have running for APM. A trace tracks a request through your services; if you assign a company identifier, an order number, a request type, or a customer segment to that trace along the way, you can subsequently group based on something the business relies on, rather than a service name. From that moment on, “where do requests drop out” is a question you can answer using your existing data.

The price lies in this sense: *assigning a business attribute to that trace*. That is instrumentation work within the application itself, and therefore a conversation with your development team, not your monitoring vendor. If no one sends that attribute along, you cannot track the process, no matter how many modules you enable. That is precisely where most process observability projects failed, even back when a product with that name still existed.

The visualization layer

Dashboards are the component everyone sees and where most time is spent. Within Observability Cloud, you work with dashboards built on incoming metrics and traces; on the Splunk Enterprise and Splunk Cloud Platform side, you build visualizations with Dashboard Studio.

The pitfall is well-known: dashboards are built for a demo and never updated afterward, while the environment does change. A dashboard that no one looks at anymore is more expensive than no dashboard at all; it gives the impression that something is being monitored. 

What needs to be right before modules deliver value

Enabling modules is not the same as having observability. Three things determine whether it works.

Instrumentation takes precedence over licenses. APM and RUM only yield results once your applications are actually instrumented. For custom applications, this is development work that belongs in a schedule, not a checkbox on a management screen.

More modules mean more notifications. Every module you activate generates alerts. Without agreements on who responds to what, the number of notifications grows faster than the number of resolved issues.

Choose based on a question you cannot answer right now. The most sensible order is not “which module do we not have yet”, but “which incident cost us the most time last month, and which module would have shortened that”.

Do you want to know which elements in your environment actually add value? On our Splunk page , you can read how we view observability, and in the Vialis case study, you can see what that looks like in practice.

Events

Knowledge session: Municipalities in control. From system to service delivery. 
  • Splunk

This knowledge session is for professionals within municipalities who want more control over their digital service delivery. You will discover how chain monitoring with Splunk provides insight into performance, availability, and the interrelationships between systems. Using your own use case, we will explore what this can mean for your municipality in practice.

ntb
ntb
Zoetermeer
SMT Experience Center
During a visit to the SMT Experience Center, you will discover how organizations combine data from IT, OT, and security environments to better understand processes, identify risks earlier, and make better-informed operational decisions.
On request
between 09:00 en 17:00
Louis Braillelaan 10,
2719 EJ Zoetermeer