Home > Cloud-native applications, Containers, Kubernetes, Microservices > Monitoring a Kubernetes environment using Prometheus or Sysdig

Monitoring a Kubernetes environment using Prometheus or Sysdig


If you have a Kubernetes cluster with pods coming online and going offline at frequent intervals, it may be worth looking at Prometheus.  Prometheus the Titan in Greek mythology stole fire from the Gods and gave it to mankind.  In the same manner Prometheus the open-source monitoring and alerting instrumentation sheds light on your cloud-native applications running as microservices in containers on-premise or in the public cloud.  Prometheus collects metrics over HTTP, it doesn’t focus on logging or events.  It comes in the form of a single binary which is to be installed on your server.

Prometheus use a pull-based mechanism, has its own simple query language (not SQL), uses a text based metrics format and has a time-series database.  So what’s the catch then?  As a user of Prometheus you have to instrument your cloud-native application with Prometheus client libraries which are available for a variety of programming languages including Java, Go, Python, .NET, PHP, Ruby.  If your aim is to visualize these metrics you need another open-source tool like Grafana. The other caveat is that Prometheus may not scale well for very large environments and it does not have long term storage or anomaly detection.

If rolling your own using open-source tools is not your thing, off-the-shelf alternatives like Sysdig are worth exploring.    The name Sysdig brought to my mind an excavator digging alongside humans in containerized homes, especially since Sysdig claims to work at the Linux kernel level and below the containers…

Digging for info

Sysdig aims to be a single platform for monitoring, run-time detection, security and forensics.  You install a Sysdig agent in a container on each host that is to be monitored.  The Sysdig back-end can run in the cloud or on-premise.  Sysdig uses  eBPF (Extended Berkeley Packet Filter) in a passive manner.  Since eBPF runs under containers in ring 0 (kernel mode), Sysdig can capture every read/write and inter-process communication with the goal of identifying user activity within the host.

Traditionally to monitor Tomcat, Redis, Elastic search you needed sidecar containers, which are small containers containing a logging agent and running in the same Kubernetes worker pod as your application.   The sidecar container shares a volume with the application container.  The application container writes logs to the shared volume from where the logging agent in the sidecar container reads them.  Since Sysdig operates at the OS kernel level you benefit from not having to run logging agents in sidecars.

Sysdig gives you KPIs which they refer to as “golden signals” for the Kubernetes cluster.  These KPIs are in the areas of availability, performance, forensics and compliance.  The forensics KPIs become useful for instance if you want to go back in time before a violation occurred and observe who shredded a bash history.  From a compliance perspective Sysdig helps you detect compliance violations.  The policy engine uses Calico to write detailed policy rules at the file, process or container level.   There is a relatively good ecosystem around Sysdig as you can send alerts from Sysdig via email or to Slack, PagerDuty, ServiceNow, Splunk, Syslog, Google security command center or the AWS security hub.

While I’d planned to talk about the VMware response to Kubernetes, monitoring seemed more interesting to me today, hence this article.  Monitoring in a Kubernetes environment is an emerging area and you can expect to see more commercial alternatives to Sysdig in the months to come.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: