Securing Kubernetes at Scale: A Comprehensive Guide to Using Falco in Production

Introduction to Falco

In the rapidly evolving landscape of containerized applications, ensuring the security and integrity of your Kubernetes clusters is paramount. As organizations migrate more of their infrastructure to the cloud and adopt Kubernetes at scale, the complexity and potential attack vectors increase exponentially. This is where Falco, the cloud-native runtime security project, steps in as a guardian of your Kubernetes environments.

Developed by Sysdig and now a part of the Cloud Native Computing Foundation (CNCF), Falco is an open-source tool designed to enhance the security of containerized deployments by monitoring and detecting anomalous activity within your Kubernetes clusters. It acts as a real-time security orchestrator, leveraging extended Berkeley Packet Filter (eBPF) technology and traditional system call monitoring to provide deep visibility into the behavior of containerized applications.

Falco works by intercepting and inspecting system calls made by the operating system kernel. It uses a powerful rule-based engine to evaluate these calls against a set of predefined security rules that define normal and abnormal application behavior. When a rule is violated, indicating a potential security threat, Falco triggers alerts. These alerts can then be routed to various endpoints for notification, analysis, and response, enabling teams to quickly address potential security incidents.

At its core, Falco is designed to be highly customizable and extensible, allowing organizations to tailor it to their specific security needs. Whether it’s monitoring for unauthorized access, detecting malicious activity, or ensuring compliance with security policies, Falco provides the necessary insights to keep your Kubernetes environments secure.

In this blog post, we will dive deeper into how to properly leverage Falco in a production Kubernetes environment. From initial setup and configuration to defining custom security rules and integrating Falco with your existing security stack, we will guide you through the best practices to maximize your Kubernetes security posture with Falco.

Installing Falco on Kubernetes Using Helm

Deploying Falco on your Kubernetes cluster is streamlined and efficient using Helm, the Kubernetes package manager that simplifies the installation and management of Kubernetes applications. Helm charts are pre-configured Kubernetes resources that package up all the necessary components of an application into a single, deployable unit. The Falco project provides an official Helm chart, making it straightforward to get Falco up and running in your environment.

Prerequisites

Before you begin, ensure that you have the following prerequisites in place:

• A Kubernetes cluster: Falco can be installed on any Kubernetes cluster, whether it’s on-premises, in a public cloud, or a managed Kubernetes service.
• Helm 3 installed: Ensure that you have Helm 3 installed on your local machine or wherever you manage your Kubernetes applications. Helm 3 is a significant improvement over previous versions, removing the need for Tiller and simplifying the architecture.

Step 1: Add the Falco Helm Repository

First, add the Falco Helm repository to your Helm client. This repository contains the official chart for deploying Falco on Kubernetes.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Step 2: Install Falco

Next, use Helm to install Falco into your Kubernetes cluster. You can customize the installation using a values.yaml file, which allows you to specify configuration options that suit your security requirements and cluster setup. For a basic installation, you can skip the custom values.yaml and use the defaults provided by the chart.

helm install falco falcosecurity/falco

If you wish to customize your installation, first download the default values.yaml file:

helm show values falcosecurity/falco > myvalues.yaml

Edit myvalues.yaml as needed, then install Falco using your customized values:

helm install falco falcosecurity/falco -f myvalues.yaml

Step 3: Verify the Installation

After installation, verify that Falco is running correctly in your cluster:

kubectl get pods -n default | grep falco

You should see the Falco pods running. If there are any issues, check the pod logs for errors:

kubectl logs -f $(kubectl get pods -n default | grep falco | awk '{print $1}')

Step 4: Configure Alerting

Falco can alert in various ways (stdout, file, HTTP[S], etc.). By default, Falco logs alerts to standard output, making them visible through the Kubernetes logs. However, for production use, you’ll likely want to integrate Falco with external systems like Slack, Email, or a SIEM system. This can be configured in the falco.yaml file, or by setting the appropriate values in your values.yaml file when using Helm.

Falco also supports forwarding alerts to Falco Sidekick, a companion service that can route alerts to a wide variety of destinations, providing a more flexible and manageable approach to handling Falco alerts.

Customizing Falco’s Security Rules

Falco’s power lies in its flexibility and the ability to tailor its security rules to the specific needs and behaviors of your applications. Customizing Falco rules enables you to fine-tune what constitutes normal and abnormal behavior in your environment, reducing false positives and ensuring that only genuine security threats are alerted.

Falco rules are defined in YAML format and can be found in the /etc/falco/falco_rules.yaml file within the Falco pods, as well as in custom files that can be specified at runtime. Here’s how you can customize these rules to whitelist legitimate activities that might otherwise trigger alerts.

Understanding Falco Rule Structure

A Falco rule is composed of several key components:

• Rule: The name of the rule.
• Description: A brief description of what the rule detects.
• Condition: The condition that triggers the rule. It is written in Falco’s rule expression language and evaluates system call data.
• Output: The message that will be output when the rule is triggered.
• Priority: The severity of the alert (e.g., INFO, NOTICE, WARNING, ERROR, CRITICAL).
• Tags: Arbitrary tags to categorize and filter rules.

Example: Whitelisting Events

Let’s go through some specific examples of whitelisting legitimate events that could otherwise trigger Falco alerts:

1. Prometheus Accessing the Kubernetes API

When Prometheus accesses the Kubernetes API to search for ServiceMonitors, it might trigger rules related to unexpected access to the Kubernetes API server. You can create a rule to whitelist this activity by specifying the user or process that Prometheus uses:

- rule: Allow Prometheus Kubernetes API Access
  desc: Allow Prometheus to access the Kubernetes API for ServiceMonitors
  condition: (k8s.api_server.request and proc.name = "prometheus") and (user.name = "prometheus" or user.name = "system:serviceaccount:monitoring:prometheus")
  output: "Prometheus API access detected (user=%user.name user_loginuid=%user.loginuid k8s.ns=%k8s.ns.name k8s.pod=%k8s.pod.name proc=%proc.name)"
  priority: INFO
  tags: [kubernetes, prometheus]

2. NGINX WAF Spawning a Shell

NGINX Web Application Firewall (WAF) may legitimately spawn a shell as part of its operation, which could trigger shell spawning detection rules. To whitelist this, identify the process or circumstances under which the shell is spawned and create an exclusion rule:

- rule: Allow NGINX WAF Shell Spawn
  desc: Allow NGINX WAF to spawn a shell process
  condition: (proc.name = "sh" or proc.name = "bash") and proc.pname = "nginx"
  output: "Shell spawned by NGINX WAF allowed (proc=%proc.name command=%proc.cmdline)"
  priority: INFO
  tags: [process, nginx, waf]

3. Calico Redirecting Network Traffic to stdout

Calico, as a network policy provider, might redirect network traffic logs to stdout, which could be mistaken for suspicious activity. To whitelist this behavior, create a rule that recognizes it as legitimate:

- rule: Allow Calico Network Traffic Redirection
  desc: Allow Calico to redirect network traffic to stdout
  condition: fd.name = "stdout" and proc.name = "calico"
  output: "Network traffic redirection by Calico allowed (proc=%proc.name command=%proc.cmdline)"
  priority: INFO
  tags: [network, calico]

Tips for Customizing Rules

• Test your rules: Before deploying custom rules in a production environment, test them in a controlled setting to ensure they don’t inadvertently block legitimate activities or flood you with false positives.
• Use Falco’s rule tester: Falco includes a rule testing tool that allows you to verify your custom rules against a set of predefined inputs to ensure they behave as expected.
• Leverage Falco’s community resources: The Falco community has developed a wide range of rules for common applications and scenarios. Before writing your own, check the Falco rules repository and Falco community for existing solutions that may meet your needs.

Customizing Falco’s security rules allows you to adapt its powerful monitoring capabilities to your specific environment, minimizing noise and focusing on detecting real threats. By understanding how to effectively whitelist legitimate activities, you can ensure that Falco becomes an integral and efficient part of your Kubernetes security posture.

Exporting Falco Events to Various Sources with Falcosidekick

Falco, by itself, is a potent tool for monitoring and alerting on suspicious activities within your Kubernetes environments. However, its integration with Falcosidekick—a simple daemon to help forward Falco’s events to multiple destinations—extends its functionality significantly. This setup enables you to send alerts to various outputs, including Slack, Prometheus, and Elasticsearch, enhancing your ability to respond to incidents and integrate with your existing monitoring and alerting infrastructure.

Understanding Falcosidekick

Falcosidekick receives events from Falco and then forwards them to configured destinations. It supports a wide range of outputs, from messaging platforms like Slack to data storage solutions such as Elasticsearch, and even monitoring tools like Prometheus. This flexibility makes it an invaluable tool for creating a centralized observability and incident response platform.

Configuring Falcosidekick with Helm

When installing Falco using Helm, you can also easily deploy Falcosidekick and configure it to forward events to Slack, Prometheus, and Elasticsearch. Below, we’ll cover how to modify the Helm chart values to include Falcosidekick and set up these specific integrations.

Prerequisites

• Ensure you have Helm installed and configured to communicate with your Kubernetes cluster.
• You have the necessary credentials or access configurations for Slack, Prometheus, and Elasticsearch.

Step 1: Add Falcosidekick to the Falco Helm Installation

First, you need to modify your values.yaml file to include Falcosidekick and its configurations for Slack, Prometheus, and Elasticsearch. If you don’t have a values.yaml file yet, you can generate one by running:

helm show values falcosecurity/falco > myvalues.yaml

Next, edit myvalues.yaml to include the Falcosidekick configuration. Below is an example configuration snippet:

falcosidekick:
  enabled: true
  config:
    # Slack output configuration
    slack:
      webhookurl: "https://hooks.slack.com/services/..."
      footer: "Falco Alerts"
    # Prometheus output configuration
    prometheus:
      enabled: true
      listenport: 9090
      customlabels: {}
    # Elasticsearch output configuration
    elasticsearch:
      hostport: "http://elasticsearch:9200"
      index: "falco"
      type: "_doc"
  customOutputs: []
  webUI:
    enabled: false

Step 2: Install or Upgrade Falco with Falcosidekick

With your values.yaml file configured, proceed to install or upgrade Falco along with Falcosidekick:

# To install
helm install falco falcosecurity/falco -f myvalues.yaml

# Or to upgrade an existing installation
helm upgrade falco falcosecurity/falco -f myvalues.yaml

Step 3: Verify the Setup

After deploying Falco with Falcosidekick, ensure that the setup is working by checking the logs of the Falcosidekick pod:

kubectl logs -l app=falcosidekick -n default

Look for log entries indicating successful event forwarding to Slack, Prometheus, and Elasticsearch.

By integrating Falco with Falcosidekick and configuring outputs to Slack, Prometheus, and Elasticsearch, you create a robust security monitoring framework that can alert your team to potential security issues in real-time, store event data for historical analysis, and integrate with your existing observability stack. This setup not only enhances your security posture but also improves your team’s ability to respond to incidents quickly and efficiently.

Remember, the configuration examples provided here are starting points. You should adjust the configurations based on your security requirements, infrastructure setup, and the specific settings of your Slack workspace, Prometheus server, and Elasticsearch cluster.

---

Navigating the complexities of Kubernetes security is crucial yet challenging. Through this guide, we’ve delved into how Falco and Falcosidekick can fortify your Kubernetes clusters, providing real-time insights and alerts on potential security threats. From installation and rule customization to integrating alerts with platforms like Slack, Prometheus, and Elasticsearch, we’ve covered the essentials to boost your security posture.

Yet, the journey to a secure Kubernetes environment doesn’t end here. It demands continuous vigilance and a deep understanding of both Kubernetes and cybersecurity landscapes. This is where Allierce, your dedicated DevOps consulting partner, comes in. Our expertise spans securing containerized environments, streamlining deployments, and crafting bespoke security strategies tailored to your needs.

Whether you’re starting fresh with Kubernetes or aiming to enhance your current setup’s security, Allierce is here to support you. We simplify the complex, ensuring your infrastructure is not only robust but also resilient against emerging threats.

Ready to elevate your Kubernetes security? Contact Allierce today. Let’s secure your infrastructure, together!