Supporting Multiple OS Types on Estimation of System Call Hook Point by Virtual Machine Monitor

Methods to hook system calls issued by a guest operating system (OS) running on a virtual machine using a virtual machine monitor are proposed. The address of the hook point is derived from the guest OS’s source code and established prior to the kernel startup process. Due to changes in system call processing in OS updates and address space layout randomization, the addresses of these hook points cannot always be pre-determined before the kernel startup process. To address this challenge, a method for estimating the system call hook point is proposed in Linux by analyzing the guest OS memory on x86-64 CPUs rather than pre-calculation. Although the method supports Linux, the method can be extended to support other OS types. In this paper, we propose a method to extend the method to support additional OSes. Specifically, we present analysis results and a novel method for estimating hook points on FreeBSD, NetBSD, and OpenBSD. The effectiveness of our proposed method is also demonstrated through evaluation.

© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

This fulltext file will be available in Feb. 2026.