Nested Virtualization on Xen Nov. 2009 Qing He Xen Summit Asia 2009 Agenda • Overview • Architecture • Principles and operations • Status Software and Services Group 2 Background • What is nested virtualization? • Virtual machines inside virtual machine — Running a VMM inside a guest • Specifically, hardware-based: e.g. VMX • Why nested virtualization? • Virtualization becoming ubiquitous — Clouds, Xen Client • Use of hardware virtualization in ordinary OS — Windows 7, XP compatibility mode • Facility for investigating VMM behavior Software and Services Group 3 The fundamental idea • Target: virtualization of VMX • Present a virtualized VMX to guest — VMX data structure — VMX instructions — VMX execution flow Software and Services Group 4 VMX revisit • VMX key concepts — Control structure: VMCS — Execution flow, VMM to guest: VMEntry — Execution flow, guest to VMM: VMExit • VMM to fix guest exits • VMCS controls the VM Guest tries to do privileged operations (e.g. mov cr0) VMExit VMEntry guest VMM • Guest running context • When the guest exits • Information exchange controls VMCS VMM to fix the situation Software and Services Group 5 Nested virtualization architecture L2 guest 1 (nested guest) Virtual VMCS Domain 0 L1 guest 1 (nested VMM) L2 guest 2 L2 vVMCS 1 vVMCS 2 L1 sVMCS 11 sVMCS 12 L0 Xen VMCS 1 Shadow VMCS Software and Services Group 6 VMX execution flow native guest VMExit normal virtualization VMM VMEntry VMCS L2 nested virtualization Virtual VMExit L1 Virtual VMEntry L0 VMCS Software and Services Group 7 Execution flow as guest switch • Consider nested guests guest L1 guest point of view also as guests • Virtual VMEntry • L1->L0; guest switch; VMM • L0->L2 (GUEST_RIP in virtual VMCS) • Virtual VMExit • L2->L0; • Virtual VMExit? guest switch; • L0->L1 (HOST_RIP in virtual VMCS) L2 • Other VMExits • Lightweight guest switch L1 • In the same vcpu context L0 ? Software and Services Group 8 Memory virtualization • No special handing for shadow memory • Pure software • However, the performance is bad — Virtual VMExits is much longer than on hardware • Nested EPT will be very helpful • Present EPT to guest • Significantly reduce number of virtual VMExits Software and Services Group 9 Status • POC for simple scenario • single cpu, one nested guest • Some VMX optimizations turned off • No suspend/resume/migration • Nested guest can boot to an early stage • BIOS booting successfully on KVM as nested VMM • Will stabilize it and refine it before send out for review Software and Services Group 10 Questions? Software and Services Group 11 Legal Information INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All dates provided are subject to change without notice. Intel is a trademark of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2007, Intel Corporation. All rights are protected. Software and Services Group 12 Software and Services Group 13