Enhancing mmap scalability by saving TLB shootdowns during page recycling

Abstract

Translation Lookaside Buffers (TLBs) significantly speed up CPU memory accesses by caching recent address translations. The operating system, unaware of the specific core using a mapping, flushes TLB entries on all cores the application is running on when addresses are unmapped to ensure security and consistency. These TLB shootdowns are expensive and bottleneck system performance and scalability. A primary cause of TLB shootdowns is memory-mapped I/O, especially during mmap-munmap cycles and page cache evictions. After deallocations, the same physical pages are often reassigned to the same process, providing an opportunity to minimize TLB shootdowns. We show that avoiding TLB shootdowns for these “recycled pages” becomes possible by modestly extending the mmap function. Therefore, we implement a “fast page recycling” (FPR) option to the mmap system call. FPR-mmaps ensure security by only sending TLB shootdowns, when a page leaves its circles of recycling and when it is assigned to another process. To guarantee consistency when FPR-mmap pointers are accessed, we adapted the virtual memory management to circumvent the ABA-problem. In contrast to previous attempts to overcome the effects of shootdowns, we do not require any hardware modifications and are able to run transparently within the Linux memory system. We evaluated various CPU, memory, and storage configurations, including persistent memory and Optane SSDs and show that FPR achieves significant performance improvements of up to 28% for real-world applications and 92% for micro-benchmarks. In addition, we show that shootdowns are responsible for bottlenecks previously attributed to other parts of the Linux kernel.