Prototype Implementation of Non-Volatile Memory Support for RISC-V Keystone Enclave

Handling confidential information has become an increasingly important concern among many areas of society. However, current computing environments have been still vulnerable to various threats, and we should think they are untrusted.
Trusted Execution Environments (TEEs) have attracted attention because they can execute a program in a trusted environment constructed on an untrusted platform.
Particularly, the RISC-V Keystone is one of the interesting TEEs since it is a flexibly customizable and fully open-source platform. On the other hand, as same as other TEEs, it must also delegate I/O processing, such as file accesses, to a host OS, resulting in the expensive overhead. For this problem, we thought utilizing byte-addressable non-volatile memory (NVM) modules is a useful solution to handle persistent data objects for TEEs.
In this paper, we introduce a prototype implementation of NVM support for the Keystone. Additionally, we evaluate it on the Freedom U500 built on a VC707 FPGA dev kit.


https://ken.ieice.org/ken/paper/20210720TC4K/

1. Prototype Implementation of
Non-Volatile Memory Support
for RISC-V Keystone Enclave
Lena Yu, Yu Omori, KeijiKimura
WasedaUniversity
Presented by Lena Yu
SWoPPCPSY2021
2021/7/20SWoPP 20211

2. Introduction
pModern systems are still vulnerable to various threats
nUntrusted environments should not handle confidential data
pSecure computing technologies exist
nHomomorphic Encryption
pComputationally expensive
nTrusted Execution Environment (TEE)
pLow performance overhead
nTEE enables secure computing on variety of platforms
pHowever TEEs have limitations on I/O operations
2021/7/20SWoPP 20212
Utilize byte-addressable non-volatile memory (NVM) modules
to handle persistent data objects for TEEs

3. Goal of this research
2021/7/20SWoPP 20213
Utilize byte-addressable non-volatile memory (NVM) modules
to handle persistent data objects for TEEs
Enable NVMM accesses from an enclave application in Keystone TEE
Make modifications to the Keystone Runtime to add page table management for the NVMM
Use two different free lists to distinguish between DRAM and NVMM free memory
As a tentative design, File Descriptor (fd) is used to distinguish mmap and munmapbetween
DRAM and NVMM
fd= -1 for DRAM
fd= -2 for NVMM
As a first step in achieving this goal

4. What is a TEE?
pTEE generally divides system into separate regions
nUntrusted area
nTrusted area
pContains the TEE, code and data inside is relatively secure compared to the untrusted
pSometimes referred to as an Enclave
pThere exist multiple TEE implementations
nIntel SGX
nARM TrustZone
nRISC-V Keystone
2021/7/20SWoPP 20214

5. Limitations of a TEE
pDelegates I/O operations to the untrusted OS
nEnclave program unable to handle I/O operations
pIncluding file system accesses
nCauses additional overhead
nPoses a security risk
pUntrusted OS can momentarily access sensitive data
2021/7/20SWoPP 20215
Introduce byte-accessible non-volatile memory as a main
memory (NVMM) in TEE to overcome these limitations

6. How might a NVMM overcome the limitations?
pCPU can store persistent data objects
nOnly requires ordinary load and store instructions
nNo need to use complicated runtime software in a TEE for management
pHost kernel does not intervene during I/O operation
nEnsures only the TEE can touch sensitive data
nAs long as the page table for NVMM is prepared
2021/7/20SWoPP 20216

7. Keystone TEE
pOpen-source project for building customizable TEE based on RISC-V
nProvides building blocks for creating a custom TEE
nTEEs can be created with only the required functionality
pReduces Trusted Computing Base (TCB)
→Fully open-source, can be modified to allow NVMM usage
2021/7/20SWoPP 20217

8. Layered Keystone Architecture
pU-Mode
nEnclave Application (Eapp)
nHost Application
pS-Mode
nEnclave Runtime
pModified for the implementation
nHost OS
pM-Mode
nSecurity Monitor (SM)
pTrusted Hardware
nRISC-V Core
nOptional Hardware features
nRoot of Trust
2021/7/20SWoPP 20218
High Privilege
Low Privilege

9. Security Monitor (SM)
pSM responsible for most security guarantees of Keystone TEE
pRISC-V primitive Physical Memory Protection (PMP)
nUsed to enforce memory isolation by SM
nPMP can grant/revoke access permissions
nS-Mode and U-Mode have no permissions by default
nM-Mode have full permissions by default
pPMP entries prioritized by index
nHighest priority PMP[0] used for SM
pSM responsible for managing PMP
nNext highest priority PMP[1] used for enclave
nLowest priority PMP[N-1] used for shared memory
2021/7/20SWoPP 20219

10. Runtime (RT)
pEach enclave has its own RT, acts like the enclaveʼs kernel
nManages enclave memory
pMemory isolated by PMP
pOS cannot access enclave memory
nImplements enclave functionality
pCustomizable enclave functionality provided in the form of plugins
nFree memory
nEdge call interface
2021/7/20SWoPP 202110

11. Free memory (Freemem)
pFreememis reserved physical memory area by enclave
nThis memory does not have to be mapped at time of creation
pAllows RT to perform page table changes
pMmapfrom Eapputilizes Free pages in Freemem
→Required for DRAM and NVMM page table management
2021/7/20SWoPP 202111
Enclave memory before
modification
Enclave memory after
modification

12. Edge Calls and System calls
pInterface that allows function calls to cross in or out of enclaves
pOutbound Call (ocall)
nFunction call that crosses out of the enclave into host
pEappcan invoke a function inside host
nShared memory used for copying between host and enclave
pSystem calls
nSome syscalls(mmap, munmap, brk) are handled in Enclave RT
nSome syscallsforwarded from Enclave to host OS through ocall
2021/7/20SWoPP 202112

13. Enclave Lifecycle
pThree distinct phases in Enclave lifecycle
2021/7/20SWoPP2021 13
InvalidAllocated
Fresh
Running
Stopped
Destroying
CREATE
CREATEEXECUTE
EXIT/STOP
DESTROY
DESTROY
DESTROY
Create
•Host allocates contiguous range of physical memory for enclave
•Enclave page tables, RT, Eappinitialized
•PMP entry set for enclave memory, status propagated through cores
Execute
•SM releases PMP permission to core containing enclave, and enters enclave
•Includes RT boot
•Page table management is done in Running state
Destroy
•Clears enclave memory
•Releases PMP entry
•Returns memory to host
•SM cleans and frees all enclave resources, PMP entries, enclave metadata
Benchmark these to measure initialization, preparation and clearing overhead of NVMM
page table and Freemem

14. Original RT Memory Management
pSv39 addressing mode used
p3-level page table
nRoot Page Table, Kernel L2, L3, DRAM L2, L3
pMemory is prepared during RT boot
nCreation of Page Table entries
pCreate root page table entry, Kernel L2 entries, and Kernel L3 entries
pCreate root page table entry, DRAM L2 entries, and DRAM L3 entries
nFreememinitialization
pAll free pages of DRAM put inside free list
pSimple Page Allocator (SPA) stores pages based on linked list
2021/7/20SWoPP 202114

15. Prototype RT Memory Management
for NVMM support
pModified to allow mapping to the NVMM
pNVMM L2, L3 page tables added
nInitialized and managed similarly to DRAM L2, L3
nCreate root page table entry, NVMM L2 entries, and NVMM L3 entries
pDRAM and NVMM free lists are used
pMemory composition for testing
nEntirety of NVMM region given to NVMM Freemem
nDRAM and NVMM regions are contiguous
2021/7/20SWoPP 202115

16. Mapping from the Eapp
pMmapfrom Eapp
nRT only supports anonymous mapping
nRT checks performed
pChecks continuous virtual address space that fits required size
pChecks DRAM/NVMM free list for number of pages available
vfd= -1 for DRAM
vfd= -2 for NVMM (tentative design for implementation)
pMaps from DRAM/NVMM Freemem
nWalks page table to find page table entry
pMapping is created if non-existent
pMunmapfrom Eapp
nPages are freed and put back into DRAM/NVMM free list
2021/7/20SWoPP 202116

17. Testing the implementation
pFirst tested if modified Keystone works
nConfirmed write/read works for the DRAM and NVMM mmapregions
pBenchmarks
nOcall overhead
nMmap5 pages from DRAM
nMmap5 pages from NVMM
nMunmap5 pages from DRAM
nMunmap5 pages from NVMM
nEnclave creation and destruction
nRT boot time
2021/7/20SWoPP 202117

18. FPGA used for Experimental Evaluation
pTest executed on a Freedom U500 VC707 FPGA Dev Kit
2021/7/20SWoPP 202118

19. Test Eapp
pClock function in the host used to measure Clock Cycles
nEapp uses Ocall to start/stop clock function
2021/7/20SWoPP 202119
Prints string in host
Fd to distinguish between DRAM and NVMM mmap
Shared
Memory
Eapp
ocall_measure_clock(x)
Enclave
Untrusted
Trusted
OS ocall_print_string(xxxx)
Output
Runtime
Clock Cycles is XXX
Enclave said: “mmap into DRAM region successful”
Clock Cycles is XXX
Enclave said: “mmap into NVM region successful”
print_string()
clock_start()
clock_stop()mmap_dram()
mmap_nvm()

20. Results
pThe test app was executed 10 times on the board, and average is taken
2021/7/20SWoPP 202120

21. Discussion
pNVM emulator uses part of DRAM region
nTiming parameters of NVM region set to same as DRAM
nTest overhead due to RT modification
pClock Cycle increase in operations after modifying the RT
nOcall had 1.66% increase
nmmaphad 0.274%
nmunmaphad 0.585%
nRT boot had 0.822% increase
pAdditional page table
pInitialization of NVMM Freememalongside DRAM Freemem
nEnclave creation had 1.04%
nEnclave destruction had 0.0894%
pNo significant increase in Clock Cycles due to RT modifications
2021/7/20SWoPP 202121

22. Limitations
pOcall overhead is large
nMultiple context switching between Enclave and host
nShared memory copy operations
nDifficult to make precise measurements
pNVMM region set to be contiguous with DRAM region
nUnrealistic for them to be contiguous
nKeystoneʼs only support contiguous enclave memory
2021/7/20SWoPP 202122

23. Related Works
pTrusted I/O path in a TEE
nSecure execution environment in TEE not enough
nAllow secure communication between TEE and peripherals
pGeneric trusted path architecture in Intel SGX
nIntel SGX does not support secure I/O and syscall
nAllow trusted paths to generic I/O devices
pSecure Storage models for TrustZone
nSecure storage of private files using encryption
2021/7/20SWoPP 202123

24. Conclusion
pImplemented a prototype NVMM support for RISC-V Keystone
nModified RT to mmapinto DRAM and NVMM regions
nModified RT did not have a significant impact on the operations
pFuture work
nTest implementation on NVMM emulator with appropriate latencies
nDesign persistent data management on the NVMM and OS
nInvestigate overhead associated with multi-core PMP synchronization
2021/7/20SWoPP 202124