Manage virtual memory in Win32
Randy Kath
Microsoft Developer Network Technology Group
Founded on: January 20, 1993
Click here to open or copy the file in the ProcessWalker sample program. The Win32 sample program requires the environment of Microsoft Windows NT.
Summary
In the Microsoft Windows NT operating system, if you have the function of each group function, and each of the functions, there is no enough understanding, then in the Win32 application, which function is decided, or which function is used to manage the memory it is hard. To simplify this problem, this technical article is mainly surrounding Win32 virtual memory management functions: which functions are available, how to use them, and what they affect the operating system. This article will discuss the following topics:
Reserved, submit, and release virtual memory
Change protection on virtual memory pages
Lock virtual memory page
Query a virtual memory of a process
There is a sample program called ProcessWalker in the Microsoft Developer Network CD, which will appear in this technical article. This sample program is useful for exploring the memory address space of a process. It also uses a virtual memory function to implement a list of mutually linked lists.
Overview
This article is one of the three related technical articles, these three articles are jujube "Manage Virtual Memory in WIN32", "Manage Memory-Mapping Files in Win32", and "Manage" in Win32 " (HEAP) Memory "Date They explain how to manage memory in applications where Win32 programming interfaces. In the overview of each article, you must specify the basic memory components in the Win32 programming model, and point out which article should be referred to if you are more interested in special fields.
The first release of the Microsoft Windows operating system describes how to manage dynamic memory based on a single global heap and multiple proprietary local stacks. All applications and systems coexist the entire bureau ( Global HEAP, and each application has its separate local heaps. At the same time, the global and partial memory management functions are also provided to provide extended features for this new memory management system. Recently, Microsoft C's runtime (CRT) library is modified to include the following features, such as the pure CRT functions such as Malloc and Free, manage the heaps in Windows. So, now developers should make a new application programming interface (API) provided by Jujube as part of the Windows 3.1 version, or insist on using portable, typical, and the CRT function familiar with people is Manage memory in the application written in Windows 3.1.
With the increasing content of Win32 API, the choice of opportunities also increases. Win32 provides three additional function groups to manage the memory of the application: Memory-mapped file function, a heap memory function, and virtual memory function. These new functions do not replace existing memory management functions in Windows 3.1; the opposite provides a new feature that makes the developer when writing a memory management section for their Win32 applications.
Figure 1. Win32 API provides different levels of memory management for the diversity of application programming.
In summary, as seen in Figure 1, there are six groups of memory management functions in Win32, all of which are designed separately. So, which function you should use? To answer this question mainly on the following two things: What is your hoped memory management type, and how the function is implemented in the operating system is implemented. In other words, is you a large database application that is building a large database application, so you want to operate a large memory structure, or you are planning some simple dynamic memory structures such as link lists or binary trees. In both cases, you need to figure out which functions provide the functions that are best suited for your intent, and how much resources are taken exactly when using each function. Table 1 classifies the memory management function group in WIN32 and points out the behavior of each related group described in this series of three technical articles. In each technical article, the effects of these functions to the system are emphasized by describing system behavior as responding to using these functions.
Table 1. Memory management functions available in Win32
Memory Sets The affected system resource related technical article Virtual memory function A process of virtual address space system page file system memory Hard disk space "Manage virtual memory in Win32" Memory - Map file function A process of virtual address Space system page file standard File I / O System Memory Hard Disk Space "Manage Memory - Mapping File" in WIN32 - Map File "Heap Memory Function A process of virtual address space system memory process stack resource structure" Manage the memory in Win32 "Global Heap Inscription Function Stack Resources "Manage Piles in Win32 Manage Piles" The Stack Resources Structure of a Process "Manage Piles in Win32 Manage Piles" C Runtime Reference Reference The Resource Structure "Manage Pile in Win32"
Each technical article is discussed by the use of Win32 functions. To learn more about how Windows NT operating systems can manage system memory, see "The Virtual-Memory Manager In Windows NT" on Microsoft Developer Network CD (Technical Articles, Win32, and Windows NT).
WINDOWS NT Memory System Overview
Windows NT uses a page-based virtual memory system that uses a 32-bit linear address. Internally, system management is called all memory in the 4096 byte segment of the page. The physical memory per page is backed up? / Font> For the temporary memory page, the disk file is used for a read-only memory page. At the same time, you can have up to 16 different page files. Codes, resources, and other read-only data are directly backed up through files created.
Windows NT provides independent, 2 GB user address space for each application (process) in the system. For applications, it seems that there is 2 GB of available memory without considering the amount of physical memory actually available. If the memory required by an application requires more memory, Windows NT is to meet this requirements, from this and / or other processes to a non-critical memory to a page file, and release These physical memory pages. As a result, in Windows NT, the global stack no longer exists. Instead, each process has its own 32-bit address space, where all memory is assigned, including code, resources, data, DLL (dynamic link library), and dynamic memory. In fact, the system is still subject to the restrictions of the available hardware resources, but implements the management of applications in the system, for the management of available resources.
Virtual memory in Win32
Windows NT is distinguished between memory and address space. Each process is assigned to a 2 GB user address space, regardless of how much physical memory is used for the actual actual use of the process. Moreover, all processes use the same range of linear 32-bit addresses, range from 0000000016-7fffffff16 without considering the address of available memory. Windows NT is responsible for putting memory page mapping to disk and from disk page back memory in an appropriate time, making each process ensures that it is addressed to the memory. Although two processes are possible to try to access memory on the same virtual address, the Windows NT virtual memory management program is to describe the position of these two memory in different physical locations. And these two addresses are not necessarily consistent with the original virtual address. This is virtual memory. Because of the existence of virtual memory, an application can manage its own address space without having to consider the impact on other processes in the system. The memory management program in Windows NT is responsible for viewing if all applications have enough physical memory for valid operations in any given time. Unlike the Windows 3.1 or earlier versions, the application under the Windows NT operating system does not have to consider this problem with other application sharing systems. Also, even in your own address space, they still share memory with other applications.
One benefit of distinguishing between memory and address space is to provide a very large file to memory for the application. You don't have to read a large file into memory, Windows NT provides support for the application to reserve the address required for the application. Then, when needed, the file portion can be browsed (physically read into memory). This is also possible to do this through the support of virtual memory.
In earlier versions of Windows, the application must first assign memory before an application operates in memory. In Windows NT, each process has been assigned, whether or not the memory is associated with the address in this segment address space is another problem. Win32 Virtual Memory Management Functions provide low-level support for address and memory of separate management processes.
The whole Win32 virtual memory function is:
Virtualalloc and VirtualFree
Virtuallock and VirtualUnlock
VirtualQuery or VirtualQueryex
VirtualProtect or VirtualProtectex
For each function, if there is any corresponding, they together form a group. If you allocate memory, please use VirtualAlloc, once you have assigned it, you must use VirtualFree to release. Similarly, a page that is locked by VirtualLock is not needed. When it is no longer needed, you must use VirtualUnlock to unlock. VirtualQuery and VirtualProtect do not correspond to the corresponding function, but they have the full functionality function (indicated in the EX extension on the function name). This allows them to be used in other processes other than the modulation process, however, when the calling process needs to have appropriate privileges. These functions will be explained in the appropriate context below.
Free, retained, and submitted virtual memory
At any given time, each address in the process can be treated as a free, retained or has been submitted. At the beginning of the process, all addresses are free, meaning they are free space and can be submitted to memory, or retained for future use. Before any free address can be used, it must first be assigned to preserved or submitted. Attempting to access a reserved or submitted address will generate an access conflict exception (Access Violation Exception).
All 2 GB of the 2 GB in a process is either for use but free, or for future use but is preserved, or it is submitted to a specific memory (in use). Figure 2 depicts a hypothetical process that includes free, retained, and submitted addresses. Figure 2. 2 Gb of a process is assigned to free, retained, and submitted memory area.
Reserved address
When the address is reserved in a process, there is no physical memory page being submitted, and maybe more importantly, the space is not retained for backup of the memory in the page file. Moreover, retaining an address range will not guarantee that there will be physical memory available in the future to submit it to these addresses. In fact, it only saves a specified free address address, which will prevent other requests for the segment address when it is necessary to use it. If there is no protection of this type, then routine operations, such as loading a DLL or resource, may occupy the specified address, and the use of it after harm.
The retention address is a fast operation, which is not related to the size of the reserved address range. Whether it retains 1 GB of address range or retains 4K address range, the speed of this function is very fast. This is not surprising, because there is no resource allocation during this operation. This function is just a virtual address descriptor (VAD) tree that enters the process. For more information on VAD, see "The Virtual-Memory Manager in Windows NT" of Developer Network CD (Technical Articles, Win32 and Windows NT Articles)
To keep a range of addresses, you need to refer to the following code to call the VirtualAlloc function:
/ * Reserved 10 MB of address space * /
LPBase = Virtualaloc (NULL,
10485760,
MEM_RESERVE,
Page_noaccess);
As shown here, the first parameter, LPADDress, use a null value, and guide the function to reserve the address range in one of the most convenient positions. In addition, it is also possible that a specified address has been passed, indicating an accurate initial address for the range to be retained. Regardless of which of the two methods, the return value of this function indicates the start position of the preserved address range, unless the function cannot complete the request. In this way, the return value of the VirtualAlloc function will be an error status value.
The second parameter indicates the address range that the function should be assigned. The size of this value can be any value from one page to 2 GB, but the VirtualAlloc is actually limited to a smaller range. The minimum value that can be retained is 64k, and the maximum value that can be retained is the largest continuous free address space in the process. The requested reserved address, resulting in a 64K address range. Conversely, requesting 2 GB will fail because there is so much available address space in any given time. (Remember, the action loaded with an application also uses some of the initial 2 GB address space.)
Note that Windows NT generates a protection facility (Safeguard) in the address space of each process. Top 65,536 bytes and low ends 65,536 bytes of each process are preserved permanently. These address space portions are preserved as a trap lost pointer (TRAY POINTERS) jujube diagram to address memory in the range of 0000000016-0000FFF16 or 7FFF000016-7FFFFFF16. It is not coincidental. In this range, it is only necessary to ignore the low four digits in these addresses (the rightmost two bytes) can easily detect the pointer. In terms of fundamentals, if the high four digits are 000016 or 7FFF16, then this pointer is invalid; all other values represent a valid address. The last two parameters in the VirtualAlalloc function are used to determine how to assign addresses and protect them with them. The address can be assigned to a MEM_COMMIT or MEM_RESERVE type. Page_readonly, Page_ReadWrite and Page_noAccess are three protection that can be applied to virtual memory. Regardless of why value is passed to the function, the reserved address is always page_noaccess, which is the system forced default value. The files that have been submitted can be read-only, or they can be readily written, or if they cannot be accessed.
Submitted memory
To use the reserved address, memory must first be submitted to this address. Submit memory to the address and reserved memory similar to Jujube call Virtualalloc, and set the dwallocation parameter at the time of call, equal to MEM_COMMIT. At this moment, the resource is submitted to the address. Every time, memory can be submitted on a page size. The maximum memory value that can be submitted only depends only on the maximum range of freedom or the retention address (but the two cannot be combined), and there is no need to consider the size of the system's available physical memory.
When the memory is submitted, the memory physical page is assigned, and the space is retained in a page file. That is, the submitted memory page always exists in the form of a physical memory page or a page file on the disk already being paged. It is also possible when submitting a bulk memory, in the initial phase, some or all of the memory is not residing in physical memory. Some memory pages have been left in the page file until it is accessed. In the system, once the memory page is submitted, the virtual memory manager treats them like all other memory pages.
In the Windows NT virtual memory system, the page table (Page Tables) is used to access the physical memory page. Each page table itself is also a memory page, like the page already submitted. Even, when submitting memory, you must also assign additional pages to the page table. So, the request for submitting a page may need to assign a page for the page table, assign a page for the requested page, and two-page space is required to back up each page in the page file. Therefore, Virtualalloc has a large time to complete a memory submission request, depending on the status of the system, and the spatial size requested.
The following example demonstrates how to submit the specified page of the preserved address in the previous example to a memory page.
/ * Submit memory for the address of page 3. * /
LPPAGE3 = Virtualalloc (lpbase (2 * 4096),
4096,
MEM_COMMIT,
Page_readwrite;
Note that for LPADDRESS, no null is specified, but specifies a specific address to accurately indicate which page of the preserved address will turn into a page submitted to memory. Moreover, the memory page is given to Page_ReadWrite protection, not like Page_noAccess in the previous example. The return address of this function is the virtual address of the first page has submitted the address.
Release virtual memory
Once the address is allocated in the form of preserved or submitted, VirtualFree is the only method that can release their method, and return them to the free address. VirtualFree can also be used to submit submitted pages, while returning these addresses to retention status. When the address is submitted, all physical memory and page file space associated with this address are released. The following example demonstrates how to release the memory page submitted in the previous example. / * Submit memory to the address of page 3. * /
Virtualfree (lpbase (2 * 4096),
4096,
MEM_DECOMMIT,
Page_noaccess);
Only the submitted address can be released. This is very important when you need to submit a wide range of addresses. For example, suppose you have some range of addresses, the subsets of multiple addresses have been submitted, while others are retained. To reserve all the addresses, the only way is to release the subsequent address subset of addressed addresses. If you try to release the submission of the entire range of addresses, it will fail because the reserved address cannot be submitted.
In contrast, the same range of addresses in one operation can be released. Because when the address is released, the status of an address is not important. The following example demonstrates the address of the 10 MB range that is retained in the first example.
/ * Release the entire 10 MB of the address. * /
Virtualfree (lpbase,
10485760,
MEM_RELEASE,
Page_noaccess);
Change the protection of virtual memory pages
Win32 provides a VirtualProtect function as a method for reproducing the files that have been submitted to the memory. For example, an application can press Page_ReadWrite to submit an address of a page and immediately fill in the data. Then, the protection of this page will be changed to Page_Readonly, which can effectively protect the data to be rewritten by any thread in the process. The following example uses the VirtualProtect function to make a page that cannot be accessed.
/ * Change page protection into readable / write. * /
VirtualProtect (LPSTACK 4096,
4096,
Page_readwrite,
LPDWOLDPROT);
The following environment can be regarded as an environment using this function. An application for buffering data receives a set of size variable data streams, depending on the specific hardware configuration and other software applications to compete with CPU time, data streams may be a certain time (at time "exceeding the process. In order to prevent this phenomenon, the application has designed a memory system that can be submitted to some memory pages at the beginning. Then, the application uses Page_noAccess protection to protect the top page of the memory so that any requests that you want to access the memory will generate an exception. Applications also use an exception handler in the outer code of the code to handle access conflicts.
When an access conflict occurs, the application can determine if the buffer has reached its limit. The application responds by changing page protection to Page_ReadWrite, allowing the buffer to receive any additional data and continue uninterrupted execution. At the same time, the application loads another thread to slow the data stream until the buffer returns to an ideal operating range. When the situation is restored to normal, the top page returns to page_noaccess, and the additional thread is over. This situation describes how to use the page protection and exception handler in Win32 to provide unique memory management opportunities.
Lock virtual memory page
The process in Windows NT has a minimum page called Working Set, which is available smoothly, and must be provided in memory during runtime. Windows NT assigns a default number of pages for a process when starting, and gradually adjusts the number, making the performance of all activated processes in the system reaches the optimal balance. When a process is running (actually, it is running when a process is running), Windows NT is "working hard" to ensure that the working group page always resides in physical memory. The process in Windows NT is authorized to use the VirtualLock and VirtualUnlock functions to suit the behavior of the system. Fundamentally, a process can establish a specific page to lock it into the working group. However, this does not give the work group of the process in the range of freedom. It does not affect the number of pages of the working group that makes it, (the system adjusts the working group for each process), and it cannot control when the working group is in memory and when not in memory. The number of pages that are locked each time a process is not more than 32. If an application will lock the memory page of the submitted in the workgroup, it may be greater than the disadvantage, because this may force other key pages in the process to be replaced. If so, the page may map the page to the disk, and the page failure will occur at any time being accessed. Thus, the process will spend a lot of CPU time but only map the key page to memory and map memory.
Keep in mind that locked a page in Win32 does not mean that the page will not be mapped to disk. Instead, it means that when the process is running, the locked memory page will be in a physical memory. Not just possible, but it is very likely that when a process is ideal, the workgroup of the entire page will be mapped to disk. When the working situation of the process is deteriorated, the Working Group page will be immediately mapped back to the memory, including the virtuallock page.
The following example is locked in the process to memory when the process is running.
/ * Lock the critical address (critical addresses) to memory. * /
Virtuallock (LPCriticaldata, 1024);
Note that the address range that is locked to the memory in this example is less than one page. It is not necessary to have a single page in memory. The last result is a memory that contains the entire page of the data in the address is locked to memory, not just the data of the address indicated by the address being locked into the memory. If the data spans the boundary of the page, the two pages will be locked.
Query a virtual memory of a process
Given the address space of a process is 2 GB, and if there is no ability to query the address information, you want to manage the full range of the address will be difficult. Because the address itself represents an independent memory, for them, these memory may be submitted, or not submitted, inquiring them, just reading the data structure that holds their status. In Windows NT, this structure is a previously mentioned virtual address descriptor tree. Win32 provides the ability to "traverse VAD structures" in the VirtualQuery and VirtualQueryex functions. Similarly, the EX suffix indicates which function can be called from one process to query another process date, if the calling process has sufficient security privileges to execute the function. The following example is separated from the ProcessWalker sample program:
/ * The next memory area in the sub-process. * /
VirtualQueryex (HchildProcess,
LPMEM,
LPLIST,
SIZEOF (Memory_basic_Information);
The main function of the ProcessWalker application is the address space that traverses a process, identifies each unique address area, and displays specific status information about each zone. It is done by each of the regions from the bottom to the upper portion each time. LPMEM is used to indicate the location of each area. It is set to 0 when it is returned from the query of each new area, increment by the query area size. The process has been repeated until the LPMEM reaches the highest system retention area. LPLIST is a pointer to the MEMORY_BASIC_INFORMATION structure, which is filled with the VirtualQueryex function. When the function returns, the structure represents information about the query area. This structure contains the following members:
Typedef struct _memory_basic_information {/ * mbi * /
PVOID BaseAddress; / * Basic Address * /
PVOID AllocationBase; / * Assign Basic Address * /
DWORD AllocationProtect; / * Initial Access Protection * /
DWord RegionSize; / * The byte size of the area * /
DWORD State; / * Submitted, retained, free * /
DWORD protect; / * Current Access Protection * /
DWORD TYPE; / * Page Type * /
Memory_basic_information;
The VirtualQuery function returns this status information for any connected address area. The function determines the size of the underlayer boundary and the size of the area, and the exact state of the address in the area. It is used to determine that the address of the area can be any address in this area. Therefore, if you want to determine how many stack spaces have been submitted at any given time, follow these steps:
Get the thread environment for the thread in the problem. Call the VirtualQuery function, provide the stack pointer address in the thread environment information as the LPMEM parameter in the function.
The query returns the size of the submitted memory and returns the address of the basic stack in the MEMORY_BASIC_INFORMATION structure in the form of regionsize and baseaddres, respectively.
The memory area is defined by VirtualQueryRegions, which is a continuous address range, its protection, type, and basic assignments are the same. Types and protection values have been described in front of this technical article. Basic assignments are the LPADDRESS parameter value, which is used when the entire memory area is allocated by the Virtualalloc function for the first time. In the Memory_basic_information structure, it is represented as the ALLOCATIONBASE field.
When the free address is reserved or submitted, the basic assignment thereof is determined. From any point of view, the memory area is not static. Once a single page in the reserved address area is submitted, the area is divided into one or more reserved areas and one submitted area. This will continue when the page memory changes. Similarly, when a plurality of Page_ReadWrite's submitted pages are changed to Page_ReadOnly protection, the area is also divided into multiple smaller areas.
to sum up
The virtual memory management function in Win32 provides the ability to manage virtual memory directly in Windows NT. The 2 Gb user address space of each process is divided into reserved, submitted or free virtual address memory area. A zone is defined as a consecutive address range, the basic assignments of each address are the same, and one or more addresses are one or more addresses in each zone, or the protection and page lock flag status bits can also be carried.
The virtual memory management function provides the application to convert the virtual address spatial page status. An application can change the memory type from the submitted change to reserved, or change the protection mode from page_readwrite to Page_Readonly to prevent access to a certain address space. An application can lock a page memory to the workgroup so that a process is at least a key memory page. The virtual memory function is considered a low-level function that it means that their speed is relatively fast, but lacks a lot of advanced functions. (Source: http://www.microsoft.com/china/msdn/library/archives/technic/develop/win32ram.asp)
