Windows 95 System Programming Secret Reading Notes, Experience
Chapter 5 Memory Management
Author: kendiv
Last Update:
Tuesday, January 11, 2005
Although this book is written according to Windows 95, many ideas and details are still very useful for us to study Windows2000 and its subsequent systems. This reading note is not for the entire book, only for the original book 3, 5, 8, 10. These four chapters are as follows:
Chapter III Module, Process, Thread
Chapter 5 Memory Management
Chapter 8 PE and COFF OBJ Format
Chapter 10 writes a Win32 API SPY
At the end of each chapter, I will transplant the original book to Windows 95 to Windows 2000, and some details mentioned in the original article will provide changes in Windows 2000. At the same time, there are many new content.
Related references:
MSDN
"Windows Core Programming" Jeffrey Richter
"32-bit assembly language program design in the Windows environment" Luo Yunbin
"IBM PC assembly language program design (fifth edition)" Peter Abel
Chapter 5 Memory Management (MEMORY Management)
Summary:
In this chapter, we will touch the concepts in Windows 95 memory management. From the Paging Mechanism of the CPU, the address space of each process, and the address space shared to each process. In Win32 API, I will introduce how the Virtualxxx function manages Pages, and will also introduce how the Heapxxx function provides higher-level memory management. The heap function derived from Win16, such as Globalxxx and Localxxx are actually only Heapxxx's light packaging.
In theory, Win32 memory management should be very similar to its three platforms (NT, Windows 9x, Win32S). However, understanding Microsoft after the trajectory of this field, you should be able to expect that Windows 95 memory management and NT and Win32s still have many differences (including good and bad). This chapter I want to discuss WINDOWS 95 version of Win32 memory. Note that many of the concepts described herein also apply to NT and Win32S.
I divide the sub-topic of memory management into two. The first sub-topic is related to the address space of the process, Memory Context, paging behavior (such as "Copy On Write"). The second sub-topic is the memory management function provided by the operating system.
Based on page-based Windows 95 memory management
If you intend to truly understand the memory management architecture of Windows 95. Then you can't avoid talking about the memory paging of Intel 80386 CPU. The memory paging technology is actually earlier than 80386, but because we only use the paging mechanism to use the paging mechanism on the 80386, I only use those terms of 80386. If you understand the pagination earlier, you can skip this section. If you feel very strange to memory pagings, or you need to charge fast, please read it safely.
Memory Paging
The main reason for paging is to provide a method to allow the operating system and the CPU to deceive the deception, so that the program is mistaken for the memory than the memory is really mounted on the machine. When a program reads and writes a unit in the memory, it may (or may not) access one unit on the actual RAM. If the program uses an address, and the address does not correspond to the actual RAM, the CPU will notify the operating system. The operating system then takes the necessary measures to link the address and the actual RAM.
If the memory usage of all executions exceeds the actual memory installation amount, the operating system may need to take a memory from the A program to the B program. Blindly uses the memory in use, it is the beginning of the disaster. So Windows 95 must save the original content in other places. This so, other places, is the hard disk. At any time, the operation system and the data used in the memory used by all applications are not stored in the RAM, which is stored in the hard disk. I said that although it is slightly rough, it is still enough to use it.
The above is said to simulate memory in paging mode and second storage space, generally referred to as "virtual memory". One basic task of Windows 95 Virtual Memory Manager (VMM module in VMM32.vxd) is to provide virtual memory, which is the minimum of the application being disturbed.
Wherever you confuse, the paging will affect the memory location of the CPU. If there is no paging, the program is handed over to the address of the CPU, which will be equivalent to the address on the Memory Bus. For example, in a real mode application, you can easily calculate the actual address according to segment: offset: Multiplying segment to 16 plus OFFSET is. However, when the paging mechanism is started, the address used by the program may be different from the address of the CPU to the memory bus. The paging mechanism introduces two indirectness to all physical addresses. When the program is handed over to the CPU, the CPU uses some bits in this 32-bit address to find the address in physical memory (it will transmit on the memory bus). The CPU is used to perform address transformation, which is controlled by operating system, which can tell the application to use any of the 4GB address space without having physical memory.
The word uses "Page" is because the CPU does not provide an indirect address in BYTE. The conversion of memory addresses is in units of 4KB. If you use the paging mechanism to specify the 0x400000 corresponding to the actual address 0x1000, the actual address 0x1001 will correspond to the program's 0x400001, the actual address 0x1FFF will correspond to the 0x400FFF of the program. However, the next program address (that is, 0x401000) is another 4kb start, so the actual address 0x2000 does not necessarily correspond to the program address 0x401000. Maybe the program address 0x401000 is mapped to the actual address 0x6000, or it is not mapped to any actual RAM. All mapping relationships have operating system paging mechanism control.
In addition to allowing the operating system to provide virtual memory, the CPU supports paging also provides a large elastic, allowing the operating system to arrange a wide variety of objects in memory. The so-called object, I refer to operating system code, application code, program data area, memory map file, etc. The memory layout used by the operating system becomes an address space layout. The Address Space Layout. I will describe the address space of Windows 95 later.
The advantage of paging is that the operating system can spread various operating system objects in the entire address range of the CPU. In terms of Intel 386 CPU family, the address range is 4GB. Theoretically positionable entire physical memory range is called the address space of the CPU (Address Space). Due to the existence of the paging mechanism, there is no need to convert those addresses first, referred to as linear address. As for the conversion, it is true to the address that is truly sent to the memory bus, called the physical address (Physical Address). Remember, in any case, the application and the API are almost all linear addresses, rather than physical memory addresses.
Once you have paging capabilities, the operating system can specify individual sections in the address space, use specific data, and reserve space for adding data. For example, when the application starts, the Windows 95 will retain 1MB address space as the Stack of the application by default. This does not mean that Windows 95 will immediately map 1MB RAM to this 1MB address space, but said that this space is 1MB, in fact, Windows 95 is mapped to the block every time (4KB) to that The address space is in. The paging mechanism allows the operating system to retain the huge memory address range, but do not need to pay the price (I refer to RAM) until RAM is really required.
At any time, in the 4GB address space of the CPU, there are four possible status of each 4KB section (ie, one page):
l State 1: Available, indicating that the page is not reserved for anyone. Anyone can have it through the dispensing action. Attempts to read and write this page will result in the "Page Fault" exception (Exception 0eh). I will describe the "Page Fault" abnormality.
l State 2: Reserved, indicating that the page has been left by someone. However, the actual physical memory has not been mapped to this address, and there is no hard disk space to be reserved for copying its content. Attempt to read this page will trigger the "Page Fault" exception (Exception 0EH). The operating system will give the owner of the page a chance to change the status to commcted and present.
l State 3: Committed and present, indicating that the page has been assigned to someone, and there is a program that has used it to store information. The paging mechanism of the CPU has also mapped 4KB physical memory to the page address. Reading and writing This address is also the physical memory that is read some mapping. This state has a child state called page locked, indicating that this page is Committed, Present, and guarantees that it will never be replaced (swapped out). When a page is in a locked state, never physical memory is mapped to this page until the lock state is released.
l State 4: Committed and not-present, this is very similar to the previous state, and the program has configured this block memory and use it to store information. Different, the operating system has decided that other places need to map to this zone of RAM. Therefore, the CPU has copied this zone to the hard disk, and tag each page of this area as a Not present (Decolification: Not present means the address is not mapped to physical memory)
As a state 1, state 2, if such Page is accessed, a PAGE FAULT will occur. Differences are that when the program reads such an address, the operating system automatically handles the Page Fault abnormal condition and re-maps a 4MB RAM. Then the operating system reads the original data in the hard disk, and then re-execute the instruction before the Page Fault. So the program does not know if there is Page Fault. Such transparency is the basis for virtual memory with hard disk simulation RAM.
Windows 95 provides some Virtualxxx APIs that make you configure a lot of Pages and change its properties. For example: Virtualalloc, VirutAlFree, and more. I will describe these functions later.
Memory paging and selector (paging vs class)
If you have written Windows 3.x applications, you may doubt how paging action is related to Selector. The 16-bit program executed in the Intel CPU protection mode must use the Selector to access a piece of memory in the CPU address space. The code segment of each Win16 program is associated into a Selector; Data Segment is also true, as is the same as any of the memory taken from Global HEAP. As long as you write Win16 programs, it is absolutely impossible to die with Selector. The most basic data in Selector is the address it points to (i.e., "base" address). On the 386 machine, Base is always between 0- (4GB-1). In other words, a Selector has the ability to point to any of the CPU address space. However, the Base address is a linear address rather than the actual address, so the Paging mechanism of the CPU is actually hidden under the selector's dominance. In Windows 3.1 and Windows 95, the 16-bit code did not think of paging and virtual memory, it just thought that there were a lot of memory to be used. The 16-bit Global Heap Manager configures a large block memory from the RING0 operating system and cut it into small pieces so that it can be taken by the Selector by the SELector. These Selector's base addresses do not need to start from the 4KB boundary, and each page in the memory area must be mapped to the actual memory.
As mentioned earlier, the Selector / Segment management system does not contribute to the level of paging. It allows the underlying paging system to provide virtual memory and assume that the memory will be there when it needs it.
If your program is executed in protection mode, you can't avoid Selector. They are absolute essentials when you access memory. Windows 95 requires at least one of the 386 machines, and a key to 386 is that you can make a segment, so that it spans the entire 4GB address space. That is to say, a Selector, Base is 0, and a Limit is 4GB. If you load such a Selector into the CS, in the DS register, you can forget the so-called "section" this, the application can point out any of the addresses according to the 32-bit offset address (Offset). In this case, the 32-bit offset address is a linear address. This mode (using base is 0, selector) is called FLAT MEMORY MODEL, which is different from SMALL, Medium, Compact, and HUGE mode of the past 16-bit. Note that although the FLAT mode makes the win32 program no longer appears, the CPU is also managed in the underlying area. This is especially important if you mix 16-bit code and 32-bit code. Because the 16-bit code cannot hide the ugly neighborhood. J
With an enlarged and open section, the application can contact any location in the CPU address space. You may be strange how the operating system protects its internal data and other places that cannot be messy by the application. For 16 programs, this is not difficult because Selector determines a specific address range, allowing program to contact, and theoretical operating system will never give a selector, allowing its base address to access it should not be accessed The place. However, Windows 3.x and Windows 95 do not prevent you from producing your own Selector and enter the "tourist stop" sway. I will use this flaw later later.
If a Win32 program uses FLAT mode, how can the operating system can constrain other programs and do not allow them to enter certain address areas? In fact, the operating system is no longer relying on the segment size, but the properties of each page are set. For example, the program should not be blindly written to something, so the operating system sets the property of the code area to read-only. Applications can read them, but any write action causes Page Fault. In the same way, the program holds a pointer that has been discarded, as if writes something to a Page that has not been configured. The operating system labeled these unconfigured PAGEs as Not-Present. Attempts to contact these addresses will result in Page Fault. In addition, the operating system can mark the Supervisor property in a range, then they can only be handled by higher permissions (part of the operating system, and vxds). Attempting to process the code processing at a lower permissions, it will cause the Page Fault. As you can see, even if there is no residential area, Windows 95 can also use paging to effectively protect the sensitive area. The only disadvantage of the paging is that the minimum unit of memory allocation is a page (one page size in Windows 95 is 4KB), not a BYTE in the 16-bit area.
Win32 process address space in Windows 95
In Windows 3.x, all programs are executed in the same address space. So any program is easy to read another program used by another program. Worse, the program can also change the memory content of other programs. This gives the Bug's ticket to hell. For example, 16-bit Windows programs in Windows 95 can even get the selector of 16-bit user dgroup and write some garbage. So Windows has to say to you.
Windows 95 gives an independent address space for each process. The so-called "independent address space", I mean that the program can only see its own memory, and the memory used by other processes is unable. More precise, Windows 95 Memory Manager uses the CPU "Based on page" Memory management philosophy to ensure that only the memory owned by the current process will be mapped to the 4GB address space of the CPU. The RAM owned by other processes does not appear in the Page Tables of the current process. The biggest advantage of this is that a problem with a problem can only destroy itself, and will not affect others.
For this feature of Windows, it has been in Unix for many years, and Windows NT is also true. We can only say that Windows 95 This desktop operating system has the most basic nature of the advanced operating system.
Although it is important to separate the memory separation of each process, some memory is still shared by all processes. That is to say, some pages in the linear address space of all processes should be mapped to the same physical memory address. why? The best example is SYSTEM DLLS. Each process requires kernel32.dll. If each process is loaded with a brand new kernel32.dll, it will be an unbearable super-waste. Therefore, other System DLLs such as Kernel32 and USER32 should reside in the shared memory. When the Windows operating system switches Page Tables to reserve those Page Tables mapped to shared memory when performing another processes. I will explain the necessity of shared memory with other examples.
Since Windows 95 segmented different processes, any discussion of how Windows 95 assigns 4GB address space will be inseparable from the so-called Memory Context concept. Memory Context is basically a series of RAM Pages, as well as linear addresses they map. In another sentence, Memory Context is a view of the operating system to a linear address of a process (view).
Every process has its own memory context. When the Windows 95 scheduler pauses a process and letting another process, it must also switch the Memory Context. Since each process has its own Memory Context, sometimes it is called Process Context, sometimes called Address Context. No matter what you call it, remember, the address itself doesn't make sense, unless you indicate which Memory Context is in this address. From the top floor, the memory layout of the WIN32 process of Windows 95 is very simple. In the 4GB address space, the most bottom 2GB (0-7fffffH) is reserved to the application, and the 2GB or more (80000000h-fffffh) remains to the operating system. These two parts have further segmentation. Figure 5-1 shows the details in the 4GB address space.
The first 4MB address space is shared by a process in the system virtual. The first part of 1MB is located, and the memory image containing MS-DOS is loaded when Windows 95 is started. The oil area of 1MB also includes a lower portion of 16 Global HEAP. All 16-bit HEAP linear addresses in Windows 3.1 are not above 2GB. If it is allocated with GMEM_FIXED attribute, then it is often below 1MB. You will see many 16-bit System DLLs in the initial 4MB of the address space because there are many (for example, KRNL386) that require "Fixed and Pagelocked Memory". This is very important, I will also discuss later.
The next area is 4MB to 2GB. This is the address space used by the Win32 process. Each Win32 process maps its own code, data, and resources to this nearly 2GB range. When the switching action of Memory Context occurs, it is actually changed to this range. The RAM Pages mapped to this area cannot be accessed by other processes unless specified. In addition to the code and data of the application, the code and data it uses any DLLS is also placed in this area. In this, you can also find the application's HEAP and STACK (each thread has a Stack).
The Win32 program is loaded to a very low address (4MB) by default. This concept is somewhat uncoordinated unless you really understand the page action. How can I have more than one program is loaded to the same address? The answer is: They share the same linear address, but they are not the same physical memory address. In general, the linear address in the process does not map the physical address of the same value. Due to the relationship between paging operations, each process can think that it has a 4MB-2GB of the entire space. It can't see the memory of other processes, and other processes cannot see it - even with each other with the same linear address. Paging "Magic" makes them actually distinguish.
The above rule "Save independent 4MB-2GB address space for each process" is: Windows 95 believes that "the same physical memory is open to multiple copies of the same program (execute individual, instance) sharing" is safe . Take the program code because the program usually does not modify its code if you perform multiple copies of the same program. Then the Windows 95 saves memory is to map the physical memory of the inner program code to the address space of each process copy.
From the perspective of the most pure operating system, if each 16-bit process has its own address space, it is best to be like 32-bit processes. Unfortunately, a large number of 16-bit programs rely on "can see memory" memory ". In order to retain compatibility for 16-bit programs, Windows 95 is bound to provide greater rights than the Win32 process to them. Windows NT 3.51 allows each Win16 process to run in its own address space, but therefore consumes more memory and leads to greater complexity. Designers of Windows 95 seem to feel that this benefit is not worthy of the cost of it. Since I have seen Windows 95, there is a question that caused my interest: 16 programs how to share their address space with different processes? Conclusion Yes: The memory used by the 16-bit program is always from 4MB or more and 2GB, and there is a so-called global sharing area.
Now let us transfer your eyes to the upper part of the 4GB. From Figure 5-1 you can see it is divided into two parts: 2GB to 3GB to share all processes, and try to use the Ring3 operating system code. At the lowest portion of this area, you will find 16-bit Global HEAP. And above, you see a memory map file. This is quite interesting and it is worth thinking.
If the memory map file is located in the area that can be shared by all processes, it is clear that any process can see it, and even do not need to map action on it (the translation: refers to the work of win32 mapviewoffile), so that such assumptions are correct of. In Windows 95, a memory map file can be accessed by all processes. This is different from Windows NT. Windows NT uses more intriguational paging mode, so that the memory mapping file can only be "mapping this file".
The top of the 2GB-3GB area is hidden in 32-bit System DLLS (kernel32, user32, etc.). To keep up to the memory map file, Ring3 System DLLS is loaded from the 3GB start. Below is the output piece of the Softice / W mod command, which is clearly expressed:
The second column is the load address of the module. Kernel32 is the first 32-bit SYSTEM DLL that is loaded, extremely close to 3GB (address: BFF700000). Next is USER32, located in BFF200000, and as best as Kernel 32. Maybe you will think that these addresses are calculated when they are loaded, not, not the case. Microsoft has a tool (Rebase.exe in Win32 SDK), which can calculate how much address space needs to be required, and then calculate the best load position so that these System DLLs can be connected closely together. When these System DLLs are linked by the compiler (the translation note: Of course not being you), Microsoft then modify the DLLS so that they have the best load location calculated by Rebase.exe. This makes all System DLLs to load the fastest time, and the Windows 95 loader does not need to do "relocation" work.
The last piece of the Windows 95 address space is 3GB-4GB (C0000000H-FFFFFFFH). Finally, this 1GB is used for Ring0 system components (which is VXDS). This can be seen from the output of the Softice / W vxd command.
Sharing memory
All programs in Win16 and all memory owned by all DLLs can be accessed by other programs and DLLs. This is because each Win16 process uses the same area descriptor table (LDT). Therefore, it is very easy to share the memory between the processes: Just let two (above) programs can use the same Selector. The memory that is ready to share is set to the GMEM_SHARE attribute, which is not necessary. Yes, it is not necessary to pay attention to the warning of Microsoft. Now let's compare Windows 95's memory management, which distinguish the address space of each Win32 process unless you specify which block is shared. Unfortunately, specifying sharing is not just as simple as using the GMEM_SHARE property - in fact, using GMEM_SHARE in GlobalAlloc is useless. That is to say that GMEM_SHARE is useless: Win16 does not need it, because everything can be shared; Win32 is ignored at all.
Maybe you have listened to some so-called Win32 authorities said that the only way to share memory in Windows 95 or NT is to use a memory mapped file. That is indeed a way, but not the only way. If you just want to share a small amount of memory between the different executors of the same program, why should you use a cow knife? Although this book puts the focus on readable / writable data between procedures and procedures, don't forget, 4GB address space has half a half to use the system, and they can always be shared by all processes.
From the underlying, memory sharing, only the RAM of a page is mapped to the address space of more than one process. These RAMs can be mapped to the same linear address, or they can be mapped to different linear addresses.
In Windows 95, the memory shared area completed by the memory mapped file is always in different processes. The last PHYS program will expose this fact. However, this assumption in your Win32 program is very dangerous, because Windows NT does not guarantee that the memory map file has the same linear address in each process. Many Win32 programming books cover this topic with memory maps, so I don't plan to say too much here.
The simplest memory sharing method does not have much to mention. In fact, as long as the Data Section of the program is specified in the link, you can easily share this information between each executor (instance) of the same program, or between each user of the DLL. . As long as Win32 DLL's Data Section is specified as Shared, its nature is like Win16 DLL. It's really lucky, Windows 95 gives us such a simple and flexible data sharing method. You can generate multiple Data Sections in the EXE or DLL, put all the data you plan to share in one of the Data Section, and set it to Shared. As for other Data Section still use the default properties (nonshared). The Phys program demonstrates this.
In general, the Microsoft compiler puts all initialized data into one .DATA's section, then leaves it to an attribute other than image_scn_mem_shared. This will cause whenever there is an acting body (INSTANCE), the data segment will be copied, which is specifically used to the actuator. For sharing of memory, you can ask the compiler to generate a new section, the name takes you, but only the first 8 words make sense. E.g:
#pragma data_seg ("sharedat")
After #pragma, you can declare any variable you want to be shared. You should initialize these variables, otherwise they will be placed in another Data Section that stores uninitialized variables.
After the variable declaration is completed, if you want to restore the original Data Section property, just add a row:
#pragma data_seg ()
Finally, you must convey your shared heart to the compiler, you have two ways, the traditional method is to set the section attribute in the DEF file:
Section:
Sharedat Fread Write Shared
Another approach is to specify attributes in the command line parameters of the linker. RWS represents Read, Write, Shared:
LINK / Section: Sharedat, RWS
#pragma Comment (Linker, "/ Section: Sharedat, RWS
I should tell you some warnings such as "user instructions". If you initialize your data as a program code or data symbol, you will become quite interesting when the DLL is loaded with different linear addresses with different processes. Take a look at the variable declarations on this surface (in a shared Data Section):
INT I;
INT * addressof_i = & i;
The problem is not determined before the DLL is loaded. Therefore, the DLL must contain a fixed recording (FixUp Record) to tell the loader to remember the value of addressof_i. When the DLL is loaded, there is no problem. But if another process is then loaded into this DLL, the load address is not the same as the previous process, because addressof_i has been used for the first process (it is shared, isn't it, isn't it?), The loader is not Beachable to modify the value of Addressof_i. Thus, for the second process, the value of AddressOf_i is wrong. Use pointers to solve this problem. I can use a non-shared variable, put a pointer to sharing data. Since this pointer has one for each process, the loader can correct its value to make it a correct value in each process.
In addition to sharing your data, Windows 95 can also share other memory. I have said that 2GB is all shared. However, Windows 95 also slightly opens a portion of the following regions below 2GB. If you execute multiple copies of a program, or use the same DLL in more than one process, each repeated code is a waste. Although Code Section does not have image_scn_mem_shared properties, Windows 95 still loads a program code, then uses the CPU's Page Table, map the program code to other Memory Context.
This sharing code section is well practicing, the only exception is that when the DLL does not have a way to load the same linear address in different processes. Suppose foo.dll is used by two processes, and process A is loaded into the foo.dll and placed at the linear address X, and the process B uses another set of DLLs (which includes foo.dll). When the process B is loaded into the foo.dll, some other DLLs already occupy the address x, so foo.dll has to use other addresses. If your program is in this case, the solution is to reset the load base address of the DLL, and set it to a linear address that never uses other processes.
