[Original] Understanding I / O Completion Port Nonocast (original)
Welcome to this IOCP tutorial. I will give the IOCP definition and give its implementation method, and finally analyze an echo program to open the IOCP puzzle cloud, remove your heart on IOCP. OK, but I can't guarantee that you understand everything IOCP, but I will do my best. The following is the related art I will mention in this article: I / O port synchronization / asynchronous blocked / non-blocking server / client multi-threaded program Design Winsock API 2.0
Prior to this, I have developed a project, one of which needs to support the code, and only use SELECT, CONNECT, Accept, Listen, Send and Recv, plus a few #ifdef Package is used to deal with the incompatibility between Winsock and BSD socket [socket], a network subsystem is written for only a few hours, and I still make me a good aftertaste. It didn't touch it for a long time. A few days ago, we plan to make a online game, I took the initiative to bear this piece, think about this is not a small CASE, I am stealing. Online games, online games provide fun and homologous game experiences for hundreds of players, they fight each other or join the team to overcome the common enemies. I am confident that I am ready to write my network, so I found that the past block synchronous mode mode can't get a huge number of multi-player [MMP] architectures, directly denied it. So, there is an IOCP. If you can easily lose IOCP, there will be this tutorial. Please follow me to enter the topic.
What is IOCP first let's take a look at the I / O completion port for IOCP may be the most complex kernel object provided by Win32. [Advanced Windows 3rd] Jeffrey Richter This is the best way to implement high-capacity web servers. [Windows Sockets 2.0: Write Scalable Winsock Apps Using Completion Ports] Microsoft Corporation completed the port model to provide the best scalability. This model is very suitable to handle hundreds and even thousands of sockets. [Windows Network Programming 2nd] Anthony Jones & Jim Ohlund I / O Completion Ports is particularly important because they are the only technique for high load servers [must maintain many connection lines at the same time. The Completion Ports uses some threads to help balance the load caused by the I / O request. Such architectures are particularly suitable for "Scalable" servers generated in the SMP system. [Win32 multi-threaded program] Jim Beveridge & Robert Wiener
It seems that we have a complete reason to believe that IOCP is the first choice for large network architecture. What is the IOCP? Microsoft introduced this concept of IOCP in Winsock2. IOCP full name I / O Completion port, Chinese translated into I / O completion port. IOCP is an Asynchronous I / O API that delivers the I / O event to the application efficiently. Unlike the use of select () or other asynchronous methods, a socket [socket] is associated with a completion port, and then proceeds to normal Winsock operations. However, when an event occurs, this completion port will be added to a queue by the operating system. The application can then query the core layer to get this complete port. Here I am going to add some of the above concepts. Before explaining [Finish], I want to simply let the synchronization and asynchronous concepts, logically let another after doing one after another. It is to synchronize, while doing two or two things together is asynchronous. You can also take a single thread and multi-threaded metaphor. But we must separate synchronous and blockage, asynchronous, and non-blocking distal distalbsor, so-called clogging functions such as Accept (...), after calling this function, at this time thread will hang until the operating system is notified, "Hey Brothers Some people come in, "that the hangs will continue to work, which is also in line with the" producer-consumer "model. Blocking and synchronizing seems to have two points, but it is a completely different concept. Everyone knows that I / O equipment is a relatively slow device. Regardless of the printer, modem, even hard drives, compared with the CPU, it is uncomfortable, and the completion of I / O is a unwise thing. Sometimes the flow rate of data is very amazing, move the data from your file server with Ethernet speed, speed may be up to one million bytes per second, if you try to read 100KB from the file server, in the user's vision It is almost instantaneous completion, but you have to know that your thread executes this command, has been wasted 10 1 million CPU cycles. Therefore, we generally use another thread to perform I / O. Overlap IO [Overlapped I / O] is a technology of Win32, you can ask the operating system to transfer data for you and inform you when transferred. This is also the meaning of [completion]. This technology allows your program to continue to handle transactions during I / O. In fact, the inside of the operating system is threaded to complete overlapped I / O. You can get all the interests of threads without having to pay for the price of pain. Completing the so-called [port] in the port is not the port mentioned in TCP / IP, it can be said that it is completely no relationship. I didn't want to pass an I / O device [I / O DEVICE] and port [IOCP port]. It is estimated that this port is also confused. IOCP is only used to read and write operations, and it is similar to file I / O. Since it is a read and write device, we can ask for it just to deal with read and write efficient. The third part of the article You will easily discover the true intention of the IOCP design.
What is the relationship between IOCP and the network? int main () {WSAStartup (MAKEWORD (2, 2), & wsaData); ListeningSocket = socket (AF_INET, SOCK_STREAM, 0); bind (ListeningSocket, (SOCKADDR *) & ServerAddr, sizeof (ServerAddr)); listen (ListeningSocket, 5) ; int nlistenAddrLen = sizeof (clientAddr); while (TRUE) {NewConnection = accept (ListeningSocket, (SOCKADDR *) & clientAddr, & nlistenAddrLen); HANDLE hThread = CreateThread (NULL, 0, ThreadFunc, (void *) NewConnection, 0, & dwTreadId) CloseHandle (hthread);} return 0;} I believe that as long as friends who write the network should be familiar with such structures. The rear thread is hanged, waiting for a customer to issue a request, and then create a new thread to handle the request. When the new thread handles the customer request, the initial thread loops goes back to wait for another customer request. The thread processing that handles the customer request is ended. In the above concurrent model, a thread is created for each customer request. It is advantageous that the thread waiting for the request only needs to do very little work. Most time, the thread is in sleep [because the RECV is in a plug state]. But when the concurrent model is applied to the server side [Windows NT], the Windows NT team notes that the performance of these applications is unpredictable. Special, handling a lot of customer requests means that many threads are running concurrently in the system. Because all these threads are running [not hanging and waiting for something], Microsoft realizes that the NT core takes too much time to convert the context of running thread [context], threads have not got a lot of CPU time. Do their work. Everyone may also feel the bottleneck of parallel models in it created a new thread for each customer request. Creating a thread is smaller than the creation process overhead, but it is far from not overhead. We may wish to imagine: If you have a good N thread in advance, let them in the HOLD [Clog], then you can deliver all users' requests to a message queue. Then the N thread takes the message from the message queue one by one. You can avoid the shutdown of each user request. Not only reduces the resources of threads, but also improves the utilization rate of threads. Theoretical is very good, what will I do if I have to think about it? How will Microsoft don't think about it ?! The solution to this problem is a kernel object called I / O completed port, he first Windows NT3.5 is introduced. In fact, our idea should be almost the design mechanism of IOCP. In fact, it is not a message queue! You said this and [port] What are you connected? My understanding is that IOCP is an interface that is communicated with the application and operating system. As for the specific design of IOCP, I am also very difficult to say, after all, I haven't seen the code implemented, but you can simulate it, but you may ..., if you want to understand the IOCP, Jeffrey Ritchter Advanced Windows 3rd in which 13 Chapters and 14th have a lot of valuable content, you can take a look at how the system is all.
Implementation method Microsoft provides corresponding API functions for IOCP, mainly two, let's take a look at: Handle FileHandle, // Handle to File Handle ExistingCompletionPort, // Handle To I / O Completion Port Ulong_ptr CompletionKey , // Completion Key DWORD NUMBEROFCONCURRENTTHREADS / / NUMBER OF Threads to Execute ConcURRENTLY; first pay attention to this function to be used for two distinct purposes before discussing each parameter: 1. Used to create a completed port object 2. Associate a handle [Handle] and complete port together to create a completion of a port, we only need to fill in the parameter of NumberOfConcurrentThreads. It tells the system a maximum number of threads that are allowed to operate simultaneously on the port. By default, the number of wires is the same, but the number of CPUs is the same, but the experience gives us a formula: thread number = CPU number * 2 2 To make the completed port is useful, you must associate it the same or more devices. This is also done by calling Createiocompletionport. You have to pass an existing completion of the port to this function. We have to handle the online event, that is, the customer's socket is passed as a Handle. And a completed button [a 32-bit value for you, it is a pointer, the operating system doesn't care what you are. When you associate a device to the port, the system adds a message to the device list of the completion port. Another API is BOOL GetQueuedCompletionStatus (HANDLE CompletionPort, // handle to completion port LPDWORD lpNumberOfBytes, // bytes transferred PULONG_PTR lpCompletionKey, // file completion key LPOVERLAPPED * lpOverlapped, // buffer DWORD dwMilliseconds // optional timeout value); first The parameter indicates which completion port is to be monitored. Many service applications are just using an I / O completion port, and all I / O requests are completed, and they will be sent to this port. Simply put, getQueuedCompletionStatus causes the calling thread to hang until one of the specified port I / O completes a queue or until the timeout. The third data structure associated with the I / O completion port is to make the thread get information in completing the I / O item: the number of bytes transmitted, the completion keys, and the address of the overlapped structure. This information is returned to thread by passing to the LPDWNumberofByTestransferred, LPDWCompletionKey and LPOVERLAPPED parameters of GetQueuedCompletionsatus.
According to the things that have already been mentioned so far, first build a frame. Let you explain how to develop an Echo server using the completed port. Based on: 1. Initialize Winsock 2. Create a completion port 3. Create a certain amount of threads depending on the server thread 4. Prepare a socket for bind then Listen 5. Enter Cyclic Accept Waiting for the customer request 6. Create a data structure Socket and other related information 7. Connect the Socket associated with the completion port 8. After delivery, you will continue to repeat 5 to 8 processes, we will use the specific code to show the details of the details. WOW, the code of the program is posted here, it is really a scenery, can't Ctrl V can't f7. If you need the source code, you can send it to me o_nono@163.net to this article, there should be a paragraph, I brought it You have made a whirlwind tour, explore the so-called completion port. Many details cannot be detained in detail due to the relationship between the space. But I hope this article will bring you more thinking. If you have any questions, you can send it to o_nono@163.net. It can also be discussed in this discussion http://expert.9cbs.net/expert/topic/2659/2659726.xml?temp=.3871271
[[[[Above is the Windows network programming English original version, "Chinese is what I collected. 】]]] The "Complete Port" model is the most complex-species I / O model so far. however. If you need to manage a numerous socket, this model is used. It is often possible to achieve the best system performance, but unfortunately, the model is only available for the following operating system (Microsoft): Windows NT and Windows 2000 operating systems. Because of its complexity of its design, only when your application needs to manage hundreds and even thousands of sockets, and hopes that as the number of CPUs installed in the system increases, the performance of the application can also improve linearly. The "Complete Port" model should be considered. One basic criterion to be remember is, if you want to develop high-performance server applications for Windows NT or Windows 2000, you want to provide services for a large number of socket I / O (web server is a typical example of this), then I / O Complete port model is the best choice. From essentially, complete port model requires us to create a Win32 complete port object, manage overlap I / O requests by specifying the number of threads. To provide services for the completed overlapping I / O request. To be careful of. The so-called "completed port" is actually a I / O construction mechanism adopted by Win32, Windows NT, and Windows 2000. In addition to the socket handle, you can actually accept other things. However, this section is only intended to tell how to use a socket pen handle to perform huge power to complete the port model. Before using this model, first create an I / O complete port object, with any number of socket writing shanks. Manage multiple I / O requests. To do this - point, you need to call the CREATEIOCOMPLETONPORT function. This function is defined as follows: Handle FileHandle, DWord NumberOfconCurrentThreads; before we discuss each parameter, you must first note that the function is actually used for two significant purposes: ■ Create - a completed port object. ■ Associate a handle with the completion port. When you start creating - a completed port, the only interested parameter is the number of NumberOfConcurrentThreads concurrently); the three parameters of the front will be ignored. The special place of NumberOfConcurrentThreads parameters is. It defines the number of threads that allow execution on one completion port while allowing. Ideally we want each processor to be responsible - running of threads, providing service to complete ports, avoiding too frequent thread "Scene" switching. If the parameter is set to 0, how many processors have been installed in the system, allowing how many threads run at the same time! You can create an I / O completion port with the following code: completionport = createioCompletionPort (Invalid_Handle_Value, NULL, 0, 0); the language plus the role is to ask a handle. After assigning the port assigned to the completion port, it is used to calibrate the port (reference). 1. After the worker thread successfully created a completion port and completed port, the socket handle can be associated with the object. However, before the associated socket, you must first create - or multiple "worker threads" so that after I / O request delivery is delivered to the completed port object. Provide services for completion ports. At this time, everyone may feel strange, and how many threads should be created. In order to provide a service for completion port? This is actually the completion of the port model is quite "complex", because the number of service I / O requests depends on the overall design of the application. What to remember here is that the number of concurrent threads specified when we call CreateiocomletionPort, which is not the same as the number of workers' threads intended.
Earlier, we have suggested that you use the CreateiocompletionPort function to specify a thread for each processor (how much is the number of processors, specify how much thread) to avoid the entire system's overall performance. The NumberOfConcurrentThreads parameter of the CreateioCompletionPort function explicitly indicates that the system: On a completion port, only N workers threads are allowed at a time. If the number of workers created on the completed end is exceeded by n. So at the same time, only N threads are allowed to run. But in fact, the system may exceed this value in a short period of time. But it will soon reduce it to the value set in advance in the CreateiocompletionPort function. So, why is the number of workers actually created most sometimes more than the CreateiocompletionPort function? Is this necessary? As mentioned earlier. This depends primarily on the overall design of the application, assuming that our worker thread calls a function, such as Sleep () or WaitForsingleObject (), but enters the pause (lock or hang) status, then allow for another - a thread Instead of its location. In the wrap, we want to perform as many threads as possible at any time; of course, the maximum number of threads is set in advance in the CreateiocompletonPort call. This is coming. If you expect your own thread, it is possible to temporarily in the pause state, then you can create a number of threads that are more than the NumberOfConcurrentThreads parameter of Createiocompletionport. In order to give it to the potential of the system. - Once you have enough workers threads on the port to provide services to I / O requests, you can do it together with the complex handle. This requires us to call the CreateiOCompletionPort function on the existing completion port, and provide the top three parameters: FileHandle, ExistingCompletionPort, and CompletionKey - provide information about the socket. Among them, the FileHandle parameter specifies - a socket that is associated with the completion port. The EXISTINGOMPLETIONPORT parameter specifies an existing completion port. CompletionKey parameter specifies "single handle data" to be associated with a particular socket, in this parameter, the application can save any type of information corresponding to the neckline. The reason why it is called "single handle data" because it only corresponds to data associated with that socket. It can be used as a pointer to a data structure to hold the socket pen; in that structure, the handle of the socket is included, and other information related to that socket. As this chapter will also tell, the thread routine that provides services to the completion port can pass this parameter. Get information related to its set of handles. Based on our things we have learned so far. First build - a basic application framework. Program List 8-9 describes how to use the completed port model to others. To develop - a response (or "reflection" server application. In this program. We basically act as follows: 1) Create a completion port. The fourth parameter remains 0, specified on the completion port, each A processor only allows only one worker thread. 2) Determine how many processors installed in the system. 3) Creating worker threads, based on step 2), in completion of the port, I / O request provides service. In this simple example, we only create a worker thread for each processor. This is for prior expected that there will be no threads to enter "hang" "State, causes the processor idle situation (not enough thread to be performed) due to the shortcomings of the thread, and when the CreateTHRead function is called, you must provide - a worker thread, which is created by the thread. This section is created. The responsibility of the thread will be discussed later. 4) Prepare - a listener socket. In the port 5150, the entered connection request is listened. 5) Use the Accept function to accept the entered connection request.
6) Create a data structure for accommodating "single handle data." At the same time, the accepted socket is stored in the structure. 7) Call CREATEIOCOMPLETIONPORT Associates the new socket handle returned from accept to the completion port, passed the completion key (completionKey) parameter, the handle data structure is passed to the CreateiocompletionPort. 8) I / O operation is started on an accepted connection. Here, we want to deliver one or more asynchronous WSARECV or WSasend requests on the new socket by overlapping the I / O mechanism. After these I / O requests are completed, a worker thread will provide services for I / O requests while continuing future I / O requests, and will be in the operator routine specified in step 3). Experience this. 9) Repeat steps 5) -8). Until the server terminates. Program List 8.9 Completion of the port of the Port, stepwinsock () // Step 1. Create a completed port completionPort = CreateiocompletionPort (Invali_Handle_Value, NULL, 0, 0); // Step 2 Judging how many processors getSystemInfo (& systemInfo); / / Step 3: Create a working thread according to the number of processors, in this example, the number of working threads and the number of processors is the same for FOR (i = 0; i
2. Complete port and overlapping I / O After the socket handle is associated with a completion port, you can socket. Delivery or receive requests. Start I / O request processing. Next, you can start relying on the completion of the port to receive notifications about the I / O operation completion. In essence, the completion of the Ruikou model utilizes the Win32 overlapping I / O mechanism. In this mechanism. Winsock API calls like Wsasend and WSARECV will return immediately. At this point, it is necessary to be responsible for a certain time after our application. Receive the result of the call by an Overlapped structure. In completing the port model. To do this, you need to use the getQueuedCompletionStatus function. Let one or more workers threads wait on the port. The function is defined as follows: B00L getQueuedCompletionStatus (Handle CompletionPort, LPDWORD LPNUMBER0FBYTESTRANSFERRED, LPDWORD LPCOMPLETY, LPOVERLAPPED * LPOVERLAPPED, DWORD DWMILLISECONDS}
The CompletionPort parameter corresponds to the completion port waiting for the above .lpnumberofbytestransferred parameters are responsible for completing the number of bytes of actual transmission after completing the-secondary I / O operation (such as WSasend or WSARECV). The LPCompletionKey parameter returns "single handle data" for a socket that passed into the CreateCompletionPort function. As we mentioned earlier, everyone is best to save sockets in this "key" (Key). The LPOVERLAPPED parameter is used to receive overlap results of the completed I / O operation. This is actually a fairly important parameter because it uses it to obtain data for each I / O operation. The last parameter. DWMilliseConds is used to specify the time that the caller wants to wait for a completion of the packet to complete the port. If it is set to Infinite. The call will endlessly wait. 3. Single Handle Data and Single I / O Operation Data - A worker thread receives I / O after the API call from getQueuedCompletionStatus. In the LPCompletionKey and LPOVERLAPED parameters, it will contain some necessary socket information. With this information, you can continue the port on a socket by completing the port, through these parameters. Two important socket data: single handle data, and single I / O operation data. Where the LPCompletionKey parameter contains "single handle data" because the first time is associated with the completion port. Those data correspond to a specific socket handle. These data are passed through the CompletionKey parameter when we call the CreateiocompletionPort API. As mentioned earlier. The application can pass any type of data by this parameter. Typically, the application will save the socket pen with the I / O request here. The LPOVERLAPPED parameter contains a "single I / O operation data" behind it behind it. Our worker thread processing - When completing the packet (turn the data is not moving back, accepts the connection, delivering another - a thread, etc.). These information is it must be known. Single I / O operation data It is an arbitrary quantity byte to an overlapped structure. If a function requires an Overlapped structure, we must pass this - a structure to meet its search. To do this, a simple method is to define the structure. The Overlapped structure is then used as the first element of the new structure. For example, please. The above data structure can be defined to implement management of single I / O operation data: typedef struct {Overlapped overlapped; wsabuf data; char bufferl [data-bufsize]; bool operationtype;} per_io_operation_data; this structure is usually to be I / O Operation is associated with some important data elements, such as the type (send or receive request) that is just completed (send or receive request). In this structure. We believe it is very useful for data buffers for completed I / O operations. To call the WINSOCK API function, it is assigned an Overlapped structure for it, you can use your own structure "model" as an overlapped pointer, or you can simply revoke the reference to the OvBrlapped element in the structure. As shown in the following example: per_io_operation_data periodata;
Can call a function like this next side
WSARECV (Socket, ..., (overlapped *) & periodata;
Or like the following WSARECV (socket, ..., & (PERIODATA.OVERLAPPED); behind the working thread. The GetQueuedCompletionStatus function returns the overlapping structure (and completed keys). The reference to the OperationType member can be revoked. It is investigated which operation is delivered to this handle (just the return of the overlapping structure of the returned structure is the self-PER_IO_OPERATLON_DATA structure). For single I / O operation data, it is the biggest - an advantage to allow us to be on the same handle. Manage multiple I / O operations (read / write, multiple readings, multiple written, etc.). Everyone may produce such questions: Is it necessary to deliver multiple I / O operations at the same time? The answer is "scalability" of the system, or "extension ability". For example, assuming that our machine has installed multiple central processors. Every processor is moving a worker thread in a far away, then at the same time, there may be several different processors on the same socket, and perform data transmission and reception operation. In order to complete the aforementioned simple response server example, we need to provide a ServerWorkerthRead function. In the program list 8.10, we demonstrate how to design a worker thread routine, which allows you to provide services for I / O requests using single handle data and single I / O operation data.
Listing 8-10 completion port worker threads DWORD WINAPI ServerWokerThread (LPVOID CompletionPortID) {HANDLE CompletionPort = (HANDLE) ComleTionPortID; DWORD BytesTransferred; LPOVERLAPPED Overlapped; LPPER_HANDLE_DATA PerHandleData; LPPER_IO_OPERATION_DATA PerIoData; DWORD SendBytes, RecvBytes; DWORD Flages; while (TRUE) {// wait for I / O to Complete on any socket // associated with the completionport GetQueuedCompletionStatus (CompletionPort, & BytesTransferred, (LPDWORD) & PerHandleData, (LPOVERLAPPED *) & PerIoData, INFINITE); // first check o see whether an error has occurr // on the socket, if so, close the socke and clean // // per-handle and per-I / o Operation data associated with // socketiff == 0 && (Periodata-> OperationType == RECV_POSTED) && (PerIoData-> OperationType == SEND_POSTED) {// A Zero BytesTransferred indicates that the // socke has been closed by the peer, so you should // close the socket // Note: Per-handle Data was used to refresence the // Socket Assoc iated with the I / O operation; closesocket (PerHandleData-> Socket); GlobalFree (PerHandleData); GlobalFree (PerIoData); continue;} // service the completed I / O request; You // detemine which I / O request has just Completed // by looking as the OperationType Field Contained // The Per-I / O Operation Data IF (Periodata-> OperationType == RECV_POSTED) {// do someting with the receific Data // in Periodata-> Buffer}
// Post Another Wsasend or Wsarecv Operation // AS A Example We Will Post Another Wsarecv ()
Flags = 0;
// set up the per-I / O Operation Data for a Next // Overlapped Call
ZeroMemory (& (PerIoData-> Overlapped), sizeof (OVERLAPPED)); PerIoData-> DataBuf.len = DATA_BUFFER_LEN; PerIoData-> DataBuf.buf = PerIoData-> Buffer; PerIoData-> OperationType = RECV_POSTED; WSARecv (PerHandleData-> Socket , & (PERIODATA-> DATABUF), 1, & Recvbytes, & flags, & (Periodata-> overlapped), null);}} In the simple server example list listed in Program List 8-9 and Program List 8-10 (Supporting CD Also), the final payment detail is how to correctly close I / O completion port one - especially simultaneously run one or more threads, while performing I / O operation on several different sockets. An important issue to be avoided is forced to release the overlapped structure while performing overlapping I / O operations. To avoid this, the best way is to call the CloseSocket function for each socket writing handle. Any no overlapping I / O operation will be completed. - Once all socket handles are closed. It is necessary to stop the running of all workers threads on the completion port. To do this, you need to use the PostQueuedCompletionStatus function to send each worker thread - a special completion packet. The function indicates that each thread "end immediately and exit" Here is the definition PostQueuedCompletionStatus function: BOOL PostQueuedCompletionStatus (HANDLE CompletlonPort, DW0RD dwNumberOfBytesTrlansferred, DWORD dwCompletlonKey, LPOVERLAPPED lpoverlapped,); which, CompletionPort parameter specifies want to send a completion. The completion of the data packet. For the three parameters of dwnumberofbytestransferred, dwcompletionkey, and lpover. Each-all allow us to specify - value, passed directly to the corresponding parameters in the getQueuedCompletionStatus function. This is coming. - After receiving the three GetQueuedCompletionStatus function parameters passed, the worker thread can decide when to be exited according to the special value of a certain set of these three parameters. For example, 0 values can be passed with the DWCompletionPort parameter, and the worker thread will explain it into an abortment instruction. Once all workers threads are closed, you can use the CloseHandle function to close the completion port. Final safe exit program. 4. Other questions There are also several valuable technologies. Available in-step improvement of the overall I / O performance of the socket application. A technique worth considering is to test different socket buffer sizes to improve I / O performance and application's extension capabilities. For example, if a program only uses a relatively large buffer, it can only support - a WSARECV request, rather than simultaneous setting three smaller buffers. Provide support for three WSARECV requests, then the extension of the program is not very good, especially after moving to the machine where multiple processors are installed. This is because a single buffer can only handle one thread each time! In addition, single buffer design also causes certain interference to performance, if the-secondary reception operation can be performed. The potential of the network protocol driver has not been able to get full (it will often be "idle"). In other words, if you receive more data before receiving more data, you need to wait for the completion of the WSARECV operation, then between WSARECV completion and the next reception, the entire protocol is actually in the "Rest" state.
