LVS architecture analysis
林凡 ((iamafan@21cn.com)
Chen Xun Software Studio R & D Manager March 2002
In the previous three articles, we introduced the distributed architecture of the cluster, major elements, and some points of design of the cluster. In fact, only these theoretical basic knowledge is far less than enough. Any point in cluster technology is enough to write a few books. Importantly, through comprehensive introduction, we can discuss the foundation for the next case. This article will mainly introduce Linux-based cluster technology - IPVS (Based IP Virtual Server), also known as LVS (Linux Virtual Server). Through the specific analysis of LVS, understand the detailed analysis discussion of some issues that may encounter in architecture, technology implementation details and applications, step by step, gradually mastering the basic knowledge of clustering systems under Linux.
The problem is today, whether in the enterprise network, the park is still on the WAN, the development of traffic is exceeded by the most optimistic estimate; at the same time, users continue to pursue higher machine performance, and upgrade a single server system, often It has caused too high input and maintenance costs, which is much lower than expected. All this is not only for hardware, but also puts higher requirements for the software platform:
Scalability: Computer systems for good scalability enables performance to grow as cost increases. It is easy to streamline or expand it. All-weather availability support: strict business environment requires a soft hardware system to provide transparent, automatic adaptability support, ensuring that 24 × 7 system is operating normally. Manageability: The system can be very large, but it is easy to manage. Relative cost / performance advantage: Constructing such a system, which is economically economically, which is easy to customize the appropriate price target system according to the specific needs.
To resolve the previous analysis. We know that the cluster system is in cost, and the highly scalability of efficiency is the effective idea to solve this problem. Through relatively overall cost, a computer cluster, a single system is unable to provide powerless. Here we recommend the following features for the application of the Internet:
High scalability high availability high performance may be called "three high" systems.
The LVS architecture introduction Linux Virtual Server project is the founder and major developer of Dr. Zhang Wenwei (open source and Linux kernel. Famous Linux Cluster Project - Lvs (Linux Virtual Server) founders and major developers. He is currently working in parallable Distributed Treatment Key Lab, mainly engaged in cluster technology, operating system, object storage and database research. He also spent a lot of time in the development of free software, and this is a famous open source project, one Implement the solution of "three high" systems. LVS aims to solve the growing problem of high-speed development of Web business: how to increase the potential performance of Web sites in the case of limited funding.
LVS is a software tool under a Linux platform. With LVS, you can quickly and easily form a cluster system with the fourth layer load balancing. Also, by means of a third party toolkit, the functional extension of the LVS cluster can be implemented. First let's take a look at the architecture of LVS:
Figure 1: Schematic diagram of the three-layer architecture of LVS
From the figure above, we see that the abstract architecture of LVS is divided into three levels. The first layer is a load balancer, which is the unique entry of the cluster. From a client's point of view, the cluster is embodied in a single system image (SSI) based on the IP address, and the entire cluster can be used as a separate cluster as a separate IP address through its client. Host system, all access to the client is sent to this virtual IP address.
But we also found that if there is only one load balancer, it is easy to cause the load balancer as a single point of failure of the cluster, making it the most vulnerable link in the cluster. Therefore, it is necessary to provide fault tolerance mechanisms, which can automatically detect and smooth replacement when the load balancer is invalid, that is, the HA technology. In the structure above, there is a node that runs in a backup equalization, monitors the operating state of the load balancer in real time, and responds according to the detected state: alarm, take over, and recovery. The specific details will be discussed in the HA chapter. The second layer is a server group that provides actual services. After the service request sent by the client, it is transferred to the service pool by the specific server response request and returns data. Usually we will provide Web services, FTP services, or video on-demand services on the service node pool. Since a single system cannot cope with peak value data access, these loads are feasible through multiple servers.
The server node also may have a temporary failure, especially when providing a variety of services, the system's random fault or mutation of the external environment may cause a service of the node to be temporarily unavailable. Therefore, the fault tolerance mechanism of the load balancing is enabled to identify such an error, and process it in time. Similarly, when the error is excluded, the cluster can automatically identify recovery events and reintegrate the good node into the cluster.
The third layer is a storage service system that provides stable, consistent file access services for the entire cluster. This layer is an extension of the LVS cluster, which provides a single file system entry for the cluster node pool, that is, the same root (/) on each service node; and automatically completes different nodes to access the file system. The file lock, load balancing, fault tolerance, content consistent, read and write the underlying function of the transaction, providing a transparent file access service to the application layer.
The LVS cluster belongs to the loose coupling cluster system. Since the LVS implements SSI on the IP layer, there is no need to deploy special middleware or OS extensions in the cluster, comparing the compatibility of the server node OS. For internal nodes of deploying LVS, it is basically compatible with most IP applications, and there is no need to do complex portation and installation, each internal node can be regarded as a relatively independent server system. Even on the load balancer, IPvs' core functions are also transparent to user space, which does not affect the normal network application of this unit.
In fact, in reality, there are many technologies to achieve such systems. They use the load balance on a certain level, and share the network requests to zero, shared by a large number of clusters of service nodes to achieve a cluster technology that maximizes performance.
Load balancing technology actually, load balancing is not a "balance" in the traditional sense, in general, it is just aid that it is possible to congest on a local load to multiple places. If it is called "load sharing", it may be better. To be popular, the role of load balancing is like the turntable system, and the task is divided into everyone to complete, so as not to let a person exhausted. However, this sense of balance is generally static, that is, the "round value" strategy defined in advance.
Unlike the turntable day system, dynamic load balancing analyzes the data packets in real time, master the data traffic in the network, and reasonably assign the task reasonably. The structure is divided into local load balancing and geological load balancing (global load balancing), the former one is to do load balancing on the local server cluster, and the latter is to place the pair of different geographic locations, in different networks and The server cluster is loaded with load balancing.
In the load balancing system, each service node runs a separate copy of a desired server program, such as a web, ftp, telnet, or e-mail server program. For some services (such as those running on a web server), a copy of the program runs on all hosts in the cluster, and network load balancing will assign workloads between these hosts. For other services (eg, e-mail), only one host handle workload, for these services, network load balancing allows network traffic to flow onto a host, and moving the communication to other hosts when the host fails. Load balancing the structure of the structure, on the existing network structure, load balancing provides an inexpensive and effective way to extend server bandwidth and increase throughput, strengthen network data processing power, and improve network flexibility and availability. The main tasks are as follows:
Solve network congestion issues, service is available, realizing geographic location-independence provides users with better access quality to improve server response speed to improve server and other resources utilization efficiency avoiding network key sites with single point failure
For such a network load balancing technology, we will start from the different levels of network, and analyze the specific performance bottleneck. From the client application as the starting point longitudinal consideration, the load balancing technology can be divided into several different levels of different levels such as client load balancing technology, application server technology, high-level protocol exchange, network access protocol exchange, etc.:
Load balancing level
At present, there is a large number of technologies to achieve load balancing in every level, and its advantages and disadvantages are different, and for our understanding of LVS, only need to care about the load balancing technology of the network access protocol. This level of load balancing technology features:
High execution efficiency, because the underlying protocol can be deployed by the hardware system, or can be implemented at the core layer of the OS. Strong compatibility, access protocols tend to be compatible with most existing mainstream network applications, such as IP layers in the IPv4 system. The system is relatively simple, compared to the content-based high-level exchange, it does not require a complex pattern matching mechanism, mainly through port mapping, and simple rules.
Below we analyze the LVS framework and implementation methods based on load balancing technology.
The IP load balancing technology of LVS will be fundamentally, and the implementation of LVS is IP exchange, which is the previously mentioned access protocol switching technology. However, the LVS architecture has certain scalability, which can achieve many features such as high performance, high scalability, and management, becoming a real meaningful cluster system with load balancing.
First we understand the load balancing model of LVS, there are three: address translation (NAT), IP tunnel (IP tunneling) and direct route (DR) models.
◆ Address conversion mode NAT
NAT structure map and NAT package processing flow
We see that NAT's network structure is presented as a private network structure similar to a firewall, and the middle dashed line represents a network isolation belt. Through the internal IP address, the service node pool is separated from the Internet. Service nodes cannot communicate directly with the client, whether requested data or answer data, need to be loaded with IP packet processing.
The main job in NAT is to rewrite the source of the IP package, making the destination address information, so that the request data to the VIP will rewrite the internal host after rewriting; the same internal response data is overwritten after the load balancer, the VIP is sent as the source address to the source address The requester. Such modes are also called network address transformations (also known as IP address camouflage), we use this model in applications such as proxy servers, iptables, transparent gateways, which can be said that this is a relatively mature technology.
Due to the use of NAT, the network packets to enter and outline the cluster are used to overwrite the header address, which will affect the performance of the entire cluster when the load is more important, and the load balancer is easy to become a bottleneck. ◆ IP Tunnel Mode IPIP
IPIP structure diagram and iPip package processing process
The IPIP mode uses an open network structure, and the service node has a legitimate Internet IP address, which can return the answering package directly to the client through the routing path. Therefore, the load balancer only processes the request packet into the cluster, and the return package does not pass the router. Therefore, this mode is called a single work connection mode (unilateral connection mode). The connection of the load balancer and service node can be a LAN, or on a different network, it is only necessary to ensure that the load balancer can send the IP package to the service node.
After receiving the client's request package, the load balancer receives the IP packet to the IP package, form a new IP package for the selected service node, the original IP package data package in new IP Package. After the service node receives the IPIP data sent by the equalizer, the package is unwounded, and the processing result is directly returned to the client according to the client address (source address), and the source address of the response package is a cluster. Virtual address VIP.
IPIP mode technology is also reflected in other fields because the IP is reproduced, and the entire process is still transparent to the application layer. The PPTP protocol is an application for IP tunneling protocols. However, IPIP is currently implemented on Linux systems. This protocol must open the device option support in the Kernel. Bind the VIP through the Tunel device, when the service node is returned to the answer data, you can construct a response package as the source address as the source address.
◆ Direct routing mode DR
DR structure map and DR package processing flow
Like IPIP mode, the DR mode is also a single connection method, and the response data is not returned directly to the client directly. Service nodes must also have legitimate IP addresses that come to the client. Moreover, in the DR mode, the load balancer and service node must be in the same network segment.
After receiving the client request, the load balancer receives the appropriate service node, then override the MAC address section of the request package, making it a MAC address of the destination service node, and broadcast this package to the network segment where the equalizer is located. . Since each service node has a virtual network 括 device (can be Dummy0 or LO: 0), these devices are bound to the equal VIP as the equalizer, but the device does not respond to the VIP's RAP resolution, no Will conflict with the VIP address of the equalizer. After the load balancer receives the IP package that meets its own Mac, the response data is directly returned to the customer after processing, and the source address is still VIP. In this way, at the client's view, the access and acceptance response is always the VIP address of the cluster.
Comprehensive comparison Although LVS supports three load balancing modes, we have found that the LVS actually contains two models according to the input and exit mode of the load balancer, and the LVS actually contains two models: single work processing and duplex (two-way connection mode )deal with. Obviously, the NAT address conversion mode belongs to the dual work connection processing. In this mode, the load balancer not only needs to process the IP packets entering the cluster, but also handle the answering IP packets returned by the cluster internal node, one user from issuing access requests Acceptance response, the processing of the core load balancer of the cluster is therefore referred to as a duplex connection process. In other two modes, the load balancer only processes the IP request packets entering the cluster, and the response data of the internal nodes of the cluster no longer returns the client through the load balancer, but passed through the routing channel to the client directly to the client. destination. Since the equalizer processes only the IP request part of a complete connection, the response data for IP does not process, so it is called a single work connection mode.
What is the shortcomings of the two? To know, in today's web world, most network requests are relatively small, nothing more than some URL page requests, GET or POST forms, is some instructions, etc., these data is basically in several hundred to several k byte. It is relatively easy to process such an IP packet. On the contrary, the response data in the web is usually very large. If a normal web page is dozens of K, not to mention if the video, the audio flows, plus the increasing crazy network download, even if the strong processor is also Unable to withstand such a large number of IP package processing. Therefore, in the IP load balancing, if the duplex mode (NAT) is used, it is necessary to process the request to enter the cluster (the source of the IP package, the destination address), but also to do the same data returned by the service node. work. Then, with the growth of the cluster service node pool, the processing power of the load balancer will soon be saturated, and it also greatly affects the scale scalability of the LVS cluster. Using IPIP or DR mode, the load balancer only needs to handle relatively small IP packets, and for a large number of return data, the service nodes are directly returned to the client through the router, switches, and other devices. Therefore, in size scalability, the single work mode has the advantage of scalability.
There is always his truth in the existence of things. The author has originally designed these three load models, and it must be its own shortcomings. Nat is not an inendency. NAT Although the performance performance is weaker than the other models, the cluster node supports more operating systems and is relatively high in terms of network security. The following is the comparison of the three in all aspects:
NAT Mode IPIP Mode DR Mode The requirements for the service node require the service node to be any operating system service node must support the IP tunneling protocol. Currently only Linux service node supports virtual network card devices, which can disable the device's ARP response function network requirements. LAN with private IP address has a legal IP address LAN or a wide area network with legal IP addresses, service nodes and equalizers must be 10-20 nodules supported by the same network segment, and the processing of test heralders The ability is high, can support 100 service nodes, can support 100 service node gateway equalizers, the gateway service nodes for the serving node, connected to their own gateways or routers, without equalizer The service node is connected to its own gateway or router. Without the security of the equalizer service node, the internal IP, the service node is concealed, using a common IP address, the node is completely exposed, using a common IP address The node completely exposed IP requirements only one legal IP address as VIP except VIP, each service node needs to have legal IP addresses, can directly route to the client to remove VIP, each service node needs to have legal IP address, you can route directly to the client
Comparison of three modes
In summary, when we select LVS as a cluster load balancing solution, we first determine which IP load balancing structure used according to the pre-application environment. If you only have a legitimate IP address, or you need to construct a safe cluster, you can't worry about performance problems, you can use NAT mode; if you have a relatively high requirements for performance, the application is Linux, Use IPIP or DR mode will give you a surprise.
About the author: Lin Fan, is now engaged in Linux related research work in Xiamen University. The cluster technology is greatly interested in communicating with like-minded friends. You can contact him via email iamafan@21cn.com.
