Optimization and maintenance experience of heavy load Telnet BBS system
KXN
God said, there is light, so there is light;
God said, to irry, so there is BBS ...
The BBS we mentioned now, usually refers to Telnet BBS, with a TERM software connection, you can see the interface of the text, compared to today's fancy to unreasonable WWW bbs, but it is like this. BBS, countless people face it for two hours a day, I am afraid UI design experts know that they want to vomit blood. From time to time, there are predictions, predict that Telnet BBS will soon die and be more expressive WWW BBS, just year, the prophet has disappeared, but the number of users on BBS has turned over One. . . This is the charm of Telnet BBS.
The number of Telnet BBS systems is large, but from the root source, it can be roughly divided into two major families, Firebird BBS and MAPLE BBS, which takes a absolute advantage in the variant of the mainland Firebird BBS, and is the world of Maple BBS in Taiwan, due to Taiwan's computer The development history is relatively long, so the popularity of BBS is higher than the mainland, and there are several sites at the same time. However, the mainland has a population advantage. In recent years, the number of people in education network has grown rapidly. Below we introduce these two big BBS families. The first is the most popular Firebird BBS in the mainland, the most famous SMTH BBS, YTHT BBS, and Firebird 2000 three genres come.
A long time ago, there was such a group of college students, or the researcher of research institutions, they rolled over the Unix host all day, I think it is better to make a forum like a forum in the host, so they wrote a command The row program, run this program, the operator can leave a message below the interface, in order to make multiple people can operate this system, they set this program as the shell of a system, each Telnet's user, as long as this If the user's username and password are logged in, they can communicate. This is the prototype of the Internet BBS. After a period of development, this system has a considerable interaction function, but the user can not only leave a message, but also to send letters, send information, and see online users, etc.
In order to allow more people to use this system and perfect, developers of the BBS system, publish the BBS system to the network. As long as you have a UNIX host, you can get the source code and install the BBS system. Therefore, the BBS system develops in a very fast speed. In many BBS systems, some people called Pirate BBS, after the modification of some people, it is called Eagle BBS, and it has entered the Taiwan area. The Department of Jiaotong University of China has developed Phoenix BBS from him. Phoenix BBS is now most Chinese. The ancestors of the Telnet BBS system, however, its name is far less than its consecutive generation, the BBS system that further modifies the mainland BBS developer in its basis, which is given the mainland BBS developer's familiar name - Firebird BBS.
It should be said that the BBS system is time-intensive, although the function is relatively simple, but the basic architecture of the BBS system has been set, such as multi-process model, shared memory information exchange, using system signals to pass call messages, store articles and indexes with files Wait, these designs are still along the current BBS system, many of which are designed even if they are now, it is also a quite standard effective multi-process UNIX server design. Let's go into the body.
Telnet BBS is an electronic bulletin system that is popular in universities and research institutions. Unlike the WEB BBS system, the popular Web BBS system, the interface of the BBS is expressed in plain text, and the user uses the terminal software to connect the BBS system, and the text interface is generated on the server. And send it, the client software only shows the text content, belonging to a thin client application. Telnet BBS (later unless otherwise mentioned, referred to as BBS) is popular in the Taiwan and the mainland education network area, and the number of large-scale sites is generally above 10,000.
Due to historical reasons, the BBS system uses a quite traditional 1: 1 multi-process model under UNIX. The advantage of each process is handled, the advantage of this model is relatively stable, and it will not cause the entire system because of a user error. Not available, but it also brings a problem that more difficult communication between resources and processes. The complex logic of the BBS server is also difficult to implement distributed design. Therefore, BBS is usually a single machine to bear almost all loads, and the large-scale BBS server in mainland China often maintains more than 7,000 processes, and the BBS station in Taiwan has a record of 20000 process.
In the process of maintaining large BBS sites, we have accumulated some experiences such as BBS such as BBS, considering the 1: 1 process model service, there is still a wide range of applications, and is written here and readers.
Optimizing the server is a comprehensive job, not only need to modify the code, but also adjust the system parameters, including a lot of trivial content, depending on the purpose, can be divided into disk IO optimization, memory optimization, and CPU optimization based on the type of resource saving Wait a few aspects. The optimization idea described below is applied to BBS, but it also applies to other application systems.
1: Disk IO Optimization
Disk IO optimization can be said to be the most important ring in server optimization, except for a very small number of pure calculated applications, almost all overload servers are finally cards on disk IO bottlenecks.
a) Try to use IPC means such as SHM instead of file
Compared with multi-process and multithreading, the biggest trouble is that different execution environment exchange information is inconvenient. Therefore, many programmers have selected file exchange information, such as the earliest BBS design, the user's account information is existing in the file, process Read the content from the file, write the file after modification. After the improved design is to complete the account information file completely read the shared memory, all modifications are written to shared memory, and then synchronize the external process to the disk.
Even the flock does not cause too many IO synchronization operations to avoid as much as possible because Flock needs to first open files, and the Open file needs to find I nodes, so the file system's inode cache can cause other IO operations. Performance is reduced. In many cases, just a cross-process MUTEX, you can use a 0/1 signal light.
b) Use the application layer cache.
Obviously, the cache of the operating system will be disturbed by many factors. For some content that determine often access, such as the latest few articles and the latest lists of the layout, if they are buffered in SHM, performance will have substantially improved.
c) minus the size of key IO data structures
The index file of the BBS article list is made by the fixed length data structure. In this data structure, it has left a lot of reserved domains in order to make a lot of reserved domains, causing a lot of unnecessary IO, after deleting unnecessary domains, data structure change Half a small, a lot of IO. Many times, scalability and performance are actually opposite, and if it is necessary, it is also a good choice for a certain extension. d) Avoid excessive files in the same directory or use the appropriate file system
Most file systems use linear storage to file lists in the same directory, so when there are many files below a directory, open files become very slow, so they are usually to have files to different sources according to some rules. In the directory, for example, file ATEST will be stored in A / ATest, if there is too much file, you may need to have a hash next to the subdirectory.
Another way to solve excessive influence efficiency is to use a specially optimized file system, such as Reiserfs under Linux. In these file systems, the list of files in the directory is organized with a balance tree, so hundreds of thousands of files below the same directory will not reduce much.
e) Select the appropriate hardware configuration and system parameters based on the system's access mode
The BBS system uses a scattered file to store the article. Its access mode is basically a small file random reading, and the article data is relatively important, so BBS is relatively small RAID5 with a Strip size. The file system selection of Reiserfs specifically for small file optimization, the system's pre-reading length can also be toned, and the default length of Linux is 256K, some is large. If it is a big file to read and write, then RAID's Strip size and system pre-reading length should be amplified. If the file system is selected as much as possible, the file system, such as EXT2 / 3, etc., if the data is not very important, then you can cancel RAID5, in order to use RAID0 or use separate hard drives directly.
2: Memory use optimization
The multifunction mode used by the BBS system is quite consuming, in the process of BBS development, the first bottleneck is memory. Reduce the unnecessary waste of memory, save it as a system cache, thereby indirectly improving more important IO performance
a) Try to avoid dynamic initializing constants, using constant to distinguish between variables and constant distractions.
The UNIX system is in the new process of Fork, the child process and the parent process share the same space, followed by the COW mechanism, copy the modified page, if the constant can be pre-calculated (for example, some conversion tables), Should try to avoid dynamic initialization during runtime. Also because you only modify one byte, the entire page will be copied, so you should avoid the constant and the variables that will be modified together, the compiler will automatically be modified, the programmer needs to do. This is to use the Const to notify the compiler which content will not be modified.
b) Reduce the peak use of memory, especially in the stack
Many people are used to writing a program to declare a relatively large temporary array on the stack and think that this part of the memory will be automatically released after exiting the function. As everyone knows that the memory is not being recycled by the system, because the system does not know if the system can know if the stack memory is still in use, especially in the multi-threaded environment, the action system usually taken is needed. When you assign a page, it is not reclaimed before the process exits. Even through the pile of memory allocated by Malloc, the page is reclaimed, and the implementation of the library function is not determined. Therefore, in any case, there is an excessive memory allocation, it will have a certain impact on performance.
c) If possible, try to use SHM to ensure that the page will be shared by multiple processes.
3: CPU optimization
The CPU mentioned here is optimized, excluding the most basic algorithm optimization such as using HASH instead of linear lookups, but involves some and systematic relationships. a) use compilers optimized for hardware
This should be the easiest to do in all CPU-related optimizations. It is also the most easy to see the effect. The Intel CPU's Linux system uses the Intel C / C compiler, you can get good results, and even AMD's Athlon series CPU can also get A certain degree of acceleration. When the BBS is inbred, there is a lot of content, the calculation amount is relatively large, and the load is about 4 or so when using the GCC, and the load is immediately dropped to 1 or less after using the ICC compilation. Recommended compile time to optimize the specific CPU instruction set and open the Cross File Optimization Option (-IPO)
b) Use a separate process to initialize and maintain shared content, avoiding competition, resulting in logic errors
Strictly speaking this does not improve many performance, just to reduce the logic errors that often appear on the multi-process server. In the original BBS design, the creation of shared resources is created by the first access to the open failed, but the heavy load server will also open, which will fail to create a shared resource multiple times.
c) Serialization is easy to cause behavior of load rise
The BBS process needs to make a lot of initialization work in the station. When the process exits, it is also necessary to do a lot of finishing work. At this time, it is relatively large to the CPU or IO. You can make multiple processes simultaneously through a mutex. These operations, otherwise the system load may rise to a certain extent cause positive feedback, resulting in a complete crash.
d) minimize signal use
The cost of the implementation of the signal under the UNIX system is relatively large, and the signal itself is also easy to cause the process of processing logic. High load servers should try to reduce signals.
e) Use MMAP calls for large-range IO read operations
Using MMAP operations is more than traditional READ operations to reduce copying of kernel to user states. When there is a wide range of IO operations, the BBS is used in the file to search for content in the file, and the speed is increased by about 5 times. However, it should be noted that MMAP does not apply to a case where the write is written because the MMAP written is operated in a page, and the page will be rewritten, and the entire page is written on the disk. Data, the endless increase IO amount.
The above is some of our experiences in maintaining large BBS sites for readers.
BRIEF DESCRIPTION OF THE DRAWINGS: Shuimu Tsinghua BBS roughly structural map
