Improve boot speed by parallelizing Linux system services
Level: Intermediate
James hunt (jameshunt-at-uk.ibm.com software engineer, IBM2003 November 2003
This article describes how to speed up the boot speed of the Linux operating system without affecting availability. Of course, this method requires you to understand the dependencies between system services and services, if they can start parallel, let them boot in parallel instead of serial startup.
There is no doubt that Linux is an excellent system, but it is still unable to get rid of a common blame (especially those from the background with Microsoft Windows background), that is, the Linux system starts from pressing the "ON" button to us, it takes too long. In fact, they are right, Linux does need more boot time.
Although the technology I have described here is very simple, it is very simple, but it is really realized that it needs to be cautious. I hope that Linux's issuance can use this method so that users can save those configuration tasks. However, if you like an adventure, please continue to read this article.
Write before the start If you want to experience this method, you must first be familiar with Linux configuration scripts. Modifying the system's startup settings may cause danger, and may even cause your system to be unable to start. If this happens, restart the machine and enter a single user mode (Runtime 1), restore your modification, and then restart. Always remember that you want to back up all the files you have modified, in order to prevent the worst case, you also need to have at least one image backup image.
I strongly recommend that you have to modify a non-related test system before considering the way I suggested. If you have only one machine, you can use UML (User Mode Linux) very useful tools. UML is a kernel patch that compiles Linux kernels into a binary, and then you can run this kernel as you run a normal program. That is, you can run a complete Linux system on your normal system. You can imagine it to run a Linux system in a normal system. (Please refer to the reference information at the end of this article, you can find a site that can be downloaded from the UML and the Tutorial about the UML on the developerWorks website).
Using UML you can work in a test system, even if this test system is completely destroyed, it will not affect your normal system.
Overview The first part of this article describes how a Linux system starts in the background after the Linux kernel ("core" of Linux machine) is loaded. Then introduce technology that speeds up your system boot speed.
If you are already familiar with the running level and service startup script, you may want to directly jump to the limitation of the traditional service framework.
The Linux boot order and the boot process of running a Linux system can be divided into several phases. This article does not explain all different phases, as we care about the stage after the kernel is loaded.
You can run the / sbin / runlevel command to determine your system's current run level. (For more details, please refer to Man Runlevel).
The / Sbin / init program is called when the kernel is loaded and started. This program runs as root and sets to "Running" as required when starting boot. (More details on the init program, please refer to Man Init)
What is the run level? A run level is just a number, Linux distinguishes different types of high-level configurations based on this number, and the system will be booted in different high level configurations. Because most of the operating level numbers define a clear meaning, they are basically "well known". The main operational level of the Red Hat Linux system is shown in Table 1.
Table 1. Red Hat Linux
Running Level Description 0 Close 1 single user mode (generally only for management purposes only) 2 multi-user mode, not allowing network 3 multi-user mode, allowing 5 multi-user modes that are not used using network 4, allowing the use of networks, X-windows mode (graphic login interface) 6 reboot
How to initialize the system initiates through an ASCII profile (/ etc / initTab) to determine how to change the run level. Typically, init will run the /etc/rc.d/rc script based on this profile and pass the running number to this script.
Rc.sinit Scripts In the Red HAT system, INIT will first run the /etc/rc.d/rc.sysinit script before running the RC script. This script performs those required underlying settings, such as setting the system clock, check the disk error. Then mount the file system.
In this article, it is intended to start from running the RC script.
The system service RC script is responsible for starting all the services you need. As described in the name, the so-called service is a useful tool for the system. There may be many services to start. Most of the Linux system starts SSHD (Safe Shell Services), Syslog (System Log Tools), and LPD (Print Services), but there will be more services to start. For example, my Red Hat 9 system is now running 29 services, but if I start all the services, my system will have nearly 50 services running.
There is also a little important, we should understand that the service may only be started by a specific run level. For example, in addition to running level 5 (multi-user graphics mode), some form of graphics service is hardly launched because all other runs are non-graphical mode. Next we will discuss this problem in depth.
Where is the service program?
Optional Server Directory In some Linux systems, the service program is sometimes in the /etc/init.d directory.
The service program is usually found in the /etc/rc.d/init.d/ directory.
If you browse this directory, you will find quite a few (if not all is all,) the service program is actually a shell script, which is used to call other programs to complete the actual work.
How do RC scripts know which scripts are running at each run? Looking back, if we don't want to run a script under a certain period, how do we tell the system to do this? The answer is in the /etc/rc.d/ directory, in this directory, in addition to the init.d / directory we have discussed, there is a set of directories, each directory corresponding to a run level. These catalogs are named in the form of RC
Service link names These are important to the name of the symbolic link to the actual service, which follows a strict naming agreement so that the RC script knows how to handle them.
For ease of logo, each link name is a suffix in the name of the service they point to.
The prefix consists of two parts: a uppercase letter, followed by a two decimal number. The uppercase letter in the prefix is "S" (indicating "start"), or "k" (indicating "killing", or stops). The size of the two digits is from 00 to 99. The name of the service link name regular expression symbol link can be described by EGREP regular expression, [SK] [0-9] {2} [A-ZA-Z] . (For more details, please refer to Man EGREP).
Start and stop service If we decide to lead the Linux machine to the graphics mode (running level 5), the RC script will go to /etc/rc.d/rc5.d/ when the init calls the RC script and passes it. Find, and go to all symbolic links that it can find (that is, it will run the program / script to each link). It will run these links in two distant phases; first it will execute all links starting with "K" while passing to them "stop". After the execution, all these links pointed to by the service are stopped.
When the RC script stops all the services that need to be stopped, it will perform all links beginning with "S" while passing to them "start". After the execution, the services that these links point to the link are started. The RC script also passes the parameter "start" to each program.
RC passes the parameter "tart" or "stop" to every service, which is to use only one service program to start or stop that service - the server is guided by the parameter value to which the service program is passed to it. shut down.
There is an important part I have not explained - the digital part of the link name. The two decimal numbers after "S" or "k" are the order of the RC script used to determine the start link (which is the link pointing to the service). Links with smaller numbers (such as 00, 01, etc.) are run before the number larger (99 is the largest) link. We will mention this key issue again in the contents of this article.
Is it still confused now? Listing 1 lists all links in the corresponding directory of the run level 5. When booting to run level 5, the first link will be K05SASLATHD because it starts with "K" and the two decimal numbers are minimal in all links starting with "K". The first start-up link will be S05kudzu because it starts with "S", and the two decimal numbers are minimal in all links starting with "S". The last running link will be S99LOCAL.
Listing 1. Link of the pointing service program
# cd /etc/rc.d/rc5.d
# ls -al
Total 8
DRWXR-XR-X 2 root root 4096 JUL 15 09:29.
DRWXR-XR-X 10 root root 4096 jun 21 08:52 ..
LRWXRWXRWX 1 root root 19 Jan 1 2000 K05SASLATHD -> ../init.d/saslauthd
LRWXRWXRWX 1 root root 20 Feb 1 2003 k15postgreSQL -> ../init.d/postgreSQL
LRWXRWXRWX 1 root root 13 Jan 1 2000 K20NFS -> ../init.d/nfs
LRWXRWXRWX 1 root root 14 Jan 1 2000 K24IRDA -> ../init.d/irda
LRWXRWXRWX 1 root root 17 Jan 1 2000 k35winbind -> ../init.d/winbindlrwxrwxrwx 1 root root 15 Jan 1 2000 K50SNMPD -> ../init.d/snmpd
LRWXRWXRWX 1 root root 19 Jan 1 2000 K50SNMPTRAPD -> ../init.d/snmptrapd
LRWXRWXRWX 1 root root 16 jun 21 09:43 K50VSFTPD -> ../init.d/vsftpd
LRWXRWXRWX 1 root root 16 jun 21 08:57 k73ypbind -> ../init.d/ypbind
LRWXRWXRWX 1 root root 14 jun 21 08:54 k74nscd -> ../init.d/nscd
LRWXRWXRWX 1 root root 18 Feb 8 11:15 k92iptables -> ../init.d/iptables
LRWXRWXRWX 1 root root 19 Feb 1 2003 K95FirstBoot -> ../init.d/firstboot
LRWXRWXRWX 1 root root 15 Jan 1 2000 S05kudzu -> ../init.d/kudzu
LRWXRWXRWX 1 root root 14 jun 21 08:55 s09isdn -> ../init.d/isdn
LRWXRWXRWX 1 root root 17 Jan 1 2000 S10Network -> ../init.d/network
LRWXRWXRWX 1 root root 16 Jan 1 2000 S12Syslog -> ../init.d/syslog
LRWXRWXRWX 1 root root 17 Jan 1 2000 S13portmap -> ../init.d/portmap
LRWXRWXRWX 1 root root 17 Jan 1 2000 S14NFSLOCK -> ../init.d/nfslock
LRWXRWXRWX 1 root root 18 Jan 1 2000 S17keyTable -> ../init.d/keytable
LRWXRWXRWX 1 root root 16 Jan 1 2000 S20random -> ../init.d/random
LRWXRWXRWX 1 root root 16 jun 21 08:52 S24pcmcia -> ../init.d/pcmcia
LRWXRWXRWX 1 root root 15 Jan 1 2000 S25Netfs -> ../init.d/netfs
LRWXRWXRWX 1 root root 14 Jan 1 2000 S26APMD -> ../init.d/apmd
LRWXRWXRWX 1 root root 16 Jan 1 2000 S28AUTOFS -> ../init.d/autofs
LRWXRWXRWX 1 root root 14 Jan 1 2000 S55sshd -> ../init.d/sshdlrwxrwxrwx 1 root root 20 Jan 1 2000 S56RawdeVices -> ../init.d/Rawdevices
LRWXRWXRWX 1 root root 16 Jan 1 2000 S56XINETD -> ../init.d/xinetd
LRWXRWXRWX 1 root root 14 Feb 1 2003 S58NTPD -> ../init.d/ntpd
LRWXRWXRWX 1 root root 13 Jun 21 10:42 S60AFS -> ../init.d/afs
LRWXRWXRWX 1 root root 13 Jan 1 2000 S60LPD -> ../init.d/lpd
LRWXRWXRWX 1 root root 16 Feb 8 17:26 s78mysqld -> ../init.d/mysqld
LRWXRWXRWX 1 root root 18 Jan 1 2000 S80sendmail -> ../init.d/sendmail
LRWXRWXRWX 1 root root 13 Jan 1 2000 S85GPM -> ../init.d/gpm
LRWXRWXRWX 1 root root 15 mar 22 08:24 s85httpd -> ../init.d/httpd
LRWXRWXRWX 1 root root 15 Jan 1 2000 S90crond -> ../init.d/crond
LRWXRWXRWX 1 root root 13 Jan 1 2000 S90XFS -> ../init.d/xfs
LRWXRWXRWX 1 root root 17 Jan 1 2000 S95anacron -> ../init.d/anacron
LRWXRWXRWX 1 root root 13 Jan 1 2000 S95ATD -> ../init.d/atd
LRWXRWXRWX 1 root root 15 jun 21 08:57 S97RHNSD -> ../init.d/rhnsd
LRWXRWXRWX 1 root root 14 jul 15 09:29 s98wine -> ../init.d/wine
LRWXRWXRWX 1 root root 13 Feb 8 17:26 s99db2 -> ../init.d/db2
LRWXRWXRWX 1 root root 11 jun 21 08:52 S99LOCAL -> ../rc.local
#
This looks like a very complex system, but in fact it provides excellent flexibility, because if you want to temporarily prohibit service in a particular run level, just remove the appropriate symbolic link. However, manual management of these links may make people feel bored, and it is easy (especially when you are tired), so you can use a relatively good method to use the chkconfig command.
ChkConfig and Xinetd If you have a new version of ChkConfig, you will see the configuration of the XINETD (Internet Services daemon) in the last part of the main vide. In order to degrade, this portion is not included in Listing 2. How to find the activated service want to see how much service you have activated, run this command:
/ sbin / chkconfig --List
Listing 2 lists the output of this command. You can see that each line has eight columns.
The chkconfig command can also be used to switch any of the services of any service. For details, please refer to Man Book (Man Chkconfig).
Listing 2. CHKCONFIG --List | Output of Sort
AFS 0: OFF 1: OFF 2: OFF 3: on 4: Off 5: on 6: OFF
Anacron 0: OFF 1: OFF 2: on 3: on 4: on 5: On 6: Off
APMD 0: OFF 1: OFF 2: On 3: on 4: on 5: on 6: Off
ATD 0: OFF 1: Off 2: OFF 3: On 4: on 5: on 6: OFF
Autofs 0: Off 1: Off 2: OFF 3: On 4: on 5: on 6: OFF
Crond 0: OFF 1: Off 2: on 3: on 4: on 5: on 6: OFF
DB2 0: OFF 1: OFF 2: OFF 3: On 4: Off 5: on 6: OFF
FirstBoot 0: Off 1: Off 2: Off 3: Off 4: Off 5: OFF 6: OFF
GPM 0: OFF 1: OFF 2: on 3: on 4: on 5: on 6: OFF
HTTPD 0: OFF 1: OFF 2: Off 3: Off 4: Off 5: on 6: OFF
iptables 0: Off 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
Irda 0: OFF 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
ISDN 0: OFF 1: OFF 2: on 3: on 4: on 5: On 6: Off
KeyTable 0: OFF 1: on 2: on 3: on 4: on 5: on 6: OFF
Kudzu 0: OFF 1: OFF 2: Off 3: on 4: on 5: on 6: OFF
LPD 0: OFF 1: OFF 2: on 3: on 4: on 5: on 6: OFF
MySQLD 0: OFF 1: Off 2: Off 3: on 4: OFF 5: On 6: OFF
Netfs 0: OFF 1: Off 2: Off 3: on 4: on 5: on 6: OFF
Network 0: OFF 1: Off 2: on 3: on 4: on 5: on 6: OFF
NFS 0: OFF 1: Off 2: Off 3: Off 4: Off 5: Off 6: Offnfslock 0: Off 1: Off 2: Off 3: on 4: on 5: On 6: OFF
NSCD 0: OFF 1: Off 2: Off 3: Off 4: OFF 5: OFF 6: OFF
NTPD 0: OFF 1: OFF 2: OFF 3: on 4: Off 5: on 6: OFF
PCMCIA 0: OFF 1: OFF 2: On 3: on 4: on 5: On 6: Off
Portmap 0: OFF 1: Off 2: Off 3: on 4: on 5: on 6: OFF
PostgreSQL 0: OFF 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
Random 0: OFF 1: OFF 2: on 3: on 4: on 5: on 6: OFF
Rawdevices 0: Off 1: Off 2: Off 3: on 4: on 5: on 6: OFF
RHNSD 0: OFF 1: OFF 2: OFF 3: on 4: on 5: on 6: OFF
SaslauThd 0: Off 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
Sendmail 0: OFF 1: OFF 2: On 3: on 4: on 5: On 6: OFF
SNMPD 0: OFF 1: OFF 2: Off 3: Off 4: Off 5: Off 6: OFF
SNMPTRAPD 0: OFF 1: OFF 2: Off 3: Off 4: Off 5: Off 6: OFF
SSHD 0: OFF 1: OFF 2: on 3: on 4: on 5: on 6: OFF
Syslog 0: OFF 1: OFF 2: on 3: on 4: on 5: on 6: OFF
VSFTPD 0: OFF 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
WINBIND 0: OFF 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
Wine 0: OFF 1: OFF 2: on 3: on 4: on 5: on 6: OFF
XFS 0: OFF 1: Off 2: on 3: on 4: on 5: On 6: Off
XINETD 0: OFF 1: OFF 2: OFF 3: on 4: on 5: on 6: OFF
YPBIND 0: OFF 1: Off 2: Off 3: Off 4: Off 5: Off 6: OFF
The first column in Listing 2 is the name of the service, the next column is the status of the service level and each run level. For example, NTPD (NETWORK TIME DAEMON "service is configured to start only in Runtime 3 (multi-user, no graphics) and running level 5 (multi-user, with graphics), SSHD service is running 2, 3, 4 and 5 is switched to the state. Note that there is no service to start in operation stages 0 and 6. Review Table 1, the reason is obvious. Runtime 1 means that you want to turn off or stop the system, so you don't want to start any service when the machine will be "off". This is also true in running level 6.
Run level 1 - "Single User Mode" - is a special run level, which is generally used when the system has problems. All the unique applications running in Run Level 1 is the shell, allowing superusers to repair the system or let the super user modify the system in a secure environment. This is safe - just like its name "Single User Mode" - only superusers can access the system. Also, the network is disabled, so no one can log in remotely. As shown in Table 1, the only service running in a single user mode is KEYTABLE, which makes the keyboard of the super user can be used normally.
The comparison of the activation service and running service sometimes does not start because some reason cannot be started, using the following command to see which services are running:
/ sbin / service --status-all
This command will output a row or multi-line for each service, indicating whether each service is running, if running, list some specific outputs, such as the PID (process number) of the service run. The service command does not have a man page, but you can use the --help option when running this command, you can get some help information about it.
The deficiencies of the traditional service framework are that you can log in to your Linux system only when all services in the configuration starts. Waiting for 50 service startup may take several minutes, and this should be a time you enjoy the Linux system.
I have found a way to accelerate this process. Note that this method does not stop any services. In any case, the service that doesn't have to be stopped is very wisdom, not only because this can speed up the speed of boot (less service is required before the machine can log in), and because many services should be run as root user, The unused service will reduce your safety hazard.
Stealing about it, when a Linux system is booted, it runs all the services configured by a certain period of time in a continuous manner - one. This is a time consuming operation.
Perhaps a significant approach to speed up the service startup speed is to launch all the services in parallel, so they can start at the same time. Unfortunately, although this sounds very attractive, it is not feasible. The reason is that there is a dependency between the services. Linux did not explicitly explicitly explicitly, but in fact these dependencies exist. Remember the question about the link name format we have previously discussed? The two digits after "S" and "K" decide the operation order of the link (which is the service they point to). These numbers have determined a hard order to some extent, which also strengthens the dependencies between services.
The dependency between services Review Listing 1, we can see the Network Service (S10Network) will run before the NTPD service (S58NTPD). This is what we expect because NTPD services require networks to enable it to connect to a local time server. Unfortunately, this hard order cannot tell us enough information and will misunderstand. For example, in Listing 1 we can see that the LPD service (S60LPD) will run after the NetWork service. Although this is correct for those Linux systems connected to the network and using the network printer, this does not mean that the LPD service must run after the NetWork service when there is an InkJet printer on the backplane. In fact, in this case, start the LPD before starting the NetWork will be better. Take another example: Crond (Cron Daemon) service (S90crond in Listing 1) is also run after the Network is started. However, if you don't use the cron file of the remote machine file, you should let Crond start in NetWork.
Since the traditional way of launching service in Linux has been introduced, it is often tends to "safe first", let all important services start, and then start the rest.
So, although we can't launch all services in parallel, we can launch a service that does not rely on relationships between each other. When these unregistered services are started, we can start all dependency conditions (that is, those services dependent on services have started). The process is then repeated until all services are all launched.
This looks a complex problem, but fortunately, there is already a ready-made program that can be used to solve this problem. This program is not something else, it is Make.
Usually when compiling software, Make will provide the tight framework we need. All we have to do is telling Make what is the dependence between services; it can do all the difficult works of calculation cross-dependent, and use its freshly known tag -j, it can be used as many "homework" Synchronous run.
The service-dependent relationship is obtained, such as the previous I indirectly mentioned, the traditional Linux system does not explicitly represent the dependence between the service, so we have to do some difficult work now and have these dependencies. This may take a while, because you may not know what every service is doing, don't mention the relationship between services. However, if you don't have any of these work, this approach does not have any benefits for you. (As mentioned earlier, if this method has practical value, I hope that Linux distributors can use it and do these difficulties for us.)
Know your service If you run the command / sbin / chkconfig --List, you may have some service you don't know, so spend some time to figure it. A simple method is to read those who are at the beginning of the script that control the corresponding service. This way you can turn off the service that you don't have to use. Even if you need it, you can also know your system more.
Now let's do a simple example. We all know that NTPD services require a network, this shows that NTPD services rely on Network services. This depends on this dependency in the Make syntax:
NTPD: NetWork
We can also determine Netfs services (all NFS directories we need) depend on the network. On my system (you may not be the same), the AutoFS service (automatic mounting network file system) also rely on the Network service because I have automatically mount the remote file system (you mount the optical drive or soft drive). Our "dependent table" is now this: NTPD: NetWorkNetfs: NetworkAutofs: NetWork
This doesn't look, but do you know what this means? This means that once the NetWork service is started, we can launch NTPD, Netfs, and Autofs in parallel.
As a specific example, assume that all services take 10 seconds to start. It takes 40 seconds to start NetWork, NTPD, NETFS, and AutoFS services with traditional service startup methods. This technology is used, only 20 seconds - 50% of the time saving.
Why is this this? Ok, please see that the NetWork service takes 10 seconds, but because the RC script is running at runtime, the remaining three services can be started at the same time, so the three service combined startup time is 10 seconds.
In fact, the start-up time required for most services may not be 10 seconds, but since each service is to do some completely different things, the time that starts to start them will be considerably.
Sample Implementation I have an example of using the above technique in the compressed file I provide in the reference part. Including a modified RC script for calling the Make command, as well as the sample gnu makefile file, is Runlevel.MK, Start5.mk and Stop5.mk. Makefile file runlevel.mk is a control program, start5.mk file, and STOP5.mk files are the service dependent description files when running and stopping service when running level 5.
Note that the startup and stop the Makefile file is not a full service dependence list, but is just an example. At the same time, it should also be noted that if you don't modify these files, it is almost impossible to succeed, because your service list may be different from me.
Conclusion (and some supplementary description) I propose a method for speeding up Linux machine boot speed. This method allows the system to start the sequential service in the order in the start of the service in parallel, rather than the traditional serial mode to implement the boot acceleration. This approach is theoretically no problem, and can be implemented using existing system tools.
The efficiency of this method depends on the number of services that need to be activated and the time required for each service startup. Parallel feasibility depends mainly on the dependencies between the service. For some systems, using this method may only have a small improvement, but for other systems, it may significantly affect the boot speed. It can be understood that each system has a different set of services being activated, and each service requires different times to start. Again again, to use this method, you need to determine the dependencies between your specific system services.
Supplementary description:
Some service procedures are just running a program in the background, and they will withdraw itself (that is, the service is over, but the real work is still in the background). This illustrates a fact that traditional systems are imperfect, which is attempted to reduce some clock cycles within the boundaries of existing frameworks. This method described herein will make the dependencies more explicitly, and do not need service writers to go to "spoof". This approach takes into account a more efficient framework outside of these service programs. When you want "interactive" to guide your system, the techniques mentioned here are no longer applicable, because you usually do this like this when the system is wrong; in this case, you may hope Serialize all services to find out the reason. However, modify the system's startup process, let the user choose to be in the system boot to select a serial way (allowed interactive service) or "parallel" service boot mode is easy to implement. This method may also need more in-depth considerations, because if traditional systems and new systems are provided to users, they will need to synchronize how two groups about how to start information (ordered RC.D / link files and running Level Make file). A better solution is that the Linux release can automatically generate a link file from the Makefile file, because the makefile file records more information about the service than the link file. This system may not be suitable for a dedicated server, because when a service is wrong, the administrator hopes that you can see this error immediately in the console when an error occurs. However, for ordinary end users, parallelization methods can significantly speed up the boot speed under the premise that the user is allowed to see if there is a problem. Interestingly, although this method I propose is not "similar Linux" from traditional views, but the Linux Basic Standard (LSB, Linux Standards Base) does not specify the operation order of the init.d script, so this The method is likely to be adopted by the Linux publisher, and it is still in line with the LSB. This is a good news for users, as mentioned earlier, because the publisher can calculate the dependencies between all packages. There is a way to be more bold, that is, "WAIT" in the / etc / inittab file is modified to "On". This allows users to log in before the service starts complete. However, this has exceeded the scope of this article. For more detailed information, please check with Man inittab, and remember that uml is your good helper. Reference
Download a compressed package of a sample implemented with the methods discussed herein. The official GNU Make homepage has a Make program download and opportunity links. The goal of the LSB (Linux Standard Base is to define the configuration of the Linux system. The LSB public documentation contains documents about "System Initialization". UML (User Mode Linux) Home has a link to download and its document links, using UML you can start a "virtual" Linux system on a real Linux system. If you need a full guide to help you start using UML, see Tutorial "User Mode Linux Introduction" (DeveloperWorks, Jan 2003). The developerWorks Linux area has more resources prepared for Linux developers.
