President of Daniel Robbins and CEO, Gentoo Technologies, Inc.
Daniel Robbins detail the Gentoo Linux Ebuild system in detail in its last Bash instance article, which shows an excellent example of Bash capabilities. Step step by step, he shows you how to implement the Ebuild system and touches a lot of convenient Bash technology and design strategies. At the end of this article, you will have a good master of technology that is fully based on BASH-based applications and begins to encode our own automatic build system.
I have been looking forward to this third article, but it is also a Bash instance article, because since I have explained the Bash programming foundation in Articles 1 and 2, I can focus on Bash application development and programming. Such more advanced themes. In this article, it will give you a lot of actual, real-world BASH development experience through I spent a lot of time, Gentoo Linux eBuild system.
I am a chief designer of Gentoo Linux (currently a next-generation Linux OS). One of my main responsibilities is to make sure all binary packages (similar to rpm) are created correctly and work together. As you may know, the standard Linux system is not composed of a unified source tree (like BSD), which is actually composed of more than 25 core packages. This includes:
Package Description Linux actual kernel Util-Linux related to Linux related Miscellaneous program set E2FSProgs and EXT2 file system related utility collection GLIBCGNU C library
Each package is located in the respective TAR compression package and maintained by different independent developers or development groups. To create a distribution, you must download, compile and pack each package separately. Each time you have to fix, upgrade, or improve the package, you must repeat and pack steps (and, the package is indeed updated very fast). In order to help eliminate the repeating steps involved in the creation and update package, I created the eBuild system, which is almost written with BASH. In order to increase your BASH knowledge, I will gradually present how to implement the unpack and compilation section of the eBuild system. When explaining each step, it will also be discussed why some design decisions are made. At the end of this article, you will not only master large Bash programming projects, but also achieve a large part of the full automatic build system.
Why choose Bash? Bash is the basic component of the Gentoo Linux eBuild system. There are several reasons for choosing it as the main language for ebuild. First, its grammar is not complicated and is familiar to people, which is particularly suitable for calling external programs. The automatic build system is "glued code" that automatically calls external programs, and BASH is ideal for this type of application. Second, the support of the BASH to the function allows the eBuild system to use modular and easy to understand code. Third, the eBuild system utilizes the support of the BASH to environment variables, allowing the package maintenance personnel and developers to make it easy online configuration.
Constructing Process Review Whatfore I take on what the compilation and installation packages are involved before discussing the Ebuild system. For example, let's take a look at the "SED" package, this as a standard GNU text stream editing utility as all of the Linux versions. First, download the source code TAR compression package (SED-3.02.tar.gz) (see Resources). We will store this file in / usr / src / distfiles, will use the environment variable "$ distdir" to reference this directory. "$ Distdir" is the directory where all original source code TAR compression packages are a large source code base. The next step is to create a temporary directory called "Work", which stores the source code that has been decompressed. This directory will be referenced later using the "$ WORKDIR" environment variable. To do this, enter the directory with write permissions, then enter:
Unzip the SED to the temporary directory
$ MKDIR WORK
$ CD WORK
$ tar xzf /usr/src/distfiles/sed-3.02.tar.gz
Then, decompress the TAR compression package, create a directory containing all source code, named SED-3.02. The SED-3.02 directory will be referenced using the environment variable "$ srcdir" in the future. To compile the program, enter:
Unzip the SED to the temporary directory
$ CD SED-3.02
$ ./configure --prefix = / usr
(AutoConf generates an appropriate make file, this takes some time)
$ Make
(Take a little time from the source code package, it also takes a little more time)
Because only the unpacking and compilation steps are described herein, the "Make Install" step will be skipped. If you want to write a Bash script to perform all these steps, the code may be similar to:
Sample Bash script to perform an unpack / compile process
#! / usr / bin / env bash
IF [-d Work]
THEN
# REMOVE OLD WORK DIRECTORY IF IT EXISTS
RM-RF Work
Fi
Mkdir Work
CD Work
TAR XZF /USR/SRC / DistFiles/sed-3.02.tar.gz
CD SED-3.02
./configure --prefix = / usr
Make
Although this automatic compilation script can be used, it is not very flexible. Basically, the Bash script is only included in the command line entered. Although this solution can be used, it is best to make a suitable script that can quickly unread and compile any packages only by changing a few lines. In this way, the package maintenance personnel will be reduced by adding the new package to the release. Let us try to use many different environment variables to complete, make the build script more applicable:
New, more common script
#! / usr / bin / env bash
# P is the package name
P = SED-3.02
# A is the archive name
A = $ {p} .tar.gz
Export OrigDir = `pwd`
Export workdir = $ {OrigDir} / work
Export srcdir = $ {Workdir} / $ {p}
IF [-z "$ distdir"]
THEN
# set distdir to / usr / src / distfiles if not already set
Distdir = / usr / src / distfiles
Fi
Export DistDirif [-d $ {workdir}]
THEN
# REMOVE OLD WORK DIRECTORY IF IT EXISTS
RM-RF $ {Workdir}
Fi
Mkdir $ {workdir}
CD $ {Workdir}
TAR XZF $ {Distdir} / $ {a}
CD $ {srcdir}
./configure --prefix = / usr
Make
A lot of environment variables have been added to the code, but it basically does also perform the same function. However, if you want to compile any standard GNU based on AutoConf-based source code TAR compression package, simply copy the file to a new file (with the appropriate name to reflect the new package it compiled), then "$ A" and "$ p" values can be changed to a new value. All other environment variables are automatically adjusted to be properly set, and the scripts work according to the expected. Although this is very convenient, the code has improved. This code is much longer than the "Transcript" script we have started. Since one of the goals of any programming items is to reduce user complexity, it is best to shorten the code, or at least better organize code. You can do this with a clever way - dismantle the code into two separate files. The file is stored as "SED-3.02.ebuild":
Sed-3.02.ebuild
#the sed ebuild file - Very Simple!
P = SED-3.02
A = $ {p} .tar.gz
The first file is not important, only those environment variables that must be configured in each package. Below is the second file, which contains the main part of the operation. Save it as "ebuild" and make it an executable:
ebuild script
#! / usr / bin / env bash
IF [$ # -ne 1]
THEN
echo "One Argument Expected."
EXIT 1
Fi
IF [-e "$ 1"]
THEN
SOURCE $ 1
Else
Echo "eBuild File $ 1 Not found."
EXIT 1
Fi
Export OrigDir = `pwd`
Export workdir = $ {OrigDir} / work
Export srcdir = $ {Workdir} / $ {p}
IF [-z "$ distdir"]
THEN
# set distdir to / usr / src / distfiles if not already set
Distdir = / usr / src / distfiles
Fi
Export distdir
IF [-d $ {workdir}]
THEN
# REMOVE OLD WORK DIRECTORY IF IT EXISTS
RM-RF $ {Workdir}
Fi
Mkdir $ {workdir}
CD $ {Workdir}
TAR XZF $ {Distdir} / $ {a}
CD $ {srcdir}
./configure --prefix = / usr
Make
Since the building system has been dismantled into two files, I bet, you must think about it works. Basically, to compile SED, enter:
$ ./ebuild sed-3.02.ebuild
When "eBuild" is executed, it first tries "Source" variable "$ 1". What does it mean? Remember the previous article: "$ 1" is the first command line from variable - here is "Sed-3.02.ebuild". In Bash, the "Source" command reads the Bash statement from the file, then executes them, just like they directly appear in the file where the "source" command is located. Therefore, "Source $ {1}" causes the "ebuild" script to execute "$ p" and "$ a" commands in Sed-3.02.ebuild. This design changes are indeed, because if you want to compile another program, not the SED, you can simply create a new .ebuild file, then pass it as an argument to the "eBuild" script. In this way, the .ebuild file is ultimately simple, there is one place in the complex operation section of the Ebuild system, that is, "eBuild" scripts. In this way, you can upgrade or enhance the eBuild system with you to edit the "eBuild" script, while the implementation details are left outside the eBuild file. Here is a Gzip sample eBuild file: gzip-1.2.4a.ebuild
#nother really simple ebuild script!
P = gzip-1.2.4a
A = $ {p} .tar.gz
Adding functionality, we are making progress. However, I also want to add some additional functionality. I hope the eBuild script will receive a command line variable: "Compile", "unpack" or "all". This command line is telling the eBuild script which step is to perform the build process. In this way, you can tell EBUILD to unpack file, but do not compile (to view the source code file before starting compilation). To do this, a CASE statement will be added, which will test "$ 2" and then perform different operations according to its value. code show as below:
Ebuild, modified this 2
#! / usr / bin / env bash
IF [$ # -ne 2]
THEN
Echo "Please Specify Two Args - .ebuild File and Unpack, Compile or ALL"
EXIT 1
Fi
IF [-z "$ distdir"]
THEN
# set distdir to / usr / src / distfiles if not already set
Distdir = / usr / src / distfiles
Fi
Export distdir
ebuild_unpack () {
# Make Sure We're in The Right Directory
CD $ {Origdir}
IF [-d $ {workdir}]
THEN
RM-RF $ {Workdir}
Fi
Mkdir $ {workdir}
CD $ {Workdir}
IF [! -e $ {DISTDIR} / $ {a}]
THEN
Echo "$ {DISTDIR} / $} does not exist. please download first."
EXIT 1
Fi
TAR XZF $ {DistDir} / $ {a} echo "Unpacked $ {DISTDIR} / $}."
#Source is now Correctly Unpacked
}
ebuild_compile () {
# Make Sure We're in The Right Directory
CD $ {srcdir}
IF [! -d "$ {srcdir}"]
THEN
Echo "$ {srcdir} does not exist - please unpack first."
EXIT 1
Fi
./configure --prefix = / usr
Make
}
Export OrigDir = `pwd`
Export workdir = $ {OrigDir} / work
IF [-e "$ 1"]
THEN
SOURCE $ 1
Else
Echo "eBuild File $ 1 Not found."
EXIT 1
Fi
Export srcdir = $ {Workdir} / $ {p}
Case "$ {2}" in
Unpack)
ebuild_unpack
;
Compile
ebuild_compile
;
ALL)
ebuild_unpack
ebuild_compile
;
*)
Echo "Please Specify Unpack, Compile or All as the second arg"
EXIT 1
;
ESAC
I have made a lot of changes, let's review it. First, the compilation and unpacking step is placed in their respective functions, and its function names are ebuild_compile () and ebuild_unpack (). This is a good step because the code is becoming more complicated, and the new function provides a certain module, making the code more striking. At the first line of each function, the explicit "CD" to the desired directory, because, as the code becomes more modular rather than linearization, the function is executed in the wrong current working directory. The possibility also becomes large. The "CD" command explicitly enables us to be in the correct position and prevents errors in the future - this is an important step, especially when deleting files in a function.
In addition, a useful check is added at the beginning of the eBuild_Compile () function. Now, it checks to make sure that "$ srcdir" exists, if there is no existence, print an error message telling the user first unpacking the file then exiting the file. If you prefer, you can change this behavior so that in the case where "$ srcdir" does not exist, the eBuild script will automatically unpack the source code file. You can replace EBUILD_COMPILE () with the following code:
New code on eBuild_Compile ()
ebuild_compile () {
# Make Sure We're in The Right Directory
IF [! -d "$ {srcdir}"]
THEN
ebuild_unpack
Fi
CD $ {srcdir}
./configure --prefix = / usr
Make
}
One of the most obvious changes in the second edition of the eBuild scripting is the new CASE statement at the end of the code. This CASE statement just checks the second command line independent variable and then performs correct operation based on its value. If you are now:
$ EBUILD SED-3.02.EBUILD
A error message will be obtained. Now I need to tell EBUILD, as shown below:
$ EBUILD SED-3.02.ebuild Unpack
or
$ EBUILD SED-3.02.ebuild Compile
or
$ EBUILD SED-3.02.ebuild All
If a second command line independent variable outside of the above is provided, an error message (* clause) will be obtained, and then the program exits.
Make code modularity since the code is very advanced and practical, you might want to create several more advanced ebuild scripts to unpack and compile the favorite programs. If this is done sooner or later, there will be some source code that does not use autoconf ("./configure"), or may encounter other scripts that use non-standard compilation processes. Need to make some changes to the Ebuild system to adapt to these programs. But before doing, it is best to think about how to do it.
One of the wishes of "./configure --prefix = / usr; make" hardcoded to compile stage is: Most of the time it works correctly. However, it is also necessary to adapt the eBuild system to those source code that does not use AutoConf or normal Make files. To solve this problem, it is recommended that the eBuild script performs the following:
If there is a configuration script in "$ {srcdir}", then execute it as follows: ./ Configure --prefix = / usr Otherwise, skip this step. Perform the following command: make
Since ebuild is only actually presented in Configure, it can now automatically adapt to programs that do not use AutoConf but have standard Make files. However, what should I do if I have a simple "make" for some source code? There is a need for some specific code to handle these situations to override reasonable default. To do this, the eBuild_Compile () function will be converted into two functions. The first function (which can be used as a "parent" function) is still eBuild_Compile (). However, there will be a new function called User_Compile (), which only contains reasonable default operations:
Ebuild_compile () disassembled into two functions
User_compile () {
# we're already in $ {srcdir}
IF [-e configure]
THEN
#Run Configure Script if it exists
./configure --prefix = / usr
Fi
#Run Make
Make
}
ebuild_compile () {
IF [! -d "$ {srcdir}"]
THEN
Echo "$ {srcdir} does not exist - please unpack first."
EXIT 1
Fi
# Make Sure We're in The Right Directory
CD $ {srcdir}
User_compile
}
This may not be very obvious because it is now, but I will bear it. Although this code is almost the same as ebuild's previous version, it is now possible to do some could not do - User_Compile () can be overwritten in SED-3.02.ebuild. Therefore, if the default user_compile () does not satisfy the requirements, you can define a new one in the .ebuild file to include the commands necessary to compile the package. For example, there is an E2FSProgs-1.18 ebuild file that requires a slightly different "./configure" line: E2FSPROGS-1.18.EBUILD
#this ebuild file overrides the default user_compile ()
P = E2FSPROGS-1.18
A = $ {p} .tar.gz
User_compile () {
./configure --enable-Elf-shlibs
Make
}
Now, E2FSProgs will be compiled in the way we want. However, for most packages, any custom user_compile () function in the .ebuild file can be omitted, and the default user_compile () function can be used.
How do eBuild scripts know which user_compile () function wants? In fact, this is simple. In the eBuild script, define the default User_Compile () function before performing the E2FSProgs-1.18.ebuild file. If there is a user_compile () in E2FSPROGS-1.18.ebuild, it overwrite the default version of the previously defined. If not, use the default User_Compile () function.
This is a good tool, we have added a lot of flexibility without any complex code (if you don't need it). I don't talk here, but I should also modify eBuild_unpack () so that users can override the default unpacking process. This is very convenient if you want to do any patch, or if the file is included in multiple files. There is also a good idea to modify the unpacking code so that it can be defaulted to identify TAR compression packages compressed by bzip2.
The configuration file has so far, there have been a lot of inconvenient Bash technology, and now it is now. Typically, if the program is very convenient to have a configuration file in / etc. Fortunately, this is easy to use Bash. Simply create the following file, then save it to /etc/ebuild.conf:
/ECT/EBUILD.CONF
# /etc/ebuild.conf: Set System-Wide Ebuild Options in this file
# Makeopts Are Options Passed To Make
Makeopts = "- J2"
In this example, only one configuration option is included, however, you can include more. One of BASH is: By executing the file, you can analyze it. This design trick can be used in most interpreted languages. After performing /etc/ebuild.conf, define "$ makeopts" in the eBuild script. It will utilize it allows the user to pass options to the MAKE. Typically, this option will be used to allow users to tell EBUILD to perform parallel Make.
What is parallel Make? In order to improve the compilation speed of the multiprocessor system, make supports parallel compilers. This means that make simultaneously compiles the user's specified number of source files (in order to use the extra processor in the multiprocessor system), not only one source file. Enable Parallel Make by passing the -J # option to the MAKE, as shown below: make -j4 make = "make -j4" This line code indicates that Make compiled four programs simultaneously. Make = "Make -J4" argument tells make, passing the -J4 option to any subkey process started. Here is the final version of the eBuild program:
ebuild, final version
#! / usr / bin / env bash
IF [$ # -ne 2]
THEN
Echo "please specify ebuild file and unpack, compile or all"
EXIT 1
Fi
Source /etc/ebuild.conf
IF [-z "$ distdir"]
THEN
# set distdir to / usr / src / distfiles if not already set
Distdir = / usr / src / distfiles
Fi
Export distdir
ebuild_unpack () {
# Make Sure We're in The Right Directory
CD $ {Origdir}
IF [-d $ {workdir}]
THEN
RM-RF $ {Workdir}
Fi
Mkdir $ {workdir}
CD $ {Workdir}
IF [! -e $ {DISTDIR} / $ {a}]
THEN
Echo "$ {DISTDIR} / $} does not exist. please download first."
EXIT 1
Fi
TAR XZF $ {Distdir} / $ {a}
Echo "Unpacked $ {DistDir} / $ {a}."
#Source is now Correctly Unpacked
}
User_compile () {
# we're already in $ {srcdir}
IF [-e configure]
THEN
#Run Configure Script if it exists
./configure --prefix = / usr
Fi
#Run Make
Make $ makeopts make = "Make $ makeopts"
}
ebuild_compile () {
IF [! -d "$ {srcdir}"]
THEN
Echo "$ {srcdir} does not exist - please unpack first."
EXIT 1
Fi
# Make Sure We're in The Right Directory
CD $ {srcdir}
User_compile
}
Export OrigDir = `pwd`
Export workdir = $ {OrigDir} / work
IF [-e "$ 1"]
THEN
SOURCE $ 1
Else
Echo "eBuild File $ 1 Not found."
EXIT 1
Fi
Export srcdir = $ {Workdir} / $ {p}
Case "$ {2}" in
Unpack)
ebuild_unpack
;
Compile
eBuild_Compile ;;
ALL)
ebuild_unpack
ebuild_compile
;
*)
Echo "Please Specify Unpack, Compile or All as the second arg"
EXIT 1
;
ESAC
Note that /etc/ebuild.conf is executed in the beginning of the file. In addition, pay attention to use "$ makeopts" in the default user_compile () function. You may be thinking, do you use it? Fortunately, this is no problem because the variable extension occurs when User_Compile () is executed. When using user_compile (), /etc/ebuild.conf has been executed, and "$ makeopts" is also set to be correct.
Ending the text has already described a lot of Bash programming techniques, but only some fans from Bash capabilities. For example, Gentoo Linux eBuild products not only automatically disserted and compiled each package, but also:
If you do not find the source code in "$ distir", the Auto Download By using the MD5 message summary, verify that the source code is not damaged if the request is requested, the compiled application is installed to the file system being used, and records all installed files. In order to easily uninstall the package easily. If the compiled application is packaged into a TAR compression package (compressed in the form of you want) so that you can follow the other computer, or during the CD-based installation process (if you build a release CD) installation.
In addition, eBuild system products have several global configuration options, allowing users to specify options, such as what optimized in the compilation process, and whether to enable optional package support in those packets that support it (for example, GNOME and SLANG) ).
Obviously, the features that Bash can achieve far more than touch in this series of articles. Regarding this incredible tool, I hope that you have learned a lot and encourage you to use Bash to speed up and enhance the development project.
Reference
Download the source code TAR compression package (SED-3.02.tar.gz) from ftp://ftp.gnu.org/pub/gnu/sed. Read "Bash Instances: Part 1" on DeveloperWorks. Read "Bash Instances: Part 2" on DeveloperWorks. Visit the Gentoo Project home page. Visit the GNU's Bash home page. View Bash Online Reference Manual.
Transfer from: IBM DeveloperWorks China website
