Easy, safe, distributed network backup
Level: Intermediate
Carlos Justiniano (
Carlos.justiniano@ecuityinc.com)
Software designer, ECUITY INC.
The loss of important data may mean fatal damage. Despite this, there is still countless professionals ignore the backup of their data. Although the reasons may vary, the most common explanation is that the implementation routine backup is indeed cumbersome. Since machines are good at completing ordinary and repetitive tasks, the automated backup process is the key to reducing the inner boring and people and people's relapse tendency.
If you use Linux, you can already use extremely powerful tools that create custom backup solutions. The solutions in this article allow you to use an open source tool that comes with almost every Linux release to perform network backups from simple to higher and secure networks.
Simple backup
This article describes a step by step, as long as you follow the basic steps, this method is very intuitive.
Before studying more advanced distributed backup solutions, let's take a simple and powerful archive mechanism. Let's analyze a convenient script called ARC, which allows us to create a backup snapshot in the Linux shell prompt.
Listing 1. ARC shell script
#! / bin / sh tar CZVF $ 1. $ (DATE % Y% M% D% -H% M% s) .TGZ $ 1 EXIT $?
The ARC script receives a separate file or directory name as a parameter, creates a compressed archive file, and embed the current date into the name of the generated archive file. For example, if you have a directory named Beoserver, you can call the Arc script to pass the Beisterver directory name to create a compressed archive file, such as: Beoserver.20040321-014844.tgz
Use the date command to embed a date and timestamp to help you organize the archive file. The date of the date is the year, month, day, hour, minute, second - although there are more than the use of the spine. View the man name of the data command to find out other options. In addition, in Listing 1, we pass the -v (Verbose) option to the TAR. This makes the TAR show that it is archiving files. If you prefer to back up silently, remove this -V option.
Listing 2. Archive Beoserver directory
$ ls arc beoserver $ ./arc beoserver beoserver / beoServer / Bookl.dat beoserver / beoserver_ab_off beoserver / beoserver_ab_on $ ls arc beoserver beoserver.20040321-014844.tgz
Advanced backup
This simple backup is practical; however, it still contains a process of manually backed up. The best experience in the industry usually recommends backup data to multiple media and backs up to separate different geographic locations. The central idea is to avoid relying on any separate storage medium or separate location.
In the next example we will respond to this challenge, we will analyze a distributed network as shown in Figure 1, which shows system management of two remote servers and an offline storage server.
Figure 1. Distributed network
The backup files on the server # 1 and # 2 will be securely transmitted to the offline storage server, and the entire distributed backup process will be carried out regularly without artificial interference. We will use a set of standard tools (Open Safety Shell Tool Suite (OpenSSH), and tape archive (tar) and CRON task schedule service. All of our plans is to use cron to schedule, use the shell program and TAR applications to complete the backup process, using the OpenSSH Secure Shell (SSH) encryption to perform remote access, authentication, and secure shell copy (SCP) to automate file transfer. Be sure to get another manual for each tool. Secure remote access using public key / private key
In the context of digital security, the key (key) refers to a data piece used to encrypt or decrypt other data pieces. Interesting to public key private key is, using public key encrypted data, only the corresponding private key can be decrypted. You can freely release a public key so that others can encrypt your messages sent to you. One of the reasons why public key / private key mode thoroughly changes digital security is that senders and recipients do not have to share a universal password. In addition to other contributions, public key / private key encryption uses e-commerce and other secure transmissions. In this article, we will create and use public key and private key to create a very secure distributed backup solution.
Each machine to make a backup process must run the OpenSSH Secure Shell Service (SSHD), while the 22-port can be accessed through any internal firewall. If you access a remote server, it is very likely that you are using a secure shell.
Our goal will be that you can safely access the machine without manual delivery. Some people think the easiest way is to set up a password access: don't do this. This is not safe. Unexpectedly, the method we will use this article may take about an hour of time, establish a system that is equally convenient to use the "None Password" account - but it is very safe.
Let us first ensure that OpenSsh is already installed, then check its version number. When completing this article, the latest version of OpenSSH is released on February 24, 2004 3.8. You should consider using a newer and stable release, at least the version should be more than version 2.x. Access the OpenSSH Security web page to get details about the defects of the specific older version (see links in the references later in this article). So far, OpenSSH is very stable and has proved many of the defects reported by other SSH tools.
In the Shell prompt, enter SSH and give important V options to check the version number:
$ ssh -v
OpenSSH_3.5P1, SSH protocols 1.5 / 2.0, openssl 0x0090701F
If the version number returned by SS is greater than 2.x, the machine is in a relatively good state. In any case, it is recommended that all your software uses the latest stable version, which is especially important for security related software.
Our first step is to log in to an offline storage server machine (see Figure 1) using an account that will have privileged access servers 1 and 2.
$ ssh accountname@somedomain.com
After logging in to the offline storage server, use the SSH-KEYGEN program and give the -t DSA option to create a public key / key pair. The -t option is required to specify the type of key we have to generate. We will use Digital Signature Algorithm, DSA, which allows us to use the updated SSH2 protocol. See the SSH-KEYGEN manual to get more details. During the execution of SSH-KEYGEN, you will be prompted to enter the location of the SSH key storage before asking your password (passphrase). When you ask where to store keys, you only need to press Enter key, and then the SSH-KEYGEN program will create a hidden directory named .ssh (if it does not exist), and two files, a public key file and one Private key file.
A interesting feature of SSH-KEYGEN is that when prompted to enter a password, it allows you to simply press the back key. If you have not given an export, SSH-KEYGEN will generate no encrypted key! If you think, this is not a good idea. When the password is required, make sure to enter a sufficiently long character message, preferably a mixed character, not just a simple password string.
Listing 3. Always choose a good password
[OFFSITE]: $ ssh-keygen -t dsa generating public / private dsa key pair. Enter file in which to save the key (/Home/accountname/.ssh/id_dsa): Enter Passphrase: (ENTER): (ENTER passphrase) Enter same passphrase again: (enter passphrase) Your identification has been saved in /home/accountname/.ssh/id_dsa Your public key has been saved in /home/accountname/.ssh/id_dsa.pub The key fingerprint is.. : 7E: 5E: B2: F2: D4: 54: 58: 6A: FA: 6B: 52: 9C: DA: A8: 53: 1B Accountname @ Offsite
Since SSH-KEYGEN is generated. SSH directory is a hidden "dot" directory, so you have to pass a -A option to the ls command to view the newly created directory:
[OFFSITE] $ ls -a
.. .bash_logout .bash_profile .bashrc .emacs .gtkrc .ssh
Enter hidden .ssh directory and list their content:
[OFFSITE] $ cd .ssh
[OFFSITE] $ ls -lrt
ID_DSA ID_DSA.PUB
Now, in a hidden .ssh directory, we already have a private key (id_dsa) and a public key (id_dsa.pub). You can analyze the contents of each key file using a text editing tool such as VI or Emacs or simply use the LESS or CAT command. You will see that the content composed of mixed characters has passed Base64 encoding.
Then we need to copy the public key and install it to the servers 1 and 2. Do not use FTP. More reasonable is to use a secure copy program to transfer the public key to each remote machine.
Listing 4. Install the public key to the remote server
[OFFSite] $ scp .ssh / id_dsa.pub accountname@server1.com: offsite.pub accountname@server1.com's password: (Enter Password, not new passphrase!) ID_DSA.PUB 100% | ******** ******************** | 614 00:00 [Offsite] $ scp .ssh / id_dsa.pub accountname@server2.com: offsite.pub accountname @ Server2. COM's Password: (Enter Password, Not New Passphrase!) ID_DSA.PUB 100% | ************************************* | 614 00 : 00 After the new public key is installed, we can log in to each machine using the password specified when creating a private key and the public key. Now, log in to each machine and attach the contents of the OFFSITE.PUB file to a file called Authorized_Keys, which is stored in the .ssh directory of each remote machine. We can use a text editor or simply use the CAT command to attach the contents of the OFFSITE.PUB file to the Authorized_Keys file:
Listing 5. Add offsite.pub to the authorized key list
[OFFSITE] $ ssh accountname@server1.com attountname@server1.com's password: (Enter Password, Not new passphrase!) [Server1] $ Cat Offsite.pub >> ./ssh/authorized_keys
The next step is to consider some additional security. First, we modify .SSH access rights so that only owners have permissions to read, write, and execute. Then, we ensure that the Authorized_Keys file can only be accessed by the owner. Finally, delete the previously uploaded OFFSITE.PUB key file, because no need it again. Make sure to set proper access is important because the OpenSSH server may refuse to use a key with unsecure access.
Listing 6. Modify permissions using CHMOD
[Server1] $ chmod 700 .ssh [server1] $ chmod 600 ./ssh/authorized_keys [server1] $ rm offsite.pub [server1] $ exit
After the same procedure is completed on the server 2, we can return to the offline storage machine to test the access of the new password type. On the offline server you can enter the following:
[OFFSITE] $ ssh -v accountname@server1.com
When verifying your account, you can now use the new password instead of the original password to access the remote server, use the -v or verbose tag option to display debugging information. In addition to how the authentication process is observed on a high level, you can also show important information you can not get in other ways. You may not need to specify -v tags in later connections; however it is quite useful when testing.
Use SSH-Agent Automation Machine Access
The SSH-Agent program is like a doorman, which provides access to the security key as needed. After SSH-Agent is started, it will run in the background and can be used by other OpenSSH applications such as SSH and SCP programs. This allows the SSH program to request a key that has been decrypted instead of asking you the security password of the private key at a time you need. Let's take a closer study of SSH -Agent. SSH-Agent runs when it outputs the shell command:
Listing 7. SSH-Agent application
[OFFSite] $ ssh-agent ssh_auth_sock = / tmp / ssh-xx1o24ls / agent.14179; export ssh_auth_sock; ssh_agent_pid = 14180; export ssh_agent_pid; Echo Agent PID 14180;
We can use the shell's eval command to let the shell execute the output command displayed by SSH-Agent:
[OFFSite] $ Eval `SSH-Agent`
Agent PID 14198
The EVAL command tells the Shell to evaluate the command generated by the SSH-Agent program. Make sure you specify the reverse quotation mark (`) instead of single quotes! After execution, the EVAL `SSH-Agent` statement will return the process identifier of the agent. After the scene, the SSH_AUTH_SOCK and SSH_AGENT_PID shell variables have been exported. You can display them in the Shell console to see their values:
[OFFSITE] $ ECHO $ SSH_AUTH_SOCK
/TMP/SSH-XX7BHIWQ/Agent.14197
$ SSH_AUTH_SOCK (SSH Authentication Socket Abbreviation) is a location of a local socket, and the application can communicate with SSH-Agent through it. Add an Eval `SSH-Agent` statement to your ~ / .bash_profile file to make sure ssh_auth_sock and ssh_agent_pid are always registered.
SSH-Agent has now become a background process that can be viewed with TOP and PS commands.
Now we can already share our password using SSH-Agent. To do this, we must use a program named SSH-AddD, which adds our password (send) to the running SSH-Agent program.
Listing 8. SSH-ADD for free logging in
[OFFSite] $ ssh-add enter passphrase for /Home/accountname/.ssh/id_dsa: (enter passphrase) Identity Added: /Home/accountname/.ssh/id_dsa (/Home/accountname/.ssh/ID_DSA)
Now, when we accesses Server1, you will not be prompted to enter the password again:
[OFFSITE] $ ssh accountname@server1.com
[Server1] $ EXIT
If you don't believe, try to remove (kill -9) ssh-agent process and reconnect Server1. This time, you will notice that Server1 will ask the password of the private key stored in the id_dsa in the .ssh directory:
[OFFSITE] $ KILL -9 $ SSH_AGENT_PID
[OFFSITE] $ ssh accountname@server1.comenter passphrase for key '/Home/accountname/.ssh/id_dsa':
Simplify key access using Keychain
To so far, we have learned several OpenSSH programs (SSH, SCP, SSH-Agent, and SSH-ADD), and we have created and installed private key and public key to enable a secure and automated login process. You may have realized that most settings work only once. For example, create a key, install the key, and perform SSH-Agent through .bash_profile, the process of executing SSH-Agent is only taken at each machine. That is a good news.
Not ideal news is that each time we log in to offline machines, you must call SSH-ADD, and SSH-Agent is not directly compatible with the cron scheduled process we will use to automate backup. The Cron process cannot communicate with SSH-Agent is that the cron job is performed as a child process as a cron so that they do not inherit $ ssh_auth_sock shell variable.
Fortunately, there is a solution to eliminate the limitations of SSH-Agent and SSH-ADD, but also allow us to use CRON to move a variety of processes that need to be safely password access to other machines. In the three developerWorks series of Articles in 2001, OpenSSH Key Management (see Resources for Link), Daniel Robbins introduced a shell script called Keychain, which is a front end of SSH-Add and SSH-Agent Simplify the entire password process. Over time, Keychain scripts have experienced a lot of improvements, now maintained by Aron Griffis, its latest 2.3.2-1 released release from June 17, 2004.
Keychain Shell scripts are too long to be listed in this article, because many erroneous detection, rich documents and very large cross-platform code are included in the well-written scripts. However, KeyChain can be easily downloaded from the Web site (see Resources to get link).
After downloading and installing Keychain, it is very simple. Just log in to each machine and add the following two lines to each .bash_profile file:
Keychain ID_DSA
. ~ / .keychain / $ hostname-sh
When you register in each machine for the first time, KeyChain will ask you the password. However, unless the machine is restarted, the keychain will not ask you to re-enter the password when logging in later. Best, the cron task can now use the openssh command to securely access remote machines without using passwords to interactively. Better security and easier use, now we have already obtained.
Listing 9. Initialization on each machine
KeyChain 2.3.2; http://www.gentoo.org/projects/keychain Copyright 2002-2004 Gentoo Technologies, Inc .; Distributed under the GPL * Initializing /home/accountname/.keychain/localhost.localdomain-sh file .. . * Initializing /Home/AccountName/.keychain/localhost.localdomain-csh file ... * Starting ssh-agent * adding 1 key (s) ... Enter Passphrase for /Home/accountname/.ssh/id_dsa: (Enter) Passphrase) script backup process
Our next task is to create a shell script that performs the necessary backup process. The goal is to execute the full database backup of servers 1 and 2. In our example, each server runs the MySQL database server, we use the mysqldump command line tool to export some database tables into a SQL input file.
Listing 10. Server 1 DBBackup.sh shell script
! # / Bin / sh # change into the backup_agent directory where data files are stored cd / home / backup_agent # use mysqldump utility to export the sites database tables mysqldump -u sitedb -pG0oDP @ sswrd --add-drop-table sitedb. - -Tables TBL_CCODE TBL_MACHINE TBL_SESSION TBL_STATS> Userdb.sql # Compress and Archive Tar CZF Userdb.tgz Userdb.sql
On Server 2, we will set a similar script to back up the unique form given in the site database. Each script is marked as executable by the following steps:
[Server1]: $ chmod x dbbackup.sh
After setting DBBackup.sh on both servers 1 and 2, we return to offline data servers, where we will create a shell script to call each remote dbbackup.sh script and then transfer compressed (.tgz) data files.
Listing 11. With backup_remote_servers.sh shell scripts on offline data servers
#! / Bin / sh # use ssh to remotely execute the dbbackup.sh script on server 1 / usr / bin / ssh backup_agent@server1.com "/home/backup_agent/dbbackup.sh" # use scp to securely copy the newly archived Userdb.tgz file # from server 1. Note the use of the date command to timestamp # the file on the offsite data server. / usr / bin / scp backup_agent@server1.com: /Home/backup_agent/Userdb.tgz / Home / Backups / UserDB - $ (DATE % Y% M% D-% h% m% s) .TGZ # execute dbbackup.sh on server 2 / usr / bin / ssh backup_agent@server2.com "/ Home / Backup_Agent / DbBackup .sh "# Use scp to offsite server. / usr / bin / scp backup_agent@server2.com: /Home/backup_agent/transdb.tgz / home / backups / transdb - $ (DATE % Y% M % D-% h% m% s) .TGZBACKUP_REMOTE_SERVERS.SH shell script uses the ssh command to perform scripts on the remote server. Since we have set the passwordless access, the SSH command can perform commands remotely on servers 1 and 2 through offline servers. Thanks to Keychain, the entire certification process can now be done automatically.
Dispatch
Our next step is also the last step, which is the execution of the scheduled backup_remote_servers.sh shell script on the offline data storage server. We will add two entries to the CRON schedule server to request a backup script twice a day, 3: 34 is executed once, 8: 34 is executed again. Use the Edit (-e) option on an offline server to call the crontab program.
[OFFSITE]: $ crontab -e
CRONTAB calls the default editor specified by the Visual or Editor shell environment variable. Then, enter two entries and save and close the file.
Listing 12. Crontab entry on offline servers
34 3 * * * /Home/backups/remote_db_backup.sh 34 20 * * /Home/backups/remote_db_backup.sh
A Crontab row includes two main parts, a schedule section, and a backpoint of the back. The schedule is divided into multiple domains to specify when a command should be performed:
Listing 13. CRONTAB format
-- Minute | ----- Hour | | ------ Day of the Month | | | | | | | ---- Day of The Week | | | | | | | | | | | | 34 3 * * * /HOME/backups/remote_db_backup.sh Test Your Backup
You should check the backup to ensure that the program is properly done. Automatic programs can be avoided by cumbersome work, but it will never be lazy. If your data is worth backup, it also deserves a sample check.
Consider adding a CRON job to remind you to check your backup at least every month. In addition, often modifying the security key is also a good idea, and you can also schedule a cron job to remind you to do this.
Another security
To achieve higher security, consider installing and configuring an intrusion detection system (NTRusion Detection System, IDS), such as Snort. It is foreseeable that ids will notify you when the invasion is happening or in the near future. After IDS is in place, you will add other levels of security, such as digital signing and encryption for your backup.
GNU Privacy Guard (GNUPG), OpenSSSL and NCRYPT and other popular open source tools can support archive files via shell scripts, but not recommended to do this without IDS (see) More information about SNORT).
Conclude
This article shows you how to make your script execute on remote servers and how to perform secure automatic file transfer. I hope that you can get inspiration to consider protecting your own important data and use OpenSSH and Snort to build a new solution.
Reference
- in the official
OpenSSH homepage and
OpenSSH Security Page You will find download, documentation, and more.
- Read the excellent IBM DeveloperWorks article consisting of Daniel Robbins
OpenSSH Key Management (DeveloperWorks, 2001) and downloaded him
Keychain app.
- To learn from SSH, Carlos recommended O'Reilly
SSH, The Secure Shell: The Definitive Guide (O'Reilly & Associates, 2001).
-
The Snort Intrusion Detection System (IDS) is an open source code for detecting and reporting unauthorized access or suspicious behavior. If you are planning to automate the signature and encryption of the archive file, you must use an IDS.
- you can use it
Gnu Privacy Guard (GNUPG),
OpenSSL and
Ncrypt Sign and encrypt the archive backup file in the shell script.
- If you have not used them, please refer to
TCP Wrappers and
Tips in xinetd.
Perl enthusiasts will also be interested in reading "
Using Perl Automation Unix System Management (DeveloperWorks, 2001), "
Introduction of CFENGINE "(developerWorks, 2002) and" application configuration with Perl "(developerWorks, 2000), they are TED Zlatanov's work.
- DeveloperWorks Articles
Windows to Linux Tour: Part 8. Backup and Recovery (developerWorks, 2003) gives a trick about backup strategies.
-
IBM's Tivoli Storage Manager for Linux can also automate reliable backup, archive, and centralized data management on Linux computers and servers based on custom schedules. In addition,
It provides a unified environment and interface for users management, access control, and network monitoring - more features - products. - pass
The IBM DeveloperWorks Tivoli area has deeply learned Tivoli solutions.
- in
The developerWorks Linux zone can find more reference materials for Linux developers.
- Order in the Developer Bookstore Linux area
Discount Linux books.
- pass
DeveloperWorks Subscription uses the latest IBM tools and middleware to develop and test your Linux application: you can get IBM software from WebSphere, DB2, Lotus, Rational, and Tivoli, and a license that can be used within 12 months All spending is lower than what you imagine.
- Self-developerWorks
Speed-start your Linux app Download no runs on Linux, the free test version to selected developerWorks Subscription products, including WebSphere Studio Site Developer, WebSphere SDK for Web services, WebSphere Application Server, DB2 Universal Database Personal Developers Edition, Tivoli Access Manager and Lotus Domino Server. To get started faster, see how-to articles and technical support for each product.
About author
Carlos Justiniano is a software designer of ECUITY, INC. Aspects he interested include communication and distributed calculations. Carlos wrote an article for many technical magazines. He is still the founder and designer of Linux-based Chessbrain project, which has won 2005 Guinness World Record related to distributed computing. You can pass
Carlos.justiniano@ecuityinc.com Contact him.
Full article:
IBM DeveloperWorks China website
