diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a34718..e6a8573 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,7 @@ ## Latest changelog +- Uploaded first documentation +- Fixed an issue with RotateBackups - Updated obackup to log failed ssh command results - Updated ssh command filter to log failed commands - Updated ssh command filter to accept personalized commands diff --git a/DOCUMENTATION.txt b/DOCUMENTATION.txt deleted file mode 100644 index 3e0e44f..0000000 --- a/DOCUMENTATION.txt +++ /dev/null @@ -1 +0,0 @@ -## Documentation is being worked on... really... ! diff --git a/TODO.md b/TODO.md index ec89649..a6cabb3 100644 --- a/TODO.md +++ b/TODO.md @@ -1,13 +1,9 @@ -## Main version 1.83 - -Upcomming features: - -- Write proper documentation +## Main version 1.84 Bugs not yet addressed that will be done a bit later: - Recursive task creation doesn't include files in root directory but only subdirectories -- Disk usage check doesn't calculate excluded directories +- Disk usage check doesn't honor excluded directories Future plans: diff --git a/documentation.html b/documentation.html new file mode 100644 index 0000000..bb1b83a --- /dev/null +++ b/documentation.html @@ -0,0 +1,1244 @@ + + + + + + + + +Obackup v1.84 Documentation + + +
+

+Obackup v1.84 Documentation +

+

+Orsiris “Ozy” de Jong +

+

+06/24/13 +

+
+http://www.netpower.fr +
+
+
+
+
+
+Table of Contents +
+ + +
+

+1 Introduction +

+

+1.1 Basic backup concepts +

+
+Backup plans are often setup fast without any checks. There are three main points to consider: +
+ +
+You should always try to restore your backup to a test server, in order to make sure nothing is missing. Once you’ll restore, you’ll find other steps will be necessary, like reapplying file rights and ACLs, restore user permissions in databases, etc... You should always write a quick restore procedure in case someone else is going to restore your data in your holidays. +
+ +
+Immagine you’re making a backup of 100 virtualhosts. Now immagine one of your vhosts get hacked, and hundreds of gigabytes of nice but unsubtitled mangas get uploaded. The same schema will apply if suddently a user decides to upload it’s whole wildlife photo repository. If your backup solution can’t handle the whole amount of data in a given time, every other vhost or user that comes next in backup process won’t be backed up. +
+ +
+A lot of people initiate their backups from a production server to a backup server. Altough it’s easy that way, if your production server gets compromised, it contains the everything needed to access to your backup storage. The only good way to keep your backup server out of trouble is to initate your backups from the backup server that will contain the necessary keys to access your production servers. No production server has to know your where your backup server is. Professionnal backup products resolve this by adding a backup agent on the source servers. +
+
+Obackup has been designed to address this issue, with a certain degree of fault tolerance. +
+
+Example: backup directory /home +
+
+John has 3GB of data to backup. +
+
+Kelly has 2.1GB of data to backup. +
+
+Remote backup goes through an ADSL link providing 1Mb upload speed (128KB/s) which means that 5.1GB would theoretically take 696 minutes (11,5 hours). +
+
+Now immagine John just received his 22,4GB of brand new city plans, and decides to store them in his home directory. +
+
+Backup of 27.5GB would then take 3754 minutes (about 62 ½ hours). +
+
+As a matter of fact, a usual backup solution would continue backing up John’s data, and Kelly’s data would not be backed up until next run. This means Kelly’s data won’t be backed up for the next 3 days ! +
+
+Obackup tries to resolve these issues differently. +
+
+It will enumerate all folders in /home, and let each folder backup for a certain amount of time. Example of backup configuration: +
+
+
+
DIRECTORIES_RECURSE_LIST=/home
+DIRECTORIES_RECURSE_EXCLUDE=/home/lost+found
+SOFT_MAX_EXECUTION_TIME_FILE_TASK=18000
+HARD_MAX_EXECUTION_TIME_FILE_TASK=36000
+
+
+ +
+
+This backup configuration will backup every directory in /home as a separate task except lost+found, granting each one a maximum of 36000 seconds (600 minutes) time to achieve backup. If backup isn’t finised in 18000 seconds, an alert will be generated. If backup won’t finish in 36000 seconds (10 hours), an second alert will be generated, the task will be stopped and next task gets executed. +
+
+Thanks to rsync next execution of that task will resume were it stopped. +
+
+In our case, when John uploads 22.4GB of city plans, only 4,39GB of it’s city plans will be backed up (128KB x 36000 seconds), then Kelly’s documents will be backed up. +
+
+Next days, John gets another 4,39GB backup per day, until everything is synchronized without preventing other backups from running, and you (the administrator) will get email reports about what’s going on. +
+

+1.2 What exactly does Obackup +

+
+Obackup can backup files and MariaDB or MySQL databases, on both local and remote systems. +
+
+It can enumerate all databases on a system and process them as separate tasks. +
+
+It can enumerate all directories in a given path and process them as separate tasks. +
+
+Tasks are executed and warnings generated if they are long. Tasks can be stopped and next one processed if they take too long. +
+
+Backups can be done as superuser via sudo command. +
+
+Local disk space and minimum backup sizes are checked. +
+
+Connectivity to remote host can be checked before running a backup task. +
+
+Internet connectivity can be checked before running a backup task. +
+
+It will send a warning email including the whole backup process execution log if a warning is triggered. +
+
+Backups can be rotated in case you don’t use snapshots on your backup server. +
+
+Pre-processing and post-processing commands can be launched, and their results logged. +
+
+Multiple concurrent instances of obackup can be run as long as they don’t backup to into the same destination directory. +
+
+Everything can be compressed (ssh tunnel, mysql dumps, rsync file transfers). +
+
+Obackup has been successfully tested on RHEL / CentOS 5.8, CentOS 6.4, Debian Linux 6.0.7, Linux Mint 14 but should basically run on any *nix flavor. +
+

+2 Prerequisites +

+

+2.1 General packages +

+
+The following binaries are needed on both local backup system and remote source system. +
+
+
+
rsync mysql mysqldump xz lzma gzip coreutils
+
+
+ +
+

+2.2 Remote backups +

+
+Remote backups are done through an SSH tunnel. To be able to establish such a tunnel without having to enter a password, you’ll have to generate a pair of private and public RSA keys. +
+
+The private part is kept by the computer that initiates the connection. The public part is kept by the source computer that will be backed up. +
+
+The following steps will be required to generate a ssh key. Please create a dedicated backup user and log in as that user on your backup system to perform the following actions. +
+
+
+
$ ssh-keygen -t rsa
+
+
+ +
+
+This should create two files named ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub +
+
+The source server should also have a dedicated backup user (“backupuser” in this example). Both backup and source server don’t need to have the same user. +
+
+Copy the public part to the target server with scp (replace 22 with your ssh port number if needed). +
+
+
+
$ scp –P 22 ~/.ssh/id_rsa backupuser@remotesystem.tld:/home/backupuser/.ssh/authorized_keys
+
+
+ +
+
+Make sure the file is readable by the backupuser on the remote system. +
+
+
+
# chmod 600 /home/backupuser/.ssh/authorized_keys
+# chown backupuser:root /home/backupuser/.ssh/authorized_keys
+
+
+ +
+
+Now you should be able to login as “backupuser” on the target server without any password. You can try this by entering the following: +
+
+
+
$ ssh –p 22 backupuser@remotesystem.tld
+
+
+ +
+
+Be aware that only the user that generated the ssh key can remotely log in. +
+
+After having set up your backups and performed tests, you may read the chapter 2.7↓. +
+

+2.3 Databases backup +

+
+To perform MariaDB or MySQL database backups, you’ll need to create a database user that has read only permissions on all databases. +
+
+On the system that runs the sql daemon, create the database user with the following commands: +
+
+
+
$ mysql -u root -p
+mysql> CREATE USER ’sqlbackup’@’localhost’ IDENTIFIED BY ’YourPassWord!’;
+mysql> GRANT SELECT, SHOW DATABASES, LOCK TABLES on *.* to ’sqlbackup’@’localhost’;
+mysql> FLUSH PRIVILEGES;
+mysql> exit
+
+
+ +
+
+There’s no need for this user to exist as unix user, but the following file needs to exist in the user’s home directory you’re going to use as backup user. +
+
+
+
$ cat ~/.my.cnf
+[client]
+password="YourPassWord!"
+
+
+ +
+
+You should then be able to connect to mysql without specifying a password as your backup user. +
+
+
+
$ mysql –u backupuser 
+
+
+ +
+

+2.4 File backups +

+
+File backups don’t need any special configurations. You only have to worry about your backup user having enough rights to open directories and read files. +
+
+In the best case, your backup user should only have read permissions on your data. A good way is to make your user member of the files group that has read permissions. +
+
+Another way to achieve this is using ACLs if your filesystem supports them. You can add the following permissions for user “backupuser” on directory “/home/web”. Setting a default rule will add rights on new files. +
+
+
+
# setfacl -mR d:g:r-x,g:r-x /home/web
+
+
+ +
+
+Be aware that ACLs are tricky and default group permissions serve as mask for ACLs. +
+
+Make sure you can read your data with your backup user: +
+
+
+
# su backupuser
+$ cat /home/data/to/backup/test.file
+
+
+ +
+

+2.5 Performing superuser backups +

+
+Obackup can run it’s backup tasks as superuser. In order to be able to use the sudo command without having to enter a password, you’ll need to modify the source system allowing commands rsync, du and find to be executed as superuser. Edit the file /etc/sudoers and add the following +
+
+
+
backupuser ALL= NOPASSWD:/usr/bin/rsync
+backupuser ALL= NOPASSWD:/usr/bin/du
+backupuser ALL= NOPASSWD:/usr/bin/find
+
+
+ +
+
+You might check your paths to commands with the following: +
+
+
+
# which rsync
+
+
+ +
+
+You should be aware that there is a minor security risk with having rsync command run as superuser. A user who can run rsync command as superuser can upload any file he wants to the system, including a tweaked /etc/sudoers or /etc/passwd file. +
+
+Please read chapter 2.7↓ to secure your installation. +
+

+2.6 Mail transport agent +

+
+You should make sure your system can send emails so obackup can warn you if something happens. Obackup will use mutt or mail command. Please make sure you can send a test mail with at least one of the following commands run by your backup user: +
+
+
+
$ echo “your test message” | mutt -x -s “This is a test message” your@mail.tld
+$ echo “your test message” | mail -s “This is a test message” your@mail.tld
+
+
+ +
+
+Check your antispam if you don’t get your message. If you still don’t get your message, check your distributions documentation about the mail command. +
+

+2.7 Enhancing remote backup security +

+
+We may want to secure a password-less ssh access by removing non necessary services offered by SSH. +
+
+Edit the file ~/.ssh/authorized_keys created earlier and add the following line in the beginning of the file: +
+
+
+
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
+
+
+ +
+
+Also, we may want to prevent any host except of our backup server to passwordless connect. +
+
+Add the following line: +
+
+
+
from=*.my.backup.servers.domain.tld
+
+
+ +
+
+Your authorized_keys file should look like this: +
+
+
+
from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAS3NzaC1yc2EASZAABIwAAAQEApVJp/qWM3RnCnCFW610yEWy794BvB/N95fWjELbuKRtAWX2vn/2Y857mIsSGZEGJSDJZEJTZOSDFaizm+ofyPRbwU6PplF+j0KAGu9Wcw5iSjwk/1taPZ/Bu20qeRMo4155n1D2lmTJsOznhMH04yB+GbV6Hw== user@host.tld
+
+
+ +
+
+We may also restrict the ssh session to only a couple of commands we’ll need. Obackup package comes with a script called obackup_ssh_filter.sh that will only execute commands Obackup might send. +
+
+Once again edit your authorized_keys file and add the following +
+
+
+
command=”/usr/local/bin/obackup_ssh_filter.sh”
+
+
+ +
+
+Your file should then look like this: +
+
+
+
from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command=”/usr/local/bin/obackup_ssh_filter.sh" ssh-rsa AAS3NzaC1yc2EASZAABIwAAAQEApVJp/qWM3RnCnCFW610yEWy794BvB/N95fWjELbuKRtAWX2vn/2Y857mIsSGZEGJSDJZEJTZOSDFaizm+ofyPRbwU6PplF+j0KAGu9Wcw5iSjwk/1taPZ/Bu20qeRMo4155n1D2lmTJsOznhMH04yB+GbV6Hw== backupuser@remotesystem.tld 
+
+
+ +
+
+Copy then the script obackup_ssh_filter.sh to /usr/local/bin on the remote source server. +
+
+Don’t forget to make it executable and make it owned by root +
+
+
+
# chmod 755 /usr/local/bin/obackup_ssh_filter.sh
+# chown root:root /usr/local/bin/obackup_ssh_filter.sh
+
+
+ +
+
+Now, only the commands “find, du, rsync, echo, mysql, mysqldump and sudo” may be executed via the ssh tunnel. +
+
+You may enable / disable the usage of sudo command by editing the following value in the obackup_ssh_filter.sh script: +
+
+
+
SUDO_EXEC=yes
+
+
+ +
+
+Also, adding remote pre- and postexecution commands in your configuration files will not work if you use the ssh filter. You’ll have to add your main command in obackup_ssh_filter. +
+
+Example if you want to perform remote snapshots you’ll have to allow one of the following: +
+
+
+
CMD1=zfs
+CMD2=xfs
+CMD3=lvm
+
+
+ +
+

+2.8 Security for the paranoid +

+
+Executing rsync as superuser is a security risk. A way to prevent rsync usage allowing only a symlink to be executed. Thus, a attacker script using rsync would not work. This kind of security is called “security by obscurity” and should generally not be the only security process, but makes any attack harder. +
+
+First, let’s create a symlink to rsync called let’s say o_rsync +
+
+
+
# ln -s $(which rsync) $(dirname $(which rsync))/o_rsync
+
+
+ +
+
+Now edit obackup_ssh_filter.sh and change the following value: +
+
+
+
RSYNC_EXECUTABLE=o_rsync
+
+
+ +
+
+Also, edit RSYNC_EXECUTABLE value on any of your backup configuration files and you’re done. +
+

+3 Setup Obackup +

+

+3.1 Create your first backup scenario +

+
+Download your copy of obackup to your backup server and make it executable. +
+
+
+
# chmod +x obackup.sh
+
+
+ +
+
+Now copy the original host_backup.conf file to a name that will reflect your backup and the file to match your needs. +
+
+
+
# cp host_backup.conf personal_backup.sh
+
+
+ +
+
+Obackup will consider any database or directory as separate task, allowing it to skip a malfunctioning task and get the next one in list. +
+
+There are three main parameters called BACKUP_SQL, BACKUP_FILES and REMOTE_BACKUP. Disabling one of these will ignore any related configuration entries later. +
+
+The following minimum example will make a local backup of your /home directory to /backup/path/files/home, excluding lost+found directory. +
+
+The warning flag i set you if the whole backup is smaller than 1MB, or if there is less than 1GB of free space on drive. +
+
+Also, it will exit backup if it can’t ping www.kernel.org nor google.com +
+
+If task takes more than 1 hour (3600 seconds), the warning flag is set. If task takes more than 2 hours (7200 seconds), the warning flag is set and it’ll get stopped. +
+
+At least five copies of directory /home are kept in /backup/path/files/home. +
+
+At the end of the script, if the warning flag is set, an email is sent to your@mail.address +
+
+
+
#!/bin/bash
+BACKUP_ID="personal_backup"
+BACKUP_SQL=no
+BACKUP_FILES=yes
+LOCAL_FILE_STORAGE="/backup/path/files"
+LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=no
+BACKUP_SIZE_MINIMUM=1024
+LOCAL_STORAGE_WARN_MIN_SPACE=1048576
+REMOTE_BACKUP=no
+REMOTE_3RD_PARTY_HOST="www.kernel.org"
+PATH_SEPARATOR_CHAR=";"
+DIRECTORIES_SIMPLE_LIST="/home"
+DIRECTORIES_RECURSE_LIST=""
+DIRECTORIES_RECURSE_EXCLUDE_LIST="/home/lost+found"
+RSYNC_EXCLUDE_PATTERN="*/tmp"
+SOFT_MAX_EXEC_TIME_FILE_TASK=3600
+HARD_MAX_EXEC_TIME_FILE_TASK=7200
+DESTINATION_MAILS="your@mail.address"
+SOFT_MAX_EXEC_TIME_TOTAL=30000
+HARD_MAX_EXEC_TIME_TOTAL=36000
+ROTATE_BACKUPS=yes
+ROTATE_COPIES=5
+
+
+ +
+
+Now try your setup by specifying parameter --dry. +
+
+
+
# ./obackup.sh /path/to/personal_backup.conf --dry
+
+
+ +
+
+Backup should only enumerate what will be processed. If destination directory doesn’t exist, you may get a warning about destination disk space. +
+
+If everything worked out right, you might process the actual backup. +
+
+
+
# ./obackup.sh /path/to/personal_backup.conf
+
+
+ +
+

+3.2 Create regular backup scenarios +

+
+Creating regular backup is quite simple as long as you don’t schedule two backups in a shorter time span than your HARD_MAX_EXEC_TIME_TOTAL value. +
+
+Just create a crontab entry and add parameter --silent so your local mailbox won’t get filled up. +
+
+Example, having a backup scheduled every hour in /etc/crontab +
+
+
+
00 * * * * localbackupuser /usr/local/bin/obackup.sh /home/localbackupuser/databases_backup.sh --silent
+
+
+ +
+
+You may find the backup log under /var/log/obackup-version-your_backup.log +
+

+4 Configuration appendix +

+

+4.1 Command line parameters +

+
+Obackup takes at minimum one parameter, which must be a valid configuration file. +
+
+
+
$ ./obackup.sh /path/to/your/backup.conf
+
+
+ +
+
+Other parameters obackup will take are +
+
+--dry Will make obackup run a simulation only. +
+
+--silent Will run obackup silently, to be used in a cron schedule. +
+
+--help Will print Obackup version and usage. +
+

+4.2 Configuration file parameters +

+
+Set this to whatever you want to identify your backup. This value is also in the log name and in the warning mails. +
+
+
+
BACKUP_ID=personal_backup
+
+
+ +
+
+Enable / disable database backups +
+
+
+
BACKUP_SQL=yes|no
+
+
+ +
+
+Enable / disable files backup +
+
+
+
BACKUP_FILEs=yes|no
+
+
+ +
+
+Database backups storage absolute path on backup system +
+
+
+
LOCAL_SQL_STORAGE=/backup/path/sql
+
+
+ +
+
+Files backups storage absolute path on backup system +
+
+
+
LOCAL_FILE_STORAGE=/backup/path/files
+
+
+ +
+
+Keep absolute paths in your backup. Very usefull to know where exactly to restore files to the source server. +
+
+
+
LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=yes|no
+
+
+ +
+
+Generate an alert if backup size is lower than given value in KB. This generally works to detect unmounted local devices. +
+
+
+
BACKUP_SIZE_MINIMUM=99999999
+
+
+ +
+
+Generate an alert if local storage space is lower than given value in KB. +
+
+
+
LOCAL_STORAGE_WARN_MIN_SPACE=99999999
+
+
+ +
+
+Backups may be executed as root to enable system files backup like /etc. See prerequisites in chapter 2.5↑. +
+
+
+
SUDO_EXEC=yes|no
+
+
+ +
+
+Paranoia option. Don’t change this unless you read chapter 2.8↑. +
+
+
+
RSYNC_EXECUTABLE=rsync
+
+
+ +
+
+Backup a remote host via ssh or not. +
+
+
+
REMOTE_BACKUP=yes|no
+
+
+ +
+ +
+Location of the private RSA key +
+
+
+
SSH_RSA_PRIVATE_KEY=~/.ssh/id_rsa
+
+
+ +
+
+The name of the user on the remote system that has the corresponding public RSA key +
+
+
+
REMOTE_USER=backupuser
+
+
+ +
+
+Remote system hostname or IP address +
+
+
+
REMOTE_HOST=your.host.local
+
+
+ +
+
+Remote SSH port +
+
+
+
REMOTE_PORT=22
+
+
+ +
+
+Enable / disable ssh compression. Leave this enabled unless your connection to remote system is high speed (LAN) +
+
+
+
SSH_COMPRESSION=yes|no
+
+
+ +
+
+Ping remote host before launching backup. Be sure the host responds to ping. Failing to ping will skip current task. +
+
+
+
REMOTE_HOST_PING=yes|no
+
+
+ +
+ +
+
+
REMOTE_3RD_PARTY_HOST="www.kernel.org"
+
+
+ +
+ +
+Database backup user that has read only rights on all databases +
+
+
+
SQL_USER=sqlbackup
+
+
+ +
+
+Save all databases +
+
+
+
DATABASES_ALL=yes|no
+
+
+ +
+
+Database exclude list only when DATABASES_ALL=yes, list separated by space character +
+
+
+
DATABASES_ALL_EXCLUDE_LIST="test otherdb"
+
+
+ +
+
+If DATABASES_ALL=no, the following space separated list of databases will be processed. +
+
+
+
DATABASES_LIST="mydb myseconddb workdb"
+
+
+ +
+
+Maximum backup execution time per database. Soft value generates a warning only. Hard value generates a warning, stops current task and processes next one in list. Time is specified in seconds. +
+
+Can be set to 0 if you don’t want a task to be stopped. +
+
+
+
SOFT_MAX_EXEC_TIME_DB_TASK=3600
+HARD_MAX_EXEC_TIME_DB_TASK=7200
+
+
+ +
+
+Preferred database dump compression program. Leave this set to xz for best results. Compression level 5 is a good compromise between cpu, memory hunger and compression ratio. +
+
+
+
COMPRESS_PROGRAM=xz|lzma|gz
+COMPRESS_RATIO=1-9
+
+
+ +
+
+Sets database dump compression to be done on remote site. Can also be set locally to lower remote system cpu usage in expense for more bandwidth. Don’t forget to set at least ssh compression if you set this value to no. +
+
+
+
COMPRESSION_REMOTE=yes|no
+
+
+ +
+ +
+Path separator charracter for directories list. You can change this to whatever unholy separator you want in your directories list below. (try to use unconventional separators like “|”). +
+
+
+
PATH_SEPARATOR_CHAR=";"
+
+
+ +
+
+Double quoted directory list separated by the specified PATH_SEPARATOR_CHAR. Every directory will be processed as a separate task. +
+
+
+
DIRECTORIES_SIMPLE_LIST="/home/dir1;/var/log;/etc"
+
+
+ +
+
+Double quoted recursive directory list separated by the specified PATH_SEPARATOR_CHAR. Every first level subdirectory of any directory in the list will be processed as a separated task. +
+
+Example: “/home;/var” will create tasks “/home/dir1”, “/home/dir2”, ... “/home/dirN”, “/var/log”, “/var/lib” ... “/var/whatever” +
+
+
+
DIRECTORIES_RECURSE_LIST="/home;/var"
+
+
+ +
+
+Double quoted directory exclude list separated by the specified PATH_SEPARATOR_CHAR. Every directory in this list will be skipped from the recursive list. In the above example you could skip “/home/dir3” +
+
+
+
DIRECTORIES_RECURSE_EXCLUDE_LIST="/home/backupuser;/home/lost+found;/var/tmp"
+
+
+ +
+
+Rsync style exclude patterns. Exclusions happen from the root of the backed-up folder. Be aware that every recurse list task will have it’s own root. +
+
+Example: if recuse list contains /home and you want to exclude the tmp directories in any task’s root you may set RSYNC_EXCLUDE_PATTERN to “*/tmp”. +
+
+Example 2: If you want to skip any file with extension .lck anywhere in your backup, you may set RSYNC_EXCLUDE_PATERN to “**/*.lck” +
+
+
+
RSYNC_EXCLUDE_PATTERN="*/tmp;*/sessions;**/*.lck"
+
+
+ +
+
+Preserve ACLs. Please check that your filesystem supports ACLs and is mounted with it’s support or rsync will get you loads of errors. +
+
+
+
PRESERVE_ACL=yes|no
+
+
+ +
+
+Preserve Xattr. The same applies as for ACLs +
+
+
+
PRESERVE_XATTR=yes|no
+
+
+ +
+
+Use rsync compression for file transfers. Leave this disabled unless your’re not using SSH compression. +
+
+
+
RSYNC_COMPRESS=yes|no
+
+
+ +
+
+Maximum backup execution time per file task. Soft value generates a warning only. Hard value generates a warning, stops current task and processes next one in list. Time is specified in seconds. +
+
+Can be set to 0 if you don’t want a task to be stopped. +
+
+
+
SOFT_MAX_EXEC_TIME_FILE_TASK=3600
+HARD_MAX_EXEC_TIME_FILE_TASK=7200
+
+
+ +
+ +
+Alert email adresses separated by a space character. Alert emails will be sent to the following addresses at the and of the backup script. +
+
+
+
DESTINATION_MAILS="your@mail.address"
+
+
+ +
+
+Maximum backup execution time for the whole backup process. Soft value generates a warning only. Hard value generates a warning and stops the whole backup process. +
+
+
+
SOFT_MAX_EXEC_TIME_TOTAL=30000
+HARD_MAX_EXEC_TIME_TOTAL=36000
+
+
+ +
+
+Backup rotation. In case you don’t use snapshots on your backup server, you may use backup rotation instead. If you set ROTATE_BACKUPS=yes, obackup will keep ROTATE_COPIES number of copies. +
+
+The newest one will be called like the task name. The older ones will have a suffix “.obackup.N” where N is the number of the backup. The higher N is, the older the backup is. +
+
+
+
ROTATE_BACKUPS=yes|no
+ROTATE_COPIES=N
+
+
+ +
+
+Commands that can be run before and / or after backup execution. Remote execution will only happen if REMOTE_BACKUP=yes +
+
+This can be handy to initiate a snapshot script or run any service stops before backing up. +
+
+
+
LOCAL_RUN_BEFORE_CMD=""
+LOCAL_RUN_AFTER_CMD=""
+REMOTE_RUN_BEFORE_CMD=""
+REMOTE_RUN_AFTER_CMD=""
+
+
+ +
+
+Maximum command execution time. Soft value generates a warning only. Hard value generates a warning and stops the whole backup process. +
+
+Can be set to 0 if you don’t want a command to be stopped. +
+
+
+
MAX_EXEC_TIME_PER_CMD_BEFORE=3600
+MAX_EXEC_TIME_PER_CMD_AFTER=0
+
+
+ +
+

+5 Troubleshooting +

+
+Obackup has been tested successfully on multiple systems for a wide variety of backup plans. Please check the following steps before requesting help. +
+

+5.1 Local backups +

+
+Obackup logs every of it’s actions to /var/log/obackup-version-your_backup_id.log +
+
+Please check the log file if something went wrong. +
+
+You might try running obackup as root to check if your problem is user permission related. +
+

+5.2 Remote backups +

+
+Remote backups are more tricky. +
+
+You might check that you can log in remotely with the command +
+
+
+
$ ssh -p 22 remotebackupuser@remotehost.tld
+
+
+ +
+
+Also, you might check that you can use mysql and mysql dump commands remotely +
+
+
+
$ ssh -p 22 remotebackupuser@remotehost.tld mysql -u sqlbackupuser ’SHOW DATABASES’;
+$ ssh -p 22 remotebackupuser@remotehost.tld mysqldump -u sqlbackupuser mysql
+
+
+ +
+
+You can temporarily disable ssh security by removing lines you added in chapter 2.7↑. +
+
+Additionnaly, you can check obackup_ssh_filter log in ~/.ssh/obackup_ssh_filter.log +
+
+You might try running obackup with SUDO_EXEC to check if your problem is user permission related. +
+
+
+
+ + + +
+ + diff --git a/obackup.sh b/obackup.sh deleted file mode 100755 index 46764f4..0000000 --- a/obackup.sh +++ /dev/null @@ -1,1086 +0,0 @@ -#!/bin/bash - -###### Remote (or local) backup script for files & databases -###### (L) 2013 by Ozy de Jong (www.badministrateur.com) -OBACKUP_VERSION=1.84 -OBACKUP_BUILD=0807201301 - -DEBUG=no -SCRIPT_PID=$$ - -LOCAL_USER=$(whoami) -LOCAL_HOST=$(hostname) - -## Log a state message every $KEEP_LOGGING seconds. Should generally not be equal to soft or hard execution time so your log won't be unnecessary big. -KEEP_LOGGING=1801 - -## Global variables and forked command results -DATABASES_TO_BACKUP="" # Processed list of DBs that will be backed up -DATABASES_EXCLUDED_LIST="" # Processed list of DBs that won't be backed up -TOTAL_DATABASES_SIZE=0 # Total DB size of $DATABASES_TO_BACKUP -DIRECTORIES_RECURSE_TO_BACKUP="" # Processed list of recursive directories that will be backed up -DIRECTORIES_EXCLUDED_LIST="" # Processed list of recursive directorires that won't be backed up -DIRECTORIES_TO_BACKUP="" # Processed list of all directories to backup -TOTAL_FILES_SIZE=0 # Total file size of $DIRECTORIES_TO_BACKUP - -# /dev/shm/obackup_dblist_$SCRIPT_PID Databases list and sizes -# /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID Recursive directories list -# /dev/shm/obackup_local_sql_storage_$SCRIPT_PID Local free space for sql backup -# /dev/shm/obackup_local_file_storage_$SCRIPT_PID Local free space for file backup -# /dev/shm/obackup_fsize_$SCRIPT_PID Size of $DIRECTORIES_TO_BACKUP -# /dev/shm/obackup_rsync_output_$SCRIPT_PID Output of Rsync command -# /dev/shm/obackup_config_$SCRIPT_PID Parsed configuration file -# /dev/shm/obackup_run_local_$SCRIPT_PID Output of command to be run localy -# /dev/shm/obackup_run_remote_$SCRIPT_PID Output of command to be run remotely - -# Alert flags -soft_alert_total=0 -error_alert=0 - -function Log -{ - echo "TIME: $SECONDS - $1" >> "$LOG_FILE" - if [ $silent -eq 0 ] - then - echo "TIME: $SECONDS - $1" - fi -} - -function LogError -{ - Log "$1" - error_alert=1 -} - -function TrapError -{ - local JOB="$0" - local LINE="$1" - local CODE="${2:-1}" - if [ $silent -eq 0 ] - then - echo " /!\ Error in ${JOB}: Near line ${LINE}, exit code ${CODE}" - fi -} - -function TrapStop -{ - LogError " /!\ WARNING: Manual extit of backup script. Backups may be in inconsistent state." - if [ "$DEBUG" == "no" ] - then - CleanUp - fi - exit 1 -} - -function Spinner -{ - if [ $silent -eq 1 ] - then - return 1 - fi - - case $toggle - in - 1) - echo -n $1" \ " - echo -ne "\r" - toggle="2" - ;; - - 2) - echo -n $1" | " - echo -ne "\r" - toggle="3" - ;; - - 3) - echo -n $1" / " - echo -ne "\r" - toggle="4" - ;; - - *) - echo -n $1" - " - echo -ne "\r" - toggle="1" - ;; - esac -} - -function Dummy -{ - exit 1; -} - -function StripQuotes -{ - echo $(echo $1 | sed "s/^\([\"']\)\(.*\)\1\$/\2/g") -} - -function EscapeSpaces -{ - echo $(echo $1 | sed 's/ /\\ /g') -} - -function CleanUp -{ - rm -f /dev/shm/obackup_dblist_$SCRIPT_PID - rm -f /dev/shm/obackup_local_sql_storage_$SCRIPT_PID - rm -f /dev/shm/obackup_local_file_storage_$SCRIPT_PID - rm -f /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID - rm -f /dev/shm/obackup_fsize_$SCRIPT_PID - rm -f /dev/shm/obackup_rsync_output_$SCRIPT_PID - rm -f /dev/shm/obackup_config_$SCRIPT_PID - rm -f /dev/shm/obackup_run_local_$SCRIPT_PID - rm -f /dev/shm/obackup_run_remote_$SCRIPT_PID -} - -function SendAlert -{ - CheckConnectivityRemoteHost - CheckConnectivity3rdPartyHosts - cat "$LOG_FILE" | gzip -9 > /tmp/obackup_lastlog.gz - if type -p mutt > /dev/null 2>&1 - then - echo $MAIL_ALERT_MSG | $(which mutt) -x -s "Backup alert for $BACKUP_ID" $DESTINATION_MAILS -a /tmp/obackup_lastlog.gz - if [ $? != 0 ] - then - Log "WARNING: Cannot send alert email via $(which mutt) !!!" - else - Log "Sent alert mail using mutt." - fi - elif type -p mail > /dev/null 2>&1 - then - echo $MAIL_ALERT_MSG | $(which mail) -a /tmp/obackup_lastlog.gz -s "Backup alert for $BACKUP_ID" $DESTINATION_MAILS - if [ $? != 0 ] - then - Log "WARNING: Cannot send alert email via $(which mail) with attachments !!!" - echo $MAIL_ALERT_MSG | $(which mail) -s "Backup alert for $BACKUP_ID" $DESTINATION_MAILS - if [ $? != 0 ] - then - Log "WARNING: Cannot send alert email via $(which mail) without attachments !!!" - else - Log "Sent alert mail using mail command without attachment." - fi - else - Log "Sent alert mail using mail command." - fi - else - Log "WARNING: Cannot send alert email (no mutt / mail present) !!!" - return 1 - fi -} - -function LoadConfigFile -{ - if [ ! -f "$1" ] - then - LogError "Cannot load backup configuration file [$1]. Backup cannot start." - return 1 - elif [[ $1 != *.conf ]] - then - LogError "Wrong configuration file supplied [$1]. Backup cannot start." - else - egrep '^#|^[^ ]*=[^;&]*' "$1" > "/dev/shm/obackup_config_$SCRIPT_PID" - source "/dev/shm/obackup_config_$SCRIPT_PID" - fi -} - -function CheckEnvironment -{ - sed --version > /dev/null 2>&1 - if [ $? != 0 ] - then - LogError "GNU coreutils not found (tested for sed --version). Backup cannot start." - return 1 - fi - - - if [ "$REMOTE_BACKUP" == "yes" ] - then - if ! type -p ssh > /dev/null 2>&1 - then - LogError "ssh not present. Cannot start backup." - return 1 - fi - - if [ "$BACKUP_SQL" != "no" ] - then - if ! type -p mysqldump > /dev/null 2>&1 - then - LogError "mysqldump not present. Cannot start backup." - return 1 - fi - fi - fi - - if [ "$BACKUP_FILES" != "no" ] - then - if ! type -p rsync > /dev/null 2>&1 - then - LogError "rsync not present. Backup cannot start." - return 1 - fi - fi -} - -# Waits for pid $1 to complete. Will log an alert if $2 seconds exec time exceeded unless $2 equals 0. Will stop task and log alert if $3 seconds exec time exceeded. -function WaitForTaskCompletition -{ - soft_alert=0 - SECONDS_BEGIN=$SECONDS - while ps -p$1 > /dev/null - do - Spinner - sleep 1 - EXEC_TIME=$(($SECONDS - $SECONDS_BEGIN)) - if [ $(($EXEC_TIME % $KEEP_LOGGING)) -eq 0 ] - then - Log "Current task still running." - fi - if [ $EXEC_TIME -gt $2 ] - then - if [ $soft_alert -eq 0 ] && [ $2 != 0 ] - then - LogError "Max soft execution time exceeded for task." - soft_alert=1 - fi - if [ $EXEC_TIME -gt $3 ] && [ $3 != 0 ] - then - LogError "Max hard execution time exceeded for task. Stopping task execution." - return 1 - fi - fi - done -} - - -## Runs local command $1 and waits for completition in $2 seconds -function RunLocalCommand -{ - CheckConnectivity3rdPartyHosts - $1 > /dev/shm/obackup_run_local_$SCRIPT_PID & - child_pid=$! - WaitForTaskCompletition $child_pid 0 $2 - wait $child_pid - retval=$? - if [ $retval -eq 0 ] - then - Log "Running command [$1] on local host succeded." - else - Log "Running command [$1] on local host failed." - fi - - Log "Command output:" - Log "$(cat /dev/shm/obackup_run_local_$SCRIPT_PID)" -} - -## Runs remote command $1 and waits for completition in $2 seconds -function RunRemoteCommand -{ - CheckConnectivity3rdPartyHosts - if [ "$REMOTE_BACKUP" == "yes" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Cannot run remote command." - return 1 - else - $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT "$1" > /dev/shm/obackup_run_remote_$SCRIPT_PID & - fi - child_pid=$! - WaitForTaskCompletition $child_pid 0 $2 - wait $child_pid - retval=$? - if [ $retval -eq 0 ] - then - Log "Running command [$1] succeded." - else - LogError "Running command [$1] failed." - fi - - if [ -f /dev/shm/obackup_run_remote_$SCRIPT_PID ] - then - Log "Command output: $(cat /dev/shm/obackup_run_remote_$SCRIPT_PID)" - fi - fi -} - -function RunBeforeHook -{ - if [ "$LOCAL_RUN_BEFORE_CMD" != "" ] - then - RunLocalCommand "$LOCAL_RUN_BEFORE_CMD" $MAX_EXEC_TIME_PER_CMD_BEFORE - fi - - if [ "$REMOTE_RUN_BEFORE_CMD" != "" ] - then - RunRemoteCommand "$REMOTE_RUN_BEFORE_CMD" $MAX_EXEC_TIME_PER_CMD_BEFORE - fi -} - -function RunAfterHook -{ - if [ "$LOCAL_RUN_AFTER_CMD" != "" ] - then - RunLocalCommand "$LOCAL_RUN_AFTER_CMD" $MAX_EXEC_TIME_PER_CMD_AFTER - fi - - if [ "$REMOTE_RUN_AFTER_CMD" != "" ] - then - RunRemoteCommand "$REMOTE_RUN_AFTER_CMD" $MAX_EXEC_TIME_PER_CMD_AFTER - fi -} - -function SetCompressionOptions -{ - if [ "$COMPRESSION_PROGRAM" == "xz" ] && type -p xz > /dev/null 2>&1 - then - COMPRESSION_EXTENSION=.xz - elif [ "$COMPRESSION_PROGRAM" == "lzma" ] && type -p lzma > /dev/null 2>&1 - then - COMPRESSION_EXTENSION=.lzma - elif [ "$COMPRESSION_PROGRAM" == "gzip" ] && type -p gzip > /dev/null 2>&1 - then - COMPRESSION_EXTENSION=.gz - COMPRESSION_OPTIONS=--rsyncable - else - COMPRESSION_EXTENSION= - fi - - if [ "$SSH_COMPRESSION" == "yes" ] - then - SSH_COMP=-C - else - SSH_COMP= - fi -} - -function SetSudoOptions -{ - ## Add this to support prior config files without RSYNC_EXECUTABLE option - if [ "$RSYNC_EXECUTABLE" == "" ] - then - RSYNC_EXECUTABLE=rsync - fi - - if [ "$SUDO_EXEC" == "yes" ] - then - RSYNC_PATH="sudo $(which $RSYNC_EXECUTABLE)" - COMMAND_SUDO="sudo" - else - RSYNC_PATH="$(which $RSYNC_EXECUTABLE)" - COMMAND_SUDO="" - fi -} - -function CreateLocalStorageDirectories -{ - if [ ! -d $LOCAL_SQL_STORAGE ] && [ "$BACKUP_SQL" != "no" ] - then - mkdir -p $LOCAL_SQL_STORAGE - fi - - if [ ! -d $LOCAL_FILE_STORAGE ] && [ "$BACKUP_FILES" != "no" ] - then - mkdir -p $LOCAL_FILE_STORAGE - fi -} - -function CheckSpaceRequirements -{ - if [ "$BACKUP_SQL" != "no" ] - then - if [ -d $LOCAL_SQL_STORAGE ] - then - # Not elegant solution to make df silent on errors - df -P $LOCAL_SQL_STORAGE > /dev/shm/obackup_local_sql_storage_$SCRIPT_PID 2>&1 - if [ $? != 0 ] - then - LOCAL_SQL_SPACE=0 - else - LOCAL_SQL_SPACE=$(cat /dev/shm/obackup_local_sql_storage_$SCRIPT_PID | tail -1 | awk '{print $4}') - LOCAL_SQL_DRIVE=$(cat /dev/shm/obackup_local_sql_storage_$SCRIPT_PID | tail -1 | awk '{print $1}') - fi - - if [ $LOCAL_SQL_SPACE -eq 0 ] - then - LogError "Local sql storage space reported to be 0Ko." - elif [ $LOCAL_SQL_SPACE -lt $TOTAL_DATABASES_SIZE ] - then - LogError "Local disk space may be insufficient to backup files (available space is lower than non compressed databases)." - fi - else - LOCAL_SQL_SPACE=0 - LogError "SQL storage path [$LOCAL_SQL_STORAGE] doesn't exist." - fi - fi - - if [ "$BACKUP_FILES" != "no" ] - then - if [ -d $LOCAL_FILE_STORAGE ] - then - df -P $LOCAL_FILE_STORAGE > /dev/shm/obackup_local_file_storage_$SCRIPT_PID 2>&1 - if [ $? != 0 ] - then - LOCAL_FILE_SPACE=0 - else - LOCAL_FILE_SPACE=$(cat /dev/shm/obackup_local_file_storage_$SCRIPT_PID | tail -1 | awk '{print $4}') - LOCAL_FILE_DRIVE=$(cat /dev/shm/obackup_local_file_storage_$SCRIPT_PID | tail -1 | awk '{print $1}') - fi - - if [ $LOCAL_FILE_SPACE -eq 0 ] - then - LogError "Local file storage space reported to be 0Ko." - elif [ $LOCAL_FILE_SPACE -lt $TOTAL_FILES_SIZE ] - then - LogError "Local disk space may be insufficient to backup files (available space is lower than full backup)." - fi - else - LOCAL_FILE_SPACE=0 - LogError "File storage path [$LOCAL_FILE_STORAGE] doesn't exist." - fi - fi - - if [ "$LOCAL_SQL_DRIVE" == "$LOCAL_FILE_DRIVE" ] - then - LOCAL_SPACE=$LOCAL_FILE_SPACE - else - LOCAL_SPACE=$(($LOCAL_SQL_SPACE+$LOCAL_FILE_SPACE)) - fi - - if [ $BACKUP_SIZE_MINIMUM -gt $(($TOTAL_DATABASES_SIZE+$TOTAL_FILES_SIZE)) ] - then - LogError "Backup size is smaller then expected." - elif [ $LOCAL_STORAGE_WARN_MIN_SPACE -gt $LOCAL_SPACE ] - then - LogError "Local disk space is lower than warning value [$LOCAL_STORAGE_WARN_MIN_SPACE Ko]." - fi - Log "Local Space: $LOCAL_SPACE Ko - Databases size: $TOTAL_DATABASES_SIZE Ko - Files size: $TOTAL_FILES_SIZE Ko" -} - -function CheckTotalExecutionTime -{ - #### Check if max execution time of whole script as been reached - if [ $SECONDS -gt $SOFT_MAX_EXEC_TIME_TOTAL ] - then - if [ $soft_alert_total -eq 0 ] - then - LogError "Max soft execution time of the whole backup exceeded while backing up $BACKUP_TASK." - soft_alert_total=1 - fi - if [ $SECONDS -gt $HARD_MAX_EXEC_TIME_TOTAL ] - then - LogError "Max hard execution time of the whole backup exceeded while backing up $BACKUP_TASK, stopping backup process." - exit 1 - fi - fi -} - -function CheckConnectivityRemoteHost -{ - if [ "$REMOTE_HOST_PING" != "no" ] && [ "$REMOTE_BACKUP" != "no" ] - then - ping $REMOTE_HOST -c 2 > /dev/null 2>&1 - if [ $? != 0 ] - then - LogError "Cannot ping $REMOTE_HOST" - return 1 - fi - fi -} - -function CheckConnectivity3rdPartyHosts -{ - if [ "$REMOTE_3RD_PARTY_HOSTS" != "" ] - then - remote_3rd_party_success=0 - for $i in $REMOTE_3RD_PARTY_HOSTS - do - ping $i -c 2 > /dev/null 2>&1 - if [ $? != 0 ] - then - LogError "Cannot ping 3rd party host $i" - else - remote_3rd_party_success=1 - fi - done - if [ $remote_3rd_party_success -ne 1 ] - then - LogError "No remote 3rd party host responded to ping. No internet ?" - return 1 - fi - fi -} - -function ListDatabases -{ - SECONDS_BEGIN=$SECONDS - Log "Listing databases." - CheckConnectivity3rdPartyHosts - if [ "$REMOTE_BACKUP" == "yes" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Stopping current task." - Dummy & - else - $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT "mysql -u $SQL_USER -Bse 'SELECT table_schema, round(sum( data_length + index_length ) / 1024) FROM information_schema.TABLES GROUP by table_schema;'" > /dev/shm/obackup_dblist_$SCRIPT_PID & - fi - else - mysql -u $SQL_USER -Bse 'SELECT table_schema, round(sum( data_length + index_length ) / 1024) FROM information_schema.TABLES GROUP by table_schema;' > /dev/shm/obackup_dblist_$SCRIPT_PID & - fi - child_pid=$! - WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_DB_TASK $HARD_MAX_EXEC_TIME_DB_TASK - wait $child_pid - retval=$? - if [ $retval -eq 0 ] - then - Log "Listing databases succeeded." - else - LogError "Listing databases failed." - if [ -f /dev/shm/obackup_dblist_$SCRIPT_PID ] - then - LogError "Command output: $(cat /dev/shm/obackup_dblist_$SCRIPT_PID)" - fi - return $retval - fi - - while read line - do - db_name=$(echo $line | cut -d' ' -f1) - db_size=$(echo $line | cut -d' ' -f2) - - if [ "$DATABASES_ALL" == "yes" ] - then - db_backup=1 - for j in $DATABASES_ALL_EXCLUDE_LIST - do - if [ "$db_name" == "$j" ] - then - db_backup=0 - fi - done - else - db_backup=0 - for j in $DATABASES_LIST - do - if [ "$db_name" == "$j" ] - then - db_backup=1 - fi - done - fi - - if [ $db_backup -eq 1 ] - then - if [ "$DATABASES_TO_BACKUP" != "" ] - then - DATABASES_TO_BACKUP="$DATABASES_TO_BACKUP $db_name" - else - DATABASES_TO_BACKUP=$db_name - fi - TOTAL_DATABASES_SIZE=$((TOTAL_DATABASES_SIZE+$db_size)) - else - DATABASES_EXCLUDED_LIST="$DATABASES_EXCLUDED_LIST $db_name" - fi - done < <(cat /dev/shm/obackup_dblist_$SCRIPT_PID) - return 0 -} - -function BackupDatabase -{ - CheckConnectivity3rdPartyHosts - if [ "$REMOTE_BACKUP" == "yes" ] && [ "$COMPRESSION_REMOTE" == "no" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Stopping current task." - exit 1 - fi - $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT mysqldump -u $SQL_USER --skip-lock-tables --single-transaction --database $1 | $COMPRESSION_PROGRAM -$COMPRESSION_LEVEL $COMPRESSION_OPTIONS > $LOCAL_SQL_STORAGE/$1.sql$COMPRESSION_EXTENSION - elif [ "$REMOTE_BACKUP" == "yes" ] && [ "$COMPRESSION_REMOTE" == "yes" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Stopping current task." - exit 1 - fi - $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT "mysqldump -u $SQL_USER --skip-lock-tables --single-transaction --database $1 | $COMPRESSION_PROGRAM -$COMPRESSION_LEVEL $COMPRESSION_OPTIONS" > $LOCAL_SQL_STORAGE/$1.sql$COMPRESSION_EXTENSION - else - mysqldump -u $SQL_USER --skip-lock-tables --single-transaction --database $1 | $COMPRESSION_PROGRAM -$COMPRESSION_LEVEL $COMPRESSION_OPTIONS > $LOCAL_SQL_STORAGE/$1.sql$COMPRESSION_EXTENSION - fi - exit $? -} - -function BackupDatabases -{ - for BACKUP_TASK in $DATABASES_TO_BACKUP - do - Log "Backing up database $BACKUP_TASK" - SECONDS_BEGIN=$SECONDS - BackupDatabase $BACKUP_TASK & - child_pid=$! - WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_DB_TASK $HARD_MAX_EXEC_TIME_DB_TASK - wait $child_pid - retval=$? - SECONDS_END=$SECONDS - EXEC_TIME=$(($SECONDS_END - $SECONDS_BEGIN)) - if [ $retval -ne 0 ] - then - LogError "Backup failed." - else - Log "Backup succeeded." - fi - - CheckTotalExecutionTime - done -} - -# Fetches single quoted directory listing including recursive ones separated by commas (eg '/dir1';'/dir2';'/dir3') -function ListDirectories -{ - SECONDS_BEGIN=$SECONDS - Log "Listing directories to backup." - OLD_IFS=$IFS - IFS=$PATH_SEPARATOR_CHAR - for i in $DIRECTORIES_RECURSE_LIST - do - CheckConnectivity3rdPartyHosts - if [ "$REMOTE_BACKUP" == "yes" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Stopping current task." - Dummy & - else - $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT "$COMMAND_SUDO find $i/ -mindepth 1 -maxdepth 1 -type d" > /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID & - fi - else - $COMMAND_SUDO find $i/ -mindepth 1 -maxdepth 1 -type d > /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID & - fi - child_pid=$! - WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_FILE_TASK $HARD_MAX_EXEC_TIME_FILE_TASK - wait $child_pid - retval=$? - if [ $retval != 0 ] - then - LogError "Could not enumerate recursive directories in $i." - if [ -f /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID ] - then - LogError "Command output: $(cat /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID)" - fi - return 1 - else - Log "Listing of recursive directories succeeded for $i." - fi - - while read line - do - file_exclude=0 - for k in $DIRECTORIES_RECURSE_EXCLUDE_LIST - do - if [ "$k" == "$line" ] - then - file_exclude=1 - fi - done - - if [ $file_exclude -eq 0 ] - then - if [ "$DIRECTORIES_TO_BACKUP" == "" ] - then - DIRECTORIES_TO_BACKUP="'$line'" - else - DIRECTORIES_TO_BACKUP="$DIRECTORIES_TO_BACKUP$PATH_SEPARATOR_CHAR'$line'" - fi - else - DIRECTORIES_EXCLUDED_LIST="$DIRECTORIES_EXCLUDED_LIST$PATH_SEPARATOR_CHAR'$line'" - fi - done < <(cat /dev/shm/obackup_dirs_recurse_list_$SCRIPT_PID) - done - DIRECTORIES_TO_BACKUP_RECURSE=$DIRECTORIES_TO_BACKUP - - for i in $DIRECTORIES_SIMPLE_LIST - do - if [ "$DIRECTORIES_TO_BACKUP" == "" ] - then - DIRECTORIES_TO_BACKUP="'$i'" - else - DIRECTORIES_TO_BACKUP="$DIRECTORIES_TO_BACKUP$PATH_SEPARATOR_CHAR'$i'" - fi - done - - IFS=$OLD_IFS -} - -function GetDirectoriesSize -{ - # remove the path separator char from the dir list with sed 's/;/ /g' - dir_list=$(echo $DIRECTORIES_TO_BACKUP | sed 's/'"$PATH_SEPARATOR_CHAR"'/ /g' ) - Log "Getting files size" - CheckConnectivity3rdPartyHosts - if [ "$REMOTE_BACKUP" == "yes" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Stopping current task." - Dummy & - else - $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT "echo $dir_list | xargs $COMMAND_SUDO du -cs | tail -n1 | cut -f1" > /dev/shm/obackup_fsize_$SCRIPT_PID & - fi - else - echo $dir_list | xargs $COMMAND_SUDO du -cs | tail -n1 | cut -f1 > /dev/shm/obackup_fsize_$SCRIPT_PID & - fi - child_pid=$! - WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_FILE_TASK $HARD_MAX_EXEC_TIME_FILE_TASK - wait $child_pid - retval=$? - if [ $retval != 0 ] - then - LogError "Could not get files size." - if [ -f /dev/shm/obackup_fsize_$SCRIPT_PID ] - then - LogError "Command output: $(cat /dev/shm/obackup_fsize_$SCRIPT_PID)" - fi - return 1 - else - Log "File size fetched successfully." - TOTAL_FILES_SIZE=$(cat /dev/shm/obackup_fsize_$SCRIPT_PID) - fi -} - -function RsyncExcludePattern -{ - OLD_IFS=$IFS - IFS=$PATH_SEPARATOR_CHAR - for excludedir in $RSYNC_EXCLUDE_PATTERN - do - if [ "$RSYNC_EXCLUDE" == "" ] - then - RSYNC_EXCLUDE="--exclude=$(EscapeSpaces $excludedir)" - else - RSYNC_EXCLUDE="$RSYNC_EXCLUDE --exclude=$(EscapeSpaces $excludedir)" - fi - done - IFS=$OLD_IFS -} - -function RsyncArgs -{ - RSYNC_ARGS=-rlptgoDE - if [ "$PRESERVE_ACLS" == "yes" ] - then - RSYNC_ARGS=$RSYNC_ARGS"A" - fi - - if [ "$PRESERVE_XATTR" == "yes" ] - then - RSYNC_ARGS=$RSYNC_ARGS"X" - fi - - if [ "$RSYNC_COMPRESS" == "yes" ] - then - RSYNC_ARGS=$RSYNC_ARGS"z" - fi -} - -function Rsync -{ - i="$(StripQuotes $1)" - if [ "$LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS" == "yes" ] - then - local_file_storage_path="$(dirname $LOCAL_FILE_STORAGE$i)" - else - #### Leave the last directory path if recursive task when absolute paths not set so paths won't be mixed up - if [ "$2" == "recurse" ] - then - local_file_storage_path="$LOCAL_FILE_STORAGE/$(basename $(dirname $i))" - else - local_file_storage_path="$LOCAL_FILE_STORAGE" - fi - fi - if [ ! -d $local_file_storage_path ] - then - mkdir -p "$local_file_storage_path" - fi - - CheckConnectivity3rdPartyHosts - if [ "$REMOTE_BACKUP" == "yes" ] - then - CheckConnectivityRemoteHost - if [ $? != 0 ] - then - LogError "Connectivity test failed. Stopping current task." - exit 1 - fi - rsync_cmd="$(which $RSYNC_EXECUTABLE) $RSYNC_ARGS --delete $RSYNC_EXCLUDE --rsync-path=\"$RSYNC_PATH\" -e \"$(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -p $REMOTE_PORT\" \"$REMOTE_USER@$REMOTE_HOST:$1\" \"$local_file_storage_path\" > /dev/shm/obackup_rsync_output_$SCRIPT_PID 2>&1" - else - rsync_cmd="$(which $RSYNC_EXECUTABLE) $RSYNC_ARGS --delete $RSYNC_EXCLUDE --rsync-path=\"$RSYNC_PATH\" \"$1\" \"$local_file_storage_path\" > /dev/shm/obackup_rsync_output_$SCRIPT_PID 2>&1" - fi - #### Eval is used so the full command is processed without bash adding single quotes round variables - if [ "$DEBUG" == "yes" ] - then - Log $rsync_cmd - fi - eval $rsync_cmd - exit $? -} - -#### First backup simple list then recursive list -function FilesBackup -{ - OLD_IFS=$IFS - IFS=$PATH_SEPARATOR_CHAR - for BACKUP_TASK in $DIRECTORIES_SIMPLE_LIST - do - BACKUP_TASK=$(StripQuotes $BACKUP_TASK) - Log "Beginning file backup $BACKUP_TASK" - SECONDS_BEGIN=$SECONDS - Rsync $BACKUP_TASK & - child_pid=$! - WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_FILE_TASK $HARD_MAX_EXEC_TIME_FILE_TASK - wait $child_pid - retval=$? - SECONDS_END=$SECONDS - EXEC_TIME=$(($SECONDS_END-$SECONDS_BEGIN)) - if [ $retval -ne 0 ] - then - LogError "Backup failed on remote files." - if [ -f /dev/shm/obackup_rsync_output_$SCRIPT_PID ] - then - LogError "$(cat /dev/shm/obackup_rsync_output_$SCRIPT_PID)" - fi - else - Log "Backup succeeded." - fi - CheckTotalExecutionTime - done - - for BACKUP_TASK in $DIRECTORIES_TO_BACKUP_RECURSE - do - BACKUP_TASK=$(StripQuotes $BACKUP_TASK) - Log "Beginning file backup $BACKUP_TASK" - SECONDS_BEGIN=$SECONDS - Rsync $BACKUP_TASK "recurse" & - child_pid=$! - WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_FILE_TASK $HARD_MAX_EXEC_TIME_FILE_TASK - wait $child_pid - retval=$? - SECONDS_END=$SECONDS - EXEC_TIME=$(($SECONDS_END-$SECONDS_BEGIN)) - if [ $retval -ne 0 ] - then - LogError "Backup failed on remote files." - if [ -f /dev/shm/obackup_rsync_output_$SCRIPT_PID ] - then - LogError "$(cat /dev/shm/obackup_rsync_output_$SCRIPT_PID)" - fi - else - Log "Backup succeeded." - fi - CheckTotalExecutionTime - done - IFS=$OLD_IFS -} - -# Will rotate everything in $1 -function RotateBackups -{ - for backup in $(ls -I "*.obackup.*" $1) - do - copy=$ROTATE_COPIES - while [ $copy -gt 1 ] - do - if [ $copy -eq $ROTATE_COPIES ] - then - rm -rf "$1/$backup.obackup.$copy" - fi - path="$1/$backup.obackup.$(($copy-1))" - if [[ -f $path || -d $path ]] - then - mv $path "$1/$backup.obackup.$copy" - fi - copy=$(($copy-1)) - done - - # Latest file backup will not be moved if script configured for remote backup so next rsync execution will only do delta copy instead of full one - if [[ $backup == *.sql.* ]] - then - mv "$1/$backup" "$1/$backup.obackup.1" - elif [ "$REMOTE_BACKUP" == "yes" ] - then - cp -R "$1/$backup" "$1/$backup.obackup.1" - else - mv "$1/$backup" "$1/$backup.obackup.1" - fi - done -} - -function Init -{ - # Set error exit code if a piped command fails - set -o pipefail - set -o errtrace - - trap TrapStop SIGINT SIGQUIT - if [ "$DEBUG" == "yes" ] - then - trap 'TrapError ${LINENO} $?' ERR - fi - - LOG_FILE=/var/log/obackup_$OBACKUP_VERSION-$BACKUP_ID.log - MAIL_ALERT_MSG="Warning: Execution of obackup instance $BACKUP_ID (pid $SCRIPT_PID) as $LOCAL_USER@$LOCAL_HOST produced errors." - -} - -function DryRun -{ - Log "/!\ DRY RUN as $LOCAL_USER@$LOCAL_HOST (PID $SCRIPT_PID)" - - SetCompressionOptions - SetSudoOptions - - if [ "$BACKUP_SQL" != "no" ] - then - ListDatabases - fi - if [ "$BACKUP_FILES" != "no" ] - then - ListDirectories - GetDirectoriesSize - fi - - Log "DB backup list: $DATABASES_TO_BACKUP" - Log "DB exclude list: $DATABASES_EXCLUDED_LIST" - Log "Dirs backup list: $DIRECTORIES_TO_BACKUP" - Log "Dirs exclude list: $DIRECTORIES_EXCLUDED_LIST" - - CheckSpaceRequirements -} - -function Main -{ - Log "Backup launched as $LOCAL_USER@$LOCAL_HOST (PID $SCRIPT_PID)" - - SetCompressionOptions - SetSudoOptions - - if [ "$BACKUP_SQL" != "no" ] - then - ListDatabases - fi - if [ "$BACKUP_FILES" != "no" ] - then - ListDirectories - GetDirectoriesSize - fi - CreateLocalStorageDirectories - CheckSpaceRequirements - - # Make Backup - if [ "$BACKUP_SQL" != "no" ] - then - if [ "$ROTATE_BACKUPS" == "yes" ] - then - RotateBackups $LOCAL_SQL_STORAGE - fi - BackupDatabases - fi - if [ "$BACKUP_FILES" != "no" ] - then - if [ "$ROTATE_BACKUPS" == "yes" ] - then - RotateBackups $LOCAL_FILE_STORAGE - fi - RsyncExcludePattern - RsyncArgs - FilesBackup - fi - # Be a happy sysadmin (and drink a coffee ? Nahh... it's past midnight.) -} - -function Usage -{ - echo "Obackup $OBACKUP_VERSION $OBACKUP_BUILD" - echo "" - echo "usage: obackup backup_name [--dry] [--silent]" - echo "" - echo "--dry: will run obackup without actually doing anything, just testing" - echo "--silent: will run obackup without any output to stdout, usefull for cron backups" -} - -# Command line argument flags -dryrun=0 -silent=0 - -if [ $# -eq 0 ] -then - Usage - exit -fi - -for i in "$@" -do - case $i in - --dry) - dryrun=1 - ;; - --silent) - silent=1 - ;; - --help|-h) - Usage - exit - ;; - esac -done - -CheckEnvironment -if [ $? == 0 ] -then - if [ "$1" != "" ] - then - LoadConfigFile $1 - if [ $? == 0 ] - then - Init - DATE=$(date) - Log "--------------------------------------------------------------------" - Log "$DATE - Obackup v$OBACKUP_VERSION script begin." - Log "--------------------------------------------------------------------" - if [ $dryrun -eq 1 ] - then - DryRun - else - OLD_IFS=$IFS - RunBeforeHook - Main - IFS=$OLD_IFS - RunAfterHook - fi - CleanUp - else - LogError "Configuration file could not be loaded." - exit - fi - else - LogError "No configuration file provided." - exit - fi -fi - -if [ $error_alert -ne 0 ] -then - SendAlert - LogError "Backup script finished with errors." -else - Log "Backup script finshed." -fi