From 6bddcea7594a9bbe329f3181f720cffdf33c7de1 Mon Sep 17 00:00:00 2001 From: deajan Date: Mon, 24 Jun 2013 16:06:56 +0200 Subject: [PATCH 1/5] Fixed typo in Rotate function --- obackup.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/obackup.sh b/obackup.sh index 7a89859..26f2817 100755 --- a/obackup.sh +++ b/obackup.sh @@ -2,7 +2,7 @@ ###### Remote (or local) backup script for files & databases ###### (L) 2013 by Ozy de Jong (www.badministrateur.com) -OBACKUP_VERSION=1.83 +OBACKUP_VERSION=1.84 OBACKUP_BUILD=2306201301 DEBUG=no @@ -917,7 +917,7 @@ function RotateBackups then cp -R "$1/$backup" "$1/$backup.obackup.1" else - mv "$1/backup" "$1/$backup.obackup.1" + mv "$1/$backup" "$1/$backup.obackup.1" fi done } From c5db8a6e794b80984ea71fe069d00a1f29460539 Mon Sep 17 00:00:00 2001 From: deajan Date: Mon, 24 Jun 2013 21:24:14 +0200 Subject: [PATCH 2/5] Uploaded initial documentation file. --- DOCUMENTATION.txt | 1 - documentation.html | 1244 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1244 insertions(+), 1 deletion(-) delete mode 100644 DOCUMENTATION.txt create mode 100644 documentation.html diff --git a/DOCUMENTATION.txt b/DOCUMENTATION.txt deleted file mode 100644 index 3e0e44f..0000000 --- a/DOCUMENTATION.txt +++ /dev/null @@ -1 +0,0 @@ -## Documentation is being worked on... really... ! diff --git a/documentation.html b/documentation.html new file mode 100644 index 0000000..bb1b83a --- /dev/null +++ b/documentation.html @@ -0,0 +1,1244 @@ + + + + + + + + +Obackup v1.84 Documentation + + +
+

+Obackup v1.84 Documentation +

+

+Orsiris “Ozy” de Jong +

+

+06/24/13 +

+
+http://www.netpower.fr +
+
+
+
+
+
+Table of Contents +
+
+
+Section 1: Introduction +
+
+
+Subsection 1.1: Basic backup concepts +
+
+Subsection 1.2: What exactly does Obackup +
+
+
+Section 2: Prerequisites +
+
+
+Subsection 2.1: General packages +
+
+Subsection 2.2: Remote backups +
+
+Subsection 2.3: Databases backup +
+
+Subsection 2.4: File backups +
+
+Subsection 2.5: Performing superuser backups +
+
+Subsection 2.6: Mail transport agent +
+
+Subsection 2.7: Enhancing remote backup security +
+
+Subsection 2.8: Security for the paranoid +
+
+
+Section 3: Setup Obackup +
+
+
+Subsection 3.1: Create your first backup scenario +
+
+Subsection 3.2: Create regular backup scenarios +
+
+
+Section 4: Configuration appendix +
+
+
+Subsection 4.1: Command line parameters +
+
+Subsection 4.2: Configuration file parameters +
+
+
+Section 5: Troubleshooting +
+
+
+Subsection 5.1: Local backups +
+
+Subsection 5.2: Remote backups +
+
+
+ +
+

+1 Introduction +

+

+1.1 Basic backup concepts +

+
+Backup plans are often setup fast without any checks. There are three main points to consider: +
+
    +
  • +As long as you did not try to restore your backup, you won’t know if it’s reliable. +
    • + +
+
+You should always try to restore your backup to a test server, in order to make sure nothing is missing. Once you’ll restore, you’ll find other steps will be necessary, like reapplying file rights and ACLs, restore user permissions in databases, etc... You should always write a quick restore procedure in case someone else is going to restore your data in your holidays. +
+
    +
  • +A single problem in your backup process can render the whole backup process unreliable. +
    • + +
+
+Immagine you’re making a backup of 100 virtualhosts. Now immagine one of your vhosts get hacked, and hundreds of gigabytes of nice but unsubtitled mangas get uploaded. The same schema will apply if suddently a user decides to upload it’s whole wildlife photo repository. If your backup solution can’t handle the whole amount of data in a given time, every other vhost or user that comes next in backup process won’t be backed up. +
+
    +
  • +Your backup process should be secure +
    • + +
+
+A lot of people initiate their backups from a production server to a backup server. Altough it’s easy that way, if your production server gets compromised, it contains the everything needed to access to your backup storage. The only good way to keep your backup server out of trouble is to initate your backups from the backup server that will contain the necessary keys to access your production servers. No production server has to know your where your backup server is. Professionnal backup products resolve this by adding a backup agent on the source servers. +
+
+Obackup has been designed to address this issue, with a certain degree of fault tolerance. +
+
+Example: backup directory /home +
+
+John has 3GB of data to backup. +
+
+Kelly has 2.1GB of data to backup. +
+
+Remote backup goes through an ADSL link providing 1Mb upload speed (128KB/s) which means that 5.1GB would theoretically take 696 minutes (11,5 hours). +
+
+Now immagine John just received his 22,4GB of brand new city plans, and decides to store them in his home directory. +
+
+Backup of 27.5GB would then take 3754 minutes (about 62 ½ hours). +
+
+As a matter of fact, a usual backup solution would continue backing up John’s data, and Kelly’s data would not be backed up until next run. This means Kelly’s data won’t be backed up for the next 3 days ! +
+
+Obackup tries to resolve these issues differently. +
+
+It will enumerate all folders in /home, and let each folder backup for a certain amount of time. Example of backup configuration: +
+
+
+
DIRECTORIES_RECURSE_LIST=/home
+DIRECTORIES_RECURSE_EXCLUDE=/home/lost+found
+SOFT_MAX_EXECUTION_TIME_FILE_TASK=18000
+HARD_MAX_EXECUTION_TIME_FILE_TASK=36000
+
+
+ +
+
+This backup configuration will backup every directory in /home as a separate task except lost+found, granting each one a maximum of 36000 seconds (600 minutes) time to achieve backup. If backup isn’t finised in 18000 seconds, an alert will be generated. If backup won’t finish in 36000 seconds (10 hours), an second alert will be generated, the task will be stopped and next task gets executed. +
+
+Thanks to rsync next execution of that task will resume were it stopped. +
+
+In our case, when John uploads 22.4GB of city plans, only 4,39GB of it’s city plans will be backed up (128KB x 36000 seconds), then Kelly’s documents will be backed up. +
+
+Next days, John gets another 4,39GB backup per day, until everything is synchronized without preventing other backups from running, and you (the administrator) will get email reports about what’s going on. +
+

+1.2 What exactly does Obackup +

+
+Obackup can backup files and MariaDB or MySQL databases, on both local and remote systems. +
+
+It can enumerate all databases on a system and process them as separate tasks. +
+
+It can enumerate all directories in a given path and process them as separate tasks. +
+
+Tasks are executed and warnings generated if they are long. Tasks can be stopped and next one processed if they take too long. +
+
+Backups can be done as superuser via sudo command. +
+
+Local disk space and minimum backup sizes are checked. +
+
+Connectivity to remote host can be checked before running a backup task. +
+
+Internet connectivity can be checked before running a backup task. +
+
+It will send a warning email including the whole backup process execution log if a warning is triggered. +
+
+Backups can be rotated in case you don’t use snapshots on your backup server. +
+
+Pre-processing and post-processing commands can be launched, and their results logged. +
+
+Multiple concurrent instances of obackup can be run as long as they don’t backup to into the same destination directory. +
+
+Everything can be compressed (ssh tunnel, mysql dumps, rsync file transfers). +
+
+Obackup has been successfully tested on RHEL / CentOS 5.8, CentOS 6.4, Debian Linux 6.0.7, Linux Mint 14 but should basically run on any *nix flavor. +
+

+2 Prerequisites +

+

+2.1 General packages +

+
+The following binaries are needed on both local backup system and remote source system. +
+
+
+
rsync mysql mysqldump xz lzma gzip coreutils
+
+
+ +
+

+2.2 Remote backups +

+
+Remote backups are done through an SSH tunnel. To be able to establish such a tunnel without having to enter a password, you’ll have to generate a pair of private and public RSA keys. +
+
+The private part is kept by the computer that initiates the connection. The public part is kept by the source computer that will be backed up. +
+
+The following steps will be required to generate a ssh key. Please create a dedicated backup user and log in as that user on your backup system to perform the following actions. +
+
+
+
$ ssh-keygen -t rsa
+
+
+ +
+
+This should create two files named ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub +
+
+The source server should also have a dedicated backup user (“backupuser” in this example). Both backup and source server don’t need to have the same user. +
+
+Copy the public part to the target server with scp (replace 22 with your ssh port number if needed). +
+
+
+
$ scp –P 22 ~/.ssh/id_rsa backupuser@remotesystem.tld:/home/backupuser/.ssh/authorized_keys
+
+
+ +
+
+Make sure the file is readable by the backupuser on the remote system. +
+
+
+
# chmod 600 /home/backupuser/.ssh/authorized_keys
+# chown backupuser:root /home/backupuser/.ssh/authorized_keys
+
+
+ +
+
+Now you should be able to login as “backupuser” on the target server without any password. You can try this by entering the following: +
+
+
+
$ ssh –p 22 backupuser@remotesystem.tld
+
+
+ +
+
+Be aware that only the user that generated the ssh key can remotely log in. +
+
+After having set up your backups and performed tests, you may read the chapter 2.7↓. +
+

+2.3 Databases backup +

+
+To perform MariaDB or MySQL database backups, you’ll need to create a database user that has read only permissions on all databases. +
+
+On the system that runs the sql daemon, create the database user with the following commands: +
+
+
+
$ mysql -u root -p
+mysql> CREATE USER ’sqlbackup’@’localhost’ IDENTIFIED BY ’YourPassWord!’;
+mysql> GRANT SELECT, SHOW DATABASES, LOCK TABLES on *.* to ’sqlbackup’@’localhost’;
+mysql> FLUSH PRIVILEGES;
+mysql> exit
+
+
+ +
+
+There’s no need for this user to exist as unix user, but the following file needs to exist in the user’s home directory you’re going to use as backup user. +
+
+
+
$ cat ~/.my.cnf
+[client]
+password="YourPassWord!"
+
+
+ +
+
+You should then be able to connect to mysql without specifying a password as your backup user. +
+
+
+
$ mysql –u backupuser 
+
+
+ +
+

+2.4 File backups +

+
+File backups don’t need any special configurations. You only have to worry about your backup user having enough rights to open directories and read files. +
+
+In the best case, your backup user should only have read permissions on your data. A good way is to make your user member of the files group that has read permissions. +
+
+Another way to achieve this is using ACLs if your filesystem supports them. You can add the following permissions for user “backupuser” on directory “/home/web”. Setting a default rule will add rights on new files. +
+
+
+
# setfacl -mR d:g:r-x,g:r-x /home/web
+
+
+ +
+
+Be aware that ACLs are tricky and default group permissions serve as mask for ACLs. +
+
+Make sure you can read your data with your backup user: +
+
+
+
# su backupuser
+$ cat /home/data/to/backup/test.file
+
+
+ +
+

+2.5  Performing superuser backups +

+
+Obackup can run it’s backup tasks as superuser. In order to be able to use the sudo command without having to enter a password, you’ll need to modify the source system allowing commands rsync, du and find to be executed as superuser. Edit the file /etc/sudoers and add the following +
+
+
+
backupuser ALL= NOPASSWD:/usr/bin/rsync
+backupuser ALL= NOPASSWD:/usr/bin/du
+backupuser ALL= NOPASSWD:/usr/bin/find
+
+
+ +
+
+You might check your paths to commands with the following: +
+
+
+
# which rsync
+
+
+ +
+
+You should be aware that there is a minor security risk with having rsync command run as superuser. A user who can run rsync command as superuser can upload any file he wants to the system, including a tweaked /etc/sudoers or /etc/passwd file. +
+
+Please read chapter 2.7↓ to secure your installation. +
+

+2.6 Mail transport agent +

+
+You should make sure your system can send emails so obackup can warn you if something happens. Obackup will use mutt or mail command. Please make sure you can send a test mail with at least one of the following commands run by your backup user: +
+
+
+
$ echo “your test message” | mutt -x -s “This is a test message” your@mail.tld
+$ echo “your test message” | mail -s “This is a test message” your@mail.tld
+
+
+ +
+
+Check your antispam if you don’t get your message. If you still don’t get your message, check your distributions documentation about the mail command. +
+

+2.7  Enhancing remote backup security +

+
+We may want to secure a password-less ssh access by removing non necessary services offered by SSH. +
+
+Edit the file ~/.ssh/authorized_keys created earlier and add the following line in the beginning of the file: +
+
+
+
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
+
+
+ +
+
+Also, we may want to prevent any host except of our backup server to passwordless connect. +
+
+Add the following line: +
+
+
+
from=*.my.backup.servers.domain.tld
+
+
+ +
+
+Your authorized_keys file should look like this: +
+
+
+
from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAS3NzaC1yc2EASZAABIwAAAQEApVJp/qWM3RnCnCFW610yEWy794BvB/N95fWjELbuKRtAWX2vn/2Y857mIsSGZEGJSDJZEJTZOSDFaizm+ofyPRbwU6PplF+j0KAGu9Wcw5iSjwk/1taPZ/Bu20qeRMo4155n1D2lmTJsOznhMH04yB+GbV6Hw== user@host.tld
+
+
+ +
+
+We may also restrict the ssh session to only a couple of commands we’ll need. Obackup package comes with a script called obackup_ssh_filter.sh that will only execute commands Obackup might send. +
+
+Once again edit your authorized_keys file and add the following +
+
+
+
command=”/usr/local/bin/obackup_ssh_filter.sh”
+
+
+ +
+
+Your file should then look like this: +
+
+
+
from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command=”/usr/local/bin/obackup_ssh_filter.sh" ssh-rsa AAS3NzaC1yc2EASZAABIwAAAQEApVJp/qWM3RnCnCFW610yEWy794BvB/N95fWjELbuKRtAWX2vn/2Y857mIsSGZEGJSDJZEJTZOSDFaizm+ofyPRbwU6PplF+j0KAGu9Wcw5iSjwk/1taPZ/Bu20qeRMo4155n1D2lmTJsOznhMH04yB+GbV6Hw== backupuser@remotesystem.tld 
+
+
+ +
+
+Copy then the script obackup_ssh_filter.sh to /usr/local/bin on the remote source server. +
+
+Don’t forget to make it executable and make it owned by root +
+
+
+
# chmod 755 /usr/local/bin/obackup_ssh_filter.sh
+# chown root:root /usr/local/bin/obackup_ssh_filter.sh
+
+
+ +
+
+Now, only the commands “find, du, rsync, echo, mysql, mysqldump and sudo” may be executed via the ssh tunnel. +
+
+You may enable / disable the usage of sudo command by editing the following value in the obackup_ssh_filter.sh script: +
+
+
+
SUDO_EXEC=yes
+
+
+ +
+
+Also, adding remote pre- and postexecution commands in your configuration files will not work if you use the ssh filter. You’ll have to add your main command in obackup_ssh_filter. +
+
+Example if you want to perform remote snapshots you’ll have to allow one of the following: +
+
+
+
CMD1=zfs
+CMD2=xfs
+CMD3=lvm
+
+
+ +
+

+2.8  Security for the paranoid +

+
+Executing rsync as superuser is a security risk. A way to prevent rsync usage allowing only a symlink to be executed. Thus, a attacker script using rsync would not work. This kind of security is called “security by obscurity” and should generally not be the only security process, but makes any attack harder. +
+
+First, let’s create a symlink to rsync called let’s say o_rsync +
+
+
+
# ln -s $(which rsync) $(dirname $(which rsync))/o_rsync
+
+
+ +
+
+Now edit obackup_ssh_filter.sh and change the following value: +
+
+
+
RSYNC_EXECUTABLE=o_rsync
+
+
+ +
+
+Also, edit RSYNC_EXECUTABLE value on any of your backup configuration files and you’re done. +
+

+3 Setup Obackup +

+

+3.1 Create your first backup scenario +

+
+Download your copy of obackup to your backup server and make it executable. +
+
+
+
# chmod +x obackup.sh
+
+
+ +
+
+Now copy the original host_backup.conf file to a name that will reflect your backup and the file to match your needs. +
+
+
+
# cp host_backup.conf personal_backup.sh
+
+
+ +
+
+Obackup will consider any database or directory as separate task, allowing it to skip a malfunctioning task and get the next one in list. +
+
+There are three main parameters called BACKUP_SQL, BACKUP_FILES and REMOTE_BACKUP. Disabling one of these will ignore any related configuration entries later. +
+
+The following minimum example will make a local backup of your /home directory to /backup/path/files/home, excluding lost+found directory. +
+
+The warning flag i set you if the whole backup is smaller than 1MB, or if there is less than 1GB of free space on drive. +
+
+Also, it will exit backup if it can’t ping www.kernel.org nor google.com +
+
+If task takes more than 1 hour (3600 seconds), the warning flag is set. If task takes more than 2 hours (7200 seconds), the warning flag is set and it’ll get stopped. +
+
+At least five copies of directory /home are kept in /backup/path/files/home. +
+
+At the end of the script, if the warning flag is set, an email is sent to your@mail.address +
+
+
+
#!/bin/bash
+BACKUP_ID="personal_backup"
+BACKUP_SQL=no
+BACKUP_FILES=yes
+LOCAL_FILE_STORAGE="/backup/path/files"
+LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=no
+BACKUP_SIZE_MINIMUM=1024
+LOCAL_STORAGE_WARN_MIN_SPACE=1048576
+REMOTE_BACKUP=no
+REMOTE_3RD_PARTY_HOST="www.kernel.org"
+PATH_SEPARATOR_CHAR=";"
+DIRECTORIES_SIMPLE_LIST="/home"
+DIRECTORIES_RECURSE_LIST=""
+DIRECTORIES_RECURSE_EXCLUDE_LIST="/home/lost+found"
+RSYNC_EXCLUDE_PATTERN="*/tmp"
+SOFT_MAX_EXEC_TIME_FILE_TASK=3600
+HARD_MAX_EXEC_TIME_FILE_TASK=7200
+DESTINATION_MAILS="your@mail.address"
+SOFT_MAX_EXEC_TIME_TOTAL=30000
+HARD_MAX_EXEC_TIME_TOTAL=36000
+ROTATE_BACKUPS=yes
+ROTATE_COPIES=5
+
+
+ +
+
+Now try your setup by specifying parameter --dry. +
+
+
+
# ./obackup.sh /path/to/personal_backup.conf --dry
+
+
+ +
+
+Backup should only enumerate what will be processed. If destination directory doesn’t exist, you may get a warning about destination disk space. +
+
+If everything worked out right, you might process the actual backup. +
+
+
+
# ./obackup.sh /path/to/personal_backup.conf
+
+
+ +
+

+3.2 Create regular backup scenarios +

+
+Creating regular backup is quite simple as long as you don’t schedule two backups in a shorter time span than your HARD_MAX_EXEC_TIME_TOTAL value. +
+
+Just create a crontab entry and add parameter --silent so your local mailbox won’t get filled up. +
+
+Example, having a backup scheduled every hour in /etc/crontab +
+
+
+
00 * * * * localbackupuser /usr/local/bin/obackup.sh /home/localbackupuser/databases_backup.sh --silent
+
+
+ +
+
+You may find the backup log under /var/log/obackup-version-your_backup.log +
+

+4 Configuration appendix +

+

+4.1 Command line parameters +

+
+Obackup takes at minimum one parameter, which must be a valid configuration file. +
+
+
+
$ ./obackup.sh /path/to/your/backup.conf
+
+
+ +
+
+Other parameters obackup will take are +
+
+--dry Will make obackup run a simulation only. +
+
+--silent Will run obackup silently, to be used in a cron schedule. +
+
+--help Will print Obackup version and usage. +
+

+4.2 Configuration file parameters +

+
+Set this to whatever you want to identify your backup. This value is also in the log name and in the warning mails. +
+
+
+
BACKUP_ID=personal_backup
+
+
+ +
+
+Enable / disable database backups +
+
+
+
BACKUP_SQL=yes|no
+
+
+ +
+
+Enable / disable files backup +
+
+
+
BACKUP_FILEs=yes|no
+
+
+ +
+
+Database backups storage absolute path on backup system +
+
+
+
LOCAL_SQL_STORAGE=/backup/path/sql
+
+
+ +
+
+Files backups storage absolute path on backup system +
+
+
+
LOCAL_FILE_STORAGE=/backup/path/files
+
+
+ +
+
+Keep absolute paths in your backup. Very usefull to know where exactly to restore files to the source server. +
+
+
+
LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=yes|no
+
+
+ +
+
+Generate an alert if backup size is lower than given value in KB. This generally works to detect unmounted local devices. +
+
+
+
BACKUP_SIZE_MINIMUM=99999999
+
+
+ +
+
+Generate an alert if local storage space is lower than given value in KB. +
+
+
+
LOCAL_STORAGE_WARN_MIN_SPACE=99999999
+
+
+ +
+
+Backups may be executed as root to enable system files backup like /etc. See prerequisites in chapter 2.5↑. +
+
+
+
SUDO_EXEC=yes|no
+
+
+ +
+
+Paranoia option. Don’t change this unless you read chapter 2.8↑. +
+
+
+
RSYNC_EXECUTABLE=rsync
+
+
+ +
+
+Backup a remote host via ssh or not. +
+
+
+
REMOTE_BACKUP=yes|no
+
+
+ +
+
    +
  • +The following options only apply is REMOTE_BACKUP=yes +
    • + +
+
+Location of the private RSA key +
+
+
+
SSH_RSA_PRIVATE_KEY=~/.ssh/id_rsa
+
+
+ +
+
+The name of the user on the remote system that has the corresponding public RSA key +
+
+
+
REMOTE_USER=backupuser
+
+
+ +
+
+Remote system hostname or IP address +
+
+
+
REMOTE_HOST=your.host.local
+
+
+ +
+
+Remote SSH port +
+
+
+
REMOTE_PORT=22
+
+
+ +
+
+Enable / disable ssh compression. Leave this enabled unless your connection to remote system is high speed (LAN) +
+
+
+
SSH_COMPRESSION=yes|no
+
+
+ +
+
+Ping remote host before launching backup. Be sure the host responds to ping. Failing to ping will skip current task. +
+
+
+
REMOTE_HOST_PING=yes|no
+
+
+ +
+
    +
  • +Check for internet access by pinging one or more hosts before remote backup tasks. Leave this empty do disable the check. Failing to ping will skip current task. +
    • + +
+
+
+
REMOTE_3RD_PARTY_HOST="www.kernel.org"
+
+
+ +
+
    +
  • +Database options, only apply if BACKUP_SQL=yes +
    • + +
+
+Database backup user that has read only rights on all databases +
+
+
+
SQL_USER=sqlbackup
+
+
+ +
+
+Save all databases +
+
+
+
DATABASES_ALL=yes|no
+
+
+ +
+
+Database exclude list only when DATABASES_ALL=yes, list separated by space character +
+
+
+
DATABASES_ALL_EXCLUDE_LIST="test otherdb"
+
+
+ +
+
+If DATABASES_ALL=no, the following space separated list of databases will be processed. +
+
+
+
DATABASES_LIST="mydb myseconddb workdb"
+
+
+ +
+
+Maximum backup execution time per database. Soft value generates a warning only. Hard value generates a warning, stops current task and processes next one in list. Time is specified in seconds. +
+
+Can be set to 0 if you don’t want a task to be stopped. +
+
+
+
SOFT_MAX_EXEC_TIME_DB_TASK=3600
+HARD_MAX_EXEC_TIME_DB_TASK=7200
+
+
+ +
+
+Preferred database dump compression program. Leave this set to xz for best results. Compression level 5 is a good compromise between cpu, memory hunger and compression ratio. +
+
+
+
COMPRESS_PROGRAM=xz|lzma|gz
+COMPRESS_RATIO=1-9
+
+
+ +
+
+Sets database dump compression to be done on remote site. Can also be set locally to lower remote system cpu usage in expense for more bandwidth. Don’t forget to set at least ssh compression if you set this value to no. +
+
+
+
COMPRESSION_REMOTE=yes|no
+
+
+ +
+
    +
  • +File backup options, only apply if BACKUP_FILES=yes +
    • + +
+
+Path separator charracter for directories list. You can change this to whatever unholy separator you want in your directories list below. (try to use unconventional separators like “|”). +
+
+
+
PATH_SEPARATOR_CHAR=";"
+
+
+ +
+
+Double quoted directory list separated by the specified PATH_SEPARATOR_CHAR. Every directory will be processed as a separate task. +
+
+
+
DIRECTORIES_SIMPLE_LIST="/home/dir1;/var/log;/etc"
+
+
+ +
+
+Double quoted recursive directory list separated by the specified PATH_SEPARATOR_CHAR. Every first level subdirectory of any directory in the list will be processed as a separated task. +
+
+Example: “/home;/var” will create tasks “/home/dir1”, “/home/dir2”, ... “/home/dirN”, “/var/log”, “/var/lib” ... “/var/whatever” +
+
+
+
DIRECTORIES_RECURSE_LIST="/home;/var"
+
+
+ +
+
+Double quoted directory exclude list separated by the specified PATH_SEPARATOR_CHAR. Every directory in this list will be skipped from the recursive list. In the above example you could skip “/home/dir3” +
+
+
+
DIRECTORIES_RECURSE_EXCLUDE_LIST="/home/backupuser;/home/lost+found;/var/tmp"
+
+
+ +
+
+Rsync style exclude patterns. Exclusions happen from the root of the backed-up folder. Be aware that every recurse list task will have it’s own root. +
+
+Example: if recuse list contains /home and you want to exclude the tmp directories in any task’s root you may set RSYNC_EXCLUDE_PATTERN to “*/tmp”. +
+
+Example 2: If you want to skip any file with extension .lck anywhere in your backup, you may set RSYNC_EXCLUDE_PATERN to “**/*.lck” +
+
+
+
RSYNC_EXCLUDE_PATTERN="*/tmp;*/sessions;**/*.lck"
+
+
+ +
+
+Preserve ACLs. Please check that your filesystem supports ACLs and is mounted with it’s support or rsync will get you loads of errors. +
+
+
+
PRESERVE_ACL=yes|no
+
+
+ +
+
+Preserve Xattr. The same applies as for ACLs +
+
+
+
PRESERVE_XATTR=yes|no
+
+
+ +
+
+Use rsync compression for file transfers. Leave this disabled unless your’re not using SSH compression. +
+
+
+
RSYNC_COMPRESS=yes|no
+
+
+ +
+
+Maximum backup execution time per file task. Soft value generates a warning only. Hard value generates a warning, stops current task and processes next one in list. Time is specified in seconds. +
+
+Can be set to 0 if you don’t want a task to be stopped. +
+
+
+
SOFT_MAX_EXEC_TIME_FILE_TASK=3600
+HARD_MAX_EXEC_TIME_FILE_TASK=7200
+
+
+ +
+
    +
  • +General options +
    • + +
+
+Alert email adresses separated by a space character. Alert emails will be sent to the following addresses at the and of the backup script. +
+
+
+
DESTINATION_MAILS="your@mail.address"
+
+
+ +
+
+Maximum backup execution time for the whole backup process. Soft value generates a warning only. Hard value generates a warning and stops the whole backup process. +
+
+
+
SOFT_MAX_EXEC_TIME_TOTAL=30000
+HARD_MAX_EXEC_TIME_TOTAL=36000
+
+
+ +
+
+Backup rotation. In case you don’t use snapshots on your backup server, you may use backup rotation instead. If you set ROTATE_BACKUPS=yes, obackup will keep ROTATE_COPIES number of copies. +
+
+The newest one will be called like the task name. The older ones will have a suffix “.obackup.N” where N is the number of the backup. The higher N is, the older the backup is. +
+
+
+
ROTATE_BACKUPS=yes|no
+ROTATE_COPIES=N
+
+
+ +
+
+Commands that can be run before and / or after backup execution. Remote execution will only happen if REMOTE_BACKUP=yes +
+
+This can be handy to initiate a snapshot script or run any service stops before backing up. +
+
+
+
LOCAL_RUN_BEFORE_CMD=""
+LOCAL_RUN_AFTER_CMD=""
+REMOTE_RUN_BEFORE_CMD=""
+REMOTE_RUN_AFTER_CMD=""
+
+
+ +
+
+Maximum command execution time. Soft value generates a warning only. Hard value generates a warning and stops the whole backup process. +
+
+Can be set to 0 if you don’t want a command to be stopped. +
+
+
+
MAX_EXEC_TIME_PER_CMD_BEFORE=3600
+MAX_EXEC_TIME_PER_CMD_AFTER=0
+
+
+ +
+

+5 Troubleshooting +

+
+Obackup has been tested successfully on multiple systems for a wide variety of backup plans. Please check the following steps before requesting help. +
+

+5.1 Local backups +

+
+Obackup logs every of it’s actions to /var/log/obackup-version-your_backup_id.log +
+
+Please check the log file if something went wrong. +
+
+You might try running obackup as root to check if your problem is user permission related. +
+

+5.2 Remote backups +

+
+Remote backups are more tricky. +
+
+You might check that you can log in remotely with the command +
+
+
+
$ ssh -p 22 remotebackupuser@remotehost.tld
+
+
+ +
+
+Also, you might check that you can use mysql and mysql dump commands remotely +
+
+
+
$ ssh -p 22 remotebackupuser@remotehost.tld mysql -u sqlbackupuser ’SHOW DATABASES’;
+$ ssh -p 22 remotebackupuser@remotehost.tld mysqldump -u sqlbackupuser mysql
+
+
+ +
+
+You can temporarily disable ssh security by removing lines you added in chapter 2.7↑. +
+
+Additionnaly, you can check obackup_ssh_filter log in ~/.ssh/obackup_ssh_filter.log +
+
+You might try running obackup with SUDO_EXEC to check if your problem is user permission related. +
+
+
+
+ +
+ +
+ + From 5c9d3ee8eab88fefb28e82f2adba9bf76ff99e43 Mon Sep 17 00:00:00 2001 From: deajan Date: Mon, 24 Jun 2013 22:25:44 +0300 Subject: [PATCH 3/5] Update CHANGELOG.md --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a34718..e6a8573 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,7 @@ ## Latest changelog +- Uploaded first documentation +- Fixed an issue with RotateBackups - Updated obackup to log failed ssh command results - Updated ssh command filter to log failed commands - Updated ssh command filter to accept personalized commands From 324416d28c648aa010e0435132f98ea209c143ed Mon Sep 17 00:00:00 2001 From: deajan Date: Mon, 24 Jun 2013 22:53:40 +0200 Subject: [PATCH 4/5] Fixed a typo in ListDatabases local lookup --- obackup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/obackup.sh b/obackup.sh index 26f2817..695a0f6 100755 --- a/obackup.sh +++ b/obackup.sh @@ -529,7 +529,7 @@ function ListDatabases $(which ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT "mysql -u $SQL_USER -Bse 'SELECT table_schema, round(sum( data_length + index_length ) / 1024) FROM information_schema.TABLES GROUP by table_schema;'" > /dev/shm/obackup_dblist_$SCRIPT_PID & fi else - mysql -u $SQL_USER -Bse 'SELECT table_schema, round(sum( data_length + index_length ) / 1024) FROM information_schema.TABLES GROUP by table_schema;' > /dev/shm/oback_dblist_$SCRIPT_PID & + mysql -u $SQL_USER -Bse 'SELECT table_schema, round(sum( data_length + index_length ) / 1024) FROM information_schema.TABLES GROUP by table_schema;' > /dev/shm/obackup_dblist_$SCRIPT_PID & fi child_pid=$! WaitForTaskCompletition $child_pid $SOFT_MAX_EXEC_TIME_DB_TASK $HARD_MAX_EXEC_TIME_DB_TASK From fcd10edeb527443467b84a7bf1cef720585d0280 Mon Sep 17 00:00:00 2001 From: deajan Date: Tue, 25 Jun 2013 09:38:23 +0200 Subject: [PATCH 5/5] Update TODO.md --- TODO.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/TODO.md b/TODO.md index ec89649..a6cabb3 100644 --- a/TODO.md +++ b/TODO.md @@ -1,13 +1,9 @@ -## Main version 1.83 - -Upcomming features: - -- Write proper documentation +## Main version 1.84 Bugs not yet addressed that will be done a bit later: - Recursive task creation doesn't include files in root directory but only subdirectories -- Disk usage check doesn't calculate excluded directories +- Disk usage check doesn't honor excluded directories Future plans: