Fixing small typosé

This reverts commit 7ee1872718.

CHANGELOG.md

@@ -4,11 +4,6 @@ SHORT FUTURE IMPROVEMENTS
 - Rewrite rsync exclude patterns using \"pattern\" instead of escaped chars
 - Clean most of recursive task creation code
 
-FAR FUTURE IMPROVEMENTS
------------------------
-
-- (Secret world domination... Still need to get bald and get a cat)
-
 KNOWN ISSUES
 ------------
 
@@ -16,9 +11,74 @@ KNOWN ISSUES
 - Bandwidth parameter is ignored for SQL backups
 - Missing symlink support when run from MSYS environment
 
+UNDER WORK
+----------
+
+- Commands like cp should have their stderr redirected to log file
+- Mysqldump must be checked for not telling success if a table is damaged (also check for event table error)
+- Mysqldump commands error msg must be logged
+
+
 CHANGELOG
 ---------
-
+- New function to kill child processes
+- Fixed no_maxtime not honored
+- Improved some logging, also added highlighting to stdout errors
+- Backported some fixes from Osync
+	- Small improvements on install script
+	- Copy ssh_filter.sh from osync project
+	- Small improvements in obackup-batch.sh time management
+- Quick and dirty hack to get the full last run log in SendAlert email
+- Added detection of obackup.sh script in obackup-batch.sh to overcome mising path in crontab
+- Moved command line arguments after config file load for allowing command line overrides
+- Added a config file option equivalent to --dontgetsize
+- Added basic install script from osync project
+- Added obackup-batch.sh from osync project to rerun failed backups in row
+- Delta copy algorithm is now used even for local copies (usefull for network drives), this can be overriden in config file
+- Added --dontgetsize parameter to backup huge systems immediatly
+- Fixed multiple keep logging messages since sleep time between commands has been lowered under a second
+- Create local subdirectories if not exist before running rsync (rsync doesn't handle mkdir -p)
+- Backported some fixes from Osync
+	- Lowered sleep time between commands
+	- Lowered debug sleep times
+	- Fixed a bug with exclude pattern globbing preventing multiple exludes
+	- Lowered default compression level for email alerts (for low end systems)
+- Prevent exclude pattern globbing before the pattern reaches the rsync cmd
+- Fixed some typos with ported code from osync rendering stats and partial downloads unusable
+- Added delete on destination option for files that vanished from source
+- Fixed ignoring compression level in conf file
+- Added experimental partial downloads support for rsync so big files can be resumed on slow links
+- Fixed dry mode sql backup output
+- Prevented triggering TrapError if there are no child processes to terminate on TrapQuit
+- Improved mysql debug logs
+- Prevent creation of backup-id less log file when DEBUG is set
+- WARNING: Default behavior is now to copy the referrent files and directories from symlinks (this can reach files outside the backup root)
+- Recursive directory search now includes symlinks (find -L option. -type d cannot be replaced by -xtype d because of portability issues with BSD)
+- Dry mode does not create target directories anymore
+- Dry mode also tries mysqldumps now (check for error messages being logged)
+- Added experimental partial download support
+- Added Rsync exclude files suppport from osync
+- Fixed another issue with existing symlinks to directories on target on non recursive backups
+- Fixed remaining rsync -E option preventing obackup to work correctly on MacOS X
+- Fixed an issue with existing symlinks to directories on target
+- Prevent changed IFS to make ping commands fail
+- Added RotateCopies execution time (spinner support)
+- redirect stderr for mysqldump to catch problems
+- Moved msys specific code to Init(Local|Remote)OSSettings except in TrapQuit that needs to work at any moment
+- Added support for multithreaded gzip (if pigz is installed)
+- Merged back changes from osync codebase
+	- Enhanced debugging
+	- Added language agnostic system command output
+	- Enhanced log sending
+	- Better handling of OS specific commands
+	- Improved WaitForTaskCompletion when DEBUG enabled or SILENT enabled
+	- Enhanced OS detection
+- More correct error message on remote connection failure
+- Gzipped logs are now deleted once sent
+- Fixed some typos (thanks to Pavel Kiryukhin)
+- Improved OS detection and added prelimnary MacOS X support
+- Improved execution hook logs
+- Improved RunLocalCommand execution hook
 - 02 Nov. 2013: v1.84 RC3
 - Updated documentation
 - Minor rewrites in recursive backup code

README.md

@@ -27,6 +27,12 @@ Currently, Obackup also runs on FreeBSD and Windows MSYS environment, altough it
 
 Feel free to drop me a mail for limited support in my free time.
 
+## Warning
+
+Starting with Obackup 1.84RC4, the default behavior is modified.
+Obackup now follows symlinks and treats them as the referent files / dirs, following symlinks even outside the backup root, which IMHO is more secure in terms of backups.
+You may disable this behavior in the config file.
+
 ## Installation
 
 You can download the latest obackup script from authors website.
@@ -64,6 +70,8 @@ that's actually launched and it's result.
 One you're happy with a test run, you may run obackup as a cron task with the "--silent" parameter so output will not be written to stdout.
 All backup activity is logged to "/var/log/obackup_backupname.log" or current directory if /var/log is not writable.
 
+You may mix "--silent" and "--verbose" parameters to output verbose input only in the log files.
+
 ## Final words
 
 Backup tasks aren't always reliable, connectivity loss, insufficient disk space, hacked servers with tons of unusefull stuff to backup... Anything can happen.

exclude.list.example

@@ -0,0 +1,7 @@
+.AppleDouble/
+._*
+.DS_Store
+Thumbs.db
+System Volume Information
+$Recycle.Bin
+

host_backup.conf

@@ -2,7 +2,7 @@
 
 ###### Remote (or local) backup script for files & databases
 ###### (L) 2013 by Ozy de Jong (www.netpower.fr)
-###### Config file rev 0211201301
+###### Config file rev 2015090801
 
 ## ---------- GENERAL BACKUP OPTIONS
 
@@ -32,8 +32,10 @@ LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=yes
 
 ## Generate an alert if backup size is lower than given value in Kb (this can also help identifying empty mount dirs)
 BACKUP_SIZE_MINIMUM=1024
+## You may disable testing backup size
+DISABLE_GET_BACKUP_FILE_SIZE=no
 
-## Generate an alert if local storage free space is lower than given value in Kb.
+## Generate an alert if local storage free space is lower than given value in Kb. Keep in mind that disabling backup file size test will only test min space against SQL backup size.
 LOCAL_STORAGE_WARN_MIN_SPACE=1048576
 
 ## ---------- MISC OPTIONS
@@ -58,7 +60,7 @@ REMOTE_PORT=22
 ## ssh compression should be used unless your remote connection is good enough (LAN)
 SSH_COMPRESSION=yes
 ## Remote rsync executable path. Leave this empty in most cases
-REMOTE_RSYNC_PATH=""
+RSYNC_REMOTE_PATH=""
 
 ## Check for connectivity to remote host before launching remote backup tasks. Be sure the hosts responds to ping. Failing to ping will skip current task.
 REMOTE_HOST_PING=yes
@@ -70,21 +72,20 @@ REMOTE_3RD_PARTY_HOSTS="www.kernel.org www.google.com"
 ## Database backup user
 SQL_USER=backupuser
 
-## Enabling the following option will save all databases on local or remote given SQL instance except the ones specified in the exlude list.
+## Enabling the following option will save all databases on local or remote given SQL instance except the ones specified in the exclude list.
 ## Every found database will be backed up as separate backup task.
 DATABASES_ALL=yes
 DATABASES_ALL_EXCLUDE_LIST="test;mysql"
-## Alternatively, if DATABASES_ALL=no, you can specifiy a list of databases to backup separated by spaces.
+## Alternatively, if DATABASES_ALL=no, you can specify a list of databases to backup separated by spaces.
 DATABASES_LIST=""
 
-## Max backup execution time per Database task. Soft max exec time generates a warning only. Hard max ecec time generates a warning and stops current backup task.
+## Max backup execution time per Database task. Soft max exec time generates a warning only. Hard max exec time generates a warning and stops current backup task.
 ## If a task gets stopped, next one in the task list gets executed. Time is specified in seconds.
 SOFT_MAX_EXEC_TIME_DB_TASK=3600
 HARD_MAX_EXEC_TIME_DB_TASK=7200
 
-## Preferred SQL dump compression. Can be set to xz, lzma or gzip.
-## Generally, xz level 5 is a good compromise between cpu, memory hunger and compress ratio. Gzipped files are set to be rsyncable.
-COMPRESSION_PROGRAM=xz
+## Preferred SQL dump compression. Compression methods can be xz, lzma, pigz or gzip (will fallback from xz to gzip depending if available)
+## Generally, level 5 is a good compromise between cpu, memory hunger and compress ratio. Gzipped files are set to be rsyncable.
 COMPRESSION_LEVEL=3
 
 ## SQL Dump compression should be done on remote side but can also be done locally to lower remote system usage (will take more bandwidth, check for ssh compression)
@@ -96,10 +97,10 @@ COMPRESSION_REMOTE=yes
 DIRECTORIES_SIMPLE_LIST="/var/named;/var/lib"
 
 ## There's a special backup schema in Obackup called 'backup task division'  which creates one backup task per level 1 subdirectory of a directory.
-## This is VERY usefull to backup multiple virtualhosts as separate tasks without having to specifiy each one separately.
-## This may also be usefull dividing big data directories in subdirectories tasks.
+## This is VERY useful to backup multiple virtualhosts as separate tasks without having to specify each one separately.
+## This may also be useful dividing big data directories in subdirectories tasks.
 
-## Directories backup task division backup: Semicolon separated directories of which every level 1 subdirectory will be backed up recursively as a spearate backup task.
+## Directories backup task division backup: Semicolon separated directories of which every level 1 subdirectory will be backed up recursively as a separate backup task.
 ## Example: "/home;/var" will create tasks "/home/dir1", "/home/dir2", ... "/home/dirN", "/var/log", "/var/lib"... "/var/whatever"
 DIRECTORIES_RECURSE_LIST="/home"
 ## You may optionally exclude subdirectories from task division. On the above example you could exclude /home/dir2 by adding it to DIRECTORIES_RECURSE_EXCLUDE_LIST
@@ -108,7 +109,13 @@ DIRECTORIES_RECURSE_EXCLUDE_LIST="/home/backupuser;/home/lost+found"
 ## Rsync exclude patterns, used by simple and division lists
 RSYNC_EXCLUDE_PATTERN="*/tmp;*/ftp/www/cache/cachefs;*/sessions"
 
-## List separator char. You may set an alternative seperator char for your directories lists above.
+## File that contains the list of directories or files to exclude from sync on both sides. Leave this empty if you don't want to use an exclusion file.
+## This file has to be in the same directory as the config file
+## Paths are relative to sync dirs. One element per line.
+RSYNC_EXCLUDE_FROM=""
+#RSYNC_EXCLUDE_FROM="exclude.list"
+
+## List separator char. You may set an alternative separator char for your directories lists above.
 PATH_SEPARATOR_CHAR=";"
 
 ## Preserve ACLS. Make sure source and target FS can hold same ACLs or you'll get loads of errors.
@@ -116,16 +123,33 @@ PRESERVE_ACL=no
 ## Preserve Xattr. MAke sure source and target FS can hold same Xattr or you'll get loads of errors.
 PRESERVE_XATTR=no
 
-## Let RSYNC compress file transfers. Do not use this on local-local backup schemes. Also, this is not usefull if SSH compression is enabled.
+## Transforms symlinks into referent files/dirs
+COPY_SYMLINKS=yes
+## Treat symlinked dirs as dirs. CAUTION: This also follows symlinks outside of the replica root.
+KEEP_DIRLINKS=yes
+## Preserve hard links. Make sure source and target FS can manage hard links or you will lose them.
+PRESERVE_HARDLINKS=no
+
+
+## Let RSYNC compress file transfers. Do not use this on local-local backup schemes. Also, this is not useful if SSH compression is enabled.
 RSYNC_COMPRESS=no
 
 ## Max execution time per file backup task. Soft is warning only. Hard is warning, stopping backup and processing next one one file list. Tilme is specified in seconds
 SOFT_MAX_EXEC_TIME_FILE_TASK=3600
 HARD_MAX_EXEC_TIME_FILE_TASK=7200
 
+## Keep partial uploads that can be resumed on next run, experimental feature
+PARTIAL=no
+
+## Delete files on destination that vanished from source. Do not turn this on unless you enabled backup rotation  or a snapshotting FS like zfs to keep those vanished files on the destination.
+DELETE_VANISHED_FILES=no
+
+## Use delta copy algortithm (usefull when local paths are network drives), defaults to yes
+DELTA_COPIES=yes
+
 ## ---------- ALERT OPTIONS
 
-## Alert email adresses separated by a space character
+## Alert email addresses separated by a space character
 DESTINATION_MAILS="your@mail.address"
 
 ## Windows (MSYS environment) only mail options (used by sendemail.exe)
@@ -148,14 +172,14 @@ ROTATE_COPIES=7
 ## ---------- EXECUTION HOOKS
 
 ## Commands can will be run before and / or after backup execution (remote execution will only happen if REMOTE_BACKUP is set).
-## This is usefull to make a snapshot before backing up data, or even handle snapshots of backed up data.
+## This is useful to make a snapshot before backing up data, or even handle snapshots of backed up data.
 LOCAL_RUN_BEFORE_CMD=""
 LOCAL_RUN_AFTER_CMD=""
 
 REMOTE_RUN_BEFORE_CMD=""
 REMOTE_RUN_AFTER_CMD=""
 
-## Max execution time of commands before they get force killed. Leave 0 if you don't wan't this to happen. Time is specified in seconds.
+## Max execution time of commands before they get force killed. Leave 0 if you don't want this to happen. Time is specified in seconds.
 MAX_EXEC_TIME_PER_CMD_BEFORE=0
 MAX_EXEC_TIME_PER_CMD_AFTER=0
 

install.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+SCRIPT_BUILD=2015082501
+
+## Obackup install script
+## Tested on RHEL / CentOS 6 & 7
+## Please adapt this to fit your distro needs
+
+if [ "$(whoami)" != "root" ]
+then
+  echo "Must be run as root."
+  exit 1
+fi
+
+mkdir /etc/obackup
+cp ./host_backup.conf /etc/obackup/host_backup.conf.example
+cp ./exclude.list.example /etc/obackup
+cp ./obackup.sh /usr/local/bin
+cp ./obackup-batch.sh /usr/local/bin
+cp ./ssh_filter.sh /usr/local/bin
+chmod 755 /usr/local/bin/obackup.sh
+chmod 755 /usr/local/bin/obackup-batch.sh
+chmod 755 /usr/local/bin/ssh_filter.sh
+chown root:root /usr/local/bin/ssh_filter.sh
+

obackup-batch.sh

@@ -0,0 +1,170 @@
+#!/usr/bin/env bash
+
+PROGRAM="Obackup-batch" # Batch program to run obackup instances sequentially and rerun failed ones
+AUTHOR="(L) 2013-2014 by Orsiris \"Ozy\" de Jong"
+CONTACT="http://www.netpower.fr/obackup - ozy@netpower.fr"
+PROGRAM_BUILD=2508201501
+
+## Runs an obackup instance for every conf file found
+## If an instance fails, run it again if time permits
+
+## Configuration file path. The path where all the obackup conf files are, usually /etc/obackup
+CONF_FILE_PATH=/etc/obackup
+
+## If maximum execution time is not reached, failed instances will be rerun. Max exec time is in seconds. Example is set to 10 hours.
+MAX_EXECUTION_TIME=36000
+
+## Specifies the number of reruns an instance may get
+MAX_RERUNS=3
+
+## Log file path
+if [ -w /var/log ]
+then
+        LOG_FILE=/var/log/obackup-batch.log
+else
+        LOG_FILE=./obackup-batch.log
+fi
+
+# No need to edit under this line ##############################################################
+
+function Log
+{
+        prefix="TIME: $SECONDS - "
+        echo -e "$prefix$1" >> "$LOG_FILE"
+
+        if [ $silent -eq 0 ]
+        then
+                echo -e "$prefix$1"
+        fi
+}
+
+function CheckEnvironment
+{
+        ## Obackup executable full path can be set here if it cannot be found on the system
+        if ! type -p obackup.sh > /dev/null 2>&1
+        then
+                if [ -f /usr/local/bin/obackup.sh ]
+                then
+                        OBACKUP_EXECUTABLE=/usr/local/bin/obackup.sh
+                elif [ -f ./obackup.sh ]
+                then
+                        OBACKUP_EXECUTABLE=./obackup.sh
+                else
+                        Log "Could not find obackup.sh"
+                        exit 1
+                fi
+        else
+                OBACKUP_EXECUTABLE=$(type -p obackup.sh)
+        fi
+
+	## Check for CONF_FILE_PATH
+        if [ ! -d "$CONF_FILE_PATH" ]
+        then
+                Log "Cannot find conf file path $CONF_FILE_PATH"
+                Usage
+        fi
+}
+
+function Batch
+{
+	## Get list of .conf files
+	for i in $(ls $CONF_FILE_PATH/*.conf)
+	do
+		if [ "$RUN" == "" ]
+		then
+			RUN="$i"
+		else
+			RUN=$RUN" $i"
+		fi
+	done
+
+	RERUNS=0
+	while ([ $MAX_EXECUTION_TIME -gt $SECONDS ] || [ $MAX_EXECUTION_TIME -eq 0 ]) && [ "$RUN" != "" ] && [ $MAX_RERUNS -gt $RERUNS ]
+	do
+		Log "Obackup instances will be run for: $RUN"
+		for i in $RUN
+		do
+			$OBACKUP_EXECUTABLE "$i" $opts
+			if [ $? != 0 ]
+			then
+				Log "Run instance $(basename $i) failed"
+				if [ "RUN_AGAIN" == "" ]
+				then
+					RUN_AGAIN="$i"
+				else
+					RUN_AGAIN=$RUN_AGAIN" $i"
+				fi
+			else
+				Log "Run instance $(basename $i) succeed."
+			fi
+		done
+		RUN="$RUN_AGAIN"
+		RUN_AGAIN=""
+		RERUNS=$(($RERUNS + 1))
+	done
+}
+
+function Usage
+{
+        echo "$PROGRAM $PROGRAM_BUILD"
+        echo $AUTHOR
+        echo $CONTACT
+        echo ""
+        echo "Batch script to sequentially run obackup instances and rerun failed ones."
+        echo "Usage: obackup-batch.sh [OPTIONS]"
+        echo ""
+        echo "[OPTIONS]"
+	echo "--path=/path/to/conf      Path to obackup conf files, defaults to /etc/obackup"
+	echo "--max-reruns=X            Number of runs  max for failed instances, (defaults to 3)"
+	echo "--max-exec-time=X         Retry failed instances only if max execution time not reached (defaults to 36000 seconds)"
+	echo "--no-maxtime		Run obackup without honoring conf file defined timeouts"
+        echo "--dry                     Will run obackup without actually doing anything; just testing"
+        echo "--silent                  Will run obackup without any output to stdout, used for cron jobs"
+        echo "--verbose                 Increases output"
+        exit 128
+}
+
+silent=0
+dry=0
+verbose=0
+opts=""
+for i in "$@"
+do
+        case $i in
+                --silent)
+                silent=1
+		opts=$opts" --silent"
+                ;;
+                --dry)
+                dry=1
+		opts=$opts" --dry"
+                ;;
+                --verbose)
+                verbose=1
+                opts=$opts" --verbose"
+		;;
+		--no-maxtime)
+		opts=$opts" --no-maxtime"
+		;;
+		--path=*)
+		CONF_FILE_PATH=${i##*=}
+		;;
+		--max-reruns=*)
+		MAX_RERUNS=${i##*=}
+		;;
+		--max-exec-time=*)
+		MAX_EXECUTION_TIME=${i##*=}
+		;;
+		--help|-h|-?)
+		Usage
+		;;
+		*)
+		Log "Unknown param '$i'"
+		Usage
+		;;
+	esac
+done
+
+CheckEnvironment
+Log "$(date) Obackup batch run"
+Batch

ssh_filter.sh

@@ -1,10 +1,13 @@
 #!/bin/bash
 
-##### Obackup / Osync ssh command filter build 2306201301
-##### This script should be located in /usr/local/bin in the remote system that will be backed up / synced
+##### Osync ssh command filter build 2015070203
+##### This script should be located in /usr/local/bin in the remote system to sync / backup
 ##### It will filter the commands that can be run remotely via ssh.
 ##### Please chmod 755 and chown root:root this file
 
+##### Obackup needed commands: rsync find du mysql mysqldump (sudo)
+##### Osync needed commands: rsync find du echo mkdir rm if df (sudo)
+
 ## If enabled, execution of "sudo" command will be allowed.
 SUDO_EXEC=yes
 ## Paranoia option. Don't change this unless you read the documentation and still feel concerned about security issues.
@@ -30,22 +33,29 @@ function Go
 case ${SSH_ORIGINAL_COMMAND%% *} in
 	"$RSYNC_EXECUTABLE")
 	Go ;;
-	"mysqldump")
-	Go ;;
-	"mysql")
-	Go ;;
-	"echo")
-	Go ;;
 	"find")
 	Go ;;
 	"du")
 	Go ;;
+	"mysql")
+	Go ;;
+	"mysqldump")
+	Go ;;
 	"$CMD1")
+	if [ "$CMD1" != "" ]
+	then
 		Go ;;
+	fi
 	"$CMD2")
+	if [ "$CMD2" != "" ]
+	then
 		Go ;;
+	fi
 	"$CMD3")
+	if [ "$CMD3" != "" ]
+	then
 		Go ;;
+	fi
 	"sudo")
 	if [ "$SUDO_EXEC" == "yes" ]
 	then
@@ -56,24 +66,42 @@ case ${SSH_ORIGINAL_COMMAND%% *} in
 		then
 			Go
 		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo find"* ]]
+		then
+			Go
+                elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo mysql"* ]]
+                then
+                        Go
+                elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo mysqldump"* ]]
                 then
                         Go
 		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD1"* ]]
+		then
+			if [ "$CMD1" != "" ]
 			then
 			Go
+			fi
 		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD2"* ]]
 		then
-			Go
-		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD3"* ]]
+			if [ "$CMD2" != "" ]
 			then
 			Go
+			fi
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD3"* ]]
+		then
+			if [ "$CMD3" != "" ]
+			then
+			Go
+			fi
 		else
 			Log "Command [$SSH_ORIGINAL_COMMAND] not allowed."
+			exit 1
 		fi
 	else
 		Log "Command [$SSH_ORIGINAL_COMMAND] not allowed. sudo not enabled."
+		exit 1
 	fi
 	;;
 	*)
 	Log "Command [$SSH_ORIGINAL_COMMAND] not allowed."
+	exit 1
 esac