Fixed wrong ofunctions ver

Better kill childs works flawlessly

CHANGELOG.md

@@ -1,26 +1,43 @@
-SHORT FUTURE IMPROVEMENTS
--------------------------
-
-- Rewrite rsync exclude patterns using \"pattern\" instead of escaped chars
-- Clean most of recursive task creation code
-
 KNOWN ISSUES
 ------------
 
 - Backup size check does not honor rsync exclude patterns
 - Bandwidth parameter is ignored for SQL backups
 - Missing symlink support when run from MSYS environment
-
+- Mysqldump errors aren't taken in account
-UNDER WORK
-----------
-
-- Commands like cp should have their stderr redirected to log file
-- Mysqldump must be checked for not telling success if a table is damaged (also check for event table error)
-- Mysqldump commands error msg must be logged
-
 
 CHANGELOG
 ---------
+
+README: FreeBSD execution needs mailer (not found), sudo missing, bash needed, sed missing (see if StripQuotes mandatory)
+
+06 Aug 2016: obackup v2.0 released
+- Made logging begin before remote checks for sanity purposes
+- RunAfterCommands can get executed when trapquit
+- Improved process killing and process time control
+- Added optional statistics for installer
+- Added an option to ignore knownhosts for ssh connections (use with caution, this can lead to a security issue)
+- Improved mail fallback
+- More logging enhancements
+- Improved upgrade script
+- Revamped rsync patterns to allow include and exclude patterns
+- Better SQL and file backup task separation (rotate copies and warnings are defined for sql and/or file)
+- Added reverse backup, now backups can be local, pushed or pulled to or from a remote system
+- Better fallback for SendAlert even if disk full
+- Added an alert email sent on warnings while backup script is running
+- Way better logging of errors in _GetDirectoriesSizeX, _BackupDatabaseX, _CreateStorageDirectoriesX
+- Added bogus config file checks & environment checks
+- Full code refactoring to use local and remote code once
+- Fully merged codebase with osync
+	- Added (much) more verbose debugging (and possibility to remove debug code to gain speed)
+	- Replace child_pid by $? directly, add a better sub process killer in TrapQuit
+	- Added some automatic checks in code, for _DEBUG mode (and _PARANOIA_DEBUG now)
+	- Improved Logging
+	- Updated obackup to be fully compliant with coding style
+- Fixed creation of bogus subdirectories in some cases
+- A long list of minor improvements and bug fixes
+
+v0-1.x - Jan 2013 - Oct 2015
 - New function to kill child processes
 - Fixed no_maxtime not honored
 - Improved some logging, also added highlighting to stdout errors

CODING_STYLE.TXT

@@ -0,0 +1,147 @@
+Coding style used for my bash projects (v2.1 Oct 2015)
+
+++++++ Header
+
+Always use the following header
+
+----BEGIN HEADER
+#!/usr/bin/env bash
+
+PROGRAM="program-name" # Long description
+AUTHOR="(L) 20XX-20YY by Orsiris \"Ozy\" de Jong"
+CONTACT="http://www.example.com me@example.com"
+PROGRAM_BUILD=YYYYMMDDVV
+
+## Optional instructions
+----END HEADER
+
+Using bind style versionning:
+YYYYMMDDVV (Year, Month, Day, Revision): Example: 2015012402 = 2nd revision of 24 Jan 2015
+
+#!/usr/bin/env bash instead of #!/bin/bash
+
+Change old scripts with
+for i in $(grep -r '#!/bin/bash' * |cut -f1 -d':'); do sed -i 's&#!/bin/bash&#!/usr/bin/env bash&g' $i; done
+
+
+type instead of type -p for bash test (other shells don't know -p)
+++++++ Indentation
+
+Using tabs
+Transform old shell scripts using unexpand command
+
+++++++ Comments
+
+Some command # comment
+## Some comment on a new line
+################################################# Some separation
+
+++++++ Work comments
+
+Whenever there is some idea to postpone, use #TODO[-version]:[dev-name:] some remark
+A marker must be left where on the line a dev is working (when the work isn't finished). Marker is #WIP:dev-name: some remark
+dev-name is mandatory if more than one person is coding
+Example: #TODO-v2.1:deajan: need to do something
+
+++++++ Variables
+
+All local variables are lowercase, separated by _ (ex: low_wait)
+All global variables full upercase, separated by _ (ex: EXEC_TIME)
+All environment variables (verbose, silent, debug, etc) have prefix _ and are full upercase, separated by _ (ex: _PARANOIA_DEBUG)
+
+++++++ Functions
+
+Every word in a function begins with an uppercase (ex: SomeFunctionDoesThings)
+
+Define functions this way. Use sed ':a;N;$!ba;s/\n{\n/ {\n/g' to adapt when opening bracket is on a new line.
+
+function something {
+	
+}
+
+If function has some arguments, use local variable names that are more readable than $1...$n. Explain via comments what those variables contain if needed.
+
+function anotherthing {
+	local var_name="${1}"
+	local other_var_name="${2}" # This variable contains stuff
+}
+
+Functions should always have return status
+function thirdthing {
+	some_command
+	return $?
+}
+
+++++++ Sub functions
+
+When a function is a subroutine of another function, it is called _SomethingAsSubFunction
+
+++++++ Function argument check
+
+Bash does not provide any checks against missing function arguments. Also, missing quotes can lead to an inconsistent number of arguments.
+Every function call will be checked by __CheckArguments which takes the number of arguments, $# (the real number of args given), the parent function name and the parent function's arguments.
+__CheckArguments will trigger a critical error if number of arguments if incorrect. This will also prevent silent typo errors.
+Ex:
+
+function Something {
+	local some="${1}"
+	local other="${2}"
+	local args="${3}"
+	__CheckArguments 3 $# $FUNCNAME "$*"
+
+__CheckArguments will only trigger if script is called with DEBUG=yes
+Also, with PARANOIA_DEBUG=yes, __CheckArguments will recount all arguments given by "$*" and compare. This can mislead if arguments contain spaces.
+
+++++++ If statements
+
+If statements will be fully written (word "if" must be used). then is written on the same line.
+(Use sed ':a;N;$!ba;s/]\n\t*then/]; then/g' to convert files to this format... Replace "],new line, zero or more tabs, then" by "; then")
+if [ something ]; then
+	stuff
+else
+	other stuff
+fi
+
+++++++ Logging
+
+A logging function is available with the following levels of logging:
+
+- DEBUG: Only log this when DEBUG flas is set in program. Any command forged for eval should be logged by this.
+- NOTICE: Standard messages
+- WARN: Requires attention
+- ERROR: Program produced an error but continues execution
+- CRITICAL: Program execution is halted
+
+++++++ Eval
+
+Most commands should be logged to a tmp file.
+The basic way of doing is:
+
+cmd='"something '$somevar'" > some_file 2>&1'
+eval $cmd &
+WaitForTaskCompletion $! 0 0 $FUNCNAME
+
+Remote commands should exist as:
+
+cmd=$SSH_CMD' "some; commands \"'$VARIABLE'\" some; other; commands" > some_file 2>&1'
+
+++++++ File variables
+
+All eval cmd should exit their content to a file called "$RUNDIR/osync.$FUNCNAME.$SCRIPT_PID"
+Dots are used instead of '_' so variables can be separated with a forbidden char in variables, so they get detected.
+
+++++++ Finding code errors
+
+Use shellcheck.net now and then (ignore SC2086 in our case)
+
+Use a low tech approach to find uneven number of quotes per line
+
+tr -cd "'\n" < my_bash_file.sh | awk 'length%2==1 {print NR, $0}'
+tr -cd "\"\n" < my_bash_file.sh | awk 'length%2==1 {print NR, $0}'
+
+++++++ ofunctions
+
+As obackup and osync share alot of common functions, ofunctions.sh will host all shared code.
+Dev programs n_osync.sh and n_obackup.sh will source ofunctions.sh
+Release programs will still include ofunctions.sh in order to enhance ease of use.
+

LICENCE.TXT

@@ -1,4 +1,4 @@
-Copyright (c) 2013, Orsiris "Ozy" de Jong. ozy@netpower.fr
+Copyright (c) 2013-2016, Orsiris "Ozy" de Jong. ozy@netpower.fr
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

README.md

@@ -1,12 +1,11 @@
 obackup
 =======
 
-A small robust file & database backup script for local to local or remote to local backups via ssh.
+A robust file & database backup script that works for local and remote push or pull backups via ssh.
-Works especially well for multiple virtualhost backups with 'backup divide task' functionnality.
 
 ## About
 
-OBackup is designed from ground to make the backup process as reliable as possible.
+obackup is designed to make the backup process as reliable as possible.
 It divides the whole backup process into tasks, allowing each task to execute for a certain amount of time.
 If a task doesn't finish in time, it's stopped and the next task in list is processed.
 Before a task gets stopped, a first warning message is generated telling the task takes too long.
@@ -14,22 +13,19 @@ Every action gets logged, and if a warning has been generated, a task gets stopp
 
 Remote backups are initiated from the backup server instead of the production server, so hacked servers won't get ssh access to the backup server.
 
-OBackup can enumerate and backup all MariaDB / MySQL databases present on a server.
+obackup can enumerate and backup all MariaDB / MySQL databases present on a server.
 It can also enumarate all subdirectories of a given path and process them as separate tasks (usefull for multiple vhosts).
 It will do several checks before launching a backup like execution checks, dryruns, checking backup size and available local disk space.
 
-Obackup can execute local and remote commands before and after backup execution,
+obackup can execute local and remote commands before and after backup execution,
 thus providing an easy way to handle snapshots (see https://github.com/deajan/zsnap for a zfs snapshot management script).
 It may also rotate backups for you.
 
-As of today, obackup has been tested successfully on RHEL / CentOS 5, CentOS 6, Debian 6.0.7 and Linux Mint 14.
+As of today, obackup has been tested successfully on RHEL / CentOS 5, 6 and 7, Debian 6 and 7, Linux Mint 14 and 17, FreeBSD 8.3 and 10.3.
-Currently, Obackup also runs on FreeBSD and Windows MSYS environment, altough it is not fully tested yet.
+Currently, obackup also runs on MacOSX and Windows MSYS environment.
-
-Feel free to drop me a mail for limited support in my free time.
 
 ## Warning
 
-Starting with Obackup 1.84RC4, the default behavior is modified.
 Obackup now follows symlinks and treats them as the referent files / dirs, following symlinks even outside the backup root, which IMHO is more secure in terms of backups.
 You may disable this behavior in the config file.
 
@@ -38,10 +34,11 @@ You may disable this behavior in the config file.
 You can download the latest obackup script from authors website.
 You may also clone the following git which will maybe have some more recent builds.
 
-    $ git clone git://github.com/deajan/obackup.git
+    $ git clone -b "v2.0" git://github.com/deajan/obackup.git
-    $ chmod +x ./obackup.sh
+    $ cd obackup
+    $ ./install.sh
 
-Obackup needs to run with bash shell, using any other shell will most probably fail.  
+obackup needs to run with bash shell, using any other shell will most probably fail.  
 Once you have grabbed a copy, just edit the config file with your favorite text editor to setup your environment and you're ready to run.
 A detailled documentation can be found on the author's site.
 You can run multiple instances of obackup scripts with different backup environments. Just create another configuration file,
@@ -75,7 +72,7 @@ You may mix "--silent" and "--verbose" parameters to output verbose input only i
 ## Final words
 
 Backup tasks aren't always reliable, connectivity loss, insufficient disk space, hacked servers with tons of unusefull stuff to backup... Anything can happen.
-Obackup will sent your a warning email for every issue it can handle.
+obackup will sent your a warning email for every issue it can handle.
 Nevertheless, you should assure yourself that your backup tasks will get done the way you meant it. Also, a backup isn't valuable until you're sure
 you can successfully restore. Try to restore your backups to check whether everything is okay. Backups will keep file permissions and owners,
 but may loose ACLs if destination file system won't handle them. 
@@ -83,7 +80,4 @@ but may loose ACLs if destination file system won't handle them.
 ## Author
 
 Feel free to mail me for limited support in my free time :)
-Orsiris "Ozy" de Jong | ozy@netpower.fr
+Orsiris de Jong | ozy@netpower.fr
-
-
- 

dev/merge.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+## MERGE 2016080601
+
+## Merges ofunctions.sh and n_program.sh into program.sh
+## Adds installer
+
+PROGRAM=obackup
+VERSION=$(grep "PROGRAM_VERSION=" n_$PROGRAM.sh)
+VERSION=${VERSION#*=}
+
+PARANOIA_DEBUG_LINE="__WITH_PARANOIA_DEBUG"
+PARANOIA_DEBUG_BEGIN="#__BEGIN_WITH_PARANOIA_DEBUG"
+PARANOIA_DEBUG_END="#__END_WITH_PARANOIA_DEBUG"
+MINIMUM_FUNCTION_BEGIN="#### MINIMAL-FUNCTION-SET BEGIN ####"
+MINIMUM_FUNCTION_END="#### MINIMAL-FUNCTION-SET END ####"
+
+function Unexpand {
+	unexpand n_$PROGRAM.sh > tmp_$PROGRAM.sh
+}
+
+function MergeAll {
+
+	sed "/source \"\.\/ofunctions.sh\"/r ofunctions.sh" tmp_$PROGRAM.sh | grep -v 'source "./ofunctions.sh"' > debug_$PROGRAM.sh
+	chmod +x debug_$PROGRAM.sh
+}
+
+function MergeMinimum {
+        sed -n "/$MINIMUM_FUNCTION_BEGIN/,/$MINIMUM_FUNCTION_END/p" ofunctions.sh > tmp_minimal.sh
+        sed "/source \"\.\/ofunctions.sh\"/r tmp_minimal.sh" tmp_$PROGRAM.sh | grep -v 'source "./ofunctions.sh"' | grep -v "$PARANOIA_DEBUG_LINE" > debug_$PROGRAM.sh
+	rm -f tmp_minimal.sh
+        chmod +x debug_$PROGRAM.sh
+}
+
+
+function CleanDebug {
+
+# sed explanation
+#/pattern1/{         # if pattern1 is found
+#    p               # print it
+#    :a              # loop
+#        N           # and accumulate lines
+#    /pattern2/!ba   # until pattern2 is found
+#    s/.*\n//        # delete the part before pattern2
+#}
+#p
+
+	sed -n '/'$PARANOIA_DEBUG_BEGIN'/{p; :a; N; /'$PARANOIA_DEBUG_END'/!ba; s/.*\n//}; p' debug_$PROGRAM.sh | grep -v "$PARANOIA_DEBUG_LINE" > ../$PROGRAM.sh
+	chmod +x ../$PROGRAM.sh
+}
+
+function CopyCommons {
+	sed "s/\[prgname\]/$PROGRAM/g" common_install.sh > ../tmp_install.sh
+	sed "s/\[version\]/$VERSION/g" ../tmp_install.sh > ../install.sh
+	if [ -f "common_batch.sh" ]; then
+		sed "s/\[prgname\]/$PROGRAM/g" common_batch.sh > ../$PROGRAM-batch.sh
+	fi
+	chmod +x ../install.sh
+	chmod +x ../$PROGRAM-batch.sh
+	rm -f ../tmp_install.sh
+}
+
+Unexpand
+if [ "$PROGRAM" == "osync" ] || [ "$PROGRAM" == "obackup" ]; then
+	MergeAll
+else
+	MergeMinimum
+fi
+CleanDebug
+CopyCommons
+rm -f tmp_$PROGRAM.sh

host_backup.conf.example

@@ -1,73 +1,79 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
-###### Remote (or local) backup script for files & databases
+###### obackup - Local or Remote, push or pull backup script for files & mysql databases
-###### (L) 2013 by Ozy de Jong (www.netpower.fr)
+###### (C) 2013-2016 by Orsiris de Jong (www.netpower.fr)
-###### Config file rev 2015090801
+###### obackup v2.x config file rev 2016052501
 
-## ---------- GENERAL BACKUP OPTIONS
+###### GENERAL BACKUP OPTIONS
 
-## Backup identification name.
+## Backup identification string.
-BACKUP_ID="your backup name"
+INSTANCE_ID="test-backup"
 
-## Log file location. Leaving this empty will create log file at /var/log/obackup_version_BACKUP_ID.log (or current directory if /var/log doesn't exist)
+## Log file location. Leaving this empty will create log file at /var/log/obackup.INSTANCE_ID.log (or current directory if /var/log doesn't exist).
 LOGFILE=""
 
-## Backup databases
+## Elements to backup
-BACKUP_SQL=no
+SQL_BACKUP=yes
-## Backup files
+FILE_BACKUP=yes
-BACKUP_FILES=yes
 
-## ---------- LOCAL BACKUP STORAGE OPTIONS
+## Backups can be done local, pulled from another server or pushed to a backup server. Available options are [local,pull,push].
+## Pulled backups are the safest option, as the backup server contains the RSA key and cannot be compromised by another server.
+BACKUP_TYPE=local
 
-## Local storage paths where to put backups
+###### BACKUP STORAGE
-LOCAL_SQL_STORAGE="/home/storage/backup/sql"
+
-LOCAL_FILE_STORAGE="/home/storage/backup/files"
+## Storage paths of the backups (absolute paths of the local or remote system)
+SQL_STORAGE="/home/storage/backup/sql"
+FILE_STORAGE="/home/storage/backup/files"
+
+## Backup encryption using GPG and duplicity. Feature not ready yet.
+ENCRYPTION=no
 
 ## Create backup directories if they do not exist
 CREATE_DIRS=yes
 
 ## Keep absolute source path in your backup, eg: /your/backup/storage/the/remote/server/files
-## You should leave this enabled if you intend to use 'backup task division' functionality of OBackup, or everything will end up in the same directory.
+## You should leave this enabled if you intend to use 'backup task division' functionality of oBackup, or everything will end up in the same directory.
-LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=yes
+KEEP_ABSOLUTE_PATHS=yes
 
-## Generate an alert if backup size is lower than given value in Kb (this can also help identifying empty mount dirs)
+## Generate an alert if backup size is lower than given value in Kb (this can also help identifying empty mount dirs).
 BACKUP_SIZE_MINIMUM=1024
-## You may disable testing backup size
-DISABLE_GET_BACKUP_FILE_SIZE=no
 
-## Generate an alert if local storage free space is lower than given value in Kb. Keep in mind that disabling backup file size test will only test min space against SQL backup size.
+## Check backup size before proceeding
-LOCAL_STORAGE_WARN_MIN_SPACE=1048576
+GET_BACKUP_SIZE=yes
 
-## ---------- MISC OPTIONS
+## Generate an alert if storage free space is lower than given value in Kb.
+## Keep in mind that disabling backup file size test will only test min space against SQL backup size.
+SQL_WARN_MIN_SPACE=1048576
+FILE_WARN_MIN_SPACE=1048576
 
-## Bandwidth limit Kbytes / second for file backups. Leave 0 to disable limitation.
+###### REMOTE ONLY OPTIONS
-BANDWIDTH=0
 
-## If enabled, file backups will be processed as superuser. See documentation for /etc/sudoers configuration ("find", "du" and "rsync" need to be allowed). Requiretty needs to be disabled.
+## In case of pulled or pushed backups, remote system URI needs to be supplied.
-SUDO_EXEC=no
+REMOTE_SYSTEM_URI="ssh://backupuser@remote.system.tld:22/"
-## Paranoia option. Don't change this unless you read the documentation.
-RSYNC_EXECUTABLE=rsync
 
-## ---------- REMOTE BACKUP OPTIONS
+## You can specify a RSA key (please use full path). If not defined, the default ~/.ssh/id_rsa will be used. See documentation for further information.
+SSH_RSA_PRIVATE_KEY="/root/.ssh/id_rsa"
 
-## The following options allow this Obackup instance to connect to a remote system via an ssh tunnel.
-## Needs public RSA key need to be put into ~/.ssh/authorized_keys in remote users home directory.
-REMOTE_BACKUP=no
-SSH_RSA_PRIVATE_KEY=~/.ssh/id_rsa
-REMOTE_USER=backupuser
-REMOTE_HOST=yourhost.local
-REMOTE_PORT=22
 ## ssh compression should be used unless your remote connection is good enough (LAN)
 SSH_COMPRESSION=yes
+
+## Ignore ssh known hosts verification. DANGER WILL ROBINSON DANGER: This can lead to security risks. Only enable if you know what you're doing.
+SSH_IGNORE_KNOWN_HOSTS=no
+
 ## Remote rsync executable path. Leave this empty in most cases
 RSYNC_REMOTE_PATH=""
 
 ## Check for connectivity to remote host before launching remote backup tasks. Be sure the hosts responds to ping. Failing to ping will skip current task.
 REMOTE_HOST_PING=yes
+
 ## Check for internet access by pinging one or more 3rd party hosts before remote backup tasks. Leave empty if you don't want this check to be be performed. Failing to ping will skip current task.
 REMOTE_3RD_PARTY_HOSTS="www.kernel.org www.google.com"
 
-## ---------- DATABASE BACKUP OPTIONS
+## If enabled, commands will be executed as superuser on remote side. See documentation for /etc/sudoers configuration ("find", "du", "tee" and "rsync" need to be allowed). Requiretty needs to be disabled.
+SUDO_EXEC=no
+
+###### DATABASE SPECIFIC OPTIONS
 
 ## Database backup user
 SQL_USER=backupuser
@@ -75,9 +81,9 @@ SQL_USER=backupuser
 ## Enabling the following option will save all databases on local or remote given SQL instance except the ones specified in the exclude list.
 ## Every found database will be backed up as separate backup task.
 DATABASES_ALL=yes
-DATABASES_ALL_EXCLUDE_LIST="test;mysql"
+DATABASES_ALL_EXCLUDE_LIST="test"
 ## Alternatively, if DATABASES_ALL=no, you can specify a list of databases to backup separated by spaces.
-DATABASES_LIST=""
+#DATABASES_LIST="somedatabase"
 
 ## Max backup execution time per Database task. Soft max exec time generates a warning only. Hard max exec time generates a warning and stops current backup task.
 ## If a task gets stopped, next one in the task list gets executed. Time is specified in seconds.
@@ -88,30 +94,31 @@ HARD_MAX_EXEC_TIME_DB_TASK=7200
 ## Generally, level 5 is a good compromise between cpu, memory hunger and compress ratio. Gzipped files are set to be rsyncable.
 COMPRESSION_LEVEL=3
 
-## SQL Dump compression should be done on remote side but can also be done locally to lower remote system usage (will take more bandwidth, check for ssh compression)
+###### FILES SPECIFIC OPTIONS
-COMPRESSION_REMOTE=yes
 
-## ---------- FILES BACKUP OPTIONS
+## File backups are divided in tasks. Every directory in DIRECTORY_LIST will be processed as a unique task.
+## Every subdirectory of each directory in RECURSIVE_DIRECTORY_LIST will be processed as a unique task.
+## Example: RECURSIVE_DIRECTORY_LIST="/home;/var" will create backup tasks tasks "/home/dir1, "/home/dir2", ... "/home/dirN", "/var/log", "/var/lib"... "/var/something".
+## You can exclude directories from the avove backup task creation, ex: avoid backing up "/home/dir2" by adding it to RECURSIVE_EXCLUDE_LIST.
 
-## Directories backup list. List of semicolon separated directories that will be backed up recursively. Every directory will be processed as one backup task.
+## Directories backup list. List of semicolon separated directories that will be backed up.
-DIRECTORIES_SIMPLE_LIST="/var/named;/var/lib"
+DIRECTORY_LIST="/var/named"
+RECURSIVE_DIRECTORY_LIST="/home"
+RECURSIVE_EXCLUDE_LIST="/home/backupuser;/home/lost+found"
 
-## There's a special backup schema in Obackup called 'backup task division'  which creates one backup task per level 1 subdirectory of a directory.
+## Rsync exclude / include order (the option set here will be set first, eg: include will make include then exclude patterns)
-## This is VERY useful to backup multiple virtualhosts as separate tasks without having to specify each one separately.
+RSYNC_PATTERN_FIRST=include
-## This may also be useful dividing big data directories in subdirectories tasks.
 
-## Directories backup task division backup: Semicolon separated directories of which every level 1 subdirectory will be backed up recursively as a separate backup task.
+## List of files / directories to incldue / exclude from sync on both sides (see rsync patterns, wildcards work).
-## Example: "/home;/var" will create tasks "/home/dir1", "/home/dir2", ... "/home/dirN", "/var/log", "/var/lib"... "/var/whatever"
+## Paths are relative to sync dirs. List elements are separated by a semicolon.
-DIRECTORIES_RECURSE_LIST="/home"
+RSYNC_INCLUDE_PATTERN=""
-## You may optionally exclude subdirectories from task division. On the above example you could exclude /home/dir2 by adding it to DIRECTORIES_RECURSE_EXCLUDE_LIST
+RSYNC_EXCLUDE_PATTERN=""
-DIRECTORIES_RECURSE_EXCLUDE_LIST="/home/backupuser;/home/lost+found"
+#RSYNC_EXCLUDE_PATTERN="tmp;archives"
 
-## Rsync exclude patterns, used by simple and division lists
+## Files that contains lists of files / directories to include / exclude from sync on both sides. Leave this empty if you don't want to use an exclusion file.
-RSYNC_EXCLUDE_PATTERN="*/tmp;*/ftp/www/cache/cachefs;*/sessions"
-
-## File that contains the list of directories or files to exclude from sync on both sides. Leave this empty if you don't want to use an exclusion file.
 ## This file has to be in the same directory as the config file
 ## Paths are relative to sync dirs. One element per line.
+RSYNC_INCLUDE_FROM=""
 RSYNC_EXCLUDE_FROM=""
 #RSYNC_EXCLUDE_FROM="exclude.list"
 
@@ -147,18 +154,27 @@ DELETE_VANISHED_FILES=no
 ## Use delta copy algortithm (usefull when local paths are network drives), defaults to yes
 DELTA_COPIES=yes
 
-## ---------- ALERT OPTIONS
+## Bandwidth limit Kbytes / second for file backups. Leave 0 to disable limitation.
+BANDWIDTH=0
+
+## Paranoia option. Don't change this unless you read the documentation.
+RSYNC_EXECUTABLE=rsync
+
+###### ALERT OPTIONS
 
 ## Alert email addresses separated by a space character
 DESTINATION_MAILS="your@mail.address"
 
-## Windows (MSYS environment) only mail options (used by sendemail.exe)
+## Windows specific (msys / cygwin environment) only mail options (used with mailsend.exe from muquit, http://github.com/muquit/mailsend or from sendemail.exe from Brandon Zehm, http://caspian.dotconf.net/menu/Software/SendEmail/
-SENDER_MAIL="alert@your.system"
+SENDER_MAIL="alert@your.system.tld"
-SMTP_SERVER=smtp.your.isp.com
+SMTP_SERVER=smtp.your.isp.tld
+SMTP_PORT=25
+# encryption can be tls, ssl or none
+SMTP_ENCRYPTION=none
 SMTP_USER=
 SMTP_PASSWORD=
 
-## ---------- GENERAL BACKUP OPTIONS
+###### GENERAL BACKUP OPTIONS
 
 ## Max execution time of whole backup process. Soft max exec time generates a warning only.
 ## Hard max exec time generates a warning and stops the whole backup execution.
@@ -166,10 +182,12 @@ SOFT_MAX_EXEC_TIME_TOTAL=30000
 HARD_MAX_EXEC_TIME_TOTAL=36000
 
 ## Backup Rotation. You may rotate backups if you don't use snapshots on your backup server.
-ROTATE_BACKUPS=no
+ROTATE_SQL_BACKUPS=no
-ROTATE_COPIES=7
+ROTATE_SQL_COPIES=7
+ROTATE_FILE_BACKUPS=no
+ROTATE_FILE_COPIES=7
 
-## ---------- EXECUTION HOOKS
+###### EXECUTION HOOKS
 
 ## Commands can will be run before and / or after backup execution (remote execution will only happen if REMOTE_BACKUP is set).
 ## This is useful to make a snapshot before backing up data, or even handle snapshots of backed up data.
@@ -185,3 +203,6 @@ MAX_EXEC_TIME_PER_CMD_AFTER=0
 
 ## Stops whole backup execution if one of the above commands fail
 STOP_ON_CMD_ERROR=no
+
+## Run local and remote after backup cmd's even on failure
+RUN_AFTER_CMD_ON_ERROR=no

install.sh

@@ -1,25 +1,311 @@
 #!/usr/bin/env bash
 
-SCRIPT_BUILD=2015082501
+PROGRAM=obackup
+PROGRAM_VERSION=2.0
+PROGRAM_BINARY=$PROGRAM".sh"
+PROGRAM_BATCH=$PROGRAM"-batch.sh"
+SCRIPT_BUILD=2016052601
 
-## Obackup install script
+## osync / obackup / pmocr / zsnap install script
-## Tested on RHEL / CentOS 6 & 7
+## Tested on RHEL / CentOS 6 & 7, Fedora 23, Debian 7 & 8, Mint 17 and FreeBSD 8 & 10
 ## Please adapt this to fit your distro needs
 
-if [ "$(whoami)" != "root" ]
+#TODO: silent mode and no stats mode
-then
+
-  echo "Must be run as root."
+CONF_DIR=/etc/$PROGRAM
+BIN_DIR=/usr/local/bin
+SERVICE_DIR_INIT=/etc/init.d
+# Should be /usr/lib/systemd/system, but /lib/systemd/system exists on debian & rhel / fedora
+SERVICE_DIR_SYSTEMD_SYSTEM=/lib/systemd/system
+SERVICE_DIR_SYSTEMD_USER=/etc/systemd/user
+
+## osync specific code
+OSYNC_SERVICE_FILE_INIT="osync-srv"
+OSYNC_SERVICE_FILE_SYSTEMD_SYSTEM="osync-srv@.service"
+OSYNC_SERVICE_FILE_SYSTEMD_USER="osync-srv@.service.user"
+
+## pmocr specfic code
+PMOCR_SERVICE_FILE_INIT="pmocr-srv"
+PMOCR_SERVICE_FILE_SYSTEMD_SYSTEM="pmocr-srv.service"
+
+## Generic code
+
+## Default log file
+if [ -w /var/log ]; then
+        LOG_FILE="/var/log/$PROGRAM-install.log"
+elif ([ "$HOME" != "" ] && [ -w "$HOME" ]); then
+        LOG_FILE="$HOME/$PROGRAM-install.log"
+else
+    	LOG_FILE="./$PROGRAM-install.log"
+fi
+
+# Generic quick logging function
+function _QuickLogger {
+        local value="${1}"
+       	local destination="${2}" # Destination: stdout, log, both
+
+        if ([ "$destination" == "log" ] || [ "$destination" == "both" ]); then
+                echo -e "$(date) - $value" >> "$LOG_FILE"
+        elif ([ "$destination" == "stdout" ] || [ "$destination" == "both" ]); then
+                echo -e "$value"
+       	fi
+}
+
+function QuickLogger {
+	local value="${1}"
+
+	if [ "$_SILENT" -eq 1 ]; then
+		_QuickLogger "$value" "log"
+	else
+		_QuickLogger "$value" "stdout"
+	fi
+}
+
+function urlencode() {
+    # urlencode <string>
+
+    local LANG=C
+    local length="${#1}"
+    for (( i = 0; i < length; i++ )); do
+        local c="${1:i:1}"
+        case $c in
+            [a-zA-Z0-9.~_-]) printf "$c" ;;
+            *) printf '%%%02X' "'$c" ;; 
+        esac
+    done
+}
+
+function SetOSSettings {
+	USER=root
+
+	local local_os_var
+
+	local_os_var="$(uname -spio 2>&1)"
+	if [ $? != 0 ]; then
+		local_os_var="$(uname -v 2>&1)"
+		if [ $? != 0 ]; then
+			local_os_var="$(uname)"
+		fi
+	fi
+
+	case $local_os_var in
+		*"BSD"*)
+		GROUP=wheel
+		;;
+		*"Darwin"*)
+		GROUP=admin
+		;;
+		*)
+		GROUP=root
+		;;
+		*"MINGW32"*|*"CYGWIN"*)
+		USER=""
+		GROUP=""
+		;;
+	esac
+
+	if ([ "$USER" != "" ] && [ "$(whoami)" != "$USER" ]); then
+	  	QuickLogger "Must be run as $USER."
 		exit 1
 	fi
 
-mkdir /etc/obackup
+	OS=$(urlencode "$local_os_var")
-cp ./host_backup.conf /etc/obackup/host_backup.conf.example
+}
-cp ./exclude.list.example /etc/obackup
-cp ./obackup.sh /usr/local/bin
-cp ./obackup-batch.sh /usr/local/bin
-cp ./ssh_filter.sh /usr/local/bin
-chmod 755 /usr/local/bin/obackup.sh
-chmod 755 /usr/local/bin/obackup-batch.sh
-chmod 755 /usr/local/bin/ssh_filter.sh
-chown root:root /usr/local/bin/ssh_filter.sh
 
+function GetInit {
+	if [ -f /sbin/init ]; then
+		if file /sbin/init | grep systemd > /dev/null; then
+			init="systemd"
+		else
+			init="initV"
+		fi
+	else
+		QuickLogger "Can't detect initV or systemd. Service files won't be installed. You can still run $PROGRAM manually or via cron."
+		init="none"
+	fi
+}
+
+function CreateConfDir {
+	if [ ! -d "$CONF_DIR" ]; then
+		mkdir "$CONF_DIR"
+		if [ $? == 0 ]; then
+			QuickLogger "Created directory [$CONF_DIR]."
+		else
+			QuickLogger "Cannot create directory [$CONF_DIR]."
+			exit 1
+		fi
+	else
+		QuickLogger "Config directory [$CONF_DIR] exists."
+	fi
+}
+
+function CopyExampleFiles {
+	if [ -f "./sync.conf.example" ]; then
+		cp "./sync.conf.example" "/etc/$PROGRAM/sync.conf.example"
+	fi
+
+	if [ -f "./host_backup.conf.example" ]; then
+		cp "./host_backup.conf.example" "/etc/$PROGRAM/host_backup.conf.example"
+	fi
+
+	if [ -f "./exlude.list.example" ]; then
+		cp "./exclude.list.example" "/etc/$PROGRAM"
+	fi
+
+	if [ -f "./snapshot.conf.example" ]; then
+		cp "./snapshot.conf.example" "/etc/$PROGRAM/snapshot.conf.example"
+	fi
+}
+
+function CopyProgram {
+	cp "./$PROGRAM_BINARY" "$BIN_DIR"
+	if [ $? != 0 ]; then
+		QuickLogger "Cannot copy $PROGRAM_BINARY to [$BIN_DIR]. Make sure to run install script in the directory containing all other files."
+		QuickLogger "Also make sure you have permissions to write to [$BIN_DIR]."
+		exit 1
+	else
+		chmod 755 "$BIN_DIR/$PROGRAM_BINARY"
+		QuickLogger "Copied $PROGRAM_BINARY to [$BIN_DIR]."
+	fi
+
+	if [ -f "./$PROGRAM_BATCH" ]; then
+		cp "./$PROGRAM_BATCH" "$BIN_DIR"
+		if [ $? != 0 ]; then
+			QuickLogger "Cannot copy $PROGRAM_BATCH to [$BIN_DIR]."
+		else
+			chmod 755 "$BIN_DIR/$PROGRAM_BATCH"
+			QuickLogger "Copied $PROGRAM_BATCH to [$BIN_DIR]."
+		fi
+	fi
+
+	if [  -f "./ssh_filter.sh" ]; then
+		cp "./ssh_filter.sh" "$BIN_DIR"
+		if [ $? != 0 ]; then
+			QuickLogger "Cannot copy ssh_filter.sh to [$BIN_DIR]."
+		else
+			chmod 755 "$BIN_DIR/ssh_filter.sh"
+			if ([ "$USER" != "" ] && [ "$GROUP" != "" ]); then
+				chown $USER:$GROUP "$BIN_DIR/ssh_filter.sh"
+			fi
+			QuickLogger "Copied ssh_filter.sh to [$BIN_DIR]."
+		fi
+	fi
+}
+
+function CopyServiceFiles {
+	# OSYNC SPECIFIC
+	if ([ "$init" == "systemd" ] && [ -f "./$OSYNC_SERVICE_FILE_SYSTEMD_SYSTEM" ]); then
+		cp "./$OSYNC_SERVICE_FILE_SYSTEMD_SYSTEM" "$SERVICE_DIR_SYSTEMD_SYSTEM" && cp "./$OSYNC_SERVICE_FILE_SYSTEMD_USER" "$SERVICE_DIR_SYSTEMD_USER/$SERVICE_FILE_SYSTEMD_SYSTEM"
+		if [ $? != 0 ]; then
+			QuickLogger "Cannot copy the systemd file to [$SERVICE_DIR_SYSTEMD_SYSTEM] or [$SERVICE_DIR_SYSTEMD_USER]."
+		else
+			QuickLogger "Created osync-srv service in [$SERVICE_DIR_SYSTEMD_SYSTEM] and [$SERVICE_DIR_SYSTEMD_USER]."
+			QuickLogger "Can be activated with [systemctl start osync-srv@instance.conf] where instance.conf is the name of the config file in /etc/osync."
+			QuickLogger "Can be enabled on boot with [systemctl enable osync-srv@instance.conf]."
+			QuickLogger "In userland, active with [systemctl --user start osync-srv@instance.conf]."
+		fi
+	elif ([ "$init" == "initV" ] && [ -f "./$OSYNC_SERVICE_FILE_INIT" ]); then
+		cp "./$OSYNC_SERVICE_FILE_INIT" "$SERVICE_DIR_INIT"
+		if [ $? != 0 ]; then
+			QuickLogger "Cannot copy osync-srv to [$SERVICE_DIR_INIT]."
+		else
+			chmod 755 "$SERVICE_DIR_INIT/$OSYNC_SERVICE_FILE_INIT"
+			QuickLogger "Created osync-srv service in [$SERVICE_DIR_INIT]."
+			QuickLogger "Can be activated with [service $OSYNC_SERVICE_FILE_INIT start]."
+			QuickLogger "Can be enabled on boot with [chkconfig $OSYNC_SERVICE_FILE_INIT on]."
+		fi
+	fi
+
+	# PMOCR SPECIFIC
+	if ([ "$init" == "systemd" ] && [ -f "./$PMOCR_SERVICE_FILE_SYSTEMD_SYSTEM" ]); then
+		cp "./$PMOCR_SERVICE_FILE_SYSTEMD_SYSTEM" "$SERVICE_DIR_SYSTEMD_SYSTEM"
+		if [ $? != 0 ]; then
+			QuickLogger "Cannot copy the systemd file to [$SERVICE_DIR_SYSTEMD_SYSTEM] or [$SERVICE_DIR_SYSTEMD_USER]."
+		else
+			QuickLogger "Created pmocr-srv service in [$SERVICE_DIR_SYSTEMD_SYSTEM] and [$SERVICE_DIR_SYSTEMD_USER]."
+			QuickLogger "Can be activated with [systemctl start pmocr-srv] after configuring file options in [$BIN_DIR/$PROGRAM]."
+			QuickLogger "Can be enabled on boot with [systemctl enable pmocr-srv]."
+		fi
+	elif ([ "$init" == "initV" ] && [ -f "./$PMOCR_SERVICE_FILE_INIT" ]); then
+		cp "./$PMOCR_SERVICE_FILE_INIT" "$SERVICE_DIR_INIT"
+		if [ $? != 0 ]; then
+			QuickLogger "Cannot copy pmoct-srv to [$SERVICE_DIR_INIT]."
+		else
+			chmod 755 "$SERVICE_DIR_INIT/$PMOCR_SERVICE_FILE_INIT"
+			QuickLogger "Created osync-srv service in [$SERVICE_DIR_INIT]."
+			QuickLogger "Can be activated with [service $PMOCR_SERVICE_FILE_INIT start]."
+			QuickLogger "Can be enabled on boot with [chkconfig $PMOCR_SERVICE_FILE_INIT on]."
+		fi
+	fi
+}
+
+function Statistics {
+        if type wget > /dev/null; then
+                wget -qO- "$STATS_LINK" > /dev/null 2>&1
+                if [ $? == 0 ]; then
+                        return 0
+                fi
+	fi
+
+        if type curl > /dev/null; then
+                curl "$STATS_LINK" -o /dev/null > /dev/null 2>&1
+                if [ $? == 0 ]; then
+                        return 0
+                fi
+	fi
+
+       	QuickLogger "Neiter wget nor curl could be used for. Cannot run statistics. Use the provided link please."
+        return 1
+}
+
+function Usage {
+	echo "Installs $PROGRAM into $BIN_DIR"
+	echo "options:"
+	echo "--silent		Will log and bypass user interaction."
+	echo "--no-stats	Used with --silent in order to refuse sending anonymous install stats."
+	exit 127
+}
+
+_SILENT=0
+_STATS=1
+for i in "$@"
+do
+	case $i in
+		--silent)
+		_SILENT=1
+		;;
+		--no-stats)
+		_STATS=0
+		;;
+		--help|-h|-?)
+		Usage
+	esac
+done
+
+SetOSSettings
+CreateConfDir
+CopyExampleFiles
+CopyProgram
+GetInit
+CopyServiceFiles
+
+STATS_LINK="http://instcount.netpower.fr?program=$PROGRAM&version=$PROGRAM_VERSION&os=$OS"
+
+QuickLogger "$PROGRAM installed. Use with $BIN_DIR/$PROGRAM"
+if [ $_STATS -eq 1 ]; then
+	if [ $_SILENT -eq 1 ]; then
+		Statistics
+	else
+		QuickLogger "In order to make install statistics, the script would like to connect to $STATS_LINK"
+		read -r -p "No data except those in the url will be send. Allow [Y/n]" response
+		case $response in
+			[nN])
+			exit
+			;;
+			*)
+			Statistics
+			exit $?
+			;;
+		esac
+	fi
+fi

obackup-batch.sh

@@ -1,15 +1,15 @@
 #!/usr/bin/env bash
+SUBPROGRAM=obackup
+PROGRAM="$SUBPROGRAM-batch" # Batch program to run osync / obackup instances sequentially and rerun failed ones
+AUTHOR="(L) 2013-2016 by Orsiris de Jong"
+CONTACT="http://www.netpower.fr - ozy@netpower.fr"
+PROGRAM_BUILD=2016052501
 
-PROGRAM="Obackup-batch" # Batch program to run obackup instances sequentially and rerun failed ones
+## Runs an osync /obackup instance for every conf file found
-AUTHOR="(L) 2013-2014 by Orsiris \"Ozy\" de Jong"
-CONTACT="http://www.netpower.fr/obackup - ozy@netpower.fr"
-PROGRAM_BUILD=2508201501
-
-## Runs an obackup instance for every conf file found
 ## If an instance fails, run it again if time permits
 
-## Configuration file path. The path where all the obackup conf files are, usually /etc/obackup
+## Configuration file path. The path where all the osync / obackup conf files are, usually /etc/osync or /etc/obackup
-CONF_FILE_PATH=/etc/obackup
+CONF_FILE_PATH=/etc/$SUBPROGRAM
 
 ## If maximum execution time is not reached, failed instances will be rerun. Max exec time is in seconds. Example is set to 10 hours.
 MAX_EXECUTION_TIME=36000
@@ -18,60 +18,70 @@ MAX_EXECUTION_TIME=36000
 MAX_RERUNS=3
 
 ## Log file path
-if [ -w /var/log ]
+if [ -w /var/log ]; then
-then
+	LOG_FILE=/var/log/$SUBPROGRAM-batch.log
-        LOG_FILE=/var/log/obackup-batch.log
 else
-        LOG_FILE=./obackup-batch.log
+	LOG_FILE=./$SUBPROGRAM-batch.log
 fi
 
 # No need to edit under this line ##############################################################
 
-function Log
+function _logger {
-{
+	local value="${1}" # What to log
-        prefix="TIME: $SECONDS - "
+	echo -e "$value" >> "$LOG_FILE"
-        echo -e "$prefix$1" >> "$LOG_FILE"
+}
 
-        if [ $silent -eq 0 ]
+function Logger {
-        then
+	local value="${1}" # What to log
-                echo -e "$prefix$1"
+	local level="${2}" # Log level: DEBUG, NOTICE, WARN, ERROR, CRITIAL
+
+	prefix="$(date) - "
+
+	if [ "$level" == "CRITICAL" ]; then
+		_logger "$prefix\e[41m$value\e[0m"
+	elif [ "$level" == "ERROR" ]; then
+		_logger "$prefix\e[91m$value\e[0m"
+	elif [ "$level" == "WARN" ]; then
+		_logger "$prefix\e[93m$value\e[0m"
+	elif [ "$level" == "NOTICE" ]; then
+		_logger "$prefix$value"
+	elif [ "$level" == "DEBUG" ]; then
+		if [ "$DEBUG" == "yes" ]; then
+			_logger "$prefix$value"
+		fi
+	else
+		_logger "\e[41mLogger function called without proper loglevel.\e[0m"
+		_logger "$prefix$value"
 	fi
 }
 
-function CheckEnvironment
+function CheckEnvironment {
-{
+	## osync / obackup executable full path can be set here if it cannot be found on the system
-        ## Obackup executable full path can be set here if it cannot be found on the system
+	if ! type $SUBPROGRAM.sh > /dev/null 2>&1
-        if ! type -p obackup.sh > /dev/null 2>&1
 	then
-                if [ -f /usr/local/bin/obackup.sh ]
+		if [ -f /usr/local/bin/$SUBPROGRAM.sh ]
 		then
-                        OBACKUP_EXECUTABLE=/usr/local/bin/obackup.sh
+			SUBPROGRAM_EXECUTABLE=/usr/local/bin/$SUBPROGRAM.sh
-                elif [ -f ./obackup.sh ]
-                then
-                        OBACKUP_EXECUTABLE=./obackup.sh
 		else
-                        Log "Could not find obackup.sh"
+			Logger "Could not find $SUBPROGRAM.sh" "CRITICAL"
 			exit 1
 		fi
 	else
-                OBACKUP_EXECUTABLE=$(type -p obackup.sh)
+		SUBPROGRAM_EXECUTABLE=$(type -p $SUBPROGRAM.sh)
 	fi
 
 	## Check for CONF_FILE_PATH
-        if [ ! -d "$CONF_FILE_PATH" ]
+	if [ ! -d "$CONF_FILE_PATH" ]; then
-        then
+		Logger "Cannot find conf file path $CONF_FILE_PATH" "CRITICAL"
-                Log "Cannot find conf file path $CONF_FILE_PATH"
 		Usage
 	fi
 }
 
-function Batch
+function Batch {
-{
 	## Get list of .conf files
-	for i in $(ls $CONF_FILE_PATH/*.conf)
+	for i in $CONF_FILE_PATH/*.conf
 	do
-		if [ "$RUN" == "" ]
+		if [ "$RUN" == "" ]; then
-		then
 			RUN="$i"
 		else
 			RUN=$RUN" $i"
@@ -81,21 +91,20 @@ function Batch
 	RERUNS=0
 	while ([ $MAX_EXECUTION_TIME -gt $SECONDS ] || [ $MAX_EXECUTION_TIME -eq 0 ]) && [ "$RUN" != "" ] && [ $MAX_RERUNS -gt $RERUNS ]
 	do
-		Log "Obackup instances will be run for: $RUN"
+		Logger "$SUBPROGRAM instances will be run for: $RUN" "NOTICE"
 		for i in $RUN
 		do
-			$OBACKUP_EXECUTABLE "$i" $opts
+			$SUBPROGRAM_EXECUTABLE "$i" $opts &
-			if [ $? != 0 ]
+			wait $!
-			then
+			if [ $? != 0 ]; then
-				Log "Run instance $(basename $i) failed"
+				Logger "Run instance $(basename $i) failed" "ERROR"
-				if [ "RUN_AGAIN" == "" ]
+				if [ "$RUN_AGAIN" == "" ]; then
-				then
 					RUN_AGAIN="$i"
 				else
 					RUN_AGAIN=$RUN_AGAIN" $i"
 				fi
 			else
-				Log "Run instance $(basename $i) succeed."
+				Logger "Run instance $(basename $i) succeed." "NOTICE"
 			fi
 		done
 		RUN="$RUN_AGAIN"
@@ -104,43 +113,36 @@ function Batch
 	done
 }
 
-function Usage
+function Usage {
-{
 	echo "$PROGRAM $PROGRAM_BUILD"
 	echo $AUTHOR
 	echo $CONTACT
 	echo ""
-        echo "Batch script to sequentially run obackup instances and rerun failed ones."
+	echo "Batch script to sequentially run osync or obackup instances and rerun failed ones."
-        echo "Usage: obackup-batch.sh [OPTIONS]"
+	echo "Usage: $SUBPROGRAM-batch.sh [OPTIONS]"
 	echo ""
 	echo "[OPTIONS]"
-	echo "--path=/path/to/conf      Path to obackup conf files, defaults to /etc/obackup"
+	echo "--path=/path/to/conf      Path to osync / obackup conf files, defaults to /etc/osync or /etc/obackup"
 	echo "--max-reruns=X            Number of runs  max for failed instances, (defaults to 3)"
-	echo "--max-exec-time=X         Retry failed instances only if max execution time not reached (defaults to 36000 seconds)"
+	echo "--max-exec-time=X         Retry failed instances only if max execution time not reached (defaults to 36000 seconds). Set to 0 to bypass execution time check."
-	echo "--no-maxtime		Run obackup without honoring conf file defined timeouts"
+	echo "--no-maxtime		Run osync / obackup without honoring conf file defined timeouts"
-        echo "--dry                     Will run obackup without actually doing anything; just testing"
+	echo "--dry                     Will run osync / obackup without actually doing anything; just testing"
-        echo "--silent                  Will run obackup without any output to stdout, used for cron jobs"
+	echo "--silent                  Will run osync / obackup without any output to stdout, used for cron jobs"
 	echo "--verbose                 Increases output"
 	exit 128
 }
 
-silent=0
-dry=0
-verbose=0
 opts=""
 for i in "$@"
 do
 	case $i in
 		--silent)
-                silent=1
 		opts=$opts" --silent"
 		;;
 		--dry)
-                dry=1
 		opts=$opts" --dry"
 		;;
 		--verbose)
-                verbose=1
 		opts=$opts" --verbose"
 		;;
 		--no-maxtime)
@@ -159,12 +161,12 @@ do
 		Usage
 		;;
 		*)
-		Log "Unknown param '$i'"
+		Logger "Unknown param '$i'" "CRITICAL"
 		Usage
 		;;
 	esac
 done
 
 CheckEnvironment
-Log "$(date) Obackup batch run"
+Logger "$(date) $SUBPROGRAM batch run" "NOTICE"
 Batch

ssh_filter.sh

@@ -1,95 +1,94 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
-##### Osync ssh command filter build 2015070203
+##### osync / obackup ssh command filter
 ##### This script should be located in /usr/local/bin in the remote system to sync / backup
 ##### It will filter the commands that can be run remotely via ssh.
 ##### Please chmod 755 and chown root:root this file
 
 ##### Obackup needed commands: rsync find du mysql mysqldump (sudo)
 ##### Osync needed commands: rsync find du echo mkdir rm if df (sudo)
+SCRIPT_BUILD=2016031401
 
 ## If enabled, execution of "sudo" command will be allowed.
 SUDO_EXEC=yes
 ## Paranoia option. Don't change this unless you read the documentation and still feel concerned about security issues.
 RSYNC_EXECUTABLE=rsync
 ## Enable other commands, useful for remote execution hooks like remotely creating snapshots.
-CMD1=
+CMD1=""
-CMD2=
+CMD2=""
-CMD3=
+CMD3=""
 
 LOG_FILE=~/.ssh/ssh_filter.log
 
-function Log
+function Log {
-{
 	DATE=$(date)
 	echo "$DATE - $1" >> $LOG_FILE
 }
 
-function Go
+function Go {
-{
+	eval "$SSH_ORIGINAL_COMMAND"
-	eval $SSH_ORIGINAL_COMMAND
 }
 
 case ${SSH_ORIGINAL_COMMAND%% *} in
 	"$RSYNC_EXECUTABLE")
 	Go ;;
+	"echo")
+	Go ;;
 	"find")
 	Go ;;
 	"du")
 	Go ;;
-	"mysql")
+	"mkdir")
 	Go ;;
-	"mysqldump")
+	"rm")
+	Go ;;
+	"df")
+	Go ;;
+	"mv")
 	Go ;;
 	"$CMD1")
-	if [ "$CMD1" != "" ]
+	if [ "$CMD1" != "" ]; then
-	then
+		Go
-		Go ;;
 	fi
+	;;
 	"$CMD2")
-	if [ "$CMD2" != "" ]
+	if [ "$CMD2" != "" ]; then
-	then
+		Go
-		Go ;;
 	fi
+	;;
 	"$CMD3")
-	if [ "$CMD3" != "" ]
+	if [ "$CMD3" != "" ]; then
-	then
+		Go
-		Go ;;
 	fi
+	;;
 	"sudo")
-	if [ "$SUDO_EXEC" == "yes" ]
+	if [ "$SUDO_EXEC" == "yes" ]; then
-	then
+		if [[ "$SSH_ORIGINAL_COMMAND" == "sudo $RSYNC_EXECUTABLE"* ]]; then
-		if [[ "$SSH_ORIGINAL_COMMAND" == "sudo $RSYNC_EXECUTABLE"* ]]
-		then
 			Go
-		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo du"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo du"* ]]; then
-		then
 			Go
-		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo find"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo find"* ]]; then
-		then
 			Go
-                elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo mysql"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo mkdir"* ]]; then
-                then
 			Go
-                elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo mysqldump"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo rm"* ]]; then
-                then
 			Go
-		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD1"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo echo"* ]]; then
-		then
+			Go
-			if [ "$CMD1" != "" ]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo df"* ]]; then
-			then
+			Go
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo mv"* ]]; then
+			Go
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD1"* ]]; then
+			if [ "$CMD1" != "" ]; then
 			Go
 			fi
-		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD2"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD2"* ]]; then
-		then
+			if [ "$CMD2" != "" ]; then
-			if [ "$CMD2" != "" ]
-			then
 			Go
 			fi
-		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD3"* ]]
+		elif [[ "$SSH_ORIGINAL_COMMAND" == "sudo $CMD3"* ]]; then
-		then
+			if [ "$CMD3" != "" ]; then
-			if [ "$CMD3" != "" ]
-			then
 			Go
 			fi
 		else

upgrade-v1.x-2.x.sh

@@ -0,0 +1,202 @@
+#!/usr/bin/env bash
+
+PROGRAM="obackup config file upgrade script"
+SUBPROGRAM="obackup"
+AUTHOR="(C) 2015 by Orsiris \"Ozy\" de Jong"
+CONTACT="http://www.netpower.fr/obacup - ozy@netpower.fr"
+OLD_PROGRAM_VERSION="v1.x"
+NEW_PROGRAM_VERSION="v2.x"
+PROGRAM_BUILD=2016041201
+
+## type -p does not work on platforms other than linux (bash). If if does not work, always as$
+if ! type "$BASH" > /dev/null; then
+        echo "Please run this script only with bash shell. Tested on bash >= 3.2"
+        exit 127
+fi
+
+function Usage {
+	echo "$PROGRAM $PROGRAM_BUILD"
+	echo $AUTHOR
+	echo $CONTACT
+	echo ""
+	echo "This script migrates $SUBPROGRAM $OLD_PROGRAM_VERSION config files to $NEW_PROGRAM_VERSION."
+	echo ""
+	echo "Usage: $0 /path/to/config_file.conf"
+	echo "Please make sure the config file is writable."
+	exit 128
+}
+
+function LoadConfigFile {
+	local config_file="${1}"
+
+	if [ ! -f "$config_file" ]; then
+		echo "Cannot load configuration file [$config_file]. Sync cannot start."
+		exit 1
+	elif [[ "$1" != *".conf" ]]; then
+		echo "Wrong configuration file supplied [$config_file]. Sync cannot start."
+		exit 1
+	else
+		egrep '^#|^[^ ]*=[^;&]*'  "$config_file" > "./$SUBPROGRAM.$FUNCNAME.$$"
+		source "./$SUBPROGRAM.$FUNCNAME.$$"
+		rm -f "./$SUBPROGRAM.$FUNCNAME.$$"
+	fi
+}
+
+function RewriteConfigFiles {
+	local config_file="${1}"
+
+	if ((! grep "BACKUP_ID=" $config_file > /dev/null) && ( ! grep "INSTANCE_ID=" $config_file > /dev/null)); then
+		echo "File [$config_file] does not seem to be a obackup config file."
+		exit 1
+	fi
+
+	echo "Backing up [$config_file] as [$config_file.save]"
+	cp -p "$config_file" "$config_file.save"
+	if [ $? != 0 ]; then
+		echo "Cannot backup config file."
+		exit 1
+	fi
+
+	echo "Rewriting config file $config_file"
+
+	sed -i'.tmp' 's/^BACKUP_ID=/INSTANCE_ID=/g' "$config_file"
+	sed -i'.tmp' 's/^BACKUP_SQL=/SQL_BACKUP=/g' "$config_file"
+	sed -i'.tmp' 's/^BACKUP_FILES=/FILE_BACKUP=/g' "$config_file"
+	sed -i'.tmp' 's/^LOCAL_SQL_STORAGE=/SQL_STORAGE=/g' "$config_file"
+	sed -i'.tmp' 's/^LOCAL_FILE_STORAGE=/FILE_STORAGE=/g' "$config_file"
+	if ! grep "^ENCRYPTION=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^FILE_STORAGE=*/a\'$'\n''ENCRYPTION=no\'$'\n''' "$config_file"
+	fi
+	sed -i'.tmp' 's/^DISABLE_GET_BACKUP_FILE_SIZE=no/GET_BACKUP_SIZE=yes/g' "$config_file"
+	sed -i'.tmp' 's/^DISABLE_GET_BACKUP_FILE_SIZE=yes/GET_BACKUP_SIZE=no/g' "$config_file"
+	sed -i'.tmp' 's/^LOCAL_STORAGE_KEEP_ABSOLUTE_PATHS=/KEEP_ABSOLUTE_PATHS=/g' "$config_file"
+	sed -i'.tmp' 's/^LOCAL_STORAGE_WARN_MIN_SPACE=/SQL_WARN_MIN_SPACE=/g' "$config_file"
+	if ! grep "^FILE_WARN_MIN_SPACE=" "$config_file" > /dev/null; then
+		VALUE=$(cat $config_file | grep "SQL_WARN_MIN_SPACE=")
+		VALUE=${VALUE#*=}
+		sed -i'.tmp' '/^SQL_WARN_MIN_SPACE=*/a\'$'\n''FILE_WARN_MIN_SPACE='$VALUE'\'$'\n''' "$config_file"
+	fi
+	sed -i'.tmp' 's/^DIRECTORIES_SIMPLE_LIST=/DIRECTORY_LIST=/g' "$config_file"
+	sed -i'.tmp' 's/^DIRECTORIES_RECURSE_LIST=/RECURSIVE_DIRECTORY_LIST=/g' "$config_file"
+	sed -i'.tmp' 's/^DIRECTORIES_RECURSE_EXCLUDE_LIST=/RECURSIVE_EXCLUDE_LIST=/g' "$config_file"
+	sed -i'.tmp' 's/^ROTATE_BACKUPS=/ROTATE_SQL_BACKUPS=/g' "$config_file"
+	if ! grep "^ROTATE_FILE_BACKUPS=" "$config_file" > /dev/null; then
+		VALUE=$(cat $config_file | grep "ROTATE_SQL_BACKUPS=")
+		VALUE=${VALUE#*=}
+		sed -i'.tmp' '/^ROTATE_SQL_BACKUPS=*/a\'$'\n''ROTATE_FILE_BACKUPS='$VALUE'\'$'\n''' "$config_file"
+	fi
+	sed -i'.tmp' 's/^ROTATE_COPIES=/ROTATE_SQL_COPIES=/g' "$config_file"
+	if ! grep "^ROTATE_FILE_COPIES=" "$config_file" > /dev/null; then
+		VALUE=$(cat $config_file | grep "ROTATE_SQL_COPIES=")
+		VALUE=${VALUE#*=}
+		sed -i'.tmp' '/^ROTATE_SQL_COPIES=*/a\'$'\n''ROTATE_FILE_COPIES='$VALUE'\'$'\n''' "$config_file"
+	fi
+	REMOTE_BACKUP=$(cat $config_file | grep "REMOTE_BACKUP=")
+	REMOTE_BACKUP=${REMOTE_BACKUP#*=}
+	if [ "$REMOTE_BACKUP" == "yes" ]; then
+		REMOTE_USER=$(cat $config_file | grep "REMOTE_USER=")
+		REMOTE_USER=${REMOTE_USER#*=}
+		REMOTE_HOST=$(cat $config_file | grep "REMOTE_HOST=")
+		REMOTE_HOST=${REMOTE_HOST#*=}
+		REMOTE_PORT=$(cat $config_file | grep "REMOTE_PORT=")
+		REMOTE_PORT=${REMOTE_PORT#*=}
+
+		REMOTE_SYSTEM_URI="ssh://$REMOTE_USER@$REMOTE_HOST:$REMOTE_PORT/"
+
+		sed -i'.tmp' 's#^REMOTE_BACKUP=yes#REMOTE_SYSTEM_URI='$REMOTE_SYSTEM_URI'#g' "$config_file"
+		sed -i'.tmp' '/^REMOTE_USER=*/d' "$config_file"
+		sed -i'.tmp' '/^REMOTE_HOST=*/d' "$config_file"
+		sed -i'.tmp' '/^REMOTE_PORT=*/d' "$config_file"
+
+		sed -i'.tmp' '/^INSTANCE_ID=*/a\'$'\n''BACKUP_TYPE=pull\'$'\n''' "$config_file"
+	else
+		if ! grep "^BACKUP_TYPE=" "$config_file" > /dev/null; then
+			sed -i'.tmp' '/^INSTANCE_ID=*/a\'$'\n''BACKUP_TYPE=local\'$'\n''' "$config_file"
+		fi
+	fi
+
+	# Add new config values from v1.1 if they don't exist
+	if ! grep "^ENCRYPTION=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^FILE_STORAGE=*/a\'$'\n''ENCRYPTION=no\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^CREATE_DIRS=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^ENCRYPTION=*/a\'$'\n''CREATE_DIRS=yes\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^GET_BACKUP_SIZE=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^BACKUP_SIZE_MINIMUM=*/a\'$'\n''GET_BACKUP_SIZE=yes\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^RSYNC_REMOTE_PATH=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^SSH_COMPRESSION=*/a\'$'\n''RSYNC_REMOTE_PATH=\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^SSH_IGNORE_KNOWN_HOSTS=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^SSH_COMPRESSION=*/a\'$'\n''SSH_IGNORE_KNOWN_HOSTS=no\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^REMOTE_HOST_PING=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^RSYNC_REMOTE_PATH=*/a\'$'\n''REMOTE_HOST_PING=yes\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^COPY_SYMLINKS=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^PRESERVE_XATTR=*/a\'$'\n''COPY_SYMLINKS=yes\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^KEEP_DIRLINKS=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^COPY_SYMLINKS=*/a\'$'\n''KEEP_DIRLINKS=yes\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^PRESERVE_HARDLINKS=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^KEEP_DIRLINKS=*/a\'$'\n''PRESERVE_HARDLINKS=no\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^RSYNC_PATTERN_FIRST=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^LOGFILE=*/a\'$'\n''RSYNC_PATTERN_FIRST=include\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^RSYNC_INCLUDE_PATTERN=" "$config_file" > /dev/null; then
+	        sed -i'.tmp' '/^RSYNC_EXCLUDE_PATTERN=*/a\'$'\n''RSYNC_INCLUDE_PATTERN=""\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^RSYNC_INCLUDE_FROM=" "$config_file" > /dev/null; then
+	        sed -i'.tmp' '/^RSYNC_EXCLUDE_FROM=*/a\'$'\n''RSYNC_INCLUDE_FROM=""\'$'\n''' "$config_file"
+	fi
+
+        if ! grep "^PARTIAL=" "$config_file" > /dev/null; then
+                sed -i'.tmp' '/^HARD_MAX_EXEC_TIME_FILE_TASK==*/a\'$'\n''PARTIAL=no\'$'\n''' "$config_file"
+        fi
+
+	if ! grep "^DELETE_VANISHED_FILES=" "$config_file" > /dev/null; then
+		sed -i'.tmp' '/^PARTIAL=*/a\'$'\n''DELETE_VANISHED_FILES=no\'$'\n''' "$config_file"
+	fi
+
+	if ! grep "^DELTA_COPIES=" "$config_file" > /dev/null; then
+                sed -i'.tmp' '/^PARTIAL=*/a\'$'\n''DELTA_COPIES=yes\'$'\n''' "$config_file"
+        fi
+
+	if ! grep "^BANDWIDTH=" "$config_file" > /dev/null; then
+                sed -i'.tmp' '/^DELTA_COPIES=*/a\'$'\n''BANDWIDTH=0\'$'\n''' "$config_file"
+        fi
+
+	if ! grep "^STOP_ON_CMD_ERROR=" "$config_file" > /dev/null; then
+                sed -i'.tmp' '/^MAX_EXEC_TIME_PER_CMD_AFTER=*/a\'$'\n''STOP_ON_CMD_ERROR=no\'$'\n''' "$config_file"
+        fi
+
+	if ! grep "^RUN_AFTER_CMD_ON_ERROR=" "$config_file" > /dev/null; then
+                sed -i'.tmp' '/^STOP_ON_CMD_ERROR=*/a\'$'\n''RUN_AFTER_CMD_ON_ERROR=no\'$'\n''' "$config_file"
+        fi
+
+	rm -f "$config_file.tmp"
+}
+
+if [ "$1" != "" ] && [ -f "$1" ] && [ -w "$1" ]; then
+	CONF_FILE="$1"
+	# Make sure there is no ending slash
+	CONF_FILE="${CONF_FILE%/}"
+	LoadConfigFile "$CONF_FILE"
+	RewriteConfigFiles "$CONF_FILE"
+else
+	Usage
+fi