rsync with cron job every minute issue

I am transferring files over the network using rsync in cron job that runs every 1 one minute, task is whenever new file come it has to start uploading on remote server:

Since I am new so doing big mistake here:

* * * * * rsync -azP /local_path user@x.x.x.x:/remote_path > /dev/null

rsync create temp file (at remote end) and on next minute it creates temp file again (at remote end) after few minutes my networks starts getting choke and no upload bandwidth left, then I kill all rsync processes and copy at at a time.

Need this automated please help.

Thanks

2 Answers

Looks like this is because the command takes longer than a minute to run, so the commands pile up until your system is starved of resources.

You can ensure that only one instance of that rsync runs by using a lock file. util-linux includes a file locking wrapper called flock, which you can use in your crontab like so:

* * * * * /usr/bin/flock -n '/tmp/example.lock' -c 'rsync -azP /local_path user@x.x.x.x:/remote_path' > /dev/null

In the above example, the lock file path is /tmp/example.lock, but you can set it to be anything sensible. -n will prevent flock commands from piling up because it'll exit immediately instead of wait for the lock to be released to continue.

Alternatively, you can also prevent your cron command from piling up by making a script with a fairly simple lock that ensures only one instance of the script can be running at a time:

#!/bin/bash
PIDFILE=/tmp/example.pid
if [ -f "$PIDFILE" ]
then PID="$(cat "$PIDFILE")" ps -p $PID > /dev/zero 2>&1 if [ $? -eq 0 ] then echo "Instance of this script is still running as PID $PID" exit 1 fi
fi
echo $$ > $PIDFILE
if [ $? -ne 0 ]
then echo "Could not create PID file: $PIDFILE" exit 1
fi
# YOUR CODE GOES BELOW
rsync -az /local_path user@x.x.x.x:/remote_path
# YOUR CODE GOES ABOVE
rm -f "$PIDFILE"

^{_{Locking mechanism inspired by Preventing duplicate cron job executions by Benjamin Cane}}

Note that the main limitation in the script is if the PID file isn't removed for whatever reason and another process has the same PID, the script will erroneously quit.

Save this script somewhere, like in /usr/local/bin/example.sh, chmod +x /usr/local/bin/example.sh, and then call it from your crontab like this:

* * * * * /usr/local/bin/example.sh > /dev/null

First of all remove -P option, cron really won't to see progress.

You need to implement some locking mechanism, don't call rsync directly from cron, but make some script instead that would be used in cron. In the script, check first condition if rsync already running (grep ps output for example), if it running then simply exit script, otherwise run rsync.