FileSyncTask: Using Phing to synchronize files and directories

I needed to automate the task of synchronizing files from one server to another, so I wrote a Phing task. Finally today I found some time to finish writing the documentation.

Overview

FileSyncTask is a Phing extension for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer. FileSyncTask can copy or display directory contents and copy files, optionally using compression and recursion.

Rather than using FTP or some other form of file transfer, FileSyncTask uses rsync to copy only the diffs of files that have actually changed. Only actual changed pieces of files are transferred, rather than the whole file, which results in transferring only a small amount of data and are very fast. FTP, for example, would transfer the entire file, even if only one byte changed. The tiny pieces of diffs are then compressed on the fly, further saving you file transfer time and reducing the load on the network.

FileSyncTask can be used to synchronize Website trees from staging to production servers and to backup key areas of the filesystems.

Usage

There are 4 different ways of using FileSyncTask:

  1. For copying local files.
  2. For copying from the local machine to a remote machine using a remote shell program as the transport (ssh).
  3. For copying from a remote machine to the local machine using a remote shell program.
  4. For listing files on a remote machine.

The SSH client called by FileSyncTask uses settings from the build.properties file:

sync.source.projectdir=/home/development.com/public
sync.destination.projectdir=/home/staging.com
sync.remote.host=server.com
sync.remote.user=user
sync.destination.backupdir=/home/staging.com/backup
sync.exclude.file=/home/development.com/build/sync.exclude

Listing files

The “listonly” option will cause the modified files to be listed instead of transferred. You must specify a source and a destination, one of which may be remote.

<taskdef name="sync" classname="phing.tasks.ext.FileSyncTask" />
<sync
    sourcedir="${sync.source.projectdir}"
    destinationdir="${sync.destination.projectdir}"
    listonly="true"
    verbose="true" />

Excluding irrelevant files

To exclude files from synchronizations, open and edit the sync.exclude file under the build/ directory. Each line can contain a file, a directory, or a pattern:

*~
.svn
.htaccess
public/index.development.php
public/images/uploads/*
build/*
log/*
tmp/*

Copying files to a remote machine

The following task definition will transfer files from a local source to a remote destination:

<taskdef name="sync" classname="phing.tasks.ext.FileSyncTask" />
<sync
    sourcedir="${sync.source.projectdir}"
    destinationdir="${sync.remote.user}@${sync.remote.host}:${sync.destination.projectdir}"
    excludefile="${sync.exclude.file}"
    verbose="true" />

Example

Directory structure

In order to separate the sync settings from the main build file, I’ve created a file called sync.properties:

development.com
|-- build
|   |-- build.properties
|   |-- build.xml
|   |-- sync.exclude
|   `-- sync.properties
`-- public
    `-- index.php

XML build file

Phing uses XML build files that contain a description of the things to do. The build file is structured into targets that contain the actual commands to perform:

<?xml version="1.0" ?>
<project name="example" basedir="." default="build">
<property name="version" value="1.0" />

<!-- Public targets -->
<target name="sync:list" description="List files">
  <phingcall target="-sync-execute-task">
    <property name="listonly" value="true" />
  </phingcall>
</target>

<target name="sync" description="Copy files">
  <phingcall target="-sync-execute-task">
    <property name="listonly" value="false" />
  </phingcall>
</target>

<!-- Private targets -->
<target name="-init" description="Load main settings">
  <tstamp />
  <property file="build.properties" />
</target>

<target name="-sync-execute-task" depends="-init">
  <property file="sync.properties" />
  <if>
    <not>
      <isset property="sync.verbose" />
    </not>
    <then>
      <property name="sync.verbose" value="true" override="true" />
      <echo message="The value of sync.verbose has been set to true" />
    </then>
  </if>
  <property name="sync.remote.auth" value="${sync.remote.user}@${sync.remote.host}" />
  <taskdef name="sync" classname="phing.tasks.ext.FileSyncTask" />
  <sync
    sourcedir="${sync.source.projectdir}"
    destinationdir="${sync.remote.auth}:${sync.destination.projectdir}"
    backupdir="${sync.remote.auth}:${sync.destination.backupdir}"
    excludefile="${sync.exclude.file}"
    listonly="${listonly}"
    verbose="${sync.verbose}" />
</target>
</project>

Execute task

$ phing sync:list

Outputs:

Buildfile: /home/development.com/build/build.xml

example > sync:list:
[phingcall] Calling Buildfile '/home/development.com/build/build.xml'
            with target '-sync-execute-task'

example > -init:
[property] Loading /home/development.com/build/build.properties

example > -sync-execute-task:
[property] Loading /home/development.com/build/sync.properties
[echo] The value of sync.verbose has been set to true

Execute Command
----------------------------------------
rsync -razv --list-only -b --backup-dir

Sync files to remote server
----------------------------------------
Source:        /home/development.com/public
Destination:   user@server.com:/home/staging.com
Backup:        user@server.com:/home/staging.com/backup

Exclude patterns
----------------------------------------
*~
.svn
.htaccess
public/index.development.php
public/images/uploads/*
build/*
log/*
tmp/*

(list of files that have changed)

BUILD FINISHED
Total time: 1.9763 second

More information

Related Articles

Advertisements

16 thoughts on “FileSyncTask: Using Phing to synchronize files and directories

  1. Pingback: Narendra Dhami

  2. Hi! I just started exploring phing, and find your task extension. Overall it is a little overhead comparing to raw rsync, but I can see it as more reliable to other not-so unix folks!
    Anyhow, there is a little typo You made in build.xml example:
    backupdir=”${sync.remote.auth}:${sync.destination.backupdir}”
    should be:
    backupdir=”${sync.destination.backupdir}”
    as this will create awkward folder structure.

    The other suggestion I would like to make is error output. If rsync fails, there is just error code (number) printed, instead of more usefull – text error message.

    Keep up the great work!

  3. Pingback: Juozas devBlog

  4. Thanks for this extension to Phing. I use this one a lot. But have found a few things missing.

    1) Even though the extension displays an error message in case that the rsync fails. But in cases where the phing command is run in the background, there is no way to ascertain whether the rsync completed successfully or not. A solution would be to send an email to the webmaster stating that the task fails along with the error code.

    2) No option to choose individual files only. The task currently rsync’s the whole folder stated as source. In some deployment cases there is a need to only push out some files instead of all the files in a folder.
    One solution would be is to use rsync’s –include-from flag to only include the files mentioned in the sync.include file.

  5. Great Task!

    What was the reasoning for using an external file to mark files for exclusion?

    Can I not nest a FileList/FileSet within the sync element and use include/exclude?

  6. This task is a huge help for me!

    Can I ask why you overwrite options if they’re set via the XML? In the link just above this comment, what I’m talking about is on line 217. I’m trying to set additional options without editing the task; I want to pass in chmod options to update the directory after sync. My chmod options look like this:

    –chmod=u+rwX,g+rX,o+rX

    But if I say:

    I lose the -raz at the beginning, and where the “verbose” tag is appended, it causes it to be invalid syntax. So, if I pass in options, including the -raz, it still breaks if verbose is on, creating an rsync call that looks like this:

    rsync -raz –chmod=u+rwX,g+rX,o+rX v -b –backup-dir=…

    Note how my options are passed in but the V is just floating there?

    Might I suggest adding an “additionalOptions” tag that just gets appended into the options at the end of all the setters? I’ve updated the above code paste with my suggested changes:

    http://codepad.org/LXsCwVSm

  7. I’m sorry, in my post above, my example sync xml got blocked out. It was supposed to say

    options=”–chmod=u+rwX,g+rX,o+rX”

    within the xml sync tag in the build script.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s