Sync a Remote Directory
Mirrors a directory between the Clarive server and a target server, in either direction, transferring only the parts that differ. Permissions, timestamps and symbolic links are preserved, and the transfer is compressed on the wire.
Use it when you are moving a tree rather than a file, and especially when the same tree moves repeatedly during a job and you want the second pass to be cheap. Ship File Remotely is the better choice when you need a backup of what you overwrote or a rollback of the transfer, because this op has neither. Retrieve a remote file is the one-file version of the inbound direction.
This op only works on servers connected over SSH. A server reached over Clax, Balix, FTP or a worker
connection stops the job as soon as the op runs, because syncing is not implemented for those.
Both machines also need rsync installed, since the Clarive server drives the transfer from its own
side and tunnels it over the SSH connection the server resource is configured with.
Fields¶
Server¶
The server to sync with. Required. Only resources that can hold an agent appear in the picker, and
the picker holds one of them at a time. The field also accepts a ${var} entry, and that is the way
to reach several servers from one op: a variable that expands to a comma-separated string is split,
and each name is resolved as a server resource.
When that happens, the servers are synced one after another against the same Local Path. On
Remote to local every server writes into that one directory and the results merge, with later
servers overwriting earlier ones file by file.
A name that does not resolve to a server resource stops the job. This op does not check whether a server resource is marked inactive, unlike most of its neighbours, so an inactive server is still contacted.
User¶
The account to connect as. Left blank, the connection uses the user configured on the agent attached to the server resource, and when that agent has no user either, the transfer goes out as whatever account the Clarive server itself runs under.
Direction¶
Which way the copy runs. Required, and defaults to Local to remote.
Local to remote makes Local Path the source and Remote Path the destination. Remote to local
reverses them. Nothing else on the form changes, so a step that quietly does the wrong thing is
usually this field.
Remote Path¶
The directory on the target server. Required. Expanded first against the stash,
then against the attributes of the server being contacted, so ${hostname} and any custom parameter
you added to the server resource resolve here.
When this is the source, the path is handed to the target's shell, so * and ? expand there and
/opt/build/dist/* copies the contents of that directory. When it is the destination, only the last
directory in the path is created for you. A destination whose parent directory is missing fails, and
since transfer failures are silent here, create the parent first with the Create Remote Directory op
when you are writing into a tree that may not exist yet.
The trailing slash matters. /opt/build/dist/ as a source copies what is inside the directory.
/opt/build/dist copies the directory itself, creating dist under the destination.
Local Path¶
The directory on the Clarive server. Required, expanded under the same two-pass rule as
Remote Path, and subject to the same trailing-slash rule.
Wildcards do not expand on this side. The local path is passed through as written, so * in it
matches a literal asterisk rather than a set of files.
Delete extraneous files from destination¶
Unticked, files present at the destination but missing from the source are left alone, so the sync only ever adds and updates.
Ticked, they are deleted, which makes the destination an exact copy of the source. Combine that with
a wrong Direction or a source directory that failed to populate and you will empty the destination.
Leave it unticked unless you need the mirror to be exact.
Failure and logging¶
The job log gets one line per server, written before the transfer starts:
Syncing directories '/local/path' and '*user@myserver*:/remote/path'. The right-hand side is built
from the server resource's name, not its host name, so it does not always match what you typed into
the resource. With User blank the line reads @myserver.
Nothing else is logged. The transfer's own file-by-file output is captured and then dropped, so the job log never tells you which files moved or how many bytes went across.
A failed transfer does not stop the job. The exit code comes back and is then dropped rather than
raised, so a source directory that cannot be read and a destination the account cannot write to both
leave the op reporting success and the rule carrying on. The same goes for rsync being absent on
either machine. Verify the outcome yourself when it matters, for example with
Run a Remote Script listing the destination afterwards.
What does stop the job: a server value that does not resolve to a server resource, a server that cannot be connected to at all, and a server whose connection is not SSH.
The form has no error mode and no return-code handling of its own. The op writes nothing into the
stash either; set a Return Key in the op properties panel and you get the number 1 stored under it
whatever the transfer did. There is also no transfer timeout, so a sync against a host that accepts
the connection and then stalls sits there until the Timeout set in the op properties panel cuts it
off.
This op takes no backups and does not participate in rollback. A rollback pass will run it again in
the same direction unless you untick Run Rollback in the op properties panel. See
rollback.
Combining with other ops¶
Pair it with Run a Remote Script on either side: push a source tree out, build it there, then pull the result back with a second sync in the other direction.
Use Publish local file to log afterwards to keep one file from a retrieved tree attached to the job, since the synced directory itself lives only in the workspace.
When the payload is a single file, Ship File Remotely and Retrieve a remote file are simpler and, in the outbound case, safer.
Examples¶
Push a built site out to a web server and remove anything the build no longer produces.
Server web_host
User webuser
Direction Local to remote
Local Path ${job_dir}/${project}/dist/
Remote Path /srv/www/${project}/
Delete extraneous files from destination ticked
Collect a report directory produced by a remote test run, leaving anything already present alone.
Server ${build_server}
Direction Remote to local
Remote Path /home/clarive/jobs/${job_name}/reports/
Local Path ${job_dir}/reports/
Delete extraneous files from destination unticked