Ship File Remotely
Copies files from the Clarive server out to one or more target servers. It builds a list of local files, filters it, works out where each file lands on the target, takes a backup of whatever it is about to overwrite, and sends it. On a rollback run the same op puts the backups back.
This is the op to use for deployment rather than scripting a copy with Run a Remote Script, because the backup and rollback behaviour, the already-sent bookkeeping and the asset tracking are only available here. For pulling a file the other way use Retrieve a remote file, and for keeping two directories aligned use Sync a Remote Directory.
Fields¶
Server¶
The target servers. Only resources that can hold an agent appear in the picker, and you can select more than one. The whole shipping sequence runs once per server, in the order they appear.
The field accepts ${var} entries as well as picked resources, and a variable that expands to a
comma-separated string is split into several targets.
A server marked inactive logs Server <name> is inactive. Skipped and is passed over. Leave the
picker empty and the step fails outright with Server not configured, before any file list is built,
so a variable that expands to nothing stops the job rather than shipping nowhere.
User¶
The account to connect as. Left blank, the connection uses the user configured on the agent attached
to the server resource. The value also appears in the user@server prefix on the log lines, and it
is part of the path where backups are kept, so changing it between the forward run and the rollback
run makes the rollback unable to find its backups.
Recursive¶
Unticked, Local Path is treated as a shell pattern: it can contain * and ?, and every plain
file it expands to is shipped. Directories are ignored.
Ticked, Local Path is walked as a directory tree and every plain file under it at any depth is
shipped. Wildcards stop working in this mode, so /tmp/build/* with Recursive ticked matches
nothing and the step ships zero files. Point it at the directory itself.
This field has no effect when Local Mode is Nature Items.
Local Mode¶
Where the list of local files comes from. A new op arrives on Nature Items, not on Local Files.
Nature Items takes the item paths left in the job by APPLY NATURE
and resolves them under the job directory. Local Path is hidden in this mode. If no nature block
has run, or the nature matched no items, the list is empty and nothing ships.
Local Files builds the list from Local Path, which the form then shows.
Local Path¶
The local file, pattern or directory to ship. Defaults to ${job_dir}/${project}.
Expanded twice: once against the stash, then against the attributes of the target
server, so ${hostname} and any custom parameter on the server resource resolve here.
The second pass is not repeated cleanly for each target. The first server's expansion is kept and
handed to the next one, so with two or more servers every target after the first reads the local
files that the first server's attributes pointed at. Keep server attributes out of Local Path when
you ship to more than one machine. Remote Path does not have this problem and re-expands properly
per server.
Exist Mode Local¶
What to do when a server ends up with nothing shipped. Skip if no files were found is the default.
Skip if no files were found logs Could not find any file locally to ship to '<user@server>' as a
warning and moves on to the next server. Fail if no files were found stops the job with
Error: No local files were found. A deployment that quietly ships nothing is the most common way
for this op to appear to have worked, so use Fail on steps that must move something.
The count it tests is files that got past the include and exclude filters, not files found on disk.
A directory full of files that every filter rejected counts as nothing found and trips this setting.
Either way the Shipped 0 files entry is written first, so a failing step leaves that line above the
failure.
Relative Path¶
How the destination path under Remote Path is built for each file.
| Relative Path | Destination |
|---|---|
File Only, no Path |
Remote Path plus the file name, so everything lands flat in one directory |
Keep Relative Path from job directory |
Remote Path plus the file's path relative to the job directory |
Specify Anchor Path |
Remote Path plus the file's path relative to Anchor Path |
File Only, no Path is the default and will silently collapse two files with the same name from
different directories onto each other.
The include and exclude filters are matched against this computed relative path, not against the full
local path. Under File Only, no Path that means your patterns only ever see file names.
Anchor Path¶
The base the relative path is measured from, shown only when Relative Path is Specify Anchor Path.
Defaults to ${job_dir}/${project}. It expands ${var} against the stash, but unlike Local Path
and Remote Path it is never expanded against the target server's attributes, so a server parameter
written here stays as literal text.
A file that is not underneath the anchor produces a relative path full of ../ segments, which then
get appended to Remote Path and land the file outside the directory you aimed at. Keep the anchor
at or above every file in the list.
Remote Path¶
The destination directory on the target. Required, and the form will not let you save the op without it. Expanded against the stash and then against the target server's attributes.
It is always treated as a directory, whatever Relative Path is set to and however many files are
in the list. The computed relative path is appended to it, so pointing it at
/opt/app/myapp.war with one file called myapp.war writes /opt/app/myapp.war/myapp.war. Name
the directory and let the file name come from the file.
The directory each file needs is created on the target if it is not already there.
Exist Mode Remote¶
Whether to send a file that has already been sent during this job.
Skip, if file already sent by any task in job chain is the default and consults a ledger the job
keeps under the stash name sent_files. A file counts as already sent when the same local path, the
same content, the same file timestamps and size, and the same destination path on the same hostname
were shipped earlier in the job or in a job it chained from.
Timestamps being part of that fingerprint matters. Regenerate the file with byte-identical content and it ships again, because the modification time moved. Re-running the job starts a new execution and every fingerprint changes with it, so nothing is ever skipped across runs.
Reship, even file has already been shipped to node disables the ledger and sends every time.
A skipped file still gets backed up and audited, and Chown and Chmod are still applied to it. What
it does not get is a tracked asset record, because tracking only happens on a file that was actually
sent.
Backup Mode¶
Whether to fetch the file being overwritten before overwriting it. Backups are what the rollback path reads, so this setting decides whether a rollback can restore anything.
Backup Existing Files is the default and the setting that takes backups. Each remote path is backed
up once per job: if a backup already exists from an earlier step, it is not overwritten, so the
backup always holds the state from before the job started. A remote file that does not exist yet is
recorded as having nothing to back up, which is how the rollback path knows to delete it later. A
failure while reading the backup logs a warning and the ship continues.
Backups are taken on a forward run only. On a rollback run this setting is not consulted at all.
No Backup skips the whole thing.
Backup Existing Files or Fail does not take backups. Despite the name, selecting it behaves like
No Backup, which means a rollback of this step will find nothing to restore. Use Backup Existing
Files whenever you want a working rollback.
Rollback Mode¶
What this op does when the job is running in rollback. It has no effect on a forward run.
Rollback from local files if exist is the default. For each file, the backup taken on the forward
run is sent back over the remote file. When there is no backup, meaning the file did not exist before
the deployment, the remote file is deleted.
Must Rollback from local files behaves the same when a backup exists and fails the step when one is
missing, instead of deleting.
No Rollback leaves the rollback logic out, so the op ships the same local files again. That is
rarely what you want. To keep the op out of rollback entirely, untick Run Rollback in the op
properties panel.
As soon as one file gets past the filters, this op marks its step as needing rollback, so a later failure in the job brings the job back through here without you configuring anything. That includes a file the ledger skipped, so a step that sent nothing new still asks for rollback.
The automatic marking only holds while the op's Needs Rollback? property is untouched. Open the op
properties panel and save it, and whatever that combo shows gets written down and wins. It shows
No Rollback Necessary by default, so a visit to the panel for an unrelated reason can switch the
automatic marking off. Check it reads Rollback Needed After on any op you opened. See
rollback.
Asset Track Mode¶
Track Assets Deployed records each file that lands on a target, with its checksum or its size and
date, and attaches the record to the job. No Tracking is the default.
Tracking is what later drift audits compare against, and it is a prerequisite for the field below. See asset tracking.
Audit Asset Drift¶
Audit Tracked Asset Before Overwriting checks, for each remote file that already exists and has
been tracked before, that it still matches what Clarive last deployed. A mismatch fails the step with
Tracked asset ... was modified before anything is overwritten, which is how you stop a deployment
from destroying a hand-patched file.
No Auditing is the default. The check does nothing on files that have never been tracked, so turn
on Asset Track Mode first and let a deployment run before expecting this to catch anything.
Chown¶
An owner specification applied to each file after it arrives, in the form the target platform
expects, such as appuser or appuser:appgroup. Left blank, ownership is left alone.
Failures are logged as a warning and the job continues, so a file that ended up owned by the wrong account leaves a warning line and a green step.
Chmod¶
A permission specification applied to each file after it arrives, such as 755. Left blank,
permissions are left alone. Failures are logged as a warning, as with Chown.
Max Transfer Chunk Size¶
Splits large files into numbered parts, sends the parts, then concatenates and deletes them on the
target with a remote command. Whole numbers only. The value is compared against the file size in
bytes, so 1048576 means files above one megabyte are split.
Left blank, files of one megabyte and above going to a Clax agent whose destination directory already exists are split into two parts automatically. That automatic size then sticks for the rest of the file list on that server, so every later file bigger than the first big one gets chunked too. Everything else is sent in one piece.
Set it when a link or an agent drops long transfers. The reassembly step runs a shell command on the target, so the target needs a working shell for chunked transfers.
Copy File Attributes¶
Ticked, the file's modification time travels with it. Over an SSH agent the permission bits travel too, and on a target where the connecting user is not allowed to set the mode the transfer then fails where it would otherwise have succeeded. A Clax agent carries the timestamp and nothing else.
Unticked, which is the default, the file arrives with the target's default ownership and mode, and
you set them with Chown and Chmod.
Filters¶
Two grids of regular expressions, each with Add and Delete, new rows prefilled with .*.
Include Paths acts as a whitelist. With no rows, every file passes. With one row or more, a file
has to match at least one of them. Exclude Path acts as a blacklist applied afterwards: a file
matching any row is dropped.
Both are matched against the computed relative path from Relative Path, not the local path on disk,
so a pattern like ^src/ only works under Keep Relative Path from job directory or
Specify Anchor Path. Filtered-out files are only reported at debug level, so a filter that is too
strict looks like a step that found nothing.
What appears in the job log¶
One entry per server reading Shipped N files to server user@name, with a per-file breakdown
attached as detail: which file went where, which were skipped because of the ledger, which were
backed up, and any chunking. A server that matched no files adds a warning.
A transfer failure stops the job with Ship failed for server ..., which carries only the underlying
error. The breakdown of what had been done up to that point goes into a separate debug entry, so on a
failed ship you need the Debug filter on to find out which files made it across.
Combining with other ops¶
Build the payload first with Run command or local script or Checkout Job Environment (all repos), then ship it. Use Replace Strings in between to substitute environment-specific values into config files before they leave the Clarive server.
After shipping, Run a Remote Script is what restarts the service or runs the installer. Since this op marks itself as needing rollback, a failure in that later step brings the job back through here and the backups go back.
Wrap the whole sequence in TRY statement and CATCH statement when you want to report a failed deployment rather than stop the job.
Examples¶
Ship one generated file to a fixed directory, keeping a backup so rollback works.
Server app_host
Local Mode Local Files
Local Path ${job_dir}/${project}/myapp.war
Exist Mode Local Fail if no files were found
Relative Path File Only, no Path
Remote Path /opt/tomcat/webapps/
Backup Mode Backup Existing Files
Rollback Mode Rollback from local files if exist
Chown appuser:appgroup
Chmod 644
Ship a whole tree to several servers, preserving the directory layout under an anchor and skipping build leftovers.
Server ${prod_nodes}
Recursive ticked
Local Mode Local Files
Local Path ${job_dir}/${project}/dist
Relative Path Specify Anchor Path
Anchor Path ${job_dir}/${project}/dist
Remote Path /srv/${project}/
Exist Mode Remote Reship, even file has already been shipped to node
Filters > Exclude Path \.map$
^tmp/
Ship exactly the files a nature selected, tracking them so later drift audits have something to compare against, and refuse to overwrite anything that was changed outside Clarive.
Server mainframe_proxy
Local Mode Nature Items
Relative Path Keep Relative Path from job directory
Remote Path ${deploy_root}
Backup Mode Backup Existing Files
Asset Track Mode Track Assets Deployed
Audit Asset Drift Audit Tracked Asset Before Overwriting