cla export - export Clarive configuration as HCL

cla export: writes the configuration of a Clarive installation as HCL files.

The files contain no database identifiers. Objects are named by their address, and one object refers to another by address rather than by id, so the same files can be imported into a different installation and the relationships still land on the right objects.

Usage

cla export -c ENV [--include PATTERN]... [--exclude PATTERN]...
           [--topics CATEGORY]... [--path DIR] [--file NAME] [--pack] [--split[=LIST]]
           [--stdout] [--no-deps] [--internal] [--external LIST] [--all-external]
           [--no-moved] [--all-attributes] [--pedantic]
           [--secret b64|plain|omit|encrypt] [--secret-key KEY] [--indent N] [-v]

Options

Option	Default	What it does
-c ENV	—	Which installation to read. Required.
--include PATTERN	everything	Keep objects matching this address or pattern. Repeatable, comma-separated.
--exclude PATTERN	none	Drop objects matching this. Repeatable, comma-separated. Applied after includes.
--filter PATTERN	—	Older spelling of both: a leading ! means exclude. Still accepted.
--topics CATEGORY	none	Also export topics of these categories, by name. No --topics all exists.
--path DIR	./cla-objects/<env>	Where to write.
--split	off	One file per object instead of one per family.
--split=LIST	none	One file per object only for these classes, by class, feature or plugin name. Globs, comma-separated.
--pack	off	Everything into one clarive-pack.hcl.
--file NAME	—	Everything into a file you name. Implies --pack.
--stdout	off	Print the document instead of writing files.
--no-deps	deps on	Do not pull in what the selection references.
--internal	off	Also export runtime classes: revisions, jobs, events, assets.
--external LIST	none	Also export these external classes, by class, feature or plugin name. Globs, comma-separated, repeatable.
--all-external	off	Also export every external class.
--no-moved	moved on	Do not emit moved {} blocks for renamed objects.
--all-attributes	off	Write attributes left at their default too.
--pedantic	off	Write nothing at all unless the export is error free.
--secret MODE	b64	b64, plain, omit or encrypt.
--secret-key KEY	—	Key for --secret encrypt. The importing side needs the same one.
--indent N	2	Spaces per nesting level, 1 to 8. cla import reads any width.
-v	off	Report per-family counts, pulled dependencies, unchanged files and timings.

With no options, every exportable object is written to ./cla-objects/. Objects are grouped one file per family; rules get a file each:

cla-objects/generic_server.hcl              every generic_server
cla-objects/bl.hcl                          every bl
cla-objects/user.hcl                        every user
cla-objects/role.hcl                        every role
cla-objects/resource.GitRepository.hcl      every GitRepository
cla-objects/topic.config-item.hcl           every topic in that category
cla-objects/pipeline.deploy-to-prod.hcl     one rule
cla-objects/event.on-topic-created.hcl      one rule
cla-objects/ws.acme-hook.hcl                one rule

The file name is the object's address with its own label removed, so the layout follows the addressing rules rather than a separate convention.

Rules are the exception because they are big: a pipeline or a workflow is hundreds of lines of code-shaped HCL, edited on its own. Two of them in one file would turn every unrelated edit into a merge conflict. Everything else is a few lines per object and is read as a group.

Inside a grouped file, blocks are sorted by address, and files are written in sorted order, so re-exporting an unchanged installation produces byte-identical files.

--split restores one file per object:

cla-objects/generic_server.back-office.hcl
cla-objects/generic_server.front-desk.hcl
cla-objects/role.deploy-manager.hcl

This matters at scale. An installation with hundreds of thousands of objects exported per object is hundreds of thousands of files; grouped, it is a few hundred.

One file per object for some classes

Between the two, a class can have a file per object while every other class stays grouped. That suits a class whose objects are big, or are edited one at a time, so that a change to one never conflicts with a change to another:

cla-objects/generic_server.hcl                 every generic_server
cla-objects/resource.foo_object.bar.hcl        one foo_object
cla-objects/resource.foo_object.baz.hcl        one foo_object

There are two ways to ask for it:

1. The class says so. A class consumes the marker role Baseliner::Role::HCL::FilePerObject:

perl package BaselinerX::CI::foo_object; use Baseliner::Moose; with 'Baseliner::Role::HCL::FilePerObject';

A plugin lists the role in its class definition, read from the source the same way as ::HCL::Exportable (see External classes):

js ci.createClass('FooObject', { roles: ['::HCL::FilePerObject'], });

1. --split=LIST names it. Each name matches a class, or the feature or plugin it comes from, exactly as --external matches them: case-insensitive whole names, * and ? as wildcards.

cla export --split=foo_object cla export --split=FooFeature # every class in the FooFeature feature cla export --split=generic_server,user

A bare --split still gives every object its own file. The default for the classes of features and plugins is the same as for any other class: grouped, unless the role or --split says otherwise. The role only changes the layout. It does not let an external class through, which is what Baseliner::Role::HCL::Exportable and --external are for.

Import reads a directory the same whichever way its objects are spread over files. Switching a class from one layout to the other leaves the old files in place, though, and cla import then reports HCL221 Duplicate address. Delete the class's old files before re-exporting into the same directory.

Rules are exported too, as the block keyword their type is named by: pipeline, workflow, event, form, dashboard and the rest. A rule's steps are written as nested blocks, and anything a step points at — a server, a project, another rule — is written as an address, so the step lands on the right object after an import even though the two installations number things differently.

Choosing what to export

--include keeps objects, --exclude drops them. Both take addresses or patterns, both are repeatable, and both accept comma-separated lists. They combine: includes decide the set, then excludes subtract from it.

cla export --include generic_server
cla export --include 'role.*,project.*'
cla export --exclude resource.DBDest,resource.foo_object
cla export --include generic_server --exclude generic_server.scratch-01

With only --exclude, everything starts included. With any --include, only what is included is considered.

Pattern rules

• A pattern is a glob unless it contains regular expression characters (^ $ ( ) [ ] + \), in which case it is a regular expression.
• * matches within one address segment; ** crosses segments.
• ? matches one character within a segment.
• A pattern that names a prefix also matches everything under it, so --exclude role drops the whole family and --exclude resource.DBDest drops every object of that class. Prefix widening applies only to patterns with no wildcard of their own.
• Addresses match in every form they have, so a second-order object answers to both resource.foo_object.foo and its short address.

generic_server            every generic_server
generic_server.*          the same thing, spelled with a wildcard
generic_server.web*       servers whose slug starts with web
topic.config-item         every topic in that category
**                        everything
resource.**               every second-order resource class

--filter is the older spelling: one option, with a leading ! for exclusion. It still works and cla import and cla diff still take it, but --include and --exclude say which direction a pattern goes without a sigil to remember.

Dependencies

Whatever the selection points at comes with it. Exporting a server that has a proxy also exports the proxy, transitively, so the resulting set can be imported somewhere else without dangling references. The one exception is a reference into an external class, which is kept but not followed: see External classes. --no-deps turns that off; the summary always reports how many objects were pulled in, and -v names them.

Note that --exclude does not survive the dependency walk: if an included object references an excluded one, the reference would dangle, so the target is pulled back in. Use --no-deps if you need the exclusion to be absolute.

Internal classes

--internal also exports classes that are normally left out: git revisions, jobs, events, assets and other things that are runtime state rather than configuration, exist in enormous numbers, and mean nothing in another installation.

External classes

Only the classes that ship with Clarive are exported by default. A class from anywhere else is external, and its objects are left out:

Where the class comes from	Example	The summary says
A feature, features/<name>/lib/BaselinerX/CI/<class>.pm	foo_server	feature FooFeature
A plugin outside the core distribution, through ci.createClass()	VaultEndpoint	plugin cla-vault-plugin
Nowhere: the objects are in the database, the code is not installed	a removed feature's class	no code installed

Where a class comes from is decided by where its code is, never by its name. Classes created by the plugins in Clarive's own plugins/ directory are core.

External classes belong to whoever wrote them, so Clarive doesn't guess whether their objects mean anything on another installation, or whether that installation even has the feature. They are also often most of the database: a single feature can hold hundreds of thousands of objects, against a few thousand objects of core configuration.

Nothing is left out quietly. The summary lists the external classes it skipped, biggest first, with the flag that brings them back:

Selecting...
  125620 object(s) of external classes left out:
    foo_object                120000  feature FooFeature
    bar_record                  4000  no code installed
    baz_item                     900  no code installed
    qux_data                     500  no code installed
    quux_entry                   120  no code installed
    and 12 more class(es), -v lists them
    To export them: --external=<class, feature or plugin>, or --all-external for every external class
  3000 selected

There are three ways to let an external class through:

1. Its author opts in. A feature class consumes the marker role Baseliner::Role::HCL::Exportable:

perl package BaselinerX::CI::foo_server; use Baseliner::Moose; with 'Baseliner::Role::CI::Server'; with 'Baseliner::Role::HCL::Exportable';

A plugin lists the role in its class definition:

js ci.createClass('FooServer', { roles: ['Server', '::HCL::Exportable'], });

Plugin init scripts do not run under cla export, so for a plugin class the role is read from the source of the createClass() call. That reading skips strings, comments and regular expressions, but it is not a full JavaScript parser: very unusual syntax inside the call can hide the role or pick up one from the next call. Keep roles a plain list of strings, and if the summary still lists the class as external, --external names it.

1. --external LIST names it. Each name matches a class, or the feature or plugin it comes from, so one name can cover a whole family of classes. Names are case-insensitive whole names; * and ? are wildcards:

cla export --external foo_server cla export --external FooFeature # every class in the FooFeature feature cla export --external 'bar_*,cla-vault-plugin'

1. --all-external lets every external class through.

Selection still applies on top: --external FooFeature --include foo_server writes the servers without the objects. When an --include matches nothing but would have matched external objects, the error says so and gives the command:

[error] Nothing matched. Check the filter, or run without one to export everything
[error] 5 external object(s) match it, and external classes are left out. To export them:

  cla export --external=foo_server

References. When an exported object points at an external one, the reference is kept and written as the external object's address, because dropping it would lose configuration. The external object is not pulled in with it, not even with dependencies on. The summary counts these references, and -v lists each one:

  12 external object(s) are referenced and not exported; the target installation must already have them

If the target installation doesn't have one of those objects, cla import stops with HCL230 Unresolved reference. --allow-missing skips the objects that depend on it instead.

Files from an earlier export are removed. Export overwrites the files it writes and leaves every other file alone. A directory exported before external classes were left out, or by an --all-external run, would still hold foo_server.hcl, and the next import would apply it. So when a run covers an external class and leaves it out, it deletes that class's files from the target directory, in the grouped and the --split layout alike, and names each one:

  foo_server.hcl removed: its class is external and left out
  resource.foo_object.hcl removed: its class is external and left out
...
  removed   2

An export whose --include doesn't cover the class, such as --include 'generic_server.*', doesn't touch its files. --stdout, --pack and --file remove nothing.

Import is not affected. cla import applies whatever the files contain, so an external class in the files imports like any other. cla import --prune only considers the classes present in the files, so leaving a class out of an export can never put its objects up for deletion.

Topics

Topics are never exported unless you ask for them, by category:

cla export --topics 'Config Item'
cla export --topics 'Config Item,Deployment Window'
cla export --topics 'Config Item' --include 'topic.config-item.smtp*'

Most topics are the work an organisation records — changesets, releases, incidents — and there are millions of them. Some categories, though, hold configuration: a settings topic, a deployment window, an environment matrix. --topics is for those, and it takes category names, not a pattern, so that an export can never sweep up a category nobody meant to include. There is no --topics all.

Categories are matched by name or by the slug of the name, so --topics 'Config Item' and --topics config-item mean the same thing. A name no category answers to is an error, and the message lists the categories that do exist.

The count of topics found in each category is printed before anything is written, so an unexpectedly large category is visible immediately:

category Config Item: 12 topic(s)

A topic block carries the topic's title, its status, and the fields its category's form defines:

topic "config-item" "smtp-settings" {
  title  = "SMTP Settings"
  status = status.active

  fields {
    smtp_host = "mail.example.invalid"
    smtp_port = 2525
    notify    = [user.foo, user.bar]
    project   = [project.acme-web]
  }
}

It does not carry comments, attachments, activity, status history, effort or timestamps. Those are the record of what people did to a topic, not configuration, and copying them onto another installation would invent a history that never happened there.

A reference from a gated topic to a topic of a category you did not name is dropped, with a warning — that is what stops one settings category from dragging a hundred thousand changesets along behind it.

cla import needs no flag: topic blocks in the files are applied like anything else, and --prune considers only the categories the files actually name. The category itself must already exist on the target installation; a category that is missing is reported in the plan, before anything is written.

Where it goes

• --path DIR — write to another directory instead of ./cla-objects/.
• --split — one file per object instead of one per family. Rules are already one per file either way. --split=LIST does it only for the classes named.
• --pack — everything into one file, clarive-pack.hcl.
• --file NAME — everything into a file you name.
• --stdout — print instead of writing.

A file whose content has not changed is left alone, so an export into a git working copy only touches what actually differs.

Indentation

Files are written with two spaces per nesting level. --indent N writes N instead, anywhere from 1 to 8:

cla export --indent=4

The width applies to every file the run writes, heredoc bodies included: an indented heredoc (<<-EOT) sits one level inside its block at whatever width that is, and the indentation of its lines relative to each other is kept.

Indentation is presentation only. cla import, cla diff and cla import-plan read a four-space file exactly as they read a two-space one, so changing the width never shows up as a change to the installation. It does show up in git, once, as every line moving; pick a width for a repository and keep it.

cla fmt rewrites files at two spaces unless told otherwise, so run it with the same width, cla fmt --indent=4, or it will undo the export's choice. The UI editor always saves at two.

Progress

The command reports as it goes. All of it goes to standard error, so cla export --stdout still pipes a clean document.

$ cla export --path ./cla-objects
Reading configuration from the database...
  resources  612
  roles      14
  rules      118
  topics     0
  744 object(s) found
Selecting...
  744 selected
Encoding 744 object(s)...
Writing to /opt/clarive/cla-objects/prod
  generic_server.hcl
  pipeline.deploy-to-prod.hcl
744 object(s) exported, 0 pulled in as dependencies
  files     147
  written   2
  unchanged 145
  into      /opt/clarive/cla-objects/prod

object(s) exported counts objects; files counts the files they were grouped into. The two are only the same number under a bare --split.

Only files that were actually written are listed. Unchanged files are counted, not named, so on a second export the output is the changeset.

Reading an installation is a full scan of master_doc — hundreds of thousands of documents on a large one. Each sweep reports as it finishes, and on a terminal a running count of documents scanned updates in place while the slow sweep runs, so a long read never looks like a hung command.

An error does not stop the export. Objects that could not be written in full are reported, everything else is written, and the command exits 2. The error and warning counts are repeated after the summary, because on a real export the diagnostics themselves scroll off the top of the screen.

-v adds: the address of every object the dependency closure pulled in, unchanged files by name, and elapsed time per phase.

Perl warnings from plugins are normally silenced; -v un-silences them, so a noisy third-party plugin may add lines that have nothing to do with the export.

$ cla export --path ./cla-objects --include generic_server.front-desk -v --split
Reading configuration from the database...
  resources  612
  roles      14
  rules      118
  topics     0
  744 object(s) found (11.4s)
Selecting...
  1 selected (0.2s)
  1 pulled in as dependencies
  + generic_server.back-office
Encoding 2 object(s)...
Writing to /opt/clarive/cla-objects/prod
  generic_server.back-office.hcl (unchanged)
  generic_server.front-desk.hcl
2 object(s) exported, 1 pulled in as dependencies
  files     2
  written   1
  unchanged 1
  into      /opt/clarive/cla-objects/prod
  took      12.9s

--topics prints a count per category before anything is written, because a category somebody believed held six settings topics may hold sixty thousand changesets.

Secrets

Attributes that hold credentials are never written in the clear by default.

• --secret b64 — base64, the default. Not encryption: it keeps a password out of plain sight in a file someone might paste into a ticket.
• --secret plain — write the real value.
• --secret omit — leave the attribute out entirely. An import then keeps whatever the target already has.
• --secret encrypt --secret-key KEY — encrypt with the same cipher Clarive uses for stored credentials. The importing side needs the same key.

Diagnostics

An error does not stop the export. The objects involved are written without the part that could not be represented, everything else is written normally, and the command exits 2. --pedantic restores all-or-nothing: with it, an export that has any error produces no output at all.

Three things make an error traceable after the run:

1. A comment in the file, where the missing thing should have been.

hcl role "baz" { name = "Baz" # HCL108: Dropped the grant for action.topics.jobs: its bound on id_category # could not be written, and a grant with no bound allows everything }

A file that silently lacks an attribute is indistinguishable from one where it was never set. Comments are ignored on import, so a commented file still round-trips.

1. export.log, written into the output directory alongside the .hcl files. It holds the same diagnostics, uncoloured, plus the suggested --exclude. cla import only reads *.hcl, so the log is never mistaken for an object.

2. A suggested --exclude that makes the run clean:

``` To export everything else and leave these out:

 cla export --exclude=role.foo,role.bar,role.baz

```

On a terminal, errors are red and warnings yellow. Colour is never written to export.log or returned to the UI.

Every diagnostic names the object it came from, by address:

role.qux: warning[HCL106]: Refused to write status-13 for id_status: it is a database id and no address could be found for it (x14)
workflow.foo: warning[HCL107]: The value of expr has UserGroup-215 written inside it. That is a database id and it will not mean the same thing elsewhere

Identical messages are folded and counted — (x14) means fourteen objects hit the same problem. If the warning list was cut short, the command says so (and N more warning(s) not shown) rather than leaving a ceiling looking like a total.

HCL106 — refused to write a database id. A reference could not be turned into an address, so the attribute was left out of the file. This is data loss: after an import that field will be empty. Usual causes are a reference to an object that no longer exists, and a value that was typed into a picker by hand rather than chosen.

HCL107 — a database id inside an opaque value. The id is embedded in a free-text field (expr, code, condition, filter), so it is written, as-is. Nothing is lost, but the id means something different on the target installation. These need editing by hand — there is no safe mechanical rewrite for an id buried in a code fragment.

HCL102 — a reference was dropped. The target of a declared relationship was not found in the database.

HCL108 — a grant was dropped. A role grant had a bound (id_status, projects, bl) whose value could not be written. This is an error, not a warning, and it stops the export with exit 2. A grant that keeps only some of its bounds allows more than it did, and a grant with no bounds at all allows everything — so the grant is refused rather than written wider than it is. Fix the dangling bound, or drop the grant from the role.

Exit codes

• 0 — exported.
• 2 — nothing matched the selection, a --topics category does not exist, a file could not be written, or a diagnostic was raised as an error (see HCL108).

Examples

Export everything into a git working copy and see what changed:

cla export --path ./cla-objects
git diff

Move one project and everything it depends on to another installation:

cla export --include project.acme-web --pack --file acme.hcl
# copy acme.hcl to the other machine
cla import acme.hcl

Look at a single object without writing anything:

cla export --stdout --include role.deploy-manager

--all-attributes

By default an export leaves out any attribute sitting at the value its form would have filled in anyway. The Rule Designer saves every field of an operation's form whether or not anyone touched it, so a step that was given four values is stored with eighteen; printing all of them buries the four that were chosen.

ship "Ship a File Remotely" {          ship "Ship a File Remotely" {
  backup_mode   = "none"                 anchor_path      = "${job_dir}/..."
  from          = "${job_dir}/..."       audit_tracked    = "none"
  host          = generic_server.web     backup_mode      = "none"
  local_mode    = "local_files"          copy_attrs       = 0
  rollback_mode = "none"                 exclude_path     = []
  to            = "/tmp/"                exist_mode       = "skip"
  user          = "jboss"                ... eleven more ...
}                                      }
      default                                --all-attributes

Note what is KEPT on the left: backup_mode = "none" stays because the form's default is "backup", so "none" is a choice somebody made. Only values that match what the form would have written are dropped.

This is safe because the omission is symmetric: cla import puts back exactly what the export left out, from the same table. Absence in a file means "the default", never "remove it".

Pass --all-attributes when the file is an archive, or when it will be imported by a different Clarive release. The compact form reads back identically only against the defaults table of the release that wrote it; if that table changes, an old compact file imports with the new defaults. The full form has no such dependency.