Artifact Repository Manager
Clarive has an integrated artifact repository manager that is capable of managing the delivery of artifacts (or components) to build tools such as Maven or similar.
The key features of the Clarive artifact manager are:
- server repository of any kind of files (or artifacts), from binary executables to packaged libraries and metadata.
- files published are accessible through HTTP(S) navigation.
- a in-tool artifact browser with search and tagging.
- remote repository proxying.
- database artifact indexing a storage.
- access-control to artifacts and directories.
- repository groups.
The Clarive artifact repository management system is controlled by a set of three Resources:
- Local Artifact Repository (ArtifactLocal).
- Remote Artifact Repository (ArtifactRemote).
- Repository Group (ArtifactGroup).
Repository management in Clarive is uncomplicated, unlike other tools on the market. A local repository points to a location on the filesystem (or database) that contains a structure of artifacts. A remote repository points to a URL that can provide a given artifact. A group is an ordered set of both local and remote artifacts.
To create your first repository, start by creating a Resource of type Local Artifact Repository. Read below for details.
All repositories served by Clarive start from the base url
All resources returned by
/artifacts/repo have its Content-Type set to
application/download. If you want to get artifacts with the correct
MIME type for each extension, replace the URL part
When using the
/artifact/static/ URL prefix, repository files are returned
Content-Type header that matches the extension.
This is the extension-to-content-type MIME types
for artifacts served with
Local Artifact Repository (ArtifactLocal)¶
A Clarive local artifact repository is a filesystem location where artifacts are stored and made accessible through a URL path.
The ArtifactLocal Resource represents a location on the filesystem that holds a directory structure of artifacts, with a corresponding URL prefix.
For example, for a Local Repository Resource called "MyRepo" you may want to define a local directory for storing the files at the following location:
and a URL prefix of:
The URL base for this repository would be:
Creating a Local Artifact Repository Resource¶
The minimum requirement for creating a local repository is to set a Resource
Name, define a
URL Prefix and a
Local Path to
- Name - the name does not influence any of the URL or directory related operations.
- URL Prefix - this is the URL path part after
/artifacts/repowhere repositories and their files will be accessible to users, scripts and build systems that depend on it.
- Local Path to Repository - the physical location of the artifact repository on the server filesystem, also called a root path. If the local path does not exist, it will be created by Clarive.
- Repository Mode - Could be Maven or NPM depending on the type of artifacts that it will store.
Local Path Auto-Selection¶
If no local path is defined, upon saving the Resource, Clarive will set the path to a combination of the global config
config.artifacts.path and the URL prefix.
After saving, reload the Resource to see the new value.
Access Mode: Public¶
Local repositories with Public access mode can be navigated through the interface and the artifacts it contains can be downloaded by any user that has the URL.
Access Mode: Hidden¶
By default, local repositories are public. But in Locked mode, the repositories are not accessible by anyone, except through a repository group.
Locked mode is therefore ideal for defining local storage locations that can later be combined into a repository group.
Exact Matching Mode¶
By default, local repository files are matched exactly to the requested filename.
To perform near matches, use the Prefix mode.
Prefix Matching Mode, White and Blacklisting¶
In the Prefix matching mode, files served by a local directory do not need to be exact. Clarive will attempt to match all possible files that start with the filename requested. A local artifact repository may also have white/blacklisted artifacts that allow for administrators to better control what types of files are returned by Clarive.
To understand how white/blacklisting works, first we need to make the distinction between the two types of input lists:
- files - a filename requested by the user, build tool etc.
- images - a file found by Clarive on the filesystem.
If more than one image matches for a given file request, the first image matched is returned.
Requested file: http://clarive/artifacts/repo/myrepo/file.jar Matching images: http://clarive/artifacts/repo/myrepo/file_v01.jar http://clarive/artifacts/repo/myrepo/file_v00.jar File served: http://clarive/artifacts/repo/myrepo/file_v01.jar
All lists are regular expressions, so they need to be written with separating
A list of file patterns that can be requested by the user or build tool. Patterns can match any part of the file name or requested URL.
An empty file whitelist means ANY file pattern can be requested. If the file does not match the pattern, it is discarded immediately.
A list of file suffixes that cannot be requested by the user or build tool.
An empty blacklist of files means NONE is blacklisted.
Note: even if a file is blacklisted, it may be served due to an image whitelist. See below.
Requested file: http://clarive/artifacts/repo/myrepo/file.jar Blacklisted files: -source|-sources Matching images: http://clarive/artifacts/repo/myrepo/file.jar http://clarive/artifacts/repo/myrepo/file-sources.jar File served, which excludes the blacklisted files: http://clarive/artifacts/repo/myrepo/file.jar
Images are files found in the artifact repository that match the filename detected.
The whitelisted image pattern will match any part of the path or file name requested.
Important: even if a file is blacklisted, if there are any matching whitelisted images, the images will be served back to the requestor by Clarive. However, only whitelisted images that have the requested filename will be returned.
Images are files found in the artifact repository that match the filename detected.
Remote Artifact Repository (ArtifactRemote)¶
A remote artifact repository is a URL location from where Clarive can download artifacts and serve them to the user.
Remote artifact repositories allows Clarive to act as a proxy between the user and the remote artifact.
Creating a Remote Artifact Repository¶
To create a remote artifact repository, you need only the Resource
URL to Remote Repository fields.
- Name - the name is just the Resource name and does not impact on how the repository is used or accessed remotely.
- URL to Remote Repository - a URL to an artifact repository that can deliver the files requested. Typically this URL will be served over the internet, although it could also point to other repositories within the intranet or private cloud of an organization.
Remote to Local Association¶
In order to enable a remote repository for use, first it is necessary to have a local repository that bridges to the remote one.
Having a local-remote repository association creates a locally cached remote repository. That way, files that are frequently requested by builds will be cached locally (in a server directory), and may be indexed, searched and traversed by users.
Repository Group (ArtifactGroup)¶
A repository group is an ordered set of local repositories that are logically bundled into one URL prefix.
Why use repository groups?
- to have local repositories organized in different ways and locations without impacting users and builds that can break with a URL prefix reorg.
- to have a set of failback repositories that allow the user or build system to try several locations before finally returning a 404 (not found) code.
We recommend the use of repository groups, precisely for these reasons. Groups are a single entry point for users to the artifact repository hierarchy.
URL mapping from group to local¶
URL prefixes of local repositories will be replaced by the group prefix when a local repository is accessed through a group.
For example, suppose we have the following local repository:
and normally, within the local repository, we can access the following file:
and the following is a group prefix for a group that contains the above local repository:
Then the following URL will also be made available to users:
effectively replacing the previous
mylocal/ URL path part with the new one.
Repository Error Messages¶
The repositories may generate the following errors and error messages:
400 Bad Request¶
This error message means that the repository prefix does not exist in the Clarive repository management system.
Here is an example in Maven:
[ERROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.5: Could not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.5 from/to central (http://clarive:8080/artifacts/repo/local): Failed to transfer file: http://clarive:8080/artifacts/repo/local/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom. Return code is: 400, ReasonPhrase:Bad Request. -> [Help 1]
Check your URL prefix, which is the part that comes right after the
404 File Not Found¶
This error indicates that an artifact was not found in the repository, but the repository itself exists.
In Maven, the error could look somewhat like this. Notice that the error code is not visible here:
[ERROR] Failed to execute goal on project my-app: Could not resolve dependencies for project com.mycompany.app:my-app:jar:1: Failed to collect dependencies at com.dbcx.core:dropbox-core-sdk:jar:LATEST: Failed to read artifact descriptor for com.dbcx.core:dropbox-core-sdk:jar:LATEST: Failed to resolve version for com.dbcx.core:dropbox-core-sdk:jar:LATEST: Could not find metadata com.dbcx.core:dropbox-core-sdk/maven-metadata.xml in local (/opt/mavenprj/.m2/repository) -> [Help 1]
The 404 error could be caused either by a misconfiguration in the user's or process build/packaging script, or by the fact that the organization has decided not to make the file available to the user through blacklisting.
Using the Clarive Artifact Repository with Package Managers¶
The Clarive artifact repository management system can be used together with many different package management systems, not just build systems like Maven.
Private repositories is an ideal method for publishing and controlling the versions of packages used within an organization.
Below are some examples of using Clarive with many popular package managers.
The NodeJS NPM package manager uses the concept of a
registry, which is a URL that contains a NPM repository.
To have Clarive proxy an NPM registry, first create a remote that points to a public Node registry, such as (the official public Node registry):
Remote URL: https://registry.npmjs.org/
Then create a local repository to cache the artifacts downloaded and connect it to the remote.
Once the Clarive Resources are created, you can set the npm registry on the command-line. Suppose you set the URL prefix to
local-npm, you can now install Node JS packages through Clarive:
npm install --registry http://clarive:8080/artifacts/repo/local-npm babel
or simply configure the registry in your
$HOME/.npmrc file that points to the Clarive NPM repository you just created:
; pointing the local NPM repository in Clarive registry = http://clarive:8080/artifacts/repo/local-npm
If you want to publish NPM packages in a Clarive repository, you'll need to configure a publish user and password. Then, you'll need to stablish a session to the repository:
Once connected correctly, you'll be able to publish packages:
npm publish ./
npm publish ./ --registry http://clarive:8080/artifacts/repo/local-npm
The Python PyPI package management system can use Clarive to proxy and control packages downloaded.
First create a remote artifact Resource that points to the remote URL of a Python PyPI repository:
Remote URL: http://pypi.python.org/simple/
then create a local repository to cache the artifacts downloaded and connect it to the remote.
Now simply set the URL of the Clarive artifact manager url-prefix using the
-i) of Python's
easy_install -i http://clarive:8080/artifacts/repo/pypi/ [package_name]
Perl and CPAN¶
Using Clarive with CPAN package managers is simple, just create a remote that points to one of many CPAN mirrors around the internet. For example:
Remote URL: https://cpan.metacpan.org
and create the local Resource with a URL prefix
cpan and add the remote Resource to the Remote Repositories field.
Now you can install Perl packages proxying through Clarive.
cpanm --mirror-only --mirror http://clarive/artifacts/repo/cpan Moose