crcache acts as a layer to obtain, use, and discard compute resources - be it virtual machines, chroots or even physical machines. This is a common requirement for testing environments, and having an abstraction layer allows a single project setup to scale in dramatically different ways just by the user reconfiguring their crcache config.
This manual covers both part planned works and implemented works. If something doesn’t work please file a bug.
The default configuration is to have a single source local which runs commands locally.
Configuration is looked up in ~/.config/crcache and $(pwd)/.crcache. Where something is defined in both places, the first found definition wins, allowing local configuration to supercede any configuration supplied in a project (which might be version controlled and thus harder to change without side effects).
Each source is a subdirectory of a config root - $root/sources/$sourcename. Sources define how to provision one or more compute resources.
A source called local will replace the implicit definition of the local source.
The file source.conf is a .ini file that controls basic metadata for the source:
[DEFAULT]
; What sort of source is this?
type=[local|pool|ssh]
; Do not discard instances if less than this many are running.
; Defaults to 0 - avoids caching expensive resources w/out warning.
reserve=int
; Do not scale out beyond this many instances.
; Defaults to 0 - no limit.
maximum=int
; Override the concurrency of returned instances, rather than probing.
; Defaults to 0 - autoprobe.
concurrency=int
; For pools only
sources=sourcename,sourcename,...
; For ssh only
ssh_host=string
If a directory called provision.d exists as a sibling to source.conf then its contents will be run as they are provisioned (using run-parts). The resource name is supplied to the scripts as the first parameter - the script can call crcache run to execute commands on the resource.
Likewise for discard.d immediately before discarding an instance.
Provides details of sources and resources:
$ crcache status
source cached in-use max
local 0 1 1
pool 1 0 1
$ crcache status --query available pool
1
Checks a compute resource out for use:
$ crcache -s pool acquire
pool-0
$ crcache status pool
source cached in-use max
pool 0 1 1
Runs a command on a checked out resource:
$ crcache run pool-0 echo foo
foo
Get a shell on the resource:
$ crcache run pool-0
...
Returns a compute resource from use:
$ crcache release pool0
$ crcache status pool
source cached in-use max
pool 1 0 1
Each source stores the instances it has obtained and has cached in the crcache store, stored in $HOME/.cache/crcache/state.db.
The internal API is largely uninteresting for users - and see the DESIGN and DEVELOPER documentation if you are interested. That said, one possibly common need is creating additional source types, and so we cover that here.
Source types are looked up by looking for a python module with the same name in the cr_cache.source. package namespace. They can be installed as a third-party using namespace packages, or patched into the main crcache source tree. Source modules should include a Source class, which the source type loader looks for - you can subclass source.AbstractSource or just implement its contract. The loader will instantiate a Source instance with a ConfigParser and a get_source callback (which permits sources to layer on other sources).
Sources are responsible for four things:
Other operations, such as enforcing a limit on the number of instances, caching of instances, are taken care of by crcache infrastructure.