Server Configuration

Server configuration is specified by a TOML file from the file system. It can be specified via lorry --config <PATH>.

Private Token

The private token used for basic authentication in Git pushes can be specified either in the configuration file as gitlab-private-token = gplat-ABC or as an environment variable by setting LORRY_GITLAB_PRIVATE_TOKEN=gplat-ABC. If both the configuration file and environment are set the environment variable will take preference.

Notes on Threading

Lorry is a multi-threaded server process that will map blocking git clone operations onto each available thread the server has configured. By default the server will spawn as many clone operations as it has cores available which is a reasonable default. Note that when using the Git binary for fetch and push operations that it has its own concept of concurrency. If 8 threads are available on a Lorry host it is recommend to configure the git binary to use only a single thread.

For example:

n-threads = 8
[clone]
engine = GitBinary
n-threads = 1

Annotated Example

An annotated file is provided below.

# Path to a SQLite database that is used to schedule mirroring operations and
# store historical information.
statedb = "./db/lorries.sqlite"
# An optional username that is used as part of basic authentication on the
# downstream Git mirror.
username = "oauth2"
# Hostname of the downstream mirror and optional port number. NOTE that
# scheme e.g. https:// should not be included here.
hostname = "127.0.0.1:9999"
# Path to where Git repositories and raw-file assets will be stored on disk.
# NOTE that this directory is safe to delete however doing so will require that
# the mirrors be re-created causing extra CPU and network utilization.
working-area = "./workd"
# The maximum number of redirections Lorry should follow when resolving 
# raw-file mirrors.
maximum-redirects = 1
# Optional URL from with a Lorry configuration can be cloned from and will be
# updated periodically.
# confgit-url = "https://my-git-repository.example.org/lorry-config"
# an optional branch to use as part of the remote git configuration
# If specified Lorry will read its configuration from this directory and 
# reload it periodically.
configuration-directory = "./examples/controller"
# Logging level 
log-level = "INFO"
# The port that Lorry should listen to for incoming network connections on.
port = 3000
# If calls to the Gitlab API can be done over insecure HTTP. Not recommended
# for production settings.
gitlab-insecure-http = true
# An optional path to a program which will return authentication credentials
# for Git push operations.
askpass-program = "./contrib/lorry-askpass"

# If sha256sums are required for all raw file mirrors.
sha256sums_required = true

# Username which should be configured for automated commits of Lorry raw-file
# mirrors. This name will show up in the Git log.
username = "lorry@example.org"

# An optional path to the gitconfig configuration file to use during commands
# where Lorry shells out to the git binary such as fetch operations when 
# engine = GitBinary. Lorry will dynamically modify this file changing settings
# that are required for its normal operation but will not clobber it.
# git-config-path = "/etc/lorry/.gitconfig"

# The number of threads that Lorry will spawn to mirror individual 
# repositories. It will default to the number of cores available on the 
# currnetly running system.
# n-threads = 8

# git clone related configuration
[clone]
# the engine to use for cloning (and fetching) operations. this setting can
# either be gitbinary or libgit2. note that using the git binary for cloning
# large repositories has considerably better performance than using the 
# libgit2 bindings.
engine = gitbinary
# the number of threads to use for cloning remote repositories. note that this
# settings only effects cloning when the gitbinary is in use. libgit2 does not
# support multi-threaded operations. due to lorry being multi-threaded "1" is
# typically a good setting here because several concurrent clone operations
# can degrade performance considerably.
n-threads = 1

Mirror Configuration

The mirror configuration is where the link between upstream repositories and downstream mirrors are defined. There are some basic requirements for this:

  • Must be a git repository
  • Must have a JSON file called lorry-controller.conf in the root of the repository.

The repository is specified in the confgit-url and confgit-branch settings for the controller, and is cloned or updated when the Read Configuration endpoint is accessed.

Lorry Controller Configuration

The main lorry-controller.conf file consists of a list of objects, containing the following required keys and values:

  • type - String that should be set to lorries.
  • interval - String in ISO8601 duration format. For example PT3H corresponds to a 3-hour duration. Specifies the interval for mirroring the various lorry configs in this group.
  • timeout - String in ISO8601 duration format, see interval. If mirroring one of the lorry configs takes longer than the timeout, it will be cancelled.
  • prefix - String specifying the downstream group prefix. This is prefixed to the individual lorry names in this group on the downstream repository.
  • globs - Array of Strings specifying the file globs containing the individual lorry configurations. For example, the folder/*.lorry example given will look for all .lorry files in the folder directory ( relative to the lorry-controller.conf file)

An example configuration file would look like this:

[
  {
    "type": "lorries",
    "interval": "PT1M",
    "timeout": "PT1M",
    "prefix": "lorry-mirrors/github",
    "globs": [
      "github.lorry"
    ]
  }
]

This would mirror any repositories specified in a github.lorry file every minute.

The individual lorry mirror configurations are YAML files. These are in the form:

mirror-name:
  type: mirror type
  # further mirror config

The mirror type currently can be either git or raw-file. This determines the extra mirror configuration required.

When the mirror type is git, the extra configuration options are:

  • url Required - String of the git URL for the repository.
  • check-certificates Default: true - Boolean if the SSL/TLS certificate for the specific repository should be checked. If the worker level check-certificates option is set to false, this will not turn the checking of certificates back on, it can only disable the checking of certificates for the current mirror.
  • ref-patterns Optional - List of glob patterns that define which git references to mirror.
  • ignore-patterns Optional - List of glob patterns to exclude from mirrors. NOTE that these take precedence over ref-patterns.

When the mirror type is raw-file, the extra configuration options are:

  • urls Required - List of URL mappings, with the following keys:
    • url Required - String of the file URL to download.
    • destination Required - String of the directory to store the downloaded file in.
  • check-certificates Default: true - Boolean if the SSL/TLS certificate for the files should be checked. If the worker level check-certificates option is set to false, this will not turn the checking of certificates back on, it can only disable the checking of certificates for the current mirror.
  • sha256sum Optional - The expected sha256sum of the raw file

With the above, an example lorry mirror configuration could look like the following:

octocat/hello-world:
  type: git
  url: https://github.com/octocat/Hello-World.git

raw-files:
  type: raw-file
  urls:
    - destination: target-directory
      url: https://my-file-host.tld/directory/more-directory/file.tar
      sha256sum: 3a1a7d59eb62f8710a46d86faea9ab9600f948660aed33acd4846658def0ef83
    - destination: another-target-directory
      url: https://my-file-host.tld/directory/another_file.tar

If this was used with the controller configuration above, then the two repositories created in your GitLab group would be at:

  • lorry-mirrors/github/octocat/hello-world
  • lorry-mirrors/github/raw-files