Troubleshooting

Diagnostic Steps for Troubleshooting Failed Mirrors

Step 1

Check the UI in Lorry for the particular mirror and look for any logs or error messages.

Step 2

Access the logs of the running Lorry instance and select Schedule Run from the user interface of the mirror that is failing. Monitor the log output and look for any suspicious error messages.

Step 3

Inspect Lorry's working directory.

find workd -maxdepth 3
workd
workd/raw-assets_lorry-assets                           <-- A raw file mirror
workd/raw-assets_lorry-assets/raw-files                 <--- Raw file mirrors store data here 
workd/raw-assets_lorry-assets/raw-files/lorry-tiny.png
workd/raw-assets_lorry-assets/raw-files/.gitattributes
workd/raw-assets_lorry-assets/git-repository            <--- Git repository is stored here
workd/raw-assets_lorry-assets/git-repository/logs
workd/raw-assets_lorry-assets/git-repository/hooks
workd/raw-assets_lorry-assets/git-repository/lfs
workd/raw-assets_lorry-assets/git-repository/info
workd/raw-assets_lorry-assets/git-repository/index
workd/raw-assets_lorry-assets/git-repository/config
workd/raw-assets_lorry-assets/git-repository/description
workd/raw-assets_lorry-assets/git-repository/COMMIT_EDITMSG
workd/raw-assets_lorry-assets/git-repository/refs
workd/raw-assets_lorry-assets/git-repository/objects
workd/raw-assets_lorry-assets/git-repository/FETCH_HEAD
workd/raw-assets_lorry-assets/git-repository/HEAD
workd/lorry_lorry                                       <-- A normal mirror without raw files
workd/lorry_lorry/raw-files
workd/lorry_lorry/git-repository
workd/lorry_lorry/git-repository/hooks
workd/lorry_lorry/git-repository/info
workd/lorry_lorry/git-repository/config
workd/lorry_lorry/git-repository/description
workd/lorry_lorry/git-repository/refs
workd/lorry_lorry/git-repository/objects
workd/lorry_lorry/git-repository/FETCH_HEAD
workd/lorry_lorry/git-repository/HEAD

Step 4

Verify the integrity of the mirror repository

cd workd/lorry_lorry/git-repository
git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (4011/4011), done.

Check the status of a raw-file mirror

cd workd/raw-assets_lorry-assets/git-repository
git --work-tree=../raw-files status

Step 5

Delete the mirror repository on disk and allow Lorry to initialize it again.

rm -rf workd/lorry_lorry

You can select Schedule Run in the web interface to start the mirroring operation again. NOTE that if your downstream repository is corrupt or otherwise in an invalid state Lorry will first clone from there prior to fetching new updates.

Step 6

Stop the Lorry process and delete the mirrored repository from your downstream and then delete the working directory as described in Step 5.

Start Lorry again and monitor it's logs to confirm the subsequent clone and initialization was successful.

List-lorries Endpoint Analysis Tool

scrips/lorries_analysis.py Is a script for gathering the status data of the mirrored repositories, to assist with debugging systematic failures.

See ./scrips/lorries_analysis.py --help for usage.