DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

Version 2, December 2004

Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>

Everyone is permitted to copy and distribute verbatim or modified

copies of this license document, and changing it is allowed as long

as the name is changed.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

0. You just DO WHAT THE FUCK YOU WANT TO.

git-cache

Context

If you repeatedly clone a Git repository, for instance for a software you use to hack, you could bothered by always waiting during the clone operation, particularly long for big softwares. On a other side, if you have multiple copies of a Git repository on your computer, it takes place you could prefer not to use.

Git has a feature "--reference"/alternates, which gives the possibility to indirect commits to another directory on the same computer. The use case is:

$ cd ~/test

$ git clone https://example.org/git/repo.git normal-repo

(you wait minutes or hours, and the resulting directory is big)

$ git clone --reference ~/test/repo https://example.org/git/repo.git referenced-repo

(you wait seconds, and the resulting directory is small - only the size of the files, no size used by the history)

Both clones contain the whole history of the Git repository. Internally, the second clone contains a file `.git/objects/info/alternates` containing the path of the first directory, where git searches the commits it doesn’t find the in the second clone. Obviously, if you delete the first clone, the second is useless and will display tons of errors.

git-cache

git-cache creates a central cache directory on a computer, which contains most of the commits of regularly cloned Git repositories.

If you are a regular hacker of a software, say MediaWiki, you can cache most commits and then only download recent commits. Use case:

$ sudo git cache init # cache is located in /var/cache/git-cache

$ git cache add mediawiki https://git.wikimedia.org/git/mediawiki/core.git

You can now use the cache directory with:

git clone --reference /var/cache/git-cache https://git.wikimedia.org/git/mediawiki/extensions/TorBlock.git

(1607 seconds, the cache takes 314 Mio)

$ git clone --reference /var/cache/git-cache https://git.wikimedia.org/git/mediawiki/core.git clone-1

(46 seconds, the directory takes 98 Mio, the .git subdirectory takes 17 Mio because there were new commits)

$ git clone --reference /var/cache/git-cache https://git.wikimedia.org/git/mediawiki/core.git clone-2

(58 seconds, the directory takes 98 Mio, the .git subdirectory takes 17 Mio because there were new commits)

Usage

__Installation:__ Copy git-cache inside your git commands directory (on Ubuntu: /usr/lib/git-core) and be sure it can be executed by the users (mode 755).

__Use:__ All commands have the format

git cache ACTION PARAMETERS

Details:

# General maintenance commands

git cache init [DIR] initialise the cache directory

git cache delete --force delete the cache directory

# Daily commands

git cache add NAME URL add a cached Git repository

git cache show [NAME] show cached Git repositoryies/repository

git cache fetch fetch all cached Git repositories

git cache rm --force NAME remove a cached Git repository

(Any other command will be applied to the cache directory,

e.g. `git cache gc` or `git cache remote show`.)

__Location of the cache directory:__ The default cache directory contains all cached repositories (each Git repository is a remote). If it is created by root, the cache directory is `/var/cache/git-cache`, else it is `~/.cache/git-cache`; it can also be another directory specified in the init command. This directory could become big, be sure you have enough place.

~~You can want to cron `git cache fetch` to automatically retrieve new commits.~~ (bug)

Development

Bug: git cache fetch does not retrieve new commits.

This is a first version, and feedback is needed to improve its daily usage. Some questions I wonder:

- Are the subcommands sufficiently explicit?

- Should we regularly run `git gc` or even `git gc --aggressive`?

- How git behaves when some commits are both local and in the cache? Does it remove local objects to gain place?

- etc.

Internally the cache directory is a big bare directory containing all remote repositories, even if they do not share commits (sort of orphans branches). Should the cache directory be splitted by repository: /var/cache/git-cache/mediawiki, /var/cache/git-cache/visualeditor, etc. Particularly in the second case, the name of the remote repository is useless, and I would prefer use unique names, e.g. md5(URL) and completely hide this to the user (possibly with soft links from the URL to the directory for usability inside the cache directory).

Licence

- Original author: [Seb35](https://github.com/Seb35)

- Licence: [WTFPL 2.0](http://www.wpfpl.net)

References

This program is a generalisation to arbitrary Git repositories of an idea and implementation by [Randy Fay](http://randyfay.com/content/reference-cache-repositories-speed-clones-git-clone-reference) specifically for Drupal. Hopefully this generalisation is sufficiently simple to stay useable and practical.

CACHE=$(git config cache.directory)

if [ ! -d "$CACHE" -a ! "$1" = "init" ]

echo "Inexistant cache directory, please create it with \`git cache init [DIR]'"

exit 1

if [ "$1" = "init" ]

if [ -d "$CACHE" -a -f "$CACHE/HEAD" -a ! "$2" = "$CACHE" ]

then

echo "Cache directory is already initialised. You can use \`git cache delete --force' to delete it."

echo "$CACHE"

exit 1

fi

# Try to create directory in the following order:

# 1) given by argument

# 2) /var/cache/git-cache if root or sudo

# 3) ~/.cache/git-cache else

CACHE="$2"

TYPE=global

[ "$CACHE" = "" -a -w /var/cache ] && CACHE=/var/cache/git-cache && TYPE=system

[ "$CACHE" = "" ] && CACHE=~/.cache/git-cache

mkdir -p "$CACHE"

cd "$CACHE"

git init --bare

chgrp -R users .

chmod -R g+w .

git config --$TYPE cache.directory "$CACHE"

elif [ "$1" = "delete" ]

if [ ! "$2" = "--force" ]

then

echo "Please add: 1) the --force option; to confirm you want to delete the whole cache directory."

exit 1

fi

if [ ! "$(git config --system cache.directory)" = "" ]

then

CACHE="$(git config --system cache.directory)"

git config --system --remove-section cache 2>/dev/null && rm -rf "$CACHE" && echo "System cache directory $CACHE deleted."

fi

if [ ! "$(git config --global cache.directory)" = "" ]

then

CACHE="$(git config --global cache.directory)"

git config --global --remove-section cache 2>/dev/null && rm -rf "$CACHE" && echo "Global user cache directory $CACHE deleted."

fi

elif [ "$1" = "add" ]

if [ "$2" = "" -o "$3" = "" ]

then

echo "Please add: 1) the remote name and 2) the remote url to be added in the cache directory."

exit 1

fi

# Fetch new Git repository in a temporary directory for speed reasons

#dir=$(mktemp -d)

#cd $dir

#git init --bare

#git remote add --mirror=fetch "$2" "$3"

#git fetch --all

# Copy it in the main cache directory

cd "$CACHE"

#git remote add --mirror=fetch "$2" $dir

#git fetch --all

#git remote set-url "$2" "$3"

git remote add "$2" "$3"

git fetch "$2"

#rm -rf $dir

# Display use

echo

echo "You can now use the cached directory with:"

echo

echo " git clone --reference $CACHE $3"

echo

elif [ "$1" = "rm" ]

if [ ! "$2" = "--force" -o "$3" = "" ]

then

echo "Please add: 1) the --force option and 2) the remote name; to confirm you want to delete this Git repository from the cache directory."

exit 1

fi

cd "$CACHE"

git remote rm "$3"

elif [ "$1" = "fetch" ]

cd "$CACHE"

git fetch --all --prune

elif [ "$1" = "show" ]

cd "$CACHE"

git remote -v | grep \(fetch\) | grep "$2" | awk "{print \"git clone --reference $CACHE \"\$2}"

elif [ "$1" = "help" ]

echo

echo "Use:"

echo

echo " git cache init [DIR] initialise the cache directory"

echo " git cache delete --force delete the cache directory"

echo

echo " git cache add NAME URL add a cached Git repository"

echo " git cache show [NAME] show all cached Git repositories"

echo " git cache fetch fetch all cached Git repository"

echo " git cache rm --force NAME remove a cached Git repository"

echo

echo " (Any other command will be applied to the cache directory.)"

echo " e.g. \`git cache gc\` or \`git cache remote show\`.)"

echo

echo "Standard cache directory is either:"

echo

echo " /var/cache/git-cache if created by root (then usable by every user)"

echo " ~/.cache/git-cache else"

echo

echo "Once created, the cache directory is written in the configuration parameter"

echo "'cache.directory', either in the system or global user configuration."

echo

cd "$CACHE"

git "$@"