git-lfs/docker/README.md

# README #

## TL;DR version ##
1. Run the dockers

        ./docker/run_dockers.bsh
        
2. **Enjoy** all your new package files in

        ./repos/

##Using the Dockers##

All these commands need to either be run as root OR as a user in the docker group
Adding your username to the docker group is probably the easiest

### Running Dockers ###

In order to run the dockers, the docker has to be built, and then run with a
lot of arguments to get the mount points right, etc... A convenient script is 
supplied to make this all easy. Simply run

    ./docker/run_docker.bsh
    
All the git-lfs_* images are built automatically, and then run. The default 
command is the build script. The currently checkout version of git-lfs is 
copied into the docker, and ```git clean -xdf``` is run to remove any 
non-tracked files, but non-committed changes are kept. Then git-lfs is built, 
and a packages is created (deb or rpm)

To only run certain docker images, supply them as arguments, e.g.

    ./docker/run_docker.bsh git-lfs_debian_7
    ./docker/run_docker.bsh git-lfs_debian_*
    ./docker/run_docker.bsh git-lfs_*[6-8]
    
And only those images will be run.

There are a few environment variables you can set to easily adjust the behavior
of the ```run_docker.bsh``` script.

REPO_HOSTNAME - Override the hostname for all the repos generated/tested (see below)

BUILD_LOCAL - Set to 1 (default) to use the currently checked out version of
the git-lfs to build against. If it's not 1 the released archived is downloaded
and built against. Currently only works for RPMs. DEB always builds the currently
checkout version. Build local only affect the version of the code used in generating
the rpms, not the scripts running to generate the rpms (e.g. ```./rpm/build_rpms.bsh```)

DOCKER_AUTOBUILD - Default 1. ```run_docker.bsh``` always calls ```build_docker.bsh``` to
ensure your docker image is up-to-date. Sometimes you may not want this. If set
this to 0, it will not build docker images before running

AUTO_REMOVE - Default 1. Docker containers are automatically deleted on 
exit. If set to 0, the docker containers will not be automatically deleted upon 
exit. This can be useful for a post mortem analysis (using other docker commands
not covered here). Just make sure you clean up the docker containers manually.

###Development with Dockers###

Sometimes you don't want to just build git-lfs and destroy the container, you
want to get in there, run a lot of command, DEVELOP! To do this, the best
command to run is bash, and then you have an interactive shell to use. To do this

    ./docker/run_docker.bsh {image(s)} -- bash

After listing the image(s) you want to run, add a double dash (--) and then any 
command (and arguments) you want executed in the docker. Remember, the command
you are executing has to be part of the docker image.

### Building Dockers (Optional) ###

```run_dockers.bsh``` calls ```build_dockers.bsh```, but you can still call the
script manually to get it all out of the way once while you go make some 
tea/coffee.

In order to use the docker **images**, they have to be built so that they are
ready to be used. For OSes like Debian, this is a fairly quick process. 
However CentOS takes considerably longer time, since it has to build go, ruby,
or git from source, depending on the distro. Fortunately, you can build the 
docker images JUST once, and you won't have to build it again (until the version
changed.) The build script uses a downloaded release from github of git-lfs to
bootstrap the CentOS image and build/install all the necessary software. Currently
the only way to change what version the image is built off of is by changing the
URL in the Dockerfile for git-lfs_centos_*. 

This means all the compiling, yum/apt-get/custom dependency compiling is done 
once and saved. (This is done in CentOS by using the already existing 
```./rpm/rpm_build.bsh``` script to bootstrap the image and saving the image.)

The script that takes care of ALL of these details for you is

    ./docker/build_dockers.bsh
    
All the git-lfs_* images will be built automatically. These are all a
developer would need to test the different OSes. And create the git-lfs rpm or
deb packages.

However, in order to distribute git-lfs or build dependencies, the packages 
that were installed for you by ```build_docker.bsh``` need to be saved too.
This is currently only a CentOS problem. In order to generate THOSE rpms, 
the git-lfs-full-build_* will use NON-bootstrapped images to build every package
and git-lfs and generate rpms. This takes as long as building the image in the
first place, but you don't get the benefit of a saved state. The 
```./rpm/rpm_build.bsh``` script will build all of its dependencies when you
run the dockers, making the rpms available. These images can be built by running

    ./docker/docker_build.bsh ./docker/git-lfs-full-build_*

This is most important for CentOS 6 where git 1.8.2 or newer is not available, 
only git 1.7.1 is available, so every user either has to build git from source, 
or use the rpms generated by the ```git-lfs-full-build_centos_6``` image. This 
will only needs to be done once.

(To manually build a docker image, run 
```docker build -t $(basename ${DockerDir}) -f ${DockerDir}/Dockerfile ./docker```


##Deploying/Building Repositories##

When ```./docker/run_dockers.bsh``` is done building git-lfs and the rpms/deb,
it actually creates a repository for distribution too. Each distro gets a repo
generated in ```./repos/{DISTRO_NAME}/{VERSION #}```. Just drop the repo
directory onto a webserver and you have a fully functioning Linux repo. (See
Testing the Repositories below for more detail)

The two major packages included are:
git-lfs-....* - the git-lfs package
git-lfs-repo-release....* - A package to install the repo.

When using ```BUILD_LOCAL=1```, all UNTRACKED files are removed during RPM 
generation (except any stray directories containing a .git folder will not be
cleared. This shouldn't be the case, unless you are temporarily storing another
git repo in the git repo. This is a safety mechanism in git, so just keep in mind
if you are producing packages.)

### Setting the website URL ###

The git-lfs-repo-release must contain the URL where the repo is to be hosted.
The current default value is 'git-lfs.github.com' but this can be overridden
using the REPO_HOSTNAME env var, e.g.

    REPO_HOSTNAME=www.notgithub.uk.co ./docker/run_dockers.bsh
    
Now all the git-lfs-repo-release....* files will point to that URL instead

### GPG signing ###

For private repo testing, GPG signing can be skipped. apt-get and yum can 
install .deb/.rpm directly without gpg keys and everything will work. This 
section is for distribution in a repo. Most if not all this functionality is 
automatically disabled when there is no signing key present
(```./docker/signing.key```).

In order to sign packages, you need to generate and place GPG keys in the right place

1. gpg --gen-key

    1. 1 - RSA and RSA
    2. 4096 bits
    3. Some length of time or 0 for infinite
    4. y for yes
    5. Signer name (Will become part of the key and uid)
    6. Email address (Will become part of the key and uid)
    7. Comment (Will become part of the key)
    8. O for Okay
    9. Enter a very secure password, make sure you will not forget it
    10. Generate Entropy!
    
2. gpg -a --export > ./docker/public.key

3. gpg -a --export-secret-keys > ./docker/signing.key

Keep in mind, signing.key must NEVER be accidentally committed to the repo. 

To prevent MANY passphrase entries at random times, the gpg-agent is used to
cache your signing key. This is done by running gpg-agent in the host, and passing
the connection to each docker image. This will be done for you automatically by
calling the ```./docker/preload_key.bsh``` script. This can be called manually
before any other command just to get the pass phrase entry out of the way before
you start running everything.

GPG agent TTL is set to 5 hours. This should be plenty to build everything. If this is
not good for you, set the GPG_MAX_CACHE and GPG_DEFAULT_CACHE environment variables
(in seconds)

### GPG capabilities by Distro ###

Debian WILL work with 4096 bit RSA signing subkeys like [1] suggests, but will
also work with 4096 bit RSA signing keys.

CentOS will NOT work with subkeys[3]. CentOS 6 and 7 will work with 4096 bit RSA 
signing keys

CentOS 5 will NOT work with v4 signatures. The rpms will be so unrecognizable 
that it can't even be installed with --nogpgcheck. It should work with RSA on v3.
However, I could not get it to. It builds v3 correctly, but for some reason
the GPG check fails for RSA. However ```yum install --nogpgcheck``` does work! CentOS 5
will NOT work with 2048 bit DSA keys... I suspect 2048 is too big for it to
fathom. CentOS 5 WILL work with 1024 bit DSA keys. Either sign it with the key
instructions above and install it with the ```yum install --nogpgcheck``` OR
create a NEW DSA key just for CentOS 5 and sign with that. 

[1] https://www.digitalocean.com/community/tutorials/how-to-use-reprepro-for-a-secure-package-repository-on-ubuntu-14-04

[2] https://iuscommunity.org/pages/CreatingAGPGKeyandSigningRPMs.html#exporting-the-public-gpg-key

[3] http://www.redhat.com/archives/rpm-list/2006-November/msg00105.html

### Testing the Repositories ###

To test that all the OSes can download the rpm/debs, install, and run the tests
again, run

    ./test_dockers.bsh
    
(which is basically just ```./docker/run_dockers.bsh ./docker/git-lfs-test_*```)

Remember to set REPO_HOSTNAME if you changed it for ```./docker/build_docker.bsh```
This can also be used to run a local test (on ```localhost:{Port Number}```, for
example)

An easy way to test the repositories locally, is to run them on a simple webserver such as

    cd ./repos
    python -m SimpleHTTPServer {Port number}

or

    cd ./repos
    ruby -run -ehttpd . -p{Port Number}


## Adding addition OSes ##

To add another operating system, simply follow the already existing pattern, 
and all the scripts will pick them up. A new Dockerfile should go in a directory
named

    ./docker/git-lfs_{OS NAME}_{OS VERSION #}
    
where **{OS NAME}** and **{OS VERSION #}** should not contain underscores (_).
Any files that needs to be added to the docker image can be dropped in the 
```./docker``` directory, since that is the context root they are built against 
(not the directory containing the Dockerfile like most dockers)

The docker image should run a script that write it's repo files to the /repo
directory inside the docker container. Writing to /repo in the docker will cause the
files to end up in

    ./repos/{OS NAME}/{OS VERSION #}/

## Docker Cheat sheet ##

Install https://docs.docker.com/installation/

* list running dockers

    docker ps
    
* list stopped dockers too

    docker ps -a
    
* Remove all stopped dockers

    docker rm $(docker ps --filter=status=exited -q)
    
* List docker images

    docker images

* Remove unused docker images

    docker rmi $(docker images -a --filter=dangling=true -q)
    
* Run another command (like bash) in a running docker

    docker exec -i {docker name} {command}


# Troubleshooting #

1. I started one of the script, and am trying to stop it with Ctrl+C. It is
ignoring many Ctrl+C's

    This happens a lot when calling programs like apt-get, yum, etc... From the
    host, you can still use ps, pgrep, kill, pkill, etc... commands to kill the
    PIDs in a docker. You can also use ```docker ps``` to find the container
    name/id and then used ```docker stop``` (signal 15) or ```docker kill```
    (signal 9) to stop the docker
    
2. How do I re-enter a docker after it failed/succeeded?

    Dockers are immediately deleted upon exit. The best way to work in a docker
    is to run bash. This will let you to run the main build command and then
    continue.
    
3. That answer's not good enough. How do I resume a docker?

    Well, first you have to set the environment variable ```AUTO_REMOVE=0``` 
    before running the image you want to resume. This will keep the docker 
    around after stopping. (Be careful! They multiply like rabbits.) Then
    
        docker commit {container name/id} {new_name}
    
    Then you can ```docker run``` that new image.

4. Everything in the ./repos directory is owned by root

    That is currently a side effect of the dockers being run as root.