2015-07-18 02:52:50 +00:00
|
|
|
# README #
|
|
|
|
|
|
|
|
## TL;DR version ##
|
2015-07-19 22:59:14 +00:00
|
|
|
1. Run the dockers
|
2015-07-18 02:52:50 +00:00
|
|
|
|
|
|
|
./docker/run_dockers.bsh
|
2015-07-19 22:59:14 +00:00
|
|
|
|
|
|
|
2. ???
|
2015-07-18 02:52:50 +00:00
|
|
|
|
|
|
|
3. **Enjoy** all your new package files in
|
|
|
|
|
|
|
|
./docker/repos/
|
|
|
|
|
|
|
|
##Using the Dockers##
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
### Running Dockers ###
|
|
|
|
|
|
|
|
In order to run the dockers, the docker has to be built, and then run with a
|
|
|
|
lot of arguments to get the mount points right, etc... A convenient script is
|
|
|
|
supplied to make this all easy. By running
|
|
|
|
|
|
|
|
./docker/run_docker.bsh
|
|
|
|
|
|
|
|
All the git-lfs_* images are built automatically, and then run. The default
|
|
|
|
command is the build script. The currently checkout version of git-lfs is
|
|
|
|
copied into the docker, and ```git clean -xdf``` is run to remove any
|
|
|
|
non-tracked files, but non-commited changes are kept. Then git-lfs is built,
|
|
|
|
and a pacakges is created (deb or rpm)
|
|
|
|
|
|
|
|
To only run certain docker images, supply them as arguments, e.g.
|
|
|
|
|
|
|
|
./docker/run_docker.bsh git-lfs_debian_7
|
|
|
|
./docker/run_docker.bsh git-lfs_debian_*
|
|
|
|
./docker/run_docker.bsh git-lfs_*[6-8]
|
|
|
|
|
|
|
|
And only those images will be run.
|
|
|
|
|
|
|
|
There are a few environment variables you can set to easily adjust the behavior
|
|
|
|
of the run_docker.bsh script.
|
|
|
|
|
|
|
|
REPO_HOSTNAME - Override the hostname for all the repos generated/tested (see below)
|
|
|
|
|
|
|
|
BUILD_LOCAL - Set to 1 (default) to use the currently checked out version of
|
|
|
|
the git-lfs to build against. If it's not 1 the released archived is downloaded
|
|
|
|
and built against. Currently only works for RPMs. DEB always builds the currently
|
|
|
|
checkout version. Build local only affect the version of the code used in generating
|
|
|
|
the rpms, not the scripts running to generate the rpms.
|
|
|
|
|
|
|
|
DOCKER_AUTOBUILD - Default 1. run_docker.bsh always calls build_docker.bsh to
|
|
|
|
ensure your docker image is uptodate. Somtimes you may not want this. If set
|
|
|
|
this to 0, it will not build docker images before running
|
|
|
|
|
|
|
|
AUTO_REMOVE - Default 1. Docker instances are automatically deleted on program
|
|
|
|
exit. If set to 0, the docker instance will not be automatically deleted upon
|
|
|
|
exit. This can be useful for a post mortem analysis (using other docker commands
|
|
|
|
not covered here). Just make sure you clean up the docker instances manually.
|
|
|
|
|
|
|
|
###Development with Dockers###
|
|
|
|
|
|
|
|
Sometimes you don't want to just build git-lfs and destroy the container, you
|
|
|
|
want to get in there, run a lot of command, DEVELOP! To do this, the best
|
|
|
|
command to run is bash, and then you have an interactive shell to use. To do this
|
|
|
|
|
|
|
|
./docker/run_docker.bsh {image(s)} -- bash
|
|
|
|
|
|
|
|
After listing the image(s) you want to run, add a double dash (--) and then any
|
|
|
|
command (and arguments) you want executed in the docker. Remember, the command
|
|
|
|
you are executing has to be part of the docker image.
|
|
|
|
|
2015-07-19 22:59:14 +00:00
|
|
|
### Building Dockers (Optional) ###
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
```run_dockers.bsh``` calls build_dockers ```build_dockers.bsh```, but you
|
2015-07-19 22:59:14 +00:00
|
|
|
can still call the script manually to get it all out of the way once while you
|
2015-07-23 02:45:02 +00:00
|
|
|
go make some tea/coffee.
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-19 22:59:14 +00:00
|
|
|
In order to use the docker **images**, they have to be built so that they are
|
|
|
|
ready to be used. For OSes like Debian, this is a fairly quick process.
|
2015-07-18 02:52:50 +00:00
|
|
|
However CentOS takes considerably longer time, since it has to build go, ruby,
|
2015-07-23 02:45:02 +00:00
|
|
|
or git from source, depending on the distro. Fortunately, you can build the
|
|
|
|
docker images JUST once, and you won't have to build it again (until the version
|
|
|
|
changed.) The build script uses a downloaded release from github of git-lfs to
|
|
|
|
bootstap the centos image and build/install all the neccessary software. Currently
|
|
|
|
the only way to change what version the image is built off of is by changing the
|
|
|
|
url in the Dockerfile for git-lfs_centos_*.
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
This means all the compiling, yum/apt-get/custom dependency compiling is done
|
|
|
|
once and saved. (This is done in CentOS by using the already existing
|
|
|
|
```./rpm/rpm_build.bsh``` script to bootstrap the image and saving the image.)
|
|
|
|
|
|
|
|
The script that takes care of ALL of these details for you is
|
2015-07-18 02:52:50 +00:00
|
|
|
|
|
|
|
./docker/build_dockers.bsh
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
All the git-lfs_* images will be built automatically. These are all a
|
|
|
|
developer would need to test the different OSes. And create just git-lfs rpm or
|
|
|
|
deb pacakges.
|
|
|
|
|
|
|
|
However, in order to distribute git-lfs or build dependencies, the packages
|
|
|
|
that were installed for you by ```build_docker.bsh``` need to be saved too.
|
|
|
|
This is currently only a CentOS problem. In order to genereate THOSE rpms,
|
|
|
|
the git-lfs-full-build_* will use NON-bootstrapped images to build every pacakge
|
|
|
|
and git-lfs and generate rpms. This takes as long as building the image in the
|
|
|
|
first place, but you don't get the benifit of a saved state. The
|
|
|
|
```./rpm/rpm_build.bsh``` script will build all of its dependencies when you
|
|
|
|
run the dockers, making the rpms available. These images can be built by running
|
2015-07-18 02:52:50 +00:00
|
|
|
|
|
|
|
./docker/docker_build.bsh ./docker/git-lfs-full-build_*
|
|
|
|
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
This is most important for CentOS 6 where git 1.8.2 or newer is not available,
|
|
|
|
only git 1.7.1 is available, so every user either has to build git from source,
|
|
|
|
or use the rpms generated by the ```git-lfs-full-build_centos_6``` image.
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
(To manually build a docker image, run
|
|
|
|
```docker build -t $(basename ${DockerDir}) -f ${DockerDir}/Dockerfile ./docker```
|
2015-07-18 02:52:50 +00:00
|
|
|
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
##Deploying/Building Repositories##
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
When ```./docker/run_dockers.bsh``` is done building git-lfs and the rpms/deb,
|
|
|
|
it actually creates a repository for distubution too. Each distro gets a repo
|
|
|
|
gerenated in ```./docker/repos/{DISTRO_NAME}/{VERSION #}```. Just drop the repo
|
|
|
|
directory onto a webserver and you have a fully functioning linux repo. (See
|
|
|
|
Testing the Repositories below for more detail)
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
The two major packages included are:
|
|
|
|
git-lfs-....* - the git-lfs package
|
|
|
|
git-lfs-repo-release....* - A package to install the repo.
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
When using ```BUILD_LOCAL=1```, all UNTRACKED files are removed during RPM
|
|
|
|
generation, except any stray directories containing a .git folder will not be
|
|
|
|
cleared. This shouldn't be the case, unless you are temporarily storing another
|
|
|
|
git repo in the git repo. This is a safty mechanism in git, so just keep in mind
|
|
|
|
if you are producing packages.
|
|
|
|
|
2015-07-18 19:16:54 +00:00
|
|
|
### Setting the website URL ###
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
The git-lfs-repo-release must contain the URL where the repo is to be hosted.
|
|
|
|
The current default value is 'git-lfs.github.com' but this can be overridden
|
|
|
|
using the REPO_HOSTNAME env var, e.g.
|
|
|
|
|
|
|
|
REPO_HOSTNAME=www.notgithub.uk.co ./docker/run_dockers.bsh
|
|
|
|
|
|
|
|
Now all the git-lfs-repo-release....* files will point to that url instead
|
|
|
|
|
2015-07-18 19:16:54 +00:00
|
|
|
### GPG signing ###
|
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
For private repo testing, GPG signing can be skipped. apt-get and yum can
|
|
|
|
install .deb/.rpm directly without gpg keys and everything will work. This
|
|
|
|
section is for distribution in a repo. Most if not all this functionality is
|
2015-07-23 02:45:02 +00:00
|
|
|
automatically disabled when there is no signing key present
|
|
|
|
(```./docker/signing.key```).
|
2015-07-20 05:40:24 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
In order to sign packages, you need to generate and place GPG keys in the right place
|
2015-07-20 05:40:24 +00:00
|
|
|
|
|
|
|
1. gpg --gen-key
|
|
|
|
|
|
|
|
1. 1 - RSA and RSA
|
|
|
|
2. 4096 bits
|
|
|
|
3. Some length of time or 0 for infinite
|
|
|
|
4. y for yes
|
|
|
|
5. Signer name (Will become part of the key and uid)
|
|
|
|
6. Email address (Will become part of the key and uid)
|
|
|
|
7. Comment (Will become part of the key)
|
|
|
|
8. O for Okay
|
|
|
|
9. Enter a very secure password, make sure you will not forget it
|
|
|
|
10. Generate Entropy!
|
|
|
|
|
|
|
|
2. gpg -a --export > ./docker/public.key
|
|
|
|
|
|
|
|
3. gpg --export-secret-keys > ./docker/signing.key
|
2015-07-18 19:16:54 +00:00
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
Keep in mind, signing.key must NEVER be accidentally commited to the repo.
|
2015-07-19 22:59:14 +00:00
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
To prevent MANY passphrase entries at random times, the gpg-agent is used to
|
|
|
|
cache your signing key. This is done by running gpg-agent in the host, and passing
|
|
|
|
the connection to each docker image. This will be done for you automatically by
|
|
|
|
calling the ```./docker/preload_key.bsh``` script. This can be called manually
|
|
|
|
before any other command just to get the pass phrase entry out of the way before
|
|
|
|
you start running everything.
|
2015-07-19 22:59:14 +00:00
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
GPG agent ttl set to 5 hours, should be plenty to build everything. If this is
|
|
|
|
not good for you, set the GPG_MAX_CACHE and GPG_DEFAULT_CACHE environment variables
|
|
|
|
(in seconds)
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
### GPG capabilited by Distro ###
|
|
|
|
|
|
|
|
Debian WILL work with 4096 bit RSA signing subkeys like [1] suggests, but will
|
|
|
|
also work with 4096 bit RSA signing keys.
|
|
|
|
|
|
|
|
CentOS will NOT work with subkeys. CentOS 6 and 7 will work with 4096 bit RSA
|
|
|
|
signing keys
|
|
|
|
|
|
|
|
CentOS 5 will NOT work with v4 signatures. The rpms will be so unrecognizable
|
|
|
|
that it can't even be installed with --nogpgcheck. It should work with RSA on v3.
|
|
|
|
However, I could not get it to it builds v3 correctly now, but for some reason
|
|
|
|
the GPG check fails. However ```yum install --nogpgcheck``` does work! CentOS 5
|
|
|
|
will NOT work with 2048 bit DSA keys... I suspect 2048 is too big for it to
|
|
|
|
fathom. CentOS 5 WILL work with 1024 bit DSA keys. Either sign it with the key
|
|
|
|
instructinos above and install it with the ```yum install --nogpgcheck``` OR
|
|
|
|
create a NEW DSA key just for CentOS 5.
|
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
[1] https://www.digitalocean.com/community/tutorials/how-to-use-reprepro-for-a-secure-package-repository-on-ubuntu-14-04
|
|
|
|
[2] https://iuscommunity.org/pages/CreatingAGPGKeyandSigningRPMs.html#exporting-the-public-gpg-key
|
|
|
|
[3] http://www.redhat.com/archives/rpm-list/2006-November/msg00105.html
|
2015-07-19 22:59:14 +00:00
|
|
|
|
2015-07-19 07:35:13 +00:00
|
|
|
### Testing the Repositories ###
|
|
|
|
|
2015-07-20 05:40:24 +00:00
|
|
|
To test that all the OSes can download the rpm/debs, install, and run the tests
|
|
|
|
again, run
|
|
|
|
|
|
|
|
./test_dockers.bsh
|
|
|
|
|
|
|
|
(which is basically just ```./docker/run_dockers.bsh ./docker/git-lfs-test_*```)
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
Remember to set REPO_HOSTNAME if you changed it for ```./docker/build_docker.bsh```
|
|
|
|
This can also be used to run a local test (on ```localhost:{Port Number}```, for
|
|
|
|
example)
|
2015-07-20 05:40:24 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
An easy way to test the repositories locally, is to run host them on a webserver such as
|
2015-07-20 05:40:24 +00:00
|
|
|
|
|
|
|
cd ./docker/repos
|
|
|
|
python -m SimpleHTTPServer {Port number}
|
|
|
|
|
|
|
|
or
|
|
|
|
|
|
|
|
cd ./docker/repos
|
|
|
|
ruby -run -ehttpd . -p{Port Number}
|
|
|
|
|
2015-07-19 07:35:13 +00:00
|
|
|
|
2015-07-18 02:52:50 +00:00
|
|
|
## Adding addition OSes ##
|
|
|
|
|
|
|
|
To add another operating system, simply follow the already existing pattern,
|
2015-07-23 02:45:02 +00:00
|
|
|
and all the script will pick them up. A new Dockerfile should go in a directory
|
2015-07-18 02:52:50 +00:00
|
|
|
named
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
./docker/git-lfs_{OS NAME}_{OS VERSION #}
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
where **{OS NAME}** and **{OS VERSION #}** should not contain underscores (_).
|
2015-07-18 02:52:50 +00:00
|
|
|
Any files that need to be added to the docker image can be dropped in the
|
2015-07-23 02:45:02 +00:00
|
|
|
```./docker/common``` directory, since that is the root they are built against
|
|
|
|
(not the directory containing the Dockerfile like most dockers)
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
The docker image should run a script that write it's repo files to the /repo
|
|
|
|
direcrtory inside the docker, and writing to /repo in the docker will cause the
|
|
|
|
files to end up in
|
2015-07-18 02:52:50 +00:00
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
./docker/repos/{OS NAME}/{OS VERSION #}/
|
2015-07-18 02:52:50 +00:00
|
|
|
|
|
|
|
## Docker Cheat sheet ##
|
|
|
|
|
|
|
|
http://docs.docker.com/ Install -> Docker Engine -> Installation on ...
|
2015-07-19 22:59:14 +00:00
|
|
|
|
|
|
|
* list running dockers
|
|
|
|
|
|
|
|
docker ps
|
|
|
|
|
|
|
|
* list stopped dockers
|
|
|
|
|
|
|
|
docker ps -a
|
|
|
|
|
|
|
|
* Remove all stopped dockers
|
|
|
|
|
|
|
|
docker rm $(docker ps --filter=status=exited -q)
|
|
|
|
|
|
|
|
|
|
|
|
# Troubleshooting #
|
|
|
|
|
|
|
|
1. I started one of the script, and am trying to stop it with Ctrl+C. It is
|
|
|
|
ignoring many Ctrl+C's
|
|
|
|
|
|
|
|
This happens a lot when calling programs like apt-get, yum, etc... From the
|
|
|
|
host, you can still use ps, pgrep, kill, pkill, etc... commands to kill the
|
|
|
|
PIDs in a docker.
|
|
|
|
|
|
|
|
2. How do I re-enter a docker after it failed/suceeded?
|
|
|
|
|
|
|
|
Dockers are immediately deleted upon exit. The best way to work in a docker
|
|
|
|
is to run bash. This will let you to run the main build command and then
|
|
|
|
continue.
|
|
|
|
|
|
|
|
3. That answer's not good enough. How do I resume a docker?
|
|
|
|
|
2015-07-23 02:45:02 +00:00
|
|
|
Well, first you have to set the environment variable ```AUTO_REMOVE=0```
|
|
|
|
before running the image you want to resume. This will keep the docker
|
|
|
|
around after stopping. (Be careful! They multiply like rabbits.) Then
|
2015-07-19 22:59:14 +00:00
|
|
|
|
|
|
|
```docker commit {container name/id} {new_name}```
|
|
|
|
|
|
|
|
Then you can ```docker run``` that new image.
|
|
|
|
|