Raphael's Blog

The first thing to do is of course fetch the sources:

$ svn export https://neo.yahostme.com/svn/DeBaBaReTools/trunk/ DeBaBaReTools

$ git clone git://git.debian.org/git/users/atomo64-guest/debabaretools.git

Build and install the package:

$ cd DeBaBaReTools; dpkg-buildpackage -b; dpkg -i ../debabaretools_0.2_all.deb

$ dpkg -L debabaretools
/.
/etc
/etc/DeBaBaReTools
/etc/DeBaBaReTools/common.conf
/usr
/usr/bin
/usr/bin/wannaBuild
/usr/bin/needsBuild
/usr/bin/needsReBuild
/usr/bin/buildQueue
/usr/bin/checkSums
/usr/bin/wannaUpload
/usr/bin/processIncoming
/usr/share
/usr/share/doc
/usr/share/doc/debabaretools
/usr/share/doc/debabaretools/copyright
/usr/share/doc/debabaretools/ENV
/usr/share/doc/debabaretools/changelog.gz
/usr/share/doc/debabaretools/TODO
/usr/share/DeBaBaReTools
/usr/share/DeBaBaReTools/backends
/usr/share/DeBaBaReTools/backends/config
/usr/share/DeBaBaReTools/backends/config/reprepro.sh
/usr/share/DeBaBaReTools/backends/probe.sh
/usr/share/DeBaBaReTools/backends/fetch
/usr/share/DeBaBaReTools/backends/fetch/dget.sh
/usr/share/DeBaBaReTools/backends/verbosity.sh
/usr/share/DeBaBaReTools/backends/lock
/usr/share/DeBaBaReTools/backends/lock/default.sh
/usr/share/DeBaBaReTools/backends/setDefault.sh
/usr/share/DeBaBaReTools/backends/buildQueue
/usr/share/DeBaBaReTools/backends/buildQueue/plaintext.sh
/usr/share/DeBaBaReTools/backends/signature
/usr/share/DeBaBaReTools/backends/signature/gpgv.sh
/usr/share/DeBaBaReTools/backends/check
/usr/share/DeBaBaReTools/backends/check/default.sh
/usr/share/DeBaBaReTools/backends/build
/usr/share/DeBaBaReTools/backends/build/pbuilder.sh
/usr/share/DeBaBaReTools/backends/upload
/usr/share/DeBaBaReTools/backends/upload/dput.sh
/usr/share/DeBaBaReTools/backends/parsers
/usr/share/DeBaBaReTools/backends/parsers/default.sh
/usr/share/DeBaBaReTools/backends/repository
/usr/share/DeBaBaReTools/backends/repository/shared.sh
/usr/share/DeBaBaReTools/backends/repository/reprepro.sh
/usr/share/DeBaBaReTools/backends/misc.sh
/usr/share/DeBaBaReTools/backends/codenamesTable
/usr/share/DeBaBaReTools/backends/codenamesTable/release.sh
/usr/share/DeBaBaReTools/backends/codenamesTable/reprepro.sh
/usr/share/DeBaBaReTools/backends/config/default.sh
/usr/share/DeBaBaReTools/backends/fetch/default.sh
/usr/share/DeBaBaReTools/backends/buildQueue/default.sh
/usr/share/DeBaBaReTools/backends/signature/default.sh
/usr/share/DeBaBaReTools/backends/build/default.sh
/usr/share/DeBaBaReTools/backends/upload/default.sh
/usr/share/DeBaBaReTools/backends/repository/default.sh
/usr/share/DeBaBaReTools/backends/codenamesTable/default.sh

Now you have to create a base reprepro repository. I strongly recommend setting the REPREPRO_BASE_DIR ENV variable if possible.

A recommended repository structure is (in repoitory/):

.:
./build:
./build/cache:
./build/data:
./build/data/logs:
./build/tmp:
./conf:
distributions
incoming
options
tiffany.py
uploaders
./db:
./debian:
./debian/dists:
etch
oldstable
sarge
sid
stable
testing
unstable
./debian/dists/...
./debian/pool/...
./incoming:
./incoming/accepted:
./incoming/new:
./incoming/rejected:
./pub:
./pub/UploadQueue:

Note that the build/, incoming/ and pub/ directory structures are DeBaBaReTools' defaults, if you want to change them you will have to set some variables at the configuration file ($HOME/.DeBaBaReToolsrc).

The distributions file must at least contain the Codename, Suite and Architectures entries for DeBaBaReTools to work correctly. Note that the current code used to correlate the suites with the codenames greps for the given codename/suite including two lines before and two after the match, so the entry must be on that scope.
E.g.

$ cat conf/distributions
Codename: sid
Suite: unstable
Architectures: i386 amd64 source
Components: main contrib non-free
Uploaders: uploaders
DebIndices: Packages Release .bz2 .gz .
DscIndices: Sources Release . .gz .bz2
Contents: .bz2 .gz .
Tracking: minimal includechanges keepsources

conf/incoming must name at least one rule. This is because when installing, DeBaBaReTools will grep for all the rule names and execute them one by one.
The rest of reprepro's setup is trivial.

When building packages with pbuilder, DeBaBaReTools expects the basetgz's to be named like $PBUILDER_CACHE/$CODENAME.tgz where $PBUILDER_CACHE is an environment variable defaulting to /var/cache/pbuilder.

Tweaking DeBaBaReTools:
Because we are using reprepro and pbuilder we'll give some hints to DeBaBaReTools so it uses a reprepro-specific and/or pbuilder-specific backend where possible.

$ echo 'HINT_ALL="reprepro pbuilder"' > $HOME/.DeBaBaReToolsrc

The .DeBaBaReToolsrc file is sourced by all scripts, so you might even set variables like $REPREPRO_BASE_DIR there.

/usr/share/doc/debabaretools/ENV documents only some of the multiple environment variables that can be specified to customise DeBaBaReTools' behaviour. It is important to say that variables which are empty will be filled with a default value.

You may now want to modify .dput.cf by adding a 'local' target for your repository:

[local]
fqdn = localhost
method = local
incoming = /home/raphael/Deb/repository/pub/UploadQueue
allow_unsigned_uploads = 0
run_dinstall = 0

So let the fun begin!

To make sure everything is ok let's run processIncoming in verbose mode:

VERBOSE=1 processIncoming
processIncoming v0.1 started.
processIncoming v0.1 stopped.

Not getting that output means something is not ok.

All the scripts will try to source a file called $HOME/.${APP_NAME}rc, where $APP_NAME is the name of the script, e.g. processIncoming. This can be used to customise them even more.

Tweaking the scripts behaviour:

processIncoming offer some options such as:

• NEW_POLICY, default: binary. Meaning: uploads introducing new binaries will be moved to $NEW
  
• REJECTED_POLICY: move(default)/kill: action taken on non-reclaimed (by a .changes file) files
  
• CHECK_SIGNATURE: 1(default)/0, whether to check signatures on the .changes file or not
  
• INSTALL: 1/0(default), whether to run the repository's install rules, this will usually be set at run-time (e.g. daily cronjob)
  
• KEYRING: default: uploaders.gpg, the keyring with keys allowed to upload
  

wannaBuild:

• BUILD_LOGS_DIR: $BUILD_DATA/logs, directory where build logs are stored
  
• SIGN: see the ENV file for details
  
• SIGN_COMMAND: see the ENV file for details
  
• DEB_HOST_ARCH: see the ENV file for details
  
• BUILD_DATA_CACHE: $BUILD_DIR/cache, directory where tools such as dget may store data
  
• KEYRING: secring.gpg (needs to be changed), keyring file containing the keys to accept
  
• UPDATE_ENVIRONMENT: see the ENV file for details
  
• GAINROOT: see the ENV file for details
  
• DISTROS_TO_BUILD: $SUPPORTED_DISTROS (based on the repository's backend information), list of distributions (e.g. unstable) wannaBuild is going to build for.
  

When you should call which DeBaBaReTools script:
processIncoming takes care of accepting package uploads and installing them when told to do so.
needsBuild processes the .changes files accepted by processIncoming and tells the build queue manager that a given package needs to be built, or when the binary package is also uploaded, or is already built.
wannaBuild only takes care of building source packages (note that arch: all-onnly packages WON'T be built, they must be uploaded together with the source package).

So the workflow is:
processIncoming -> needsBuild -> INSTALL=1 processIncoming

A package won't be installed in the repository if it isn't handled before by needsBuild.

Here's a demonstration (note the usage of VERBOSE=1):

$ dput local kcometen3_1.1-1_source.changes
Checking Signature on .changes
...
Uploading to local (via local to localhost):
Successfully uploaded packages.
Not running dinstall.
sh: /usr/bin/mini-dinstall: No such file or directory
Error while executing command.
Continuing...

The package is now at pub/UploadQueue

$ VERBOSE=1 processIncoming
processIncoming v0.1 started.
Processing /home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1_source.changes...
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1.diff.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1.dsc" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1_source.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1.orig.tar.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
processIncoming v0.1 stopped.

After checking sums and signature, the upload is accepted.

$ VERBOSE=1 needsBuild
needsBuild v0.2 started.
Architectures: i386 amd64 all
Now processing /home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1_source.changes
Adding kcometen3 to be built in i386 for the sid distribution
Adding kcometen3 to be built in amd64 for the sid distribution
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1.dsc" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1.orig.tar.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1.diff.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1_source.changes" to "/home/raphael/Deb/repository/incoming"
needsBuild v0.2 stopped.

The package is now marked to be built on i386 and amd64 for sid (note that distro to codename conversion is performed) and moved to incoming/ so processIncoming is able to install it.

$ VERBOSE=1 processIncoming
processIncoming v0.1 started.
processIncoming v0.1 stopped.

There's nothing under pub/UploadQueue and nothing is installed (do you remember the INSTALL=0 option? )

$ INSTALL=1 VERBOSE=1 processIncoming
processIncoming v0.1 started.
No distribution found for 'kcometen3_1.1-1_source.changes'!
There have been errors!
No distribution found for 'kcometen3_1.1-1_source.changes'!
There have been errors!
No distribution found for 'kcometen3_1.1-1_source.changes'!
There have been errors!
making diffs between /home/raphael/Deb/repository/debian/dists/sid/main/source/Sources and /home/raphael/Deb/repository/debian/dists/sid/main/source/Sources.new:
generating diff
Exporting indices...
processIncoming v0.1 stopped.

The errors can be safely ignored, it is just that reprepro couldn't install kcometen3_1.1-1_source.changes for the specified install rule (I've one for each distro, that's why).

$ VERBOSE=1 wannaBuild
wannaBuild v0.1 started.
Fetching list of packages to be built for unstable
Fetching source package kcometen3 v1.1-1
Use of uninitialized value in string eq at /usr/bin/dget line 377.
cat: /home/raphael/Deb/repository/build/data/needsBuild.unstable: No such file or directory
Starting build backend...
Executing sudo pbuilder --update --basetgz /var/cache/pbuilder/unstable.tgz
Starting build process
Verifying signature of /home/raphael/Deb/repository/build/kcometen3jb3362/kcometen3_1.1-1_i386.changes...
gpgv: no valid OpenPGP data found.
gpgv: the signature could not be verified.
Please remember that the signature file (.sig or .asc)
should be the first file given on the command line.
Fetching list of packages to be built for testing
Fetching list of packages to be built for stable
Fetching list of packages to be built for oldstable
wannaBuild v0.1 stopped.

There are still some things that need to be fixed (e.g. the no such file or directory message).
Other than that, the output is just fine: the package wasn't automatically signed so when trying to check the signature it failed to find it thereby not uploading the package automatically.

Now we cd to build/kcometen3jb3362/, review the build log, the .changes file, and we finally sign it.
Afterwards, we upload the package and install it:

$ VERBOSE=1 processIncoming
processIncoming v0.1 started.
Processing /home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1_i386.changes...
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1_i386.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcometen3_1.1-1_i386.deb" to "/home/raphael/Deb/repository/incoming/accepted/"
processIncoming v0.1 stopped.

$ VERBOSE=1 needsBuild
needsBuild v0.2 started.
Architectures: i386 amd64 all
Now processing /home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1_i386.changes
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1_i386.deb" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcometen3_1.1-1_i386.changes" to "/home/raphael/Deb/repository/incoming"
needsBuild v0.2 stopped.

$ INSTALL=1 VERBOSE=1 processIncoming
processIncoming v0.1 started.
No distribution found for 'kcometen3_1.1-1_i386.changes'!
There have been errors!
No distribution found for 'kcometen3_1.1-1_i386.changes'!
There have been errors!
No distribution found for 'kcometen3_1.1-1_i386.changes'!
There have been errors!
making diffs between /home/raphael/Deb/repository/debian/dists/sid/main/binary-i386/Packages and /home/raphael/Deb/repository/debian/dists/sid/main/binary-i386/Packages.new:
generating diff
Exporting indices...
processIncoming v0.1 stopped.

done

We can now setup some cronjobs:
*/10 * * * * processIncoming
*/5 * * * * needsBuild
@daily INSTALL=1 processIncoming

needsReBuild and uploadQueue still need to be written, so don't expect them to do anything right now.

I hope people start using it so I receive more feedback

Update: code is now in a git repository, no longer in svn at neo.