photo of Cosimo Streppone

Random hacking

Assume nothing. Code defensively. Keep it simple, stupid!

"DebPAN", a production-grade Debian CPAN repository

Tuesday, July 6, 2010 2:58:19 PM

, , , , ,

The problem

This is a proposal I came up with after talking to Gabor Szabo about his Perl Ecosystem Development proposal.

One of the major "problems" we face while developing and deploying production Perl-based systems with Debian is that the state of the Debian CPAN modules is depressingly outdated. As an example, we're using Catalyst in Lenny, and that dates back to 2008 for the most parts.

This is just not enough.

A solution: maintain your own APT repository

Our current solution is to manually package every bit and maintain our own internal Opera APT repository that our servers and applications depend on. That's not optimal for two reasons:

  • we have to package lots of modules, due to interdependencies, dedicating a fair amount of time to this activity that is not exactly "productive"
  • we can't trust our systems to be Debian anymore, since we're updating bits and pieces with the bold assumption that everything will work fine

Of course, 99.999% will work fine, since it's Perl, but the problem is that one day this could fall down on our heads.

So, given the problem, what are the solutions?

  • Continuing to manually keep an apt repository. Downside: Some(tm) waste of time
  • Use a different packaging/deployment system, like PAR::Repository. I would personally like this, but it doesn't eliminate the need to maintain an own repository. You just don't use dh-make-perl and friends, that are, IMHO, nice to have and useful
  • The "DebPAN"

The DebPAN

I know Jeremiah Foster, and at a couple of Perl events I heard him talking to other CPAN/Perl developers about these issues. His answer would probably be to file a request for packaging for the modules we're interested in.

The reason why that doesn't work is that the lead time for a given RFP to land on Debian stable is unacceptably long, and I realize that is for a good reason. After all, it's supposed to be stable, right?

What about a "DebPAN" repository?

That could be a 3rd party APT repository, something like debpan.perl.org:

  • maintained by a close group of Perl/Debian/CPAN developers
  • guaranteed to have a selection of the most important modules (more on that...) in a reasonably recent version
  • targeted to Debian stable, and maybe other distributions? I think Ubuntu 10.04 suffers from this same problem, but much less than Debian Lenny, just to pick two versions
  • maybe even with patches applied?, but that might be way too much, actually

The can of worms

Of course, lots of problems can arise. However, if we think this is a good idea, then we should try to have something even minimal up and running. Then we'll worry about all the problems...

However, who gets to decide the most useful modules? That should go by popular demand I guess. Even looking at Debian requests for packaging stats, maybe? I can also imagine that bigger companies using Perl would be interested in this to potentially save lots of "infrastructural work".

I'd be really interesting to know other people opinions on this, especially if they use Debian stable, Debian developers, or the Debian-Perl group itself.

Cache::Memcached::Mock, instant in-process memcached mockPerl 6 LWP::Simple gets chunked transfers support

Comments

Anonymous Tuesday, July 6, 2010 3:42:59 PM

Jaldhar writes: Do you know about backports.org? (http://www.backports.org/) This is a central repository for packages backported from testing. Not unstable so you still don't have the absolute bleading edge but you have a reasonable assumption of stability. Only DDs can upload .debs there afaik but I think it would go a long way towards solving the problems you've described.

Cosimo Strepponecstrep Tuesday, July 6, 2010 4:00:23 PM

Sure, I know about backports, and we used it sometimes, but what I had in mind is targeted specifically to the Perl world. It would provide a cohesive set of up to date CPAN modules.

Maybe backports covers that already. I'll look into it.

Anonymous Tuesday, July 6, 2010 4:07:12 PM

mo writes: what's the difference to http://debian.pkgs.cpan.org/? It's not maintained anymore but it is the same idea, isn't it?

Cosimo Strepponecstrep Tuesday, July 6, 2010 4:18:31 PM

@mo: yes, it looks like the same. One difference could be that a completely automated cpan-to-deb doesn't guarantee consistency or the general cohesion of the archive.

Say that you package Plack, you should also ideally package the most used 3rd-party middleware plugins. Just an example...

Anonymous Tuesday, July 6, 2010 4:21:09 PM

phaylon writes: I think the problem with these hand-pick approaches is that people aren't usually just interested in the most popular modules. If I'm building a web application, I'm not only interested in my web framework and ORM, but also all other modules that are somehow related to my core business model. And that can be just about anything. I personally don't use this approach. I like using local-lib or even a whole separate business perl that is separate from the system one. But I always wondered if the process of building .deb's out of Perl distribution tarballs can't be automated to a certain degree. You might not be able to automatically convert all of them, but if for example the distribution page on CPAN would show a "Provided by DebPAN" field, the community effect might kick in and people might want to provide distributions that build cleanly. One might even start sending emails in a CPAN Testers style to inform authors of build fails.

Anonymous Tuesday, July 6, 2010 4:56:29 PM

Daniel Ruoso writes: I am one of the initiators of the Debian Perl Group (having been MIA mostly in the last years). And one of the things I have been wondering since then but haven't the energy to do so was to build what you called DebPAN. In my head, this repository would be built from the same packages available on Debian, just doing a re-build targetting the stable Perl (just like backports, but using a buildd). New versions of packages as well as new packages would be handled by the Debian Perl Group and uploaded to Debian unstable, being automatically re-built targetting the stable Perl. There's one other important thing, tho, which is the stability of the repository. Even if some CPAN authors take much care in the API stabiltiy, that is not the case for every module, in that sense, it's important to have a stability of the distribution. This is what Debian does already, but it's life-cycle is too long for the Modern Perl. What I would suggest, additionally to doing an automated build of all perl modules in Debian unstable targetting Debian stable, would be to maintain a shorter life cycle for a "stable perl distro" (maybe 6-month-release cycle)...

Anonymous Tuesday, July 6, 2010 8:59:16 PM

poisonbit writes: My perl modules over Debian leny use to go using Local::Lib. You can deploy the modules by application/user/etc without touching the system modules. Other options are: dh_make_perl (works as a charm having the -dev deppends installed when needed by XS modules) minicpan cpan (directly, but I see this much worse than Local::Lib) Anyway, I like too much the idea of having this issue solved in a repo, but I see more safe "no to change" system modules. Greetings

Anonymous Thursday, July 8, 2010 12:43:17 AM

Hercynium writes: I did a talk on a closely related subject at this year's YAPC::NA. The boiled-down thesis of the talk was that the CPAN is a constantly moving target - there is no "stable," at least, not one that will work for everybody all the time. To take control of the situation, use a minicpan, under version control, and install everything to either a private perl (like with perlbrew) or using local::lib. With such a setup, deploying your perl code is always 100% repeatable. This is simply impossible to have using the CPAN directly. If an minicpan update breaks something, roll back to the previous commit and fix. For the past year, on all the systems I have deployed, the only cpan dist I ever install to the system perl is local::lib. After that, nobody is ever allowed to run cpan as root - including myself. (of course, libfoo.*-perl debian packages are just fine) I have avoided the whole "just build .deb packages from every dist you need" because a. we have a mixed env - CentOS and Debian, b. I've had odd breakages and frustrating dependency issues building my own .debs from cpan dists, and c. the minicpan route really is quite easy :)

Anonymous Thursday, July 8, 2010 3:29:53 PM

Jozef Kutej writes: For some time I'm also experimenting with packaging. You can have a look at Debian::Apt::PM that can index Perl modules in Debian repositories. This solves the problem of mapping the minimal Debian package version to required Perl module version. There is also a dh-make-pm script that can recursively build Debian packages. dh-make-pm uses CPAN::Patches::Plugin::Debian to apply custom/local changes to the debian/ folder to automatically fix the c lib dependencies or add patches. If that sounds interesting to anyone then let me know.

How to use Quote function:

  1. Select some text
  2. Click on the Quote link

Write a comment

Comment
(BBcode and HTML is turned off for anonymous user comments.)

If you can't read the words, press the small reload icon.


Smilies

More about me