UNIX snapshot update 'one liners' and scripts
Friday, October 2, 2009 7:17:00 AM
Originally posted by ruario:
The debian packages are by far our most popular Linux download so they got a repository first. Additionally there is more that one rpm update system (apt-rpm, Urpmi, Zypper, Smart Package Manager, etc.), and if you start considering non .rpm distros as well the multitude of update systems on Linux/UNIX gets ever larger (slackpkg, slapt-get, Swaret, pacman/Yaourt, Portage, etc.). As you can imagine the decision about what to support is therefore quite hard. Additionally, a number of distros (Gentoo, Arch, SUSE, Linux Mint, etc.) include us in their own official or community repositories anyway meaning that further repositories are often unnecessary. All that said, we are of course constantly reviewing this situation and we may add an rpm based repository in the future.
Also, it is worth mentioning that snapshots are not currently available on the debian repository anyway, so the users of debian based systems are also downloading these snapshots 'manually'. For more information on this read the "Why no snapshot auto-updates?" section of my recent blog post.
If you feel it is too 'manual', it would be fairly simple to knock up a shell script utilising wget (or curl) to auto grab your favoured package and install it (I'm not recommending you do so but it could be done), e.g. a script that does something like:
1. Pull the rss feed.
2. Extract the first blog post link and grab that page.
3. Extract the 'snapshot.opera.com/unix/....' link.
4. Parse the sub-directory and build number to work out the download link to your favoured package.
5. Grab the package and pass this to your package manager for install (or extract and install with the tar package).
This got me thinking about exactly how easy this would be to do and it occurred to me that I could probably do it with a one liner (albeit a fairly long and ugly one liner). Here is what I came up with after a just a couple of minutes thinking.
Edit: I recently wrote something a little longer that will detect your architecture and download the latest 10.6x+ snapshot build, unpack it and set and title pref so that you can keep track of which build is which:
Before I go any further let me state that these examples have no error checking. They are meant as very quick demonstrations, to show what can be done with relatively little effort. On the other hand, despite the lack of error checking they are all quite usable. Most possible failures would give the same net result: failure to download.
Note: Since I am not error checking in these examples I have not provided any example that passes the file straight to the package manager for install, as I'm slightly concerned that a novice user might take such an example and run it without concerning themselves with the potential problems they may encounter if things did go wrong. However it is trivial to actually implement this even as a one liner (if you so desired), as many package managers can handle a URL as an input string.
The following example is the simplest and demonstrates automatic download of the most recent Linux G3 i386 .rpm package, but you could easily alter this to whatever is appropriate for your system:
wget -nv http://my.opera.com/desktopteam/blog/ -O - | grep snapshot.opera.com\/unix | head -1 | sed -e "s/.*\(http.*\)\" target.*/\1/" | wget -i - -r -erobots=off -nd -np -A gcc4.shared.qt3.i386.rpm
For this first example (only) here is a brief explanation of how the various parts work:
1. Get the HTML contents of the Opera Desktop Team Blog.
wget -nv http://my.opera.com/desktopteam/blog/ -O - |\
2. Find all references to snapshot.opera.com/unix in the raw HTML (these are links to the UNIX snapshot download pages), ignoring everything else.
grep snapshot.opera.com\/unix |\
3. Delete all but the first (most recent) link.
head -1 |\
4. Remove all the extraneous HTML leaving only the actual link itself.
sed -e "s/.*\(http.*\)\" target.*/\1/" |\
5. Using this link, search through the available sub directories for a package that ends 'gcc4.shared.qt3.i386.rpm' and download it, saving nothing else.
wget -i - -r -erobots=off -nd -np -A gcc4.shared.qt3.i386.rpm
Here is another way, this time using curl instead to download the latest equivalent .deb package:
curl -s http://my.opera.com/desktopteam/blog/ | grep snapshot.opera.com\/unix | head -1 | sed -e "s/.*\(http.*\)\" target.*/--url \1intel-linux\//" | curl -s -K - | grep -e title.*intel-linux -e gcc4.*qt3.i386.deb | tr -d '\n' | sed -e "s/.*\(.unix.*linux\).*a href=\"\(opera.*deb\)\".*/--url http:\/\/snapshot.opera.com\1\/\2/" | curl -# -K - -O
That looks a fair bit more complex, purely because curl (for all its power) does not have native recursion. On the plus side it would be slightly faster, as it is not recursively checking all subdirectories to locate the correct package.
One advantage of these examples being done on single line is that they don't even need to be saved as scripts, a shell alias is enough, i.e. you could alias 'osnap' (Opera snapshot) for the entire command.
A even neater trick with the tar packages, is extracting their contents on the fly. After download you can simply 'cd' into the created directory and run Opera in place via './opera &', which would mean that the main opera installation (both the install and preferences) wouldn't be touched. Great for trying the snapshots, with the least risk of messing up your setup trying an experimental build!
Here is an example, using wget's recursive capabilities for simplicity once again:
wget -nv http://my.opera.com/desktopteam/blog/ -O - | grep snapshot.opera.com\/unix | head -1 | sed -e "s/.*\(http.*\)\" target.*/\1/" | wget -i - --spider -r -erobots=off -nd -np 2>&1 | grep \/opera.*.gcc4.*qt3.i386.tar.bz2 | sed -e "s/.*http/http/" | wget -i - -O - | tar xjf -
Note: In this example I use the '--spider' option. Although this option has been around a while, a bug in older versions mean that my intended usage would fail. However it does work with the version found in most modern distros (1.11.4) and the very latest version (1.12).
If the above fails for you because you have an older version of wget, here it is without making use of '--spider' or recursion. I'll use the same technique as in my curl example but this time using wget once again:
wget -nv http://my.opera.com/desktopteam/blog/ -O - | grep snapshot.opera.com\/unix | head -1 | sed -e "s/.*\(http.*\)\" target.*/\1intel-linux\//" | wget -nv -i - -O - | grep -e title.*intel-linux -e gcc4.*qt3.i386.tar.bz2 | tr -d '\n' | sed -e "s/.*\(.unix.*linux\).*a href=\"\(opera.*bz2\)\".*/http:\/\/snapshot.opera.com\1\/\2/" | wget -i - -O - | tar xjf -
Of course writing an actual script rather than a one liner would allow for error checking, only downloading if a newer version is actually available, and so on. And before anyone asks, no I won't be creating such a script and sharing it!
Finally, let me remind you all to read the associated snapshot blog post before installing any build, so that we can highlight known issues and major problems. If you don't and something goes wrong, you can only blame yourself!