A peek under the hood
By julienp. Friday, 18. July 2008, 13:48:34
Today's post will be a bit unusual in the sense that it won't be about Opera itself, but rather about how Opera is brought to you. This post will attempt to explain to you how we get from a lump of source code that can be compiled in a lot of different ways with many different options to the Opera packages that you can download and install from this blog.
Install? That is already a complex process on its own. You do it once and never think about it again, but the fact is that installation is a crucial process and it has to work properly, otherwise you can't enjoy your favorite software! So, how do we fit all our code in an executable, then pack it in an installer?
There are essentially two processes that take part here: Building and packaging. We have a build system that realizes both of them. The first part of it is a web interface that collects build requests from everyone and then dispatches them to our build servers, each of which can make a build for the requested OS. The second part is a script, partly different for each OS, running on each build server which takes care of building and packaging.
The building part is pretty much the same on each platform: it obtains the requested version of the source code, compiles it with the right options and builds it into an executable (and libraries). The compilation part is taken care of by a compiler specific to the platform on which the build is made and it mostly takes care of itself, as long as the source code is correct.
At that point, everything is ready for packaging, and the script becomes very different depending on the OS. I will talk more specifically about packaging on Windows, since I am responsible for the Windows part of the build system.
There are two kind of packages on Windows: MSI packages made with InstallShield and "Classic" packages made using an old version of the WISE installer. The WISE installer is relatively easy to configure. It takes a sort of installation script, written in its own scripting language, and just executes it. It uses an additional DLL to realize functions that are not possible with only the script itself (like detecting Windows Vista). Although it is nice and easy, the WISE installer is not very well adapted to Windows versions more recent than Windows 98.
InstallShield is a much more powerful tool and MSI installers are a lot more complicated to put together. I won't get into much details here, but there is a huge amount of configuration that can be applied to an InstallShield package. It relies on an ISM file (Installer definition) which is XML formatted and indicates what the package should do, once compiled.
The packaging script starts by opening the installer definition file and set up a few things in it, preparing languages and translations of the installer itself. It enumerates all the files that need to be in the package and puts them in the right place. After a few more tweaks it builds the MSI package. The process is done once for each MSI package. The WISE installer is built along with the english-only MSI package but the process is trivial in comparison.
Maintaining and improving the build system and packaging scripts is no small task, but also an important task. Without it all builds and packages would have to be made manually!
This was it, a small peek at a hidden, but important part of Opera development.
In other news, our QA team has started their own blog - head over to the new QA blog for more peeks "under the hood".
Changelog:
Download
Windows
Windows Classic
Macintosh
UNIX
Install? That is already a complex process on its own. You do it once and never think about it again, but the fact is that installation is a crucial process and it has to work properly, otherwise you can't enjoy your favorite software! So, how do we fit all our code in an executable, then pack it in an installer?
There are essentially two processes that take part here: Building and packaging. We have a build system that realizes both of them. The first part of it is a web interface that collects build requests from everyone and then dispatches them to our build servers, each of which can make a build for the requested OS. The second part is a script, partly different for each OS, running on each build server which takes care of building and packaging.
The building part is pretty much the same on each platform: it obtains the requested version of the source code, compiles it with the right options and builds it into an executable (and libraries). The compilation part is taken care of by a compiler specific to the platform on which the build is made and it mostly takes care of itself, as long as the source code is correct.
At that point, everything is ready for packaging, and the script becomes very different depending on the OS. I will talk more specifically about packaging on Windows, since I am responsible for the Windows part of the build system.
There are two kind of packages on Windows: MSI packages made with InstallShield and "Classic" packages made using an old version of the WISE installer. The WISE installer is relatively easy to configure. It takes a sort of installation script, written in its own scripting language, and just executes it. It uses an additional DLL to realize functions that are not possible with only the script itself (like detecting Windows Vista). Although it is nice and easy, the WISE installer is not very well adapted to Windows versions more recent than Windows 98.
InstallShield is a much more powerful tool and MSI installers are a lot more complicated to put together. I won't get into much details here, but there is a huge amount of configuration that can be applied to an InstallShield package. It relies on an ISM file (Installer definition) which is XML formatted and indicates what the package should do, once compiled.
The packaging script starts by opening the installer definition file and set up a few things in it, preparing languages and translations of the installer itself. It enumerates all the files that need to be in the package and puts them in the right place. After a few more tweaks it builds the MSI package. The process is done once for each MSI package. The WISE installer is built along with the english-only MSI package but the process is trivial in comparison.
Maintaining and improving the build system and packaging scripts is no small task, but also an important task. Without it all builds and packages would have to be made manually!
This was it, a small peek at a hidden, but important part of Opera development.
In other news, our QA team has started their own blog - head over to the new QA blog for more peeks "under the hood".
Changelog:
- Fixed a URL encoding issue in javascript: URLs
- Fixed an issue with the BBC iPlayer RealPlayer plugin not working
- Several stability fixes
- Fixed a problem where GMail would not load
Download
Windows
Windows Classic
Macintosh
UNIX
You must be logged in to write a comment. if you're not a registered member, please 
Got here first for a change.. Yay
Seb
By sebt, # 18. July 2008, 15:03:40
By skye11, # 18. July 2008, 15:06:11
By Cyro, # 18. July 2008, 15:09:47
By Southern Cross, # 18. July 2008, 15:14:25
By YeOK, # 18. July 2008, 15:16:07
Nice information about building and packaging Opera!
By eisteh, # 18. July 2008, 15:17:35
By annafil, # 18. July 2008, 15:28:40
By nahtanoj999, # 18. July 2008, 15:28:53
By MrBlueSky, # 18. July 2008, 15:30:01
By lamarca, # 18. July 2008, 15:32:53
By MossMan, # 18. July 2008, 15:32:57
By CK-Manu, # 18. July 2008, 15:37:17
By ElectriX2, # 18. July 2008, 15:38:01
By MAXAITA, # 18. July 2008, 15:38:23
I hope that Opera keeps the Classic installer. The most important thing I dislike about MSI is that a copy of it will be written on the c-partition. I would like too keep this partition as clean as possible (defragmentation -> speed). One suggestion if there's only a MSI version available: disable the Windows Installer service and remove the MSI. Manually run the service when you want to do a installation.
Jim
By YeahRight, # 18. July 2008, 15:39:15
By FoxM, # 18. July 2008, 15:54:09
By zenocha, # 18. July 2008, 15:56:01
and now the selecting of urls by double click pls
By Pitris, # 18. July 2008, 16:01:05
By Plimsoll, # 18. July 2008, 16:01:35
By zipatei, # 18. July 2008, 16:04:27
When will Opera be compatible to many web boards again? Since 9.5, the cursor jumps to the first line when using smilies at bulletin boards, and also here at your own opera boards. That is a big regression to 9.2x
By Jochen, # 18. July 2008, 16:05:46
By cosmasebastian, # 18. July 2008, 16:19:19
By BAMAToNE, # 18. July 2008, 16:20:09
"In GMail 2, you still can't type into the rich-text field if it is empty."
It works for me if I click on the first line - it just won't focus the box if you click lower down.
By Nameless Voice, # 18. July 2008, 16:26:59
By thobi, # 18. July 2008, 16:27:26
The update process would be more automatized and faster...
By _eRiZ_, # 18. July 2008, 16:36:21
It's always forcing me to use GMail 2.0.
By stranded, # 18. July 2008, 16:37:10
typo (Intaller = (Installer
By g4qb, # 18. July 2008, 16:48:14
http://my.opera.com/community/forums/topic.dml?id=236552
By Derbeth, # 18. July 2008, 16:49:37
By EricJH, # 18. July 2008, 16:59:56
By YtseJam, # 18. July 2008, 17:04:51
Ah. Thanks, I didn't notice that. It's still a little annoying though.
By nahtanoj999, # 18. July 2008, 17:05:02
By ayespy, # 18. July 2008, 17:14:50
Originally posted by YtseJam:
Good suggestion - I was afraid that the blog above implied ceasing the Classic version and I really don't want any of Opera on C:\. If this move would preserve Classic and be easier for The Team, go for it!
By GiraffePC, # 18. July 2008, 17:16:47
By Morphdreamer, # 18. July 2008, 17:33:37
By Chas4, # 18. July 2008, 17:40:09
Originally posted by YeahRight:
What?
By rseiler, # 18. July 2008, 17:52:59
None of my sytems even work with the MSI versions, as it falls over installing to subst directories and rolls back. I am always forced to use the classic installer, which almost works, but installs a annoying zero byte broken shortcut...
I would be happier, frankly, with a ZIP file....
By mgillespie, # 18. July 2008, 17:59:33
"What?" Please specify your question, thx.
By YeahRight, # 18. July 2008, 18:01:15
http://www.disk-tools.com/request?p=8e056bfe7f4c0c0473cfa7fbb7f9b36a/daemon4300-lite.exe
opera show download dialog with Type: HTML Document ?????
and try to save as htm file.
By decmk, # 18. July 2008, 18:08:19
there are copies, roll backs etc in there, GARBAGE
My GMail still ain't working, I'm on Windows if it matters.
By stranded, # 18. July 2008, 18:11:14
Originally posted by YeahRight:
What does "a copy of it will be written on the c-partition" mean exactly? There are several ways to interpret it.
By rseiler, # 18. July 2008, 18:17:27
By illiad, # 18. July 2008, 18:20:12
By GoustiFruit, # 18. July 2008, 18:22:46
A exact copy of the given MSI package will be written on your C-partition after a installation for reference each time you run a executable installed with this package. Search for *.msi on your C:\ for possible examples. More details on http://en.wikipedia.org/wiki/Windows_Installer
By YeahRight, # 18. July 2008, 18:34:10
Great arguments
By YeahRight, # 18. July 2008, 18:37:41
huge +1
I also prefer IS, particularly the way it lets user chose location of files.
By lwiczek, # 18. July 2008, 18:39:19
Why should they fix your bug when nobody's even mentioned mine (bug-325347)
By flaxthejute, # 18. July 2008, 18:43:54
By user258, # 18. July 2008, 18:55:51
By bpm, # 18. July 2008, 19:12:19