oSync - Opera profile synchronization / bookmark publisher

oSync - Opera profile synchronization / bookmark publisher

Hi, I would like you to test my new synchronization script for Opera, oSync.
oSync is written in Python and uses the personal file space on my.opera.com to store files. It can be used to sync profile files between two computers, or always keep a HTML version of your bookmarks online, or both!

Screenshots/flyer: Link

Latest version:

http://sourceforge.net/projects/osync


Old versions:

oSync 0.5.0 (b141, Windows binaries)
oSync-0.5.0-b141_BIN.zip
oSync 0.5.0 (b141, Source dist)
oSync-0.5.0-b141_SRC.zip

oSync 0.5.0 (b139, Windows binaries)
Does not require Python. Recommended if you're running Windows and not interested in the source code.
oSync-0.5.0-b139_BIN.zip

oSync 0.5.0 (b139, Source dist)
oSync-0.5.0-b139_SRC.zip
Do require Python. Recommended if you're not running Windows or interested in the source code. This version, on Windows, opens a "DOS window" when running sync_windowed (unlike the Windows binaries).

Getting started guide (included above)
getting_started.pdf


Features:

• Synchronize any profile files through my.opera.com.
• Confirmed working on Windows and Linux.
• Compression, request caching, checksum and encryption support.
• File filters prevents synchronization of trivial changes.
• Very efficient network usage.
• Backups replaced local files.

Integrated as well as standalone tools:

• Bookmark file cleaner.
• Bookmark to html converter

New releases RSS-feed

This first post will be updated continuously (the posts right below this one are old ones).

Nice work, it worked fine from my machine. I'll try it on my laptop tomorrow and see if I can sync them up.

A couple of suggestions,

1) would it be possible to allow other files to be synced up like wand.dat, notes etc?

2) is the script able to be run from inside a folder or do all the files have to be in the profile directory, I like to keep my directory as clean as I can so I like to put all my additional files like ua.ini editor in a seperate folder.

Originally posted by alex_p:

Nice work, it worked fine from my machine. I'll try it on my laptop tomorrow and see if I can sync them up.

A couple of suggestions,

1) would it be possible to allow other files to be synced up like wand.dat, notes etc?

2) is the script able to be run from inside a folder or do all the files have to be in the profile directory, I like to keep my directory as clean as I can so I like to put all my additional files like ua.ini editor in a seperate folder.

I'm glad to hear it worked

1) I'll probably include that in the next release. An ini-file will control which files that should be synced.

2) Same goes here. I totally agree.

Note ... for non-windows users, a link to the actual script would be nice. Linux generally comes with python so the files could be run directly with some minor editting - though I'm sure that editing would include changing the file paths. (The path to your profile in Linux is "~/.opera/", for example.)

Works fine, I have just synced my laptop from my desktop. There were no problems and it worked fine first time.

Nice work.

minisu: this sounds great! my question is exactly how you compare files, is it only by file time stamp, or do you compare file contents? So if I have computer A and computer B, and I add bookmarks to both then try to synch, what happens?

Originally posted by sgunhouse:

Note ... for non-windows users, a link to the actual script would be nice. Linux generally comes with python so the files could be run directly with some minor editting - though I'm sure that editing would include changing the file paths. (The path to your profile in Linux is "~/.opera/", for example.)

All links should be relative, but I have to test some os-dependent functions in Linux first.

Originally posted by non-troppo:

minisu: this sounds great! my question is exactly how you compare files, is it only by file time stamp, or do you compare file contents? So if I have computer A and computer B, and I add bookmarks to both then try to synch, what happens?

As of now, only by file time stamp. That means the newest file will replace the older. Worth looking into .


New version - oSync 0.02a:

WHAT'S NEW:
 Support for multiple files.
 Backups are now made for replaced files (/backup).
 Now operates from its own directory.

Download here: See first post for latest release

Nice update, a couple more questions

Do you have to have this in a folder in the profile dircetory or can it be in a subfolder?

ie instead of profile>sync could I have profile>extras>sync ?

After the initial setup on one machine is it possible to just copy the files over to another machine and it will work or do I have to run the setup again?

Originally posted by alex_p:

Do you have to have this in a folder in the profile dircetory or can it be in a subfolder?

ie instead of profile>sync could I have profile>extras>sync ?

After the initial setup on one machine is it possible to just copy the files over to another machine and it will work or do I have to run the setup again?

1) With 0.03a you can place it wherever you want, as long as it's in or somewhere below your profile directory.

2) Will probably work... Tell me if it's not.

New version - oSync 0.03a:

WHAT'S NEW:
  Compression of server files -> Faster sync.
  Operates from profile dir or any of its subdirs.

Download here: See first post for latest release

non-troppo & others: Have started planning a comparing/diff system...

I can't seem to get this version to work at all.

The upload from my desktop machine seems to work fine, I can see the file in the file archive but if I copy over the folder and try it doesn't download to my laptop. Also if I unzip a new copy into the laptop's folder and run setup it still doesn't download.

Originally posted by alex_p:

I can't seem to get this version to work at all.

The upload from my desktop machine seems to work fine, I can see the file in the file archive but if I copy over the folder and try it doesn't download to my laptop. Also if I unzip a new copy into the laptop's folder and run setup it still doesn't download.

Strange, works here... No error messages?

Originally posted by minisu:

non-troppo & others: Have started planning a comparing/diff system...

Very good! I assume Python has good file comparison libraries available, you'll just need to modify it to work on a per bookmark rather than per line system.

By the way, diplo started work on a bookmark synchronizer, but it is coded in .NET I believe.

I would suggest parsing the bookmarks in XBEL XML first (diplo does this), then probably using some XML compare function would be easier. It was actually the Python XML group that created XBEL I believe:

http://pyxml.sourceforge.net/topics/xbel/
http://gazonk.org/~eloj/projects/operatoxbel.html - Opera to XBEL converter in python
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/135131 - A Python recipe for merging XBEL files!

It seems for python most of the hard work is already done...

Originally posted by non-troppo:

By the way, diplo started work on a bookmark synchronizer, but it is coded in .NET I believe.

I would suggest parsing the bookmarks in XBEL XML first (diplo does this), then probably using some XML compare function would be easier.

Great news! Thanks for the info too.

ALL: Current version does NOT work. I know why and I have fixed the problem. Stay tuned for next release that at the moment includes encryption, OS/Browser integration, hash sums...

I've posted some thoughts on file syncing previously...

All bookmarks have a Created and Last visited time (in UNIX time format, seconds since 1/1/1970), that would make things simpler. The one major problem is determining if a bookmark was added to file a or deleted from file b (for a bookmark which is only present in file a), to determine this you'd need to know the time of the last sync operation. If the bookmark is older than the last sync, then it must have been deleted from file b; if it is newer than the last sync then it was added to file a.

I don't think "Created" is modified if you edit a bookmark, in the event one has two different versions of the same bookmark I suppose you use the one with the most recent "Last visited" time.

New version - oSync 0.1.0 a:
I rewrote almost the whole program.
oSync now operates via an alternative Opera desktop icon and the user don't have to interact at all (after setup). Enjoy!

WHAT'S NEW:

 -Completely rewritten frame code
     * More readable source code
     * More modular
     * One single executable
 -Encryption
     * More secure
     * No need to specify server file name
 -Checksums for fast comparing
     * More economic network use

 -File filtering before hashing
     * Ignores trivial changes in files
     * More economic network use

 -Optimized module imports
     * Reduced memory footprint
     * Less non-standard modules (see info.txt for details)

 -Fixed bug reported by alex_p
 -Added desktop shortcut linked to script
 -Added info.txt

Download here: osync.exe

0.1.0a doesn't actually work at all for me. The previous version did work in uploading a file, I didn't try the other end yet.

- McAfee stops the opera.vbs script, because it is suspicious. After telling McAfee to allow it, I get an error message saying that the right file couldn't be found by the system. Code 8007000. Line 3, character 1.

- The setup program appears to work alright. I even thought it had already put the file on my.opera.com, but that was not the case.

- I don't know much about VBscripting , but this line from
oShell.run "../..\opera.exe",1,1
... seems weird, as the Profile and osync are located in the Appdata map, far and away from 'Program Files'.

Originally posted by Rijk:

0.1.0a doesn't actually work at all for me. The previous version did work in uploading a file, I didn't try the other end yet.

- McAfee stops the opera.vbs script, because it is suspicious. After telling McAfee to allow it, I get an error message saying that the right file couldn't be found by the system. Code 8007000. Line 3, character 1.

- The setup program appears to work alright. I even thought it had already put the file on my.opera.com, but that was not the case.

- I don't know much about VBscripting , but this line from
oShell.run "../..\opera.exe",1,1
... seems weird, as the Profile and osync are located in the Appdata map, far and away from 'Program Files'.

Entirely forgot Opera for multiple users that stores profile in AppData! oSync assumes that profile dir is directly under Opera dir. This also makes your desktop icon a undefined-file-type one, am I right?

Try changing

oShell.run "../..\opera.exe",1,1

into

oShell.run "[absolute path to opera.exe]",1,1

I Will definitely fix this very soon.

Fixed it! Help me test it...

osync_b44.exe

Sorry, still doesn't work. The installer suggested a wrong path, there should be no \program\ before the opera.exe. I entered the right path, but the script still didn't work. If that Opera.exe is not running, the script fails immediately. If Opera is running, the script fails after a short while, after some local files (temp.zip) are created.

Sorry for the inconvenience. Are you running build 43 or 44 (see readme.txt)?

If you are running b43, please try b44 instead or create a directory called "backup" in the osync dir (build 43 was only up during a short period and I didn't bother post a new post about b44).

If you are running b44, or if b43 still doesn't work; please run oSync.exe from the command line interpreter and post the error messege here, or... download build 46 from below (inculdes basic error handling) and post data in error.log here.

[Deleted, new version will be out around 14:00 CET today]

Originally posted by Rijk:

The installer suggested a wrong path, there should be no \program\ before the opera.exe. I entered the right path, but the script still didn't work.

I don't quite get the circumstances...
1. Where do you got your Opera.exe (absolute path)?
2. Where do you got your Opera Plug-in path (absolute path)?

Before I know that I cannot be sure to fix your problem, but I hope that the newest version will fix it: osync_b49.exe

I'm almost 100% certain that I fixed (at least) the first of your problems in the latest build:

osync_b50.exe

Looks like a great application. . . now if I could only get setup to work.

this error keeps getting thrown as osync prepares for server access.


"Exception called in module cookielib after 35151 ms."




please help,

Originally posted by jbuc:

Looks like a great application. . . now if I could only get setup to work.

this error keeps getting thrown as osync prepares for server access.

"Exception called in module cookielib after 35151 ms."

The same thing happened to me a few times yesterday but have never happened before that or since, so I think it is just a time-out error since I had problems to reach my.opera.com now and then that day. So try again today and let me know what happens.


Some screens of current progress on next release:
s1.png (no more console windows, will look much prettier at release)
s2.png (easy to use installation and unistallation, osync can be place whereever you want and automatically detects opera and profile directory)

Let me know what you think.

I've tried it several times since then with the same issue. How long before it times out? I'm getting the error a couple of seconds after it tries to connect.


btw 0.2.0 looks amazing

One of the issues I'm having w/ b50 is that osync always assumes the version on the remote server is the latest one, and when I try to sync, I get only an option to get from the server to local.

Is there some kind of date-stamp check being made to see if the version that's local may be newer than the remote, and if so, it should upload automagically (or give you the option to do so).

Originally posted by aagha:

One of the issues I'm having w/ b50 is that osync always assumes the version on the remote server is the latest one, and when I try to sync, I get only an option to get from the server to local.

Is there some kind of date-stamp check being made to see if the version that's local may be newer than the remote, and if so, it should upload automagically (or give you the option to do so).

Thank you all for helping me with the testing

oSync does no longer use timestamps (as it used to before 0.1.0). I changed that since the date often doesn't say anything about which files you want (you don't always want the newest). Since 0.1.0 osync uses md5 checksums that is stored locally and in a server meta file and operates like this (simplified):

If checksum of server files == checksum of local files: Local data and server data are already identical - Quit

If checksum of server files == checksum when local client last uploaded: No other client have uploaded new files since last time local client uploaded - Upload new files and quit

If checksum when clients last were synced (found in server meta file) == checksum when local client last uploaded: Clients are in sync - safe to download (replacing local files) and quit

Not any of the above: Clients are offsync (two profiles have grown independently from each other) - ask user what to do (download & replace files or abort)


It seems to me that you have used your profiles independently without syncing in between. To just replace local files or ask the user what to do makes little sense to me since you'll always lose data this way. To prevent users to forget syncing I'm working on integration improvements for the next release. Other planned/implemented news (except already mentioned in previous posts) are "no need to run setup on more than one client" and "caching of even more network data".

And here's some screen of the current build: s3.pngs4.png
Judging from this thread, people have problems with oSync which I don't got (not very suprising). This is bound to happen since I only got two pretty identical computers to test on, which both are used to develop on (which is bad from a testing POV). Therefore I would like to invite a few beta testers to help me make sure that the next release is bug free. PM me if you are interested.

New test release, check the top post

I need testers running Windows Me or older!

Running Windows XP SP2

I can't enter my password, the only thing I can do in that field is press enter to skip the step. Also, I was unable to locate where the .cfg file was so that I could manually insert my password.

Originally posted by Vasto:

Running Windows XP SP2

I can't enter my password, the only thing I can do in that field is press enter to skip the step. Also, I was unable to locate where the .cfg file was so that I could manually insert my password.

The password is entered, but nothing is seen (so that noone can see what's typed). Try enter your password and ignore that you can't see it.

@minisu:

I still see a profile folder detection problem in build 75...

If Opera.exe lives in "C:\Program files\Opera 8\", the profile folder for multi-user is:
"C:\Documents and settings\<username>\Application Data\Opera\Opera 8\profile\"
You missed the 'Opera' in this path

But I could edit 'installer.dat' and run setup.exe again, and then it worked.

Originally posted by minisu:

Originally posted by Vasto:

Running Windows XP SP2

I can't enter my password, the only thing I can do in that field is press enter to skip the step. Also, I was unable to locate where the .cfg file was so that I could manually insert my password.

The password is entered, but nothing is seen (so that noone can see what's typed). Try enter your password and ignore that you can't see it.

Why do this the Unix way? You could just display "*" instead of any entered letter.
It is far more user-friendly and is the common practice for entering passwords.

* note, I haven't tried the program, I'm guessing here that no "*" are used here.

Exception called in module setup after 14719 ms.
(see info.txt for possible solutions)

Traceback (most recent call last):
File "setup.py", line 90, in ?
IOError: [Errno 2] No such file or directory: 'C:/Documents and Settings/Michael/Application Data/Opera/profile/notes.adr'

@Vasto: that's what I got as well at first, because the program used the wrong file paths. It should be "Application Data/Opera/Opera/profile/"

Workaround:
Edit installer.dat so the paths are correct, delete settings.ini and all younger files, and run 'setup.exe'. At least, that worked for me.

Originally posted by Rijk:

@minisu:

I still see a profile folder detection problem in build 75...

If Opera.exe lives in "C:\Program files\Opera 8\", the profile folder for multi-user is:
"C:\Documents and settings\<username>\Application Data\Opera\Opera 8\profile\"
You missed the 'Opera' in this path

But I could edit 'installer.dat' and run setup.exe again, and then it worked.

Thank you for pointing this out. I will try to ensure that this was the last you saw of this bug. Will be fixed within 24 hours.

Originally posted by YtseJam:

Originally posted by minisu:

The password is entered, but nothing is seen (so that noone can see what's typed). Try enter your password and ignore that you can't see it.

Why do this the Unix way? You could just display "*" instead of any entered letter.
It is far more user-friendly and is the common practice for entering passwords.

I used no terminal echo since it was the easiest way to do it. I guess I could change that, you'll see tomorrow.

As promised: oSync-0.3.0-b77.exe

Fixed Multiuser path and password echo

It works now.

Sorry. I tried to install oSync(0.3.0 b77), but at the "Create Shortcut:...Manual sync..." it always freezes. What could be wrong by me? (Win'98 at home)

I hate to be non-constructive, but I fail to see the point in a synchronization program that doesn't synchronize. That is, oSync doesn't merge changes and only uses the latest version of data. I could just as well upload the files to a server and then copy them to another computer.

I understand that with current Opera's bookmark file format making real synchronization would be painful, but is it not at all possible?

Or maybe I'm missing something important here?

Originally posted by vinczej:

Sorry. I tried to install oSync(0.3.0 b77), but at the "Create Shortcut:...Manual sync..." it always freezes. What could be wrong by me? (Win'98 at home)

To find out what's wrong I need to know the following:
*Registry key: HKCR\HTTP\shell\open\command
*Are you using Opera as your standard application for http?

(this may be an OS related)

Originally posted by van_grieg:

I hate to be non-constructive, but I fail to see the point in a synchronization program that doesn't synchronize. That is, oSync doesn't merge changes and only uses the latest version of data. I could just as well upload the files to a server and then copy them to another computer.

I understand that with current Opera's bookmark file format making real synchronization would be painful, but is it not at all possible?

Or maybe I'm missing something important here?

I find your question legitimate . Actually my first approach on oSync was merging. The full answer is rather lengthy though:

How oSync works
oSync don't just use the latest version of data. It uses checksums to keep track of different versions of data which have a few benefits:

1. You save time and bandwidth since oSync will only download data that isn't equal or very similar (uses invidual file filters) to local data.

2. oSync will never replace any data with other data that does not derive from the synced data, without asking the user (refered to as "offsync").

All this means that as long as you always sync before starting Opera and after closing Opera, you'll always have the most recent files.

Merging is only needed if you for some reason don't sync before/after using Opera:

Case1: You forget to sync - Can't happen if you use autosync.
Case2: You have no internet access - Why would you then want to use Opera?
Case3: You simply don't want to access a server that often - Then you'll not have the most recent versions to work with, which is the point with syncing after all.

And if despite this manages to skip syncing while updating your bookmarks on more than one client, oSync will detect and warn you so that no files will be replaced behind your back.

How merging works
The problem with merging is:
1. Determine if a bookmark/note/folder was added on one client or deleted on the other.
2. Accurately keep track of moved bookmarks/notes/folders.
3. The Wand data, and other files, can't be merged (it's encrypted). If you want to sync such files, you would have to sync as often as oSync does - and then there would be no reason not to sync bookmarks/notes just as often.

First thought might be to go around 1) and 2) by adding mirror files to your adr-files in which you would add ID:s to all items. This wouldn't work though since the user can change all atrributes that you base that ID on.

I don't have anything against merging as idea, so anyone are welcome to contribute such module to oSync.


Edit: Other reasons to use oSync instead of "upload the files to a server and then copy them to another computer" are the compression and encryption.

Originally posted by minisu:

I don't have anything against merging as idea, so anyone are welcome to contribute such module to oSync.

Taking 3) into account, the only party that can add merging is Opera Software.

Originally posted by minisu:

You forget to sync - Can't happen if you use autosync.

Yes, it can. In my normal usage, Opera is never started or shut down. I have two laptops, both have Opera installed. I never turn those laptops off - only Hybernate one of them, the other one is always on for weeks (if not months). About the only exception from that rule is Opera crashing (which probably won't cause syncing, but I may be wrong).

Otherwise, I find your arguments very convincing. Would it be possible to trigger syncing when files are modified rather than starting/closing Opera or Windows?

Thanks for your efforts anyway!

Originally posted by van_grieg:

In my normal usage, Opera is never started or shut down. I have two laptops, both have Opera installed. I never turn those laptops off - only Hybernate one of them, the other one is always on for weeks (if not months). About the only exception from that rule is Opera crashing (which probably won't cause syncing, but I may be wrong).

Opera crashing should lead to sync (haven't tested that though).

Originally posted by van_grieg:

Otherwise, I find your arguments very convincing. Would it be possible to trigger syncing when files are modified rather than starting/closing Opera or Windows?

Unfortunately, the problem with never closing Opera is that Opera will not (and must not) let other applications write to the files. As long as there is no such command in Opera as "Unload/Load profile files", though that would be almost the same thing as restarting Opera.

Originally posted by minisu:

the problem with never closing Opera is that Opera will not (and must not) let other applications write to the files.

Which is perfectly fine. See, I don't need local files being updated automatically. The problem is keeping the online version always up-to-date.

I already keep my bookmark/wand files online, and I could write a script that would upload those files on Opera shutdown (although your tool is better of course). But to remember to shut it down is just as tricky as to remember to upload files.

Indeed, maybe a button in Opera that would update the online versions of the files? I don't think there is any prohibition on READING the files Opera has open, I mean, Acronis will image WINDOWS while it's running lol.

And then if you want to sync up, you have to restart Opera.

There's one other thing, you would want to have an option somewhere for either having your autosave.win to be concurrent accross Opera installs or not depending on what people wanted to do.

Originally posted by jp10558:

Indeed, maybe a button in Opera that would update the online versions of the files? I don't think there is any prohibition on READING the files Opera has open, I mean, Acronis will image WINDOWS while it's running lol.

And then if you want to sync up, you have to restart Opera.

Actually I already made buttons for that in an earlier unreleased version. I guess you'll see them in a future version. Nice idea!

Originally posted by jp10558:

There's one other thing, you would want to have an option somewhere for either having your autosave.win to be concurrent accross Opera installs or not depending on what people wanted to do.

I don't quite understand what you mean. You can set "Active session" to "n" ("no") in the setup.

Originally posted by jp10558:

Indeed, maybe a button in Opera that would update the online versions of the files?

Mmmm, no. This brings us back to syncing manually and thus misses the point. Uploading should be automatic or it will be forgotten sooner or later. Triggering upload automatically on file modification and manual download appears like the best compromise to me so far.

Minisu, what do you think?

hi

i downloaded this and when it installs, it only goes as far as creating a shortcut on the desktop for a manual sync before it does nothing for a very long time. is that because this is an alpha release?

Originally posted by vinczej:

Sorry. I tried to install oSync(0.3.0 b77), but at the "Create Shortcut:...Manual sync..." it always freezes. What could be wrong by me? (Win'98 at home)

Originally posted by ctk:

i downloaded this and when it installs, it only goes as far as creating a shortcut on the desktop for a manual sync before it does nothing for a very long time. is that because this is an alpha release?

It seems like you got the same problem. To have a chance to solve this I need to know a few things about your systems:

1. ctk, are you running Windows Me or older?
2. Are you using Opera as your standard application for http (standard web browser)?
2. What's in this registry key: HKCR\HTTP\shell\open\command (run regedit.exe)?

And yes, this problem is test-release related and won't exist in a stable release (which can only be released with your help, I got no non-XP computer to test on).

If you need help with some of this things, don't hesitate to contact me via MSN or PM.

1. on this machine xp pro (i am going to put this on my personal computer which is xp home)
2. i am using opera as my main browser, but the default is ie
3. "C:\Program Files\Internet Explorer\iexplore.exe" -nohome

Originally posted by ctk:

1. on this machine xp pro (i am going to put this on my personal computer which is xp home)
2. i am using opera as my main browser, but the default is ie
3. "C:\Program Files\Internet Explorer\iexplore.exe" -nohome

That's the problem. oSync uses your default browser path as assumed path for Opera in the test release. I'll try to figure out a better way to do that. If I got the time an update will be out later today (CET).

Edit: I haven't figured out a absolutely certain way to find all the paths to all Opera installations (yet). At the moment I have two ways to detect Opera path; either the system/user default browser or the default Opera installation (in the Windows shell). I've decided to stick with the former for now since it's easier for the user to change and more likely to be the correct path if the user got several Opera installations.

If you want to stick with IE as your default browser you can change to Opera just before the oSync installation and then switch back.

If somone knows a good way to detect Opera paths, please let me now!

Originally posted by van_grieg:

[...] Uploading should be automatic or it will be forgotten sooner or later. Triggering upload automatically on file modification and manual download appears like the best compromise to me so far.

Minisu, what do you think?

Agreed, but... automatically upload on file modification would require a background process, which is not the job for Python. Locking oSync in upload mode is very easy on the other hand, and in most cases it will manage that on its own.

Because of this, I won't add this to oSync in the near future. But maybe someone else will - currently I'm making the source code ready for public and multi OS support.