Skip navigation.

Ruari's thoughts

Uploading files to My Opera with cURL

, , ,

Since I have been playing around with cURL of late, I started thinking about how to use it to upload files to My Opera's file space. It can actually be done fairly easily, here some examples.

Note: The following examples will work on Linux, UNIX and Mac OS X. Whilst it is possible to achieve similar results on Windows, I have not covered this.

Before you can upload files you need an auto-login cookie. You could extract the cookie information from your Opera cookie store, but it is just as easy to get cURL to generate a new login cookie. Just change 'username' and 'password' to your own My Opera username and password in the following command and then run it:

$ curl -L -c ~/.myoperacookie -d "location=&user=username&passwd=password&remember=1" http://my.opera.com/community/login/index.pl > /dev/null

Next save the following as shell script. Before saving this you'll need to edit the 'username' parts to make it your own My Opera username:

#!/bin/bash
curl -b ~/.myoperacookie http://my.opera.com/username/files/addpic.pl -F file=@"$1" -F dir="$2" -o /dev/null
file=`echo $1 | sed -e "s/.*\///" | sed -e "s/ /%20/g"`
curl -s -I "http://files.myopera.com/username/$2/$file" -e http://my.opera.com/ | grep HTTP | sed -e "s/HTTP\/1.1 200 OK/Upload Successful/" | sed -e "s/HTTP.*/Upload Failed/"

Note: Make sure you make the script executable and save it somewhere in your execute path.

To use the script simply type the script's name (I called my script 'moput', meaning 'My Opera Put'), followed by the name and path to file you want to upload, and the name of the My Opera directory you want to upload to. For example, to upload a local file called 'funny.jpg' from your local '~/Pictures' directory to your My Opera Pictures directory, you could issue the following:

$ moput ~/Pictures/funny.jpg Pictures

Note: If the My Opera directory doesn't exist it will be created. If you want it to be a subdirectory this is possible to, e.g. 'files/Pictures'. If you only mention the local file and don't mention a My Opera directory, it will placed in 'files' but you will be told that the 'Upload Failed'.

Now the obvious questions are, "How is this more efficient than the regular uploader? What if I have lots of files to upload, do I have to upload them one at a time?"

Firstly, for me at least it is more efficient as I prefer work on the command line. It also allows me to upload files from remote machines that I connect to via ssh (hence no GUI). However, I appreciate that many people won't see these as benefits. In answer to the main question, yes you can queue up lots of files to upload, and no it doesn't even require having to make a more complex script. Instead you can just make use of the other tools on your system. Suppose for example I wanted to upload all the .jpg files in the current working directory to My Opera and save them in the remote Pictures/jpegs folder. For this you could use the UNIX 'find' command as follows:

$ find *.jpg -exec moput {} Pictures/jpegs \;

Here is another variant, which would copy all PDF files in the current working directory and one sub directory deep, and save them in in the remote 'Documents' folder:

$ find . -maxdepth 2 -type f -name "*.pdf" -exec moput {} Documents \;

Note: If you are not familiar with the 'find' command, read the 'man' page or consult one of the many online tutorials before generating more variant commands. 'find' is incredibly powerful, meaning plenty of things are possible, including stuff that you might not intend!

As many Mac users tend to shy away from the command line, here is a nice Graphical option, using an Automator workflow.

Before we can create the Automator workflow, we need a My Opera login cookie. This is exactly the same as my early instructions but I will repeat it here for convenience.

To make a login cookie, start the Terminal application and type the following, remembering to change the 'username' and 'pasword' to reflect your own My Opera login credentials:

curl -L -c ~/.myoperacookie -d "location=&user=username&passwd=password&remember=1" http://my.opera.com/community/login/index.pl > /dev/null

Now we need to create an Automator workflow as follows:

  1. Start Automator
  2. In "Select a starting point to open a new workflow", highlight "Files & Folders"
  3. Select "Get content from": "my Mac" & "Use files & folder selected in the Finder when workflow runs"
  4. Under "Actions > Utlilities" drag "Run Shell Script" to the workflow on the right, below the initial entry.
  5. Change the "Pass input" option to "as arguments"
  6. Replace the example code provided by Automator with the following, remembering to change the two 'username' references to reflect your own My Opera username:


    for item in "${@}"
    do
    curl -b ~/.myoperacookie http://my.opera.com/username/files/addpic.pl -F file=@"${item}" -F dir=files -o /dev/null
    file=`echo ${item} | sed -e "s/.*\///" | sed -e "s/ /%20/g"`
    curl -s -I "http://files.myopera.com/username/files/$file" -e http://my.opera.com/ | grep HTTP | sed -e "s/HTTP\/1.1 200 OK/0/" | sed -e "s/HTTP.*/1/"
    done | sort -r | head -1 | sed -e "s/0/File(s) uploaded successfully./" | sed -e "s/1/Check My Opera files: Items may be missing./"
    

  7. Under "Actions > Utlilities" drag "Run AppleScript" to the workflow on the right, below the previous entry.
  8. Replace the example code provided by Automator with the following:

    on run {input}
    
    tell application "Finder"
    display dialog input buttons {"OK"}
    end tell
    
    return input
    end run
    

  9. Select "File > Save as Plug-in" and name it appropriately (e.g. 'Send Files to My Opera'), as a Plug-in for "Finder".
  10. Quit Automator


To use: Simply highlight the files you wish to upload in a Finder window, right click with your mouse (or Ctrl+click) and choose "More > Automator > Send Files to My Opera". Automator will then start in the background (you can see the status at the top of the screen). When it is done you will get a message telling you if the files uploaded or not. Done! :wink:

Of course if all this seems like too much work, you could just use Unite to share files instead! :D

cURL is the new Wget? I don't think so!What happened to the Widgets?

Comments

Espen André Øverdahl 13. October 2009, 06:48

Brilliant. Just tried it and it worked! Sweet.

Ruari Ødegaard 13. October 2009, 06:59

Cool glad you liked it. It should be possible to do something similar to upload Photo Albums as well but I haven't really looked into that yet.

it-s 13. October 2009, 19:04

Awesome! Thank you so much. This is even better then FTP :D

Espen André Øverdahl 13. October 2009, 19:06

I've been using this all day and I'm more and more impressed by how easy it is now to upload files. Thanks again!

Ruari Ødegaard 13. October 2009, 19:33

Hey, don't thank me! Thank Daniel Stenberg (author of cURL)!! :D

You can actually do stuff link this on pretty much any site that has a file upload option. It sometimes takes a little work to decipher how to construct the POST request, and you often have to include a step or two for authentication, but this is usually more than paid for in time saved later if you use a site a lot. If you so desired you could also write simple scripts to list files or delete them as well.

The document "Using cURL to automate HTTP jobs" over at the main cURL website does a good job of explaining how this all works. Also reading the man page will probably also give you further ideas about stuff you could automate.

Oh, and whilst in this instance I used cURL, I suggest having a quick skim of Wget's manual as well. As I mentioned in my previous blog post, sometimes it is Wget that works best for this stuff. :wink:

Dustin Wilson 13. October 2009, 19:48

How clever. Never occurred to me in the slightest to use cURL to handle that, but now that I think of it it's the perfect solution.

Ruari Ødegaard 13. October 2009, 20:26

@EspenAO (and others who read this earlier): Since I first wrote this I have edited the blog post slightly. I was originally throwing away the returned HTML content. However it has since occurred to me that this might have some useful information related to failures so I switched the ending from:

> /dev/null

To:

| grep -i -e error -e proxy | sed -e "s/.*/\*\*\*Upload Problems\*\*\*/" | sort -u

This should pick up on any Proxy timeout problems and authentication errors (for example if the cookie expires).

Other keywords could obviously be added as '-e keyword' in the grep command for other failure pages.

There shouldn't be too many false positives as it is not actually the my.opera.com/username/files/ page that is being echoed back to curl on successful transfer but rather a simple '302' page (the option '-L' would have caused curl to follow on to the files page).

Charles Schloss 14. October 2009, 03:58

:cool:

I wonder what Tamil thinks of this?

Ruari Ødegaard 14. October 2009, 05:58

Originally posted by Chas4:

I wonder what Tamil thinks of this?


So, you think Tamil doesn't know about automating content posting on web sites? Haven't you always wondered how those first posts are so consistently achieved on the Desktop Team Blog? p:

Actually in all seriousness, I don't personally know (or care) if Tamil does this.

Either way Tamil is great asset to our community! :wink:

Aleksander Aas 14. October 2009, 06:29

I want a Windows version :wink:

Ruari Ødegaard 14. October 2009, 07:50

Well, Windows users could install Cygwin to give them access to a UNIX-like environment under Windows, which should allow all the original examples to run (with at most minimal tweaking). However, I grant you that that would feel like something of an over kill just to gain the ability to upload files from the command line.

Otherwise, you can still get a Windows version of cURL from links on the bottom of the cURL downoad page and create a batch file.

Having not actually used Windows for quite some time (my home and work machines are all Linux, UNIX or Mac) I can give some hints from my vague memories of Windows, but I can't guarantee any of it will work. I'm almost certain to forget something critical. :frown:

That said .. what the hell! Here goes nothing!! Once cURL is installed somewhere in your path you should be able to run the following commands to create your login cookie:

curl -L -c "%userprofile%\myoperacookie.txt" -d "location=&user=username&passwd=password&remember=1" http://my.opera.com/community/login/index.pl -o "%temp%\myopera.tmp"
del "%temp%\myopera.tmp"
attrib +h "%userprofile%\myoperacookie.txt"

Hopefully this will create a hidden login cookie file in your home directory called 'myoperacookie.txt'. Open it in Notepad to confirm it looks correct.

Assuming that worked, save the following in a text file called 'moput.bat':

curl -b "%userprofile%\myoperacookie.txt" http://my.opera.com/username/files/addpic.pl -F file=@%1 -F dir="%2" 

After placing 'moput.bat' somewhere in your execute path, you could (in theory at least) run commands like:

moput "%userprofile%\My Documents\funny.jpg" Pictures

Obviously changing "%userprofile%\My Documents\funny.jpg" to whatever file you wanted to upload.

Or if you were already in the correct directory, perhaps:

moput serious.txt Documents

With regards to replacement for UNIX 'find', or some other way to queue up lots of files, that is beyond my memory without a Windows machine to test with. Indeed I am not even 100% certain that any of the examples I gave above will work at all.

Perhaps some Windows command line Guru could come along and produce real examples. I can't imagine it would be too hard on Windows. Indeed I assume that 'PowerShell' would be a better option than 'cmd.exe', but as I have never used PowerShell, I will not even attempt that!

Tracio 14. October 2009, 10:55

Very good stuff. Thanks for posting it.

I've included your command in a function and place it in my .zshrc . I've changed it a little bit so it can batch upload files from a single command. Furthermore, by using zsh's extended file globbing, the use of "find" can be avoided in most cases.

function uploader () {
for item in "${@}"; do
curl -b ~/path_to_my_opera_cookie http://my.opera.com/my_username/files/addpic.pl -F file=@"${item}" -F dir="My_directory" | grep -i -e error -e proxy | sed -e "s/.*/\*\*\*Upload Problems\*\*\*/" | sort -u
done
}


Thus a

% uploader **/*.jpg(.)


will batch upload all .jpg files present in your current directory and all subdirectories.

Ruari Ødegaard 14. October 2009, 11:24

@Tracio: Very nice!

Ruari Ødegaard 15. October 2009, 07:57

Hmm ... I just thought of a better way to check that the upload is successful. After upload, I now use the cURL '-I' option to fetch the HTTP-header of the uploaded file. If I get back a "HTTP/1.1 200 OK" then the file must have uploaded correctly. If not, it failed. Simple eh. :wink:

I also need to tell cURL to send (my.opera.com) referer page information to the server so that the file header is returned and not the header for the download page.

I have updated the main blog post with this variant.

Espen André Øverdahl 15. October 2009, 08:02

How difficult (or challenging) would it be to have this in some sort of application? Can it be used in a Widget, or are we only limited to the terminal option?

Ruari Ødegaard 15. October 2009, 08:24

Rather than a Widget this would actually be easier with a Unite Application, as you can nominate an area of the file system that the application is allowed access to.

You wouldn't be able to reuse anything I have written here but that doesn't matter too much as the Unite Framework would allow you to do the same thing.

However a better option might be a Unite Application called something like "My Opera Files Sync", which would work as follows: You start the service and nominate a directory on your hard disk, then any files placed in that folder are automatically uploaded to your My Opera Files store.

Indeed it should be possible to create a similar Unite Application for My Opera Photos.

Espen André Øverdahl 15. October 2009, 08:41

Originally posted by ruario:

However a better option might be a Unite Application called something like "My Opera Files Sync"

This, and:

Originally posted by ruario:

it should be possible to create a similar Unite Application for My Opera Photos.

this would make me a happy man.

Great suggestions!

Ruari Ødegaard 15. October 2009, 10:04

I'm not going to write a Unite Application myself however as:

1. My JavaScript skills are not up to it.
2. The current solution (with cURL) does the job for me.

Hopefully someone else will be inspired though.

Ruari Ødegaard 15. October 2009, 10:06

@EspenAO Mac users could wrap some Apple Script around the basic cURL call to make something graphical without too much effort.

I might have a look at that if I find a moment.

Tracio 15. October 2009, 10:29

A very ugly hack would be to call the terminal from Opera (it can be done from a menu item or button) telling it to execute a one line for loop. If you hardcode a folder and change the command a bit to watch for new files, it will upload new files to my.opera. For example, as a menu item:

Item, "MyOpera-Uploader"="Execute program, "gnome-terminal --execute for item in $(find ~/path_to_dir_being_watched -Btime -1d); do curl -b ~/path_to_my_opera_cookie http://my.opera.com/my_username/files/addpic.pl -F file=@"${item}" -F dir="MyOperaDir" -o /dev/null ;done""


That will automatically upload all the files present in the directory which have been modified in the last day, assuming that Opera can pass a for loop to gnome-terminal... I can't test the menu item code stuff right now but the for loop works fine.

Needless to say that a cron job (or a launchd plist on OSX) would work too, that way you don't even need to execute the script, it would do everything automatically.

Ruari Ødegaard 15. October 2009, 13:10

@EspenAO: Ok, I wanted to say this publicly earlier but timing was hard as I wasn't sure exactly when we would make an announcement. Anyway, with the direction we are heading with OWD, i.e. stand alone 'applications' and the ability to use the File I/O API. Yes you should be able to create a Widget as well, not just UNITE. (Both for the file and Photo uploader ideas)

Ruari Ødegaard 15. October 2009, 18:36

@EspenAO It occurred to me that on a Mac the simplest way to wrapper the script and make it graphical is actually Automator. Despite having a Mac at home I have never bothered to use Automator before but I have just fired it up and tried it. True to form, it is very straight forward and hence is perfect for procedural stuff like this.

I now have added Automator instructions to the end of the original blog. Once setup, a user can simply highlight a selection of files in the Finder and right click (or control click) to send them to their My Opera file storage area.

Daniel Aleksandersen 16. October 2009, 06:14

You should upload the Automator workflow to Apple.com/Downloads.

Ruari Ødegaard 16. October 2009, 06:32

Originally posted by danaleks:

You should upload the Automator workflow to Apple.com/Downloads.



Nice idea but in the example I provided, the username is hardcoded, hence anyone wanting this has to re-create the workflow themselves using my instructions.

I'm sure you could get around this by asking for the username and password only the first time (and them storing these), but that requires something more complex and I feel that now I have done these examples to death. :wink:

My intention was more to show what could be done 'fairly' easily not to write an entire application, widget or unite service.

So if someone wants something more complex or with more options, I think they will need to write it themselves. awww

endless love 17. October 2009, 06:11

:up:

Ruari Ødegaard 20. October 2009, 20:14

By the way you can also use the same technique for batch uploads to the File Inbox Unite Application.

Here is a very simple cURL script example (with no error checking or confirmation of upload):

#!/bin/bash
unitesessionid=XXX
unitenonce=`curl -s -b "unite-session-id=$unitesessionid" http://devicename.username.operaunite.com/file_inbox/ | grep unite-nonce | sed -e "s/.*value..\(.*\)\".*/\1/"`
curl -L -b "unite-session-id=$unitesessionid" http://devicename.username.operaunite.com/file_inbox/ -F unite-action=upload -F unite-nonce=$unitenonce -F file1=@"$1" -o /dev/null

You would need to configure this as follows:

XXX should be replaced with the value from a 'unite-session-id' cookie, taken from a browser that is logged into the 'File Inbox' Application as a visitor.

devicename should be replaced with the device name for the unite service you are connecting to.

username should be replaced with the My Opera username of the owner of the unite service you are connecting to.

In Opera you could get the unite-session-id cookie value from the Opera cookie manager (in the Advanced Preferences). Though, as the cookie will expire reasonably frequently, you will either need to update this value manually from time to time or write another cURL script to auto-update it for you. Personally I linked my copy of the script to my w3m cookie file. Then I can just re-login with w3m to get the script working again, whenever the cookie expires. To do this, I just need to change the line that sets the 'unitesessionid' variable to:

unitesessionid=`grep unite-session-id ~/.w3m/cookie | sed -e "s/.*unite-session-id\t\([a-zA-Z0-9][a-zA-Z0-9]*\).*/\1/"`

In case you are not familiar with it, w3m is a handy terminal web browser that I use from time to time on machines with no GUI running. I figure I can recommend it in good confidence because its a great little browser, and it is not exactly direct competition! :wink:

How to use Quote function:

  1. Select some text
  2. Click on the Quote link

Write a comment

Comment
(BBcode and HTML is turned off for anonymous user comments.)

If you can't read the words, press the small reload icon.


Smilies

Download Opera, the fastest and most secure browser
December 2009
S M T W T F S
November 2009January 2010
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31