Ruarí Ødegaard

Uploading files to My Opera with cURL

Monday, October 12, 2009 9:23:18 AM

Automator, not-supported, myopera, cURL, GNU Parallel

Since I have been playing around with cURL of late, I started thinking about how to use it to upload files to My Opera's file space. It can actually be done fairly easily, here some examples.

Note: The following examples will work on Linux, UNIX and Mac OS X. Whilst it is possible to achieve similar results on Windows, I have not covered this.

Before you can upload files you need an auto-login cookie. You could extract the cookie information from your Opera cookie store, but it is just as easy to get cURL to generate a new login cookie. Just change 'username' and 'password' to your own My Opera username and password in the following command and then run it:

$ curl -Lc ~/.myoperacookie -d "user=username&passwd=password&remember=1" https://my.opera.com/community/login/index.pl > /dev/null

Next save the following as shell script. Before saving this you'll need to edit the 'username' part to make it your own My Opera username:

#!/bin/bash
if curl -sLb ~/.myoperacookie http://my.opera.com/username/files/addpic.pl -F file=@"${1}" -F dir="${2}" | grep -q "name.*${2}/${1/*\/}" 
then
	echo "${1/*\/} was successfully uploaded"
else
	echo "Uploading ${1/*\/} failed"
	exit 1
fi

Note: Make sure you make the script executable and save it somewhere in your execute path.

To use the script simply type the script's name (I called my script 'moput', meaning 'My Opera Put'), followed by the name and path to file you want to upload, and the name of the My Opera directory you want to upload to. For example, to upload a local file called 'funny.jpg' from your local '~/Pictures' directory to your My Opera Pictures directory, you could issue the following:

$ moput ~/Pictures/funny.jpg Pictures

Note: If the My Opera directory doesn't exist it will be created. If you want it to be a subdirectory this is possible to, e.g. 'files/Pictures'. If you only mention the local file and don't mention a My Opera directory, it will placed in 'files'.

Now the obvious questions are, "How is this more efficient than the regular uploader? What if I have lots of files to upload, do I have to upload them one at a time?"

Firstly, for me at least it is more efficient as I prefer work on the command line. It also allows me to upload files from remote machines that I connect to via ssh (hence no GUI). However, I appreciate that many people won't see these as benefits. In answer to the main question, yes you can queue up lots of files to upload, and no it doesn't even require having to make a more complex script. Instead you can just make use of the other tools on your system. Suppose for example I wanted to upload all the .jpg files in the current working directory to My Opera and save them in the remote Pictures/jpegs folder. For this you could use the UNIX 'xargs' command as follows:

$ ls *.jpg | xargs -I@ -d"\n" moput @ Pictures/jpegs

Here is another variant, which would copy all PDF files in the current working directory and one sub directory deep, and save them in in the remote 'Documents' folder. This time using 'find':

$ find . -maxdepth 2 -type f -name "*.pdf" -exec moput {} Documents \;

Note: If you are not familiar with the 'find' command, read the 'man' page or consult one of the many online tutorials before generating more variant commands. 'find' is incredibly powerful, meaning plenty of things are possible, including stuff that you might not intend!

As many Mac users tend to shy away from the command line, here is a nice Graphical option, using an Automator workflow.

Before we can create the Automator workflow, we need a My Opera login cookie. This is exactly the same as my early instructions but I will repeat it here for convenience.

To make a login cookie, start the Terminal application and type the following, remembering to change the 'username' and 'pasword' to reflect your own My Opera login credentials:

curl -Lc ~/.myoperacookie -d "user=username&passwd=password&remember=1" https://my.opera.com/community/login/index.pl > /dev/null

Now we need to create an Automator workflow as follows:

Start Automator
In "Select a starting point to open a new workflow", highlight "Files & Folders"
Select "Get content from": "my Mac" & "Use files & folder selected in the Finder when workflow runs"
Under "Actions > Utlilities" drag "Run Shell Script" to the workflow on the right, below the initial entry.
Change the "Pass input" option to "as arguments"

Replace the example code provided by Automator with the following, remembering to change the 'username' reference to reflect your own My Opera username:

for item in "${@}"
do
	curl -Lb ~/.myoperacookie http://my.opera.com/username/files/addpic.pl -F file=@"${item}" -F dir=files | grep -c "name.*files/${item/*\/}" 
done | sort -r | head -1 | sed -e "s/1/File(s) uploaded successfully./" | sed -e "s/0/Check My Opera files: Items may be missing./"

Under "Actions > Utilities" drag "Run AppleScript" to the workflow on the right, below the previous entry.

Replace the example code provided by Automator with the following:

on run {input}
	
	tell application "Finder"
		display dialog input buttons {"OK"}
	end tell
	
	return input
end run

Select "File > Save as Plug-in" and name it appropriately (e.g. 'Send Files to My Opera'), as a Plug-in for "Finder".
Quit Automator

To use: Simply highlight the files you wish to upload in a Finder window, right click with your mouse (or Ctrl+click) and choose "More > Automator > Send Files to My Opera". Automator will then start in the background (you can see the status at the top of the screen). When it is done you will get a message telling you if the files uploaded or not. Done!

Of course if all this seems like too much work, you could just use Unite to share files instead!

cURL is the new Wget? I don't think so!What happened to the Widgets?

Comments

Espen André ØverdahlEspenAO # Tuesday, October 13, 2009 6:48:42 AM

Brilliant. Just tried it and it worked! Sweet.

Ruarí Ødegaardruario # Tuesday, October 13, 2009 6:59:11 AM

Cool glad you liked it. It should be possible to do something similar to upload Photo Albums as well but I haven't really looked into that yet.

it-s # Tuesday, October 13, 2009 7:04:30 PM

Awesome! Thank you so much. This is even better then FTP

Espen André ØverdahlEspenAO # Tuesday, October 13, 2009 7:06:40 PM

I've been using this all day and I'm more and more impressed by how easy it is now to upload files. Thanks again!

Ruarí Ødegaardruario # Tuesday, October 13, 2009 7:33:19 PM

Hey, don't thank me! Thank Daniel Stenberg (author of cURL)!!

You can actually do stuff link this on pretty much any site that has a file upload option. It sometimes takes a little work to decipher how to construct the POST request, and you often have to include a step or two for authentication, but this is usually more than paid for in time saved later if you use a site a lot. If you so desired you could also write simple scripts to list files or delete them as well.

The document "Using cURL to automate HTTP jobs" over at the main cURL website does a good job of explaining how this all works. Also reading the man page will probably also give you further ideas about stuff you could automate.

Oh, and whilst in this instance I used cURL, I suggest having a quick skim of Wget's manual as well. As I mentioned in my previous blog post, sometimes it is Wget that works best for this stuff.

Dustin WilsonKhadgar # Tuesday, October 13, 2009 7:48:47 PM

How clever. Never occurred to me in the slightest to use cURL to handle that, but now that I think of it it's the perfect solution.

Ruarí Ødegaardruario # Tuesday, October 13, 2009 8:26:53 PM

@EspenAO (and others who read this earlier): Since I first wrote this I have edited the blog post slightly. I was originally throwing away the returned HTML content. However it has since occurred to me that this might have some useful information related to failures so I switched the ending from:

> /dev/null

To:

| grep -i -e error -e proxy | sed -e "s/.*/\*\*\*Upload Problems\*\*\*/" | sort -u

This should pick up on any Proxy timeout problems and authentication errors (for example if the cookie expires).

Other keywords could obviously be added as '-e keyword' in the grep command for other failure pages.

There shouldn't be too many false positives as it is not actually the my.opera.com/username/files/ page that is being echoed back to curl on successful transfer but rather a simple '302' page (the option '-L' would have caused curl to follow on to the files page).

Charles SchlossChas4 # Wednesday, October 14, 2009 3:58:35 AM

I wonder what Tamil thinks of this?

Ruarí Ødegaardruario # Wednesday, October 14, 2009 5:58:58 AM

Originally posted by Chas4:

I wonder what Tamil thinks of this?

So, you think Tamil doesn't know about automating content posting on web sites? Haven't you always wondered how those first posts are so consistently achieved on the Desktop Team Blog?

Actually in all seriousness, I don't personally know (or care) if Tamil does this.

Either way Tamil is great asset to our community!

Aleksander AasAleksander # Wednesday, October 14, 2009 6:29:41 AM

I want a Windows version

Ruarí Ødegaardruario # Wednesday, October 14, 2009 7:50:52 AM

Well, Windows users could install Cygwin to give them access to a UNIX-like environment under Windows, which should allow all the original examples to run (with at most minimal tweaking). However, I grant you that that would feel like something of an over kill just to gain the ability to upload files from the command line.

Otherwise, you can still get a Windows version of cURL from links on the bottom of the cURL downoad page and create a batch file.

Having not actually used Windows for quite some time (my home and work machines are all Linux, UNIX or Mac) I can give some hints from my vague memories of Windows, but I can't guarantee any of it will work. I'm almost certain to forget something critical.

That said .. what the hell! Here goes nothing!! Once cURL is installed somewhere in your path you should be able to run the following commands to create your login cookie:

curl -L -c "%userprofile%\myoperacookie.txt" -d "location=&user=username&passwd=password&remember=1" http://my.opera.com/community/login/index.pl -o "%temp%\myopera.tmp"
del "%temp%\myopera.tmp"
attrib +h "%userprofile%\myoperacookie.txt"

Hopefully this will create a hidden login cookie file in your home directory called 'myoperacookie.txt'. Open it in Notepad to confirm it looks correct.

Assuming that worked, save the following in a text file called 'moput.bat':

curl -b "%userprofile%\myoperacookie.txt" http://my.opera.com/username/files/addpic.pl -F file=@%1 -F dir="%2"

After placing 'moput.bat' somewhere in your execute path, you could (in theory at least) run commands like:

moput "%userprofile%\My Documents\funny.jpg" Pictures

Obviously changing "%userprofile%\My Documents\funny.jpg" to whatever file you wanted to upload.

Or if you were already in the correct directory, perhaps:

moput serious.txt Documents

With regards to replacement for UNIX 'find', or some other way to queue up lots of files, that is beyond my memory without a Windows machine to test with. Indeed I am not even 100% certain that any of the examples I gave above will work at all.

Perhaps some Windows command line Guru could come along and produce real examples. I can't imagine it would be too hard on Windows. Indeed I assume that 'PowerShell' would be a better option than 'cmd.exe', but as I have never used PowerShell, I will not even attempt that!

Tracio # Wednesday, October 14, 2009 10:55:27 AM

Very good stuff. Thanks for posting it.

I've included your command in a function and place it in my .zshrc . I've changed it a little bit so it can batch upload files from a single command. Furthermore, by using zsh's extended file globbing, the use of "find" can be avoided in most cases.

function uploader () {
	for item in "${@}"; do
		curl -b ~/path_to_my_opera_cookie http://my.opera.com/my_username/files/addpic.pl -F file=@"${item}" -F dir="My_directory" | grep -i -e error -e proxy | sed -e "s/.*/\*\*\*Upload Problems\*\*\*/" | sort -u
	done
}

Thus a

% uploader **/*.jpg(.)

will batch upload all .jpg files present in your current directory and all subdirectories.

Ruarí Ødegaardruario # Wednesday, October 14, 2009 11:24:02 AM

@Tracio: Very nice!

Ruarí Ødegaardruario # Thursday, October 15, 2009 7:57:09 AM

Hmm ... I just thought of a better way to check that the upload is successful. After upload, I now use the cURL '-I' option to fetch the HTTP-header of the uploaded file. If I get back a "HTTP/1.1 200 OK" then the file must have uploaded correctly. If not, it failed. Simple eh.

I also need to tell cURL to send (my.opera.com) referer page information to the server so that the file header is returned and not the header for the download page.

I have updated the main blog post with this variant.

Espen André ØverdahlEspenAO # Thursday, October 15, 2009 8:02:05 AM

How difficult (or challenging) would it be to have this in some sort of application? Can it be used in a Widget, or are we only limited to the terminal option?

Ruarí Ødegaardruario # Thursday, October 15, 2009 8:24:16 AM

Rather than a Widget this would actually be easier with a Unite Application, as you can nominate an area of the file system that the application is allowed access to.

You wouldn't be able to reuse anything I have written here but that doesn't matter too much as the Unite Framework would allow you to do the same thing.

However a better option might be a Unite Application called something like "My Opera Files Sync", which would work as follows: You start the service and nominate a directory on your hard disk, then any files placed in that folder are automatically uploaded to your My Opera Files store.

Indeed it should be possible to create a similar Unite Application for My Opera Photos.

Espen André ØverdahlEspenAO # Thursday, October 15, 2009 8:41:02 AM

Originally posted by ruario:

However a better option might be a Unite Application called something like "My Opera Files Sync"

This, and:

Originally posted by ruario:

it should be possible to create a similar Unite Application for My Opera Photos.

this would make me a happy man.

Great suggestions!

Ruarí Ødegaardruario # Thursday, October 15, 2009 10:04:30 AM

I'm not going to write a Unite Application myself however as:

1. My JavaScript skills are not up to it.
2. The current solution (with cURL) does the job for me.

Hopefully someone else will be inspired though.

Ruarí Ødegaardruario # Thursday, October 15, 2009 10:06:58 AM

@EspenAO Mac users could wrap some Apple Script around the basic cURL call to make something graphical without too much effort.

I might have a look at that if I find a moment.

Tracio # Thursday, October 15, 2009 10:29:10 AM

A very ugly hack would be to call the terminal from Opera (it can be done from a menu item or button) telling it to execute a one line for loop. If you hardcode a folder and change the command a bit to watch for new files, it will upload new files to my.opera. For example, as a menu item:

Item, "MyOpera-Uploader"="Execute program, "gnome-terminal --execute for item in $(find ~/path_to_dir_being_watched -Btime -1d); do curl -b ~/path_to_my_opera_cookie http://my.opera.com/my_username/files/addpic.pl -F file=@"${item}" -F dir="MyOperaDir" -o /dev/null ;done""

That will automatically upload all the files present in the directory which have been modified in the last day, assuming that Opera can pass a for loop to gnome-terminal... I can't test the menu item code stuff right now but the for loop works fine.

Needless to say that a cron job (or a launchd plist on OSX) would work too, that way you don't even need to execute the script, it would do everything automatically.

Ruarí Ødegaardruario # Thursday, October 15, 2009 1:10:18 PM

@EspenAO: Ok, I wanted to say this publicly earlier but timing was hard as I wasn't sure exactly when we would make an announcement. Anyway, with the direction we are heading with OWD, i.e. stand alone 'applications' and the ability to use the File I/O API. Yes you should be able to create a Widget as well, not just UNITE. (Both for the file and Photo uploader ideas)

Ruarí Ødegaardruario # Thursday, October 15, 2009 6:36:27 PM

@EspenAO It occurred to me that on a Mac the simplest way to wrapper the script and make it graphical is actually Automator. Despite having a Mac at home I have never bothered to use Automator before but I have just fired it up and tried it. True to form, it is very straight forward and hence is perfect for procedural stuff like this.

I now have added Automator instructions to the end of the original blog. Once setup, a user can simply highlight a selection of files in the Finder and right click (or control click) to send them to their My Opera file storage area.

Daniel Aleksandersendaniel # Friday, October 16, 2009 6:14:16 AM

You should upload the Automator workflow to Apple.com/Downloads.

Ruarí Ødegaardruario # Friday, October 16, 2009 6:32:27 AM

Originally posted by danaleks:

You should upload the Automator workflow to Apple.com/Downloads.

Nice idea but in the example I provided, the username is hardcoded, hence anyone wanting this has to re-create the workflow themselves using my instructions.

I'm sure you could get around this by asking for the username and password only the first time (and them storing these), but that requires something more complex and I feel that now I have done these examples to death.

My intention was more to show what could be done 'fairly' easily not to write an entire application, widget or unite service.

So if someone wants something more complex or with more options, I think they will need to write it themselves.

endless lovepersianweblog # Saturday, October 17, 2009 6:11:40 AM

Ruarí Ødegaardruario # Tuesday, October 20, 2009 8:14:23 PM

By the way you can also use the same technique for batch uploads to the File Inbox Unite Application.

Here is a very simple cURL script example (with no error checking or confirmation of upload):

#!/bin/bash
unitesessionid=XXX
unitenonce=`curl -s -b "unite-session-id=$unitesessionid" http://devicename.username.operaunite.com/file_inbox/ | grep unite-nonce | sed -e "s/.*value..\(.*\)\".*/\1/"`
curl -L -b "unite-session-id=$unitesessionid" http://devicename.username.operaunite.com/file_inbox/  -F unite-action=upload -F unite-nonce=$unitenonce -F  file1=@"$1" -o /dev/null

You would need to configure this as follows:

XXX should be replaced with the value from a 'unite-session-id' cookie, taken from a browser that is logged into the 'File Inbox' Application as a visitor.

devicename should be replaced with the device name for the unite service you are connecting to.

username should be replaced with the My Opera username of the owner of the unite service you are connecting to.

In Opera you could get the unite-session-id cookie value from the Opera cookie manager (in the Advanced Preferences). Though, as the cookie will expire reasonably frequently, you will either need to update this value manually from time to time or write another cURL script to auto-update it for you. Personally I linked my copy of the script to my w3m cookie file. Then I can just re-login with w3m to get the script working again, whenever the cookie expires. To do this, I just need to change the line that sets the 'unitesessionid' variable to:

unitesessionid=`grep unite-session-id ~/.w3m/cookie | sed -e "s/.*unite-session-id\t\([a-zA-Z0-9][a-zA-Z0-9]*\).*/\1/"`

In case you are not familiar with it, w3m is a handy terminal web browser that I use from time to time on machines with no GUI running. I figure I can recommend it in good confidence because its a great little browser, and it is not exactly direct competition!

Dustin WilsonKhadgar # Wednesday, February 2, 2011 3:38:27 AM

Hey Ruarí, I've been using this script for some time now, and for a bit now (ever since the new My Opera) it's quit working because the generation of the cookie isn't working. I've looked at the form on the login page and nothing has changed... it's just that no cookie is popping out. Would you have any clue why?

Ruarí Ødegaardruario # Wednesday, February 2, 2011 9:11:21 AM

Hmm ... actually I don't know but I'll have a look later. I must admit I generally don't upload much to My Opera these days because I have an account on people.opera.com (for Opera employees) but occasionally I might need to do this so it would be nice to have it working again.

Ruarí Ødegaardruario # Wednesday, February 2, 2011 9:59:52 AM

Ah hah: http://my.opera.com/community/login/index.pl

Now has to be: https://my.opera.com/community/login/index.pl

It should work again if you make that change. I am actually glad we did this as logins really should be encrypted. I just wish I could have my whole My Opera session encrypted!

P.S. I updated the examples above.

Espen André ØverdahlEspenAO # Wednesday, February 2, 2011 10:43:52 AM

I have a few difficulties with this again. Last time I did this, I had a techy friend help me. I'm a complete programmer nab, so excuse my questions if they are completely off...

1. How do I "Next save the following as shell script."?

2. How do I "Make sure you make the script executable and save it somewhere in your execute path."????

Ruarí Ødegaardruario # Wednesday, February 2, 2011 11:41:09 AM

Espen, you are on a Mac right? Ok, this gets techy pretty fast. Basically your path ($PATH) includes all the directories that MacOS looks for executable files. Though it is not enough just to place files within them. They need to be explicitly set as executable as well.

However, rather than add this script to one of the standard directories it is probably better to make your own directory. Here are some instructions to do just that:

1. Start the terminal (you can find it in Applications/Utilites/Terminal)
2. Type the following and press enter after each new line:

echo "export PATH=\$HOME/.bin:\$PATH" >> ~/.profile
source ~/.profile 
mkdir -p ~/.bin

Ok, now don't close the terminal as we'll use it for the next part, making the script file and making it executable.

1. Enter the following in the terminal (and press enter after each line) to create, make executable and then edit the script:

touch ~/.bin/moput
chmod +x ~/.bin/moput
open ~/.bin/moput

2. Copy the code (with any alternations you need) into the TextEdit window, save and exit.

Ruarí Ødegaardruario # Wednesday, February 2, 2011 12:33:06 PM

If you have GNU Parallel installed you can change this example:

ls *.jpg | xargs -I@ -d"\n" moput @ Pictures/jpegs

To:

parallel moput {} Pictures/jpegs ::: *.jpg

This is neater, easier to read and will be faster if you are uploading lots of files, since it will do several simultaneous uploads, rather than doing each sequentially.

Dustin WilsonKhadgar # Wednesday, February 2, 2011 5:39:44 PM

Originally posted by ruario:

Now has to be: https://my.opera.com/community/login/index.pl

Rats! Thanks! On another note would you know of a way to get a listing of all the folders without having to grab the HTML through cURL and mine it?

Espen, you can also do it this way. Open terminal and type in:

chmod +x /path/to/your/script

Upon success close Terminal. Click on the desktop somewhere to make Finder be the active application and then press ⌘⇧G. This will open a window with a dialog box asking you to enter in a path. Type in /usr/local/bin. If it doesn't exist (and if I remember correctly it doesn't on a fresh install) you'll have to create the folders:

sudo mkdir -p /usr/local/bin

It'll ask for a password. Just put yours in and repeat the previous step. Lastly you just drag your script over to the folder that's open. It'll bitch and ask for a password. Open up Terminal again and your script should work.

Ruarí Ødegaardruario # Wednesday, February 2, 2011 9:27:45 PM

Originally posted by Khadgar:

On another note would you know of a way to get a listing of all the folders without having to grab the HTML through cURL and mine it?

No I don't but if it is just the list of directories you want (not the contents) it is pretty easy to grab the list:

curl -sLb ~/.myoperacookie http://my.opera.com/username/files/ | sed -n "s,.*Directory: \(.*\)</td.*,\1,p"

Change 'username' in the above example to something appropriate, like perhaps 'Khadgar'

Ruarí Ødegaardruario # Wednesday, February 2, 2011 9:45:37 PM

Or perhaps you are after the following (you'll need to change the two 'username' references):

curl -sLb ~/.myoperacookie http://my.opera.com/username/files/ | sed -n "s,.*class=\"name.*/username/\(.*\)\">.*,\1,p"

Dustin WilsonKhadgar # Thursday, February 3, 2011 1:02:37 AM

Thanks! Yeah the first one was what I was looking to do. Right now I'm making an automator workflow that accepts files from the Finder, creates .myoperacookie if it doesn't exist, grabs a list of folders, shows the folders for the user to select in a nice dialog box, and then lastly uploads the files to the folder selected. I have it working perfectly, but I'm interested in what you commented earlier about Parallel. My experience with Parallel is quite limited (and I do have it installed), but what if I had a list of files (like what you'd get from using Automator's "Get Selected Finder Items") what would the syntax be for that? Pretty much what I'm thinking about doing here is detecting the presence of parallel, and if it doesn't exist then it'll just upload the files the regular way.

And when I'm done I'll share my Automator workflow with everyone, obviously.

Ruarí Ødegaardruario # Thursday, February 3, 2011 6:31:06 AM

I don't recall how Automator formats a list when you "Get Selected Finder Items" (I am not by a Mac right now) but if you can somehow present this list to GNU Parallel, then Parallel will be able to do what you are after. In my above example I appended a list to the end of the Parallel command but you could instead pipe a list into Parallel or have Parallel read from a file that contains the list, e.g.

ls *.jpg | parallel moput {} Pictures/jpegs

parallel -a list_of_files_to_upload moput {} Pictures/jpegs

Also, just to be clear. If you are using Parallel it should be GNU Parallel as there is another program of the same name which works differently. In case you are unsure about detecting GNU Parallel. Here is a small detection script that you could tweak to your needs:

if command -v parallel >/dev/null 2>&1
then
	if parallel --version | grep -q "GNU parallel"
	then 
		echo "GNU Parallel is installed"
	else 
		echo "The Parallel installed is not GNU Parallel"
	fi	
else
	echo "GNU Parallel is not available"
fi

Originally posted by Khadgar:

And when I'm done I'll share my Automator workflow with everyone, obviously.

Cool, it sounds great!

Dustin WilsonKhadgar # Thursday, February 3, 2011 7:05:41 AM

When I saw you mentioned GNU parallel I downloaded it and compiled it for my machine, so I know I personally have GNU parallel. However, not everyone will... so the check you provided there would work nicely; before I was just doing a simple command -v check. It'll be more fool proof with what you provided.

Well I'm essentially passing everything through to a "Run Shell Script" action, so I'm accepting everything as arguments because I have 2 other bits of information to receive (username and the remote folder the user selected to upload it to). Firstly I've set moput as a function. After I set the first two arguments to variables and shift what's left is just the list of files to be uploaded in $@, and that's where I'm having difficulty — in passing $@ to parallel. Either I can't find any examples of it or I'm just missing something (and I could be because of the late hour

Ruarí Ødegaardruario # Thursday, February 3, 2011 10:55:07 AM

Presuming that you set your upload directory as the a variable called "directory" and another called "username", you can probably do something like this:

if parallel -H1 curl -sLb ~/.myoperacookie http://my.opera.com/${username}/files/addpic.pl -F file=@{} -F dir="${directory}" \>/dev/null ::: $@ 
then 
	echo "All files were successfully uploaded"
else
	echo "Failed trying to upload a file. Further uploads have been cancelled."
	echo "Check My Opera files, before proceeding!"
fi

It probably needs some tweaking to work with the rest of the stuff you have but perhaps it gives you some ideas.

Ruarí Ødegaardruario # Thursday, February 3, 2011 11:37:07 AM

Originally posted by Khadgar:

to create the folders:
sudo mkdir /usr/local; sudo mkdir /usr/local/bin

By the way you can do that as one command:

sudo mkdir -p /usr/local/bin

Dustin WilsonKhadgar # Thursday, February 3, 2011 11:10:42 PM

Aye. Only thing is that what you wrote there won't ever error unless parallel itself has a problem. It doesn't error if there's a problem uploading the files. Here's what I ended up doing to get it to work, but I'm sure there's an easier way:

 if ! parallel -H1 "if ! curl -sLb ~/.myoperacookie http://my.opera.com/username/files/addpic.pl -F file=@{} -F dir=\"${FOLDER}\" | grep -q \"name.*${FOLDER}/${1/*\/}\"; then exit 1; fi" \>/dev/null ::: $@; then 
  displayError "Failed trying to upload a file. Check My Opera files before proceeding again."
  exit 1
 fi

displayError is a function which uses osascript to display a dialog box with the error. The only problem I see with this is that the dialog box will not display until all of parallel's tasks are done.

I purposely left 'username' in there so it would error.

Ruarí Ødegaardruario # Friday, February 4, 2011 5:52:42 AM

Originally posted by Khadgar:

Only thing is that what you wrote there won't ever error unless parallel itself has a problem. It doesn't error if there's a problem uploading the files.

Sure it does, I read the man page to check how and when parallel changes its exit code before I wrote my example and then manually checked it. Parallel with '-H1' will shut with an error that it inherits from the failing job, if one of the curl jobs fails.

Originally posted by Khadgar:

The only problem I see with this is that the dialog box will not display until all of parallel's tasks are done.

That is also not correct and is the whole point of the '-H1'. '-H1' means that Parallel will halt execution as soon as it realises that one of the jobs failed. Only jobs in the middle of execution will continue. If you want to kill everything immediately use '-H2'.

Ruarí Ødegaardruario # Friday, February 4, 2011 5:56:59 AM

Originally posted by GNU Parallel man page:

-H <0|1|2>

0 Do not halt if a job fails. Exit status will be the number of jobs failed. This is the default.

1 Do not start new jobs if a job fails, but complete the running jobs including cleanup. The exit status will be the exit status from the last failing job.

2 Kill off all jobs immediately and exit without cleanup. The exit status will be the exit status from the failing job.

I also checked it myself by getting parallel+curl to upload a set of files, one of which does not exist and checking the value of '$?'. It ends up with an exit code inherited from the failed curl job.

Ruarí Ødegaardruario # Friday, February 4, 2011 6:10:07 AM

Also I realised you might not need this bit (which is why it is missing from my snippet above):

| grep -q \"name.*${FOLDER}/${1/*\/}\"

because you can just use the exit code of parallel (which parallel -H1 will take from the failing curl task if there is a problem) to confirm that things went ok and shove the output to /dev/null

Either way has a disadvantage. Grepping the page means that you see that the file is listed in the output but will fail if the format of the page ever changes. Furthermore if the web based error page mentions the file and folder it was trying to upload to you may get a false positive. Checking only the curl exit status means that all you know is if the sever accepted the complete post request, not if the file really was added to your list. However, it seems unlikely that it would take the file and then not add it to your list, though I must admit I have not actually tested this.

Dustin WilsonKhadgar # Friday, February 4, 2011 8:52:49 AM

I can explain better. First let's disregard my last comment for a minute and focus on your comment where you showed me how to feed $@ into parallel. After seeing that example I had an example that should work for what I'm attempting to do so I could read the manual and compare it to what you wrote to understand what was going on. That's how I've learned programming. I learn by example.

In the code in your comment you're feeding curl into parallel, telling curl to send stdout to dev/null, and expecting parallel to return false when a problem occurs so the subsequent actions could occur depending on the outcome. In the process of programming my script I intentionally left everything as you wrote it into my script so I could test to see what happens when an error does occur when valid files are told to be uploaded to a location the user has no permission to upload to. I feed the script files on my computer which do exist to a curl command within parallel that will not be able to upload anything because the folder and username it's being told to upload to is incorrect. In this situation curl then receives a 404 page, retrieves the HTML data it's being fed, and then sends it to nothingness like it's being told to. There's no error to output. Parallel goes through all the tasks it's being told to without an error, so it outputs nothing. In outputting nothing the script then echoes "All files were successfully uploaded" when obviously they weren't.

I then attempted a way around that problem. I fed something that was similar to your initial if statement into parallel. However, despite the -H1 flag parallel was only quitting when it had attempted to upload all the files. When I initially made the comment I thought it really was doing that in error; it was not. Even with 7 files it looks like it was running all seven simultaneously, letting the others finish out upon the first error. It ran all 7, but it still was doing what it was told to do. So, yes. I was incorrect in that assessment, and I'm now fine with how that behaves.

Before doing all that I failed to look at curl's flags more closely. The reason why it wasn't returning an error upon receiving a 404 from My Opera's server was because the -f flag wasn't set. If that's set everything works as intended, and like you said the grep isn't needed.

So in a few days when I get time to hook this service into Growl (only runs if it's available) I'll make it available. Also, I'm sorry for clogging up your comments with all this.

Ruarí Ødegaardruario # Friday, February 4, 2011 12:15:49 PM

Originally posted by Khadgar:

That's how I've learned programming. I learn by example.

That is how I also learned the very limited bits of scripting I do.

Originally posted by Khadgar:

In the process of programming my script I intentionally left everything as you wrote it into my script so I could test to see what happens when an error does occur when valid files are told to be uploaded to a location the user has no permission to upload to.

Ah hah, yes I see the problem!

Originally posted by Khadgar: