an Opera refresher

hello,

I searched for "refresher" and got no response. I also looked over these threads and they don't exactly cover what I need:
http://my.opera.com/community/forums/topic.dml?id=298220
http://forums.g4tv.com/showthread.php?t=170985

I am looking for a way to either develop (or buy, if it already exists) an application that refreshes a URL using Opera and every time it reloads the page, saves the HTML with a time stamp; e.g., My_Page_10_12_2011_08_00_15.html. (I don't really care what the file name format is. The important thing is that the reload should not overwrite the last page that was saved.)

I am not sure if this is a request for an extension or a widget.

You can purchases refreshers that reload pages and save HTML for IE. However, I don't see any that do this for Opera.

There's another possibility I have been thinking of. Is there an application available for Blackberries that could allow you to configure a hot key (e.g., F10) that reloads an Opera browser, does a View/Source and saves the HTML using a system-generated name. The trick again is that the file name should not overwrite the previous one.

Opera is my browser of choice.

thanks,

You could try using AutoIT to create a script that clicks the right sequence of buttons to do this. Opera Actions might work too, but I don't see any way to save a page with a specific name. You could also use the two together by, for example, creating a keyboard shortcut with Opera actions that opens the URL in a new tab and opens the "Save as" dialog, then making a AutoIT script that presses the shortcut key, waits for the dialog to open, then enters the file name and hits Enter.

Extensions would be ideal for this, but they unfortunately can't access the file system at all.

Originally posted by spadija:

Extensions would be ideal for this, but they unfortunately can't access the file system at all.

What about File I/O API (http://dev.opera.com/libraries/fileio/)? Is it strictly for widgets?

file io api was explicitly disabled for extensions at one point.

i hope they will reconsider this.

Spadija, thanks! I have a lot to learn; however, I'm really glad there is a possibility of accomplishing what I need.

thanks again.

I'm sorry, but solution based on AutoIT is not easy both in programming and for end user. The following alternatives are possible:

1. Develop it as widget + use File I/O API for saving files. This is very easy to implement, the only drawback is that widget can not get url from address bar of browser automatically, user will need to enter url manually in widget interface. Widget docks: http://dev.opera.com/articles/widgets/, File I/O API docs: http://dev.opera.com/articles/view/file-i-o-api-for-widgets/

2. Develop it as extension + connect to the following Unite app for saving files: http://unite.opera.com/application/332/

3. Heavy hammers that always work: extension + use NPAPI plugin or Java applet for saving in files. Drawback: neither plugin nor applet can be embedded directly into oex file.

sergey-pypyrev, thanks a lot for the additional information. I would like to see if I can use AutoIt in conjunction with either a widget or an extension, as I don't have the luxury of typing in the URL manually for this project.

I have a lot of reading to do. Thanks again!

Hi sergey-pyprev,

Develop it as widget + use File I/O API for saving files. This is very easy to implement, the only drawback is that widget can not get url from address bar of browser automatically, user will need to enter url manually in widget interface.

When you say "need to enter url manually in widget interface" are you saying that I need to type it in? Or, can it be entered once in a config file?

Also, if you (or anyone else) can point me to an example(s) of widget(s) using the file I/O I'd appreciate it. That would help me with the learning curve.

thanks!

There are rather old, but both of my widgets use the file I/O API to save text data. Search Organizer, CSS Fix. If you right click the "Install" button and "Save linked content as", you can save the widgets without opening them, then unpack them as zip files to get to their source code.

Originally posted by cafescott:

When you say "need to enter url manually in widget interface" are you saying that I need to type it in? Or, can it be entered once in a config file?

It can be done either way.

Hi Sergey-pyprev and Spadija,

thanks! this is tremendously helpful.

Hi,

I see my task in building this refresher widget as reading the HTML from a particular URL into memory and then writing it to a HTML file using the File I/O library. However, I don't want to read the HTML using the Microsoft.XMLHTTP class. From past experience the code I've written using XMLHTTP has pulled the HTML into memory as if it were from an IE browser. I really want to convince the source web site that I am reading the HTML in from an Opera browser.

I've taken a look at the Opera libraries:
http://dev.opera.com/libraries/

Maybe I'm missing it, but I don't see a library routine that reads HTML from a URL.

Does anyone have an idea how to read HTML from a URL and appear to the host website that I'm doing this from an Opera browser?

thanks,

I don't know how XMLHTTPRequest worked in old versions of Opera, but in recent versions it reports user agent as Opera. So, it should get the same HTML as regular page loading.

Alternative is to go with extension. Extension can get html from document.documentElement.outerHTML. But it will be more difficult to implement saving file in extension.

Also, don't use Microsoft.XMLHTTP (it doesn't exist in Opera anyways). Just use XMLHttpRequest, or if you want something easier, just use jQuery's $.get() function (set $.support.cors to "true" or else it won't work in widgets and extensions). jQuery (and many other JavaScript libraries) abstracts XMLHttpRequest to make it easier to use.

Spadija, thanks again. I really appreciate the info!

hey sergey-pypyrev, I forgot to thank you also. Thanks!

does $.support.cors play nice with Opera? I created a simple HTML page in the hopes of seeing whether I can scrape some HTML. The file (index.html) is run off of localhost on my own server. (The jquery-1.6.4.js file exists.):

* * *
<!DOCTYPE html>
<head>
<title>Refresher</title>
<script type="text/javascript" src="js/jquery-1.6.4.js"></script>
</head>
<body>

<script type="text/javascript" >
$(document).ready(function() {
$.support.cors = true;
alert('after support statement');

$.get('http://www.yahoo.com', function(data) {
$("#result").html(data);
alert('Load was performed.');
});
});
</script>

<div id="result"></div>

Do we see anything?

</body>
</html>


* * * *

I get the Javascript alert after setting $.support to true. However, at that point nothing happens. I see the statement "Do we see anything?" on a blank page.

Does anyone have any ideas about how to grab some external HTML and display it on a locally-run browser? thanks.

Originally posted by cafescott:

does $.support.cors play nice with Opera?

You can only do cross-site requests inside the context of an extension with permission to do so ( i.e. from the background process with <access origin="*" /> in config.xml)

Trying to do this from a regular webpage is subject to normal security restrictions, and Opera doesn't yet support CORS, so you can't do cross-domain XHR requests from a normal webpage.

The $.support.cors = true line is there only to make jQuery aware that it can do cross-origin requests, otherwise it short-circuits the request with an error without even sending it. So this is more of a case of incorrect assumptions in jQuery and not actually enabling CORS in the browser.

See also:
http://my.opera.com/community/forums/topic.dml?id=988782

Hi d4n3,

I apologize for the fact that you keep telling people the same thing about cross-site requests. Is there any particular place where the origin="*" statement goes in the config.xml?

I made a simple widget. When it runs I get a white screen with my text: "Do we see anything?" The URL doesn't load. The two alerts don't execute. (BTW, any tips on how to debug widgets?) Here's my stuff.

index.html:
<!DOCTYPE html>
<html>
<head>
<title>Refresher</title>
<script type="text/javascript" src="js/jquery-1.6.4.js"></script>
</head>
<body bgcolor="#FFFFFF">

<script type="text/javascript" >
$(document).ready(function() {
$.support.cors = true;
alert('after support statement');

$.get('http://online.wsj.com/home-page', function(data) {
$("#result").html(data);
alert('Load was performed.');
});
});
</script>

<div id="result"></div>

Do we see anything?

</body>
</html>


config.xml:
<?xml version='1.0' encoding='utf-8'?>
<widget defaultmode="application" network="public">
<widgetname>Simple Refresher</widgetname>
<access origin="*"/>
<description>Loads HTML and saves it.</description>
<width>800</width>
<height>1000</height>
<author>
<name>John Doe</name>
<email>nope@nowhere.com</email>
<link>http://nowhere.com</link>
<organization></organization>
</author>
<id>
<host>http://online.wsj.com/home-page</host>
<name>Simple Refresher</name>
<revised>2011-10-19</revised>
</id>
<feature name="http://xmlns.opera.com/fileio" required="true"/>
<security>
<access>
<protocol>http</protocol>
</access>
<content plugins="no"/>
</security>
</widget>


Any help is greatly appreciated. Thanks.

Well, I just discovered the problem. My archive didn't have the JQuery script file!! Stoooopid!!

I'm getting the result I expected. Thanks d4n3!

hi,
I've encountered a problem that maybe someone here can help with. My little widget now does a pretty good job of refreshing a page. However, there are differences between the page that the widget accesses versus what I obtain using another Opera browser. I think this is because the website delivers different content based on whether the request is asynchronous or not.

I have found this page:
http://stackoverflow.com/questions/6849686/how-to-change-ajax-default-settings

It recommends this code:
$.ajaxSetup({
async: false
});

Unfortunately, it doesn't work. A good example is yahoo.com. My widget doesn't match what shows up when I use another Opera browser. Here's the code. (The config.xml is the same as what I posted earlier.)

index.html:
<!DOCTYPE html>
<html>
<head>
<title>Refresher</title>
<script type="text/javascript" src="jquery-1.6.4.js"></script>
</head>
<body bgcolor="#FFFFFF">

<script type="text/javascript" >
$(document).ready(function() {
$.support.cors = true;
$.ajaxSetup({
async: false
});
var intI = 0;
var refreshId = setInterval(function(){

$.get('http://www.yahoo.com', function(data) {
$("#result").html(data);

intI = intI + 1;
$("iteration").html(intI);

});
});

},600000);

</script>

<div id="iteration"></div>
<div id="result"></div>

Do we see anything?

</body>
</html>


Is it possible to still use "get" and make a synchronous request? Or, do I have to throw away the elegance of it and use the more clunky versions of retrieving a URL? (Will this clunky code actually work?)

<edit>
Of course, another problem is the idea of iterating based on setInterval or setTimeout. My understanding is that these calls are inherently asynchronous. So, it may be impossible to repeat a synchronous get HTML request.
</edit>

Many thanks for help.

hi,

I simplified my code to:

$(document).ready(function() {
$.support.cors = true;
$.ajaxSetup({
async: false
});
$.get('http://www.yahoo.com', function(data) {
$("#result").html(data);
});
});

I am still making an asynchronous request. This suggests:

1. $.ajaxSetup ({ asynch: false })
Is not working;

or
2. It is impossible to call "$.get" synchronously.

Well, I sure hope JQuery offers a truly synchronous solution. thanks for reading; and thanks for the help.

jQuery.get is just a shorthand for jQuery.ajax, with some presets, but as far as I can see you can't set the async: false in get().

You could just use $.ajax directly - i.e. :

$.ajax("http://www.yahoo.com", {async: false});

That being said, I highly doubt there would be a difference between data sent with a synchronous or asynchronous request, the server has no way of distinguishing between them, they are identical (you can verify this using dragonfly's network tab or Wireshark). Synchronous requests block the event thread (script execution, UI), so there is really no good reason you would ever want to use synchronous requests.

What is more likely happening is that the widget and the browser use different cookies, so you may be getting a "plain" page in the widget and a "personalized" page in the browser, where you are logged in. Widgets do not have access to cookies set in the browser, widgets run as standalone browser instances with a seperate cookie store.

There is an option for extensions to get shared cookies from the browser (requires a cookie sharing permission), so you can get cookie-customized (i.e. logged in) content from the background - see Cookie sharing in Opera extensions.

Another possibility is that yahoo's front page uses a lot of javascript widgets which are only displayed as a result of scripts modifying the DOM on the page. Just doing an ajax request to the page will only load the raw HTML of the page, so javascript widgets will not be displayed.

You could try loading the page in an iframe instead. That would allow scripts to run.

Hi d4n3 and spadija, thanks a lot for your input. I am starting to use an iFrame while keeping the "get" syntax and the first time I ran it, I got output that was much closer to what I see when I look at yahoo from an independent browser. This suggests you are both right; i.e., we can get the output that is expected asynchronously, and yahoo's javascript is responsible for their widgets.

(It would be interesting to experiment by using the Ajax syntax that specifically includes a statement about a synchronous call; but at this moment I have no reason to believe it would produce anything differently.)

I'm trying to get it to reload every 30 seconds or so and in so doing have screwed up the project. All its doing right now is showing a black screen. (btw, ever have a PC that is running so slow you just want to shoot it? That's what I'm experiencing now.)

Thanks again for the big help. Here's my index.html right now. (I understand that the asynch initialization can be removed as it is not doing anything.)

<!DOCTYPE html>
<html>
<head>
<title>Refresher</title>
<script type="text/javascript" src="jquery-1.6.4.js"></script>
</head>

<script type="text/javascript" >
$(document).ready(function() {
$.support.cors = true;
$.ajaxSetup({
async: false
});

var intI = 0;
var refreshId = setInterval(function(){

$.get('http://www.yahoo.com', function(data) {
$("#result").html(data);

intI = intI + 1;
$("iteration").html(intI);

});
});

},30000);

</script>

<body>
<iframe id="iframeID" width=600 height=1000>
<div id="iteration"></div>
<div id="result"></div>
</iframe>
</body>


</html>

I think that setting html of iframe will not work. You should set attribute src of iframe to 'http://www.yahoo.com/'. When you'll do this - you may have troubles in accessing iframe document. In regular html page this will be denied due to same origin policy. Maybe, widget will allow this. You should try this, if it will not work - you should switch to development of extension instead of widget.

One more thing. Earlier you wrote that you want to get html of document in the same way as user sees it after right click -> view source. If this is your task - then you should not use iframe or find other ways to execute javascript on page. You should just get response of ajax - this will match view source content (and this is closer to how search engines see page). Of course, it will not include various changes done by javascript.

Unfortunately, if he switches to building an extension, he can't save the files. (You can create a new tab with a dataurl, but that forces the user to manually save each file.) Theoretically, it would be possible to have an extension (which can't save files but can get pages) send the data to a Unite service (which can save files), but working with Unite isn't nearly as easy as widgets or extensions.

There is already Unite app that can save files (I provided link earlier). Extension will need to upload files to this app. There is also an option to save files using either Java applet or NPAPI plugin. All these variants are not as easy as file I/O API in widget. But they do allow to solve this task (and they are not so difficult really).

Let's wait first whether cafescott will be able to get html from iframe. If widget will allow this - of course, it will be preferred solution.

hey guys, thanks again for the thoughtful replies. I'll post back when I know more. Thanks.

hi, I now think the best solution is to get AutoIt involved. I just realized that the "Main Bar" toolbar has a handy "save" button which is fortunately enabled by Ctrl-S. When this is keyed in a dialogue box appears. It is helpful if the last file you saved was in "Web archive (single file)" mode. This way you don't have to choose it. At this point you type in a unique name and click enter. I am hoping that AutoIt can refresh an open Opera browser and then handle the saving tasks.

In addition, there is a very handy Opera extension called "Source" which is available from Opera's extension library. Once installed, Ctrl-F9 gives you the page source. Ctrl-A selects all text, Ctrl-C copies it. At this point you can start Notepad. You have to change from ANSI to Unicode format before saving. (I don't know if Autoit can choose Unicode from the list. If not, perhaps there is a way to configure Notepad, or something similar, to always save in Unicode.) Hopefully AutoIt can do these things.

I've tested both saving methods with yahoo, and I haven't detected any differences between what the browser originally shows and what the HTML page displays (when you drop it into a blank browser). If AutoIt can handle which ever method is used for saving, there is no reason to pursue a complicated JQuery solution which might ultimately be elusive.

I've appreciated all the help. This is a nice forum for information. Thanks Sergey-pyprev, Spadija and d4n3.