"Fix the Web" - updated with development roadmap

"Fix the Web" - updated with development roadmap

I'm excited to announce the development of an extension, currently dubbed "Fix the Web", which will address the problem of the many web pages that are broken in Opera as a result of poor coding techniques. If you are a developer, your help would be greatly appreciated in this project. It is in early stages of actual development, but I have a fairly complete plan already.

First I'll describe the extension and then I'll go over the development road map plan.

Extension Plan

My main goal for this extension is to address the problem of broken websites and broken features on websites. Currently, you can go to Opera > Help > Report site problem..., but only the Opera developer team can see those reports. The problem with that is that the problem remains unfixed until the Opera team patches it or the website owner fixes the page.

I hope to design this extension to offer an extraordinarily effective tool to apply patches to websites that have coding mistakes. I'll outline the design ideas I, and others helping me, have put together so far:

When the user encounters a problematic web site, he can click the extension icon and a form to report the site will pop up:



The long-term plan is to implement MyOpera OAuth so that you can login with your Opera account so when you report an issue, it will report it under your Opera username. This will allow us to implement additional features such as when someone replies to your error report, you will get notified. I'm not sure yet if this is possible with the MyOpera OAuth API (https://auth.opera.com/service/oauth/application/new). Please let me know if you know anything about if this API would work.

As you are browsing, a badge will appear on the extension toolbar icon showing how many error reports have been submitted either for that site or for that page (in the preferences, you'll be able to specify which). If you click the icon, the popup will appear and you can click the menu item that says, "View problems reported on this site":



...and this frame will appear (it would be styled nicer):



The inspiration for the comments page comes from the question-answer system at stackoverflow.com. The person who originally can choose an answer. Others who are authenticated could vote on their favorite reply.

When you click "View recent bug reports" in the popup window, it will bring you to a page inside the extension that displays a sortable paginated table of bug reports, by default ordering the reports from newest to oldest.

Another big part of the project is the patches script. It will be an open source script that fixes patches on certain web pages - similar to, but more specific than, browser.js. When a bug report is received, any developer interested in fixing the bug can do the following:

FIRST (important) contact the webmaster and explain the problem and ask him to fix it. If that doesn't work, THEN you can apply a patch via the patches.js file that will be an injected script.

Here is my current idea for database design:

reports table (error reports):

id (unique, int, auto_increment)
username (string)
language (string)
category (string)
time (timestamp)
report (string)
page (string)
domain (string) # for if you want to select reports by website instead of by page; I think I have more ideas of how I'd use it in the extension later
opera_version (string) # opera.version
additional_details (string) # build number, plugins, operating system, etc.

report_comments table (comment replies to the error reports):

id (unique, int, auto_increment)
report_id (int)
username (string)
category (string) # problem severity: minor, major, unusable
time (timestamp)
comment (string)
opera_version (string)
additional_details (string) # build number, plugins, operating system, etc.
is_solution (boolean, default = false)
read_by_op (boolean, default = false) # for alerting the person who wrote the original bug report of this new comment

comment_ratings table (for the comment rating system):

id (unique, int, auto_increment)
comment_id (int)
username (string)
rating (int) # rating can be 1 or -1

Extension Development Roadmap

I still do not have a definite roadmap, but here are some of my first priorities:

- refine & complete the error reporting form in popup.html
- build the website problem submission system
- build the comments frame and comment rating system
- build the table of reports in an internal extension page (possibly index.html)
- (eventually) implement MyOpera OAuth authorization (if possible)

Contribute!

This project has three different parts, so I put each part in its own repository on Github. Most of the work will happen in the main repository, the actual extension files: https://github.com/cyberstream/Fix-the-Web The Javascript patches file is in its own repository here: https://github.com/cyberstream/Fix-the-Web-Patch-Script The server-side files are here: https://github.com/cyberstream/Fix-the-Web-Server-Side

Active development has already started, so if you want to get in involved, jump in now! If you have any feedback whatsoever (including extension name ideas; currently my favorite suggestion is "Fix the Web", but other suggestions will be considered) regarding any part of this extension project, please let us hear your thoughts below!

Development Roadmap

Here is the roadmap I created today for this project.

1. Fetch the patches.js script from Github and put it in the injected scripts file. (this is functional now, but needs to be optimized - see comment in background.js of the extension files: https://github.com/cyberstream/Fix-the-Web)

2. Create a report form in the popup window. done Processing the form when it is submitted will be handled later (in step #5).

3. Make the comment frame appear on the page when "View problems reported on this site" is clicked in the popup. Ideally, the frame should fill the page width-wise, and the height should be adjustable.

4. Create a database on the server for the extension. Three tables will be needed: reports, report_comments, comment_ratings. See the proposed database design in the previous post.

5. Make the error reporting form work and send its contents to the server to be inserted into the database.

6. Add a badge to the extension's toolbar icon with the number of error reports about the current website or page (whether it counts error reports on the current page or domain should be changeable in the preferences).

7. In the frame created in #3, show error reports for the current domain or for the current page (again, controlled by the same option as in #6). Only show error reports, browser version, and additional details (the additional details should be hidden until you click a link to expand them). Don't allow replies yet.

8. Create a table of reports in an internal extension file, maybe index.html...? This is the page that will show up when you click the "View recent bug reports" option in the popup. Reports in the table should be paginated and sortable, although only pagination is important at this point. Sorting and searching can come later. The table should include an excerpt of the report, the page it was reported from, the date and time it was reported, and the version of Opera. There will be an icon on the report's row that you can click so a modal window or frame will be displayed with the full report and all its details.

9. The options page is next. The options should include:
- update interval for checking for an updated patches.js file. (optional values: whenever browser starts, every 15, 30, 45 minutes, 1 hour, 2 hours, 3 hours, 6 hours, or every 12 hours)
- option specifying what reports to show in the badge count and in the reports frame. Value can be: reports for the current page or reports for the current domain
- enable notification when someone replies to one of my error reports. (see #13)
- other options?

10. Translate the extension into other languages. When the form is submitted, detect the locale of the extension (e.g. EN, ES, TU, FR, DE, NO, etc) and automatically include it in the report. Also implement a sort-by-locale option in the comments frame (see image above).

11. Release version 1.0 beta?

12. Implement the MyOpera OAuth API. Require users to be authenticated with their My Opera account in order to submit error reports or reply to error reports.

13. Expounding on #12, once the user is authenticated, he can report website errors, post comments on error reports, and vote comments up or down. The comments and reports will be shown in a threaded layout in the comments frame (see #3 and #7). The person who originally posted the reply will be able to choose a comment that solved their problem. Also, when you report a website problem and someone replies to your report, you will have the option to be notified.

14. Release 1.0 final

If you would like to participate in this project, head of to Github and fork it! https://github.com/cyberstream/Fix-the-Web

Now, we're expecting comments and contributions of My Opera community.

Absolutely! Even if you can't want to get involved with the coding, we would love to hear any suggestions or comments you have about this project. The only way to make it a good extension is if we know what Opera users want.

I don't know HTML and JavaScript enough - so I may not be of any help.

However I think the extension plan isn't very good or is complicated.
If I were to start the very same project-

Create a My Opera group - "Fix the Web"
Start a forum within the group.

Fix the Web extension purposes-
- get latest version of patches.js from Github repository (similar to how Ghostery extension gets updated bug libraries) and patch websites
- Give a link to the forums in the "Fix the Web" My Opera group so that users can create new forum thread to report website problems.

Hi all,
I've just found some time to check the project which realizes one of my dreams.

I don't have too much time to look into code, i have some advices. i wish i have more time to fork and send a pull request but i don't have too much time now :(

* I'm not sure if opera extensions (even userscripts which are included in includes directory) has its own scope, but don't use global JavaScript variables such as "resizeFrame" and "CONFIG" which may override these global variables on current page, and I think Opera crew, who check the extensions' codes may reject these just because of this (firefox does not allow such plugins afaik). Just create a new namespace like var fixTheWeb = {}; and bind all methods under this namespace like fixTheWeb.resizeFrame. At least put a "var" in front of these variables so they are scoped by the closure they are.

* I cannot understand (dont have too much time to dig into code) but, why dont you just include this "patches.js" with creating a <script> tag and setting the "src" attribute and append it to <head> instead of fetching content and putting content in it?

* separate db credentials from db.php so you dont have to execute that "create table if exsists" queries on every request. maybe another file named "install.php" will be fine

* I dont know opera api allows you to send messages/notifications but you can authenticate user with opera OAuth api. Will this be only available to my opera users? if so, you may want to revise your database schema. maybe creating a "users" table and relate this with other tables

* Idea behind ajax_request_handler.php is seems like to privide a JSON api which is good. You better split the queries, request handling and response parts to make it more maintainable. (MVC?)

* separate the markup, javascript and css on serverside index.php. you also have some little mistake which makes this markup invalid. "fix the index.php ;)"

Keep up good work :)

Originally posted by YuXeL:

* I'm not sure if opera extensions (even userscripts which are included in includes directory) has its own scope, but don't use global JavaScript variables such as "resizeFrame" and "CONFIG" which may override these global variables on current page, and I think Opera crew, who check the extensions' codes may reject these just because of this (firefox does not allow such plugins afaik). Just create a new namespace like var fixTheWeb = {}; and bind all methods under this namespace like fixTheWeb.resizeFrame. At least put a "var" in front of these variables so they are scoped by the closure they are.

They do have their own scope. The global namespace in an extension include script is not "window". To place something in the regular global scope, you would have to explicitly reference it using "window".

Originally posted by YuXeL:

* I cannot understand (dont have too much time to dig into code) but, why dont you just include this "patches.js" with creating a <script> tag and setting the "src" attribute and append it to <head> instead of fetching content and putting content in it?

Because then it wouldn't run in extension scope and wouldn't have access to all the nice functions extensions get. It would also require constant web requests to that script with every page load instead of one request every once in a while to download and store the script. Even if all of those requests result in "You've got the latest version. Carry on," the extra time with every page load could add up over time, especially if the remote site goes down.


By the way, I have a nice little library called OperaExtOptions which handles loading and saving every imaginable input type along with resetting options to their default values. There's a little more information about it here on my blog. You are free to use it if you think it will be helpful.

Hey everyone!

I'm sorry I haven't been replying to your posts. I didn't see them until today.

There are some excellent ideas here.

@Swapnil99pro: Thanks for your suggestion. I just created the group. You can view it and join it here: http://my.opera.com/fix-the-web/about/ It is a fantastic idea; it will help us stay much more organized as we develop the extension.

I did create a forum for reporting website problems, but I will close it when I release the extension because you will report site problems through the extension then.

Originally posted by YuXeL:

I dont know opera api allows you to send messages/notifications but you can authenticate user with opera OAuth api. Will this be only available to my opera users? if so, you may want to revise your database schema. maybe creating a "users" table and relate this with other tables

Currently you can authenticate a user with the Opera Link API, but it doesn't have another feature I would need for the extension, so I will be using Twitter and Facebook OAuth. That is the main obstacle that is preventing me from releasing the first version right now. I am working on implementing it as I get the time to do it.

Originally posted by YuXeL:

Idea behind ajax_request_handler.php is seems like to privide a JSON api which is good. You better split the queries, request handling and response parts to make it more maintainable. (MVC?)

I will keep that in mind for the future.

Originally posted by YuXeL:

separate the markup, javascript and css on serverside index.php. you also have some little mistake which makes this markup invalid. "fix the index.php "

I think that was fixed a couple days ago. Thanks for mentioning it!

Thank you for your comments, spadija. Your library looks very useful, but we've already created something in our extension that does that. I'll probably use it if I ever create another extension. Thanks!

First of all - thanks for the invite to this group. I joined Fix the Web group today.
I will start working on the project and patches in April as I am busy for this month.

Thanks for your willingness to participate in contributing to this project. I appreciate it a lot!

That is just fine. By April, I think we can have the extension working well and focus on Fixing the Web one web page at a time!

Originally posted by YuXeL:

separate the markup, javascript and css on serverside index.php. you also have some little mistake which makes this markup invalid. "fix the index.php

Yes, this is in my mind. I will separate css js and html markup on index.php before first release.

One suggestion more, let's assign patches a number, PATCH-1, PATCH-2 and so on as // comments in patches.js - this would allow easier tracking of patches; for example when a user wants to report that the site is changed and some patch is not required, the patch can be simply referred by the number.

That's a good idea. Also, we should probably add the Opera version that patch was added in. Whenever a new version of Opera comes out, we can test the extension to see which patches are still necessary. If they are still necessary, then we can updated the version number in the comment to show that the web page still has problems in that version.

I will add these rules to the patches script how-to page on the Fix the Web blog.

Version 1 Alpha is out for testing! See more info on Fix the Web's blog: http://my.opera.com/fix-the-web/blog/2012/03/04/version-1-alpha-released

Works as expected. Good job FtW team.

The second alpha release is out! It mainly fixes problems with the button and badge: http://my.opera.com/fix-the-web/blog/2012/03/05/version-1-0-alpha-2-release

First alpha version published! https://addons.opera.com/en/addons/extensions/details/fix-the-web/

looks like some big changes on the way, no update for a long time!

Keep watching...work is progressing - slowly - but progressing nonetheless.

Sorry cyberstream - I know it's April and I haven't done anything for the project.
I will write some patches in end of April but I currently have an issue-
I am not a developer - so I haven't used GitHub before. My forks of the project repositories are outdated - I mean the recent changes done to cyberstream/Fix-the-web and other repositories are not being reflected on swapnil99pro/fix-the-web....
Is it OK to add the patch to my fork? What do I need to do to make you notified so that you merge my fork changes?

Swapnil99pro: Basically, when you fork a repo on github you should create two "remotes". One for the original repo (read-only) and one for the fork (this will be probably the "origin" when you clone it). You can then use the read-only repo to update your files. If you have already done some work before updating, you can use "stash" to save the changes and then pop them to the updated version.

commands:
create remote: git remote add upstream git://github.com/cyberstream/Fix-the-Web-CSS-Patches.git
stash changes: git stash
update repo: git pull upstream master
pop changes: git stash pop

there might be some merging issue you'll have to resolve

When you are finished, push your changes to the fork and use the "Pull request" option on github.com.

Cheers,
BS-Harou

BS-Harou: All this is terribly complicated for a new user of gitHub. I know it's my fault, but when I tried your first command (create remote); it gave me an error: fatal - not a git repository (or something like that - meaning the same).

cyberstream: And there is one more suggestion: make a post on the group blog - with list of all patches, in unordered lists - one for each website (not webpage) patched. All patches, whether CSS or JS applied to that website would be listed with patch numbers. (Just like Opera used to maintain one for BrowserJS - http://web.archive.org/web/20100722091750/http://www.opera.com/docs/browserjs/ )

Swapnil99pro: Have you already created the repo somewhere? (git clone ....) If so, you must firstly go to the folder of the repo and use the commands there. Otherwise, you must firstly clone the repo from github with this command (for the repo with Patches):

git clone git://github.com/Swapnil99pro/Fix-the-Web-CSS-Patches.git ftwpatches

then go to the newly created folder "ftwpatches" and create the second "remote":
git remote add upstream git://github.com/cyberstream/Fix-the-Web-CSS-Patches.git

When you are finished with your changes use:
git commit -am "description of the changes"
git push origin master

Later you can update your repo with newest changes of the original repo:
git stash
git pull upstream master
git stash pop

Thankyou very much, that was better - the git clone, git remote add upstream... and git pull upstream master commands worked - haven't tried the rest since I do not have any commits at the moment.

Is git push, similar to doing a pull request from the web interface.
Also one question - does git pull upstream master update my repo in the web interface too?

At the moment there are three repos. One on your machine (master), one is the original repo on github (remote: upstream) and one is your fork on github (remote: origin).

"git pull upstream master" downloads newest data from "upstream" to "master"
"git push origin master" moves data from "master" to "origin" (updates web interface of your fork)
"pull request" on github.com is attempt to move data from "origin" to "upstream"

Once your pull request is accepted, your repo on github (origin) should be automatically updated to the newest version of "upstream". Or at least I think so

Originally posted by BS-Harou:

Once your pull request is accepted, your repo on github (origin) should be automatically updated to the newest version of "upstream".

That's the most important part. THANKYOU VERY MUCH FOR YOUR TIME.

Originally posted by Swapnil99pro:

That's the most important part. THANKYOU VERY MUCH FOR YOUR TIME.

Ppulling newest chanegs from upstream and pushing them to origin works as well

Thanks for the fantastic explanations, @BS-Harou. Maybe they'll help other people learn Github too.

I've still been very busy with other obligations, and I've run into some problems in the extension that I need to fix in addition to completing the new optimizations. Hence, it may be a little while before I can get an update pushed to Github. The best thing to do right now is to work on creating and adding CSS patches and pushing those to the repo on Github.

Originally posted by Swapnil99pro:

cyberstream: And there is one more suggestion: make a post on the group blog - with list of all patches, in unordered lists - one for each website (not webpage) patched. All patches, whether CSS or JS applied to that website would be listed with patch numbers. (Just like Opera used to maintain one for BrowserJS - http://web.archive.org/web/20100722091750/http://www.opera.com/docs/browserjs/ )

Something like this is already planned. Thank you for the suggestion anyway.

Is it just me or you guys moved the project server-side to a different address.

Originally posted by Swapnil99pro:

Is it just me or you guys moved the project server-side to a different address.

OperaTurkiye.net having some trouble about server. Greench are working on it.

Let's organize the project so that it's easier to work on.

1. For every patch submitted, there should be a thread created in the Fix the web group forums. (create a subforum for that).

Reason: Websites never remain the same, or with a newer Opera release, the patch may not be needed. With each patch having it's own thread in the forum, we can track changes more easily. On every Opera release, some people of the group can test the websites and post if they need an update or need to be removed.

2. // This one might be hard to do and incredibly time-taking.

http://www.operaturkiye.net/fix-the-web/ interface should be enhanced - allowing us to categorize reports. Some 2 or 3 people from the group can mark reports as "CSS patch required", "JavaScript patch required", "Won't fix" (for reports of website problems that exist in other browsers also, or for Opera Core bugs causing problems), "Details required" (for when the problem is not stated clearly, or we need code of the website because it requires a log-in).

3. There should be a blogpost once in a month (or frequently when there are more patch contributors) listing all websites and their problems fixed in that month by patches and removed patches. (just like http://my.opera.com/sitepatching/blog/ ).

Awaiting opinions from the project owners and members.

I'm sorry the project's pace has nearly stopped recently. I have a hard time finding time to work on it in the summer months, but I have big plans for when I can pick it back up again. One area I want to improve is adding the patches. Instead of adding them through Github, I plan to make it possible to add them directly through the extension. All patches tentatively will also be viewable through the extension, like in AutoPatchWork.

I'll have to think more about #1. I don't want the project to get disorganized, so I only want to add new things to the project if I absolutely must.

#2 would be useful, but it would be labor-intensive too. It is a good suggestion, so I'll consider it.

#3 would be nice as it would improve communication between us developers and the users.

Thank you for offering your suggestions. Even though I don't have time to work on the project right now, I do have time to think about improvements to the project, so keep the comments coming!