Project V: Undo and Storage
Saturday, 18. July 2009, 00:32:53
I was thinking about my undo problem again and although I had a solution mostly worked out, perhaps there is an easier way. I wanted to enable the sharing of data amongst multiple users. So you could work on a part of the project while someone else works on something else. Also, I wanted to be able to make changes in someone else's workspace. I thought I had to build a custom back-end platform to enable all of this. Then again, perhaps not. Would there be any drawback in using a wiki to store projects? The data itself would mostly just be junk if you were to view it through a web page, but comments, copyright, inputs, outputs, component name, usage, author, history of changes, etc. would all be there.
I think I'm going to go this route. You can load up a local copy and the IDE will mark another page stating that you're working on it. Of course, the IDE would check this list first and inform you that it's being worked on, giving you the chance to visually connect to the active worksheet on someone else's machine if they allow it. In essence, I would use certain pages as a tracker to monitor who is working on what as well as a way to connect to each other P2P style in order to be notified of changes, especially for chat and concurrent updates on the same worksheet.
I think I'm going to go this route. You can load up a local copy and the IDE will mark another page stating that you're working on it. Of course, the IDE would check this list first and inform you that it's being worked on, giving you the chance to visually connect to the active worksheet on someone else's machine if they allow it. In essence, I would use certain pages as a tracker to monitor who is working on what as well as a way to connect to each other P2P style in order to be notified of changes, especially for chat and concurrent updates on the same worksheet.
Local copies will keep a list of UNDO for each worksheet (usually one possible worksheet per component). You may submit your changes at any time and the IDE will let you know if someone else submitted something before you could add your changes, allowing you to compare changes. And this is possible in Project V through the use of layers where you can superimpose different worksheets one on top of another like transparencies. I'll try and make sure the changes are highlighted in this case if it's something like a description or new input or something like that.
What I like is that this would solve most of my problems and would eliminate a LOT of work. No need to figure out database stuff. I can just add HTTP code and I basically have all the back-end functionality I need while being distributed at the same time. It also solves my problem on how to publish software, especially generic components that everyone needs. Being an admin also allows one to lock components and everyone can manage their own projects on their own sites if they wish. They can even have a project that spans different sites.
There are GPL versions and BSD versions of wikis. So the IDE simply needs a template for different commands (like how to fetch a component page, history and the talk page, etc. as well as submitting changes) and that's it. I already have MediaWiki installed here so I'm gonna do that until there's a real reason to use something else. Writing something to convert the data would be simpler than being locked into a specific database format.
Once I have undo on a per component basis, I'll start adding functionality to fetch and update components through the wiki and finish off component linking. Heck, I REALLY like this idea. Sharing information and projects will be a snap. And then we'll be able to actually start beta testing on a distributed network.
Is there a reason I wouldn't do this? I don't see any glaring flaws with this idea right now. All these problems have been solved already. Not sure why I would re-invent the wheel.
If you have cool ideas or whatnot about this, please share them. I think this opens up a lot of possibilities. Like posting tutorials, videos, documentation, right in the component's page, etc. So even without the IDE, you can still find the info you want. Plus search functionality, especially if online with Google, would again solve a lot of problems. Yeah, I'm liking this more and more.
WIDGETIZE
spc476 # 18. July 2009, 23:57
Vorlath # 19. July 2009, 14:51
The reason I liked a wiki is because you'd have something visual with a web browser as a reference if you want it. I think most programmers are already using the web to look up docs as is and would not be a problem until I can build internal docs. Besides, I think the web would be the way to go anyhow, even if docs are viewable internally.
I'm still going to use a wiki for now. Changing to another system is simple. I'll make sure it works with git as well. Some wikis have systems for user managements and restricting pages by group. That's something I don't want to be writing again.
Anonymous # 20. July 2009, 11:47
My cousin wrote a wiki system in Python (hes refactoring it at the moment, when done it will be on launchpad) which uses bazaar as a backend. This means that versioning and history is handled automagically for him by bazaar. This also means you can have multiple local repositories which are changed independently (a group of users could have their own wiki, which is periodically merged with the main one, for example). One interesting thing, which he implemented, is that the wiki data is stored as simple HTML (with additional external metadata), meaning that you can simply do a bazaar checkout and get the html, point a web server at it and have a static version of the wiki content being served!
I am considering using bazaar as a backend for my own dataflow project, so that projects being worked on with it can a) be versioned automatically with full history and audit trail, b) be shared between multiple users with local repositories for versioning between merges, and c) integrate seemlessly with my cousins wiki, allowing the projects to be viewed and modified in a web interface too. I'm thinking that this can be built up into a full project management web frontend with build system, wiki for documentation, code browsing (at least, the text version - unless I generate images.. we'll see), ticket system, change tracking etc etc.
Anonymous # 20. July 2009, 11:51
PS: glad to hear you're working on Project V! Been too long since your last Project V post... I almost got worried; who else would I discuss/argue about dataflow to :-D
I actually started my own blog now too, where eventually I hope to describe my own ideas on programming and dataflow, but at the moment the content is pityfull. I'd rather not insult you with a link just yet, but once I've had a chance to work on it, I'll drop you a line and maybe you can give me some feedback. Or at least, maybe we can start up a new dialogue and further improve each others ideas on this stuff. (Or more likely, mine, since you appear to have everything thought out fairly well. I guess you had a few years of a headstart :-D)
Vorlath # 20. July 2009, 16:56
And yes, I should have had all this done at least a year ago. (I know, I know. That's not what you said. I still wish I could have done more earlier.) But that time has not been wasted. A great many details have come together quite nicely. I have a great architecture. And I now understand a wide range of areas that you simply cannot read about in any book or on any blog (other than mine). For me, that's already a success.
Anyhow, I'm going to integrate directly with the MediaWiki blog with possibility to configure for other versioning system. The data will be formatted and then converted for that specific storage system. I can use dataflow mechanisms for this even though it'll be internally written.
Post a link to your blog. Doesn't matter if there's only a few things there. I'm always interested in dataflow ideas since they are so scarce (yet growing in number).
Anonymous # 20. July 2009, 17:53
I totally understand! My own project was dreamt up over a year ago and should have been at least in a usable (if simplistic) state right now - instead its only barely left the design phase... I find it much too dificult to dedicate time to this nowadays, work eats up too much time and afterwards I want to get away from computers.. *sigh*
My blog is really in a rough draft state; the last few posts are in need of refactoring really, as I more or less just dumped my thoughts at the time. The earlier ones are ok, though they really just mirror some of the things you've said (probably incorrectly :-D).
Anyway, here you go: http://didntread.wordpress.com/
Vorlath # 20. July 2009, 18:17
I assure everyone that Project V WILL get done.
I can't state a specific date, but I will get it done. Getting projects completed is actually my specialty. But Project V mostly started as a hobby. It's turned into something having it's own life and is definitely not the standard way I usually go about it since Project V and this blog are in a symbiotic relationship and I've never had a blog before (or worked on a project that was intricately linked to a blog). I was learning a lot along the way and did not want to rush it. It'll be a MUCH better product because of it. But now's the time to get on with the show. I still have another project to finish. However, I will make time for Project V.
Anonymous # 20. July 2009, 18:23
I never meant to imply that I thought it wouldn't get done, just that you would post much less often than before and we would be left wondering what the status was until one day BAM completion announcement.
Vorlath # 20. July 2009, 19:27
Vorlath # 23. July 2009, 06:38
I don't know why, but sometimes I read some of my old blog posts like the ones you linked and I can't stop reading them. It's kind of stupid, but I'm going "this is good shit" and then realizing that yeah, DUH I wrote that stuff. My older articles had a lot better topics. But that goes with the territory I suppose. Keeping things fresh after several years is tough to do. Which is why it's high time for me to finish Project V and get something going.
In a way, your blog reminds me a lot of mine in the early days, at least with the sense of discovery. Of trying to piece together varied and often confounded information in order to make it simpler, better and more usable.
Anonymous # 24. July 2009, 11:18
Hey, thanks for the kind words. Blah, I couldn't care less about what the internet police thinks; if the topics make sense, then they make sense. If others disagree, then like you said before, they're the ones losing out - not me.
I hope you don't mind my linking to your material, though. I should have asked first, but I guess it never occurred to me to do so. I feel like I owe it to you to cite your work when mine is based off it.
I guess your older posts (and my current ones) are more theoretical, creative posts about ideas and theories and concepts, rather than something concrete like the implementation, so they're more fun and interesting to read. I quite enjoy reading your old posts, they have an entertainment quality to them while still being enlightening and useful. I hope that I can manage something similar. I have got some good feedback about my writing style and content from friends, so unless they were saying it to make me feel better, I think that perhaps I'm on the right track. I hope so anyway.
But yeah, hurry up and finish Project V haha :-D I'm itching to try it out for real. I think having an implementation, even a simple one, would help others to understand the concepts. Thats what I hope to do with my own blog soon (unfortunately, like you, I'm being delayed by work stuff.. off to the middle-east in two weeks, so have very little time to work on these things :-( thats life, I guess)
Vorlath # 24. July 2009, 14:38
I agree with you about older posts. It's almost like brainstorming where you don't have limits, but where you can see how different concepts fit together, and which don't. And yeah, your posts read well. The only thing I noticed is sometime the transition to a different idea is abrupt, but that goes with the territory when looking at different things. Hope you take that a constructive since I don't have any complaints.
Yeah, I got a good reaction to just my demo that only allowed you to create new implementations and delete them. Imagine if you could create inputs and outputs and link them!!!
Anonymous # 24. July 2009, 16:53
Thanks for the feedback! I appreciate it. I'm trying to edit the posts as I find errors or think of better ways to write the ideas down. I'll see can I make the different ideas flow better. Yep, definitely a brainstorming thing. Speaking of posts reading well, I actually really like your writing style. I enjoyed reading this post, for example: http://my.opera.com/Vorlath/blog/2007/01/04/i-don-t-know
Yes, the demo was interesting. I imagine that once you can construct simple pieces of code, more people will start to understand whats going on. A few friends have already told me that it would be easier to understand the ideas if they could see them in action or play around with them, so I guess thats what is needed. Its certainly an interesting area to play around in though and I can see a lot of good coming from this.
Vorlath # 24. July 2009, 22:44
Anonymous # 25. July 2009, 09:40
I guess nobody commented on it for the same reasons I didn't - I'm more interested in comenting on and discussing the technical posts/articles. That doesn't mean that the non-technical ones aren't worth reading though.
Thanks for the feedback, its appreciated. Can't improve if I don't know what I'm doing right or wrong.
Anonymous # 30. July 2009, 08:26
I just saw this article on slashdot: http://games.slashdot.org/story/09/07/30/0437239/Making-a-Game-of-Hardware-Design
The part that made me want comment was this: "a web-game to harness the natural human abilities for electronic design automation". Does this mean that humans have a natural ability for dataflow? :-P
Anonymous # 30. July 2009, 08:34
I should have tried the game before commenting here.. I was pretty disappointed. I still believe humans have a natural ability for dataflow though.