Blog Home

Developing Datawasp.

4 years ago I had just come off a frustrating project involving management of a 40 strong development team on a really demanding project.   The project involved lots of internal data sharing (what project doesn't) and we used spread sheets to do this but they were inadequate.  The only alternative was to develop bespoke relational database solutions to each problem but the requirements changed every week and database solutions are too brittle.

When the project finished (not very well) I decided the world needed a data sharing solution in between these two extremes and I was the person to develop it for them.

As you might be able to guess from the timescale there was  a lot of feature creep and direction changing during development.

As originally conceived the project was just a formalized data store with multi user access to a shared file.

The idea was that you defined some object types and then created and populated them.

Major revision 1 - Feature explosion

I quickly realized that

a) this was only marginally more useful than a spread sheet for storing data
b) lacked the calculations of a spreadsheet which allowed some kind of reporting
b) it was as hard to use as a database. Harder because it did not draw on any widely used paradigms.

Slowly I added bits to give it more power.

Nested tables
Calculations
More data types
Data formatting
Embedded Files
Rich text

The power grew but the learning curve got steeper and steeper as I added more and more concepts.

Major revision 2 - Replace features with calculation system

To make the system simpler I trashed a load of features and replaced them a single calculation system which could achieve pretty much everything that the feature explosion had done.  The features that were replaced were reporting features and features which allowed rows from different tables to be combined. The calculation system allowed the user to do all these things and more but required them only to learn the calculation system.

To facilitate the easy use of the calculation system I developed a dialog to help create expressions which made it as simple as humanly possible - presenting all the available options in each scenario in a drop down menu.  In addition I stuck to the same kind of syntax you would find in a spread sheet.  The only addition was the use of the 'dot' operator to dereference properties of objects.  At this stage the whole thing was still presented to the user as an object store.

At this stage I decided that the finished article has to meet the following requirement.

You had to be able to use it out of the box.. No training - no poring over help files  - no tutorials. 

This came from my experience with games.  Right back in the far distant past 1980s games used to come with manuals explaining how they worked.  Then it became clear that the customer preferred products where they did not have to learn anything before playing and that is the way the industry went.  Data management has not - so far - taken this view.   There are training courses on how to use a word processor - let alone a data base.  At the time I felt that my application met this requirement. Well its not actually a crime being wrong.

Major revision 3 - From object store to spread sheet.

After about 2 years I thought I was largely finished but I thought that before I got into hard core testing and polishing I should try my product out on people.  So I wrote out a set of instructions laying out a hypothetical situation where someone was downloading a trial version for a specific situation.  I gave this sheet to a few friends and sat over their shoulder whilst they tried it out.

I started with my girlfriend (of 20 years and 3 kids. A lot more than a girlfriend but language is inadequate)  after half an hour she was thoroughly lost and I was nervous.   I tried it out on her sister with the same result. Now I was very scared.

I went through a period of making revisions and trying the result out on fresh victims. I felt that once some one had had one shot they were useless for the acid test of the first impression.

This period lasted about a year.  The big changes I made were:-

1) The abandonment of object oriented language. Objects became rows and properties became columns. This made a big difference as surprisingly few people are familiar with the language of object oriented programming but most people know what columns and rows are.

2) The data design was merged into the main display.  Up until this point the structure was designed in its own display mode.  Each 'object' had its own window and property lists. It was very much like the design mode of the databases I thought were too hard to use.  I decided that spread sheets worked because people were unaware they were designing a data structure at all and would deny it if you accused them of it. I decided that I had to cut down the number of concepts as well as the number of features.   The result is that in Datawasp to add a property to an object type you just add a column to table - blisfully ignorant of the huge ramifications this has. I.e. the object store in every machine attached to the database has to be changed while the user is using it - along with the display which might actually be being used at the time.  Not only this but the store on disk every record of that type has to be read and re written.

3) I conglomerated as many data design dialogs as possible into one huge one.  Instead of 10 small dialogs you have one big one. This seem like a bad thing but what happens is that people get a feel almost straight away for what is available.  If you add a text column you are presented with the possibility of adding a sub table or an embedded file or a calculation. Also you notice that all you ever see is this one dialog. That's it; its all you have to know.

4) I switched away from menus and towards buttons on the display linked to my one mega dialog.  The buttons on the main display get you past the blank page problem.
I feel you never want leave the user in a dead zone where they have to educate themselves before proceeding.  When you create a new database now you get a blank page with a single friendly button saying 'Add Paragraph'.

The result of this is pretty much the same as the first release version but the work was not finished.  Oh no not by long chalk.

Major revision 4 -Testability

This change had less to do with design than to do with testing.

As a game developer I am aware of the huge testing effort that goes into game development.  There is typically a room full of people playing the game in a very organized way to exorcise every bit of code and every resource and if possible break it.  The typical game will have between 2 and 5 thousand bugs registered and resolved before launch.  Up to this point I was kind of ignoring this problem.

Once the design was settled in my mind I could not longer ignore this.

To start with I tried it the manual way.  Carefully wrote out a list of test scripts which I laboriously exorcised one after the other. The word boring does not even begin to describe the mind numbing tedium...

I realized however that there was a larger problem than merely covering all the ground.  As soon as I fixed one bug I was invalidating every passed test.  And I had to make changes just thru most of my test scripts.  No only was it going to take 2 months to get thru my tests but at the end I was going to have to start again. And if I made one change I was going to have to start again.  Kafka and Poe would have smiled - I almost broke down.

I solved the problem by writing a scripting system which linked multiple applications to a single scripting hub which could record then play back the users actions on several application instances. 

Writing the hub was relatively easy (that doesn't mean it was easy just that it was easier than what remained) the hard part was going back into my code and rewriting it so that every user action recorded itself with the hub and then was capable of being replayed under instruction from the scripting hub.

What made this work really horrible was the thought that
a) the work was not adding any features.
b) the nagging doubt that it might be faster just to test the manual way
c) the extra code to do the testing could also be a source of bugs.
d) at some point the testing code was going to have to come out and that this would amount to a change which would invalidate the previous testing. (I eventually  'solved' this by the brilliant expedient of leaving the testing code in.)

All was well however  when the system started working.   I added a bit which snapshotted the interface during recording and compared the stored snapshot with a new one during playback.

Eventually I was able to use 2 computers to run 20,000 test steps in about 3 hours.

The final version went out with a completely green test result on every one of those steps.  Hooray and hooray again

Major revision 5 - The licensing system.

This is where I taught myself web design and database design and server-side scripting    I am still not sure if I am too paranoid about IP protection (anti piracy measures) but at the time I was convinced that I needed some.  I was not really concerned with true hackers - I felt there was little that could be done about these. My real concern was casual abuse. People handing the software to friends inside their organization or in other organizations.

At first I looked around for off the shelf solutions but could not find one that did not have what seemed to me major flaws.  In the end I decided to develop my own.  It was at that time that I had the suspicion that subconsciously I was scared of launching and was finding ways to delay.   I carried on any way: learned html' learned asp.net' learned SQL (what an irony)'  learned Javascript

I am really happy with the final solution but a little bit of me still wonders if it is not just a grand delaying tactic - if so score one for my self conscious mind.

So now I was ready to launch.  So what is this thing....marketing....

Click here for an account of my 'marketing' efforts so far.