Historical Overview

When the first versions of our VNC Manager were released way back in 1999, the data was stored in a Microsoft Access database.  Despite the popular (and often justified) opinion of Microsoft’s Access database, there are some applications that use data “just right” for that platform and VNCScan was one of them.

After frustrations with requiring multiple runtimes for Access, the decision was made to move the data to an XML format.  A lot of code was written to manage that process smoothly on the thousands of existing VNCScan installs deployed all over the globe.  The XML format has been working pretty well over the past 10 years but there are some pretty serious issues that the time has come to address.

Issues with XML Databases

The largest sacrifice of moving to XML was the ability to share one set of data with many administrators.  When the back end was database driven, multiple computers could pull from the same data at the same time with very good performance and reliability.  After moving to flat XML files, doing so often led to groups disappearing and data files that became corrupted so badly that it prevented the program from starting up.  We had to remove any support for sharing data with XML. because of this.

For example, if John and Mike both point their VNCScan consoles at the same data location, they will constantly be stepping on each others changes.  An XML file must be read into memory, modified, then placed back onto the disk as a complete file (overwriting whatever is there).

If John reads the data into his console, then Mike makes a change to the group, and then John makes a different change to HIS copy of the group, Johns change will wipe out Mikes change even if they change different properties of the group.  If they both try to write their changes simultaneously, they end up with a corrupted XML file.

The Great Database Debate

A while back, I made a blog post asking opinions of using an Access back end verses using a Microsoft SQL Server back end.  I received a lot of emails with great responses from so many of you!

On one hand, the Access database format needs to be compacted and repaired from time to time to get rid of orphaned data and “white space” in the database.  It also does not do well with a lot of threads hitting it at the same time.

On the other hand, Access is a very portable database format that requires no runtime (any more) to use in your program and holds up well with the type of data accesses that VNCScan performs.  I’ve also taken performance heavily into consideration with each line of code that I have written to mitigate any “over use” of the database.

If we were to use a SQL database, we would need to deploy the Microsoft SQL runtimes (MSDE) or require that every customer has a SQL server on premises.  For the style of data that VNCScan employs, both of those options seemed like overkills at this point.

The Future

The initial database version of VNCScan will be backed by a Microsoft Access database.  The code was written in a way that will make it VERY easy to port it to a SQL database in the near future.  While we have tested VNCScan extensively on the Access database, we will be watching “wide eyed” for any indication that the platform isn’t good enough.

I believe that by June, we will have a version that will have a choice between Access and SQL. We’re only releasing the first version as Access based to cure the ills that plague the XML.  Even with its shortcomings, Access databases are a hundred times better than XML.

The Beta

We are in the final stages of “dog fooding” the database version.  That means that we’re using it internally so that we feel any hardships before the beta testers do.

Within the next day or so, you will see a bog post with instructions and a link to try the beta.  I strongly suggest that you back up your <My Documents>\VNCScan folder before installing the beta.

In late 2003, I decided to move away from Access in favor of flat XML.  To this day, I still debate with myself which was better because each have their issues.  The XML file format for data made it possible to ship a smaller product with less dependencies but it severely limited the multi-user capabilities for sharing data.

The XML format is just plain text files. When one console reads the data and then writes a change, there’s a chance that another console that is also reading from that same textual XML file can write back what it thinks is accurate data and overwrite the first consoles changes.  It can get messy real fast.

At least with the Access database, you could have a few users adding and reading data at the same time without overwriting the whole file just for one change to a single field. 

I realize that Microsoft is steering its developers to their mini desktop SQL server instead of Access but it just seems like a lot of overhead for the type of data being stored. 

Today, all of the routines needed to read and write to an Access database is built into the existing DotNet Runtime.  What drove me to this XML format is no longer a problem. 

So, what do you think?  Should we make the move back to the Access database to facilitate a better multi-user experience or stick with the XML?