The Engine Behind this Board Games Website
Over the last few weeks I’ve been building the core engine behind FMG’s main concept.
The engine is literally the workhorse of the website - working away behind the scenes and providing the main functionality that will (hopefully) make the site so appealing and useful to board gamers!
On Concept…
I’m still a little reluctant to give the main concept behind the games engine away at this stage as it encapsulates the entire raison d’etre for the site. I will however talk a little bit about the engine and what it has to do. Given that this is the largest and most complex area of the final site, this module is naturally taking the longest to design and develop, and therefore, will also be the key area in need of the most thorough testing.
The purpose of the games engine is to take input from website users, process it, and store this away in the database for later usage to fulfill the concept behind the site. Normally, engines like this are easy enough to build; but for FMG the engine is much more complex than I had originally thought. I appreciate there’s a little missing context here from a reader’s perspective, but please bear with me and see if you get a feel for what’s I’m babbling about 
The Base Behind The Engine’s Data
Behind any website that builds an online service like this where users can login, save some data, send emails, process a bunch of stuff, and then remember all this next time you come back, there needs to be some sort of storage mechanism. From a web site’s perspective this is the database. Not your normal Microsoft Access style database, but an online database that can store huge amounts of data, and manage many people connecting to it to retrieve such data many times a second.
The database stores all the various data about the website: everything from a user’s login details, to the entire games directory. This data has to be formatted and retrieved using a very specific syntax (a structured query- and data definition- language called SQL).
Storing data correctly is the first hurdle to allowing data to be efficiently retrieved in a way that ’scales up without the website breaking down’. The second challenge is finding the right syntax to get the data out of the database in a similarly efficient way such that users don’t wait for 5 minutes every time they want to view a page on the site.
Game Data Storage and Representation
For the most part, representing the game engine data in the database is a fairly simple task - you store information about the various entities you want to represent on the website, and also store the relationships between them - easy stuff, and databases are great at handling this one-to-many and (to a degree) many-to-many style relationships (e.g. a game has one or more publishers or a game may have one or more names).
For FMG’s games engine, representing the data becomes a little more tricky. The relationships are not as simple as one-to-many or even a simple many-to-many that can be decomposed. FMG’s relationships are many-to-many, infinitely hierarchical, and span multiple cross-linked hierarchies to boot (i.e. data could is related to many other pieces of data in an ever growing hierarchy of inter-related hierarchies!! Does your brain hurt yet?! Mine does and I’m a programmer!).
Databases are not good at storing this sort of inter-related hierarchical information - and its harder still to find efficient ways to extract this information in a performant way! SQL is not really the right tool for the job….still, it’s the only tool that will have to do the job at the moment (that I have access to). Good job I picked the easiest board games website idea to build first then!
So, most of the previous week was spent trying to find an efficient way to get game engine data in and out of the database. It’s been a complex mess of scribbling, hacking, and database design that really challenged the way I’ve previously thought about databases and their usage. I’ve been forced to upgrade my knowledge of databases and learn some advanced database programming techniques (not a bad thing!) to find out the most sensible way to represent this data.
A Guru in Shining Armour
For the most complex part I hit a brick wall. I figured out some of the most obvious stuff, but just could not think of a way to represent the complex dependencies and relationships without some massive combinatorial explosion in the database. In the end I took a punt - I emailed Joe Celko - a highly respected and world renowned database programming guru, to see if he could shed some light. I didn’t think this last desperate act of madness would work so didn’t expect a replay…I imagine Joe gets bombarded with this sort of annoying cry for help all the time…
He replied.
By the time I sent my message here in NZ, went to bed, then let the world wake up in the US, Joe had replied! He came back with some great and helpful feedback, all for free. I was stunned - not so much to the content of the response, more that I got any response! You’ve got to respect someone who’s at the top of their game in their work, consults to Fortune 500s, and has written many books amongst other things, to help out a fellow techie in need, and quickly too. You could put it down to the challenge of the complexity, or just that Joe’s a real nice guy that loves his work and likes to help out people where he can - either way, respect to you, Mr Celko!
Promising Directions
Joe gave me some interesting directions to follow, many that again led me to realise that my database (MySQL) wasn’t easily going to be up to the job. After much more research and banging my head against a few walls, I did however come up with a semi-viable strategy that seems like it would work.
Of the massive complex dependencies the games engine has to manage - I decided that the combinatorial explosion of data didn’t have to be quite so explosive. After all, the fundamentals of the game engine are based on a set of rules that I’ve constructed, and rules which I can restrict just as easily.
In the end, I decided to implement some constraints in the engine - you’ll see more what I mean when you finally get to use the site, but for now, these restrictions and limitations of the games engine don’t make the site particularly unusable, and the sacrifice means hugely increased performance over how I was initially thinking of representing that part of the database.
The games engine is not complete by a long shot, but I’ve bitten off a huge chunk of pie and devoured it whole so I’m looking very positive for the remainder.
The Front-End
The last seven days have been spent mostly working on some of the front-end data entry pages to the games engine. Working on these now has helped me solidify some of the more complex rules and error conditions that need to be embedded into the engine before data goes into the database. Having these screens built and functional also gives me a much more effective way of testing the database representations directly without having to fumble away trying to manually build all of the complex relationships in the database whenever I need to run a few tests.
I do have some new screenshots - but can’t release these yet until I get a little more of the core development finished.
Where Next?
I estimate the games engine is about 50% built now with the main headache mostly gone. It still needs quite a bit of testing especially with lots of data in the database to simulate real usage. Hopefully this is where the beta-testers will be able to help out later too.
Next steps are to built the email notification part of the games engine, and some latter-engine-pages before this module can be marked complete - a major achievement that I’m very much looking forward to!
Oh yes, back in the real world, I’ve just started a new online business venture with a few old- and new- work colleagues, so my ’spare time’ (which was never spare anyway) is looking even shorter as it ever was whilst I wind down my old web business - good job I’m passionate about board games, and about getting this site up an running as soon as I can!
Thanks again for reading - hopefully it made some sort of sense…?
Feel free to post a comment below.
-Dan
Comments (1)
May 1st, 2008 at 8:41 pm
Sounds intriguing. I look forward to seeing this complex web of hierarchies and how well you manage to populate the data!
Add Your Comment