Lau-Labs

I’ve finally found my gSpot. Not only as a developer, designer and passionate OpenSource advocate, but as a human too. I know my limits, and as a full time partner at Laulima, father and husband, I’ve also come to accept that I am incapable of giving back to the community as much as I would like. I was once a self-proclaimed BuddyPress fanatic and had grand plans for gPress, which was my first foray into providing freely distributed OpenSource geo-location tools. Prior to that, I had worked with a “premium” theme company creating commercial geo-location themes, and was originally brought on at Laulima to provide additional geo-location services to TravelBlog.

Unfortunately, with all that time previously spent in Google Maps and jQuery, I had never spent the time creating a proper plugin. I would usually cobble things together with some copy, pasting and repeated tweaking. I’d also started introducing marker clustering and customized HTML / CSS3 InfoWindows through a collection of other external odds and ends that was quickly growing into a complete unmanageable mess. That stops here!

I’ve now developed a jQuery plugin called gSpot, from which all future browser-based geo-location work I do will be extended. I’ve also embraced GitHub, from where I will be distributing all future OpenSource work that I personally develop. Speaking of which, I’ve also managed to convince Laulima to start using GitHub, from where we will be distributing mongoBase.

For those that don’t already know, Laulima is developing it’s own OpenSource CMS called MongoPress, which is a mongoDB powered platform providing high-performance NoSQL web-management. The underlying CMS is freely available from MongoPress.org as a self-hosted download. Laulima are ultimately planning to commercialize MongoPress.com by launching a hosted version of the system with an integrated “verified” app-store. This fully-managed hosted phase of Laulima’s plans is currently in an invite-only state, and was pushed forward ahead of schedule specifically for the AWS Startup Challenge that we recently participated in. However, as outlined within our official roadmap, the true focus of Laulima’s current efforts are concentrating on a 0.3 release.

MongoPress 0.3 will be our biggest update to date, a complete re-write from the ground-up, and rather than simply make a rock-solid OOP compliant core strictly for MongoPress, we decided to take things one step further by developing a core foundation and set of modules that would work for any application developed for mongoDB. It’s no secret that we have fallen for mongoDB. Truth be known, we’re head-over heals, or at least I am. There’s no turning back for me and Laulima is committed to the point of organizing Malaysia’s first and only official MongoDB User Group. In fact, I’d even go as far as saying that mongoDB is partly responsible for me finding my gSpot, for it allows me to develop at such a rapid pace whilst keeping all my logic contained within a single language. Nonetheless, this development is not rapid enough for my liking, which is why I have pushed so hard for mongoBase to get such prominence within our current work-flow.

As a user of Automattic’s BackPress, from which I was just about capable of getting GeoPress launched, I quickly realized the importance of a solid framework. But unlike the mess that is BackPress, which appears to have been cobbled together by working backwards and extracting things from core, I wanted to do things right with MongoPress. It is for this reason that mongoBase has now become our top priority. MongoBase will be the foundation for MongoPress, which is why everything is being completely re-written for the 0.3 release. We want a foundation at core that has been specifically built as a foundation, which can be used for anything from a simple one page prototypes for visualizing new mongoDB based ideas, through to fully realized CMS platforms and everything in-between.

When I get an idea, which come more often than I can handle, I’d like to be able to more easily develop those ideas, which is why I am now so focused on foundations. It is with these foundations that I am also working on my most ambitious project to date, which is currently codenamed ” MongoMail ” and is my attempt to put an end to IMAP and POP3 emailing systems with a long-term vision that would take me a million more hours than I have freely available in my spare time. It’s a project so grand in scale that I would have been crazy to even think about starting it… It’s also a project that I started several weekends ago…

Replacing the antiquated technologies currently known as email may seem impossible to some, and probably would be if the first stage of that unfathomable quest did not support IMAP (the underlying technology central to gMail, Microsoft Exchange and other such platforms). It would after all be illogical to assume you could completely replace the most used form of online communication with another form of communication that did not support the current form, which is why I am a man on a mission with a million and one things left to do…

So with that said and done, you might be asking where I currently stand…?

Well, I have at least made a start, without which, nothing would ever happen…

I already have a web-based PHP / mongoDB driven email client that can sync with IMAP and can be fully customized through themes and plugins. I have a default theme for this email client that revolutionizes the way we look at emails by threading conversations and auto-generating media galleries from attachments. Everything is asynchronous, AJAX powered and breathtakingly simple in the sexiest of minimal and mobile compatible ways.

We also posted a couple of images on Twitter today, where you can now get a quick glimpse of the working copy I am using from my localhost and take a peep at our re-invented inbox or how one forwards messages of the future… I say “we” because Laulima has graciously allowed me to spend a certain amount of my time continuing to develop the MongoMail project and can see the game-changing potential of this new territory. With that said, and due to Laulima’s insatiable commitment to the OpenSource community, phase one of this ambitious project will hopefully be available for download before the end of the year!

MongoPress 0.2.2 is live! – http://mongopress.org

Having made it through the 0.2 series in one piece, we are finally ready to start our most gruelling release cycle yet – the 0.3 series will see us returning to core and completely re-writing many of the core functions in an effort to drastically improve performance, stability and especially the way in which plugins and inherent caching is handled, but until then, we are hoping the 0.2.2 changes will keep you interested.

With new style-sheets for hand-held devices and administration pages, along with print-only style-sheets for themes, my favourite new set of features for this release are the advance AJAX fetchers included in core, where you need only provide elements with specific classes (currently supporting fetch-feed, fetch-avatar and fetch-content), which when also given the relevant data-attributes; MongoPress will then perform AJAX requests to collect content and pump it into the selected container. Very cool stuff indeed – and as such, we have even updated the default theme included in core to show how this works – where AJAX is now used to collect the first featured-article and display it on the homepage as seen at – http://mongopress.org

On a more personal note, I’m really happy to see we now have an AJAX-powered core-contributor / credits page in place – not because it has my name on it, but because it contains our first public contributor – Dattas – thanks again for your input!

During this release cycle, we also held our first Kuala-Lumpur MongoDB User-Group, for which we conducted a rather thorough series of benchmarks aimed specifically at re-creating the environment of a CMS system, which was recently published here on the labs – http://labs.laulima.com/mongodb-vs-mysql-performance-benchmarks-cms – this clearly shows that our efforts with MongoPress have not been in vein and that any website or web-application with more than one simultaneous user and (or) visitor is going to benefit from the performance that MongoDB is able to offer, and that’s not even considering other important native functionality such as replica-sets, sharding, GridFS and geo-location!

Over the past few years there has been a gradual rise in prominence for a new breed of databases that is quickly gaining traction. NoSQL databases offer many advantages to developers over traditional Relational SQL databases, both in terms of development flexibility and the speed and instant scalability. The question that will be forefront in the mind of any developer working on a large project is “how does it perform?”. The bottom line for many projects will be “which database will run my application the best” – we at Laulima also had these questions, and after an extensive but not exhaustive search failed to find any definitive third-party statistics for the simple tasks.

Laulima is responsible for the technology and design of two high profile projects, http://www.travelblog.org/ is a PHP MySQL based travel CMS, and http://www.mongopress.org/ is a new MongoDB (NoSQL) PHP OpenSource CMS – so our immediate interest was in measuring the performance on these two specific databases. Both MySQL and mongoDB are open-source, are also both considered to be the standard of their classes.

And so in light of hosting the first-ever Kuala-Lumpur MongoDB User-Group, we wanted to present some easy to understand, realistic benchmarks for typical tasks that are usually associated with CMS platforms and how they might construct a page or interact with their users from a database-level.

Deciding on Typical CMS Usage Scenarios

With our experience developing TravelBlog, we have seen a single page require as many as 200 separate transactions to generate a page. This is admittedly a large number, and through the use of caching we have reduced this number down to a level where visitors rarely notice a pause. However, with a need to choose a benchmarking figure we selected this as our typical worst case “social-network” style page load. Analysing the typical number of reads Vs update operations, we opted for a ratio with 10 reads Vs 1 update.

Our benchmarks have used these numbers and ratios as the basis, they are not arbitrary but may differ from other less “heavy” CMS systems – we have also open-sourced our scripts and encourage those interested to download and adjust the parameters and server set-up as required. We are well aware that in specific usage scenarios, the results found could be very different.

Breaking Down Typical CMS Operations

NoSQL operates in a different way to SQL – where some of the operations that are fundamental to SQL are not the same in NoSQL. For instance; auto-incremented IDs and the fetching of random data do not make sense from a technical perspective in a NoSQL environment, and so the operations we simulated include:

• MySQL: Select – Update – Insert – Delete
• MongoDB: Find – FindAndModify – Save – Remove

We will randomly loop-through the PHP functions, meaning that certain functions will be called more than the others but the difference is is only in the region of 5%. We have taken high-profile website operations as the example, with lots of reads and select operations because the users and visitors are rarely inserting and updating data in comparison. We conducted the benchmarks using this operation in mind and both scripts have a query to run selects based on indexed column / fields, where no index but the index field will be run more than 5 consecutive times. All selects, updates and delete queries are based on random “lorem ipsum” data collected from an external text file.

Ensuring Fairness

We wrote our scripts in PHP, the language of nearly all our projects and 75% of all production websites (http://en.wikipedia.org/wiki/PHP#Usage). We used Apache/2.2.9 (Debian), PHP Version 5.2.6-1+lenny13 from Debian Lenny package manager, MongoDB 1.8.2, MySQL 5.0+ with MongoDB driver version 1.2.4 from pecl, MySQL client API driver version 5.0.51a and Apache Bench (ab) in order to conduct our benchmarks concurrently. We stripped down our scripts to use the most commonly used drivers for each database, did not include any additional third-party frameworks, and as such error handling was also minimized. These are scripts intended purely for determining performance, and should not be used as the basis of an application.

In both cases we were not only measuring the pure performance of the database, but were also looking at the overhead from PHP and Apache, but seeing as both set-ups are identical, and both are industry standards, we feel that the impact of this is fair on the tested databases. The differences in the performance of the scripts should be a result of the database alone and so to ensure this, each benchmark run was conducted on a clean boot-up, through the localhost address (so that network issues would not impact performance) and no PHP code optimizer (eaccelerator, apc, xcache) was installed. Both databases saved their data in XFS partitions to get the best performance and no applications other than those used for the benchmarks were in use.

Our Server

Our server utilised the following specifications:

• Linux Debian Lenny Kernel 2.6.26-2-amd64
• Intel(R) Core(TM) i5 CPU 661 @ 3.33GHz
• 4GB RAM
• HDD SATA Model Number: WDC WD5000AADS-00S9B0 (500GB)

This is a relatively new desktop workstation that we dual boot into a Linux server for cases like this. Unfortunately, we did not have the resources to test on a production level server (12 core, 16gb Linux servers in most cases) – but we anticipate that the levels of performance would hold true assuming that they were configured correctly.

The Results

This first graph shows MongoDB under-performing when given only a single task to do at a time:

But when MongoDB is given simultaneous connections and tasks, thinks become interesting:

Concluding Thoughts

The full logs for these benchmarks can be seen at – http://www.mongopress.org/presentations/benchmarks/benchmark_results_001.txt

So forgetting about other MongoDB golden-nuggets, such as Replica-Sets, Sharding, GridFS and Geo-Location, not to mention the inherently relevant way of storing data as documents whilst providing developers with more native ways to easily extend the database and in-turn allow for more rapid development; MongoDB clearly outperforms MySQL at even the simplest of tasks when given simultaneous connectivity, which is something every web-site and or web-based application should strive to need.

You may also be interested in How To: Import 7 Million Locations from GeoNames to MySQL, then into MongoDB so they can be compared and optimized for Location Queries.

Version 0.2.1 features a critical fix for AJAX powered admin-pages, as well as a whole bunch of other minor improvements as listed in the following change-set – http://trac.mongopress.org/trac/query?status=closed&milestone=0.2.1. We’ve also released two public themes – http://svn.mongopress.org/themes/ – mongotope (live demo) and mongollery (live demo). With the first Kuala Lumpur MongoDB User Group now over, we can also provide links to a couple of MongoPress-powered presentations (that use another theme we will soon be releasing). This includes Introducing MongoDB and Introducing MongoPress.

Our focus for the next release will be tying off a few loose ends prior to starting on our next major release. In 0.3 we plan to radically re-write most of core – specifically plugins and the way that web-pages are created from objects – 0.3 is going to be a solid bone-crunching release that will surely cement our platform with the necessary foundations for our future.

We hosted KL MUG #1 (our first MongoDB user-group) last night, which was generously sponsored by 10gen, the company behind MongoDB. Not only was it our first user-group, it was also the first time our company have hosted any form of public gathering – and to make things even more nerve-racking, it was also the first time I planned to give a presentation.

Introducing MongoDB & Why I Love it!

The presentation featured above is powered by MongoPress and will soon be added to our public repository of themes, along with mongotope and mongollery – which are already available to download via SVN – http://svn.mongopress.org/themes/

Getting back to the topic at hand – I was somewhat surprised that anyone made it – especially with it being the first one – so really, thanks ever so much for coming and showing so much excitement for the coming months – for all those skeptics out there – the proof is in the pudding – so here goes:

We spent the first hour sitting in a circle getting to know each other and eating some damn fine pizza, after which, I presented a quick introduction to MongoDB which focused on the reasons why I (as a front-end designer / developer who should hate all things administrative and databasey) love it so much – Ali then took centre stage and provided a live hands-on session where he set-up an instance on Amazon EC2 and got the latest mongo drivers installed and ready to rock. We then discussed potential topics for next month, which included an interest in replica-sets and sharding, NodeJS and MongoDB as well as streaming video and audio with GridFS.

Please do join the Meet-Up group – http://www.meetup.com/KL-MongoDB-User-Group/, which is the preferred method that 10gen have asked us to organise things through – so until next time; may the force be with you! : -)