I’ve finally found my gSpot. Not only as a developer, designer and passionate OpenSource advocate, but as a human too. I know my limits, and as a full time partner at Laulima, father and husband, I’ve also come to accept that I am incapable of giving back to the community as much as I would like. I was once a self-proclaimed BuddyPress fanatic and had grand plans for gPress, which was my first foray into providing freely distributed OpenSource geo-location tools. Prior to that, I had worked with a “premium” theme company creating commercial geo-location themes, and was originally brought on at Laulima to provide additional geo-location services to TravelBlog.

Unfortunately, with all that time previously spent in Google Maps and jQuery, I had never spent the time creating a proper plugin. I would usually cobble things together with some copy, pasting and repeated tweaking. I’d also started introducing marker clustering and customized HTML / CSS3 InfoWindows through a collection of other external odds and ends that was quickly growing into a complete unmanageable mess. That stops here!

I’ve now developed a jQuery plugin called gSpot, from which all future browser-based geo-location work I do will be extended. I’ve also embraced GitHub, from where I will be distributing all future OpenSource work that I personally develop. Speaking of which, I’ve also managed to convince Laulima to start using GitHub, from where we will be distributing mongoBase.

For those that don’t already know, Laulima is developing it’s own OpenSource CMS called MongoPress, which is a mongoDB powered platform providing high-performance NoSQL web-management. The underlying CMS is freely available from MongoPress.org as a self-hosted download. Laulima are ultimately planning to commercialize MongoPress.com by launching a hosted version of the system with an integrated “verified” app-store. This fully-managed hosted phase of Laulima’s plans is currently in an invite-only state, and was pushed forward ahead of schedule specifically for the AWS Startup Challenge that we recently participated in. However, as outlined within our official roadmap, the true focus of Laulima’s current efforts are concentrating on a 0.3 release.

MongoPress 0.3 will be our biggest update to date, a complete re-write from the ground-up, and rather than simply make a rock-solid OOP compliant core strictly for MongoPress, we decided to take things one step further by developing a core foundation and set of modules that would work for any application developed for mongoDB. It’s no secret that we have fallen for mongoDB. Truth be known, we’re head-over heals, or at least I am. There’s no turning back for me and Laulima is committed to the point of organizing Malaysia’s first and only official MongoDB User Group. In fact, I’d even go as far as saying that mongoDB is partly responsible for me finding my gSpot, for it allows me to develop at such a rapid pace whilst keeping all my logic contained within a single language. Nonetheless, this development is not rapid enough for my liking, which is why I have pushed so hard for mongoBase to get such prominence within our current work-flow.

As a user of Automattic’s BackPress, from which I was just about capable of getting GeoPress launched, I quickly realized the importance of a solid framework. But unlike the mess that is BackPress, which appears to have been cobbled together by working backwards and extracting things from core, I wanted to do things right with MongoPress. It is for this reason that mongoBase has now become our top priority. MongoBase will be the foundation for MongoPress, which is why everything is being completely re-written for the 0.3 release. We want a foundation at core that has been specifically built as a foundation, which can be used for anything from a simple one page prototypes for visualizing new mongoDB based ideas, through to fully realized CMS platforms and everything in-between.

When I get an idea, which come more often than I can handle, I’d like to be able to more easily develop those ideas, which is why I am now so focused on foundations. It is with these foundations that I am also working on my most ambitious project to date, which is currently codenamed ” MongoMail ” and is my attempt to put an end to IMAP and POP3 emailing systems with a long-term vision that would take me a million more hours than I have freely available in my spare time. It’s a project so grand in scale that I would have been crazy to even think about starting it… It’s also a project that I started several weekends ago…

Replacing the antiquated technologies currently known as email may seem impossible to some, and probably would be if the first stage of that unfathomable quest did not support IMAP (the underlying technology central to gMail, Microsoft Exchange and other such platforms). It would after all be illogical to assume you could completely replace the most used form of online communication with another form of communication that did not support the current form, which is why I am a man on a mission with a million and one things left to do…

So with that said and done, you might be asking where I currently stand…?

Well, I have at least made a start, without which, nothing would ever happen…

I already have a web-based PHP / mongoDB driven email client that can sync with IMAP and can be fully customized through themes and plugins. I have a default theme for this email client that revolutionizes the way we look at emails by threading conversations and auto-generating media galleries from attachments. Everything is asynchronous, AJAX powered and breathtakingly simple in the sexiest of minimal and mobile compatible ways.

We also posted a couple of images on Twitter today, where you can now get a quick glimpse of the working copy I am using from my localhost and take a peep at our re-invented inbox or how one forwards messages of the future… I say “we” because Laulima has graciously allowed me to spend a certain amount of my time continuing to develop the MongoMail project and can see the game-changing potential of this new territory. With that said, and due to Laulima’s insatiable commitment to the OpenSource community, phase one of this ambitious project will hopefully be available for download before the end of the year!

MongoPress 0.2.2 is live! – http://mongopress.org

Having made it through the 0.2 series in one piece, we are finally ready to start our most gruelling release cycle yet – the 0.3 series will see us returning to core and completely re-writing many of the core functions in an effort to drastically improve performance, stability and especially the way in which plugins and inherent caching is handled, but until then, we are hoping the 0.2.2 changes will keep you interested.

With new style-sheets for hand-held devices and administration pages, along with print-only style-sheets for themes, my favourite new set of features for this release are the advance AJAX fetchers included in core, where you need only provide elements with specific classes (currently supporting fetch-feed, fetch-avatar and fetch-content), which when also given the relevant data-attributes; MongoPress will then perform AJAX requests to collect content and pump it into the selected container. Very cool stuff indeed – and as such, we have even updated the default theme included in core to show how this works – where AJAX is now used to collect the first featured-article and display it on the homepage as seen at – http://mongopress.org

On a more personal note, I’m really happy to see we now have an AJAX-powered core-contributor / credits page in place – not because it has my name on it, but because it contains our first public contributor – Dattas – thanks again for your input!

During this release cycle, we also held our first Kuala-Lumpur MongoDB User-Group, for which we conducted a rather thorough series of benchmarks aimed specifically at re-creating the environment of a CMS system, which was recently published here on the labs – http://labs.laulima.com/mongodb-vs-mysql-performance-benchmarks-cms – this clearly shows that our efforts with MongoPress have not been in vein and that any website or web-application with more than one simultaneous user and (or) visitor is going to benefit from the performance that MongoDB is able to offer, and that’s not even considering other important native functionality such as replica-sets, sharding, GridFS and geo-location!

Over the past few years there has been a gradual rise in prominence for a new breed of databases that is quickly gaining traction. NoSQL databases offer many advantages to developers over traditional Relational SQL databases, both in terms of development flexibility and the speed and instant scalability. The question that will be forefront in the mind of any developer working on a large project is “how does it perform?”. The bottom line for many projects will be “which database will run my application the best” – we at Laulima also had these questions, and after an extensive but not exhaustive search failed to find any definitive third-party statistics for the simple tasks.

Laulima is responsible for the technology and design of two high profile projects, http://www.travelblog.org/ is a PHP MySQL based travel CMS, and http://www.mongopress.org/ is a new MongoDB (NoSQL) PHP OpenSource CMS – so our immediate interest was in measuring the performance on these two specific databases. Both MySQL and mongoDB are open-source, are also both considered to be the standard of their classes.

And so in light of hosting the first-ever Kuala-Lumpur MongoDB User-Group, we wanted to present some easy to understand, realistic benchmarks for typical tasks that are usually associated with CMS platforms and how they might construct a page or interact with their users from a database-level.

Deciding on Typical CMS Usage Scenarios

With our experience developing TravelBlog, we have seen a single page require as many as 200 separate transactions to generate a page. This is admittedly a large number, and through the use of caching we have reduced this number down to a level where visitors rarely notice a pause. However, with a need to choose a benchmarking figure we selected this as our typical worst case “social-network” style page load. Analysing the typical number of reads Vs update operations, we opted for a ratio with 10 reads Vs 1 update.

Our benchmarks have used these numbers and ratios as the basis, they are not arbitrary but may differ from other less “heavy” CMS systems – we have also open-sourced our scripts and encourage those interested to download and adjust the parameters and server set-up as required. We are well aware that in specific usage scenarios, the results found could be very different.

Breaking Down Typical CMS Operations

NoSQL operates in a different way to SQL – where some of the operations that are fundamental to SQL are not the same in NoSQL. For instance; auto-incremented IDs and the fetching of random data do not make sense from a technical perspective in a NoSQL environment, and so the operations we simulated include:

• MySQL: Select – Update – Insert – Delete
• MongoDB: Find – FindAndModify – Save – Remove

We will randomly loop-through the PHP functions, meaning that certain functions will be called more than the others but the difference is is only in the region of 5%. We have taken high-profile website operations as the example, with lots of reads and select operations because the users and visitors are rarely inserting and updating data in comparison. We conducted the benchmarks using this operation in mind and both scripts have a query to run selects based on indexed column / fields, where no index but the index field will be run more than 5 consecutive times. All selects, updates and delete queries are based on random “lorem ipsum” data collected from an external text file.

Ensuring Fairness

We wrote our scripts in PHP, the language of nearly all our projects and 75% of all production websites (http://en.wikipedia.org/wiki/PHP#Usage). We used Apache/2.2.9 (Debian), PHP Version 5.2.6-1+lenny13 from Debian Lenny package manager, MongoDB 1.8.2, MySQL 5.0+ with MongoDB driver version 1.2.4 from pecl, MySQL client API driver version 5.0.51a and Apache Bench (ab) in order to conduct our benchmarks concurrently. We stripped down our scripts to use the most commonly used drivers for each database, did not include any additional third-party frameworks, and as such error handling was also minimized. These are scripts intended purely for determining performance, and should not be used as the basis of an application.

In both cases we were not only measuring the pure performance of the database, but were also looking at the overhead from PHP and Apache, but seeing as both set-ups are identical, and both are industry standards, we feel that the impact of this is fair on the tested databases. The differences in the performance of the scripts should be a result of the database alone and so to ensure this, each benchmark run was conducted on a clean boot-up, through the localhost address (so that network issues would not impact performance) and no PHP code optimizer (eaccelerator, apc, xcache) was installed. Both databases saved their data in XFS partitions to get the best performance and no applications other than those used for the benchmarks were in use.

Our Server

Our server utilised the following specifications:

• Linux Debian Lenny Kernel 2.6.26-2-amd64
• Intel(R) Core(TM) i5 CPU 661 @ 3.33GHz
• 4GB RAM
• HDD SATA Model Number: WDC WD5000AADS-00S9B0 (500GB)

This is a relatively new desktop workstation that we dual boot into a Linux server for cases like this. Unfortunately, we did not have the resources to test on a production level server (12 core, 16gb Linux servers in most cases) – but we anticipate that the levels of performance would hold true assuming that they were configured correctly.

The Results

This first graph shows MongoDB under-performing when given only a single task to do at a time:

But when MongoDB is given simultaneous connections and tasks, thinks become interesting:

Concluding Thoughts

The full logs for these benchmarks can be seen at – http://www.mongopress.org/presentations/benchmarks/benchmark_results_001.txt

So forgetting about other MongoDB golden-nuggets, such as Replica-Sets, Sharding, GridFS and Geo-Location, not to mention the inherently relevant way of storing data as documents whilst providing developers with more native ways to easily extend the database and in-turn allow for more rapid development; MongoDB clearly outperforms MySQL at even the simplest of tasks when given simultaneous connectivity, which is something every web-site and or web-based application should strive to need.

You may also be interested in How To: Import 7 Million Locations from GeoNames to MySQL, then into MongoDB so they can be compared and optimized for Location Queries.

Version 0.2.1 features a critical fix for AJAX powered admin-pages, as well as a whole bunch of other minor improvements as listed in the following change-set – http://trac.mongopress.org/trac/query?status=closed&milestone=0.2.1. We’ve also released two public themes – http://svn.mongopress.org/themes/ – mongotope (live demo) and mongollery (live demo). With the first Kuala Lumpur MongoDB User Group now over, we can also provide links to a couple of MongoPress-powered presentations (that use another theme we will soon be releasing). This includes Introducing MongoDB and Introducing MongoPress.

Our focus for the next release will be tying off a few loose ends prior to starting on our next major release. In 0.3 we plan to radically re-write most of core – specifically plugins and the way that web-pages are created from objects – 0.3 is going to be a solid bone-crunching release that will surely cement our platform with the necessary foundations for our future.

We hosted KL MUG #1 (our first MongoDB user-group) last night, which was generously sponsored by 10gen, the company behind MongoDB. Not only was it our first user-group, it was also the first time our company have hosted any form of public gathering – and to make things even more nerve-racking, it was also the first time I planned to give a presentation.

Introducing MongoDB & Why I Love it!

The presentation featured above is powered by MongoPress and will soon be added to our public repository of themes, along with mongotope and mongollery – which are already available to download via SVN – http://svn.mongopress.org/themes/

Getting back to the topic at hand – I was somewhat surprised that anyone made it – especially with it being the first one – so really, thanks ever so much for coming and showing so much excitement for the coming months – for all those skeptics out there – the proof is in the pudding – so here goes:

We spent the first hour sitting in a circle getting to know each other and eating some damn fine pizza, after which, I presented a quick introduction to MongoDB which focused on the reasons why I (as a front-end designer / developer who should hate all things administrative and databasey) love it so much – Ali then took centre stage and provided a live hands-on session where he set-up an instance on Amazon EC2 and got the latest mongo drivers installed and ready to rock. We then discussed potential topics for next month, which included an interest in replica-sets and sharding, NodeJS and MongoDB as well as streaming video and audio with GridFS.

Please do join the Meet-Up group – http://www.meetup.com/KL-MongoDB-User-Group/, which is the preferred method that 10gen have asked us to organise things through – so until next time; may the force be with you! : -)

Well folks, it’s been a real interesting and somewhat intense couple of weeks…

Around the same time as winning the monthly developer contest at 10gen, I started work on a new administration interface for MongoPress whilst Ali set himself on the task of re-writing the installation process and better organising the general flow of perma-trail interactions with super-charged private nonces and cookie key protection of sensitive data. As a result, we’ve hit all new levels of stability and security. We’ve got a fresh look and feel that utilises HTML5 in new and interesting ways with drag and drop batch image uploading and AJAX powered page transitions with push states and history. Our installation process is as simple as it gets – boasting an all new “One Step Process” with inline importing of content. The default theme has been given a new object type called “places“, which have been used for us to test the new geoRSS mu-plugin presently added to core, whilst also allowing us to showcase the new gMap shortcode functionality for displaying places on a map. With all this and countless other changes and bug-fixes, we’re proud to announce our first major milestone since launch. Our first “production ready” release. Our baby; MongoPress 0.2

For those still unfortunate enough to not have an environment to install MongoDB and check-out MongoPress, we wanted to show you a couple of teasers from the administrative pages, where the first image shows the dashboard and the second shows drag and drop batch image uploading.

In addition to this, there are now several websites out there also using MongoPress.

• http://smalley.my/
• http://www.libertyresourcing.co.uk/

It’s been just over two weeks since we first launched MongoPress 0.1 – since then; we’ve had over 1,000 downloads, which considering the niche server requirements needed to run MongoPress; is impressive. In the city of Shenzhen, we’ve even been ranked in the top 20,000 most visited websites!

Since the initial buzz of the release v0.1 we’ve concentrated on making MongoPress even more stable and easier to install. Valuable feedback from Mitch, means that all PHP notice and warning messages have been fixed. v0.1.2 is a big step up from our first release, we have successful installs on Windows 7 with XAMP, Debian Squeeze with Apache and Nginx.

Some highlights from the 40+ tickets we closed during the 0.1.1 and 0.1.2 development cycles includes:

• Improved Plugin Functionality – but even still, this is our weakest area and will be vastly improved upon by 0.2
• Improved Site Options – can now change the site name, description and cookie expiration rate from the administrative pages
• Improved Core Mu-Plugins – download counter improvements and NEW contact form plugin added – these will be removed from core by 0.2
• Working Enque Methods for Theme and Admin pages (in headers and footers)
• Attribute Filters for all “a” and “img” tags – useful for SEO, etc
• Fully Working Administrative Themes (see themes/default/admin/_README.txt)
• Geo-Location Functionality at Core (add lng lat via custom fields then query objects based on location)
• Short-Code Functionality for Adding Images and Downloads from GridFS
• Pretty Media Slugs (such as http://yourdomain/media/image.jpg linked to GridFS)
• Improved HTML5 Structure
• Improved Language Detection
• Improved Stability – now developed in an ERROR_ALL environment

MongoPress needs your help. We need people from all around the world to know that there is a NoSQL CMS that is powered by three of the most stable frameworks available in their fields. MongoDB, PHP and jQuery – by using this as our base, and derivations of several WordPress APIs – we have a platform that will be easy for developers to understand and manipulate, where we are allowing all facets of the installation to be customised and themed, from the administrative pages, structure and content; everything can be themed, translated and customised through hooks and filters and vast selection of unique functions only available with MongoDB; such as instant replication, load-balancing, distributed media storage and serving capabilities; right through to fully integrated geo-location formatting and querying. We have a LONG way left to go, especially in refining our administrative UX, but since this can already be fully themed, we employ you to do a better job for us! In the meantime, we want to continue focusing on unique features, and have a whole host left up our sleeves…

Let us know about your successful installations – on Mac OS, on WinXP on Android phones – we’re excited to know where MongoPress is being used or tested.

Thanks again for all of your support thus far!

• Download MongoPress
• Follow MongoPress on Twitter
• Like MongoPress on Facebook

Sure enough; with MongoDB, we were able to perform the exact same query with the exact same data in only 0.02 seconds – over 8,000 times faster than MySQL, so with a love for WordPress and an instant obsession with MongoDB, I was quick to register the mongopress domain. What I assumed would be a relatively easy task to build a plugin that could connect WordPress and MongoDB, quickly grew into something of epic proportions, and after digging deeper into both WordPress and MongoDB, I realized that WordPress had been built from the ground-up with SQL and a grand table-schema in mind. It did not take long for the penny to drop, and my mind started working over-time on the possibilities of a CMS that was built from the ground-up to best utilise MongoDB.

With that in mind, we would like to introduce you to MongoPress; where we have taken some of the best features from WordPress, and combined them into our our own lightweight MongoDB-based framework. With an emphasis on security, speed and limitless flexibility; MongoPress has the essential features needed for managing content on the web. We have tried to make as few assumptions as possible, allowing for a truly customised experience. We have tried and will continue trying to only include the absolute essentials in core, and if it could be added as a plugin, it will – from RSS and Comments to more traditional analytics and sitemaps; these will all be optional plugins. Instead, we will be concentrating on core MongoDB capabilities, such as out-of-the-box scaling through support for replica-sets and sharding, as well as geo-location features and GridFS support for media.

What is MongoPress presently NOT…?

MongoPress is NOT recommended for production use and has only be released early as a technology preview.

MongoPress is NOT a fully-loaded all-frills CMS and is lacking a lot of features people have come to expect from WordPress, but as a result, it is also much more flexible and can be easily customised with design, content and administrative templates.

So what can we do with MongoPress 0.1 …?

MongoPress 0.1 includes the following working features:

• Lightening Fast Reflexes
• Local / Remote MongoDB Support with Replica-Sets and Sharding
• Really Simple Installation Wizards / Process
• Powerful Perma-Trail Functionality
• WordPress Plugin API (hooks and filters)
• WordPress Translation Functionality
• Non-GUI-Based Theme Management
• Administrative Access to MongoDB Objects
• Additional Publication View Mode for Editing Objects
• SHA(256) SALTED Security and Authentication
• HTML Importation / Content Template Functionality
• GridFS Media Support for Images and ZIPs (tracking downloads and views)

What did not make it into the initial release…?

The following items did not make it into our 0.1 initial release but are presently scheduled for 0.2:

• Core GUI for Theme / Plugin Management
• Ability to Develop Themes in HTML
• Core Geo-Location Support
• HTML5 Audio / Video Support for Default Theme

What plugins have we already built…?

The following list are included mu-plugins that are in extremely crude states, but demonstrate existing flexibility that allows the bravest of souls to run in a production environment, as we are for MongoPress itself:

• Download Counter (with GridFS support)
• Default Header (with GridFS support)
• Analytics Plugin (for Google Analytics)

What plugins are at the top of our list…?

The following list are plugins we plan to develop if not already in development when required:

• Contact Form Plugin
• Comment Plugin
• RSS Plugin
• Sitemap Plugin

How can you help…?

If you think that we might be on to something and want to help-out, please get in touch. If you simply want to show support, let people know about us. Write a post or send a tweet, shout it from the roof-tops if you can. We need to hear from you guys. We need to know that this is something you want, and so the more we hear from you, the better our ability becomes in responding to your needs.

Follow us on Twitter – at @mongopress

Download our nightly-build via SVN – at http://svn.mongopress.org/trunk/

You can keep track of bugs and up-coming features via MongoTrac – at http://trac.mongopress.org

So, like everyone else who’s been paying any kind of attention to recent database technology trends, I’d heard about this No-SQL stuff, heard it was fast, and far more approachable to your average developer due to the JSON based formatting, but had never had the time to sit down and really investigate things. And then I heard about MongoDB, specifically about their superb geo-location functions – http://www.mongodb.org/display/DOCS/Geospatial+Indexing – and as you may know already, I’m a bit of a geo-location nut-case, so was instantly hooked when I also heard that Foursquare was using MongoDB too.

So where to start…? We first needed some geo-data, and where better to get that than from GeoNames – where you can download the allCountries.zip file (containing 7 Million+ records at over 200 MB), or simply grab the country-specific information for the country you reside.

We will need to get the data into MySQL, not only so that we can compare the speeds for proximity searches, but also so that we can learn a valuable lesson on how to export from MySQL into MongoDB, which really is both flexible and easy.

If we plan to use the GeoNames data, we MUST first create a table using the following SQL (with the field names being specifically important for the next stage):

CREATE TABLE `geonames` (
    `geonameid` int(10) unsigned NOT NULL default '0',
    `name` varchar(200) NOT NULL default '',
    `ansiname` varchar(200) NOT NULL default '',
    `alternatenames` varchar(2000) NOT NULL default '',
    `latitude` double NOT NULL default '0',
    `longitude` double NOT NULL default '0',
    `feature_class` char(1) ,
    `feature_code` varchar(10) ,
    `country_code` char(2),
    `cc2` varchar(60),
    `admin1_code` varchar(20) default '',
    `admin2_code` varchar(80) default '',
    `admin3_code` varchar(20) default '',
    `admin4_code` varchar(20) default '',
    `population` bigint(11) default '0',
    `elevation` int(11) default '0',
    `gtopo30` int(11) default '0',
    `timezone` varchar(40),
    `modification_date` date default '0000-00-00',
    PRIMARY KEY (`geonameid`)
) CHARACTER SET utf8 ;

With the geonames table created, we can import the data contained within the TXT file we just downloaded from GeoNames by using SQL similar to the following (where you will need to replace the location of the TXT file to the location you just extracted it to from the ZIP file you downloaded):

LOAD DATA INFILE 'C:/xampp/htdocs/allCountries.txt' INTO TABLE geonames(
    geonameid, name, ansiname, alternatenames, latitude, longitude, feature_class, feature_code, country_code, cc2, admin1_code, admin2_code, admin3_code, admin4_code, population, elevation, gtopo30, timezone, modification_date
)

We should now have over 7 million places in our database (assuming you were brave enough to use the allCountries.txt file, which to be honest, may be too big for the next stages and cause you several problems like it did when running it on my personal PC at home). Once you have the records in your MySQL database, do not forget to add an index to the “latitude” and “longitude” fields in order to give the tests a fair chance later.

At this stage, we should probably start thinking about getting MongoDB set-up, and fortunately, the online documentation and general community-spirit for MongoDB is a jolly-fine one at that. For the purpose of this comparison, we will be using MongoDB and PHP to test the results, but in a future “How To”, I also hope to document my findings for NodeJS too!

Let’s download MongoDB – http://www.mongodb.org/downloads – then run-through the getting-started guides – http://www.mongodb.org/display/DOCS/Quickstart. I found set-up on the PC and Mac equally as easy. Simply download the package, save somewhere sensible, create the necessary “data” folder, then run the two recommended console executables “mongod.exe” and “mongo.exe” and if 3+3 equals 6 everything’s working properly! For the benefit of getting to the meat and bones of this How To, I’m going to assume that setting-up MongoDB was a breeze and that you were then capable of also copying the correct .dll (for windows) or .so (for Mac) into the correct “extension” folder as specified by your php.ini file and documented here – PHP Drivers for Mongo.

We are now ready to import the MySQL data into MongoDB, and although the method I am about to show you is not the preferred method for large data-sets (which should use the import function), showing you this way teaches you the most fundamental MongoDB functions, which includes adding new collections (tables) and objects (records), but also updating those objects and then retrieving the objects, all from PHP in syntax we as PHP developers should be very comfortable with. In fact, the following code provides everything you need in order to not only import the data from MySQL to MongoDB, but also to run comparative tests to see how fast each of the databases can run and return the same results.

Before you begin, please note that importing all 7 Million records using the methods below from MySQL to MongoDB is going to take some time, several hours, if not all day (and then some, especially using these methods), so if you are doing this as a way to get started and want some immediate feedback, you REALLY should consider importing one of the country specific .txt files from GeoNames, or using the country_code and (or) limit variables to migrate only a portion of the whole MySQL data into MongoDB.

Either way, feel free to use, improve and comment on the following code as required (as a reminder, this is not the best way to import large data-sets to MongoDB but have done it this way so that it’s easy to understand the fundamentals):

// We need this function for running stop-watch later...
function ms_microtime_float(){
    list($usec, $sec) = explode(" ", microtime());
    return ((float)$usec + (float)$sec);
}

// We need this function for making the most optimised MySQL proximity query available...
// Simply provide a database name, your current Lat / Lng and optional DB Limits and Maximum Distances...
function ms_get_locations_by_distance_using_sql($database_name='geonames',$lat,$lng,$limit=100,$distance=100,$country_specific=false){
    if($country_specific){ $extra_where_clause = "AND country_code = 'MY'"; }else{ $extra_where_clause=''; }
    if($limit>0){ $limit = 'LIMIT 0, '.$limit; }else{ $limit=NULL; }
    $lng1 = $lng - $distance / abs(cos(deg2rad( $lat ) ) *69 );
    $lng2 = $lng + $distance / abs(cos(deg2rad( $lat ) ) *69 );
    $lat1 = $lat - ( $distance /69 );
    $lat2 = $lat + ( $distance /69 );
    /* THIS REALLY IS THE BEST WAY TO PERFORM PROXIMITY SEARCH IN MYSQL */
    $nearest_locations_ordered_by_distance = "SELECT *,
        ( 6371 * acos( cos( radians( $lat ) ) * cos( radians( latitude ) ) * cos( radians( longitude ) - radians( $lng ) ) + sin( radians( $lat ) ) * sin( radians( latitude ) ) ) ) AS distance
        FROM $database_name
        WHERE longitude between $lng1 and $lng2
        AND latitude between $lat1 and $lat2
        $extra_where_clause
        HAVING distance < $distance
        ORDER BY distance
        $limit";
    return $nearest_locations_ordered_by_distance;
}

/*
 *  THESE ARE OUR OPTIONS
 *
 *  In order to initially import data from MySQL to MongoDO use:
 *
 *  $collect_mongo_results = false;
 *  $loop_through_sql = true; // SET TO FALSE IF ONLY INTERESTED IN MONGO RESULTS
 *  $add_to_mongo = true;
 *  $print_arrays = false;
 *
 *  In order to compare times for collecting results in both MySQL and MongoDB use:
 *
 *  $collect_mongo_results = true;
 *  $loop_through_sql = true; // SET TO FALSE IF ONLY INTERESTED IN MONGO RESULTS
 *  $add_to_mongo = false;
 *  $print_arrays = true;
 *
*/
$collect_mongo_results = false;
$loop_through_sql = true;
$add_to_mongo = true;
$print_arrays = false; // Be careful, with 7 million results, this can take time
$limit = -1; // Set this to -1 if you wish to import everything from MySQL to MongoDB else add limit as number
$maximum_distance = 100; // This represents the maximum distance in KM for proximity search
$database_name = 'geonames'; // This will also be used for the MongoDB name
$table_name = 'countries'; // This will also be used as the MongoDB collection name
$mysql_host = 'localhost';
$mysql_username = 'root';
$mysql_password = '';
// These default Lat / Lng Coordinates are for Malaysia
$current_latitude = 3.190908; // Do not use a string!
$current_longitude = 101.682243; // Do not use a string!
// This country code is for Malaysia
// Change to false if wanting to run queries on all records...
$country_code = false; // This is used to perform sub-set queries on specific countries...
/* END OF OPTIONS */

// Connect to Mongo
$m = new Mongo();
// Select / Create a Database
$db = $m->$database_name;
// Select / Create a Collection (table)
$collection = $db->$table_name;

/* THIS ALLOWS SCRIPTS TO RUN WITHOUT TIMING OUT */
set_time_limit(0);

if($collect_mongo_results){

    /* START TIMING MONGO */
    $time_start = ms_microtime_float();

    // Start building MongoDB query
    /* PAY EXTRA ATTENTION TO ORDER OF LONGITUDE THEN LATITUDE */
    /* THIS IS VERY IMPORTANT FOR MONGODB - OTHERWISE RESULTS WILL BE WRONG !!! */
    $this_latlng = array('lng'=>$current_longitude,'lat'=>$current_latitude);

    $mongo_near_query = array('geoNear'=>$table_name,'near'=>$this_latlng,'$spherical'=>true,'$maxDistance'=>$maximum_distance/6378,'num'=>$limit);
    $cursor = $db->command($mongo_near_query);

    /* THE SAME QUERY CAN BE RUN IN MONGO CONSOLE USING THE FOLLOWING */
    //-> db.runCommand({geoNear:"countries",near:[101,3],spherical:true,maxDistance:100/6378,num:10})

    if($print_arrays){
        // iterate through the results
        foreach ($cursor['results'] as $obj) {
            echo '<br />This is MongoDB Array:<br />';
            echo '<pre>';
            print_r($obj);
            echo '</pre>';
        }
    }
    /* END MONGO TIMER AND DISPLAY RESULTS */
    $time_end = ms_microtime_float();
    $time = $time_end - $time_start;
    $mongo_results = "<br />Collected and Printed using MongoDB in $time seconds<br />";
    echo $mongo_results;

}

if($loop_through_sql){

    /* START SQL TIMER */
    $time_start = ms_microtime_float();

    // Connect to MySQL
    $link = mysql_connect($mysql_host, $mysql_username, $mysql_password);
    if (!$link) {
        die('Could not connect: ' . mysql_error());
    }

    // Connect to
    if (!mysql_select_db($database_name)) {
        die('Could not select database: ' . mysql_error());
    }

    /* WHICH QUERY TO RUN...? DEPENDS ON OPTIONS ABOVE */
    if($add_to_mongo){
        if($country_code){ $where_clause = "WHERE country_code = 'MY'"; }
        $query = 'SELECT * FROM '.$table_name.$where_clause;
    }else{
        $query = ms_get_locations_by_distance_using_sql($table_name,$current_latitude,$current_longitude,$limit,$maximum_distance,$country_code);
    }

    // Perform Query
    $result = mysql_query($query);

    // Check result
    // This shows the actual query sent to MySQL, and the error. Useful for debugging.
    if (!$result) {
        $message  = 'Invalid query: ' . mysql_error() . "\n";
        $message .= 'Whole query: ' . $query;
        die($message);
    }

    // Use result
    // Attempting to print $result won't allow access to information in the resource
    // One of the mysql result functions must be used
    // See also mysql_result(), mysql_fetch_array(), mysql_fetch_row(), etc.
    while ($row = mysql_fetch_assoc($result)) {
        if($print_arrays){
            echo '<br />This is MySQL Array:<br />';
            echo '<pre>';
            print_r($row);
            echo '</pre>';
        }
        /* END SQL TIMER */
        $time_end = ms_microtime_float();
        $time = $time_end - $time_start;

        /* NOW NEED TO ADD DATA TO MONGO TOO */
        if($add_to_mongo){
            $obj = array();
            foreach($row as $key => $value){
                $obj[$key] = $value;
            }
            // This is the MongoDB PHP Function for inserting objects (records)
            $collection->insert($obj);
        }
        // In order to use MongoDB Geo-Spatial Indexing we need a Lng / Lat field
        // This shows us how to use the MongoDB PHP Update function
        $collection->update(array("geonameid" => $row['geonameid']), array('$set' => array("latlng" => array('lng'=>(float)$row['longitude'],'lat'=>(float)$row['latitude']))));
        // Create Geo-Spaital Index In Mongo Console (once this field has been added) using:
        // db.countries.ensureIndex({latlng:'2d'})
    }

    /* THEN PRINT RESULTS */
    echo "<br />Collected and Printed using MySQL in $time seconds<br />";
    if($collect_mongo_results){
        echo "WHERE AS ".$mongo_results;
    }

    // Free the resources associated with the result set
    // This is done automatically at the end of the script
    mysql_free_result($result);
    mysql_close($link);

}

Several things to remember before running or after checking the results of the comparisons is whether you remembered to added the MySQL indexes to your latitude and longitudes, not to mention whether you added the geo-spatial index to your MongoDB, to do which, you will need to use the following command from within the Mongo Console (geeky I know – there are some GUIs, but this is much easier than it looks):

db.countries.ensureIndex({latlng:'2d'})

Some other simple yet useful commands for the console include dropping databases:

db.geonames.drop()

By default, opening the Mongo Console launches the “test” database, so to switch from that to geonames is as simple as typing:

use geonames

In conclusion, MongoDB is FAST. In searching through 65,411 records / objects (as I myself only imported the places for Malaysia in order to save time and get to the learning part), and displaying only those within a 100 KM radius, and then having those results limited to 10, ordered by distance with the closest first, and echoed out onto the page took 0.23555 seconds for MySQL, where as that very same query providing the very same results took only 0.00152 seconds with MongoDB – and on several occasions, it even came in around the 0.00095 mark! And to think that that’s with just one user accessing the query. The biggest problem with MySQL and the reason I initially sought alternatives was the fact that only one person can access the database at one time, where concurrent queries get added to a queue and need to wait their turn. With MongoDB, this is not the case, so really, truth be known, there is no comparison between them! But we will be running some concurrent tests soon, as well as running the same queries using the complete 7 Million record set…

But how do we import all 7 Million+ records and perform tests on those…?

By NOT using the methods above! I tried several times and several different ways to import all 7 million records, and the quickest most reliable method I could find was to first export the MySQL as CSV. I tried several times to use a mysqldump from console, but could never get the formatting right. Believe it or not, the most Mongo-Friendly method was to export from phpMyAdmin in CSV for MS Excel format (with the first row being field names) as this was able to then utilize the mongoimport function. However, whenever I tried to export the entire set, I ran into problems. What I ended-up doing was exporting a million records at a time as CSV.

Once you have all 8 CSV files, you can then import them into MongoDB using the following console commands:

mongoimport --db geonames --collection countries --type csv --file geonames_part1.csv --headerline --upsert

Do this for each part, and remember this must be done from a standard terminal console, not from the MongoDB console, which was my first mistake! After we have all 7 Million plus records into the database, we are going to need to add in the latlng / loc field, which does not yet exist and is needed by MongoDB for the geo-spatial indexing. To do this, the quickest method I found was to use a JS query. To do that, you will first need to create a new JavaScript with contents such as:

db.countries.find().forEach(function(data) {
    db.countries.update({_id:data._id},{$set:{loc:{lng:parseFloat(data.longitude),lat:parseFloat(data.latitude)}}});
})

We then need to query the MongoDB using that JS file, which is perhaps my most favourite MongoDB feature and can be performed from a regular terminal (not the MongoDB console) using syntax such as:

mongo geonames add_latlng.js

Once we have that in place, we can add the Geo-Spatial Index (from the Mongo Console) by typing:

db.countries.ensureIndex({loc:"2d"})

The results are phenomenal:

Querying 7.8 Million records using MySQL and finding the 10 nearest locations ordered by distance took an average of 165 seconds!

With MongoDB, that same query took only 0.02 seconds!

Other than that, what’s next…?

MongoPress – for it is this that started the future of NoSQL CMS : -)

You may also be interested in MongoDB Vs MySQL – Benchmarks Re-Creating Typical CMS Functionality to Compare Performance of MySQL and MongoDB in PHP / Apache.