MongoTips by John Nunemaker

Another ObjectId Trick

2012-05-30T10:38:48-04:00

A while back I posted A Few ObjectId Tricks. While not a new trick, this one is relatively new to me and quite useful.

As you know, the key _id is automatically indexed for each document in a collection. What you might not know, is that you can use objects for _id.

It seems obvious now, but I had not thought of it. A few days back, I re-read foursquare’s article on MongoDB Strategies for the Disk-Averse. They make heavy use of objects if _id in their analytics.

I work on Gauges, a web analytics tool, and I use several techniques to more efficiently store data. What I have been doing up until now is string _id’s with multiple values smushed in.

{"_id":  "<oid>:<hash>"}

oid is typically a bson object id and hash is some kind of hash on whatever I’m doing a write for. I do writes against _id and the hash determines uniqueness without storing the full url of the page we are tracking.

The crappy part is strings can’t be serialized as efficiently as an object id or an integer. The foursquare article pointed out that I could just store my _id’s like this:

{"_id":  {"i": <oid>, "h": <hash>}}

So what is the benefit of objects as _id’s? I see at least four.

1. Fewer indexes

If you were not using objects or concatenated values for _id, those values would need to be with keys inside the document. If you wanted to write against those keys, you would need to index them. This means you would have an _id that is indexed and also a secondary compound index (ie: [[i, 1], [h, 1]]).

More indexes means more writes and more RAM. Using an object for _id or a concatenated value saves this extra index and thus saves you efficiency and RAM on the server side.

2. Document Size

Object ids and integers can be serialized in bson more efficiently than strings. Switching from a string that is a mashup of object id and hash to an object can cut several bytes from each document.

require 'rubygems'
require 'bson'

oid = BSON::ObjectId.new

puts BSON.serialize(:_id => "#{oid}:123456789").size # 49
puts BSON.serialize(:_id => {:i => oid, :h => 123456789}).size #37

In the example above, the size difference was 37 verse 49 bytes. That saves almost a fourth of the document size, simply by using an object instead of a string mashup. Your mileage may vary, but applied across the millions of documents we track every month, this is a non-trivial amount of savings.

3. More Query-able (than concatenated values)

The example below shows using a range query using objects as _id, with greater than or equal to and less than or equal to. This would be painful with concatenated values in some scenarios and impossible in others.

require 'pp'
require 'rubygems'
require 'mongo'

conn = Mongo::Connection.new
db = conn.db('test')
col = db['test']
col.remove

oid = BSON::ObjectId.new

(1..3).each do |day|
  col.save(:_id => {:i => oid, :d => day})
end

# Get all documents
pp col.find.to_a

# [{"_id"=>{"i"=>BSON::ObjectId('4fc62a0f4c114f273c000001'), "d"=>1}},
#  {"_id"=>{"i"=>BSON::ObjectId('4fc62a0f4c114f273c000001'), "d"=>2}},
#  {"_id"=>{"i"=>BSON::ObjectId('4fc62a0f4c114f273c000001'), "d"=>3}}]

# Only get those matching a given day for a given oid
pp col.find({
  :_id => {
    '$gte' => {:i => oid, :d => 1},
    '$lte' => {:i => oid, :d => 1},
  },
}).to_a

# [{"_id"=>{"i"=>BSON::ObjectId('4fc62a0f4c114f273c000001'), "d"=>1}}]

We could change the $lte :d value to 2 or to 3 and those documents would then be included as well. Sure, you can query relatively the same thing using strings, but you would need to generate all the _id’s and use a $in query to pull them all out. An added bonus is that you get the documents back sorted ascending, which is what I have wanted when using objects as _id’s.

Note: if you try to execute the code above on Ruby 1.8, it will not work. Ruby 1.8 hashes are not ordered and ordering is important for _id objects. Use BSON::OrderedHash instead of a plain hash if on 1.8. The same applies to whatever language you use if you aren’t using Ruby and said language does not default to ordered hashes or dictionaries.

4. Objects are easier to grok

Values that are concatenated are not very intent revealing. With objects, however, the keys reveal what the values are.

Granted if you shorten the keys to save space, they might not reveal as much, but typically in this scenario, you create a mapping of short to long keys, which could be used to quickly deduce the key names and thus what the values are.

This point might not be a major one, but objects definitely feel cleaner and more obvious to me than concatenated values.

Lower Lock % and Number of Slow Queries

2012-01-30T12:17:05-05:00

Gauges tracks several websites. Some get a lot of traffic and others don’t. The sites that get a lot of traffic tend to stay hot and sit in RAM. The sites that get little traffic, eventually get pushed out of RAM.

This is why for Gauges, we can have 1GB of RAM on the server and over 14GB of indexes, yet Mongo hums along. Even better yet, of that 1GB of RAM, we only use around 175MB.

RAM, What Needs to Fit There

You have probably heard that MongoDB recommends keeping all data in RAM if you can. If not, they suggest at least keeping your indexes in RAM.

What I haven’t heard people say as much, is that you really just need to keep your active data or indexes in RAM. For stats, the active data is the current day or week, which is a much smaller data set than all of the data for all time.

Write Heavy

The other interesting thing about Gauges is that we are extremely write heavy, as you would expect for a stats app. Requests due to the tracking script loading on a website are over 95% of all requests to Gauges.

Some of these track requests are for sites that rarely get hit, of which, the data that needs to be updated has been pushed out of RAM and is just sitting on disk.

Global Lock is Global

As you probably know, MongoDB has a global lock. The longer your writes take, the higher your lock percentage is. Updating documents that are in RAM is super fast.

Updating documents that have been pushed to disk, first have to be read from disk, stored in memory, updated, then written back to disk. This operation is slow and happens while inside the lock.

Updating a lot of random documents that rarely get updated and have been pushed out of RAM can lead to slow writes and a high lock percentage.

MongoDB for Analytics

2015-07-13T11:39:00-04:00

Just over a month ago, I presented on storing stats in MongoDB at MongoChi 2011. 10Gen posted the video recently, so I thought I would share it here. The first 5 minutes or so are a bit rough, but it improves after that. I’ll be presenting the same thing at MongoSV next week if you want to catch it live.

I’ve also embedded the slides below.

Replacing RabbitMQ with MongoDB

2011-10-08T22:49:04-04:00

Interesting post on the Boxed Ice blog about how they are using MongoDB capped collections for a queue system.

Until today I did not realize you could index a capped collection. Not sure why I was confused about that, but now I totally understand how capped collections would work great for a queue.

Counters Everywhere, Part 2

2011-09-13T12:43:14-04:00

For those that follow me here, but not over at RailsTips, I recently released a new post on storing stats in MongoDB. The topic of this post is around storing large ranges of data.

MongoDB 2.0 Released

2011-09-13T12:37:41-04:00

Yesterday, MongoDB 2.0 was released. Personally, I am excited about the concurrency improvements and the more compact index storage. Also, for the haters, journaling is now on by default. Great release!

Counters Everywhere, Storing Stats in Mongo

2011-06-28T12:01:24-04:00

Over at RailsTips, I posted on how I use a single document to store a lot of good stuff in Gaug.es, the real-time stat app we are making at Ordered List.

Customizing Your Mongo Shell Prompt

2011-06-28T07:19:54-04:00

Sometimes I think all I do is link to Kristina, but she has written several great posts, so enjoy another.

MongoHQ Funded

2011-06-25T10:20:20-04:00

MongoHQ has raised $417K from a few investment firms to help grow the service.

Geo Spatial Indexing

2011-06-25T10:19:55-04:00

In depth post on how Mongo’s geo spatial indexing works.

A Few ObjectId Tricks

2011-03-02T10:52:19-05:00

One of the things that I was not aware of until recently is how handy MongoDB’s object ids actually are. Below are a few tips based on some things I have been doing lately.

To and From Strings

First off, it is quite simple to switch back and forth between object id and string, which is useful in JSON/XML serialization and in finding Mongo documents from params.

id = BSON::ObjectId.new
# => BSON::ObjectId('4d6e5acebcd1b3fac9000002') 
id.to_s
# => "4d6e5acebcd1b3fac9000002" 
BSON::ObjectId.from_string(id.to_s)
# => BSON::ObjectId('4d6e5acebcd1b3fac9000002')

Generation Time

Switching back and forth between strings is simple and obvious. Something a little more interesting is that pretty much every driver supports extracting the generation time from an object id. This means that you can stop using created_at in your Mongo documents and instead just pull it from the object id. We are doing this in Gaug.es quite often.

BSON::ObjectId.new.generation_time
# => 2011-03-02 15:01:08 UTC

The generation time is UTC and you can easily use ActiveSupport’s awesome TimeZone stuff to move the time into different zones.

id = BSON::ObjectId.new
id.generation_time.in_time_zone(Time.zone)

Miss created at? Just make a simple method that wraps this and returns the generation time in the current zone.

def created_at
  id.generation_time.in_time_zone(Time.zone)
end

Date Range Queries

Where it gets even more interesting is that you can also create object id’s from time. This means you can use object id’s in range queries, say to find out how many people signed up for your sweet new app today.

sites = Gauges.db['sites'] # get Mongo::Collection
start_at = Time.zone.now.beginning_of_day
time = BSON::ObjectId.from_time(start_at)
sites.find('_id' => {'$gte' => time}).count

Above I just did a count query, but you could actually iterate the documents or whatever as with any normal Mongo query. The benefit is that _id is automatically indexed, so you do not have to have an extra field plus an extra index.

Nothing earth shattering here, but I have found these really helpful on Gaug.es and thought I would share.

MongooseJS 1.0

2011-02-01T15:03:49-05:00

Mongoose, the MongoDB object modeling tool for node.js that powers LearnBoost, just hit 1.0.

MongoDB 1.8 Map/Reduce Changes

2011-01-27T12:57:01-05:00

Map/reduce output has changed in 1.7.5 (and thus on up). This post does a great job explaining why temporary collections are no longer an option and how to use map/reduce with the updates.

Single Server Durability

2011-01-27T12:54:31-05:00

The MongoDB team has released 1.7.5 (unstable) which includes single server durability. Using the —dur option when on start turns on journaling. The documentation on journaling is also a good read.

Why Command Helpers Suck

2011-01-25T12:01:18-05:00

Kristina Chodorow posts about why the command helpers are harmful and what you should be doing instead.

Every command helper is just a find() in disguise! This means you can do (almost) anything with a database command that you could with a query.