Comments on Just a little Python: MongoDB Schema Design at Scale

nice

2016-10-06T05:51:58.418-04:00

nice

If you only want a single metric, the query is str...

2013-09-11T10:22:41.450-04:00

If you only want a single metric, the query is straightforward. If you need multiple metrics, you can do a regex query on _id for '^20101010/', which will be reasonably fast. If you're *always* getting the same set of metrics, however, I'd recommend storing them alongside one another in the document instead.

The question of whether to use multiple *collections* or a single collection is pretty much a wash performance-wise. Separating your documents into different collections is only really *necessary* when you have different query patterns (and therefore a need for different indexes, sharding approaches, etc.).

Hope this helps!

Well, you can store the three counters in three co...

2013-09-11T10:19:16.137-04:00

Well, you can store the three counters in three collections. I don't think there's a good way to make it too much more compact, though; you don't want to "pack" the arrays since that means MongoDB can't do an in-place update, and BSON's not particularly efficient as a storage protocol (compared with relational DBs, that is).

Great post. I have some questions for you. You are...

2013-09-10T17:34:40.382-04:00

Great post. I have some questions for you. You are using "20101010/metric-1" as an _id. So you are using a single collection for multiple metrics. How do you query that?
Also, having in mind that I have multiple devices with multiple metrics what's better to do. Create one collection per device and store multiple metrics or create one collection for every single metric?

I want to structure my data this way but I need a ...

2013-09-09T05:45:31.114-04:00

I want to structure my data this way but I need a counter for each value(daily, hourly, minute). Any idea of how to store that efficiently? I firstly tried this solution but I want my data to be more compact.

Hi Anon, Monary looks pretty cool; I hadn't h...

2013-01-15T20:51:26.456-05:00

Hi Anon,

Monary looks pretty cool; I hadn't heard of it before. I don't think it's really applicable to this case, however, as it's focused on reading a "stripe" of values from many documents into a single numpy array. What I'm trying to do here is to repeatedly update a single document.

Thanks for the comment, though. Monary looks very interesting; I'm going to have to find a place to use it. :-)

What about using alternative connector like Monary...

2013-01-15T17:48:29.149-05:00

What about using alternative connector like Monary which instead of encapsulating into dictionaries, it relies on NumPy arrays?

https://bitbucket.org/djcbeach/monary/wiki/Home

2012-11-07T15:21:02.218-05:00

This comment has been removed by the author.

Thanks for the comment, Vivek! Glad you found it u...

2012-10-28T19:56:11.855-04:00

Thanks for the comment, Vivek! Glad you found it useful :)

Simple and insightful article, thank you Rick. I&#...

2012-10-27T12:58:16.058-04:00

Simple and insightful article, thank you Rick.
I'm making schema changes right-away :)

Hi Sergey, Thanks for the comment! It turns out t...

2012-10-13T10:21:34.087-04:00

Hi Sergey,

Thanks for the comment! It turns out that arrays in BSON are actually stored as dicts where they keys are "1", "2", "3", etc. (surprising but true!), so you wouldn't actually get any faster using them. I"ve brought up this interesting design decision with 10gen folks multiple times, so someday we may have better-performing arrays, but for now they're just documents with a different type code.

I am just curious if accessing 'dict' keys...

2012-10-05T03:45:19.622-04:00

I am just curious
if accessing 'dict' keys in mongo document is slow
is accessing 'list' indexes slow too ?

could we replace
'minute': { '0000': N0, '0001': N1, ... '1439': N1439 }
with
'minute': [N0...N1439]

In this case, I simply kept the minute of the day ...

2012-10-02T11:59:02.248-04:00

In this case, I simply kept the minute of the day (numbered 0-1439) as the key of the embedded document. You could also use

"23": { "00":..., "59": ... }

... which is probably what I would do in the future.

"23": { ..., "1439": 2819 } Is...

2012-10-02T10:31:49.005-04:00

"23": { ..., "1439": 2819 }
Is that really correct? Shouldn't it be something like:
"23": { ..., "2339": 2819 }
Or have I missed out something?

Thanks for the comment! I have to admit, the insig...

2012-09-28T10:57:01.566-04:00

Thanks for the comment! I have to admit, the insight on O(N) object traversal is not mine; it was discovered by the 10gen MMS team when optimizing the monitoring service. It was a fascinating and weird enough 2nd order effect that I thought others would be interested as well.

Thanks again!

Great insights! especially with regard to the 0(N)...

2012-09-27T18:46:58.824-04:00

Great insights! especially with regard to the 0(N) object traversal on updates.

2012-09-27T18:44:17.155-04:00

This comment has been removed by the author.

Thanks for the comment! I'm glad you liked it....

2012-09-27T11:53:12.609-04:00

Thanks for the comment! I'm glad you liked it.

Wonderful! Thank you, we need more of this kind of...

2012-09-27T09:16:41.682-04:00

Wonderful! Thank you, we need more of this kind of posts :)

Thanks for the comment!

2012-09-26T10:46:00.074-04:00

Thanks for the comment!

Great post! Implement, measure, adjust.

2012-09-26T00:11:12.679-04:00

Great post! Implement, measure, adjust.

Leo, Thanks so much for the comment! I'm glad...

2012-09-25T13:31:29.943-04:00

Leo,

Thanks so much for the comment! I'm glad you found the post useful. It's always nice to see real numbers I think (especially in scatter plots ;-) )

Rick, Thank you very much for putting this togeth...

2012-09-25T13:16:30.625-04:00

Rick,

Thank you very much for putting this together. I have to say that this is one of the best blog posts I have ever read on MongoDB design and scalability. And it has just convinced me of getting rid of growing documents in my models.

Leo