Shulou(Shulou.com)06/01 Report--

Recently, the number of url deduplicates in a certain period of time was counted, and an error was reported due to the huge amount of data.

Distinct failed: {"errmsg": "exception: distinct too big, 16mb cap", "code": 17217, "ok": 0} at src/mongo/shell/collection.js:1108

Finally, the solution through mapreduce is as follows:

/ / define the map function map=function () {emit (this.url, {"count": 1});} / / define the reduce function reduce=function (key,values) {var total=0; for (var ivalues 0; I < values.length; ivalues +) {total+= values [I] .count } return {count:total}} / / executes the mapreduce function, where the value of out is the collection db.runCommand ({"mapreduce": "visit", "map": map, "reduce": reduce, "query": {"vtime": {"$gte": 1412611200, "$lte": 1413907119}}, "out": "test.tmp"})

The options for mapreduce are explained as follows:

"out": {replace: "collection name"}-mapreduce output will replace the original collection,collection if it does not exist, create "out": {merge: "collection name"}-merge the new and old data, replace the old one, and add the non-existing data to "out": {reduce: "collection name"}-- when the old data exists Add new data (i.e. new value=old value+mapreduce value) "out": {inline:1}-- No collection will be created, and the result will be saved in memory only if the result is less than 16MB.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.