ES aggregates Learning Notes-- HyperLogLog and BloomFilter

The aggregation of ES is one of its major features. However, for performance reasons, the aggregation of ES is in units of fragmented Shard, not Index, so

The accuracy of some aggregations needs to be noted. For example: TermAggregations.

The cardinality aggregation of es uses the hyperloglog algorithm. Out of curiosity, I got to know it.

In massive data scenarios, we usually encounter two problems:

Data weighting. For example, in the push message scenario, repeated messages disturb the user, and when the user issues coupons, it is a loss.

Pv/uv statistics. In such scenarios, there is no need to haggle over pennies for accuracy.

How to solve these two kinds of problems efficiently?

For data weighting, we can use a Bloom filter. The java sample column code is as follows:

BloomFilter bloomFilter = BloomFilter.create (new Funnel () {private static final long serialVersionUID = 1L; @ Override public void funnel (String arg0, PrimitiveSink arg1) {arg1.putString (arg0, Charsets.UTF_8);}, 1024 / 1024 / 32); bloomFilter.put ("asdf"); bloomFilter.mightContain ("asdf")

For counting, we can use the HyperLogLog algorithm, which is already implemented in ES.

In fact, the Bloom filter can also directly achieve the function of the HyperLogLog algorithm.

Here are a few questions left over, which can be added after careful consideration:

What are the applicable scenarios for the efficiency and memory consumption of BloomFilter and HyperLogLog algorithms under the same order of data?