Tencent open source data component Fast-Causal-Inference, which can be used for distributed vector statistical analysis and causal inference

CTOnews.com Sept. 18, Tencent announced in its official account "Tencent Open Source" that its open source distributed data science component project Fast-Causal-Inference has been announced in GitHub.

▲ image source "Tencent open source" official account it is reported that this is developed by Tencent Wechat, uses SQL interaction, based on distributed vectorization statistical analysis, causal inference calculation library, it is said to "solve the performance bottleneck of the existing statistical model base (R / Python) under big data, provide 10 billion-level data second execution Causal inference capability, at the same time reduce the threshold for the use of statistical models through SQL language, easy to use in the production environment. At present, a number of applications have been carried out within Wechat, such as Wechat video number, Wechat search and so on. "

Official introduction:

Provide Causal inference capability for mass data execution in seconds

Based on vectorization OLAP execution engine ClickHouse / StarRocks, the speed is more beneficial to maximize the user experience.

Minimalist use of SQL SQLGateway WebServer lowers the threshold for the use of statistical models through SQL, and provides minimalist SQL usage at the upper level, transparently doing engine-related SQL expansion and optimization.

Provide the causal inference ability of basic operators and higher-order operators, and the upper application encapsulation supports ttest, OLS, Lasso, Tree-based model, matching, bootstrap, DML and so on.

CTOnews.com also learned that officials said that the first version already supports the following features:

The basic causal inference tool is based on deltamethod's ttest and supports CUPED

OLS, 100 million rows of data, subsecond level

Advanced causal inference tools based on OLS IV,WLS, as well as other GLS,DID, synthetic control, CUPED,mediation are hatching

Uplift: ten million data minute level operation

Bootstrap / permutation and other data simulation frameworks to solve the problem of variance estimation without showing the solution.

Referenc

Open Source announcement | Tencent distributed data Science component

Tencent / fast-causal-inference-GitHub