Some ideas of data Intermediate products

Some ideas of data center system.

* introduction *

With the help of big data / data Lake project, enterprises generally need to improve the carding of enterprise data: taking inventory of enterprise data assets and improving the relevant processes of data governance. It is convenient for data users to use data internally. External satisfaction of the Ministry of supervision of a variety of data caliber requirements.

Basic solution * *

The basic solution to this kind of problem is to improve the data life cycle management from the perspective of metadata management (including business, technology). The metadata here contains:

Traditional data dictionary (including forms, fields) business metadata carding, including business indicators, statistical caliber, etc. (usually corresponding to dimensional data, business description data code table, mapping table carding) data data lineage, audit

* challenges and existing problems

There are many such tools in the market, and the traditional warehouse products have metadata management tools. But most of them have some problems, such as:

Unfriendly to data users. Here, data users include a) data analysts b) data maintenance operators are difficult to land: they can not be well integrated into the existing development process of enterprises, traditional enterprises have a large number of legacy systems, the metadata (data description) of these systems themselves is not perfect due to historical reasons. In the process of integrating legacy system data into big data / data Lake, these metadata need to be perfected in the course of the project, but lack of tools, or existing tools are not friendly.

* * production and selection of entry points * *

Based on the above analysis, the metadata management part of the data platform (Zhongtai) product can adopt the following design ideas:

Promote the front-end data display with the help of graph database and improve data association analysis and development customization tools for legacy systems and project development process metadata by means of NLP and traditional metadata management. (further explain the following, in most warehouses, big data project, business data analysts need to prepare Mapping documents. At present, most mapping documents exist in the form of excel, which is not conducive to maintenance, sharing and further mining. You can develop customized online mapping documentation tools as a starting point to refer to (logical) architecture diagrams:

At present, some similar products have been used on the market, which can also be used as a reference:

Http://www.stargraph.cn/ Financial Intelligence BDP https://memect.cn/ knowledge Graph Technology http://www.primeton.com/ metadata platform https://www.topquadrant.com/