Basic knowledge of distributed system design

In cloud computing, big data, Internet and other fields, there is the concept of cluster, so how to carry out cluster management? Although each has its own way of implementation, but the idea is the same, through the leader to manage a cluster, has its own election algorithm. And there are many components with similar functions in the cluster, which can be selected by users at will.

Some of my commonly used components

Cluster Management: zookeeper

Message queuing: zeromq

Cache: memcached,redis

Database: mysql,postgresql

NoSQL:mongodb (actually memcached and redis are also NoSQL)

Message queue

Compared to zeromq, rabbitmq does not support persistence, but it is very easy to use, just a library, while rabbitmq supports persistence, but requires a server for message routing.

Caching

Memcached does not support clusters, relies on zookeeper to manage clusters, only provides caching, and cached data in clusters do not need to be synchronized. Redis supports clustering, has rich data types, can achieve the read-write separation of one master and multi-standby, and has its own election algorithm.

Reverse proxy + load balancing

Nginx,haproxy, both of which are excellent reverse proxy tools, nginx can provide not only LB function, but also web server function. Nginx is easier to use and configure, but the forwarding efficiency is not as efficient as haproxy.

CDN service

Generally, you need to purchase a commercial CDN. After forming a certain scale, you can consider your own CND. At present, with the exception of BAT, few people develop their own CDN services. Because a large number of computer rooms are built, they also need a lot of money, which is not affordable to ordinary enterprises.

Firewalls

Purchase firewall equipment and configure iptables

High availability

There are many highly available solutions, such as coresync+pacemaker,keepalived, etc., keepalived is easier to use, but the function is slightly worse than the former. Data synchronization generally uses drdb.

Business split

If the business is too complex, such as large e-commerce systems, then each line of business is distributed, such as: order system, user management system, should be made into a distributed and scalable cluster.