Detailed explanation of common cross-network access in Kafka cluster

Scenario description:

When the client and the server are in different areas (across firewalls, addresses are mapped), the client will not get the broker when accessing the kafka, but the networks are interoperable. However, when client requests zookeeper across firewalls, the broker IP returned by zookeeper to client is the actual address of kafka, not the mapped address, so client access fails.

Solution 1.

1. Configuration file: listeners=PLAINTEXT:// hostname: 9092

2. Server hosts: host name of private network address

3. Client hosts: host name of public network address

Note: insecure, complex and inconsistent configuration

Solution 2: (case)

1. Notes:

Listeners=PLAINTEXT://192.168.111.134:9092

Add:

Inter.broker.protocol.version=1.0 log.message.format.version=1.0 advertised.listeners=INSIDE://192.168.111.134:9092,OUTSIDE://node7:9093 listeners=INSIDE://192.168.111.134:9092,OUTSIDE://node7:9093 listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT inter.broker.listener.name=INSIDE

Explanation: it is equivalent to opening 2 ports. If the server has multiple network cards, you can fill in the public network address in the host name part.

If unified management is adopted, hostname resolution can be used and nginx proxy can be used.

For the above changes, you need to restart 3 clusters to take effect, and the zk listening address is different, as shown in the figure (therefore, standardized configuration is recommended to take into account the business)

2. Server hosts

Host name of private network address (domain name resolution is used here)

Note: the official website also provides secure encryption and non-text encryption. If both kafka and zk require security, it is recommended to refer to the official website for unified configuration.

Explain the relevant parameters on the official website:

Host.name:

Advertised.host.name:

Advertised.listeners:

Listeners: