# kafka-logs

**Repository Path**: qspy/kafka-logs

## Basic Information

- **Project Name**: kafka-logs
- **Description**: kafka日志收集系统搭建
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2022-05-31
- **Last Updated**: 2022-05-31

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

## 1. 准备

- zookeeper 安装包下载：https://downloads.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
- kafka 安装包下载：https://downloads.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz
- jdk1.8 及以上环境：https://dragonwell.oss-cn-shanghai.aliyuncs.com/8.6.6/Alibaba_Dragonwell_8.6.6_x64_linux.tar.gz
- 准备三台机器

## 2. 集群架构

### 2.1 zookeeper 集群

| 主机IP           | 消息端口 | 通信端口      | 数据目录                                                     | 日志目录                                                     |
|----------------|------|-----------|----------------------------------------------------------|----------------------------------------------------------|
| 172.19.187.187 | 2181 | 2888:3888 | /data/software/zookeeper/apache-zookeeper-3.7.0-bin/data | /data/software/zookeeper/apache-zookeeper-3.7.0-bin/logs |
| 172.19.176.224 | 2181 | 2888:3888 | /data/software/zookeeper/apache-zookeeper-3.7.0-bin/data | /data/software/zookeeper/apache-zookeeper-3.7.0-bin/logs |
| 172.19.190.181 | 2181 | 2888:3888 | /data/software/zookeeper/apache-zookeeper-3.7.0-bin/data | /data/software/zookeeper/apache-zookeeper-3.7.0-bin/logs |


### 2.2 kafka 集群

| 主机IP           | 集群通信端口 | 节点目录                                  |
|----------------|--------|---------------------------------------|
| 172.19.187.187 | 9092   | /data/software/kafka/kafka_2.13-2.8.0 |
| 172.19.176.224 | 9092   | /data/software/kafka/kafka_2.13-2.8.0 |
| 172.19.190.181 | 9092   | /data/software/kafka/kafka_2.13-2.8.0 |


## 3. zookeeper 集群搭建

创建 zookeeper 目录
```
mkdir -p /data/software/zookeeper
```

### 3.1 解压安装包

```
tar -zxvf apache-zookeeper-3.7.0-bin.tar.gz -C /data/software/zookeeper
```

### 3.2 修改配置文件

`sudo vim /etc/profile`
```
export ZOOKEEPER_HOME=/data/software/zookeeper/apache-zookeeper-3.7.0-bin
export PATH=$PATH:$ZOOKEEPER_HOME/bin
```

生效 profile
```
source /etc/profile
```

复制 zoo_sample.cfg 改名为 zoo.cfg 并添加配置：
```
 cp /data/software/zookeeper/apache-zookeeper-3.7.0-bin/conf/zoo_sample.cfg /data/software/zookeeper/apache-zookeeper-3.7.0-bin/conf/zoo.cfg
```

```
dataDir=/data/software/zookeeper/apache-zookeeper-3.7.0-bin/data
dataLogDir=/data/software/zookeeper/apache-zookeeper-3.7.0-bin/logs
server.0=172.19.187.187:2888:3888
server.1=172.19.176.224:2888:3888
server.2=172.19.190.181:2888:3888
```

三台机器分别写入 myid（0/1/2）
```
echo 0 > /data/software/zookeeper/apache-zookeeper-3.7.0-bin/data/myid
```

## 3.3 启动 zookeeper

```
zkServer.sh start
```

查看状态
```
zkServer.sh status
```

## 4. kafka 集群搭建

### 4.1 解压 kafka

`mkdir -p /data/software/kafka/`

```
tar  -zxvf kafka_2.13-2.8.0.tgz -C /data/software/kafka/
```

### 4.2 修改配置文件

修改每个机器的 config/server.properties
```
listeners=PLAINTEXT://:9092
zookeeper.connect=172.19.187.187:2181,172.19.176.224:2181,172.19.190.181:2181/kafka
```

并将三个机器的 broker.id 分别设置为 0/1/2

### 启动 kafka

kafka 默认设置堆大小为 1G，对于小内存的机器可能不到 1G，需要进行如下设置：
```
export KAFKA_HEAP_OPTS="-Xmx512m -Xmx512m"
```
然后启动 kafka
```
bin/kafka-server-start.sh -daemon config/server.properties
```

## 5. 验证

### 5.1 创建一个 topic

```
 bin/kafka-topics.sh --create --zookeeper 172.19.187.187:2181,172.19.176.224:2181,172.19.190.181:2181/kafka --replication-factor 3 --partitions 2 --topic test-topic01
```

### 5.2 查看 topic

```
bin/kafka-topics.sh --list  --zookeeper 172.19.187.187:2181,172.19.176.224:2181,172.19.190.181:2181/kafka
```

### 5.3 生产消息

```
bin/kafka-console-producer.sh --broker-list 172.19.187.187:9092,172.19.176.224:9092,172.19.190.181:9092 --topic test-topic01
```

### 5.4 消费消息

```
bin/kafka-console-consumer.sh --bootstrap-server 172.19.187.187:9092,172.19.176.224:9092,172.19.190.181:9092 --topic test-topic01 --from-beginning
```

## 6. 安装 nginx 和 ngx_kafka_module

### 6.1 安装 ngx_kafka_module 依赖

```
git clone https://github.com/edenhill/librdkafka
cd librdkafka
./configure
make
sudo make install
```

可能需要提前安装 gcc 和 g++

```
sudo apt install gcc g++ libpcre3 libpcre3-dev  openssl  libssl-dev  zlib1g-dev
```

### 6.2 下载并解压 nginx

```
wget http://nginx.org/download/nginx-1.20.1.tar.gz
tar -zxvf nginx-1.20.1.tar.gz -C /data/software/nginx
```

### 6.3 下载 ngx_kafka_module

```
git clone https://github.com/brg-liuwei/ngx_kafka_module
```

### 6.4 编译安装 nginx

```
./configure --prefix=/usr/local/nginx --add-module=/home/ubuntu/ngx_kafka_module  --with-http_ssl_module

make

sudo make install
```