nginx qat on docker

发表于 | 分类于 | | 游览

Intel QAT 加速卡可以对HTTPS的请求进行异步请求， 加快证书处理， 降低系统性能消耗。
nginx 作为代理， 可以代理HTTPS请求， 需要重新编译，支持QAT加速卡，这样才能将请求给QAT加速卡。
Intel QAT 加速卡安装在上一遍文章已经提到， 这里就不在说了， 可以查看之前文章。
这回我们说说如何把QAT卡封装到docker容器中。
我测试过了， 就算在docker容器中安装驱动， 也需要在宿主机上安装驱动。是值代理服务可以不用在宿主机上安装。

intel qat 加速卡安装配置 ： https://sukbeta.github.io/intel-qat/
宿主机上nginx配置qat：https://sukbeta.github.io/nginx-qat/

相关URL

nginx qat docker container install ： https://01.org/sites/default/files/downloads//337020-003-qatwcontaineranddocker.pdf

安装docker

简单安装一个docker服务

1
2
3
4
yum install -y yum-utils
yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce
systemctl start docker

修改docker limit

vim /usr/lib/systemd/system/docker.service

1
2
3
4
5
6
[Service]
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
添加
LimitMEMLOCK=infinity

重启服务

1
2
systemctl daemon-reload
systemctl restart docker.service
Dockerfile qat nginx on docker

首先宿主上是安装qat驱动的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
FROM centos:7 AS build
RUN yum install -y -q epel-release centos-release-scl
RUN yum -y groupinstall "Development Tools"
RUN yum install -y wget tar gcc-c++ make bzip2 make gcc zlib-devel libtool autoconf cmake make python gcc gawk autoconf automake libtool pkg-config patch pcre-devel libxslt-devel openssl openssl-devel zlib pcre libxslt perl perl-devel pciutils libudev-devel gzip unzip net-tools lsof bzip2 bzip2-devel kernel kernel-devel && \
	rm -rf /var/yum/cache/*
#ADD QAT.tar.gz /home/
ARG ICP_ROOT=/home/QAT
RUN mkdir $ICP_ROOT
ADD QAT1.7.L.4.13.0-00009.tar.gz $ICP_ROOT
RUN cd $ICP_ROOT && ./configure && make && make install && make samples-install

ENV OPENSSL_LIB /usr/local/ssl
ENV OPENSSL_ENGINES /usr/local/ssl/lib/engines-1.1
ENV PERL5LIB $PERL5LIB:/home/openssl
ENV PATH $PATH:/usr/local/sbin
ENV SSL_INC /usr/local/ssl/include
ENV SSL_LIB /usr/local/ssl/lib
ENV QZ_ROOT /home/QATzip
ENV OPENSSL_LIB /usr/local/ssl
ENV ICP_ROOT /home/QAT
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:$QZ_ROOT/utils:/usr/lib64:/usr/local/lib64:/usr/local/ssl/lib:/usr/local/ssl/lib/engines-1.1:/usr/lib64/openssl/engines:$ICP_ROOT/build:/usr/local/lib

#RUN cd /home && git clone https://github.com/openssl/openssl.git
#ADD openssl_github.tar.gz /home/
ADD OpenSSL_1_1_1j.tar.gz /home/
RUN cd /home/ && mv openssl-OpenSSL_1_1_1j openssl
RUN yum -y install yum-utils perl* judy
RUN	cd /home/openssl && \
	./config --prefix=/usr/local/ssl -Wl,-rpath,/usr/local/ssl/lib &&\
	make depend && \
	make && \
	make install

##RUN cd /home && git clone https://github.com/intel/QAT_Engine.git
ADD QAT_Engine_github.tar.gz /home/
RUN cd /home/QAT_Engine && \
	./autogen.sh && \
	./configure \
	--with-qat_hw_dir=/home/QAT \
	--enable-qat_sw \
	--with-openssl_install_dir=/usr/local/ssl && \
	make && \
	make install

RUN cd /home/QAT_Engine/qat_contig_mem && \
	make

RUN cd /home && git clone https://github.com/intel/asynch_mode_nginx.git
RUN cd /home && wget http://nginx.org/download/nginx-1.18.0.tar.gz && tar -zxf nginx-1.18.0.tar.gz
#RUN cd /home && diff -Naru -x .git nginx-1.18.0 asynch_mode_nginx > async_mode_nginx_1.18.0.patch
ADD patch.sh /home/
RUN chmod +x /home/patch.sh && bash /home/patch.sh
RUN cd /home/nginx-1.18.0 && patch -p1 < ../async_mode_nginx_1.18.0.patch

#RUN cd /home && git clone https://github.com/intel/QATzip.git
ADD ./QATzip-master.zip /home/
RUN cd /home && unzip QATzip-master.zip && mv QATzip-master QATzip
RUN cd /home/QATzip && \
	./configure --with-ICP_ROOT=$ICP_ROOT && \
	make clean && \
	make all install
#RUN cd /home/QATzip && ./setenv.sh

ARG NGINX_INSTALL_DIR=/home/nginx
RUN cd /home/nginx-1.18.0 && \
	./configure \
        --prefix=$NGINX_INSTALL_DIR \
        --with-http_ssl_module \
        --add-dynamic-module=modules/nginx_qatzip_module \
        --add-dynamic-module=modules/nginx_qat_module/ \
        --with-cc-opt="-DNGX_SECURE_MEM -I$OPENSSL_LIB/include -I$ICP_ROOT/quickassist/include -I$ICP_ROOT/quickassist/include/dc -I$QZ_ROOT/include -Wno-error=deprecated-declarations" \
        --with-ld-opt="-Wl,-rpath=$OPENSSL_LIB/lib -L$OPENSSL_LIB/lib -L$QZ_ROOT/src -lqatzip -lz" && \
	make && \
	make install
ADD ./nginx-conf.tar.gz /home/nginx/conf/
RUN ls /home

EXPOSE 80 443
CMD ["/home/nginx/sbin/nginx", "-g", "daemon off;"]

build 

1
docker build -t qat-centos .

docker run

1
2
docker run -it --rm --ulimit memlock=-1:-1  $devpara -p 80:80 -p 443:443 qat-centos:latest /bin/bash
# /home/nginx/sbin/nginx
挂载方式运行

用挂载方式运行， 将宿主安装的目录挂载到容器中运行， 这样可以运行多个容器。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
docker run -it --rm --ulimit memlock=-1:-1  -v /home:/home -v /usr/local/ssl:/usr/local/ssl -v /usr/local/lib64:/usr/local/lib64 -v /dev/hugepages:/dev/hugepages $devpara -p 80:80 -p 443:443  centos:7 /bin/bash

#  容器中设置黄精变量
export OPENSSL_LIB=/usr/local/ssl
export OPENSSL_ENGINES=/usr/local/ssl/lib/engines-1.1
export PERL5LIB=$PERL5LIB:/home/openssl
export PATH=$PATH:/usr/local/sbin
export SSL_INC=/usr/local/ssl/include
export SSL_LIB=/usr/local/ssl/lib
export QZ_ROOT=/home/QATzip
export OPENSSL_LIB=/usr/local/ssl
export ICP_ROOT=/home/QAT
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$QZ_ROOT/utils:/usr/lib64:/usr/local/lib64:/usr/local/ssl/lib:/usr/local/ssl/lib/engines-1.1:/usr/lib64/openssl/engines:$ICP_ROOT/build:/usr/local/lib

# 容器中验证 qat 卡
/usr/local/ssl/bin/openssl  engine -t -c -vvvv qatengine

# 容器中启动nginx
/home/nginx/sbin/nginx
验证加速卡处理数据
1
cat /sys/kernel/debug/qat_dh895xcc_0000\:07\:00.0/fw_counters
感谢您的支持！