FastDFS 分布式集群搭建详解

最新推荐文章于 2025-06-30 22:41:28 发布

程序员大佬超

最新推荐文章于 2025-06-30 22:41:28 发布

阅读量4.8k

点赞数 33

CC 4.0 BY-SA版权

分类专栏： FastDFS 文章标签：分布式 FastDFS fdfs

本文链接：https://blog.csdn.net/xch_yang/article/details/136057831

FastDFS 专栏收录该内容

3 篇文章

订阅专栏

文章目录

前言

阅读本文章之前请先看上一篇单机版FastDFS安装配置详解，所以本文有些内容我就一笔带过，传送门：《FastDFS 环境搭建及使用详解》

1、整体架构

大佬超这次使用了7个服务器节点，其中2个跟踪器节点，4个存储节点，需要注意的是 nginx 有三层，第一层使用了一个 nginx 做统一入口进行负载均衡，第二层 tracker 有两个 nginx 做负载均衡，第三层 storage 有四个 nginx 用于访问FastDFS上的文件，服务器节点信息如下

节点	角色	所属组	nginx
192.168.122.1			nginx
192.168.122.22	tracker		nginx1
192.168.122.23	tracker		nginx2
192.168.122.24	storage1	Group1	nginx1
192.168.122.25	storage2	Group1	nginx2
192.168.122.26	storage3	Group2	nginx3
192.168.122.27	storage4	Group2	nginx4

在这里插入图片描述

2、安装配置FastDFS集群

每个节点先按照前面单机版的步骤安装FastDFS。然后进行集群配置，如下

2.1 配置tracker

192.168.122.22 和 192.168.122.23 节点

vi /etc/fdfs/tracker.conf

#启用配置文件，默认启用配置文件（默认值）
disabled=false 
#设置tracker的端口号，通常采用22122这个默认端口 （默认值）
port=22122 
#设置tracker的数据文件和日志目录
base_path=/opt/fastdfs/tracker
#设置http端口号，（默认值）
http.server_port=8080

需要注意的是，我们文件上传默认会负载均衡，也就是tracker里如下默认配置

# the method for selecting group to upload files
# 0: round robin
# 1: specify group
# 2: load balance, select the max free space group to upload file
store_lookup = 2

# which group to upload file
# when store_lookup set to 1, must set store_group to the group name
store_group = group2

store_lookup，该值默认是2（负载均衡策略，磁盘空间优先），0（轮询策略），1（指定group进行上传下载操作，也就是store_group的值），只有指定为1时store_group才会起作用。

2.2 配置storage

配置另外四台 storage，只有 group_name 不一样

192.168.122.24 和 192.168.122.25 节点

vi /etc/fdfs/storage.conf

#启用配置文件（默认值）
disabled=false
#组名，根据实际情况修改 (默认值)
group_name=group1
#设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 (默认值)
port=23000
#设置storage数据文件和日志目录
base_path=/opt/fastdfs/storage
#存储路径个数，需要和store_path个数匹配 （默认值）
store_path_count=1  
#实际文件存储路径
store_path0=/opt/fastdfs/storage/files
#tracker 服务器的 IP地址和端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#设置 http 端口号  (默认值)
http.server_port=8888

192.168.122.26 和 192.168.122.27 节点

vi /etc/fdfs/storage.conf

#启用配置文件（默认值）
disabled=false
#组名，根据实际情况修改 (默认值)
group_name=group2
#设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 (默认值)
port=23000
#设置storage数据文件和日志目录
base_path=/opt/fastdfs/storage
#存储路径个数，需要和store_path个数匹配 （默认值）
store_path_count=1  
#实际文件存储路径
store_path0=/opt/fastdfs/storage/files
#tracker 服务器的 IP地址和端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#设置 http 端口号  (默认值)
http.server_port=8888

3、启动集群

192.168.122.22、192.168.122.23：

#启动tracker
fdfs_trackerd /etc/fdfs/tracker.conf
#重新启动tracker
fdfs_trackerd /etc/fdfs/tracker.conf restart

192.168.122.24、192.168.122.25、192.168.122.26、192.168.122.27：

#启动storage
fdfs_storaged /etc/fdfs/storage.conf
#重新启动storage
fdfs_storaged /etc/fdfs/storage.conf restart

4、查看集群情况

fdfs_monitor /etc/fdfs/storage.conf

结果如下

[root@mongodb-1 logs]# fdfs_monitor /etc/fdfs/storage.conf
server_count=2, server_index=0

tracker server is 192.168.122.22:22122

group count: 2

Group 1:
group name = group1
disk total space = 51,175 MB
disk free space = 47,554 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.122.24
                ip_addr = 192.168.122.24  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-29 18:12:37
                up time = 2024-01-29 18:12:37
                total storage = 51,175 MB
                free storage = 49,349 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 192.168.122.25
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 2
                total_upload_count = 0
                success_upload_count = 0
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 0
                success_set_meta_count = 0
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 0
                success_upload_bytes = 0
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 0
                success_sync_in_bytes = 0
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 0
                success_file_open_count = 0
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 0
                success_file_write_count = 0
                last_heart_beat_time = 2024-01-30 10:09:11
                last_source_update = 1970-01-01 08:00:00
                last_sync_update = 1970-01-01 08:00:00
                last_synced_timestamp = 1970-01-01 08:00:00 
        Storage 2:
                id = 192.168.122.25
                ip_addr = 192.168.122.25  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-29 18:12:36
                up time = 2024-01-29 18:12:36
                total storage = 51,175 MB
                free storage = 47,554 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 2
                total_upload_count = 0
                success_upload_count = 0
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 0
                success_set_meta_count = 0
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 0
                success_upload_bytes = 0
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 0
                success_sync_in_bytes = 0
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 0
                success_file_open_count = 0
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 0
                success_file_write_count = 0
                last_heart_beat_time = 2024-01-30 10:09:09
                last_source_update = 1970-01-01 08:00:00
                last_sync_update = 1970-01-01 08:00:00
                last_synced_timestamp = 1970-01-01 08:00:00 

Group 2:
group name = group2
disk total space = 51,175 MB
disk free space = 48,108 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0

        Storage 1:
                id = 192.168.122.26
                ip_addr = 192.168.122.26  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-29 18:12:33
                up time = 2024-01-29 18:12:33
                total storage = 51,175 MB
                free storage = 48,118 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 4
                total_upload_count = 13
                success_upload_count = 13
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 13
                success_set_meta_count = 13
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 151530173
                success_upload_bytes = 151530173
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 272774924
                success_sync_in_bytes = 272774924
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 61
                success_file_open_count = 61
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 1657
                success_file_write_count = 1657
                last_heart_beat_time = 2024-01-30 10:09:06
                last_source_update = 2024-01-30 10:01:12
                last_sync_update = 2024-01-30 10:02:35
                last_synced_timestamp = 2024-01-30 10:02:35 (0s delay)
        Storage 2:
                id = 192.168.122.27
                ip_addr = 192.168.122.27  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-24 10:29:49
                up time = 2024-01-29 18:12:28
                total storage = 51,175 MB
                free storage = 48,108 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 3
                total_upload_count = 24
                success_upload_count = 24
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 24
                success_set_meta_count = 24
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 1
                success_download_count = 1
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 272773872
                success_upload_bytes = 272773872
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 29
                success_download_bytes = 29
                total_sync_in_bytes = 151530717
                success_sync_in_bytes = 151530717
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 51
                success_file_open_count = 51
                total_file_read_count = 1
                success_file_read_count = 1
                total_file_write_count = 1646
                success_file_write_count = 1646
                last_heart_beat_time = 2024-01-30 10:09:00
                last_source_update = 2024-01-30 10:02:35
                last_sync_update = 2024-01-30 10:01:13
                last_synced_timestamp = 2024-01-30 10:01:12 (0s delay)

5、nginx配置

需要在每个节点都安装nginx，但是作用不是一样的，storage上的nginx是用于访问具体的文件，并且需要使用fastdfs模块，tracker上的nginx是用于负载均衡，不需要使用fastdfs模块，而最外层nginx则是统一入口，用于负载均衡到tracker上的nginx。

5.1 配置storage的四台机器的nginx

安装nginx所需依赖

#安装nginx所需依赖
yum -y install pcre pcre-devel
yum -y install zlib zlib-devel
yum -y install openssl openssl-devel

解压 nginx 和 fastdfs-nginx-module

cd /usr/local/fastdfs
tar -zxvf nginx-1.19.10.tar.gz 
unzip fastdfs-nginx-module-master.zip

解压后进入 nginx 目录编译安装 nginx，并添加 fastdfs-nginx-module，我这里为了和原有Nginx进行区分，新安装的Nginx取名为nginx_fdfs

# 创建nginx安装目录
mkdir -p /usr/local/nginx_fdfs

cd /usr/local/fastdfs/nginx-1.19.10
# 配置，指定安装路径，添加模块
./configure --prefix=/usr/local/nginx_fdfs --add-module=/usr/local/fastdfs/fastdfs-nginx-module-master/src
# 编译
make
# 安装
make install

访问配置

把 fastdfs-nginx-module-master/src 目录下的 mod_fastdfs.conf 拷贝到 /etc/fdfs 目录下，这样才能正常启动 Nginx

cp /usr/local/fastdfs/fastdfs-nginx-module-master/src/mod_fastdfs.conf  /etc/fdfs/

在 /opt/fastdfs/ 目录下创建 nginx_mod 目录

mkdir -p /opt/fastdfs/nginx_mod

修改 mod_fastdfs.conf 配置文件

vi /etc/fdfs/mod_fastdfs.conf

#日志目录
base_path=/opt/fastdfs/nginx_mod  
#tracker服务器的IP地址以及端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#storage服务器的端口号(默认值)
storage_server_port=23000 
#组名
group_name = group1
#文件url中是否有 group 名
url_have_group_name = true 
#存储路径
store_path0=/opt/fastdfs/storage/files
#组数量
group_count = 2

[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs/storage/files

[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs/storage/files

注：group_name根据实际组名变化，这里192.168.122.24和192.168.122.25为group1，192.168.122.26和192.168.122.27为group2。

修改 nginx.conf 配置文件

vi /usr/local/nginx_fdfs/conf/nginx.conf

  server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
	    # 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求，用fastdfs的Nginx模块进行转发
        location ~ /group[1-9]/M0[0-9] {	
             ngx_fastdfs_module;  
        }
 }

启动nginx

# 测试
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -t
# 启动
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf

# 停止
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s stop
# 重载
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s reload

现在四个storage节点的图片都能进行访问，例如

http://192.168.122.24/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.25/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.26/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.27/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg

5.2 配置tracker的两台机器的nginx

192.168.122.22 和 192.168.122.23 节点

安装nginx所需依赖

#安装nginx所需依赖
yum -y install pcre pcre-devel
yum -y install zlib zlib-devel
yum -y install openssl openssl-devel

解压 nginx

cd /usr/local/fastdfs
tar -zxvf nginx-1.19.10.tar.gz

解压后进入 nginx 目录编译安装 nginx，这里不需要 fastdfs-nginx-module

# 创建nginx安装目录
mkdir -p /usr/local/nginx_fdfs

cd /usr/local/fastdfs/nginx-1.19.10
# 配置，指定安装路径，这里不需要添加模块
./configure --prefix=/usr/local/nginx_fdfs
# 编译
make
# 安装
make install

修改 nginx.conf 配置文件

vi /usr/local/nginx_fdfs/conf/nginx.conf

upstream fastdfs_group_server{
    server 192.168.122.24:80;
    server 192.168.122.25:80;
    server 192.168.122.26:80;
    server 192.168.122.27:80;
}

server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
	    # 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求
        location ~ /group[1-9]/M0[0-9] {	
             proxy_pass http://fastdfs_group_server;
        }
 }

启动nginx

# 测试
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -t
# 启动
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf

# 停止
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s stop
# 重载
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s reload

现在通过tracker上的nginx可以访问到四台storage的文件，例如

http://192.168.122.22/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.23/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg

5.3 配置统一入口

这里tracker有两个nginx，我们在最外层再增加一个nginx负载均衡，也就是我这里的192.168.122.1节点，配置如下

upstream fastdfs_group_server{
    server 192.168.122.22:80;
    server 192.168.122.23:80;
}

server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
	    # 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求
        location ~ /group[1-9]/M0[0-9] {	
             proxy_pass http://fastdfs_group_server;
        }
 }

现在通过统一入口上的nginx可以访问到文件，例如

http://192.168.122.1/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg

在这里插入图片描述

至此，集群搭建完成，我辛苦了。

熬夜写的，觉得有帮助来个赞吧！！！

更多技术干货，请持续关注程序员大佬超。
原创不易，转载请务必注明出处。