hello. Wadiz I am the service platform development manager.
Wadiz Products undergoing funding at [website/platform] undergo a rigorous pre-screening process. During this stage, we request the necessary documents. These screening documents are encrypted and securely stored and managed. In this post, I would like to introduce the process of building an in-house file storage platform using S3-compatible MinIO to ensure the safe storage of encrypted files.
Requirements and System Improvement
First, the file storage platform set a goal to kill two birds with one stone: meeting the business unit's requirements and improving the system.
Business unit requirements
“ Wadiz I would like the screening document files uploaded to be automatically deleted after N years in accordance with internal security regulations.
System improvement
Prior to the implementation of the new file storage platform, files were being managed separately for each service. In other words, encrypted files were not being managed in a single central location. According to internal security regulations, files must be automatically deleted after N years from upload; however, this process was also being managed by individual teams. Due to these circumstances, the logic for managing encrypted files was fragmented, leading to issues where changes to the relevant logic required modifying and redeploying each service. There were also problems regarding development productivity.
To summarize, it is as follows.
Now, let's take a closer look at how the file storage platform was configured.
Distributed MinIO configuration
I configured Distributed MinIO as a storage space for encrypted files. I was allocated four servers for this configuration. On each server, I connected external block storage to a mount point named '/stg' so that it could be used like a local file system. After that, I registered the MinIO domain and configured Nginx so that requests accessing the MinIO domain are forwarded to minio01 through 04.
Also, I configured Distributed MinIO by referring to the MinIO Quickstart Guide , Distributed MinIO Quickstart Guide , and How to secure access to MinIO server with TLS on the official MinIO website.
Below is the configuration environment. It has been reconstructed based on the actual environment configured.
- OS: RHEL/CentOS/Oracle Linux 7
- MinIO : 2021-04-22T15:44:28Z Version
- IP :
1.2.3.4,1.2.3.5,1.2.3.6,1.2.3.7 - Domain :
minio01.foo.com,minio02.foo.com,minio03.foo.com,minio04.foo.com - Ports Used: 9000 (MinIO default port), 443 (TLS used)
So, let's take a look at the process.
1. To configure MinIO Distributed, register /etc/hosts.
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
1.2.3.4 minio01.foo.com
1.2.3.5 minio02.foo.com
1.2.3.6 minio03.foo.com
1.2.3.7 minio04.foo.com
2. Allow ports 9000 and 443 to use MinIO and TLS.
firewall-cmd --zone=public --add-port=9000/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload
3. Install MinIO
3-1. Download the MinIO file.
[wadiz@minio01 ~]# wget https://dl.min.io/server/minio/release/linux-amd64/minio
[wadiz@minio01 ~]# cd /usr/local/bin
[wadiz@minio01 ~]# wget https://dl.min.io/server/minio/release/linux-amd64/minio
[wadiz@minio01 ~]# chmod +x minio
3-2. Create a MinIO configuration.
[wadiz@minio01 ~]# vi /etc/minio/minio.conf
MINIO_CLUSTER = --address :9000 https://minio0{1...4}.foo.com/stg/data/data{1...4}
MINIO_ACCESS_KEY = "MINIO_ACCEESS_KEY"
MINIO_SECRET_KEY = "MINIO_SECRET_KEY"
3-3. Register the system daemon for MinIO management.
[wadiz@minio01 ~]# vi /etc/systemd/system/minio.service
[Unit]
Description=Distributed Minio
Documentation=https://docs.minio.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local
User= username
Group= usergroup
PermissionsStartOnly=true
EnvironmentFile=-/etc/minio/minio.conf
ExecStart=/usr/local/bin/minio server $MINIO_CLUSTER
StandardOutput=journal
StandardError=inherit
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
4. Apply a certificate to minIO for TLS.
The certificate ~/.minio/certs After creating the directory, put private.key and public.crt in it.
[wadiz@minio01 ~]# systemctl daemon-reload
5. Run minio on 4 servers.
[root@minio01 ~]# systemctl start minio
[root@minio02 ~]# systemctl start minio
[root@minio03 ~]# systemctl start minio
[root@minio04 ~]# systemctl start minio
6. Nginx Configuration
nginx domain( minio.foo.com To ensure that requests are forwarded to minio01 ~ 04 when accessed via ), modify the upstream settings in nginx.conf.
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
# 01. minIO 서버 리스트 추가
upstream minio_servers {
server 1.2.3.4:9000;
server 1.2.3.5:9000;
server 1.2.3.6:9000;
server 1.2.3.7:9000;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name dev-minio.wadiz.kr 192.168.2.51;
root /usr/share/nginx/html;
return 301 https://$server_name$request_uri;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
}
error_page 404 /404.html;
location = /404.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
# 02. nginx 443 적용
server {
listen 443 ssl;
listen [::]:443;
# 02-1. nginx domain 설정
server_name minio.foo.com 1.2.3.8;
ignore_invalid_headers off;
proxy_buffering off;
root /usr/share/nginx/html;
# 02-2. 인증서 적용
ssl_certificate "certification.crt 경로";
ssl_certificate_key "certification.key 경로";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 60m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
ssl_prefer_server_ciphers on;
location / {
proxy_pass https://minio_servers;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 1800;
proxy_connect_timeout 1800;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
client_max_body_size 10M;
}
error_page 404 /404.html;
location = /404.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
Run nginx.
[root@nginx ~]# systemctl start nginx
Finally, we completed the installation by accessing the domain address ( minio.foo.com ) and creating the Bucket to use!
minio.foo.com
File Storage Platform Configuration
File Encryption Upload and Download
I securely stored and encrypted the SecretKey through a file service . I configured it so that it could be stored in MinIO.
Backup configuration
I created a 'File Backup Service' that backs up file services. I structured it to run periodically at specific times for backups. I configured it to store encrypted backup data in AWS S3.
In closing
After building the file storage platform, we are using it not only for funding but also for our store and membership services. Currently, it only performs basic functions, but next, we plan to develop it into a more robust file storage platform by adding features such as access logging to view request history for each service bucket and the ability to set access permissions by business unit.
I received a lot of help from various colleagues from the design phase to construction and launch into the operational environment. I also received advice and encouragement on aspects I hadn't even considered. I believe this is a great company to try new things. I look forward to continuing to make many new attempts, gaining experience, and finding a sense of fulfillment. Wadiz I want to become a person 😉
Do you still have any questions? 👀
See how developers work 👉 Click
Wadiz Curious about how project evaluations are conducted? 👉 Click
