Simple Application of distributed File system mogilefs
Brief introduction
MogileFS is an open source distributed file storage system developed by DangaInteractive, a company owned by LiveJournal. The Danga team has developed a number of well-known open source projects, including Memcached, MogileFS, Perlbal and so on.
Characteristics
1. Application layer: user space file system without special core components
2. No single point: (tracker, mogstore, database (MySQL))
3. Automatic file replication: the smallest unit of copy is not a file, but class
4. Transport neutrality, no special protocol: you can communicate through NFS or HTTP
5. Simple namespace: each file corresponds to a key: used for domain to define namespaces
component
MogileFS:
Tracker: tracking metadata
Database: storing metadata
Storage: storing data
Tracker:
Mogilefsd (daemon), responsibilities:
Replication: file replication between nodes
Deletion: deleting fil
Queryworker: file metadata access requests in response to customer requests
Reaper: replaces file replication requests in the queue after storage failure
Monitor: check the health status of hosts and devices
Database:
Mysql is generally used to store the metadata of mogilefs. It is recommended to use redundancy scheme to ensure availability (MMM,MHA).
Mogilefs provides a data structure management tool, mogdbsetup.
Storage:
Mogstored (process name), a ready mogstored node can be added to the current cluster through the mogadm command
Storage nodes need to define "device dev" as storage space; each device needs to be identified by a unique DevID in the current cluster.
Client:
The client is used to communicate with mogilefs resumes to complete data access.
Install the package:
MogileFS-Server-2.46-2.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
Perl-MogileFS-Client-1.14-1.el6.noarch.rpm
Perl-Net-Netmask-1.9015-8.el6.noarch.rpm
Perl-Perlbal-1.78-1.el6.noarch.rpm
Architecture:
1. Install mogilefs and database
# ls
MogileFS-Server-2.46-2.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm
MogileFS-Utils-2.19-1.el6.noarch.rpm
# yum localinstall * .rpm-y
# yum install mysql-server mysql- y
two。 The database authorizes remote access and inherits authorization permissions
Mysql > grant all on *. * to root@'192.168.95.%' identified by 'liaobin' with grant option
Mysql > flush privileges
3. Create the database user and authorization for mofilefs
Mysql > grant all on *. * to mguser@'192.168.95.%' identified by 'liaobin'
Mysql > flush privileges
4. Initialize mogilefs database with mogdbsetup, a tool that comes with mogilefs
# mogdbsetup-dbhost=192.168.95.30-dbname=mogilefs-dbrootuser=root-dbrootpass=liaobin-dbuser=mguser-dbpass=liaobin-yes
View the initialized database
Mysql > show databases
+-+
| | Database |
+-+
| | information_schema |
| | mogilefs |
| | mysql |
| | test |
+-+
4 rows in set (0.01sec)
5. Modify the tracker (mogilefsd) configuration file
# vim / etc/mogilefs/mogilefsd.conf
# Enable daemon mode to work in background and use syslog
Daemonize = 1
# Where to store the pid of the daemon (must be the same in the init script)
Pidfile = / var/run/mogilefsd/mogilefsd.pid
# Database connection information
Db_dsn = DBI:mysql:mogilefs:host=192.168.95.30 # configure connection information for database
Db_user = mguser # configure the database user password for mogilefs
Db_pass = liaobin
# IP:PORT to listen on for mogilefs client requests
Listen = 0.0.0.0 7001 # configure the address and port for listening
# Optional, if you don't define the port above.
Conf_port = 7001
# Number of query workers to start by default.
Query_jobs = 10 # how many worker processes are generated for queries
# Number of delete workers to start by default.
Delete_jobs = 1 # how many worker processes are generated for deletion
# Number of replicate workers to start by default.
Replicate_jobs = 5 # how many worker processes are generated for deletion
# Number of reaper workers to start by default.
# (you don't usually need to increase this)
Reaper_jobs = 1
# Number of fsck workers to start by default.
# (these can cause a lot of load when fsck'ing)
# fsck_jobs = 1
# Minimum amount of space to reserve in megabytes
# default: 100
# Consider setting this to be larger than the largest file you
# would normally be uploading.
# min_free_space = 200
# Number of seconds to wait for a storage node to respond.
# default: 2
# Keep this low, so busy storage nodes are quickly ignored.
# node_timeout = 2
# Number of seconds to wait to connect to a storage node.
# default: 2
# Keep this low so overloaded nodes get skipped.
# conn_timeout = 2
# Allow replication to use the secondary node get port
# if you have apache or similar configured for GET's
# repl_use_get_port = 1
6. Start the mogilefsd service
[root@localhost mgfs] # / etc/init.d/mogstored start
7. Add a new partition for each node as storage
# fdisk / dev/sda
.
# partx-a / dev/sda
# mkfs.ext4 / dev/sda5
# mkdir / mgfs
# vim / etc/fstab
# add the following entries
/ dev/sda5 / mgfs ext4 defaults 0 0
# mount-a
# mount
/ dev/sda5 on / mgfs type ext4 (rw)
8. The three servers create storage directories and modify permissions
95.30:
# mkdir / mgfs/dev1
# chown-R / mgfs/dev1
95.34:
# mkdir / mgfs/dev2
# chown-R / mgfs/dev2
95.35:
# mkdir / mgfs/dev3
# chown-R / mgfs/dev3
9. Modify the configuration file of store (mogstored)
# vim / etc/mogilefs/mogstored.conf
Maxconns = 10000
Httplisten = 0.0.0.0pur7500
Mgmtlisten = 0.0.0.0purl 7501
# Note here, just write to the superior directory of dev (1 ~ 2 ~ 3).
Docroot = / mgfs
10. Start the mogstored service
# / etc/init.d/mogstored start
11. Manage with mogadm
Create a storage host:
# mogadm-trackers=192.168.95.30:7001 host add store1-ip=192.168.95.30
# mogadm-trackers=192.168.95.30:7001 host add store2-ip=192.168.95.34
# mogadm-trackers=192.168.95.30:7001 host add store3-ip=192.168.95.35
View the created host:
# mogadm-trackers=192.168.95.30:7001 host list
Create a storage device:
# mogadm-trackers=192.168.95.30:7001 device add store1 1-status=alive
# mogadm-trackers=192.168.95.30:7001 device add store2 2-status=alive
# mogadm-- trackers=192.168.95.30:7001 device add store3 3-- status=alive
View the created storage device:
# mogadm-trackers=192.168.95.30:7001 device list
Create a domain domain:
Mogadm-trackers=192.168.95.30:7001 domain add img
Mogadm-trackers=192.168.95.30:7001 domain add conf
View the created domain:
Mogadm-trackers=192.168.95.30:7001 domain list
Upload file test:
Mogupload-trackers=192.168.95.30:7001-domain=img-key= "1.jpg"-file= "/ root/1.jpg"
View the uploaded file:
# mogfileinfo-trackers=192.168.95.30:7001-domain=img-key= "1.jpg"
-file: 1.jpg
Class: default
Devcount: 2
Domain: img
Fid: 9
Key: 1.jpg
Length: 34892
-http://192.168.95.35:7500/dev3/0/000/000/0000000009.fid
-http://192.168.95.30:7500/dev1/0/000/000/0000000009.fid
Access the test with fid's url:
Normal access
Nginx reverse proxy mogilefs
1. Install nginx and mogilefs modul
Install the development environment:
# yum groupinstall "Development tools"Server Platform Development"-y
# yum install pcre-devel-y
Download and extract the nginx package and the mogilefs package:
Start compiling and installing:
# cd nginx-1.6.2
#. / configure\
-- prefix=/usr/local/nginx\
-- sbin-path=/usr/local/nginx/sbin/nginx\
-- conf-path=/etc/nginx/nginx.conf\
-- error-log-path=/var/log/nginx/error.log\
-- http-log-path=/var/log/nginx/access.log\
-- pid-path=/var/run/nginx/nginx.pid\
-- lock-path=/var/lock/nginx.lock\
-- user=nginx\
-- group=nginx\
-- with-http_ssl_module\
-- with-http_flv_module\
-- with-http_stub_status_module\
-- with-http_gzip_static_module\
-- http-client-body-temp-path=/var/tmp/nginx/client/\
-- http-proxy-temp-path=/var/tmp/nginx/proxy/\
-- http-fastcgi-temp-path=/var/tmp/nginx/fcgi/\
-- http-uwsgi-temp-path=/var/tmp/nginx/uwsgi\
-- http-scgi-temp-path=/var/tmp/nginx/scgi\
-with-pcre-add-module=../nginx_mogilefs_module-1.0.4
# make & & make install
Create a service script for nginx
# vim / etc/init.d/nginx
#! / bin/sh
#
# nginx-this script starts and stops the nginx daemon
#
# chkconfig:-85 15
# description: Nginx is an HTTP (S) server, HTTP (S) reverse\
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: / etc/nginx/nginx.conf
# config: / etc/sysconfig/nginx
# pidfile: / var/run/nginx.pid
# Source function library.
. / etc/rc.d/init.d/functions
# Source networking configuration.
. / etc/sysconfig/network
# Check that networking is up.
["$NETWORKING" = "no"] & & exit 0
Nginx= "/ usr/local/nginx/sbin/nginx"
Prog=$ (basename $nginx)
NGINX_CONF_FILE= "/ etc/nginx/nginx.conf"
[- f / etc/sysconfig/nginx] & &. / etc/sysconfig/nginx
Lockfile=/var/lock/subsys/nginx
Make_dirs () {
# make required directories
User= `nginx-V 2 > & 1 | grep "configure arguments:" | sed's / [^ *] *-- user=\ ([^] *\). * /\ 1Uniple g'- `
Options= `$ nginx-V 2 > & 1 | grep 'configure arguments:' `
For opt in $options; do
If [`echo $opt | grep'. *-temp-path' `]; then
Value= `echo $opt | cut-d "="-f 2`
If [!-d "$value"]; then
# echo "creating" $value
Mkdir-p $value & & chown-R $user $value
Fi
Fi
Done
}
Start () {
[- x $nginx] | | exit 5
[- f $NGINX_CONF_FILE] | | exit 6
Make_dirs
Echo-n $"Starting $prog:"
Daemon $nginx-c $NGINX_CONF_FILE
Retval=$?
Echo
[$retval-eq 0] & & touch $lockfile
Return $retval
}
Stop () {
Echo-n $"Stopping $prog:"
Killproc $prog-QUIT
Retval=$?
Echo
[$retval-eq 0] & & rm-f $lockfile
Return $retval
}
Restart () {
Configtest | | return $?
Stop
Sleep 1
Start
}
Reload () {
Configtest | | return $?
Echo-n $"Reloading $prog:"
Killproc $nginx-HUP
RETVAL=$?
Echo
}
Force_reload () {
Restart
}
Configtest () {
$nginx-t-c $NGINX_CONF_FILE
}
Rh_status () {
Status $prog
}
Rh_status_q () {
Rh_status > / dev/null 2 > & 1
}
Case "$1" in
Start)
Rh_status_q & & exit 0
, 1
Stop)
Rh_status_q | | exit 0
, 1
Restart | configtest)
, 1
Reload)
Rh_status_q | | exit 7
, 1
Force-reload)
Force_reload
Status)
Rh_status
Condrestart | try-restart)
Rh_status_q | | exit 0
*)
Echo $"Usage: $0 {start | stop | status | restart | condrestart | try-restart | reload | force-reload | configtest}"
Exit 2
Esac
Give executive authority:
# chmod + x / etc/init.d/nginx
Create nginx users and required directories:
# useradd-r nginx
# mkdir / var/tmp/nginx/client-pv
Modify the nginx configuration file to support mogilefs reverse proxy (red font is an addition):
# vim / etc/nginx/nginx.conf
# user nobody
Worker_processes 1
# error_log logs/error.log
# error_log logs/error.log notice
# error_log logs/error.log info
# pid logs/nginx.pid
Events {
Worker_connections 1024
}
Http {
Include mime.types
Default_type application/octet-stream
# log_format main'$remote_addr-$remote_user [$time_local] "$request"'
#'$status $body_bytes_sent "$http_referer"'
#'"$http_user_agent"$http_x_forwarded_for"'
# access_log logs/access.log main
Sendfile on
# tcp_nopush on
# keepalive_timeout 0
Keepalive_timeout 65
# gzip on
Upstream stores {
Server 192.168.95.30:7001
Server 192.168.95.34:7001
Server 192.168.95.35:7001
}
Server {
Listen 80
Server_name localhost
# charset koi8-r
# access_log logs/host.access.log main
Location / {
Root html
Index index.html index.htm
}
Location / img/ {
Mogilefs_tracker stores
Mogilefs_domain img
Mogilefs_pass {
Proxy_pass $mogilefs_path
Proxy_hide_header Content-Type
Proxy_buffering off
}
}
Location / upload/ {
Mogilefs_tracker stores
Mogilefs_domain img
Mogilefs_methods PUT GET DELETE
Mogilefs_pass {
Proxy_pass $mogilefs_path
Proxy_hide_header Content-Type
Proxy_buffering off
}
}
# error_page 404 / 404.html
}
Test the nginx agent with path access:
Upload another file to test:
# mogupload-trackers=192.168.95.30:7001-domain=img-key= "2.jpg"-file= "/ root/2.jpg"
Test delete files:
# curl-X DELETE http://192.168.95.34/upload/1.jpg
