FAQ GPFS file systemes

Creation: 18 fevrier 2014
Mise à jour:
Version: 1.0
Author: Jean-Louis Bicquelet-Salaün
Location: http://jlbicquelet.free.fr
Copyright: (c) 2014 Jean-Louis BICQUELET-SALA▄N

GPFS Frequently-Asked Questions

Pour plus d'information au sujet de cette faq, contactez Jean-Louis BICQUELET


FAQ Revised: jeudi 27 f├ęvrier 2014 09:07:56


Table of Contents

1. configuration
2. status
3. suivi
4. disques
5. filesystemes

1. configuration

1.1. comment lister la configuration d'un cluster gpfs?
#mmlsconfig
Configuration data for cluster dev1.server01:
--------------------------------------------------
myNodeConfigNumber 4
clusterName dev1.server01
clusterId 111111111111111111
autoload no
minReleaseLevel 3.5.0.11
pagepool 1G
distributedTokenServer no
maxFilesToCache 3000
maxMBpS 500
unmountOnDiskFail no
worker1Threads 100
prefetchThreads 100
stealFromDoneList false
maxStatCache 12000
tiebreakerDisks VMAX639_121A_TA
adminMode allToAll

File systems in cluster dev1.server01:
-------------------------------------------
/dev/gpfs_app
/dev/gpfs_data


1.2. comment connaître le niveau de gpfs d'un cluster?
lslpp -l |grep -i gpfs
  gpfs.base                 3.5.0.15  COMMITTED  GPFS File Manager
  gpfs.gnr                  3.5.0.11  COMMITTED  GPFS Native RAID
  gpfs.msg.en_US            3.5.0.13  COMMITTED  GPFS Server Messages - U.S.
  gpfs.base                 3.5.0.15  COMMITTED  GPFS File Manager
  gpfs.docs.data             3.5.0.4  COMMITTED  GPFS Server Manpages and


1.3. comment voir la configuration d'un cluster?
#mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         dev1.server01
  GPFS cluster id:           111111111111111111
  GPFS UID domain:           dev1.server01
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    server01
  Secondary server:  (none)

 Node  Daemon node name  IP address     Admin node name  Designation
---------------------------------------------------------------------
   4   server01          192.168.1.69  server01         quorum-manager
   5   server02          192.168.1.70  server02         quorum-manager


1.4.
#cat /var/mmfs/gen/mmsdrfs
%%9999%%:00_VERSION_LINE::1340:3:324::lc:server01::0:/usr/bin/ssh:/usr/bin/scp:111111111111111111:lc2:1206025784::dev1.server01:0:0:0:0:::::0.0:
%%home%%:03_COMMENT::1:
%%home%%:03_COMMENT::2:    This is a machine generated file.  Do not edit!
%%home%%:03_COMMENT::3:
%%home%%:03_COMMENT::4:2013.10.29.12.34.23:274:4:mmadddisk gpfs_app-F new_nsd.out:pre-commit


1.5. comment lister les filesets du gpfs?
mmlsfileset gpfs_apps
Filesets in file system 'gpfs_apps':
Name                     Status    Path
root                     Linked    /apps


1.6. comment connaitre la policy du gpfs?
# mmlspolicy gpfs_app-L
GPFS: 6027-793 No policy file was installed for file system 'gpfs_apps'.



2. status

2.1. comment savoir si un cluster gpfs fonctionne?
Il faut vérifier que le démon mmfsd64 est présent.
#ps -ef | grep mmfs
    root  8733620  8716664   0   Oct 19      -  1:22 tail -100 -f /var/adm/ras/mmfs.log.2013.10.18.16.18.40.server01
    root 13266622        1   0   Oct 19      -  0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs
    root 16102342 13266622   0   Oct 19      - 21:56 /usr/lpp/mmfs/bin/aix64/mmfsd64


2.2. comment conna├«tre l'état des noeuds d'un cluster?
#mmgetstate

 Node number  Node name        GPFS state
------------------------------------------
       4      server01         active
[server01:root]/etc/init.d #mmgetstate -a

 Node number  Node name        GPFS state
------------------------------------------
       4      server01         active
       5      server02         active


2.3. Comment obtenir l'état du cluster?
#mmgetstate -a -L

 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state  Remarks
------------------------------------------------------------------------------------
       1      server021          0        0          2       down        quorum node
       2      server022         1*        1          2       active      quorum node



3. suivi

3.1. comment suivre les événements arrivant à un cluster?
Le fichier des logs du cluster est /var/adm/ras/mmfs.log.latest.
ls -l /var/adm/ras/mmfs.log.latest
lrwxrwxrwx    1 root     system           37 Dec 19 16:19 /var/adm/ras/mmfs.log.latest -> mmfs.log.2013.10.18.16.18.40.server01
[server01:root]/etc/init.d #tail -f /var/adm/ras/mmfs.log.latest
Thu Dec 19 16:19:50.853 2013: GPFS: 6027-1711 Connected to 192.168.1.10 server02 
Thu Dec 19 16:19:50.857 2013: GPFS: 6027-2750 Node 192.168.1.10 (server02) is now the Group Leader.
Thu Dec 19 16:19:51.112 2013: GPFS: 6027-300 mmfsd ready
Thu Dec 19 16:19:51 NFT 2013: mmcommon mmfsup invoked. Parameters: 192.168.1.09 192.168.1.10 all
Thu Dec 19 16:19:59.578 2013: Command: mount gpfs_app11075710
Thu Dec 19 16:20:01.950 2013: Command: err 0: mount gpfs_app11075710
Thu Dec 19 16:20:02.185 2013: Command: mount gpfs_data 11075716
Thu Dec 19 16:20:02.192 2013: Command: err 0: mount gpfs_data 11075716
Wed Jan 29 11:56:24.357 2014: Command: mmfsck /dev/gpfs_data -o
Wed Jan 29 11:56:39.718 2014: Command: err 0: mmfsck /dev/gpfs_data -o


3.2. comment lister les attributs des erreurs?
#mmlsattr -L ./dsmerror.log
file name:            ./dsmerror.log
metadata replication: 1 max 2
data replication:     1 max 2
immutable:            no
appendOnly:           no
flags:
storage pool name:    system
fileset name:         root
snapshot name:
Windows attributes:   NORMAL



4. disques

4.1. comment lister les gpfs et les LUNS d'un cluster?
#mmlsnsd

 File system   Disk name    NSD servers
---------------------------------------------------------------------------
 gpfs_app      VMAX639_21A8_14 server01,server02
 gpfs_app      VMAX639_21A9_14 server01,server02
 gpfs_app      VMAX639_21AA_14 server01,server02
 gpfs_data     VMAX639_21AC_14 server01,server02
 gpfs_data     VMAX639_21AD_14 server01,server02
 gpfs_data     VMAX639_21AF_14 server01,server02
 (free disk)   VMAX639_121A_TA server01,server02


4.2. comment associer les LUNS et les disques d'un cluster?
#mmlsnsd -m

 Disk name    NSD volume ID      Device         Node name                Remarks
---------------------------------------------------------------------------------------
 VMAX639_21A8_14 0BB0030100000006   /dev/hdisk29   server01                 server node
 VMAX639_21A8_14 0BB0030100000006   /dev/hdisk3    server02                 server node
 VMAX639_21A9_14 0BB0030100000007   /dev/hdisk30   server01                 server node
 VMAX639_21A9_14 0BB0030100000007   /dev/hdisk4    server02                 server node
 VMAX639_21AA_14 0BB0030100000008   /dev/hdisk31   server01                 server node
 VMAX639_21AA_14 0BB0030100000008   /dev/hdisk5    server02                 server node
...



5. filesystemes

5.1. comment lister les disques gpfs d'un cluster?
#mmlsfs all -P

File system attributes for /dev/gpfs_apps:
==========================================
flag                value                    description
------------------- ------------------------ -----------------------------------
 -P                 system                   Disk storage pools in file system

File system attributes for /dev/gpfs_data:
==========================================
flag                value                    description
------------------- ------------------------ -----------------------------------
 -P                 system                   Disk storage pools in file system


5.2.
#mmlsfs all -d

File system attributes for /dev/gpfs_apps:
==========================================
flag                value                    description
------------------- ------------------------ -----------------------------------
 -d                 VMAX639_21A8_14;VMAX639_21A9_14;VMAX639_21AA_14;VMAX639_21AB_14  Disks in file system

File system attributes for /dev/gpfs_data:
==========================================
flag                value                    description
------------------- ------------------------ -----------------------------------
 -d                 VMAX639_21AC_14;VMAX639_21AD_14;VMAX639_21AE_14;VMAX639_21AF_14  Disks in file system
#mmlsnsd -M

 Disk name    NSD volume ID      Device         Node name                Remarks
---------------------------------------------------------------------------------------
 VMAX639_21A8_14 0BB0030100000006   /dev/hdisk29  server01                 server node
 VMAX639_21A8_14 0BB0030100000006   /dev/hdisk3    server02                 server node
 VMAX639_21A9_14 0BB0030100000007   /dev/hdisk30   server01                 server node
 VMAX639_21A9_14 0BB0030100000007   /dev/hdisk4    server02                 server node
 VMAX639_21AA_14 0BB0030100000008   /dev/hdisk31   server01                 server node
 VMAX639_21AA_14 0BB0030100000008   /dev/hdisk5    server02                 server node
 VMAX639_21AE_14 0BB0030100000011   /dev/hdisk36   server01                 server node
...


5.3. comment savoir sur quels serveurs sont montés les gpfs?
 #mmlsmgr
file system      manager node
---------------- ------------------
gpfs_data        192.168.1.09 (server01)
gpfs_app         192.168.1.10 (server02)

Cluster manager node: 192.168.1.10 (server02)


5.4. comment connaître les montages gpfs d'un cluster?
#mmlsmount all
File system gpfs_appis mounted on 2 nodes.
File system gpfs_data is mounted on 2 nodes.


5.5. comment afficher les filesystemes ?
Avec df tout simplement.
df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         1048576    590672   44%     9775    13% /
/dev/hd2         7733248   2677048   66%    50874    14% /usr
/dev/hd9var      2097152   1689136   20%     4217     3% /var
/dev/hd3         8388608   6074048   28%     2944     1% /tmp
/dev/hd1         1048576   1045680    1%      161     1% /home
/dev/hd11admin     262144    261384    1%        5     1% /admin
/proc                  -         -    -         -     -  /proc
/dev/hd10opt     2621440   1289176   51%     8241     6% /opt
/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump
/dev/lv_syslog    4194304   4190144    1%       40     1% /var/log/syslog
/dev/sysload     1048576    715928   32%       82     1% /opt/sysload
/dev/itcmlv       262144     55280   79%     1201    16% /opt/CA
/dev/gpfs_app 209717248 159305728   25%   100270    21% /apps
/dev/gpfs_data  209717760 155372928   26%    18409     1% /data


5.6. comment obtenir la taille des filesystèmes?
#mmdf gpfs_apps
disk                disk size  failure holds    holds              free KB             free KB
name                    in KB    group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 46 GB)
VMAX639_21AB_14      26214720     2000 yes      yes        19912448 ( 76%)        215040 ( 1%)
VMAX639_21AA_14      26214720     2000 yes      yes        19927296 ( 76%)        247280 ( 1%)
VMAX639_21A9_14      26214720     2000 yes      yes        19907072 ( 76%)        253976 ( 1%)
VMAX639_21A8_14      26214720     2000 yes      yes        19906048 ( 76%)        267160 ( 1%)
                -------------                         -------------------- -------------------
(pool total)        104858880                              79652864 ( 76%)        983456 ( 1%)

                =============                         ==================== ===================
(total)             104858880                              79652864 ( 76%)        983456 ( 1%)

Inode Information
-----------------
Number of used inodes:          100270
Number of free inodes:          248914
Number of allocated inodes:     349184
Maximum number of inodes:       500224


5.7. Comment démonter les filesystemes?
mmumount all -f

5.8. Comment monter les filesystemes en écriture?
Par défaut gpfs monte les filesystemes en lecture seule. Il faut préciser si on veut écrire sur les filesystèmes.

mmmount all -o rw

5.9. Comment obtenir la liste des filesystemes montés?
mmlsmount all -L