Advanced Computational Laboratory

Front End(temporal)

  • External IP:
  • Internal IP:
  • External Name:
  • H/W : SunFire Z20v - AMD Opteron 250 / 4 Gb Ram
  • OS : Ubuntu Server 14.04
  • eth-pujc : (eth0)
  • eth-cluster : (eth1)

By default (H/W RAID)

  • SCSI (2,0,0) (sda) 2.0 TB
  • SCSI (2,1,0) (sdb) 1.6 TB


  • sda
    • swap : 20 GB
    • /boot : 4 GB
    • / : 500 GB
    • /vol-user : (PVFS/OrageFS)
  • sdb
    • /vol-scratch : (PVFS/OrangeFS)

Cluster network :

Nodes (temporal)

  • tmp-node1 :
  • tmp-node0 : (dhcp) Sun Fire
Forward traffic
# Delete and flush. Default table is "filter". Others like "nat" must be explicitly stated.
iptables --flush            # Flush all the rules in filter and nat tables
iptables --table nat --flush
iptables --delete-chain     # Delete all chains that are not in the default filter and nat table
iptables --table nat --delete-chain
# Set up IP FORWARDing and Masquerading
iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
iptables --append FORWARD --in-interface eth1 -j ACCEPT	 
echo 1 > /proc/sys/net/ipv4/ip_forward             # Enables packet forwarding by kernel

Mac IP:

Cluster File System

Use ZFS and LUSTRE for the workers and front-end.

ZFS offers consistency and support NFS ( and The idea is mount user's home directory from front-end into the workers.

LUSTRE will make a distributed file system from the worker's storage (SAS HDD). Linux kernel has support for lustre's kernel modules. There are some tools that require been build (check and A problem of LUSTRE is that data storage and clients are not recommended to run on the same machine for memory consumption problems.

Another option for distributed file system among the workers is ceph ( ceph is already in the ubuntu repository. It support Hadoop ( through a Haddop Ceph plugin.

One more to check is Gluster ( There is an official ppa repository and looks like works with NFS and ZFS.

Process Batch system

For process batch system use slurm resource manager ( HTCONDOR could be used to submit jobs for grid type of computation.

System management and deployment

There are few alternatives for system configuration and management.

To Do

  • Setup network definition
  • Setup ntp to syncronize clocks amoung nodes
  • Configure slurm ( (Missing accounting to configure)
  • Install and configure lustre Replace lustre in favor of CEPH
  • Install environment_module to select the libraries to use
  • Setup /sw to be shared between the nodes
  • Setup a git (or similar for example bitbucket) repository to keep the node's configurations (/etc files mainlly)
  • (Re)install the nodes
  • Install and configure magpie to unify hpc and big data (
  • Update Ubuntu 12.04 to 14.04
  • Admin tasks (clean users and directories)

IP address

The IP ranges for each network. It depends on its use in the cluster. The IPs has the form 192.168.XXX.YYY, where XXX and YYY are defines as follow:

XXX YYY Interface postfix
Communication 10Gbit 201 10-250 cn (Communication Network)
Communication 1Gbit 202 10-250 cn
File system Hydra 101 10-250 fsn (File system Network)
File system Lili 102 10-250 fsn
Administration 10 10-250 YYY=1: lca-admin

The front-end address (YYY) is 10.

The IP assignment is:

hostname cn vlan fsn0 vlan eth0 vlan Description
hfe 4 2 1 Hydra's front-end
hnode1 4 2 1 Hydra's node 1
hnode2 4 2 1 Hydra's node 2
hnode3 4 2 1 Hydra's node 3
hnode4 4 2 1 Hydra's node 4
lfe 5 3 1 Lili's front-end
lnode1 5 3 1 Lili's node 1
lnode2 5 3 1 Lili's node 2
lca-admin 2 1 General Administration node 3

Hostname assignment (cluster):

# Hydra's front-end    hydra-adm hfe   hydra hfe-cn # Communication (MPI)   hfe-fsn

# Hydra's node 1    hnode1-adm   hnode1      # Communication (MPI)   hnode1-fsn

# Hydra's node 2    hnode2-adm   hnode2      # Communication (MPI)   hnode2-fsn

# Hydra's node 3    hnode3-adm   hnode3      # Communication (MPI)   hnode3-fsn

# Hydra's node 4    hnode4-adm   hnode4      # Communication (MPI)   hnode4-fsn
# Lili's front-end    lili lfe   lfe-cn   lfe-fsn

# Lili's node 1    lnode1-adm   lnode1      # Communication (MPI)   lnode1-fsn

# Lili's node 2    lnode2-adm   lnode2      # Communication (MPI)   lnode2-fsn
# General administration node     lca lca-admin gc1  hydra-fsn  lili-fsn

# Other machines connected to the cluster network    Dell swtich 10G Ethernet   babbage   UPS (needs a new IP address)



  • If sinfo shows nodes in DOWN* state, check the clock synchronization between nodes and hydra (run sudo ntpdate hydra to sync on each worker)
clush -b -w hnode[1-4] sudo ntpdate hydra
  • If sinfo shows a node with DOWN state (without *), the node can be resumed running on hydra (Change hnode with the correct node name).
scontrol update NodeName=hnode State=RESUME
  • clush
hpccluster.txt · Última modificación: 2016/10/13 15:31 por callanor
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki