File Systems on DIaL3

There are a number of file systems on DIaL3 that should be used in different ways:

Areas

/home

Every user has a home directory within the /home file system. This will be your current working directory when you login to DIaL3.

The home directory should only be used to store program code, job submission scripts, configuration files and small amounts of data.

Your home directory can be referenced on the Linux command line with the shorthand ~. The simple command cd with no arguments will return you to your home directory.

/scratch

You will have a scratch directory on the scratch file system for each project that you are associated with.

The location will be /scratch/project/username - so users in multiple projects will have several to choose from. As each user's scratch directory is readable by other members of the same project, it's important to choose the correct one depending on which project is being worked on.

The directory /scratch/project/shared has special permissions to ensure that all files within are always owned by the project group.

The scratch directory used should be the main location for job files, and generally should be used as the working directory for jobs. The scratch space has quotas applied which are in line with those requested for “work” in DiRAC time applications. These are not allocations, only quotas, so can add up to more than the available storage.

Warning: Files within /scratch are not backed up.

/tmp

Each compute node has a small amount of local disc mounted on /tmp. For some jobs there may be a performance gain over /tmp in using this file system for intermediate files.

Standard compute nodes have 100GB available. The preferred way to use /tmp within a job is to refer to it using the environment variable TMPDIR. This way the job's files are cleaned-up when the job finishes automatically. Otherwise it is your job's responsibility to remove files from the local file system when it ends.

Warning: The /tmp file system is not backed up, and the contents are deleted whenever the compute node reboots for any reason.

Files on /tmp should only be considered safe for the duration of the job which they belong to.

Quotas

The default quota for a user's home directory is 100GB.

The default quota for a project is 10TB, however, individual projects may have be awarded larger quotas by the DiRAC Resource Allocation Committee.

Every user and group also has a 'file count' limit of 8M files. This may be increased on request with a suitable justification.

You can check your personal or project quotas using the command:

lfs  quota -h -p <ID>  /lustre/dirac3/

You can get your the IDs of your account and the groups you belond to using the linux id command:

  id
  uid=210110(dc-user1) gid=59019(dirac) groups=59019(dirac),70005(dp031),70009(dp033)

You can then use these IDs to query your quotas. For example, to query the quotas associated with project dp031 (which has ID 70005),

lfs  quota -h -p 70005 /lustre/dirac3/
Disk quotas for prj 70005 (pid 70005):
          Filesystem    used   quota   limit   grace   files    quota    limit grace
     /lustre/dirac3/  46.05T      0k      0k       - 7915433  8000000 60000000     -

Or to check the user quota for this account:

  lfs quota -h -p 210110 /lustre/dirac3/

  Disk quotas for prj 210110 (pid 210110):
       Filesystem    used   quota   limit   grace   files   quota   limit   grace
  /lustre/dirac3/  646.7M     80G    100G       -   21160  800000 1000000       -