High Performance Computing Service: Difference between revisions

From Brain Mapping Unit
Jump to navigationJump to search
No edit summary
No edit summary
 
(21 intermediate revisions by the same user not shown)
Line 2: Line 2:
This page is a work in progress!
This page is a work in progress!


I'm just using it as a repository for helpful hints that I've found useful!
I wrote it in 2013 and it hasn't been updated since then. The HPC documentation has been though so you're probably best starting there and editing this page as you find out useful things!
 
Kirstie x
Kirstie x


== Introduction ==
== Introduction ==
If you have many jobs that are very similar and, importantly, '''independent''' then you can use the High Performance Computing Service [[http://www.hpc.cam.ac.uk/]]. The website is helpful, but here are a few commands that I think supplement their FAQs:
If you have many jobs that are very similar and, importantly, ''independent'' then you can use the High Performance Computing Service [[http://www.hpc.cam.ac.uk/]].
 
The website is helpful, but below are a few commands that I think supplement their FAQs. Please do edit (or email me and I'll update the wiki: kw401@cam.ac.uk) if you have any further suggestions.
 
 
== Getting an account ==
 
As with everything in Cambridge there's some paperwork to fill in. Download the application form from here: http://www.hpc.cam.ac.uk/services/applying-for-resources, wait a week and then you're off.
 
 
== Logging in ==
 
If you're ''inside the Cambridge firewall'' then you can connect via putty to:
 
    '''[USERNAME]@login.hpc.cam.ac.uk'''
    eg: kw401@login.hpc.cam.ac.uk
 
You can then create a vncserver session by following these instructions: http://www.hpc.cam.ac.uk/user/visualization.html.
 
If you're ''outside the Cambridge firewall'' (at home or in a coffee shop) then you should connect to your BCNI vnc session (eg via VNC Viewer or Chicken of the VNC) and then type:
 
    '''ssh -X [USERNAME]@login.hpc.cam.ac.uk'''
    eg: ssh -X kw401@login.hpc.cam.ac.uk
 
This will transport you - in the same terminal window - to the HPC and allow you to use the graphics installed there.
 
Don't forget that if you open a ''new terminal window'' while logged into the BCNI that terminal will be connected to the BCNI! You'll have to ssh again :)


== If you want to see all the jobs you've submitted ==
== If you want to see all the jobs you've submitted ==
As you may guess, qstat gives you some statistics on the queues. To look for just '''your''' jobs type:
As you may guess, qstat gives you some statistics on the queues. To look for just ''your'' jobs type:


qstat -u <user_id>
qstat -u $USER
eg: qstat -u kw401
 
This $USER variable contains your user ID.


== If you want to delete all the jobs you've submitted ==
== If you want to delete all the jobs you've submitted ==
To just get the job IDs you need to use qselect and pipe these to qdel (which deletes the job, obviously) using xargs.
To just get the job IDs you need to use qselect and pipe these to qdel (which deletes the job) using xargs.


qselect -u <user_id> | xargs qdel
qselect -u $USER | xargs qdel
eg: qselect -u kw401 | xargs qdel


Type man xargs if you want to know more about this terribly useful command which pipes information from standard input into the next command.
Type man xargs if you want to know more about this terribly useful command which pipes information from standard input into the next command.
More specifically if you want to delete only your running jobs:
qselect -u $USER -s R | xargs qdel
Or if you want to delete only your queued jobs:
qselect -u $USER -s Q | xargs qdel

Latest revision as of 08:42, 7 April 2016

Disclaimer

This page is a work in progress!

I wrote it in 2013 and it hasn't been updated since then. The HPC documentation has been though so you're probably best starting there and editing this page as you find out useful things!

Kirstie x

Introduction

If you have many jobs that are very similar and, importantly, independent then you can use the High Performance Computing Service [[1]].

The website is helpful, but below are a few commands that I think supplement their FAQs. Please do edit (or email me and I'll update the wiki: kw401@cam.ac.uk) if you have any further suggestions.


Getting an account

As with everything in Cambridge there's some paperwork to fill in. Download the application form from here: http://www.hpc.cam.ac.uk/services/applying-for-resources, wait a week and then you're off.


Logging in

If you're inside the Cambridge firewall then you can connect via putty to:

   [USERNAME]@login.hpc.cam.ac.uk
   eg: kw401@login.hpc.cam.ac.uk

You can then create a vncserver session by following these instructions: http://www.hpc.cam.ac.uk/user/visualization.html.

If you're outside the Cambridge firewall (at home or in a coffee shop) then you should connect to your BCNI vnc session (eg via VNC Viewer or Chicken of the VNC) and then type:

   ssh -X [USERNAME]@login.hpc.cam.ac.uk
   eg: ssh -X kw401@login.hpc.cam.ac.uk

This will transport you - in the same terminal window - to the HPC and allow you to use the graphics installed there.

Don't forget that if you open a new terminal window while logged into the BCNI that terminal will be connected to the BCNI! You'll have to ssh again :)

If you want to see all the jobs you've submitted

As you may guess, qstat gives you some statistics on the queues. To look for just your jobs type:

qstat -u $USER

This $USER variable contains your user ID.

If you want to delete all the jobs you've submitted

To just get the job IDs you need to use qselect and pipe these to qdel (which deletes the job) using xargs.

qselect -u $USER | xargs qdel

Type man xargs if you want to know more about this terribly useful command which pipes information from standard input into the next command.

More specifically if you want to delete only your running jobs:

qselect -u $USER -s R | xargs qdel

Or if you want to delete only your queued jobs:

qselect -u $USER -s Q | xargs qdel