Charles Bos --- Using Rotterdam

Using Rotterdam

`Rotterdam' or `rdam' is the name of the server financed by the Department of Econometrics & O.R., or NWO, at the VU University Amsterdam. Members of the department can get an account from Laurent Callot or Charles Bos. The server is meant for all kind of computational jobs, as long as they can run on Linux.

Some definitions

`Rdam' is the machine for which you got an account.
It is known by IP address 145.108.238.26. It is reachable from outside the university as well, at this address.
You received your own login and password. Your login is personal, you cannot give your password to anybody else.
In the text below, I'll use VUNetID (abc123) as example. Replace it with your login where needed.
The present machine has 2 processors of 8 cores each, and is capable of hyperthreading on each core. Hence, you could run up to 32 processes to use the full machine. There is 32GB of memory, and plenty of local hard disk space. Furthermore, the machine contains a Tesla card, for GPGPU computations.
Each user has a limited amount of priority, e.g. if two users are working at the machine usually each is limited to 50% of the cpu-time, if there is a conflict. For the memory, there is no automatic limit. Please make sure you do not use too much memory, else the machine becomes very slow. On the other hand, 32GB seems to be enough for plenty of programs.

Entering the machine

You can enter the machine using a secure shell program, e.g. PuTTY. At the VU, send an email to ucit-servicedesk@vu.nl that you want access to PuTTY (if you don't have it already). At home, download and install http://the.earth.li/~sgtatham/putty/latest/x86/putty.zip.

Start PuTTY, fill in as host name

  145.108.238.26

and save the session under the name `Rdam'. Then hit the Open button.

Fill in your login and password. If this is the first time you login, you will be asked to change your password, do this using

  passwd

You would do well to change it to your VU password.

When you finish with PuTTY, type

  exit

  ctrl-d

or just kill the window.

For just getting to the files (from home) you could use WinSCP or FileZilla. This gives you a graphical environment to copy files to and from Rdam. FileZilla is also available on the VU.

Using your own files

After the migration (and if I have set up things straight for you, complain in case of errors), you can use

  mount ~/hdrive

to mount your h: drive. The command may ask for your username (if I didn't set it for you already). This would be

  vu/abc123

(ie vu/ followed by your vu-net id, don't forget the trailing vu/!). The password is the standard password you have on the VU network.

Lately this link to the network drive seems to work. Only with many large files at once I have some problems at times. If all else fails, work in a local directory.

With a

  cd hdrive/oxprogs

you would get into the

  h:\oxprogs

directory (assuming you have one, of course). Note the forward slashes in directory names.

If you would wish to unmount the drive, you could use

  cd
  umount ~/hdrive

Using local files

You could use local data files (which might go slightly quicker if the files are very large) or save your output locally. All files which are not saved underneath the /home/abc123/hdrive directory will end up locally. For the moment, feel free to use a couple of GBs, please ask if you need more.

Running commands

On the machine you can use Ox, gcc, octave, matlab and possibly other programs as well. Run e.g. Ox using

  cd
  cd hdrive/oxprogs
  oxl myprog

if all packages would be available. Ox on Rdam has the standard packages around, but you can add extra ones by creating

  h:\bin\ox\packages\oxprob

for instance (the $HOME/bin/ox/ directory is, on Rdam, automatically added to your OX7PATH variable). If you place the oxprob package there, creating the necessary directories, Ox should be able to find the files. Note that also Ox on your Windows machine at the VU will search this same directory, hence also on your desktop you should be able to use this same package.

Command control

You can run a job in the background, by typing

  oxl myprog &

This way, you could even run multiple jobs at once. If you have a job running, you can leave PuTTY using

  ctrl-d

(if you leave PuTTY any other way, your job will stop).

When you login to Rdam anew, you can use the command

  ps x

to see a list of jobs you have running, or use

top

to see the most active jobs of yourself and others. Stop the `top' with ctrl-c.

To run multiple jobs on Rdam, you can start working with bash-scripts. To track the number of jobs you have running, you could use something like

  for i in `seq 1 100`; do 
    oxl mysim seed $i &
    checkcpuload
  done
  wait

which would start 100 iterations of mysim, each with a different command line argument for the seed. The checkcpuload will could the number of Ox jobs you have running, and wait until a core is free. At the end, the wait command will wait until all jobs are finished.

Here I use the local command checkcpuload. With a

  checkcpuload test

you get some output on screen on the number of Ox jobs you seem to have running already. Note that checkcpuload is a local command, not available by default on sara/lisa.

Ox and multitasking

Contrary to standard installation (as on Sara for instance), here on Rdam the default is to run Ox on one thread at a time. That is to say, by default automatically the option -rp1 is added.

In case you know that running multithreaded Ox makes sense for you, use either

  oxl -rp4 mymultithreaded.ox

to test it out, or set it with a

  export OXARGS="-rp4"

to run using 4 threads. You could adapt

  export MAXCNT=4

to tell checkcpuload to run a maximum of 4 Ox jobs now, instead of the 32 it usually allows for.

On Rdam, you use a beta version of Ox7, including the option to use parallel for. Please read chapter 4 of the manual, available from Rdam at /usr/share/OxMetrics7/ox/doc/Ox.pdf.

Dropbox

Dropbox functionality is available, if

you inform me that you'd like to use it

after I start the service, use a command

    dropbox start -i
    and follow the resulting link using the lynx browser from the Rotterdam machine, to fill in your credentials.

Matlab and multitasking

Also Matlab is available on the machine. Within Putty you could start

  matlab

directly, and work from there. This has the drawback that you have to leave PuTTY running until your program finished. Alternatively, start your scripts using

  m myscript_1.m &
  m myscript_2.m &
  m myscript_3.m &

which runs your script in the background. You can now log out using

  ctrl-d

(NOT with exit), leaving the scripts running.

Other commands

Maybe check some Linux introduction on the manual for

cd	Change directory
top	See what jobs are running
fg	Get a stopped or background jobs in the foreground
bg	Put a stopped job (after pressing ctrl-z) in the background
ls	Show directory contents
mkdir mydir	Make a new directory with name `mydir`
more x.txt	Show contents of `x.txt`
cp a b	Copy file `a` to `b`
rm a	Remove file `a`
passwd	Change password

Charles S. Bos

Department of Econometrics & O.R.