Service Desk Knowledgebase: Resources

From Computer Laboratory System Administration
Revision as of 16:11, 28 October 2015 by pb22 (talk | contribs) (→‎Post-Install Tasks: Clarify ssh_known_hosts)
Jump to navigationJump to search


This is the Resources content page of the CL Wiki Service Desk Knowledgebase. Its purpose is to provide information to the Service Desk team on how to handle problems and requests about this CL service. If you are involved with the provision of this CL service please feel free to add to the knowledge about that it.

If CL staff need to tell the Service Desk team about problems with this service please email
sys-admin-aside@cl.cam.ac.uk.

Return to the Service Desk Knowledgebase SERVICE PORTFOLIO

Key Service Description & URLs

William Gates Building Floor Plans (inc. Room Codes):

CL Customer Documentation

Further CL Sys-Admin Resources

[NOTE: The Personresponsible is actually the one who is financially responsible.]

People

Equipment storage

All the monitors on the bottom shelf of the racking on the West wall of room GC20 are available.

Underpinning Services

  • Computer Lab Administration Database, Mail Server, Networking & RT

Customer-base for this Service

  • All staff and students of the Computer Laboratory

Costs

  • Visitors, Interns & RAs equipment should be funded by the Sponsor for anything substantial but short term loans are possible
  • RARS (Research Assistants registered for a PhD) should be treated as RAs but some funding bodies do not permit this (in which case Lab funding is used)
  • Staff equipment for new arrivals is funded by the department

SLA

  • N/A

Service Desk Call Handling Procedure

  • RT tickets can be escalated by changing the Queue to backoffice with the Owner set to Nobody and the Status as new. Tell the requestor:
    I am passing this request over to the experts who will be in contact when they have more questions or have some progress to report.

Delaying RT Tickets

  • To delay a ticket for automatic re-activation on a given date set the Status to Stalled then use Dates to set a Due: date and [Save Changes].

New Research Students

Processing of postgraduate student admissions by administrative staff automatically generate an RT ticket with the following format:
Academic Year RS GivenName FamilyName crsid kit
for example:
#96184: 2015/10 RS Marcel ten Wolde mw245 kit
when they are added to the database.

The body of the message contains the name and crsid of the supervisor and the email address of the Research Student (RS). For example: 

 Supervisor: gt19
 Email: mw245@cam.ac.uk

The following procedure applies ONLY to the above messages and NOT to other New Arrivals (where the New Arrivals procedure below should be used instead).

graduate admin

The auto-generation of these tickets relies on various heuristics which are sometimes wrong, and sometimes things change. The definitive source of all information relating to RSs (and RARSs) is graduate-admin@cl.cam.ac.uk. Information channels are not always clear, so if you get any information, please pass it on. If anything unexpected is revealed (a RS or supervisor says that a RS is not coming; someone says that the arrival date has been deffered; etc) forward the email graduate-admin@cl.cam.ac.uk pointing out the anomaly, and asking for confirmation that 'the system' is wrong and needs updating. Do not use an RT comment to do this, as graduate admin cannot 'reply' to an RT comment.

Phase 1

  • Before doing anything with this ticket link it as a child of this years parent ticket:
    #96194: AY2015 RS arrivals - please add new ones as stalled children and leave this ticket open
  • Note the student's name/CRSid and copy the child tickets's RT number
  • Visit the arrival tracking database, add the RT ticket number to the PhD database by clicking on Select of the appropriate record then Edit the table that appears and insert the value as a number only, set Status to "supervisor" and click on Update.
  • The next step is to return to the child RT ticket copy the supervisor's CRSid and use People then Type: Requestor and E-mail: CRSid and select CRSid@cl.cam.ac.uk and then Requestors: [tick] "Graham Titmus" <gt19@cl.cam.ac.uk> and [Save Changes] to make the supervisor as the sole requestor of the ticket.
  • Then use Display and Reply to the initial email with the following message:
 For the incoming student this request relates to,
 do you know if a new or existing group machine 
 will be used, or whether one from the Lab Pool should 
 be used please?
 
 Can the RS be given a free choice of Hardware [1] 
 and Operation System [2], or do you as Supervisor want
 to recommend particular HW or OS, or even restrict the
 choice to a particular HW or OS?  
 We will contact the student directly to decide what they
 want if you allow them to choose.
 
 [1] http://www.cl.cam.ac.uk/local/sys/platforms/equipment.html
 [2] http://www.cl.cam.ac.uk/local/sys/platforms/
 
 Regards,
  • Put the following comment into the RT ticket:
 Go to Phase 2 of the New Research Students procedure at
 https://wiki.cam.ac.uk/cl-sys-admin/Service_Desk_Knowledgebase:_Resources#Phase_2
 when the Supervisor replies with the requested information.
  • Then set the RT ticket owner as nobody with the status as stalled with a Date Due of one week hence.

Phase 2

  1. Copy the student's email address from the initiating email
  2. Click People to set the following:
    1. Type: Requestor and E-mail: student's email address for the student
    2. Type: Cc and E-mail: CRSid and select CRSid@cl.cam.ac.uk for the supervisor
    3. Requestors: [tick] <CRSid@cl.cam.ac.uk> to remove the supervisor as the requestor
  3. [Save Changes] to make the student the Requestor and the Supervisor the Cc
  • Display the RT ticket
  • If the Research Student has a CRSid, check if the student is already in the Inventory database is an assigned user of a machine (i.e. is already at the CL). If so note the details and add the machine's name to the RT Ticket's subject.
  • Update the arrival tracking database
    • update Status. If the supervisor has specified the system to use, select the appropriate value, such as Install OS, Install HW or completed. Otherwise, set it to Student - asking student for their requirements
    • update the Notes field. If the supervisor has stipulated a preferred or mandatory provision then briefly summarise the info, along with any information from the Inventory database. For example:
 CRSID mw245 
 USN 301711634 
 RT 96184 
 OS Std Linux  
 Hardware std PC and LCD  
 Machine 19138
 Status completed
 Notes keep existing machine
  • Using Reply email the incoming Research Student with the following message (editing as required):
According to the database, we are expecting you to come to the University
of Cambridge Computer Laboratory to start a PhD in October 2015. Welcome!

Are you still planning to come then?  If so then please note that you should
not expect any resources to be available ahead of the induction
date which will be notified to you by Graduate Admin (Monday October 5th).

If you are coming, you'll be allocated a pool machine as described in
http://www.cl.cam.ac.uk/local/sys/platforms/equipment.html
http://www.cl.cam.ac.uk/local/sys/platforms/ has information about the 
choice of operating system.

If they already have an assigned machine add: 

I note you are already using machine "<<machine name>>", would you like to continue
to use that and if so do you want the OS refreshed to the latest version, leaving
scratch space as it is?

If the supervisor has specified or suggested an operating system add the following (with the appropriate OS and machine type set): 

Your supervisor has suggested you start with Ubuntu 14.04 LTS on a standard Computer Lab PC
Please let us know if you don't think that is suitable.

If the supervisor has not specified anything then add: 

Please let us know what equipment and OS you would prefer.
  • Put the following comment into the RT ticket:
 Go to Phase 3 of the New Research Students procedure at
 https://wiki.cam.ac.uk/cl-sys-admin/Service_Desk_Knowledgebase:_Resources#Phase_3
 when the Research Student replies with the requested information.
  • Then set the RT ticket owner as nobody with the status as stalled with a Date Due of one week hence. If the due date is reached and the ticket reverts to open, email the Research student again as above, and stall the ticket again with the date another week hence.

Phase 3

  • If the Supervisor replies that the student is not coming: pass the info on to graduate-admin@cl.cam.ac.uk by email (not RT) , mark the ticket as Rejected, move it from being a Children: to being Referred to by:, and update the arrival tracking database to have Status of withdrawn or deferred and summarise it briefly in Notes.
  • If the Research Student says that they are not coming: thank them, email their supervisor, pass the info on to graduate-admin@cl.cam.ac.uk by email (not RT) , mark the ticket as Rejected, move it from being a Children: to being Referred to by:, and update the arrival tracking database to have Status of withdrawn or deferred and summarise it briefly in Notes.
  • If the Research Student replies with "I have not yet received a formal offer":
    • pass the info on to graduate-admin@cl.cam.ac.uk by email (not RT)
    • Stall ticket until the RS receives an offer, graduate admin convinces the RS that (s)he does have an offer, or the offer is declined (as advised by Graduate Admin)
    • Update the arrival tracking database Notes field with a *brief* summary of the info useful at *this* stage of the process with something like:
      "Has not received formal offer. graduate-admin asked to clarify"
    • Issues could be:
      • formal letter was sent, but didn't arrive
      • formal letter should have been sent, but by mistake was not
      • the automated WorkFlow has triggered a user to be put into the system too early (gt10 needs to fix)
  • When the Research Student replies with their requirements:
    • Add the options they have chosen to the PhD arrival tracking database - Select and then use Edit at the bottom of the list, e.g.
      OS: Linux
      Hardware: Std PC and LCD
      Status: Allocate - Awaiting allocation of equipment
    • Thank them for the information using a message such as:
Thank you for letting us know about your requirements.  If we need any further 
details we will contact you shortly.

In the meantime you may care to familiarise yourself with the IT ethos of the department
by reading
http://www.wiki.cl.cam.ac.uk/rowiki/SysInfo/BedtimeReading
and the pages to which it points.
  • The RT ticket then needs to be passed on to backoffice queue to provision with status set to new and owner to nobody.

New arrivals

Processing of new arrivals by administrative staff automatically generate an RT ticket with the following format:
CRSid has a HR Role starting ??/??/2015 requiring equipment v#VisitorID visitor name
when they are added to the database.

  1. Copy the RT ticket number (without the #)
  2. Click on the URL in ticket like: https://dbwebserver.ad.cl.cam.ac.uk/Administration/Visitors/VisitorDetails.aspx?ID=999
  3. Enter your CRSid & Computer Lab password if requested
  4. Click the [Edit] button at the bottom left of the form
  5. Paste in the RT_Ticket & click [Update]
  6. Desks & Rooms: Allocation of desks/rooms is done by Reception - we don't care about them until we actually need to install equipment. Reception see a parallel workflow and use the OpenRoomMap (ORM). If there seems to be a problem, contact them, and ask them which desk/room to use (but remember that they cannot reply to an email sent to them using an RT comment!) Once the 'OpenRoomMap (ORM) is updated use it to see where the user 'will be'. If room information is available [Edit] Allocated_Office & [Update].
  7. Copy notes for any specific instructions to paste into the RT ticket
  8. Return to RT
  9. Click Comment and paste specific equipment requests into the RT ticket
  10. What happens next depends on what's been requested and who has the equipment:
    • If there are no specific equipment requests Reply to the RT ticket requestor asking:
      What do you intend to provide for the person please?
      In RT set the Owner as Nobody & Status as Stalled (with a suitable Due date) & leave the Queue as Sys-Admin and then [Edit] the inventory database Equipment_state with Asking and [Update]
    • If a computer has been requested or allocated to the person but is already allocated to someone else who is still here, Reply to the RT ticket requestor asking if the new person should be made the "assigned user" of the computer (i.e. User in the inventory, the person who has 'cl-asuser' and 'sudo' privileges to do things such as install software on a managed Linux PC, & have access to the machines BMC if it has one). In RT set the Owner as Nobody & Status as Stalled (with a suitable Due date) & leave the Queue as Sys-Admin.
    • If a non-specific machine (e.g. Linux machine) is requested for an RA, reply and ask the PI whether they already have a machine, or whether they want to buy one. If they say that their grant explicitly forbids buying WSs, then they are allocated a charity case or grace and favour machine - see Special information for re-use of PCCL0xx machines for 2015/10 for more on this for the academic year 2015/16.
    • If a computer has been requested or allocated to the person and is not allocated to anyone or their end date has passed then the new person should be made the "assigned user" (i.e. User in the inventory, the person who has 'cl-asuser' and 'sudo' privileges to do things such as install software on a managed Linux PC, & have access to the machines BMC if it has one).
    • If access to servers is required Reply and CC the Sponsor/Personresponsible for each server as they need to formally agree to that access in order to meet the Cam AUP (even though almost all machines are accessible by anyone in the Computer Lab by default). In RT set the Owner as Nobody & Status as Stalled (with a suitable Due date) & leave the Queue as Sys-Admin.
  11. When all the required information has been gather escalate the RT ticket by leaving on the backoffice queue with Status as new and Owner as Nobody. Backoffice will assign a machine and then handed it over to the operators for the actual install of the OS and the positioning of the hardware.

Post-Install Tasks

When the operators have completed their tasks they will pass the RT ticket back to sys-admin for the following POST INSTALL tasks to be done. Some are needed by all systems, some only by certain ones (e.g. Linux, Windows or Macs):

  • 4.0 Check Database:
    Check the RS database or "Visitors" database work queue to check the person's details and the task is marked "post-install - Complete Post install tasks" in the case of an RS, or "Post-install tasks to be done" in the case of a visitor.
  • 4.1 WoL - Wake on LAN:
    To ensure that WoL is available, on laira run:
/usr/groups/netmaint/boot_wol_file-add.pl $HOST

If it reports: 

/usr/groups/netmaint/boot_wol_file-add.pl: FAILED: missing MAC address

then find the machine's MAC address in the Inventory then use the following to specify the MAC address (with or without colons): 

/usr/groups/netmaint/boot_wol_file-add.pl $HOST $MAC-ADDRESS

for example: 

/usr/groups/netmaint/boot_wol_file-add.pl gwendreath 0023AE91CFC1

When the entry is added, install it on the web server using 

make -C /usr/groups/netmaint wol
  • BMC Accessibilty - if present:
    If the machine has a BMC (true for most machines, not Macs) check that the address is pingable (it may take 30 seconds for it to wake up) and then check the "assigned user" access to BMC: on svr-ssh-1 or sandy run cl-amttool $host user list
 kikyo-bmc  Registered 3 AMT user accounts:
  $$uns                  1 realm    LocalAccessPermission    Enabled  <system account>
  $$OsAdmin              6 realms   LocalAccessPermission    Enabled  <system account>
  qg216                 21 realms   AnyAccessPermission      Enabled

to check that the user has AnyAccessPermission Enabled. If not or you see something like: 

cl-amttool avon user list

FAULT: 404 Not Found

do the procedures at (4.7) BMC ACL - when up if present.

  • 4.2 ssh_known_hosts (Unix):
    To be able to ssh to Linux or Mac machines, the host's ssh key needs to be known. So long as the machine is up and running an ssh server, on laira it should be possible to
/global/src/usr.bin/ssh/fetch-host-key scan $HOST

to get the info, and if there are any changed, then run: 

 sudo ssh -tt radyr '(cd /global/src/usr.bin/ssh && make dist)'
 sudo cl-update-system

to make the new ssh keys available on laira.

  • 4.3 keytab install (Linux):
    First ssh -K $HOST to the machine from a CL machine (See Waking Up a Lab Computer which has BMC if necessary). If it refuses connection, this may be because the correct keytab is not installed, so on laira, connect to it using sudo ssh $HOST which uses ssh user keys, rather than Kerberos. Check what key it has with:
sudo klist -k /etc/krb5.keytab

which should have the machine name and not noname-Linux.
If it is the wrong host's keytab, run: 

cl-onserver --keytab

If you see: 

# User key needed for kinit: please enter user password
kinit(v5): Client not found in Kerberos database while getting initial credentials

it may still have done what it needed to so check noname-linux has gone with: 

sudo klist -k /etc/krb5.keytab

If the command has failed to change noname-linux then contact gt19 to create a new keytab.

  • 4.4 User Admin (Linux):
    If oper were not told the 'assigned user', on $HOST run the following to check the assigned user:
cl-asuser cl-hostid-fix --user $CRSid

The output shows what needs to be done to make that $CRSid the "assigned user" so NO OUTPUT is good! Should things need doing to make $CRSid the "assign user" then use the following to change the assigned user: 

cl-asuser cl-hostid-fix --user $CRSid -a
  • 4.6 Tell the user - when done:
    Check that the machine name *IS* actually in the RT Ticket's subject line - if not add it first then send 'std email' to user. The email might be along the lines of:
 The machine mentioned in the Subject: line has now been re-installed for 
 you and should be ready to use. Please login and check that the basics work, 
 i.e. that you can login, access the web, and send email.  If not, please 
 reply to this ticket, which will re-open it, and we will try to sort the 
 problem.

 If you have other requests, please do NOT reply to this ticket, 
 but instead open a new ticket, and mention this one.

 Now may be a good time to look again at
 http://www.wiki.cl.cam.ac.uk/rowiki/SysInfo/BedtimeReading
 and the pages to which it points.
  • 4.7 ownfiles (Linux):
    To ensure that ownfiles data is collected, also on laira run
(umask 2; touch /usr/groups/linux/ownfiles/CKSUM/$HOST)
  • 4.8 Network account admin privs (Macs)
    Ask the user to 'reply' to the ticket once they have logged into the Mac using their network credentials, so that backoffice can grant that account admin privileges.
  • 4.9 Finishing up:
    When the above is completed, update the RS or "Visitors" work queue to mark the task to be "completed" in the case of an RS, or "OK" in the the case of a visitor (also adding the Inventory number and name of the PC after, e.g. "OK Inv#16200 ouse") [NOTE: The Equipment_state field requires OK Inventory No.???? MachineName to complete it and stop it being picked up as a "Outstanding equipment requests for arriving visitors/staff".] and update the RT Ticket with a comment of the collected history for 4.6 to 4.8 then Resolve the RT ticket.

Creating accounts for people arriving

See Account creation request for <CRSid> arriving <date>" or Outstanding equipment requests for arriving visitors/staff

Requests for machine moves

The move process involves:

  1. Reception allocate people to a free desk in a room using OpenRoomMap (ORM).
  2. When people are physically moved into a room, whoever moves them (e.g. the operators) checks that the OpenRoomMap (ORM) information is correct, reporting any erros to Reception, then moves them and when the move is complete updates the Staff List.
  3. Users can update their details within a room (e.g. if they move tables etc).


Requests for machine moves are raised by Reception after they allocate a new office.

  1. Take the RT ticket and check the equipment the person has in the Inventory.
  2. Check what phone may need moving in the VOIPList.
  3. If there is some equipment to move then list it and pass the ticket over to the operators.
  4. Set the Queue to oper, the Owner to Nobody and with Status as new.

Post move

After the operators have moved the machine they should provide details of which port the machine is now plugged into on the ticket and pass it back to the sys-admin queue to do the following checks:

1. Inventory Check:
Take the RT ticket and then update the inventory with the new machine's location.

2. Staff List Check:
Check that the information in the Staff List and the OpenRoomMap (ORM) is correct. If not then email Reception@cl.cam.ac.uk to get it updated.

3. VLAN:
Using Updating VLANs in the Cisco switches note the VLAN of the old switch port and then set the old switch port where the machine was to no longer be on the VLAN it was with no switch port access vlan vlan# then set the new switch port to access the VLAN that was removed.

4. Machine Accessibility:
Check you can access the machine remotely

5. Assigned User:
To check the "assigned user" on Linux desktop machines: On svr-ssh-1 or sandy run ssh -K $host ls -l /etc/user-config/bundles for example: 

 -rw-rw-r-- 1 qg216 sysadmin 3565 Nov  3 20:06 /etc/user-config/bundles

to check the user can install software as the "assigned user" (i.e. sysadmin). If not see Adding privileged or 'assigned' users

6. BMC Accessibilty:
To check the "assigned user" access to BMC on Linux desktop machines: On svr-ssh-1 or sandy run cl-amttool $host user list 

 kikyo-bmc  Registered 3 AMT user accounts:
  $$uns                  1 realm    LocalAccessPermission    Enabled  <system account>
  $$OsAdmin              6 realms   LocalAccessPermission    Enabled  <system account>
  qg216                 21 realms   AnyAccessPermission      Enabled

to check that the user has AnyAccessPermission Enabled. If not or you see something like: 

cl-amttool avon user list

FAULT: 404 Not Found

do the procedures at (4.7) BMC ACL - when up if present.

Leaving Procedure

Normally a leaver is only noticed by us after (1) has been completed but...

1) Tell Reception that he has left - they update the staff list at https://dbwebserver.ad.cl.cam.ac.uk/Administration/HR/HRList.aspx to say he's no longer Valid.

2) Use the Inventory Database to find what equipment he has.

2a) Anything which is 'Lab pool' (typically owned by 'RTSG', pb22, gt19, 'Lab', 'PWF' or other pseudo users) should be returned to GC20 - ask oper to do that - phone to GC12.

2b) If there is anything owned by the Host / Supervisor / PI, contact them and ask what should be done, e.g. donated to the Lab Pool, or re-deployed elsewhere.

[NOTE: STUDENT MOVES:- Student's may be moved to a Hot Desk/Writing Up area (the Library) before actually leaving. OpenRoomMap at http://www.cl.cam.ac.uk/research/dtg/openroommap/static/ shows where people 'will' be - it's used for office allocation by Reception. The Staff List at https://dbwebserver.ad.cl.cam.ac.uk/Administration/HR/HRList.aspx shows where people (and their kit) actually *ARE* and is updated by Reception. It is differences between the two which generate the WorkList for the operators to move people etc, so if the Staff List is not updated by Reception when kit (and thus people) are moved, the move remains on the WorkList. Reception@cl.cam.ac.uk may need to be asked to do this.]

User Can't install Software under Linux

See Adding privileged or 'assigned' users

Machine Hardware Problems

Graham Titmus (22 Apr '15)

Machines with noise issues etc.:
Change the RT ticket queue to hw-admin with status New AND email a comment Nick Baterham (nb@cl.cam.ac.uk) to alert him.

Operating System Upgrades

and

Dual boot requests

Piete Brooks (06 May '15)

Dual booting is discouraged in favour of running virtual machines instead. The options are either Computer Lab Windows + virtual Linux, or Computer Lab Linux + virtual Windows See http://www.cl.cam.ac.uk/local/sys/platforms/#dualboot

Reclaiming Equipment after someone leaves

1) Ask the visitor's sponsor if the visitor has left and if it's OK to reclaim the equipment
2) Once the sponsor has replied in the affirmative, add 'Reclaim' to the subject line and move the ticket to the Oper queue

Contacts

Primary

Availability

  • Monday:
  • Tuesday:
  • Wednesday:
  • Thursday:
  • Friday:
  • Saturday: Closed
  • Sunday: Closed

Hints, Tips & Known Issues

Finding out a machine's operating system

Graham Titmus (26/05/15)

Whilst it's only the best guess you can try logging into laira and running the command:
cl-hosts -p MachineName
to find out what operating system a machine is believed to have.


Typical "Standard Lab PC" (Linux & Windows) Spec.

Graham Titmus 

 Typical Standard Computer Lab PC Spec. Dated 15/04/15:  
 
 Qty     Item            Price   Total   Description 
 1       Med-ATXB        £66     £66     Med Tower ATX Case High Efficiency 80Plus 350W PSU 
 1       SH-W163A-BL     £18     £18     Samsung Internal 22xDVD-RW SATA Black 
 1       Q87M-E  £98     £98     ASUS 1150 Q87M-E Executive Series /SI M-ATX Haswell 
 1       SV300S37A/240G  £70     £70     Kingston V300 - 240GB - Read 450MB/s - Write 450MB/s - 533 DW 
 1       i5-4670 £169    £169    Intel Haswell Core i5-4670 3.4-3.8 GHz 4 Cores, 84W, HD Graphics 4600 
 2       KVR16N11/8      £53     £106    8GB 1600MHz DDR3 Non-ECC CL11 DIMM 
 2       25SATSAS35      £11     £22     StarTech.com 2.5 inch SATA/SAS SSD/HDD to 3.5 inch SATA Hard Drive Converter 
 2       HSB100SATBK     £14     £28     StarTech.com 5.25 inch Tray-Less Hot Swap Mobile Rack for 3.5 inch Hard Drive 
 1       USB2.0Header    £3      £3      USB 2.0 Header plate 
 1       Header  £5      £5      Serial Header plate 
 1       USB3.0Header    £9      £9      USB 3.0 Header plate 
 1       K/B-Mouse-Bl    £17     £17     Microsoft USB Wired Desktop 600 Keyboard+Mouse Black 
 Total:          £611.00 
 Discount:       £30.55 
 Total Ex-VAT:   £580.45 
 VAT:            £116.09 
 Total:          £696.54
 
 to which you need to add a screen - a 4K screen costs about £300 + VAT.

Categorising Keywords

  • Resources PC machines new arrivals upgrades failures
Retrieved from "https://wiki.cam.ac.uk/wiki/cl-sys-admin/index.php?title=Service_Desk_Knowledgebase:_Resources&oldid=3251"