Service Desk Knowledgebase: Resources: Difference between revisions

From Computer Laboratory System Administration
Jump to navigationJump to search
Line 210: Line 210:
#* If a computer has been requested or allocated to the person and is not allocated to anyone or their end date has passed then the new person should be made the "'''[[Service_Desk_Knowledgebase:_Linux#Adding_privileged_or_.27assigned.27_users |assigned user]]'''" (i.e. User in the inventory, the person who has 'cl-asuser' and 'sudo' privileges to do things such as install software on a managed Linux PC, & have access to the machines BMC if it has one).
#* If a computer has been requested or allocated to the person and is not allocated to anyone or their end date has passed then the new person should be made the "'''[[Service_Desk_Knowledgebase:_Linux#Adding_privileged_or_.27assigned.27_users |assigned user]]'''" (i.e. User in the inventory, the person who has 'cl-asuser' and 'sudo' privileges to do things such as install software on a managed Linux PC, & have access to the machines BMC if it has one).
#* If access to servers is required '''Reply''' and '''CC''' the Sponsor/Personresponsible for each server as they need to formally agree to that access in order to meet the Cam AUP (even though almost all machines are accessible by anyone in the Computer Lab by default).  In RT set the '''Owner''' as '''Nobody''' & '''Status''' as '''Stalled''' (with a suitable '''Due date''') & leave the '''Queue''' as '''Sys-Admin'''.  
#* If access to servers is required '''Reply''' and '''CC''' the Sponsor/Personresponsible for each server as they need to formally agree to that access in order to meet the Cam AUP (even though almost all machines are accessible by anyone in the Computer Lab by default).  In RT set the '''Owner''' as '''Nobody''' & '''Status''' as '''Stalled''' (with a suitable '''Due date''') & leave the '''Queue''' as '''Sys-Admin'''.  
# Finally when all the required information has been gather escalate the RT ticket by leaving on the '''backoffice''' queue with '''Status''' as '''new''' and '''Owner''' as '''Nobody'''.  [NOTE: The '''Equipment_state''' field requires '''OK Inventory No.<font color="red">???? MachineName</font>''' to complete it and stop it being picked up as a [[Service_Desk_Knowledgebase:_User_Accounts_and_Groups#Email:_.22Outstanding_equipment_requests_for_arriving_visitors.2Fstaff.22 | "Outstanding equipment requests for arriving visitors/staff"]].]
# Finally when all the required information has been gather escalate the RT ticket by leaving on the '''backoffice''' queue with '''Status''' as '''new''' and '''Owner''' as '''Nobody'''.  Backoffice will assign a machine and then handed it over to operators for the actual install of the OS and the positioning of the hardware. 
 
When the operators have completed their tasks they will pass the RT ticket back to sys-admin for the following '''''POST INSTALL''''' tasks to be done:
 
* '''4.0 keytab install (Linux)''': Ensure '''<font color="red">$HOST</font>''' has a correct keytab.
sudo klist -k /etc/krb5.keytab
which should have the machine name not no-name-Linux.  If it is no-name-Linux then do the command below.
If the command below fails, contact gt19 to create a new keytab. Login to '''<font color="red">$HOST</font>''' using '''sudo ssh <font color="red">$HOST</font>''' and then run:
cl-onserver --keytab
* '''4.1 User Admin - when running if needed (Linux)''':  If oper were not told the 'assigned user', on '''<font color="red">$HOST</font>''' run:
cl-asuser cl-hostid-fix --user '''<font color="red">$CRSid</font>''' -a
* '''4.2 Arrivals - when done''': update https://dbwebserver.ad.cl.cam.ac.uk/SCG/Equipment/PhDArrivals.aspx for RS or https://dbwebserver.ad.cl.cam.ac.uk/SCG/Visitors/VisitorStatus.aspx and then update RT ticket to include machine name (i.e. '''<font color="red">$HOST</font>''') in ticket Subject:
* '''4.3 Tell the user - when done''': Send final 'std email' to user. Resolve RT ticket.
* '''4.6 ssh_known_hosts - at leisure if needed (Linux)''': when the machine is running, on a '''different''' machine run
/global/src/usr.bin/ssh/fetch-host-key scan '''<font color="red">$HOST</font>'''
* '''4.8 ownfiles - at leisure if needed (Linux)''': to ensure that '''ownfiles''' data is collected, run
(umask 2; touch /usr/groups/linux/ownfiles/CKSUM/'''<font color="red">$HOST</font>''')
* '''4.9 WoL - at leisure''': to ensure that WoL is available, run:
/usr/groups/netmaint/boot_wol_file-add.pl '''<font color="red">$HOST</font>'''
* Update the [https://dbwebserver.ad.cl.cam.ac.uk/SCG/Equipment/PhDArrivals.aspx RS] or [https://dbwebserver.ad.cl.cam.ac.uk/SCG/Visitors/VisitorStatus.aspx "Visitors"] work queue to mark the task to be "'''completed'''" in the first case, or "'''OK'''" in the second (also adding the Inventory number and name of the PC after, e.g. "'''OK #16200 ouse'''") [NOTE: The '''Equipment_state''' field requires '''OK Inventory No.<font color="red">???? MachineName</font>''' to complete it and stop it being picked up as a [[Service_Desk_Knowledgebase:_User_Accounts_and_Groups#Email:_.22Outstanding_equipment_requests_for_arriving_visitors.2Fstaff.22 | "Outstanding equipment requests for arriving visitors/staff"]].]
* '''Resolve''' the RT ticket.


===Creating accounts for people arriving===
===Creating accounts for people arriving===

Revision as of 16:03, 2 October 2015


This is the Resources content page of the CL Wiki Service Desk Knowledgebase. Its purpose is to provide information to the Service Desk team on how to handle problems and requests about this CL service. If you are involved with the provision of this CL service please feel free to add to the knowledge about that it.

If CL staff need to tell the Service Desk team about problems with this service please email
sys-admin-aside@cl.cam.ac.uk.

Return to the Service Desk Knowledgebase SERVICE PORTFOLIO

Key Service Description & URLs

William Gates Building Floor Plans (inc. Room Codes):

CL Customer Documentation

Further CL Sys-Admin Resources

[NOTE: The Personresponsible is actually the one who is financially responsible.]

People

Equipment storage

All the monitors on the bottom shelf of the racking on the West wall of room GC20 are available.

Underpinning Services

  • Computer Lab Administration Database, Mail Server, Networking & RT

Customer-base for this Service

  • All staff and students of the Computer Laboratory

Costs

  • Visitors, Interns & RAs equipment should be funded by the Sponsor for anything substantial but short term loans are possible
  • RARS (Research Assistants registered for a PhD) should be treated as RAs but some funding bodies do not permit this (in which case Lab funding is used)
  • Staff equipment for new arrivals is funded by the department

SLA

  • N/A

Service Desk Call Handling Procedure

  • RT tickets can be escalated by changing the Queue to backoffice with the Owner set to Nobody and the Status as new. Tell the requestor:
    I am passing this request over to the experts who will be in contact when they have more questions or have some progress to report.

Delaying RT Tickets

  • To delay a ticket for automatic re-activation on a given date set the Status to Stalled then use Dates to set a Due: date and [Save Changes].

New Research Students

Processing of postgraduate student admissions by administrative staff automatically generate an RT ticket with the following format:
Academic Year RS GivenName FamilyName crsid kit
for example:
#96184: 2015/10 RS Marcel ten Wolde mw245 kit
when they are added to the database.

The body of the message contains the name and crsid of the supervisor and the email address of the Research Student (RS). For example:

 Supervisor: gt19
 Email: mw245@cam.ac.uk

The following procedure applies ONLY to the above messages and NOT to other New Arrivals (where the New Arrivals procedure below should be used instead).

graduate admin

The auto-generation of these tickets relies on various heuristics which are sometimes wrong, and sometimes things change. The definitive source of all information relating to RSs (and RARSs) is graduate-admin@cl.cam.ac.uk. Information channels are not always clear, so if you get any information, please pass it on. If anything unexpected is revealed (a RS or supervisor says that a RS is not coming; someone says that the arrival date has been deffered; etc) forward the email graduate-admin@cl.cam.ac.uk pointing out the anomaly, and asking for confirmation that 'the system' is wrong and needs updating. Do not use an RT comment to do this, as graduate admin cannot 'reply' to an RT comment.

Phase 1

  • Before doing anything with this ticket link it as a child of this years parent ticket:
    #96194: AY2015 RS arrivals - please add new ones as stalled children and leave this ticket open
  • Note the student's name/CRSid and copy the child tickets's RT number
  • Visit the arrival tracking database, add the RT ticket number to the PhD database by clicking on Select of the appropriate record then Edit the table that appears and insert the value as a number only, set Status to "supervisor" and click on Update.
  • The next step is to return to the child RT ticket copy the supervisor's CRSid and use People then Type: Requestor and E-mail: CRSid and select CRSid@cl.cam.ac.uk and then Requestors: [tick] "Graham Titmus" <gt19@cl.cam.ac.uk> and [Save Changes] to make the supervisor as the sole requestor of the ticket.
  • Then use Display and Reply to the initial email with the following message:
 For the incoming student this request relates to,
 do you know if a new or existing group machine 
 will be used, or whether one from the Lab Pool should 
 be used please?
 
 Can the RS be given a free choice of Hardware [1] 
 and Operation System [2], or do you as Supervisor want
 to recommend particular HW or OS, or even restrict the
 choice to a particular HW or OS?  
 We will contact the student directly to decide what they
 want if you allow them to choose.
 
 [1] http://www.cl.cam.ac.uk/local/sys/platforms/equipment.html
 [2] http://www.cl.cam.ac.uk/local/sys/platforms/
 
 Regards,
  • Put the following comment into the RT ticket:
 Go to Phase 2 of the New Research Students procedure at
 https://wiki.cam.ac.uk/cl-sys-admin/Service_Desk_Knowledgebase:_Resources#Phase_2
 when the Supervisor replies with the requested information.
  • Then set the RT ticket owner as nobody with the status as stalled with a Date Due of one week hence.

Phase 2

  1. Copy the student's email address from the initiating email
  2. Click People to set the following:
    1. Type: Requestor and E-mail: student's email address for the student
    2. Type: Cc and E-mail: CRSid and select CRSid@cl.cam.ac.uk for the supervisor
    3. Requestors: [tick] <CRSid@cl.cam.ac.uk> to remove the supervisor as the requestor
  3. [Save Changes] to make the student the Requestor and the Supervisor the Cc
  • Display the RT ticket
  • If the Research Student has a CRSid, check if the student is already in the Inventory database is an assigned user of a machine (i.e. is already at the CL). If so note the details and add the machine's name to the RT Ticket's subject.
  • Update the arrival tracking database
    • update Status. If the supervisor has specified the system to use, select the appropriate value, such as Install OS, Install HW or completed. Otherwise, set it to student
    • update the Notes field. If the supervisor has stipulated a preferred or mandatory provision then briefly summarise the info, along with any information from the Inventory database. For example:
 CRSID mw245 
 USN 301711634 
 RT 96184 
 OS Std Linux  
 Hardware std PC and LCD  
 Machine 19138
 Status completed
 Notes keep existing machine
  • Using Reply email the incoming Research Student with the following message (editing as required):
According to the database, we are expecting you to come to the University
of Cambridge Computer Laboratory to start a PhD in October 2015. Welcome!

Are you still planning to come then?  If so then please note that you should
not expect any resources to be available ahead of the induction
date which will be notified to you by Graduate Admin (Monday October 5th).

If you are coming, you'll be allocated a pool machine as described in
http://www.cl.cam.ac.uk/local/sys/platforms/equipment.html
http://www.cl.cam.ac.uk/local/sys/platforms/ has information about the 
choice of operating system.

If they already have an assigned machine add:

I note you are already using machine "<<machine name>>", would you like to continue
to use that and if so do you want the OS refreshed to the latest version, leaving
scratch space as it is?

If the supervisor has specified or suggested an operating system add the following (with the appropriate OS and machine type set):

Your supervisor has suggested you start with Ubuntu 14.04 LTS on a standard Computer Lab PC
Please let us know if you don't think that is suitable.

If the supervisor has not specified anything then add:

Please let us know what equipment and OS you would prefer.
  • Put the following comment into the RT ticket:
 Go to Phase 3 of the New Research Students procedure at
 https://wiki.cam.ac.uk/cl-sys-admin/Service_Desk_Knowledgebase:_Resources#Phase_3
 when the Research Student replies with the requested information.
  • Then set the RT ticket owner as nobody with the status as stalled with a Date Due of one week hence. If the due date is reached and the ticket reverts to open, email the Research student again as above, and stall the ticket again with the date another week hence.

Phase 3

  • If the Supervisor replies that the student is not coming: pass the info on to graduate-admin@cl.cam.ac.uk by email (not RT) , mark the ticket as Rejected, move it from being a Children: to being Referred to by:, and update the arrival tracking database to have Status of withdrawn or deferred and summarise it briefly in Notes.
  • If the Research Student says that they are not coming: thank them, email their supervisor, pass the info on to graduate-admin@cl.cam.ac.uk by email (not RT) , mark the ticket as Rejected, move it from being a Children: to being Referred to by:, and update the arrival tracking database to have Status of withdrawn or deferred and summarise it briefly in Notes.
  • If the Research Student replies with "I have not yet received a formal offer":
    • pass the info on to graduate-admin@cl.cam.ac.uk by email (not RT)
    • Stall ticket until the RS receives an offer, graduate admin convinces the RS that (s)he does have an offer, or the offer is declined (as advised by Graduate Admin)
    • Update the arrival tracking database Notes field with a *brief* summary of the info useful at *this* stage of the process with something like:
      "Has not received formal offer. graduate-admin asked to clarify"
    • Issues could be:
      • formal letter was sent, but didn't arrive
      • formal letter should have been sent, but by mistake was not
      • the automated WorkFlow has triggered a user to be put into the system too early (gt10 needs to fix)
  • When the Research Student replies with their requirements:
    • Add the options they have chosen to the PhD arrival tracking database - Select' and then use Edit at the bottom of the list, e.g.
      OS: Linux
      Hardware: Std PC and LCD
      Status: install OS
      and set the status to be
      Allocate
    • The RT ticket then needs to be passed on to oper queue to provision with status set to stalled.
    • Thank them for the information using a message such as:
Thank you for letting us know about your requirements.  If we need any further 
details we will contact you shortly.

In the meantime you may care to familiarise yourself with the IT ethos of the department
by reading
http://www.wiki.cl.cam.ac.uk/rowiki/SysInfo/BedtimeReading
and the pages to which it points.

New arrivals

Processing of new arrivals by administrative staff automatically generate an RT ticket with the following format:
CRSid has a HR Role starting ??/??/2015 requiring equipment v#VisitorID visitor name
when they are added to the database.

  1. Copy the RT ticket number (without the #)
  2. Click on the URL in ticket like: https://dbwebserver.ad.cl.cam.ac.uk/Administration/Visitors/VisitorDetails.aspx?ID=999
  3. Enter your CRSid & Computer Lab password if requested
  4. Click the [Edit] button at the bottom left of the form
  5. Paste in the RT_Ticket & click [Update]
  6. Desks & Rooms: Allocation of desks/rooms is done by Reception - we don't care about them until we actually need to install equipment. Reception see a parallel workflow and use the OpenRoomMap (ORM). If there seems to be a problem, contact them, and ask them which desk/room to use (but remember that they cannot reply to an email sent to them using an RT comment!) Once the 'OpenRoomMap (ORM) is updated use it to see where the user 'will be'. If room information is available [Edit] Allocated_Office & [Update].
  7. Copy notes for any specific instructions to paste into the RT ticket
  8. Return to RT
  9. Click Comment and paste specific equipment requests into the RT ticket
  10. What happens next depends on what's been requested and who has the equipment:
    • If there are no specific equipment requests Reply to the RT ticket requestor asking:
      What do you intend to provide for the person please?
      In RT set the Owner as Nobody & Status as Stalled (with a suitable Due date) & leave the Queue as Sys-Admin and then [Edit] the inventory database Equipment_state with Asking and [Update]
    • If a computer has been requested or allocated to the person but is already allocated to someone else who is still here, Reply to the RT ticket requestor asking if the new person should be made the "assigned user" of the computer (i.e. User in the inventory, the person who has 'cl-asuser' and 'sudo' privileges to do things such as install software on a managed Linux PC, & have access to the machines BMC if it has one). In RT set the Owner as Nobody & Status as Stalled (with a suitable Due date) & leave the Queue as Sys-Admin.
    • If a non-specific machine (e.g. Linux machine) is requested for an RA, reply and ask the PI whether they already have a machine, or whether they want to buy one. If they say that their grant explicitly forbids buying WSs, then they are allocated a charity case or grace and favour machine - see Special information for re-use of PCCL0xx machines for 2015/10 for more on this for the academic year 2015/16.
    • If a computer has been requested or allocated to the person and is not allocated to anyone or their end date has passed then the new person should be made the "assigned user" (i.e. User in the inventory, the person who has 'cl-asuser' and 'sudo' privileges to do things such as install software on a managed Linux PC, & have access to the machines BMC if it has one).
    • If access to servers is required Reply and CC the Sponsor/Personresponsible for each server as they need to formally agree to that access in order to meet the Cam AUP (even though almost all machines are accessible by anyone in the Computer Lab by default). In RT set the Owner as Nobody & Status as Stalled (with a suitable Due date) & leave the Queue as Sys-Admin.
  11. Finally when all the required information has been gather escalate the RT ticket by leaving on the backoffice queue with Status as new and Owner as Nobody. Backoffice will assign a machine and then handed it over to operators for the actual install of the OS and the positioning of the hardware.

When the operators have completed their tasks they will pass the RT ticket back to sys-admin for the following POST INSTALL tasks to be done:

  • 4.0 keytab install (Linux): Ensure $HOST has a correct keytab.
sudo klist -k /etc/krb5.keytab

which should have the machine name not no-name-Linux. If it is no-name-Linux then do the command below. If the command below fails, contact gt19 to create a new keytab. Login to $HOST using sudo ssh $HOST and then run:

cl-onserver --keytab
  • 4.1 User Admin - when running if needed (Linux): If oper were not told the 'assigned user', on $HOST run:
cl-asuser cl-hostid-fix --user $CRSid -a
/global/src/usr.bin/ssh/fetch-host-key scan $HOST
  • 4.8 ownfiles - at leisure if needed (Linux): to ensure that ownfiles data is collected, run
(umask 2; touch /usr/groups/linux/ownfiles/CKSUM/$HOST)
  • 4.9 WoL - at leisure: to ensure that WoL is available, run:
/usr/groups/netmaint/boot_wol_file-add.pl $HOST
  • Update the RS or "Visitors" work queue to mark the task to be "completed" in the first case, or "OK" in the second (also adding the Inventory number and name of the PC after, e.g. "OK #16200 ouse") [NOTE: The Equipment_state field requires OK Inventory No.???? MachineName to complete it and stop it being picked up as a "Outstanding equipment requests for arriving visitors/staff".]
  • Resolve the RT ticket.

Creating accounts for people arriving

See Account creation request for <CRSid> arriving <date>" or Outstanding equipment requests for arriving visitors/staff

Requests for machine moves

The move process involves:

  1. Reception allocate people to a free desk in a room using OpenRoomMap (ORM).
  2. When people are physically moved into a room, whoever moves them (e.g. the operators) checks that the OpenRoomMap (ORM) information is correct, reporting any erros to Reception, then moves them and when the move is complete updates the Staff List.
  3. Users can update their details within a room (e.g. if they move tables etc).


Requests for machine moves are raised by Reception after they allocate a new office.

  1. Take the RT ticket and check the equipment the person has in the equipment Inventory.
  2. Check what phone may need moving in the VOIPList.
  3. If there is some equipment to move then list it and pass the ticket over to the operators.
  4. Set the Queue to oper, the Owner to Nobody and with Status as new.


After the operators have moved the machine they should provide back details of which port the machine is now plugged into on the ticket.

1. Take the RT ticket and then update the inventory with the new machine's location.

2. Check that the information in the Staff List and the OpenRoomMap (ORM) is correct.

3. Using Updating VLANs in the Cisco switches note the VLAN of the old switch port and then set the old switch port where the machine was to no longer be on the VLAN it was with no switch port access vlan vlan# then set the new switch port to access the VLAN that was removed.

4. Check you can access the machine remotely

5. To check the "assigned user" on Linux desktop machines: On svr-ssh-1 or sandy run ssh -K $host ls -l /etc/user-config/bundles

 -rw-rw-r-- 1 qg216 sysadmin 3565 Nov  3 20:06 /etc/user-config/bundles

to check the user can install software as the "assigned user" (i.e. sysadmin). If not see Adding privileged or 'assigned' users

6. To check the "assigned user" access to BMC on Linux desktop machines: On svr-ssh-1 or sandy run cl-amttool $host user list

 kikyo-bmc  Registered 3 AMT user accounts:
  $$uns                  1 realm    LocalAccessPermission    Enabled  <system account>
  $$OsAdmin              6 realms   LocalAccessPermission    Enabled  <system account>
  qg216                 21 realms   AnyAccessPermission      Enabled

to check that the user has AnyAccessPermission Enabled. If not see (4.7) BMC ACL - when up if present.

Leaving Procedure

Normally a leaver is only noticed by us after (1) has been completed but...

1) Tell Reception that he has left - they update the staff list to say he's no longer Valid.

2) use the DB to find what equipment he has.

2a) Anything which is 'Lab pool' (typicallu owneed by 'RTSG', pb22, gt19, 'Lab', 'PWF' or other pseudo users) should be returned to GC20 - ask oper to do that - phone to GC12.

2b) If there is anything owned by the Host / Supervisor / PI, contact them and ask what should be done, e.g. donated to the Lab Pool, or re-deployed elsewhere.

User Can't install Software under Linux

See Adding privileged or 'assigned' users

Machine Hardware Problems

Graham Titmus (22 Apr '15)

Machines with noise issues etc.:
Change the RT ticket queue to hw-admin with status New AND email a comment Nick Baterham (nb@cl.cam.ac.uk) to alert him.

Operating System Upgrades

and

Dual boot requests

Piete Brooks (06 May '15)

Dual booting is discouraged in favour of running virtual machines instead. The options are either Computer Lab Windows + virtual Linux, or Computer Lab Linux + virtual Windows See http://www.cl.cam.ac.uk/local/sys/platforms/#dualboot

Reclaiming Equipment after someone leaves

1) Ask the visitor's sponsor if the visitor has left and if it's OK to reclaim the equipment
2) Once the sponsor has replied in the affirmative, add 'Reclaim' to the subject line and move the ticket to the Oper queue

Contacts

Primary

Availability

  • Monday:
  • Tuesday:
  • Wednesday:
  • Thursday:
  • Friday:
  • Saturday: Closed
  • Sunday: Closed

Hints, Tips & Known Issues

Finding out a machine's operating system

Graham Titmus (26/05/15)

Whilst it's only the best guess you can try logging into laira and running the command:
cl-hosts -p MachineName
to find out what operating system a machine is believed to have.


Typical "Standard Lab PC" (Linux & Windows) Spec.

Graham Titmus

 Typical Standard Computer Lab PC Spec. Dated 15/04/15:  
 
 Qty     Item            Price   Total   Description 
 1       Med-ATXB        £66     £66     Med Tower ATX Case High Efficiency 80Plus 350W PSU 
 1       SH-W163A-BL     £18     £18     Samsung Internal 22xDVD-RW SATA Black 
 1       Q87M-E  £98     £98     ASUS 1150 Q87M-E Executive Series /SI M-ATX Haswell 
 1       SV300S37A/240G  £70     £70     Kingston V300 - 240GB - Read 450MB/s - Write 450MB/s - 533 DW 
 1       i5-4670 £169    £169    Intel Haswell Core i5-4670 3.4-3.8 GHz 4 Cores, 84W, HD Graphics 4600 
 2       KVR16N11/8      £53     £106    8GB 1600MHz DDR3 Non-ECC CL11 DIMM 
 2       25SATSAS35      £11     £22     StarTech.com 2.5 inch SATA/SAS SSD/HDD to 3.5 inch SATA Hard Drive Converter 
 2       HSB100SATBK     £14     £28     StarTech.com 5.25 inch Tray-Less Hot Swap Mobile Rack for 3.5 inch Hard Drive 
 1       USB2.0Header    £3      £3      USB 2.0 Header plate 
 1       Header  £5      £5      Serial Header plate 
 1       USB3.0Header    £9      £9      USB 3.0 Header plate 
 1       K/B-Mouse-Bl    £17     £17     Microsoft USB Wired Desktop 600 Keyboard+Mouse Black 
 Total:          £611.00 
 Discount:       £30.55 
 Total Ex-VAT:   £580.45 
 VAT:            £116.09 
 Total:          £696.54
 
 to which you need to add a screen - a 4K screen costs about £300 + VAT.

Categorising Keywords

  • Resources PC machines new arrivals upgrades failures