Service Desk Knowledgebase: Linux

From Computer Laboratory System Administration
Jump to navigationJump to search


This is the Linux content page of the CL Wiki Service Desk Knowledgebase. Its purpose is to provide information to the Service Desk team on how to handle problems and requests about this CL service. If you are involved with the provision of this CL service please feel free to add to the knowledge about that it.

If CL staff need to tell the Service Desk team about problems with this service please email
sys-admin-aside@cl.cam.ac.uk.

Return to the Service Desk Knowledgebase SERVICE PORTFOLIO

Key Service Description & URLs

CL Customer Documentation

Further CL Sys-Admin Resources

Underpinning Services

  • ??? - Any supporting or underpinning services

Customer-base for this Service

  • Linux boxes are available to all staff and post-graduates, and well as some on the Part III Under-graduates.

Costs

  • Hardware is charged for if you are a Research Assistant or a University Teaching Officer, but free to Post-graduates.
  • Support is free.

SLA

  • ??? - Timeframes or service level agreement for fulfilling the service

Service Desk Call Handling Procedure

  • RT tickets can be escalated to the unix-admin by changing the Queue to unix-admin with the Owner set to Nobody & Status set to new. Tell the requestor:
    I am passing this request over to our Unix Admin team who, I'm sure, will be in contact shortly.

"It gave an error" or "It failed to work"

Piete Brooks (20 Feb 2015)

If someone says something like "It gave an error" or "It failed to work" on a Linux system please email them the following:

Dear ???,
Could you please send us a copy & paste of the command that you ran and the output that it generated? Also, would you please run the commands:

groups
sudo -l

and copy & paste the output of those into the same reply to this email.

Many thanks,
???

Removing a broken install

Vince Woodley (17 Feb 2015)

ssh to the machine in question then...

1) Find the process responsible for the lock with sudo lsof /var/lib/dpkg/lock

2) Check for running dpkg processes with something like ps -ef | grep dpkg

3) Ask the requestor if they'd like you to kill it:
I suspect I could do that by killing the rogue process -- shall I have a go?

4) Kill any dpkg processes shown above with sudo kill 1234 etc...
If they refuse to die try sudo kill -HUG 1234

5) Check each is dead with the same command again sudo kill 1234 (hopefully there will be no such process)

6) Find the exact name of the dropbox package with dpkg -l \*dropb\* (or similar)

7) Remove it with cl-asuser apt-get remove nautilus-dropbox
(or similar) If that fails with:-
"dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct" then cd /var/lib/dpkg/updates and delete all the files there with rm *

8) Do an update of the system with cl-update-system

9) Check the output to make sure that everything's okay repeating cl-update-system if necessary

Clock slew problem

Graham Titmus (3 Feb 2015)

First check if this is a physical machine or a VM. If a physical machine login to it. If a Xen VM then login to it and check if it is tied to the dom0 clock. 

cat /proc/sys/xen/independent_wallclock

If that returns an error then proceed as for a standalone machine. If it returns 0 then you need to find the dom0 which hosts the VM, to find that do 

cl-onserver --xe cl-vm-status all hid | grep <<machine_name>>

When logged in to the appropriate machine (using ssh or in the case of Xen you could also connect via the Xen Centre guy) first check if it is working correctly 

/usr/sbin/ntpdc -p

which should not look like this 

 remote local st poll reach delay offset disp
 =======================================================================
 *LOCAL(0) 127.0.0.1 10 64 377 0.00000 0.000000 0.03046

but have multiple lines each to a remote ntp server.

If it does look like above then restart the ntpd service 

 cl-asuser service ntpd restart

and check the output again which should now look like 

remote local st poll reach delay offset disp
=======================================================================
=morgul.deadset. 128.232.26.100 16 64 0 0.00000 0.000000 4.00000
=time-b.as43289. 128.232.26.100 16 64 0 0.00000 0.000000 4.00000
=LOCAL(0) 127.0.0.1 10 64 1 0.00000 0.000000 2.81735
=ntp.katho.be 128.232.26.100 16 64 0 0.00000 0.000000 4.00000
=server.netkolik 128.232.26.100 16 64 0 0.00000 0.000000 4.00000
=ntp1d.cl.cam.ac 128.232.26.100 2 64 1 0.00070 0.002260 2.81735
=ntp1c.cl.cam.ac 128.232.26.100 2 64 1 0.00165 0.002201 2.81743
=ntp1b.cl.cam.ac 128.232.26.100 2 64 1 0.00121 0.003449 2.81743
=ntp1a.cl.cam.ac 128.232.26.100 2 64 1 0.00058 0.002520 2.81735

Linux user can't login using graphical interface

Graham Titmus (13 Jan 2015)

Symptoms: Linux user can't login using graphical interface, they enter username and password and get a blank screen then back to login

A common cause of this is a failure to access the home directory stored on the File Server (AKA Elmer or Filer), X (the window manager) needs to write a file there when it starts the user session. To diagnose if this is the problem do the following:-

  1. Remote login to the machine using ssh -K hostname@cl.cam.ac.uk from a CL machine - check if your home directory is present (ls -al ~). If it is look to see if the users home directory is present (ls -al ~crsid).
  2. If the home directory is missing then try to restart the auto mounter (cl-asuser service autofs restart).
  3. Look at the mounted filesystems (grep ldap /proc/mounts - will show which systems have been auto mounted using data form the LDAP).


An alternative is to ask the user to check if it is the machine failing to log them in or a problem with X by getting them to try on the text console (Select with Ctrl-Alt-F2). If they can login there but have no home then it is probably a problem with the filesystem. If they cannot login at all then it is an authentication problem. They should then try from another machine that is known to work to check their login works.

Adding privileged or 'assigned' users

Linux PCs Assigned to Users:
Machines are setup with a single 'assigned user' having both cl-asuser access (due to owning the file /etc/user-config/bundles) and sudo access (due to being in a suitable group which has sudo rights). If the assigned user has not been setup (because a machine has been moved to a new user or was not done when the machine was installed):

  1. Go into laira using ssh -K laira from the slogin-serv gateway
  2. Login to the user's machine from laira using ssh -K $MachineName
  3. First run cl-asuser cl-hostid-fix --user $CRSid which will show you what it thinks needs to be done and (if it looks okay)
  4. Then run cl-asuser cl-hostid-fix --user $CRSid -a to actually do it.
  5. The user should be told that if they are currently logged in they must logout & log back in again for the changes to take effect.
  6. Then [Edit] & [Update] the machine names' entry(s) in the inventory and set the User: $CRSid and a Comment like RT#12345 User=$CRSid.

NOTE: If the user's account and home directory have not already been created (as in Create home directories for new users) you will get errors like: 

 chown jmt78 /etc/user-config/bundles
 chown: invalid user: ‘jmt78’
 chown jmt78 /etc/user-config/patches
 chown: invalid user: ‘jmt78’
 chown jmt78 /etc/user-config/hostid
 chown: invalid user: ‘jmt78’

and you will need to re-run the command after the account has been created.

Group Servers & PCs with multiple admins:
For group servers which may want multiple admins, they can use being in the sudo group to grant privileges to other users. Liaise with the machine owner to check what is wanted. To actually do it:

First ssh -K $hostname (if it's not turned on try cl-boot-mc on any of the slogin machines, or Wake-on-Lan (WoL) - wait 3-4 minutes for it to appear online) and then...

cl-asuser access: (if ACLs are enabled) is setup using sudo setfacl -m u:$CRSid:rw /etc/user-config/bundles where $CRSid should be replaced by the CRSid of the person who is to be granted privilege. cl-asuser privileges should then be available immediately.

sudo access: is setup by using an editor to add them to the relevant group (e.g. sudo or root) in the file /etc/group. To do this ssh -K $hostname and then:

  1. sudo vi /etc/group
  2. [sudo] password for abc123: enter your CL password
  3. Add the user's CRSid to the line like sudo:x:27:localadmin,sg692 by scrolling down to it with the arrow-keys and using [Shift]+A to enter --- INSERT --- mode and typing in ,$CRSid
  4. [ESC] out of insert mode
  5. Write and quit with :wg and [Enter]

(Note that sudo privileges will only take effect in new sessions.)

If there are sudo problems use groups $CRSid to check which groups the user is in, and sudo -l -U $CRSid to check the status. Check /etc/sudoers using sudo view sudoers and /etc/sudoers.d/* to check which groups give ALL access.

(4.7) BMC ACL - when up if present

Based on http://www.wiki.cl.cam.ac.uk/clwiki/SysInfo/MachineSetup#bmcacl Piete Brooks (23 Feb 2015)

  1. Make sure Pageant.EXE is running and has your private key by double clicking on CL.ppk or similar.
  2. With it running in the system tray launch PuTTY and go to the CL's slogin-serv.cl.cam.ac.uk
  3. Type kinit & press [Enter]
  4. Enter your CL Password for CRSid@AD.CL.CAM.AC.UK & press [Enter]
  5. Use ssh -K laira (or toton) & press [Enter] to get the laira:~$ prompt
  6. cd /home/$CRSid/ and [Enter]
  7. Check if the files .amtuser, .ipmi-user, .amtpw & .ipmi-pw already exist with ls -lA and [Enter]
  8. Only if they do not already exist do the following:
    • echo $CRSid | (umask 377; sudo -u $CRSid tee -a .amtuser) and [Enter]
    • sudo -u $CRSid cp -p .amtuser .ipmi-user and [Enter]
    • Create an 8 character password using at least one of each of:
      • lower case letter
      • UPPER CASE LETTER
      • digit
      • special characters: !@#$%^&*()
    • (umask 377; sudo -u $CRSid vi .amtpw) and [Enter]
      a for APPEND mode and then...
    • type/paste in the password then [ESC] and :wq and press [Enter]
    • sudo -u $CRSid cp -p .amtpw .ipmi-pw and [Enter]
  9. exit and [Enter]
  10. RDP to a machine on the Computer Lab network such as the Terminal Server ts01.ad.cl.cam.ac.uk, open a web-browser to the appropriate BMC interface URL i.e.:
    • For workstation - IAMT BMC: http://$host-bmc.cl.cam.ac.uk:16992
    • For WPCM450 on a server - IPMI BMC: http://$host.bmc.cl.cam.ac.uk
  11. [Login] as admin with the special password
  12. Go to User Accounts
  13. First select the previous CRSid and [Remove] & [Remove]
  14. Click [New] and enter the User name then the 8 character password created above in step 8 twice and select Administrator: Grant access to all pages then [Submit]
  15. You can't logout so just close the web-browser


NOTE: We do not normally expect the user to have to explicitly use the credentials - they are normally used by commands such as:

  • cl-amttool - which does raw commands to an IAMT BMC
  • ipmitool - which does raw commands to an IPMI BMC
  • cl-boot-mc - which boots a machine using the BMC
  • amtterm - which connects to the serial console of an IAMT WS

Linux Operating System Upgrades

Contacts

Primary

  • unix-admin RT queue

Other

Availability

  • Monday: 09:00-17:00
  • Tuesday: 09:00-17:00
  • Wednesday: 09:00-17:00
  • Thursday: 09:00-17:00
  • Friday: 09:00-17:00
  • Saturday: Closed
  • Sunday: Closed

Hints, Tips & Known Issues

The "inappropriate ioctl for device" error

Piete Brooks (20/03/15)

The "inappropriate ioctl for device" error is probably when the .profile uses the stty command to set your erase, kill, and interrupt characters e.g. 

 # The way certain characters are handled by the system are different between
 # Unixes.
 
 case $ARCH in
 sun*)   stty crt erase \^? kill \^x intr \^c ;;
 *)      stty erase \^? kill \^x intr \^c echoe susp \^z ;;
 esac

To fix the error you could comment out the use of stty in your .profile file or even chose to remove the .profile file.

Waking Up a Lab Computer which has BMC

Piete Brooks (20/03/15)

Give Wake-on-Lan (WoL) a try (and wait 3-4 minutes for it to appear online) but it's a highly unreliable protocol. It sends a packet into the ether and hopes it arrives - there is no ACK. In order to work, the client needs to have set everything up perfectly. A much better method for the 'assigned user' is to login to an slogin-serv machine and first run:
ping MachineName-bmc
and within (say) 10 seconds the BMC should be responsive. If that fails try running:
ping MachineName.bmc
Then run:
cl-boot-mc MachineName
which will use a much more helpful mechanism. If the machine is already awake you will see something like: 

 sandy:~: cl-boot-mc woc-base-00
 # /usr/bin/cl-boot-mc: already running: Powerstate: S0 (running) for woc-base-00-bmc
 cl-boot-mc: woc-base-00 did not need booting using amt
 sandy:~:

(You may need to hit Ctrl+C to end the process if the machine is actually asleep.)

Categorising Keywords

  • Linux Ubuntu PC Person Computer
Retrieved from "https://wiki.cam.ac.uk/wiki/cl-sys-admin/index.php?title=Service_Desk_Knowledgebase:_Linux&oldid=2347"