Moving the UID/GID of a user

This page documents a (draft) recommended procedure for moving the UID and/or GID of a user in the CL Unix LDAP tables.

Background

Modern POSIX operating systems (Linux, macOS, FreeBSD) expect users to have a numeric user identifier (UID) of at least 1000, as lower UIDs are reserved for pseudo-users allocated by the operating-system vendor. However, when the Computer Laboratory first used Unix file systems in the mid 1980s, that limit was still at 100. As a result, we still have, as of 2016, 20 users with UID < 1000.

You can check your own UID on the Linux shell with the “id” command, and list that of any other user with “getent passwd”.

A campaign to renumber these users and their files will have several benefits:

• their user identifier will no longer be confused with the identity of system pseudo-users (e.g., files owned by maj1 will finally show up in “ls -l” as owned by “maj1” and not as e.g. “avahi-autoipd”)
• the operating system will recognize them as regular users, enabling operations (such as certain xdm login functions) that are blocked for UIDs < 1000 (AccountsService has MINIMUM_UID hardwired at compile time in function user_classify_is_human()!)
• incorporating the LDAP tables on a self-managed computer will no longer risk interfering with the local /etc/passwd tables
• only users with UID ≥ 500 can be incorporated via libnss-extrausers (a convenient LDAP alternative for self-managed machines, laptops and home PCs)
• various local configuration hacks (e.g. changing the minimum_uid parameter of pam_krb5.so) to workaround the default UID ≥ 1000 assumption can be dropped on lab-managed Linux machines (improving defense in depth)

Affected users

User              CRSID    old_UID  old_GID    new_UID  new_GID     migration date/time
---------------------------------------------------------------------------------------
Martyn Johnson     maj1       101    60101        tbd      tbd       tbd
Piete Brooks       pb22       104    60104        tbd      tbd       tbd
Arthur Norman      acn1       107    60107        tbd      tbd       tbd
Mike Gordon        mjcg       110    60110        tbd      tbd       tbd
Peter Robinson     pr10       111    60111        tbd      tbd       tbd
Jon Fairbairn      jf15       128    60128        tbd      tbd       tbd
Larry Paulson      lp15       138    60138        tbd      tbd       tbd
Caroline Blackmun  ceb4       145    60145        tbd      tbd       tbd
Graham Titmus      gt19       178    60178        tbd      tbd       tbd
Ian Leslie         iml1       243    60243        tbd      tbd       tbd
Glynn Winskel      gw104      244    60244        tbd      tbd       tbd
Andy Hopper        ah12       260    60260        tbd      tbd       tbd
Alan Mycroft       am21       300    60300        tbd      tbd       tbd
Frank King         fhk1       301    60301        tbd      tbd       tbd
Martin Richards    mr10       302    60302        tbd      tbd       tbd
Derek McAuley      drm10      336    60336        tbd      tbd       tbd
Jean Bacon         jmb25      341    60341        tbd      tbd       tbd
Robin Fairbairns   rf10       344    60344        tbd      tbd       tbd
Ted Briscoe        ejb1       412    60412        tbd      tbd       tbd
Ann Copestake      aac10      432    60432        tbd      tbd       tbd

List produced with

/homes/mgk25/proj/filer/ldap_uids.pl -r100,499 -L

Prerequisites

A number of facilities should first be set up by sys-admin to make the moving progress simpler and more reliable:

1. At least one of the omnipotent NFS clients (radyr, laira, toton) must be configured to use NFSv3. On Ubuntu: add the autofs mount option "nfsvers=3" to the OPTIONS parameter in configuration file /etc/default/autofs, i.e. add OPTIONS="-Onfsvers=3"
2. If that's a VM, make sure it has enough RAM (4 GB should be fine) to support a ugid-scan job that accumulates in the region of 1 gigabyte of heap memory.
3. Install the ugid-scan search engine to make it easy to find files on the filer by uid/gid. For details, see RT#102400
4. On the UserAdminDetails interface, add for any user with uid < 1000 a "Move UID" button that allocates for the user a new UID in the range 1100-8999 (for normal users) or 9000-9499 (for pseudo-users) and updates their uid field accordingly.

Moving process

Preparation

1. Inform user of the planned date and time of the change and advise them to log out and (ideally) also terminate long-running processes during the migration period.
2. Ask the user on what other POSIX filesystems than those on elmer they own files (local disks of desktops, servers and virtual machines connected to the Unix LDAP servers)
3. Make a note of their old numeric UID and GID (e.g. in table above). We will refer to these as $old_UID and $old_GID.

Actual move

4. Update in the administrative database their LDAP UID to the new value, and make a note of it (e.g. in table above). Since their LDAP GID is autogenerated (if 100 ≤ UID < 500 then GID := UID+60000, else GID := UID ?), their GID will automatically move at the same time. We will refer to their new identifiers as $new_UID and $new_GID. As per the new departmental UID/GID allocation plan:
   • If the user is a person identified by CRSID: make sure 1100 ≤ $new_UID = $new_GID < 9000.
   • For a pseudo-user: make sure 9000 ≤ $new_UID = $new_GID < 9500.
5. If the user might have any Windows NTFS files on the filer, then wait until the new UID and GID have propagated to
   • all LDAP servers;
   • elmer’s name server database cache (NSDB), which you can force with the nfs nsdb flush command;
   • your NFSv3 client attribute cache, which should timeout after 60 seconds by default (see “man nfs”).

   After that, the user's Windows NTFS files should all show as their owner the new UID via NFSv3.

6. Log into one of the omnipotent NFSv3 clients. Check with "mount" that it really mounts elmer with nfsvers=3 (so we can see untranslated UID/GID values).
7. As "root", use ugid-find make lists of directory prefixes of the locations of files that still have the old UID or GID:

    ugid-find uid=$old_UID prefixes dirnames print >uiddir-$old_UID
    ugid-find gid=$old_GID prefixes dirnames print >giddir-$old_GID
   

8. Review the directory prefix lists to see whether they look plausible, and manually remove directories that are very unlikely to contain files that belong to the user to be migrated. (This is especially important if the renumbering is needed to resolve a collision, i.e. where multiple people have used the same numeric identifier in the past, or where the same numeric identifier was also used by an operating-system service that might own files on the filer.)
9. Then run as "root":

     xargs -r <uiddir-$old_UID chown -hcR --from=$old_UID $new_UID
     xargs -r <giddir-$old_GID chgrp -hcR --from=$old_GID $new_GID
   

Completion

9. Finally, notify the user that they can login again.
10. Remind the user to run the following commands as root on their own local Unix file systems, or do it for them, as agreed:

      chown -hcR --from=$old_UID $new_UID path ...
      chgrp -hcR --from=$old_GID $new_GID path ...
    

Some notes on the process

Dealing with Windows NTFS files

The filer can store files with either Unix or Windows NTFS access-control attributes. We do not need and want to touch files with Windows NTFS attributes for several reasons:

• Executing a chown/chgrp on a Windows NTFS file will turn it into a Unix file, and potentially destroy access-control list information as a result.
• There is no need to chown/chgrp a Windows NTFS file, because such files have Windows SIDs instead of UIDs

The trick is to first change the UID and GID of a user in the Unix LDAP tables, and then wait until that change has propagated through to the NetApp name server database cache. When a Unix user queries an NTFS file with stat(), the NetApp will first translate the SID of the file via an Active Directory LDAP lookup into a user name (CRSID). It will then translate that CRSID via a Unix LDAP lookup into the UID that stat() shows.

Once the UID and GID change in the Unix LDAP server has reached the NetApp name server database cache, all Windows NTFS files will show already the new (emulated) UID/GID via NFSv3, and therefore be ignored by the subsequent ugid-find and chown/chgrp invocations. This way, only Unix files are affected by the migration.

NFS version

It is very important that the commands

• ugid-scan
• ugid-find
• chown
• chgrp

are only executed on NFSv3 clients. This is because they are dealing here with UID/GID numbers that are no longer listed in LDAP, and the NFSv4 server will map all of these to nobody/nogroup. NFSv4 does not communicate numeric UID or GID values, instead it uses LDAP to translate these into “CRSID@cl.cam.ac.uk” strings on the wire. This would be counterproductive here.

Kerberos/AD identity

During the migration process, the Active Directory Kerberos identity (principal name, SID, etc.) remains unaffected. We are only changing the LDAP UID/GID. Therefore we are not creating a new user identity and deleting an old one in the Active Directory. This way, there will be hardly any disruption to Windows users, and all Kerberos and Active Directory metadata associated with the user remains intact.

Dealing with odd filenames

The above recipe uses the ugid-find “print” command to generate a list of LF-terminated path names. This is to make it easy to manually review and edit the generated list of directory prefixes with a text editor. However, this format might break in case one of the listed pathnames contains a linefeed character. This is very unlikely in Unix-originated filenames, but does occasionally occur with Windows Explorer generated files. The ugid-find “print0” and the xargs option -0 can be used to exchange a NUL terminated list of pathnames, which is immune to such problems.

A more sensible approach is to rename filenames that contain an LF, which is very likely an accident.

Dealing with long-running processes

If the user has long-running processes that they do not want to stop for the duration of the migration, one possible approach is to arrange that these processes have continued access to their files via their GID, while their UID is being changed.

See also

• UID/GID allocation
• ugid-scan UID/GID file search engine