UID/GID allocation: Difference between revisions

From Computer Laboratory System Administration
Jump to navigationJump to search
(→‎Potential changes: added prognosis of space exhaustion)
(Moved departmental groups and pseudo-user range from 5xxx to 9xxx (defragmentation))
Line 14: Line 14:
#* Unix tools typically display UID/GID values as decimal numbers and therefore the structure of the number space should be easy to recognize in decimal
#* Unix tools typically display UID/GID values as decimal numbers and therefore the structure of the number space should be easy to recognize in decimal
#* 4-digit just looks neater in "id", etc. than a mix of 4- and 5-digit numbers
#* 4-digit just looks neater in "id", etc. than a mix of 4- and 5-digit numbers
#* 4-digit UIDs are going to last at least until the year 2050 at current growth rates (~150 new entries every year, e.g. October 2009: max_uid=2659, October 2014: max_uid=3365).
# UID and GID allocations should be grouped together by type (regular users, pseudo-users, etc.)
# UID and GID allocations should be grouped together by type (regular users, pseudo-users, etc.)
#* This is mostly for the benefit of users who see lists of UIDs and GIDs sorted by numeric value, e.g. with the Unix "id" command or in administrative tools and tables.
#* This is mostly for the benefit of users who see lists of UIDs and GIDs sorted by numeric value, e.g. with the Unix "id" command or in administrative tools and tables.
Line 32: Line 34:
* 0000-0999: not used in LDAP, reserved for OS-specific system entries or client-specific user entries
* 0000-0999: not used in LDAP, reserved for OS-specific system entries or client-specific user entries
* 1000-1099: not used in LDAP, reserved for client-specific user entries
* 1000-1099: not used in LDAP, reserved for client-specific user entries
* 1100-4999: departmental users (real people as CRSId)
* 1100-8999: departmental users (real people as CRSId)
* 5000-5499: departmental pseudo users (role accounts, daemon accounts, group accounts, etc.)
* 9000-9499: departmental pseudo users (role accounts, daemon accounts, group accounts, etc.)
* 5500-5999: not used, to avoid collisions of corresponding personal groups with departmental groups
* 9500-9999: not used, to avoid collisions of corresponding personal groups with departmental groups
* 6000-8999: departmental users (real people as CRSId)
* 9000-9999: reserved for future allocation by department
* 10000-: not used in LDAP, reserved for client-specific temporary allocations (e.g., Linux container namespace remapping)
* 10000-: not used in LDAP, reserved for client-specific temporary allocations (e.g., Linux container namespace remapping)


Line 43: Line 43:
* 0000-0999: not used by LDAP, reserved for OS-specific entries or personal groups of client-specific user entries
* 0000-0999: not used by LDAP, reserved for OS-specific entries or personal groups of client-specific user entries
* 1000-1099: not used in LDAP, reserved for personal groups of client-specific user entries
* 1000-1099: not used in LDAP, reserved for personal groups of client-specific user entries
* 1100-5499: departmental personal groups associated with the corresponding UID name and value
* 1100-9499: departmental personal groups associated with the corresponding UID name and value
* 5500-5999: departmental groups
* 9500-9999: departmental groups
* 6000-8999: departmental personal groups associated with the corresponding UID name and value
* 9000-9999: reserved for future allocation by department
* 10000-: not used in LDAP, reserved for client-specific temporary allocations (e.g., Linux container namespace remapping)
* 10000-: not used in LDAP, reserved for client-specific temporary allocations (e.g., Linux container namespace remapping)


== Migration of legacy entries ==
== Migration of legacy entries ==


As of 2014-09-10, there are still 53 UIDs and 103 GIDs allocated in the LDAP tables in the range 0-1099. They should all be moved to higher 4-digit positions eventually, according to the new scheme. There is also one regular user at 5187 and one at 15384. Some cause more problems than others, or are easier to move than others, and should therefore be prioritized.
As of 2014-09-10, there are still 53 UIDs and 103 GIDs allocated in the LDAP tables in the range 0-1099. They should all be moved to higher 4-digit positions eventually, according to the new scheme. There is also one regular user at 15384. Some cause more problems than others, or are easier to move than others, and should therefore be prioritized.


* existing departmental groups in the range 500-999 can be moved to 5500-5999 by adding 5000 to their GID
* existing departmental groups in the range 500-999 can be moved to 9500-9999 by adding 9000 to their GID
* existing departmental pseudo-users (and associated personal groups) in the range 500-999 can be moved to 5000-5499 by adding 4500 to their UID/GID:
* existing departmental pseudo-users (and associated personal groups) in the range 500-999 can be moved to 9000-9499 by adding 8500 to their UID/GID:
  weather:501:dtg weather data gathering user
  weather:501:dtg weather data gathering user           -> 9001
  dtg-backup:502:dtg data backup user
  dtg-backup:502:dtg data backup user                   -> 9002
  qosmos:503:qosmos box file transfer user
  qosmos:503:qosmos box file transfer user             -> 9003
  wwwsvn:505:WWW SVN user id
  wwwsvn:505:WWW SVN user id                           -> 9005
  nprobe:506:nprobe pseudo-user
  nprobe:506:nprobe pseudo-user                         -> 9006
  dtg-time:507:DTG TIME project
  dtg-time:507:DTG TIME project                         -> 9007
  apache-fp:509:apache fp
  apache-fp:509:apache fp                               -> 9009
  wwwupdate:510:WWW updater
  wwwupdate:510:WWW updater                             -> 9010
  ivc:511:ivc pseudo user
  ivc:511:ivc pseudo user                               -> 9011
  ucard:512:University Card Office
  ucard:512:University Card Office                     -> 9019
  sec-repos:519:Security group svn repository
  sec-repos:519:Security group svn repository           -> 9019
  ugprac:561:Undergraduate Practicals
  ugprac:561:Undergraduate Practicals                   -> 9061
  energy:591:Energy logs from monitors
  energy:591:Energy logs from monitors                 -> 9091
  keytab:596:keytab
  keytab:596:keytab                                     -> 9096
  www-cl:888:Local www server
  www-cl:888:Local www server                           -> 9080


* particular care is needed where existing departmental groups collide numerically with existing departmental pseudo-users and their personal groups:
* particular care is needed where existing departmental groups collide numerically with existing departmental pseudo-users and their personal groups:
Line 158: Line 156:
On many Linux systems, the <code>USERGROUPS_ENAB</code> variable in <code>/etc/login.defs</code> controls whether commands like <code>useradd</code> or <code>userdel</code> automatically add or delete an associated personal group.
On many Linux systems, the <code>USERGROUPS_ENAB</code> variable in <code>/etc/login.defs</code> controls whether commands like <code>useradd</code> or <code>userdel</code> automatically add or delete an associated personal group.


== Potential changes ==
== Change history ==
 
* In the above proposal, the range allocated for pseudo-users and groups is 5xxx, just to stay visually backwards compatible with the 5xx range used previously for that purpose. If that were instead moved to the top of the 4-digit range at 9xxx, then it would avoid fragmentation of the space allocated to regular users.


* The current growth rate of regular user allocations seems in the region of 150 new entries every year (October 2009: max_uid=2659, October 2014: max_uid=3365). At that rate, the highest UID of a regular user would reach 9000 in the year 2050.
* In an earlier version of the above proposal, the range allocated for pseudo-users and groups was 5xxx, just to stay visually backwards compatible with the 5xx range used previously for that purpose. Moving that to 9xxx instead avoids fragmentation of the space allocated to regular users until about 2050 at current growth rates.


== Reviewing tools ==
== Reviewing tools ==

Revision as of 12:39, 7 July 2015

The departmental Unix LDAP servers (ldap-serv1.cl.cam.ac.uk, etc.) export lists of users and groups, equivalent to the local Unix files /etc/passwd and /etc/group. This page is a draft policy for how in future the numerical user and group IDs exported by the departmental Unix LDAP servers should be allocated.

One important goal of this policy is to ensure that self-managed machines can import the user and group tables offered by the departmental LDAP servers without risking collisions with entries defined locally on the client machine by the operating system or the user.

Rules and considerations

  1. The LDAP servers should not export any numeric UID or GID with a value below 1100.
    • This mainly helps to avoid numeric collisions with UID and GID allocations by operating systems, which can cover the whole range 0-999.
    • This helps to avoid collisions with locally created users (e.g., family members on a private laptop), which which some operating systems allocate starting at 1000 upwards.
  2. The LDAP servers should also not export any numeric UID or GID of -2, -1, 32767, 65534, or 65535.
    • This helps to avoid collisions with the POSIX return value -1 of some related system calls, as well as with the entries "nobody" and "nogroup" used on some systems.
  3. Departmental numeric UID and GID allocations should preferably only use 4-digit decimal numbers, at least until the available space there is exhausted.
    • this leaves numbers 10000- available for temporary local use, such as remapping of UIDs/GIDs by Linux containers by adding an integer multiple of 10000 for each container namespace
    • Unix tools typically display UID/GID values as decimal numbers and therefore the structure of the number space should be easy to recognize in decimal
    • 4-digit just looks neater in "id", etc. than a mix of 4- and 5-digit numbers
    • 4-digit UIDs are going to last at least until the year 2050 at current growth rates (~150 new entries every year, e.g. October 2009: max_uid=2659, October 2014: max_uid=3365).
  1. UID and GID allocations should be grouped together by type (regular users, pseudo-users, etc.)
    • This is mostly for the benefit of users who see lists of UIDs and GIDs sorted by numeric value, e.g. with the Unix "id" command or in administrative tools and tables.
  2. For each user name and numeric UID, the corresponding identical group name and numeric GID is reserved for the respective personal group.
    • Personal groups have the eponymous user as their sole member.
  3. Names of users and groups should be chosen to avoid likely collisions with operating-system allocations
  4. Files stored on the departmental file space should avoid using any numeric UID or GID value below 1100 or above 9999
    • the meaning of allocations in this range can differ between NFS client machines.

UID range allocation

  • 0000-0999: not used in LDAP, reserved for OS-specific system entries or client-specific user entries
  • 1000-1099: not used in LDAP, reserved for client-specific user entries
  • 1100-8999: departmental users (real people as CRSId)
  • 9000-9499: departmental pseudo users (role accounts, daemon accounts, group accounts, etc.)
  • 9500-9999: not used, to avoid collisions of corresponding personal groups with departmental groups
  • 10000-: not used in LDAP, reserved for client-specific temporary allocations (e.g., Linux container namespace remapping)

GID range allocation

  • 0000-0999: not used by LDAP, reserved for OS-specific entries or personal groups of client-specific user entries
  • 1000-1099: not used in LDAP, reserved for personal groups of client-specific user entries
  • 1100-9499: departmental personal groups associated with the corresponding UID name and value
  • 9500-9999: departmental groups
  • 10000-: not used in LDAP, reserved for client-specific temporary allocations (e.g., Linux container namespace remapping)

Migration of legacy entries

As of 2014-09-10, there are still 53 UIDs and 103 GIDs allocated in the LDAP tables in the range 0-1099. They should all be moved to higher 4-digit positions eventually, according to the new scheme. There is also one regular user at 15384. Some cause more problems than others, or are easier to move than others, and should therefore be prioritized.

  • existing departmental groups in the range 500-999 can be moved to 9500-9999 by adding 9000 to their GID
  • existing departmental pseudo-users (and associated personal groups) in the range 500-999 can be moved to 9000-9499 by adding 8500 to their UID/GID:
weather:501:dtg weather data gathering user           -> 9001
dtg-backup:502:dtg data backup user                   -> 9002
qosmos:503:qosmos box file transfer user              -> 9003
wwwsvn:505:WWW SVN user id                            -> 9005
nprobe:506:nprobe pseudo-user                         -> 9006
dtg-time:507:DTG TIME project                         -> 9007
apache-fp:509:apache fp                               -> 9009
wwwupdate:510:WWW updater                             -> 9010
ivc:511:ivc pseudo user                               -> 9011
ucard:512:University Card Office                      -> 9019
sec-repos:519:Security group svn repository           -> 9019
ugprac:561:Undergraduate Practicals                   -> 9061
energy:591:Energy logs from monitors                  -> 9091
keytab:596:keytab                                     -> 9096
www-cl:888:Local www server                           -> 9080
  • particular care is needed where existing departmental groups collide numerically with existing departmental pseudo-users and their personal groups:
502: dtg-backup = epson
507: dtg-time = lt-video
509: apache-fp = diss-upload
  • existing departmental regular users outside this allocation scheme should be moved to new 4-digit locations that would otherwise be allocated to the next new users, with particular priority to be be given to those in the range 101-128 (which is densely populated by Linux system users):
maj1:101:Martyn Johnson
pb22:104:Piete Brooks
acn1:107:Arthur Norman
mjcg:110:Mike Gordon
pr10:111:Peter Robinson
jf15:128:Jon Fairbairn
lp15:138:Larry Paulson
ceb4:145:Caroline Blackmun
gt19:178:Graham Titmus
iml1:243:Ian Leslie
gw104:244:Glynn Winskel
ah12:260:Andy Hopper
am21:300:Alan Mycroft
fhk1:301:Frank King
mr10:302:Martin Richards
drm10:336:Derek McAuley
jmb25:341:Jean Bacon
rf10:344:Robin Fairbairns
ejb1:412:Ted Briscoe
aac10:432:Ann Copestake
djg11:1004:David Greaves
km10:1077:Ken Moody
rmm1002:5187:Richard Mortier
sqlcache:10001:Linux cache of SQL server data
visitor1:11105:Visitor account 1
visitor2:11106:Visitor account 2
visitor3:11107:Visitor account 3
visitor4:11108:Visitor account 4
visitor5:11109:Visitor account 5
tlh20:15384:Tim Harris
Moving Linux desktop users with UID < 500 will immediately benefit them by resolving an annoying XDM malfunction on Ubuntu 14.04: they will finally be classified as human users and will therefore be able to login comfortably, as they will no longer be excluded from those offered for mouse selection in the login GUI.
  • Some active pseudo-users are currently located among the regular users:
visitor6:3225:Visitor account 6
visitor7:3226:Visitor account 7
visitor8:3227:Visitor account 8
visitor9:3228:Visitor account 9
visitor0:3229:Visitor account 0
  • Some departmental groups are currently located among the personal groups of regular users, which will soon cause problems when adding personal groups for future new users:
isabelle:3109:gp351,jpb65,lp15,wd239,zh242
dtg:3600:ab818,abr28,acr31,ags46,ah12,arb33,awm22,bdj23,drt24,dtgbackup,dtw30,fms27,gfc22,gpb29,ijw24,jas250,jsf29,lc525,ml421,oc243,rkh23,rmf25,rss39,sa497,sak70,sdp36,sja55,sjh227,tb403,thc33,wc253,xd225,zf232
dtg-cvs:3601:bdj23,jsf29,rss39
dtg-sys:3602:bdj23
dtg-ab:3603:bdj23
dtg-web:3604:arb33,bdj23,rkh23
dtg-qos:3605:gfc22,rss39
dtg-clan:3606:root,rss39
dtg-sprt:3607:acr31,arb33,rkh23
dtg-total:3608:acr31,rkh23
www-cla:3609:
www-grppr:3610:afb21,rja14
WARNING: The current (2015-04-29) highest regular user is has a UID of 3487. Once that number reaches 3600, the personal group of new users will collide with the dtg-* groups!

Background information

Linux conventions

The Linux Standard Base Core Specification specifies that UID values in the range 0 to 99 should be statically allocated by the system, and shall not be created by applications, while UIDs from 100 to 499 should be reserved for dynamic allocation by system administrators and post install scripts. See also LSB 3.0, Section 9.3: UID Ranges.

In practice, Linux distributions start allocating local regular user IDs from either 500 (Red Hat) or 1000 (SUSE, Debian). On many Linux systems, these ranges are specified in /etc/login.defs, for useradd and similar tools.

Some tools distinguish between real users and system pseudo-users based on the UID range. On Ubuntu, the LightDM display manager configuration file /etc/lightdm/users.conf contains per default the line “minimum-uid=500” and users with a lower UID are not shown in the login-screen greeter. However, if AccountsService is installed, lightdm [ allegedly] uses its definition instead, which appears to be configured at compile time.

Mac OS X conventions

Mac OS X allocates locally created users from 500 upwards and uses the UIDs and GIDs 0-499 for the operating system.

FreeBSD conventions

For both UIDs and GIDs, the range 0-49 is reserved for core OS allocations, and the range 50-999 can be allocated by package porters for use by specific software packages in the files ports/UIDs and ports/GIDs. These ranges are already quite densely populated. See also FreeBSD Porter's Handbook, Section 6.26: Adding Users and Groups.

Personal groups

For each user, a corresponding group can be created that has the same name and numeric identifier, known as the personal group. Such personal groups have no other members and make collaboration with other users in shared directories easier, by allowing users to habitually work with umask 0002. This way, newly created files can have by default write permissions enabled for group members, because this will normally only enable write access for members of the personal group, that is only for the file's owner. However, if a file is created in a shared directory that belongs to another group and has the setgid bit set, then the created file will automatically become writeable to members of that directory's group as well.

On many Linux systems, the USERGROUPS_ENAB variable in /etc/login.defs controls whether commands like useradd or userdel automatically add or delete an associated personal group.

Change history

  • In an earlier version of the above proposal, the range allocated for pseudo-users and groups was 5xxx, just to stay visually backwards compatible with the 5xx range used previously for that purpose. Moving that to 9xxx instead avoids fragmentation of the space allocated to regular users until about 2050 at current growth rates.

Reviewing tools

List and test UIDs from LDAP server:

$ /homes/mgk25/proj/filer/ldap_uids.pl

List and test GIDs from LDAP server:

$ /homes/mgk25/proj/filer/ldap_gids.pl

(Requires Ubuntu package libnet-ldap-perl)

See also