How to Free up Space on the Control Domain⚓

Summary⚓

There are a number of reasons that a control domain could become full. This is interesting because in my case, the filesystem itself was not full, but was generating random errors while trying to create backups in Xen Orchestra.

The problem itself ended up being the partition containing the log files became full and needed to be cleared out. This will all come down to exactly how the partition layout is on the domain.

Possible Errors⚓

Possible errors can include the following errors or alerts might appear in XenCenter:

• File System on Control Domain Full.
• Disk usage for the Control Domain on server 'XenServer' has reached 99.0%. XenServer's performance will be critically affected if this disk becomes full.
• Log files or other non-essential (user created) files should be removed.
• The update precheck stage failed: the server does not have enough space.

Possible symptoms can include the following:

• "No space left on the device" error when attempting to apply hotfixes, cumulative updates, or current releases as an update to the XenServer host.
• Performance of the VMs across the pool or host becomes sluggish.
• Host becomes completely unresponsive.
• XenServer console will show error: "The XenAPI connection has timed out , the console will have reduced functionality".
• All the Consoles, for hosts or VM, become unresponsive.
• Error while trying to update hotfixes: "No space left on the device".
• Any command or any process on the host takes too long to execute.
• Host will disconnect from XenCenter and will not connect back HA monitor thread stops without warning. Running VMs are no longer protected from host failure.

Verify File System⚓

To verify that a full file system is the cause of these symptoms, look for the following after running df -h

Command output on a host with the new partition layout:

Filesystem            Size  Used Avail Use% Mounted on
devtmpfs              951M  0    951M    0% /dev
tmpfs                 962M  72K  962M    1% /dev/shm
tmpfs                 962M  948K 961M    1% /run
tmpfs                 962M  0    962M    0% /sys/fs/cgroup
/dev/sda1             18G   17G  1.3G    93% /
xenstore              962M  0    962M    0% /var/lib/xenstored
/dev/loop0            45M   45M  0     100% /var/xen/xc-install
/dev/sda5             3.9G  12M  3.7G    1% /var/log
tmpfs                 193M  0    193M    0% /run/user/0

If the value of the 'Use%' for the file system mounted on / is greater than 90%, your Control Domain file system is full and you must free up space. You must also free up space in cases where the Use% is less than 90% but you are getting error messages that state that you do not have enough space to complete the action you are attempting (for example, applying an update).

Freeing up Space on the Control Domain Filesystem⚓

1. Remove additional logs

Legacy partition layout only: You can clear surplus logs for all XenServer hosts, but it will free space in the Control Domain filesystem only for those hosts that have the legacy partition layout.

1. Change directory into the /var/log directory # cd /var/log
2. Remove old compressed logs (.gz files). Do not remove *.log files as this will break log rotation and requires a reboot of the host to fix. # rm *.gz

Refer to the following screen shot for an example command. This command ensures that all the current logs files will be intact and will not be deleted.

2. Remove cached uploaded patch files**For XenServer 7.0 and earlier:1. Change directory into /var/patch
# cd /var/patch**
1. Remove all the entries except the "/applied" folder
2. Use the following command to remove all the uploaded patches and without deleting the applied folder
# find /var/patch -maxdepth 1 -type f -exec rm -v {} +**For XenServer 7.1 and later:1. Change directory into /var/update
# cd /var/update
2.** Remove all the entries except the "/applied" folder
3. Use the following command to remove all the uploaded patches and without deleting the applied folder
# find /var/update -maxdepth 1 -type f -exec rm -v {} +

3. Remove the patch-backups

1. cd /opt/xensource/patch-backup
2. ls
3. rm -rf *

4. Remove stunnel entries

1. Change to the tmp directory
# cd /tmp/
1. Remove any old stunnel log entries

Note: If these steps do not reflect the space being cleared, run #**service syslog restart a**nd the usage inside the root file system will be cleared.

5. Remove additional files

• Check **/tmp**directory for any rogue files. You can delete everything inside it as XenServer won't store any default files in it.
• Check "~" and "/" to see if there are any additional ISO files or any extra packages are present inside the directory. If yes, delete them.

Reference⚓

• https://support.citrix.com/article/CTX202821