Subject: lxc: cannot start Fedora30 and Debian Stretch in LXC containers

From: Ryutaroh Matsumoto <[email-masked]>
Date: Fri, 24 Apr 2020 18:47:13 +0900 (JST)

Package: lxc
Version: 1:3.1.0+really3.0.3-8
Severity: grave
Justification: renders package unusable

Dear Maintainer,

With the default setting, Devuan LXC package cannot start Fedora 30 and
Debian Stretch in LXC containers.
I suspect that LXC on Devuan always fails if /sbin/init in a container is systemd.

To reproduce the bug, do 

root@devuan:~# lxc-create -n stretch -t download -- -d debian -a amd64 -r stretch
Using image from local cache
Unpacking the rootfs

---
You just created a Debian stretch amd64 (20200424_05:24) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.
root@devuan:~# lxc-start -F -n stretch
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

root@devuan:~# lxc-create -n fedora30 -t download -- -d fedora -a amd64 -r 30
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Fedora 30 x86_64 (20200423_20:33) container.
root@devuan:~# lxc-start -F -n fedora30
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
root@devuan:~# exit 0

Best regards, Ryutaroh Matsumoto

-- System Information:
Distributor ID:	Debian
Description:	Devuan GNU/Linux 3 (beowulf)
Release:	3
Codename:	beowulf
Architecture: x86_64

Kernel: Linux 4.19.0-8-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=C.UTF-8 (charmap=locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
UTF-8), LANGUAGE=en_US:en (charmap=locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)
LSM: AppArmor: enabled

Versions of packages lxc depends on:
ii  debconf [debconf-2.0]  1.5.71
ii  libc6                  2.28-10
ii  libcap2                1:2.25-2
ii  libgnutls30            3.6.7-4+deb10u3
ii  liblxc1                1:3.1.0+really3.0.3-8
ii  libseccomp2            2.3.3-4
ii  libselinux1            2.8-1+b1
ii  lsb-base               10.2019051400

Versions of packages lxc recommends:
ii  apparmor                     2.13.2-10
ii  bridge-utils                 1.6-2
pn  debootstrap                  <none>
ii  dirmngr                      2.2.12-1+deb10u1
ii  dnsmasq-base [dnsmasq-base]  2.80-1
ii  gnupg                        2.2.12-1+deb10u1
ii  iproute2                     4.20.0-2
ii  iptables                     1.8.2-4
pn  libpam-cgfs                  <none>
pn  lxc-templates                <none>
pn  lxcfs                        <none>
ii  nftables                     0.9.0-2
ii  openssl                      1.1.1d-0+deb10u3
ii  rsync                        3.1.3-6
ii  uidmap                       1:4.5-1.1

Versions of packages lxc suggests:
ii  btrfs-progs  4.20.1-2
pn  lvm2         <none>
pn  python3-lxc  <none>

-- debconf information:
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US:en",
	LC_ALL = (unset),
	LC_CTYPE = "C.UTF-8",
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
  lxc/auto_update_config:

---

From: Mark Hindley <[email-masked]>
Date: Fri, 24 Apr 2020 18:02:25 +0100

Ryutaroh,

Thanks for this.

> I suspect that LXC on Devuan always fails if /sbin/init in a container is

Maybe. I have just had a quick look at Debian's lxc bugs and there are a number
relating to failures with non-systemd inits. See #706676 and #869892 for
starters.

liblxc1 depends cgroupfs-mount | systemd. I have never used cgroupfs-mount or
lxc, but can you check the instructions in
/usr/share/doc/cgroupfs-mount/README.md particularly relating to removing other
cgroupfs mounts from /etc/fstab.

Does that help?

Mark

---

From: Mark Hindley <[email-masked]>
Date: Fri, 24 Apr 2020 18:05:47 +0100

The other thing that occurred to me is whether apparmor is interfering? The
`Operation not permitted' is suspicious of that.

There is a Debian wontfix bug relating to it: #947863.

Do you have apparmor enabled? If so does disabling it help?

Thanks

Mark

---

From: Ryutaroh Matsumoto <[email-masked]>
Date: Sat, 25 Apr 2020 04:47:13 +0900 (JST)

Hi Mark,

Thanks for your help.
I was able to use Gentoo in the LXC container on Devuan.
So this is not a general problem of LXC.
It is specific to Devuan hosts and systemd containers.

Booting with apparmor=0 did not help.
I selected openrc. Replacing with sysv-rc did not help.

I wonder what is the cgroup mounts without cgroupfs-mount, so
I tried apt-get purge cgroupfs-mount, and got the following.
I have no idea how to remove cgroupfs-mount from Devuan 3:
Removing cgroupfs-mount (1.4) ...
Unmounting cgroupfs hierarchyumount: /sys/fs/cgroup/elogind: target is busy.
invoke-rc.d: initscript cgroupfs-mount, action "stop" failed.
dpkg: error processing package cgroupfs-mount (--remove):
 installed cgroupfs-mount package pre-removal script subprocess returned error exit status 32
Errors were encountered while processing:
 cgroupfs-mount
E: Sub-process /usr/bin/dpkg returned an error code (1)

The content of /proc/mounts is as follows:
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=2004912k,nr_inodes=501228,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=403952k,mode=755 0 0
/dev/vda2 / btrfs rw,noatime,space_cache,subvolid=5,subvol=/ 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
pstore /sys/fs/pstore pstore rw,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=1646140k 0 0
/dev/vda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro 0 0
securityfs /sys/kernel/security securityfs rw,relatime 0 0
cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,relatime,pids 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,relatime,rdma 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/elogind cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/elogind/elogind-cgroups-agent,name=elogind 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=403948k,mode=700 0 0

Stating that Devuan LXC does not support distros with systemd
and closing this but might be a good idea...

Best regards, Ryutaroh

From: Mark Hindley <[email-masked]>
Subject: Re: lxc: cannot start Fedora30 and Debian Stretch in LXC containers
Date: Fri, 24 Apr 2020 18:05:47 +0100

---

From: Mark Hindley <[email-masked]>
Date: Sat, 25 Apr 2020 10:48:38 +0100

> It is specific to Devuan hosts and systemd containers.

Helpful to know.

> I selected openrc. Replacing with sysv-rc did not help.

Thanks for trying.

> I have no idea how to remove cgroupfs-mount from Devuan 3:

See my answer to #428.

> and closing this but might be a good idea...

Yes, it might come to that. However, I would like to exhaust other possibilities
first.

A few other ideas to try: obviously the lxc container is looking for the host
/sys/fs/cgroup/systemd which doesn't exist. Does creating it manually help?
Does installing cgmanager help (I am not sure if that will create and manage
/sys/fs/cgroup/systemd)?

Mark

---

From: Ryutaroh Matsumoto <[email-masked]>
Date: Mon, 27 Apr 2020 11:48:03 +0900 (JST)

Hi Mark,

> Does installing cgmanager help (I am not sure if that will create and manage

None of them worked.
My Devuan is running on qemu.
I can upload the image so that you can try on your computer.

Best regards,
Ryutaroh

From: Mark Hindley <[email-masked]>
Subject: Re: lxc: cannot start Fedora30 and Debian Stretch in LXC containers
Date: Sat, 25 Apr 2020 10:48:38 +0100

---

From: Ryutaroh Matsumoto <[email-masked]>
Date: Wed, 06 May 2020 07:24:00 +0900 (JST)

Hi Mark,

From https://wiki.debian.org/LXC#Preparing_host_system_WITHOUT_Systemd_for_running_LXC

According to Gentoo wiki, if you want to run containers with systemd, the host also needs a name=systemd cgroup hierarchy mounted: mkdir -p /sys/fs/cgroup/systemd; mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd.

With that, lxc-checkconfig no longer complains and containers with systemd are able to start.

This was reported to package cgroupfs-mount in bug #939435 (patch included).

Best regards, Ryutaroh

From: Ryutaroh Matsumoto <[email-masked]>
Subject: Re: lxc: cannot start Fedora30 and Debian Stretch in LXC containers
Date: Mon, 27 Apr 2020 11:48:03 +0900 (JST)

---

From: Mark Hindley <[email-masked]>
Date: Wed, 6 May 2020 09:34:42 +0100

Control: reassign -1 cgroupfs-mount
Control: tags -1 debian

> This was reported to package cgroupfs-mount in bug #939435 (patch included).

Well done for your persitence and finding a solution!

Reassigning and tagging.

Thanks

Mark