Subject: bug#483: eudev: eudev sometimes fails to generate initial hotplug events

From: David Kuehling <[email-masked]>
Date: Mon, 15 Jun 2020 17:08:38 +0200

Package: eudev
Version: 3.2.7-6
Severity: normal
File: /etc/init.d/eudev

Dear Maintainer,

eudev sometimes fails to generate the initial hotplug events for one of
my systems.  The symptoms are that many required kernel modules are
missing after boot (no sound, no 3-D acceleration).

The error is not determinstic.  Sometimes the system boots up
successfully and everything works.  Sometimes the system boots up and
essential kernel modules are missing.

Doing '/etc/init.d/eudev stop' followed by '/etc/init.d/eudev start'
corrects the issue (i.e. results in all modules being loaded).  The
issue can also be corrected by manually running 'udevadm trigger
--action=add'.

The actual root cause seems to be a race condition in the
/etc/init.d/eudev startup script: The script first starts eudev via
start-stop-daemon, then calls 'udevadm trigger --action=add'.

On the system in question, after modifying the script, adding 'sleep 1s'
after start-stop-daemon, the startup now succeeded all the (two) times I
tried.

Note that the system in question runs from a root partition that resides
on a NBD-devices, so startup times of eudev may be very different from a
normal HDD-based system, thereby amplifying the likelhood of the race.

I have no clear idea how a proper fix for that race would look like.
Clearly 'start-stop-daemon --background' does NOT wait for the daemon to
be operational and looks like the wrong approach.  Maybe running
start-stop-daemon without "--background" and depending on eudev's
"--daemon" option is better?  Does eudev support using
start-stop-daemon's --notify-await command?  Should we run 'udevadm
trigger' in a loop until it succeeds?

-- System Information:
Distributor ID:	Debian
Description:	Devuan GNU/Linux 3 (beowulf)
Release:	3
Codename:	beowulf
Architecture: x86_64

Kernel: Linux 4.19.0-9-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)
LSM: AppArmor: enabled

Versions of packages eudev depends on:
ii  adduser      3.118
ii  debconf      1.5.71
ii  libblkid1    2.33.1-0.1+devuan1~beowulf2
ii  libc6        2.28-10
ii  libeudev1    3.2.7-6
ii  libkmod2     26-1
ii  libselinux1  2.8-1+b1
ii  lsb-base     10.2019051400
ii  procps       2:3.3.15-2+devuan1
ii  util-linux   2.33.1-0.1+devuan1~beowulf2

eudev recommends no packages.

eudev suggests no packages.

-- no debconf information

---

From: fsmithred <[email-masked]>
Date: Mon, 15 Jun 2020 16:00:29 -0400

On 6/15/20 12:15 PM, David Kuehling wrote:
>
> This is my diff:
>
> --- /etc/init.d/eudev	2019-02-09 13:22:24.000000000 +0100
> +++ /tmp/eudev	2020-06-15 17:59:00.384481497 +0200
> @@ -210,6 +210,7 @@
>  	sleep 15
>      fi
>
> +    sleep 1s
>      log_action_begin_msg "Synthesizing the initial hotplug events"
>      if udevadm trigger --action=add; then
>  	log_action_end_msg $?

This fixes a problem I'm having with live-isos booted from optical media.
It boots, but many kernel modules are missing. I used a 10-second sleep to
test, and it works.

For live-isos, the workaround I used was to stop and start eudev in
/etc/rc.local. Here's discussion about that:
https://dev1galaxy.org/viewtopic.php?pid=22137#p22137

fsmithred

---

From: fsmithred <[email-masked]>
Date: Wed, 17 Jun 2020 18:22:27 -0400

I tested with shorter sleep times. One second is enough when booting from dvd.

fsmithred

---

From: Svante Signell <[email-masked]>
Date: Mon, 13 Jul 2020 20:16:42 +0200

Hello,

regarding bug #483, can you try to insert the sleep 1 at the following
place, see the attached patch: eudev.init.diff.

This should apply to eudev-3.2.7-6. Since we plan to include 3.2.9-
7~beowulf1 in beowulf-proposed-updates can you please also test that
patch on eudev.init for that version. It is available from the unstable
release. Don't bother about spacing/tab changes. 

Thanks you very much for your help!

diff --git a/debian/eudev.init b/debian/eudev.init
index c3211a736..103a7dfbc 100644
--- a/debian/eudev.init
+++ b/debian/eudev.init
@@ -169,9 +169,10 @@ case "$1" in
     if start-stop-daemon --start --name $NAME --user root --quiet \
 	--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
 	# prevents udevd to be killed by sendsigs (see #261 & DEBIAN #791944)
-      mkdir -p $OMITDIR
-      ln -sf $PIDFILE $OMITDIR/$NAME
-      log_end_msg $?
+	mkdir -p $OMITDIR
+	ln -sf $PIDFILE $OMITDIR/$NAME
+	sleep 1
+	log_end_msg $?
     else
 	log_warning_msg $?
 	log_warning_msg "Waiting 15 seconds and trying to continue anyway"

---

From: fsmithred <[email-masked]>
Date: Mon, 13 Jul 2020 17:06:59 -0400

I tested the fix with 3.2.9-7 that I pulled from ceres and installed in
beowulf. Made a new live-iso, burned it to DVD and it loads all modules on
boot as it should. Looks good here.

fsmithred

---

From: Trek <[email-masked]>
Date: Wed, 15 Jul 2020 02:20:31 +0200

adding sleep 1 is unreliable (it really depends how fast/slow is the
hardware) as you can see in debian bug #908796

the fix was to add --notify-await to start-stop-daemon without any
additional sleep

if eudev does not implement the notify protocol, then i think it should
be started with --daemon paramenter, but this could bring back the
debian bug #791944, in case eudev does not remove the control socket on
stop or if it does not creates a pidfile

in that case a new init.d script should be written to remove the control
socket just after sendsigs, in order to fix the device-mapper/cryptsetup
shutdown issue

you can see more proposed fixes/workarounds in the bugs linked, but i
give some can help if needed

ciao!

---

From: Trek <[email-masked]>
Date: Tue, 11 Aug 2020 17:24:22 +0200

I had some time to check the sources of eudev and it does not contains
the f59118e commit, that prevented the removal of the control socket on
exit[1], so I send this patch to restore the old rock solid start-up
behavior of udev

the patch is based on the Devuan unstable ed8c5770e0 commit, but please
note that it is untested, as i don't have a Devuan system, nor eudev,
but at least it shows how to do it

basically eudev is launched with --daemon argument and all the pidfile
tracking stuff is removed, because when eudev is killed by sendsigs, it
removes the control socket and dmsetup/cryptsetup will not hang trying
to connect to eudev on shutdown

ciao!

1. https://github.com/systemd/systemd/issues/2477#issuecomment-239077407
diff -urN a/eudev.init b/eudev.init
--- a/eudev.init	2020-08-11 16:55:13.809023204 +0200
+++ b/eudev.init	2020-08-11 17:06:49.737018235 +0200
@@ -11,10 +11,8 @@
 PATH="/lib/udev:/sbin:/bin"
 NAME="udevd"
 DAEMON="/sbin/udevd"
+DAEMON_ARGS="--daemon"
 DESC="hot-plug events dispatcher"
-PIDFILE="/run/udevd.pid"
-CTRLFILE="/run/udev/control"
-OMITDIR="/run/sendsigs.omit.d"
 
 # we need to unmount /dev/pts/ and remount it later over the devtmpfs
 unmount_devpts() {
@@ -166,12 +164,9 @@
     [ -x /sbin/restorecon ] && /sbin/restorecon -R /dev
 
     log_daemon_msg "Starting $DESC" "$NAME"
-    if start-stop-daemon --start --name $NAME --user root --quiet \
-	--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
-	# prevents udevd to be killed by sendsigs (see #261 & DEBIAN #791944)
-      mkdir -p $OMITDIR
-      ln -sf $PIDFILE $OMITDIR/$NAME
-      log_end_msg $?
+    if start-stop-daemon --start --name $NAME --exec $DAEMON --quiet \
+	-- $DAEMON_ARGS; then
+	log_end_msg $?
     else
 	log_warning_msg $?
 	log_warning_msg "Waiting 15 seconds and trying to continue anyway"
@@ -205,37 +200,19 @@
 
     stop)
     log_daemon_msg "Stopping $DESC" "$NAME"
-    if start-stop-daemon --stop --name $NAME --user root --quiet \
-			 --pidfile $PIDFILE --remove-pidfile --oknodo --retry 5; then
-	# prevents cryptsetup/dmsetup hangs ( see #261 & Debian bug #791944 )
-      rm -f $CTRLFILE
-      log_end_msg $?
-    else
-      log_end_msg $?
-    fi
+    start-stop-daemon --stop --name $NAME --quiet --oknodo --retry 5
+    log_end_msg $?
     ;;
 
     restart)
     log_daemon_msg "Stopping $DESC" "$NAME"
-    if start-stop-daemon --stop --name $NAME --user root --quiet \
-      --pidfile $PIDFILE --remove-pidfile --oknodo --retry 5; then
-      # prevents cryptsetup/dmsetup hangs ( see #261 & Debian bug #791944 )
-      rm -f $CTRLFILE
-      log_end_msg $?
-    else
-      log_end_msg $? || true
-    fi
+    start-stop-daemon --stop --name $NAME --quiet --oknodo --retry 5
+    log_end_msg $? || true
 
     log_daemon_msg "Starting $DESC" "$NAME"
-    if start-stop-daemon --start --name $NAME --user root --quiet \
-	--pidfile $PIDFILE --exec $DAEMON --background --make-pidfile; then
-      # prevents udevd to be killed by sendsigs (see #261 & DEBIAN #791944)
-      mkdir -p $OMITDIR
-      ln -sf $PIDFILE $OMITDIR/$NAME
-      log_end_msg $?
-    else
-      log_end_msg $?
-    fi
+    start-stop-daemon --start --name $NAME --exec $DAEMON --quiet \
+      -- $DAEMON_ARGS
+    log_end_msg $?
     ;;
 
     reload|force-reload)

---

From: Trek <[email-masked]>
Date: Wed, 12 Aug 2020 21:01:58 +0200

sorry to write again, but my previous patch could work only if the
interpreter is set to /bin/sh without the -e option

if you don't want to remove the -e option, then this patch should
applied instead

ciao!
+++ b/eudev.init	2020-08-12 20:59:47.963533922 +0200
 
@@ -166,11 +164,8 @@
 
       log_end_msg $?
     else
 	log_warning_msg $?
@@ -205,10 +200,7 @@
 
     stop)
+    if start-stop-daemon --stop --name $NAME --quiet --oknodo --retry 5; then
       log_end_msg $?
     else
       log_end_msg $?
@@ -217,21 +209,15 @@
 
     restart)
+    if start-stop-daemon --stop --name $NAME --quiet --oknodo --retry 5; then
       log_end_msg $?
     else
       log_end_msg $? || true
     fi
 
       log_end_msg $?
     else
       log_end_msg $?

---

From: "Devuan bug Tracking System" <[email-masked]>
Date: Fri, 06 Nov 2020 16:48:05 +0000

This is an automatic notification regarding your bug report
which was filed against the eudev package:

#483: eudev: eudev sometimes fails to generate initial hotplug events

It has been closed by [email-masked].

Their explanation is attached below along with your original report.
If this explanation is unsatisfactory and you have not received a
better one in a separate message then please contact [email-masked] by
replying to this email.

-- 
483: https://bugs.devuan.org/cgi/bugreport.cgi?bug=483
Devuan Bug Tracking System
Contact [email-masked] with problems

Subject: #483: fixed in src:eudev version 3.2.9-8
From: [email-masked]
To: [email-masked]
Date: Fri, 06 Nov 2020 16:38:02 +0000

Version: 3.2.9-8

Source package eudev (3.2.9-8) added to Devuan suite unstable.

This closes bug report 483.

Thanks

DAK managing the Devuan archive

Format: 1.8
Date: Wed, 04 Nov 2020 16:48:24 +0100
Source: eudev
Binary: eudev libeudev1 libeudev-dev eudev-udeb libeudev1-udeb
Architecture: source
Version: 3.2.9-8
Distribution: unstable
Urgency: medium
Maintainer: Devuan Dev Team <[email-masked]>
Changed-By: Svante Signell <[email-masked]>
Description:
 eudev      - /dev/ and hotplug management daemon
 eudev-udeb - /dev/ and hotplug management daemon (udeb)
 libeudev-dev - libeudev development files
 libeudev1  - libeudev shared library
 libeudev1-udeb - libeudev shared library (udeb)
Closes: 483
Changes:
 eudev (3.2.9-8) unstable; urgency=medium
 .
   * debian/eudev.init:
     - Fix startup with slow boot media like DVD and USB.
       Thanks Trek <[email-masked]> for the patch (Closes: #483)
   * debian/control: Fix Vcs-Git and Vcs-Browser entries.
   * debian/watch: Use tags to find latest release.
   * debian/rules:
     - Use autogen.sh directly and remove target autoreconf.
     - Use DEB_BUILD_MAINT_OPTIONS = hardening=+all instead of CFLAGS etc.
   * debian/changelog: Fix spelling error in 3.2.7-2.
   * .gitignore: Add emacs backup files *~.
 .
   * Fixed lintian warning/info entries:
 .
   * debian/patches: man_udev.xml.patch
     Fixes: eudev: spelling-error-in-manpage
 .
   * debian/control:
     Fixes: eudev source: rules-requires-root-missing,
       out-of-date-standards-version (no changes),
       binary-control-field-duplicates-source field "section" in package eudev,
       binary-control-field-duplicates-source field "priority" in package
       libeudev1 libeudev-dev eudev-udeb libeudev1-udeb,
       xc-package-type-in-debian-control
 .
   * debian/rules:
     Fixes: eudev: hardening-no-bindnow
 .
   * debian/tests: New directory. Contents: control, udev
     Fixes: eudev source: testsuite-autopkgtest-missing
 .
   * debian/copyright:
     Fixes: eudev source: insecure-copyright-format-uri,
       wildcard-matches-nothing-in-dep5-copyright src/core/securebits.h,
       unused-file-paragraph-in-dep5-copyright
 .
   * debian/compat: Remove
   * debian/control: Replace build dependency on
     debhelper (>=10) with debhelper-compat (=12)
     Fixes: package-uses-old-debhelper-compat-version,
       extended-description-is-probably-too-short: eudev, libeudev, libeudev-dev
 .
   * debian/eudev.pre{inst,rm},eudev.post{inst,rm}
     Fixes: eudev: maintainer-script-without-set-e: pre{inst,rm}, post{inst,rm}
 .
   * debian/libeudev1.symbols: Don't include a libeudev1.symbols file to avoid:
     symbols-file-contains-current-version-with-debian-revision.
     Instead use an override for the warning: no-symbols-control-file.
 .
   * debian/source/lintian-overrides:
     eudev source: source-contains-data-from-ieee-data-oui-db
     eudev source: upstream-metadata-file-is-missing
     eudev source: debian-watch-does-not-check-gpg-signature
 .
   * debian/eudev.lintian-overrides:
     eudev: appstream-metadata-missing-modalias-provide
     eudev: spelling-error-in-binary
     eudev: improbable-bug-number-in-closes
     eudev: conflicts-with-version udev (<< 1:3.2.9+devuan4)
     # Unknown why
     eudev: hardening-no-fortify-functions lib/udev/mtd_probe
 .
   * debian/libeudev1.lintian-overrides:
     libeudev1: package-name-doesnt-match-sonames
     libeudev1: improbable-bug-number-in-closes
     libeudev1: conflicts-with-version libudev1 (<< 1:3.2.9+devuan4)
     libeudev1: no-symbols-control-file lib/x86_64-linux-gnu/libudev.so.1.6.3
 .
   * debian/libeudev-dev.lintian-overrides:
     libeudev-dev: improbable-bug-number-in-closes
     libeudev-dev: conflicts-with-version libudev-dev (<< 1:3.2.9+devuan4)
 .
   * eudev-udeb.lintian-overrides:
     # None of these overrides work: Why?
     eudev-udeb udeb: hardening-no-fortify-functions lib/udev/mtd_probe
     eudev-udeb udeb: spelling-error-in-binary bin/udevadm ressize resize
     eudev-udeb udeb: spelling-error-in-binary sbin/udevd ressize resize
     eudev-udeb udeb: debug-package-should-be-named-dbg usr/lib/debug/.dwz/
 .
   * libeudev1-udeb.lintian-overrides:
    # This override does not work: Why?
    libeudev1-udeb udeb: package-name-doesnt-match-sonames libudev1
Checksums-Sha1:
 4ca7ee690bb592afbd64d08953132463a3b3ea7d 2345 eudev_3.2.9-8.dsc
 d65744b42d59d5227fb85eb08596dd82efc6daf1 62164 eudev_3.2.9-8.debian.tar.xz
 1714353a61e6ca1b2b5d4126c08448d4101c2e19 3398 eudev_3.2.9-8_source.buildinfo
Checksums-Sha256:
 1b0077b1ce82b96a8ab914babed2bbffd697b6ff9ce5255789ae3260b3505fa9 2345 eudev_3.2.9-8.dsc
 d555e267c2543e9aa4323321bded4a904c11c5c7f392bc247fd7efe2dac69e21 62164 eudev_3.2.9-8.debian.tar.xz
 4e3a6fce281146eda3c84a75766cc42b86b12c43566382e03379dead1979755b 3398 eudev_3.2.9-8_source.buildinfo
Files:
 42a1a80e3c22e3919c9816f52e9892ee 2345 admin optional eudev_3.2.9-8.dsc
 b8a33e81ca64d2087591fc96f9740e65 62164 admin optional eudev_3.2.9-8.debian.tar.xz
 e6c4054498bb40bc94776ae4eb9960dd 3398 admin optional eudev_3.2.9-8_source.buildinfo

Subject: eudev: eudev sometimes fails to generate initial hotplug events
From: David Kuehling <[email-masked]>
To: Devuan Bug Tracking System <[email-masked]>
Date: Mon, 15 Jun 2020 17:08:38 +0200

Package: eudev
Version: 3.2.7-6
Severity: normal
File: /etc/init.d/eudev

Dear Maintainer,

-- System Information:
Distributor ID:	Debian