Subject: bug#406: apt-2.0.0: Runs very, very slow after update to version 2.0

From: Klaus Ethgen <[email-masked]>
Date: Mon, 9 Mar 2020 19:16:48 +0100

Package: apt
Version: 2.0.0

After updating apt to version 2.0.0, there are two times that takes ~10
minutes:
~> apt-get update; apt-get --auto-remove dist-upgrade && apt-get clean && sync
Erstelle alte Liste von Paketen...
Erstelle alte Liste von verfügbaren Aktualisierungen...
Erstelle alte Liste von beobachteten Paketen...
-> 10 Minutes
...
Es wurden 212 MB in 33 s geholt (6.340 kB/s).
Erstelle neue Liste von Paketen...
-> again 10 minutes.

It seems that there is all a apt-cache running that long. It seems that
daptub is running that.

Wishes
   Klaus

Ps. That system is a old sid system witch was converted to ceres. It is
    my fastest system but with many source lists.
-- 
Klaus Ethgen                                       http://www.ethgen.ch/
pub  4096R/4E20AF1C 2011-05-16            Klaus Ethgen <[email-masked]>
Fingerprint: 85D4 CA42 952C 949B 1753  62B3 79D0 B06F 4E20 AF1C

---

From: Mark Hindley <[email-masked]>
Date: Tue, 10 Mar 2020 10:18:48 +0000

Control: tags -1 debian

> After updating apt to version 2.0.0, there are two times that takes ~10

Klaus,

Thanks for this.

However, since apt is not a forked package (devuan uses Debian's package
directly) you would be much better reporting this issue directly to Debian's
BTS.

Thanks.

Mark

---

From: Andreas Messer <[email-masked]>
Date: Tue, 7 Apr 2020 19:48:43 +0200

Hello Klaus,

A devuanized version of the apt package has just been uploaded to
unstable/ceres. (2.0.1+devuan1) Should be available within the 
next hours, can you confirm the problem still exists with it?

Would you mind providing your sources.list(s) to me, such that
I can try to reproduce the problem? Thanks!

cheers,
Andreas
-- 
gnuPG keyid: 8C2BAF51
fingerprint: 28EE 8438 E688 D992 3661 C753 90B3 BAAA 8C2B AF51

---

From: Klaus Ethgen <[email-masked]>
Date: Thu, 16 Apr 2020 21:12:06 +0100

Hello Andreas,

> I can try to reproduce the problem? Thanks!

Sure.
```
/etc/apt (git-[etc:masterU])> grep -vh '^#' sources.list sources.list.d/* G -v '^$'
deb file:///home/ftp/pub/debian ceres unofficial
deb-src file:///home/ftp/pub/debian ceres unofficial
deb file:///home/ftp/pub/debian-security ceres unofficial-secured
deb http://auto.mirror.devuan.org/merged ceres main contrib non-free
deb-src http://ftp.ch.debian.org/debian sid main contrib non-free
deb http://ftp.ch.debian.org/debian experimental main contrib non-free
deb-src http://ftp.ch.debian.org/debian experimental main contrib non-free
deb [trusted=yes] file:///home/local_repo sid main
deb http://www.deb-multimedia.org/ sid main non-free
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20150521T214404Z sid main
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20161010T164513Z sid main
deb https://www.mirbsd.org/~tg/Debs/ sid wtf
```

install daptub and run `/usr/bin/daptup --hook --post`, that will take
10 Minutes.

apt is currently in the following version:
```
~> apt-cache policy apt
apt:
  Installiert:           2.0.1
  Installationskandidat: 2.0.1
  Versionstabelle:
 *** 2.0.1 500
        500 http://auto.mirror.devuan.org/merged ceres/main amd64 Packages
        100 /var/lib/dpkg/status
     1.3.1 400
        400 http://snapshot.debian.org/archive/debian/20161010T164513Z sid/main amd64 Packages
     1.0.9.9 400
        400 http://snapshot.debian.org/archive/debian/20150521T214404Z sid/main amd64 Packages
```

Regards
   Klaus

---

From: Andreas Messer <[email-masked]>
Date: Sun, 3 May 2020 19:20:30 +0200

Hi Klaus,

On Thu, Apr 16, 2020 at 09:12:06PM +0100, Klaus Ethgen wrote:
> [ Removed apt sources.list ]
> 

I had to remove some of your local repos for testing, so my test result
is probably not of much use. Running the command took about 6 seconds
on my machine.

As far as I understand daptup is another tool not part of apt package. 
So maybe it is at the moment not apt to blame. We should figure out were 
the delay occurs. Did you check that you local file repos are available 
(e.g. mounted properly before running the tool?)

Could you please install libdevel-dprof-perl and then run daptup as
follows within /tmp:

    $ sudo perl -d:DProf /usr/bin/daptup --hook --post

Then please invoke:

    $ dprofpp

And mail the output.

Best regards
Andreas

---

From: Klaus Ethgen <[email-masked]>
Date: Sun, 3 May 2020 20:37:19 +0100

Hi Andreas,

> is probably not of much use. Running the command took about 6 seconds

Strange..

> (e.g. mounted properly before running the tool?)

Yes, they are. Sure they are. And the speed change came exactly with the
new apt release.

>     $ sudo perl -d:DProf /usr/bin/daptup --hook --post

No sudo here and I did it in root $HOME:
```
~> dprofpp
Total Elapsed Time = 669.6876 Seconds
  User+System Time = 1.747609 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 44.6   0.780  0.780      1   0.7800 0.7800  Daptup::Backend::AptOrCupt::get_updates
 35.4   0.620  1.689      1   0.6196 1.6892  main::do_post
 15.4   0.270  0.270  80346   0.0000 0.0000  Term::ANSIColor::color
 1.72   0.030  0.069     11   0.0027 0.0062  main::BEGIN
 0.57   0.010  0.010      3   0.0033 0.0033  main::diff_cmd
 0.57   0.010  0.010      1   0.0100 0.0100  main::find_old_packages
 0.57   0.010  0.010      4   0.0025 0.0025  Daptup::Backend::Aptitude::BEGIN
 0.57   0.010  0.010      7   0.0014 0.0014  IO::Handle::BEGIN
 0.57   0.010  0.010      8   0.0012 0.0012  File::Spec::Unix::BEGIN
 0.57   0.010  0.010      7   0.0014 0.0014  Log::Agent::BEGIN
 0.57   0.010  0.029     21   0.0005 0.0014  File::Temp::BEGIN
 0.00   0.000  0.000      1   0.0000 0.0000  Exporter::Heavy::heavy_export_ok_tags
 0.00   0.000  0.000      1   0.0000 0.0000  Getopt::Long::GetOptionsFromArray
 0.00        - -0.000      1         -       -  main::get_watch_regex
 0.00        - -0.000      1         -       -  IO::Handle::__ANON__
```

Gruß
   Klaus

---

From: Klaus Ethgen <[email-masked]>
Date: Sun, 3 May 2020 21:18:28 +0100

Maybe the daptup.conf would be of some help.

Gruß
   Klaus

---

From: Klaus Ethgen <[email-masked]>
Date: Sun, 3 May 2020 21:58:38 +0100

Another think I seen..

`apt search '.*'` takes hours (as does all apt search or list calls).

But when I do the usual way `apt-cache search '.*'` it gives me the list
instant. (But it runs ages.)

The man page of apt says that this two should be identical and it
definitively was in the old version but now they are complete different.

By the way, even apt-cache needs some time to finish. I let it count the
lines (packages) and I get a count of 80318 lines which takes nearly 11
minutes time. My /var/lib/dpkg/available file only counts 90655
packages.

Gruß
   Klaus

---

From: Mark Hindley <[email-masked]>
Date: Wed, 11 Jan 2023 13:38:29 +0000

Control: tags -1 moreinfo
Control: retitle -1 daptup makes apt 2.0 very slow

Klaus,

Is this still an issue or can it be closed? If it is an daptup bug, it should
probably be resolved in Debian.

Thanks.

Mark

---

From: Klaus Ethgen <[email-masked]>
Date: Wed, 11 Jan 2023 14:47:29 +0100

Hi Mark,

> probably be resolved in Debian.

It is still an issue.

And a change of apt was the reason for this bug, no change in daptup.

The reason is `apt-cache search .*` which starts fast but gets slower
and slower...

And this was introduced with apt 2.0.0.

Regards
   Klaus