Upgrading AWS Ubuntu Server

MikeJones

I am running a Ubuntu 16.04.7 LTS EC2 Instance on AWS and I am having issues upgrading it.

I have tried:

sudo apt update sudo apt upgrade sudo reboot

And there are no issues until I try:

sudo do-release-upgrade

And I get this error:

Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/bionic", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeMain.py", line 238, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 2072, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade
    if not self.askDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade
    changes = self.calcDistUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade
    if not self.cache.installTasks(self.tasks):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks
    pkg.mark_install()
  File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install
    fixer.resolve(True)
SystemError: E:Unable to correct problems, you have held broken packages.
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/problem_report.py", line 497, in add_to_existing
    self.write(f)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 450, in write
    block = f.read(1048576)
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/bionic", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeMain.py", line 238, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 2072, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade
    if not self.askDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade
    changes = self.calcDistUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade
    if not self.cache.installTasks(self.tasks):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks
    pkg.mark_install()
  File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install
    fixer.resolve(True)
SystemError: E:Unable to correct problems, you have held broken packages.
=== Command terminated with exit status 1 (Thu Oct 22 03:03:11 2020) ===

Abhi

It looks like a Dependency issue. If you have proper backups, I would suggest

sudo apt autoremove

Have you seen this ?

https://askubuntu.com/questions/1214292/upgrade-path-from-16-04-to-20-04-lts-without-gui-change

Most probably you will have to do it the old fashioned way. ie, upgrade to 18.04 and then to 20.04

https://askubuntu.com/questions/1263794/how-to-upgrade-from-ubuntu-16-04-lts-to-20-04-in-virtual-machine

Abhi

Official guides from Ubuntu

https://askubuntu.com/questions/1263794/how-to-upgrade-from-ubuntu-16-04-lts-to-20-04-in-virtual-machine

16.04 to 18.04 by DO, hopefully it will work on AWS too

https://www.digitalocean.com/community/tutorials/how-to-upgrade-to-ubuntu-18-04

MikeJones

Abhi awesome thanks I will try the digital ocean guide too. They have great tutorials.

I will let you know how it goes!

MikeJones

So I followed the Digital Ocean steps and after running this:

sudo do-release-upgrade

It failed again with:

`Broken landscape-common:amd64 Depends on python3-twisted [ amd64 ] < none -> 17.9.0-2ubuntu0.1 > ( python )
Considering python3-twisted:amd64 1 as a solution to landscape-common:amd64 9999
Reinst Failed because of python3-zope.interface:amd64
MarkKeep python3-twisted [ amd64 ] < none -> 17.9.0-2ubuntu0.1 > ( python ) FU=0
Considering python3-twisted:amd64 1 as a solution to landscape-common:amd64 9999
Done

Traceback (most recent call last):
File "/tmp/ubuntu-release-upgrader-a7x268x1/bionic", line 8, in <module>
sys.exit(main())
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeMain.py", line 238, in main
if app.run():
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 2072, in run
return self.fullUpgrade()
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade
if not self.askDistUpgrade():
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade
changes = self.calcDistUpgrade()
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade
if not self.cache.installTasks(self.tasks):
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks
pkg.mark_install()
File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install
fixer.resolve(True)
SystemError: E:Unable to correct problems, you have held broken packages.
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/problem_report.py", line 497, in add_to_existing
self.write(f)
File "/usr/lib/python3/dist-packages/problem_report.py", line 450, in write
block = f.read(1048576)
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Original exception was:
Traceback (most recent call last):
File "/tmp/ubuntu-release-upgrader-a7x268x1/bionic", line 8, in <module>
sys.exit(main())
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeMain.py", line 238, in main
if app.run():
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 2072, in run
return self.fullUpgrade()
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade
if not self.askDistUpgrade():
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade
changes = self.calcDistUpgrade()
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade
if not self.cache.installTasks(self.tasks):
File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks
pkg.mark_install()
File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install
fixer.resolve(True)
SystemError: E:Unable to correct problems, you have held broken packages.
=== Command detached from window (Wed Oct 28 00:45:53 2020) ===
=== Command terminated with exit status 1 (Wed Oct 28 00:46:03 2020) ===`

Abhi

Were you able to move smoothly from 16.04 to 18.04 ? I think the following link and answer by user535733 might be helpful. The person replied is a part of the Ubuntu team ( I think so )

https://askubuntu.com/questions/1263794/how-to-upgrade-from-ubuntu-16-04-lts-to-20-04-in-virtual-machine

MikeJones

Abhi nope, still stuck.

I asked for help on the official Ubuntu Forum and someone asked me to run this command:

apt-cache policy landscape-common:amd64
Which I did and gave them the output, but not sure what next step is yet.

Abhi

From what I can see you will have to upgrade to 18.04 first and then upgrade to 20.04. I assume that is what you have done so far. If no, kindly do update.

It would be easier if you can fresh install 20.04 and avoid all these troubles. That way you have an error-free install.

MikeJones

Abhi I have not upgraded to 18.04 yet. I think thats where my issue lies.

Abhi It would be easier if you can fresh install 20.04 and avoid all these troubles. That way you have an error-free install.

If I do that I have to reconfigure my entire nginx settings and somehow import all my flarum data into the new server.... I think that seems just as much of a hassle, no?

Abhi

I understand your trouble. From the error data you posted, the issues seems to be broken packages / unmet dependencies. The issue is at the point from where you are trying to updating. ie, something is already broken on your server and from there you are trying an upgrade. Is there any chance you have a snapshot a bit older than the point from which you are trying ? Yes, it would loose you a few forum posts. But maybe a day or two old, will is cost you too much. I would suggest give it a try. I really hoped that one of those guides might have pulled you out.

PS : Did you try sudo apt autoremove ( backup please ) ? What was the result ?

Abhi

There might be hope

https://www.rosehosting.com/blog/how-to-fix-broken-packages-on-ubuntu-16-04-and-debian-9/

http://www.iasptk.com/ubuntu-fix-broken-package-best-solution/

I have not tried these, if interested, take a look

MikeJones

So I think my issue with upgrading might be this:

Reinst Failed because of python3-zope.interface:amd64 MarkKeep python3-twisted [ amd64 ]

Any more thoughts on this?

phenomlab

MikeJones Mike, this seems like the artefact of a removed PPA relating to Certbot. Have you removed any repositories recently before attempting the upgrade ?

Abhi

Could be because of partially upgrade packages.

I haven't tried :

sudo apt-get install --fix-broken

remove all Python and install it afresh

sudo apt-get purge python* && sudo apt-get autoclean && sudo apt-get install python*

MikeJones

phenomlab I dont believe so but is there anyway to check/fix?

phenomlab

MikeJones What is the result of sudo apt install certbot python3-certbot-nginx

MikeJones

phenomlab

I got this:

certbot is already the newest version (0.31.0-1+ubuntu16.04.1+certbot+1).
python3-certbot-nginx is already the newest version (0.31.0-1+ubuntu16.04.1+cert               bot+1).
python3-certbot-nginx set to manually installed.
The following packages were automatically installed and are no longer required:
  javascript-common jsonlint libargon2-0 libjs-excanvas libvpx3 libzip5
  linux-aws-headers-4.4.0-1054 linux-aws-headers-4.4.0-1055
  linux-aws-headers-4.4.0-1057 linux-aws-headers-4.4.0-1060
  linux-aws-headers-4.4.0-1061 linux-aws-headers-4.4.0-1062
  linux-aws-headers-4.4.0-1065 linux-aws-headers-4.4.0-1066
  linux-aws-headers-4.4.0-1069 linux-aws-headers-4.4.0-1070
  linux-aws-headers-4.4.0-1072 linux-aws-headers-4.4.0-1074
  linux-aws-headers-4.4.0-1075 linux-aws-headers-4.4.0-1077
  linux-aws-headers-4.4.0-1079 linux-aws-headers-4.4.0-1083
  linux-aws-headers-4.4.0-1084 linux-aws-headers-4.4.0-1085
  linux-aws-headers-4.4.0-1087 linux-aws-headers-4.4.0-1088
  linux-aws-headers-4.4.0-1090 linux-aws-headers-4.4.0-1092
  linux-aws-headers-4.4.0-1094 mercurial mercurial-common php-cli-prompt
  php-composer-semver php-composer-spdx-licenses php-json-schema
  php-symfony-console php-symfony-filesystem php-symfony-finder
  php-symfony-process
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.
ubuntu@ip-172-31-93-25:~$

MikeJones

So good news, I figured it out! I had to uninstall python3 and python-certbot-nginx and then I was able to upgrade!

Unfortunately now that that problem is solved, I am onto the next one! NGINX doesnt seem to be running: https://askubuntu.com/questions/1294969/upgraded-from-ubuntu-version-16-to-20-and-nginx-is-not-working

phenomlab

MikeJones So good news, I figured it out! I had to uninstall python3 and python-certbot-nginx and then I was able to upgrade!

That was going to be my next suggestion 🙂 You beat me to it - that's what happens when you "snooze" (I wish) I suppose 🙂

In relation to NGINX not starting, what's the output of journalctl -xe and anything in journalctl | tail ?

This is a bit old, but may still be useful
https://www.digitalocean.com/community/questions/can-t-start-nginx-job-for-nginx-service-failed

MikeJones

phenomlab

For journalctl -xe:

`-- A start job for unit session-70.scope has finished successfully.

-- The job identifier is 3818.
Nov 24 23:50:02 ip-172-31-66-31 systemd[7236]: Reached target Main User Target.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd

-- Support: http://www.ubuntu.com/support

-- A start job for unit UNIT has finished successfully.

-- The job identifier is 1.
Nov 24 23:50:02 ip-172-31-66-31 systemd[7236]: Startup finished in 52ms.
-- Subject: User manager start-up is now complete
-- Defined-By: systemd

-- Support: http://www.ubuntu.com/support

-- The user manager instance for user 1000 has been started. All services queued
-- for starting have been started. Note that other services might still be starting

-- up or be started at any later time.

-- Startup of the manager took 52538 microseconds.
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 16: Deprecated option UsePrivilegeSeparation
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 19: Deprecated option KeyRegenerationInterval
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 20: Deprecated option ServerKeyBits
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 31: Deprecated option RSAAuthentication
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 38: Deprecated option RhostsRSAAuthentication
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 31: Deprecated option RSAAuthenti>
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 38: Deprecated option RhostsRSAAu>
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Invalid user jenkins from 104.248.243.134 port 41200
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Received disconnect from 104.248.243.134 port 41200:11:>
Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Disconnected from invalid user jenkins 104.248.243.134 >
lines 2040-2070/2070 (END)
`

and for journalctl | tail:

ubuntu@ip-172-31-66-31:~$ journalctl | tail Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 16: Deprecated option UsePrivilegeSeparation Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 19: Deprecated option KeyRegenerationInterval Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 20: Deprecated option ServerKeyBits Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 31: Deprecated option RSAAuthentication Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 38: Deprecated option RhostsRSAAuthentication Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 31: Deprecated option RSAAuthentication Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 38: Deprecated option RhostsRSAAuthentication Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Invalid user jenkins from 104.248.243.134 port 41200 Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Received disconnect from 104.248.243.134 port 41200:11: Normal Shutdown, Thank you for playing [preauth] Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Disconnected from invalid user jenkins 104.248.243.134 port 41200 [preauth] ubuntu@ip-172-31-66-31:~$

phenomlab

MikeJones not the greatest of help...
What's the output of systemctl status nginx.service ?