Ubuntu Server distribution update via SSH over the internet

Standard

I have been using Ubuntu Server for development for over 10 years now. Although I am always using LTS distribution, which is supported for 5 years, there will be a time when the server needs an update only because server software needs some refresh (especially LAMP stack). Of course there is an another way – just compile new version, but in my experience this will screw up your installation (believe me – I tried).

Making distribution update on a desktop is relatively easy process – just use “Upgrade Manager” to start the upgrade procedure, sit down, relax and wait until upgrade is finished. Upgrade via SSH is much more complicated – there is no GUI to help you and SSH connection may drop any time leaving you with a server, which is half-working.

Here is my current development server configuration:

Ubuntu 14.04 LTS

–  Apache (2.2)
–  PHP (5.3)
–  MySQL (5.5)
–  PostgreSQL (9.1)
–  Ruby on Rails (using Passenger). Compiled from source 🙂
–  Tomcat (7)
–  VPN (l2tp-ipsec)
–  VNC (For creating XFCE4 remote desktop connection)

* Note. My production server has almost similar configuration, except I am using Debian for better stability.

Before starting, you should update your current system:

And you should install update manager core:

Before continuing you should check that your firewall does not block port 1022 – upgrade process will start an additional SSH process if something goes wrong.

Starting distribution update is easy:

Installation will ask many questions, read instructions carefully and press Y (or N) depending on your choice.

After upgrade you can check if distribution is successfully updated:

Continuing installation after SSH is disconnected

This happened to me several times. I was not quick enough to answer the questions and connection was dropped. When I logged in again, a new screen was started and it was impossible to continue the upgrade process.

I found an easy solution for this problem. Basically it means to reattach your screen to the upgrade process.

After that I was able to complete the upgrade process.

Problems after installation

After installation my server had many problems. It was not usable for development and major services needed some tweaks.

Especially:

– Apache virtualhosts did not work.
– Setting an additional declaration in .htaccess gave error
– PostgreSQL had two processes (9.1 and 9.3)
– Ruby was gone (yes, really gone!)
– Tomcat was not working anymore.
– It was not possible to log in via VPN anymore.
– It was not possible to log in via VNC anymore.
– It was not possible to restart server anymore. The server just hanged.

Last problem was the most complicated, because I had to go to the server room and restart the server manually. No SSH services were running anymore.

Fixing the problems.

This was not the first time for me to clean up mess after Ubuntu upgrade. I started to check system logs, read forums and soon I had working system again.

  1. Apache virtualhosts did not work.

Apache 2.4 has several security related changes, one of these changes requires that all virtual host files must have .conf extension.

To fix:

  1. Setting an additional declaration in .htaccess gave error

Apache 2.4 has the following change:

Old declaration:

New declaration:

  1. PostgreSQL had two processes

I decided to remove old version, everything worked fine after that. PS. You should still use pg_dump to backup your databases before removing old version.

4. Ruby was gone.

I decided to make a new ruby install, this time I decided to use Ubuntu packages and not to compile from source.

If you are using Rails with Apache, it is smart to use Passenger module.

Next step was to configure passenger module:

  1. Tomcat was not working anymore.

After looking long time for a good solution, I decided to purge old configuration and install Tomcat again.

After that Apache Tomcat worked again.

  1. It was not possible to log in via VPN anymore.

It seems that something has changed inside Ubuntu, which made my current ipsec configuration obsolete. I decided to search solution from the internet and found perfect configuration here

After changing configuration, ipsec started to work again and l2tp tunnel was created successfully.

  1. It was not possible to log in via VNC anymore.

Ubuntu upgrade overwrote /etc/services, which defined VNC port. I had to add this declaration into this file again:

7. It was not possible to restart server anymore. The server just hanged.

After I finished upgrade, I decided to restart my server. But the server was not coming online again and I had to go to the server room for a check.

When turning on the screen, I saw the following message:

*Deactivating swap
mount: / is busy
*Will now halt

It was obvious that server hanged during the restart/shutdown procedure.

This was most complicated problem to solve. While searching solution from the Internet, I found the following thread. This gave me hint – maybe problem might be in Broadcom wlan drivers. I decided to remove these drivers.

You may also need to blacklist BCM drivers. Edit /etc/modprobe.d/blacklist.conf file and add the following lines:

*Actually upgrading via SSH over the internet is highly discouraged and you should not use SSH when upgrading production servers over the internet.