Upgrading AllStarLink on my Cloud Server

Following upgrade recipes made this process much less daunting than I expected

I’ve been having a problem on my cloud node, hosted by Vultr.com, in which disk operations increase over time, eventually crashing the node. I kludged a solution a few months ago by having the server restart Asterisk once each day at 3:00 am.

Please consider using my referral link if you choose to open a Vultr account: https://www.vultr.com/?ref=9261098

In this graph, you can see how that restart significantly reduces disk activity at the 3:00 am restart:

Daily restart at 3:00 am fixed excessive disk writes

As I talked about this in the AllStar Link Network group on Facebook, David Gleason (of AllScan fame) suggested that I look at /var/log/syslog to see if there were errors being reported…and there were. I found a segfault being reported every 30 seconds or so, related to app_rpt.so.

More information about this particular segfault is available at https://community.allstarlink.org/t/tracing-a-segfault/19519/3. I followed the suggestion in that thread and commented out the line that started with “statpost_url” in the /etc/asterisk/rpt.conf file, then restarted Asterisk:

sudo astres.sh

After several minutes, I logged into the terminal and ran this command to see the latest syslog entries (tac is the cat command backwards

sudo tac /var/log/syslog | grep segfault

(The cat command reports the oldest information first. The tac command [cat spelled backwards!] reports in the opposite order, i.e., newest information first.)

I did that a couple of times over several minutes and confirmed that no new segfault entries were being captured.

The disk operations graph shows the dramatic effect of commenting out that one line in the rpt.conf file (the big spike was the result of running cat and tac commands multiple times):

Commenting out a command fixed the disk write problem

So that change was a success, but David also pointed out that my fix would also block the reporting of my node stats to AllStarLink.org. That’s not good. He encouraged me to upgrade ASL on my node. Upgrading seemed daunting so while I had previously thought about it, I had not found the gumption to do it.

However, David had already foreseen that some of us might find this too overwhelming to attempt, so he included helpful information on how to accomplish this upgrade at https://allscan.info/docs/diy-node.php#updating. This was the guidance I followed to upgrade my cloud node.

First, I backed up the existing installation

Why backup before an upgrade? Because it’s a best practice. You never know what might happen during an upgrade process.

I installed rdiff-backup on the existing server:

sudo apt install rdiff-backup

Then I created a Backups directory and a few subdirectories to hold specific information:

sudo mkdir /var/www/html/Backups/
sudo mkdir /var/www/html/Backups/asterisk
sudo mkdir /var/www/html/Backups/html

Next came running rdiff-backup to save configuration files and allmon2, supermon, and AllScan files:

sudo rdiff-backup /etc/asterisk/ /home/repeater/Backups/asterisk/
sudo rdiff-backup /var/www/html/ /home/repeater/Backups/html/

Once that was done, I used Filezilla to connect to the server over an SSH tunnel and pulled down a copy of the Backups directory to my local machine. That was my plan but it didn’t work because I had not allowed SSH connections through my firewall. I added a new firewall rule:

sudo ufw allow ssh

After this, I was able to connect. The directory downloaded quickly and this part was done.

Then I ran a snapshot of the entire server

Vultr has an easy way to back up the server by taking snapshots. I keep a couple of snapshots of known good configurations on hand, just in case. Before taking any steps to upgrade ASL, I ran a snapshot of the server.

Each snapshot is about 3.5 gigabytes in size, and each one costs me 20 cents per month to keep on hand. That is really cheap insurance.

Here’s my March 2023 invoice for this node:

March invoice for TomHub.us

Time to upgrade with AllScan.info directions

David Gleason summarizes the steps needed to upgrade ASL at https://allscan.info/docs/diy-node.php#updating. The short version of his notes is:

1. Log into the node by SSH as root (run "sudo su" if not already root user).

2. Run all commands listed on the ASL GitHub Prerequisites section.

3. Run the commands listed on the ASL GitHub Compiling "Manually" section.

Vultr has a neat facility in their terminal window where you can copy the commands from the GitHub instructions and paste them right into the terminal. It works very smoothly and made this process much, much easier (and less error-prone) than trying to type all the commands into the terminal window.

Of course, ASL 2.0.0 Beta is built on Debian 10. That means copying from your Windows(?) machine into the Debian shell is easy with the Ctrl Shift V keyboard shortcut.

Confusion

I encountered a couple of steps that seemed confusing. I found the wording in the Prerequisites section at https://github.com/AllStarLink/ASL-Asterisk#prerequisites to be unclear. Specifically, the instructions start with installing the AllStarLink repository, then there is a section about an alternative repository. The instructions don’t make it clear that you should do both: install the ASL repo, and install the alternate repo. I rolled the dice and completed those steps, and that was the right thing to do.

Following that, you install dependencies as shown in the instructions.

And then comes compiling where I once again had a moment of confusion. After the first “git clone” step, the instructions say deb files “will appear in the repository root folder and after compiling and packaging.” That sentence doesn’t end with a period in the directions so it was unclear whether it was supposed to be connected to the one word on the next line (“Manually”), or if the period was left out by accident, or if “Manually” meant instead of doing the first steps, you could instead do the second steps.

I pondered it for a bit and chose to do all the steps, and again, that was the right thing to do. I ran the first “git clone” and debuild commands, then ran the second “git clone” command (below the word “Manually”) and the following “make” and “make install” commands.

That worked. I highlight my own confusion here in case you encounter the same questions as you work through the recipe of steps.

Output of a successful build

If all goes well, you’ll see this result after the “make” command:

Output of the “make” step

Then run “make install” and the result should be:

Output of the “make install” step

Confirming the upgrade

I’m done…aren’t I? Did it actually work? I connected to my node through my Hamshack Hotline line and the node answered, so that was a positive sign.

I exited out of the terminal and came back in, logged in, and then ran asl-menu:

sudo asl-menu

Choice #5 in the ASL Main Menu is the Asterisk CLI (command line interface). Opening the Asterisk CLI should tell me what version of ASL is running.

However, option #5 spawned an error. I realized I should probably restart the server so I did that. After the restart, the Asterisk CLI came up cleanly and reported the Asterisk version:

To confirm that information, I hit “Ctrl C” to exit out of the CLI, then asked Asterisk to report the installed version:

sudo asterisk -V

The result confirmed what the CLI reported:

So is ASL working? Did I lose my configuration files? All of my configuration files are still there so it seems that my worry about losing them was unfounded.

I connected again through my Hamshack Hotline line and again the node answered.

AllScan is running (but I note that I’m at version 0.67 which is one release behind the current release, so I’ll address that in a bit):

Supermon is up:

Testing the disk write condition

Evaluating this upgrade would not be complete if I didn’t check whether the upgrade fixed the disk write problem. I edited the rpt.conf file, uncommenting the offending line, then restarted Asterisk.

And waited. And waited some more. It’s hard to see what’s happening because the upgrade process really loaded the disk operations, causing a big spike in the graph:

Then I ran the tac and grep commands described earlier and found no segfaults in the log. I’ll keep an eye on it but it appears to be resolved.

Conclusion

That went smoothly. I didn’t need my backups but I feel better for having saved the key configuration files.

I need to remember how I implemented the daily restart of Asterisk at 3:00 am because it appears I don’t need it. However, it’s probably not bad to restart the system daily so perhaps I’ll just leave this alone unless I spot problems.

Many thanks to David for the encouragement and for providing such helpful guidance. I let out a big sigh of relief when all the steps were completed successfully!