Information & announcements

[cobito]

The front page will stop receiving content for a few weeks. Anyway, I don't know about you, but it seems to me that the marketing of hardware manufacturers is a bit dead lately, isn't it? I think that the groups that are suffering the most from the massive layoffs of technology companies are precisely these ones. Although perhaps it is just a perception of mine.

[cobito]

During the month of September, important software update tasks will be carried out on the server:

· The operating system will be updated from Debian 11 to 12, with the large number of changes throughout the package that this will entail. This could break compatibility with the different pages, mainly with the homemade ones like the test bank and the museum, so we will have to be vigilant.

· Nodebb (forum) will be updated and it will be the first time we change the major version. Here we hope that nothing important has changed in the API and if it has, we will have to act on the front page and the test bank.

· Peertube (video) will be updated to the version that is available at that time. The current latest version brings an interesting change for future server topology thinking. In this case, a plugin will be installed that allows chat in live broadcasts (at the request of a user of the instance).

· Mastodon (social) will be updated which, like the forum, changes the major version. Here we will have to ensure communication with the front page.

Regarding hardware, we are studying changing the SSD drive where the system is installed for a mechanical disk. This will depend on the health status of the SSD at this time (the "231 SSD_Life_Left" value of SMART says 91, but I want a second opinion from CrystalDiskInfo). As for recurrent writes, we have hammered with logs (small size but constant writes) and as for voluminous writes, we have hammered with Mastodon, which was installed there thinking it wouldn't take up much but in this action it will be moved to the NAS for consuming much more than expected.

Apart from the disk, no other hardware changes are planned: we continue with the same machine and the same NAS, both already quite fair in their capacities but which should be able to hold out for at least a few more months.

Given that there are more changes than usual, Nodebb along with the Redis provided by Debian (Redis is one of the things that scares me the most about this update because of the serious problems we have had in the past), we will test a virtual machine beforehand. In addition, we will take the opportunity to update the guide for setting up a server on Debian and move it to the forum (currently it is still on dplinux.net).

As usual, all of this will be done on a different disk from the current one (whether it is decided to continue with an SSD or change to a mechanical disk) to have the current state of things insured in case of disaster.

These tasks could take Hardlimit offline between a few hours and a few days and the changes should be transparent for the user, except for the new features that the new versions of the different components bring.

[Sylver]

@cobito You're going to give yourself a terrible beating to update everything, beloved mayor. You deserve all the good in the world.

If there's anything we can contribute to, we're here.

Thank you so much for everything!

[cobito]

@sylver Thanks. For now, help won't be necessary. We'll see if the Peertube instance continues to grow, perhaps I'll need a hand in the future.

[cobito]

Coincidences of life, yesterday an important security patch for Mastodon was released and is already applied (we were on 3.5.3 and went to 3.5.10). The identifiers are CVE-2023-36459 and CVE-2023-36460. The latter seems to be especially dangerous as it is a problem in the code that processes the images and videos of the toots that would allow DoS attacks and the remote execution of arbitrary code.

[cobito]

The update is turning out to be a more complicated operation than initially anticipated. Mastodon and Blogs are still to be put into operation, in addition to a multitude of small details.

As you can see, the forum looks strange. Over the next few days I will be fixing the CSS so that it looks like it did before. When everything is finished, I will write with the details.

[cobito]

The initial idea was clear: install Debian, install the packages, move certain data from one site to another, install the different components and off you go. In a couple of hours it should be ready. It's not the first time I've done this and there shouldn't be too many difficulties. The reality: today I'm 20 years older than last week.

Some things have worked out right away:

The house pages (front page, test bench and museum) don't seem to have major incompatibilities with the new PHP 8.2 (only a small change was necessary on the front page): here a more exhaustive check will still have to be done. In addition, the people at Nodebb haven't made major changes to the forum API so all the components have continued to communicate with the registry system without the need for changes.

Regarding the forum along with its database, it has been a piece of cake. Here the only issue is that there have indeed been a large number of changes in the layout of the pages, so the CSS has become partially obsolete. I've been tweaking it, although there are still details to be polished. There are a lot of new aesthetic features and, from my point of view, the mobile version has gained a lot of ground. There is still a small permissions problem that the venerable team of moderators is having which I hope is not a big deal.

On the other hand, it's the first time in many years that we've used the official Debian version of Redis, which, from a maintenance point of view, is a big step forward. Huge step forward. I'm very happy with this.

And that's the nice part. The rest has been a descent into the underworld:

· Many blogs are not compatible with PHP 8.2, so I'll have to do a manual update for each one of them. It shouldn't be a big deal, but it has to be done. I'm going to take my time with this.

· One of the changes in the organization of the directories has been to move all the "small" content (thumbnails, plugins...) from the Peertube instance to an SSD drive to offload transactions to the NAS (it was already very full). I had a brand new SSD drive almost brand new here but, oh, what a coincidence in life!, it's incompatible with the disk controller. You plug it in and it works. After a few hours, Linux starts reporting I/O errors with the ATA. You reboot and the BIOS stops detecting it. If you turn off the computer and turn it on, it works again. I've tried several cables and several SATA connectors but no, it's the darn disk (which, by the way, works perfectly on another computer). I've had to resort to a not-so-new SSD drive (although it's in good health).

· In Peertube supposedly you can configure the directories of each type of element individually: temporary, videos, thumbnails, etc. I've configured it to distribute it between the NAS and the SSD and apparently everything worked except that from the Internet, you couldn't access the content of the NAS. Why? Because the CMS uses a bridge in Nginx to access the data directly by defining something called 'root'. The problem is that there can only be one 'root'. So, which one do you choose: the NAS or the SSD? Well, after a few twists and turns, I chose the SSD by creating symbolic links to the folders on the NAS. It's not a clean option, but it's the only one I've found.

Well, so far 9 hours have passed, there's still Mastodon to get up and running (hahaha! someone sacrifice me, please) and I'm starting to question some decisions. I'll leave it for the next day because every first Sunday of each month, the NAS does a matrix check and that makes everything go very slow for a few hours.

· ¡Mastodon! This is already a major issue. So much so that I'm going to divide it into paragraphs:

Mastodon was hosted on the system disk. But the content takes up quite a bit and the only way it can be sustainable in the long term is to move it to the NAS. The operation is simple: connect the old disk and copy the data to the NAS. The data takes up about 240GB (not excessive), is spread across about 800,000 files (which is normal because they are images, their thumbnails and short videos) and all this is spread across (attention!) two and a half million folders. Any normal person would wonder why millions of folders are necessary to store hundreds of thousands of files?

This way of distributing the data has made me realize how brutally slow it is to create two and a half million folders on mechanical disks. It's incredibly slow; I would never have imagined it. In principle, copying 240GB to the NAS should have taken around an hour. Doing this copy has taken 14 hours. 14 more hours than the instance has been down.

Once the files are copied, it's time to install the CMS. The official instructions are wrong. They say that Ruby 3.0.6 is needed. I install that version and at a certain point, it gives an error. I try the installation several more times in case I skipped something until I start analyzing the logs thoroughly (yes, this has been my fault for not having done it before). It turns out that the latest version of Mastodon uses Ruby 3.2.2.

Everything starts to go well until it starts compiling. I think that perhaps I've made another mistake and I start the installation another couple of times with the same result. At this point, I've installed Mastodon 8 times. ¡Eight! I'm starting to think that the lack of certain cognitive abilities might be preventing me from carrying out this task. Then I come across a guy on Reddit who says he's already done 12 installations. ¡Eh, look at this geek, TWELVE times, HA! -I shout at the screen, feeling my pride partially recover.

After a few minutes, I recover my fragile mental equilibrium and I start to look for the solution that some guy has given to this poor unfortunate. It turns out that I'm using a version of NodeJS that's too recent. The version I'm using is the 18 LTS that came out in April 2022. How is it possible that the latest version of Mastodon, which came out two weeks ago, is not compatible with a NodeJS that has been around for a year and a half? Come on, someone answer, any philosophers in the room?

Leaving aside the humanist issues, I look at what version Mastodon needs to work: version 16 no less. A version that has been without support since September 11. Yes, gentlemen: the majority of Mastodon instances that are up and running are using a critical component without support. What do you think? Where are those philosophers, please?

In the end, I find a hack that allows Mastodon to be compiled with NodeJS 18, it works and that's it: I restore the database, I migrate it to the new version, I update the feeds and it's up and running. NOTE: This last paragraph has been adapted for a children's audience. During the procedure, a continuous crime of blasphemy has been committed, but this time due to my own ineptitude.

And already ascending from the depths of madness, I've realized something interesting and it is that Docker has become the de facto standard. In fact, it's starting to be complicated to find documentation to do certain things "the old way". And surely, some of the difficulties have arisen from that paradigm shift. So for next time, I'll have to train myself on this because it's already the present and it also seems an incredibly convenient way to install things. I hope it's with Debian 14.

Anyway, what a long winded story. But how comfortable am I with it?

P.D.: As usual, if you see something strange, don't hesitate to comment.

[krampak]

Mother of God, what an odyssey boss!

[Fassou]

This is Atlas carrying the celestial vault, or @cobito lifting the server, now I can't remember

A great one!

[Sylver]

What a shame not to be able to return even a tiny part of what you do for us @cobito... What a great job

One way or another, it seems that Debian is still surrounded by dangers as that computer science professor told me years ago... But I bet he didn't have to face as many additions as in this case.

Anyway, a million thanks as always, and I hope to be able to contribute as much as I can to the cause.

Best regards!

[cobito]

@krampak said in Information & announcements:

My God, what an odyssey boss!

Yes, this time it has been tough. Many services and each one of their own father and mother.

@Fassou said in Information & announcements:

This is Atlas carrying the celestial vault, or @cobito lifting the server, now I can't remember

A great one!

If Atlas were made of spaghetti, there would be doubt

@Sylver said in Information & announcements:

What a shame not to be able to return even a tiny part of what you do for us @cobito... What a great effort

One way or another, it seems that Debian is still surrounded by dangers as that computer science professor told me years ago... But I bet anything that he didn't have to face as many additions as in this case.

Anyway, a million thanks as always, and I hope to be able to contribute as much as I can to the cause.

Best regards!

No shame at all. As long as you drop by every now and then to look, that's fine. Debian is not the easiest although this time it has been more bad luck than anything else.

Well, in principle, everything is already functional. I've seen that even problems that were there when viewing the museum magazines (a problem that the previous version of Imagemagick had) have been solved and the emulations work again (something strange that Nginx did that I couldn't fix).

I still have to take a look at the forum with the CSS.

[cobito]

For the past few days, a group of malicious bots have been flooding the server with requests. Due to a series of changes by Debian that I need to study, the iptables rules for the web are not working, so I have to manually enter IPs and entire networks into the multiple Nginx configurations.

Because of this, and until the "attack" ends, some services might be temporarily unavailable (today, for example, they managed to take down Redis).

[kynes]

@cobito I know what you're talking about. My store is being targeted by the tiktok bot, which is terribly aggressive and doesn't respect robots.txt

Good luck, it's a very tedious task that awaits you.

[cobito]

The forum and Peertube and Mastodon instances will be updated next weekend (it is still unclear whether on Saturday or Sunday).

In principle, it is a routine process and should not take too long.

[cobito]

A few hours ago a security patch was released that fixes a critical vulnerability in Mastodon. The flaw is caused by something called Cross Site Scripting, which allows malicious scripts to be injected into trusted websites that, in this case, would have been our instance.

The patch is already applied so in this weekend's action, one less thing. We are on version 4.2.5.

[pos_yo]

@cobito Yes, I had read something. Thanks for your work!!

[cobito]

Well, that's it for now.

Regarding Nodebb, a problem should have been resolved so that the moderation team can manage user emails according to a bug reported by @Fassou a while ago.

Regarding Peertube, which has had a major version change, the main new features for the user are:

• Thumbnails have been added to the video progress bar, so that by hovering over it, specific moments can be visualized*.
• It is now possible to create chapters in videos.
• Improvements have been made to the player that make it faster and make it adjust better to certain aspect ratios.

*Regarding thumbnails, they are currently being generated for all videos, so at this time, they will only be available in new uploads.

And I'll take this opportunity to mention that tonight is the monthly check of the RAID array, so tomorrow everything will be quite slow in general.

[Fassou]

Mr. @cobito is a genius, who makes all these upgrade and update tasks seem almost easy.

Keeping to schedule, in good shape and with Changelog

[cobito]

@Fassou Thanks man, it's a pleasure (when things work out on the first or second try at most).

[cobito]

Since yesterday, the RAID is being rebuilt due to a problem with a disk. The process is painfully slow, much more so than last time. This is causing huge wait times to access the NAS, so everything on Hardlimit will be quite slow in the coming days.

While we're at it, I'll take this opportunity to inform you that last week, an "emergency" update was made to Mastodon due to some critical security flaws on the platform.

And finally, in two or three weeks, an update campaign will be carried out that will cover the forum and the Peertube instance. It should be a routine process that will be carried out on a Saturday or Sunday.