Information & announcements

[cobito]

The initial idea was clear: install Debian, install the packages, move certain data from one site to another, install the different components and off you go. In a couple of hours it should be ready. It's not the first time I've done this and there shouldn't be too many difficulties. The reality: today I'm 20 years older than last week.

Some things have worked out right away:

The house pages (front page, test bench and museum) don't seem to have major incompatibilities with the new PHP 8.2 (only a small change was necessary on the front page): here a more exhaustive check will still have to be done. In addition, the people at Nodebb haven't made major changes to the forum API so all the components have continued to communicate with the registry system without the need for changes.

Regarding the forum along with its database, it has been a piece of cake. Here the only issue is that there have indeed been a large number of changes in the layout of the pages, so the CSS has become partially obsolete. I've been tweaking it, although there are still details to be polished. There are a lot of new aesthetic features and, from my point of view, the mobile version has gained a lot of ground. There is still a small permissions problem that the venerable team of moderators is having which I hope is not a big deal.

On the other hand, it's the first time in many years that we've used the official Debian version of Redis, which, from a maintenance point of view, is a big step forward. Huge step forward. I'm very happy with this.

And that's the nice part. The rest has been a descent into the underworld:

· Many blogs are not compatible with PHP 8.2, so I'll have to do a manual update for each one of them. It shouldn't be a big deal, but it has to be done. I'm going to take my time with this.

· One of the changes in the organization of the directories has been to move all the "small" content (thumbnails, plugins...) from the Peertube instance to an SSD drive to offload transactions to the NAS (it was already very full). I had a brand new SSD drive almost brand new here but, oh, what a coincidence in life!, it's incompatible with the disk controller. You plug it in and it works. After a few hours, Linux starts reporting I/O errors with the ATA. You reboot and the BIOS stops detecting it. If you turn off the computer and turn it on, it works again. I've tried several cables and several SATA connectors but no, it's the darn disk (which, by the way, works perfectly on another computer). I've had to resort to a not-so-new SSD drive (although it's in good health).

· In Peertube supposedly you can configure the directories of each type of element individually: temporary, videos, thumbnails, etc. I've configured it to distribute it between the NAS and the SSD and apparently everything worked except that from the Internet, you couldn't access the content of the NAS. Why? Because the CMS uses a bridge in Nginx to access the data directly by defining something called 'root'. The problem is that there can only be one 'root'. So, which one do you choose: the NAS or the SSD? Well, after a few twists and turns, I chose the SSD by creating symbolic links to the folders on the NAS. It's not a clean option, but it's the only one I've found.

Well, so far 9 hours have passed, there's still Mastodon to get up and running (hahaha! someone sacrifice me, please) and I'm starting to question some decisions. I'll leave it for the next day because every first Sunday of each month, the NAS does a matrix check and that makes everything go very slow for a few hours.

· ¡Mastodon! This is already a major issue. So much so that I'm going to divide it into paragraphs:

Mastodon was hosted on the system disk. But the content takes up quite a bit and the only way it can be sustainable in the long term is to move it to the NAS. The operation is simple: connect the old disk and copy the data to the NAS. The data takes up about 240GB (not excessive), is spread across about 800,000 files (which is normal because they are images, their thumbnails and short videos) and all this is spread across (attention!) two and a half million folders. Any normal person would wonder why millions of folders are necessary to store hundreds of thousands of files?

This way of distributing the data has made me realize how brutally slow it is to create two and a half million folders on mechanical disks. It's incredibly slow; I would never have imagined it. In principle, copying 240GB to the NAS should have taken around an hour. Doing this copy has taken 14 hours. 14 more hours than the instance has been down.

Once the files are copied, it's time to install the CMS. The official instructions are wrong. They say that Ruby 3.0.6 is needed. I install that version and at a certain point, it gives an error. I try the installation several more times in case I skipped something until I start analyzing the logs thoroughly (yes, this has been my fault for not having done it before). It turns out that the latest version of Mastodon uses Ruby 3.2.2.

Everything starts to go well until it starts compiling. I think that perhaps I've made another mistake and I start the installation another couple of times with the same result. At this point, I've installed Mastodon 8 times. ¡Eight! I'm starting to think that the lack of certain cognitive abilities might be preventing me from carrying out this task. Then I come across a guy on Reddit who says he's already done 12 installations. ¡Eh, look at this geek, TWELVE times, HA! -I shout at the screen, feeling my pride partially recover.

After a few minutes, I recover my fragile mental equilibrium and I start to look for the solution that some guy has given to this poor unfortunate. It turns out that I'm using a version of NodeJS that's too recent. The version I'm using is the 18 LTS that came out in April 2022. How is it possible that the latest version of Mastodon, which came out two weeks ago, is not compatible with a NodeJS that has been around for a year and a half? Come on, someone answer, any philosophers in the room?

Leaving aside the humanist issues, I look at what version Mastodon needs to work: version 16 no less. A version that has been without support since September 11. Yes, gentlemen: the majority of Mastodon instances that are up and running are using a critical component without support. What do you think? Where are those philosophers, please?

In the end, I find a hack that allows Mastodon to be compiled with NodeJS 18, it works and that's it: I restore the database, I migrate it to the new version, I update the feeds and it's up and running. NOTE: This last paragraph has been adapted for a children's audience. During the procedure, a continuous crime of blasphemy has been committed, but this time due to my own ineptitude.

And already ascending from the depths of madness, I've realized something interesting and it is that Docker has become the de facto standard. In fact, it's starting to be complicated to find documentation to do certain things "the old way". And surely, some of the difficulties have arisen from that paradigm shift. So for next time, I'll have to train myself on this because it's already the present and it also seems an incredibly convenient way to install things. I hope it's with Debian 14.

Anyway, what a long winded story. But how comfortable am I with it?

P.D.: As usual, if you see something strange, don't hesitate to comment.

[krampak]

Mother of God, what an odyssey boss!

[Fassou]

This is Atlas carrying the celestial vault, or @cobito lifting the server, now I can't remember

A great one!

[Sylver]

What a shame not to be able to return even a tiny part of what you do for us @cobito... What a great job

One way or another, it seems that Debian is still surrounded by dangers as that computer science professor told me years ago... But I bet he didn't have to face as many additions as in this case.

Anyway, a million thanks as always, and I hope to be able to contribute as much as I can to the cause.

Best regards!

[cobito]

@krampak said in Information & announcements:

My God, what an odyssey boss!

Yes, this time it has been tough. Many services and each one of their own father and mother.

@Fassou said in Information & announcements:

This is Atlas carrying the celestial vault, or @cobito lifting the server, now I can't remember

A great one!

If Atlas were made of spaghetti, there would be doubt

@Sylver said in Information & announcements:

What a shame not to be able to return even a tiny part of what you do for us @cobito... What a great effort

One way or another, it seems that Debian is still surrounded by dangers as that computer science professor told me years ago... But I bet anything that he didn't have to face as many additions as in this case.

Anyway, a million thanks as always, and I hope to be able to contribute as much as I can to the cause.

Best regards!

No shame at all. As long as you drop by every now and then to look, that's fine. Debian is not the easiest although this time it has been more bad luck than anything else.

Well, in principle, everything is already functional. I've seen that even problems that were there when viewing the museum magazines (a problem that the previous version of Imagemagick had) have been solved and the emulations work again (something strange that Nginx did that I couldn't fix).

I still have to take a look at the forum with the CSS.

[cobito]

For the past few days, a group of malicious bots have been flooding the server with requests. Due to a series of changes by Debian that I need to study, the iptables rules for the web are not working, so I have to manually enter IPs and entire networks into the multiple Nginx configurations.

Because of this, and until the "attack" ends, some services might be temporarily unavailable (today, for example, they managed to take down Redis).

[kynes]

@cobito I know what you're talking about. My store is being targeted by the tiktok bot, which is terribly aggressive and doesn't respect robots.txt

Good luck, it's a very tedious task that awaits you.

[cobito]

The forum and Peertube and Mastodon instances will be updated next weekend (it is still unclear whether on Saturday or Sunday).

In principle, it is a routine process and should not take too long.

[cobito]

A few hours ago a security patch was released that fixes a critical vulnerability in Mastodon. The flaw is caused by something called Cross Site Scripting, which allows malicious scripts to be injected into trusted websites that, in this case, would have been our instance.

The patch is already applied so in this weekend's action, one less thing. We are on version 4.2.5.

[pos_yo]

@cobito Yes, I had read something. Thanks for your work!!

[cobito]

Well, that's it for now.

Regarding Nodebb, a problem should have been resolved so that the moderation team can manage user emails according to a bug reported by @Fassou a while ago.

Regarding Peertube, which has had a major version change, the main new features for the user are:

• Thumbnails have been added to the video progress bar, so that by hovering over it, specific moments can be visualized*.
• It is now possible to create chapters in videos.
• Improvements have been made to the player that make it faster and make it adjust better to certain aspect ratios.

*Regarding thumbnails, they are currently being generated for all videos, so at this time, they will only be available in new uploads.

And I'll take this opportunity to mention that tonight is the monthly check of the RAID array, so tomorrow everything will be quite slow in general.

[Fassou]

Mr. @cobito is a genius, who makes all these upgrade and update tasks seem almost easy.

Keeping to schedule, in good shape and with Changelog

[cobito]

@Fassou Thanks man, it's a pleasure (when things work out on the first or second try at most).

[cobito]

Since yesterday, the RAID is being rebuilt due to a problem with a disk. The process is painfully slow, much more so than last time. This is causing huge wait times to access the NAS, so everything on Hardlimit will be quite slow in the coming days.

While we're at it, I'll take this opportunity to inform you that last week, an "emergency" update was made to Mastodon due to some critical security flaws on the platform.

And finally, in two or three weeks, an update campaign will be carried out that will cover the forum and the Peertube instance. It should be a routine process that will be carried out on a Saturday or Sunday.

[Sylver]

Many thanks for the information and dedication, Honorable Mayor @cobito

I was already surprised that yesterday the forum didn't load a couple of times in the middle of the afternoon.
I was going to notify you privately but it came back up shortly after, I imagine it was already in the process of rebuilding the RAID.

Greetings!

[Yorus]

Thanks @cobito!. Yes, yesterday I noticed slowness in Mastodon, especially in loading images, but I assumed some kind of maintenance or improvement.

[cobito]

This morning the matrix reconstruction has finished, so from now on, everything should be back to normal.

When the Peertube team has released version 6.2 of the platform (probably next week), I will schedule the updates. By the way, this version comes with some very interesting features.

[Yorus]

Yes, everything is going great, Floro, Mastodon... thanks!

[cobito]

Nodebb has been updated to version 3.8.4 (from 3.6) which doesn't bring major changes for the end user but rather new features for moderation and administration.

Peertube has also been updated to version 6.2.1 from 6.0.3. This update brings AI to Hardlimit for the first time with Whisper, a feature that automatically adds subtitles to videos. For now, subtitles will only be added to new videos and starting from the last quarter of the year, we will begin transcribing all old videos (prior to today).

Regarding Whisper, I've run a couple of videos and the results are truly amazing.

From a hardware perspective, a new machine has been added to host the Redis server. It's an Asrock J3455 board with 16GB of RAM that joins the main server and the NAS.

The reason for separating Redis is that it's a program that's very demanding of disk I/O, which slows down the entire system and consumes a lot of RAM. Additionally, it dumps the database (about 7GB) to disk every 10 minutes, so it's constantly hammering the operating system's disk buffer. By isolating it on its own machine, I hope to significantly lighten the load on the main server, which will require quite a few additional resources from Whisper.

By the way, the monthly RAID array verification will be done this morning, so until tomorrow afternoon, everything will be a bit slower than usual.

As usual, if you see something strange, please let me know.

[Fassou]

Great job Mr. Cobito!

But take a little bit of the sun, it's getting too white and it's August.

You have a free hammock right there.