Hardlimit server update

[cobito]

With the arrival of the NextGenerationHL funds, the governing team of this, your home, has decided to renew the infrastructures that support these noble lands.

After two years with the current server (the fastest and most stable to date), it will be replaced by the fourth machine that has powered Hardlimit since I took charge of the setup. With double the cores and double the RAM, it is expected to be able to process the heavy workload coming from the video section as well as provide the possibility of expanding the functionalities of both the museum and the test bank central.

Taking advantage of the change, the following software updates will be made:
· Debian 11 from the current Debian 10.
· The forum will be updated to Nodebb v1.18.5 from the current 1.4.
· The video section will be updated to Peertube 3.4.1 from the current 3.0.1.

In addition, a problem with Redis (the forum's database and the video section's cache) that was left pending a few months ago when it was updated to Debian 10 and that is currently working in a makeshift manner remains to be solved. I am in contact with the developers to see if we can find the solution, which would greatly simplify the administration tasks. But I see this as complicated in this iteration.

Now I am doing tests on the hardware. When I see it clear (probably in one or two weeks), I will proceed with the change for which I will give a few days' notice.

[_Neptunno_]

As always, a thousand thanks for your tireless work to keep our home operational
Out of curiosity, what processor and memory will the new server have? And the old one?

Greetings!!

[cobito]

@_Neptunno_ We went from 16 to 32GB of RAM. I prefer not to announce the platform for now

[Sylver]

Mayor, you are our greatest cyber fortune ever found (extensible in part to the Honorary Admins).

Eternal thanks for so much, and sincere apologies for so little!

For many more years online, cheers!

[cobito]

After the bridge break, I'm starting to get an idea of the dates. In principle, the update will take place on the weekend of December 17th. With a little luck, the problem with Redis will be solved, which is what I'm waiting for. If by that day it hasn't been solved, the update will proceed leaving the solution that is now in place.

[QVENGADOR]

@cobito said in Hardlimit server update:

NextGenerationHL

I want to be funded a PC, where do I apply for it?

[cobito]

@QVENGADOR said in Hardlimit server update:

@cobito said in Hardlimit server update:

NextGenerationHL

I want to be funded a PC, where do I apply for it?

Too late. I spent the last budget on an ivory scraper.

[QVENGADOR]

I am dying of laughter

[cobito]

It's been a few hours of sweating but it's done.

At first glance, everything seems to be working correctly. If you see anything strange, please let us know. The forum seems to need a coat of paint because they've added new graphic elements that don't fit well with the custom CSS.

Well, for those interested in what's going on behind the scenes, I'll explain a bit about the issue.

The previous server was chosen with only the forum and satellite pages in mind, without the video section. But the expansion of functionalities of pages like the museum, along with the appearance of the video section, have made this machine no longer sufficient.

The new server is the company's PC with RAM expansion:
· Core i5-3570K.
· Asus P8B75-V.
· 4 x G-Skill F3-1600C9-8GXM (32GB total).

As you can see, the PC is a powerhouse... from 8 years ago. But why this machine? The main reason is that it's the best I have available at the moment: it has reasonable performance and has proven to be completely reliable. We came from another Ivy Bridge but with only two cores and slightly lower frequency.

These are the other reasons that motivate the change:

1. The two instances of Peertube make intensive use of ffmpeg to transcode video. This implies a lot of CPU usage. When the video is in 4K, it consumes 2-3GB of RAM. To this we must add that the video section of Hardlimit and TubEdu are two independent instances, that is, each one launches its own ffmpeg and it's not uncommon for two transcodings to be done simultaneously. That, in the current CPU with 2 cores (although it has HT) sometimes results in a problem for the response time of other services.

2. The forum's database uses Redis, which resides entirely in memory. This makes the speed unbeatable (in fact, now it has taken a significant step in response time thanks to the fix that the guys in charge of the project have made [1]), but at the cost of the forum's database alone consuming around 8GB of RAM (although with the updates, now it seems to consume a little less).

3. The museum's search engine uses an index that, at the moment, occupies about 1.5GB. The speed of the search depends on the speed of the storage unit. To achieve the best speed, a RAMDisk is used, which requires an additional 1.5GB of RAM. Moreover, I want to expand the search functions so I will need more gigabytes. All this is actually optimizable, but I don't feel like doing that task at the moment.

4. The system, the multiple instances of Apache, Nginx, the processing of test bench results, etc., etc., etc., makes the 16GB of RAM we had until now exceed. That's why the museum's search has been disabled for a few months. And even so, sometimes it has given me a bit of a headache because it starts to use paging.

The new amount of RAM is sufficient for what we need. But the Ivy Bridge is not the best option. Firstly, we have the obvious reason: currently there are much more powerful CPUs. And secondly, energy efficiency is far from optimal and even more so if ffmpeg is going to be used intensively (remember that Ivy Bridge still doesn't have AVX2).

But a compromise has to be reached. On the one hand, the current context [2] makes buying new hardware not the best possible financial decision, even less so if it has to be a heavy-duty machine that is not urgent. Hardlimit is far from being a "financial resource" for me, but I don't want to buy something now that will probably be much cheaper in a year or spend an amount for which I can get something much better in a year (see for example the absurd price of DDR5 memory; that's an obscenity, man).

On the other hand, the 4 cores of Ivy Bridge will allow ffmpeg to run on several threads, which, along with the slight increase in CPU frequency and the improvement in memory latencies, will multiply the transcoding speed by 2 or 3. And that improvement seems more than reasonable for the moment.

What consumes the most resources now is the video section and the forum. We can't expect a significant increase in RAM needs from the forum because even if activity increases, it's just text (images and other things go on disk). From the video section, we don't know. At the moment (and especially on weekends) it is very demanding. If in about a year, activity has increased substantially, alternatives to the current machine will be considered.

As for the results of the change, we'll have to see how the machine behaves when Peertube jobs arrive. The forum seems to respond much faster. But the big winners are undoubtedly the blogs thanks to the replacement of a mechanical disk with an SSD where the MySQL database is stored.

Annotations

[1] Due to a bug in the Redis code, the forum's database was causing an overflow that gave an "unknown" error in recent versions. This meant that I had to run an old version of Redis on a virtual machine with an old version of Debian, with the resulting performance loss and extra RAM consumption. The solution has been to change the data types of two variables from "int" to "size_t" and the overflow no longer occurs. Size_t is a variable length equal to the maximum size of a vector that can be stored on the machine where the program is compiled. More details.

[2] The current context includes a logistical crisis that has strained supply and demand increasing prices everywhere. The current context also includes the appearance of new platforms and technologies that have had a higher starting price than usual (eg. DDR5).

[kynes]

@cobito What a big hit you take. Thanks for keeping order in this den