Information & announcements

[cobito]

The other day I got in touch with the council of wise men from Hardlimit to organize a meeting and discuss the strategies for the next course, but it turns out that they have all died of old age.

So with the savings from salaries, several budget items will be established that will be used to renew the different machines that keep this whole thing running.
The idea is to continue providing service to the current pages, some of which have seen their performance severely deteriorated in the last year due to lack of hardware resources (mainly the Peertube instance).

In addition, with the time savings that comes from not having to keep attending the useless council meetings, a series of development hours will be allocated to resume the different projects that have been abandoned for quite some time, such as the test bench and the museum for which there are several requests and a long list of wishes.

And with a little luck, new avenues can be opened to expand the range of resources offered from this house.

As for the new hardware, after the summer I will start the topic and open the first thread for you to advise me on a new machine. There are three projected, but only two are priority:

1. A powerful machine intended for hard processing: video encoding, audio transcription and experiments with AI (which will be used for very specific issues and in no case to mess up the forum or turn Hardlimit into another zombie of the dead Internet). I have this one pretty much defined in general.

2. A NAS of which I will make a copy-paste of the previous one I built (with current components) because the topic came out reasonably well. I have doubts mainly with the disks.

3. A front-end server (the least priority and which perhaps will be left until the next campaign). Here I am not clear about the topic and I have to see how the current one behaves when all the heavy load has been moved to the power machine.

In general, the three have to meet four issues above any other: they have to be very quiet, very reliable, of reduced volume (maximum micro-ATX) and they have to require very little maintenance considering that they will be on 24/7.

All the technical and economic details will be given in each new thread.

While we're at it, I take this opportunity to say that the cover, which has been on hold for a couple of weeks now, will not receive new content until September.

Happy summer.

[Yorus]

@cobito will let us know which components you need for each thing. Maybe someone has things at home that we can donate to the cause.

And well, they are always interesting threads to exchange opinions and see how you are progressing in decision making and then in assembly and preparation.

Happy summer!

[cobito]

The first phase of hardware renewal is finished.

The Ryzen 9 9900X has been put into operation and is now handling the video encodings of the Peertube instance. In addition, transcriptions have been re-enabled (they were turned off because the other server couldn't keep up) which the new machine also does. For now I can say that the performance with Whisper is brutal. With ffmpeg I still have to see more work to know how much it has improved but off the bat, each core is 3-4 times faster than before.

Regarding noise, it's an absolutely silent machine, to the point that I don't know how much noise it makes because I only hear the others.

Also, the Peertube instance has been updated to the latest version: 7.3. There are many changes for administration, but there are also visual improvements, functionality improvements and quite a few bugs fixed.

The next phase of renewal was going to be the NAS. But since I haven't bought a PC for myself since 2006, I'm going to renew my battle machine which is in dire need of it.

[cobito]

I've spent some time pushing the Ryzen 9 9900x to its limits and have come to several conclusions:

• 256GB of RAM is not necessary. The CPU falls short with models that consume around 20GB of RAM.
• NLLB is fine for translation between mainstream languages, but there's a video in Finnish out there and it's an absolute disaster. In the medium term, it's important to have robust translations in languages other than Western European ones.
• There are other superior models (much superior) like Tower+ but they are brutally slow with just the CPU.

I'm considering the different options I have without a GPU, which are not many or good, but there are some acceptable ones, even if only to get by and learn.

Regarding plugging in a graphics card, I've found this information regarding performance and prices. Using 16-bit floating point types:

• Ryzen 9 9900x: 8.6 Tflops
• Nvidia RTX 4060 ti: 177 Tflops (x20 the CPU). 450€
• Nvidia RTX 4090: 300Tflops (x35 the CPU). 2800€
• Nvidia H100: 1000Tflops (x116 the CPU). 34000€

Regarding saying that the last two options are just illustrative examples. But all this makes one think that a graphics card would give a very important boost to the issue. Unfortunately, the neglect of all these years has meant that there are other priorities regarding the acquisition of new hardware.

For now, we will continue to gain know-how so that when the time comes, the GPU issue can be better evaluated.

[pos_yo]

@cobito
Although in AI, Nvidia is unbeatable, that's beyond any doubt, AMD's GPUs are doing a good job with ROCm and it's in the latest versions of Mesa by default and with official AMD support. They still have a lot of work to do, though.

[cobito]

I've been 100% into the AI topic. Life's a funny thing, someone has made an inquiry in English on the forum that's here.

In that particular thread there's a menu where you can select the language. The Spanish version is an automated translation generated with the tools I'm putting together for Peertube.

It's all very very green still and I wasn't planning on doing anything for the forum anytime soon, but this inquiry came up and I'll use it to test everything.

For now, it's manual. That means if you reply, it will take a few hours for the reply to appear in both languages. For now, it will be the only multilingual thread.

[cobito]

Today 3 new hard drives have been added to the front server:

• One to house Peertube auxiliary files (replacing a pretty crappy SSD drive) in addition to the instance itself; now the same setup is used for Hardlimit and Tubedu, which reduces the memory footprint and makes future updates easier.
• Another one to exclusively house Mastodon (the main drive was already full).
• And another one for webs and forum, which could also serve as a starting point for the new museum library.

Today's schedule also included updating the forum and other operations that I didn't have time for, so this week's schedule is moved by a day and tomorrow, the forum will be down in the morning. Also, Mastodon will be down for the rest of the day as well because files are still being moved from one site to another. EDITED: Mastodon is now up and running. To compensate for the all-day outage, it has been updated to 4.4.8, which is the latest version available (we were coming from 4.2.x).

[cobito]

Well, we are already on the latest version of Nodebb (forum). I have to give a big review to the CSS to clean it up and put back things that have changed. But in general, everything is pretty square.

[cobito]

This week we started working on the test site. Unlike the museum, the test site is a pretty active site, with lots of visitors and participation, so we won't be making changes on the fly like we did with the museum, so version 4 will come out once it's finished.

On this version, the changes will be essentially internal focused on the migration to MariaDB. There are also a lot of code hacks in the style, for example, having the Windows versions hardcoded (among many other things). In general, the code is a monstrosity in size. It's true that it's very well organized, well commented and with a fairly coherent structure. The exaggerated complexity comes mainly from using the DBMS that I created at the time, which is what I'm eliminating now.

All this cleaning will mean a before and after in the possibilities of displaying the information and adding new functionalities, but it will be to a greater extent transparent for the user.

Another relevant change (but I don't know if feasible), will be to generate the pages in real time: at the moment, every time someone sends a result, a script processes the entire database and generates a cache, which is the one used to display the information in the front-end. What I want is for everything to be fast enough so that this pre-calculation process is not necessary.

I hope to have all this finished in a couple of weeks.

[cobito]

The database of the test bank has already been migrated to MariaDB and the motherboard page (motherboard.php) has been reimplemented. Only everything else remains (another 10 pages). I am able to dedicate much less time to it than initially planned, but the invaluable help of ChatGPT has accelerated the process by about an order of magnitude. With a bit of luck, I hope to be able to have everything ready (or almost) this week because some of the more complex things have already been done.

That said, it is unlikely that new CPUs will be added or motherboards and memories will be updated from now until the new version goes into production.

On another note, the recoding season of the Peertube instance has begun. I think I have not announced this before but I have mentioned it to someone in private (I do not remember who). Anyway, for reasons of energy efficiency, during part of the spring, summer and part of the autumn, live streams are not recoded. This is done during the cold months because I use the heat of the machines as heating for the floor (although it may seem like a joke, the savings are much more important than one might imagine). This season, in addition, the transcription of all pending videos (several thousand) will be done and in a few days, the process of translating the subtitles will begin. Here, the new Ryzen 9 9900X will be able to show off all its muscle (and all its watts).

Let's see how this week goes...

[cobito]

Version 4 of the test bench is in the "Release Candidate" phase (web only, the program as I said before will be left for later). If nothing goes wrong, it will be released this weekend.

In other news, subtitles for Peertube have started to be translated. The translation policy at the moment is:

• If the video is in English, it is translated into Spanish.
• If it is in a language other than English and compatible with the translation machinery (most major Western languages, Korean, Chinese, Japanese, Arabic, and Persian), it is translated into English.

When translations have progressed and the resource consumption of this has been better evaluated (I am having serious performance issues to which I have found a patch), it will be considered to translate everything into Spanish. The quality of the translations is generally quite good, mainly conditioned by the quality of the available transcription and the ability of the speaker to convey their ideas. That is to say, they are not perfect, sometimes mistakes are made with jargon (in this aspect, fewer errors than I expected) but at a level much more than decent.

The whole issue of translations will also condition decisions about acquiring new hardware in the future.

And about hardware, the second phase (NAS) and third phase (front-end server) of the hardware upgrade that was planned are on hold due to the rampant increase in prices, which is especially bleeding in mechanical hard drives and RAM.

[cobito]

Today the Peertube instance has been updated to version 8. Among other things, it comes with the ability to add collaborators to channels and a redesigned player.

[cobito]

Well, gentlemen, today marks the end of the quarter from an administrative standpoint. A lot has been done, but just as much has been left undone. What remains:

• Museum disc/file section (@vreyes1981 I haven't forgotten about this or your Micromanía discs). I tried to get it more or less done before Christmas, but I didn't have the time. This will be a high priority when tasks resume. Unfortunately, the lack of disk space (and the current situation in the hardware market) means that for now we will maintain the previous functionality even though the content organization has been completely restructured (this is almost finished but I want to finalize a few things before making it public). The idea is to have everything ready for when buying a new NAS is more viable than it is now and the deployment will be quick.
• Forum translation: the process has been running internally for a few days. This has been a bigger challenge than translating subtitles because the forum text is formatted in HTML and although the instructions to the model are clear and concise, sometimes it gets some HTML tags wrong (something that shouldn't be complicated to correct programmatically). Also (and this mistake probably occurred in the subtitles), it seems that the model wasn't being entirely clear about the task it had to do and the questions, it didn't always translate them literally, but rather translated the answer to the questions. If in any subtitle you see something like the definition of "free software" when the speaker said "but what is free software?", it's because of this lack of specificity on my part. The implementation on how to display the translations is outlined and is functional at an early testing level. Here, ChatGPT has messed things up for me several times and made me follow paths that led nowhere (I lost a couple of days doing nothing). The fault lies largely (apart from mine for delegating things I shouldn't have), in the poor documentation of NodeBB in some aspects although through much testing, I have managed to get a rough sketch that looks pretty good.

These are the two most priority things. Then there is an endless list of things that will come after and with which I won't bore you.

Development will resume in mid-January.

As every year, I will try to make the thread summarizing this year where the things that were done will come out.

[vreyes1981]

Hello @cobito, I just read your message.

¡¡No paasaa naa!! When it's possible, it's possible. This is done as a hobby and we're not obligated or paid for it, so don't worry, don't suffer. The important thing is that the saved material is in good hands.

That said, if we don't see each other these days, I wish you a merry Christmas.

[cobito]

This summer, a couple of Peertube users complained that they couldn't broadcast live because their connection was very slow. One was from Germany and another from Italy. I found it strange because the link is hundreds of Mbps symmetrical. After investigating a bit, I discovered that in certain cases, residential connections usually don't have good peering with other residential connections. This can happen because the routes are not good or because traffic from higher-level services is prioritized. I did some tests and, indeed, I could see that during peak hours (starting at 8 pm mainly), the bandwidth with one of the affected ISPs from here, didn't reach 1Mbps. With this situation in mind, I have realized that a residential connection is not a good option for maintaining the services that are maintained here. And more when there are strict bandwidth requirements. I was once doing tests with Cloudflare's Tunnel service. It's very easy to use, but there are limitations in terms of the protocols it supports (among which it doesn't support RTMP which is used for live broadcasts). In addition, the contract conditions do not make clear the traffic limiters, beyond responsible use. So I have looked for more complex options to configure but much more versatile and I have found OVHCloud's VPS. I have been doing some tests, and I think we are going to go there: they have installed Debian (the distro that the rest of the machines use), it has quite a lot of processing power and a sufficient amount of RAM and storage. In addition, they make it clear that the bandwidth is 400Mbps and the traffic is unlimited. All of this should be enough for the moment. The idea is that the Hardlimit data center will continue to be where it is but all traffic will go through OVH (server from France that was one of the few available when contracting the service). And with that, I hope that the problems related to bad routes/peerings disappear. In addition, I avoid exposing the public IP of my personal connection in the way it is exposed now. To be honest, I am surprised that my provider hasn't given me a warning after several DoS attacks that have been received. With this change, the domains' registrar will also move. Now we are using no-ip with its Enhanced DNS. Using the VPS it will no longer be necessary to have the dynamic DNS service, so it stops making sense to keep the domains in no-ip and they will also be moved to OVH. For the moment, the VPS will do the function exclusively of reverse proxy, although it is possible that the processing load of Peertube will end up being transferred because it turns out that CPU power is very cheap, but storage, RAM and GPU have astronomical prices. So the muscle will continue to be here. I say this only for informational purposes. The change will be made in the coming weeks and there will probably be service outages. Beyond that, the change should be transparent for you.

[_Neptunno_]

@cobito I have loved this post! Especially because, to be honest, if I read this a couple of years ago, I would have been completely lost in the technical part. Mind you, I'm not an expert now, but during this time at work I have been seeing things, I have researched on my own (also chats with Devops buddies who are great) and now everything sounds a lot more familiar.

Seeing your network diagram and how you have it set up, I am impressed with the hard work behind it. Analyzing my small homelab (where I have learned as much as I can by setting up a reverse proxy with NPM and managing my dynamic IP with DuckDNS), I realize that, although my system is very improvable, it's cool to see it work. But yours is on another level. I take my hat off to your work and the effort of maintaining the forum and all its services... what a job!!

You know, at first I didn't fully understand why you had ruled out Cloudflare (in my company we use it a lot, although I am not directly involved in its management). But after thinking about it, I realized that for your case it is unfeasible for two important reasons that you must have seen:

• The RTMP protocol: Cloudflare (in its free version) is optimized for web traffic. For PeerTube live streams you need RTMP and Cloudflare does not allow it in its free proxy; you would have to go to Enterprise solutions that cost a fortune.

• Streaming limitation: In their terms of service (policy 2.8 ) they make it clear that you cannot serve large amounts of video without a specific plan. As soon as they detect that you are moving terabytes of video through PeerTube, they would cut off the tap quickly.

Given this, I perfectly understand that you went for OVH, with whom we also work. We have some Proxmox nodes hosted there and even several MooseFS with their chunks (a beast where hundreds of disks with petabytes of information are stored). I tell you this because I like to share the things I discover, this world of systems and storage is crazy. By the way, we also have some servers at Voxility, now I can't remember certain configurations but we have several nodes in USA, Canada, Europe (UK, Germany and in another place I can't remember) and Singapore.
Anyway, we are moving many systems to AWS, but... this part is better known by Devops than my systems department jajaja

Anyway, sorry for the "lecture", I'm sure you don't know anything that I'm telling you jajaja, but I was excited to share that feeling of being able to understand such a technical post and seeing that I am learning a little more each day.
The truth is that I am super burned out at work for other reasons, but at least I get some knowledge and "fun"

Regards!!

P.D: If I let out any "technical nonsense" don't take it into account!!

[cobito]

Thanks @_Neptunno_

For me, having my homelab has always been fascinating... since my first 56kbps connection back in 2000, I was already tinkering with Counter-Strike servers, although with that bandwidth, it only allowed me (locally) and one other player.

Much later (it was actually just a few years later), when ADSL speeds started doubling, Jazztel implemented fastpath in their connections up to 20MBps (with uploads of about 1Mbps), I allowed myself to make a Counter Strike Source server for 16 players. And it was quite successful!

After that, I started hosting my personal blog on a Raspberry Pi. That's when I started learning Linux at an administrative level. And when I received Hardlimit (which was still on vBulletin), I fixed a Netbook (Atom N270 with 1GB of RAM) and it worked like a charm. Those were simpler times when only PHP and MySQL were fashionable

Little by little, I've been complicating my life until what there is now. And the truth is that, by doing so, you learn a lot of things, for example, until 6 months ago I had no idea what a peering agreement between providers was and I lived in the innocent world that my connection was (in relative terms by Mbps) equivalent to that of a data center

Continuing with the topic of VPS, CDNs and others, the problem with Cloudflare is that even with a paid contract (basic) they don't make clear the limitations of bandwidth or traffic. And, indeed, Peertube moves a few terabytes per month, which leaves me with doubt and uncertainty.

I like the VPS style better, because in the end I manage it like one more machine and in the case of OVH, they make all the characteristics and limitations totally clear. Plus, it has a very good price, all be said.

And returning to the migration, last night I was doing the first tests setting up a basic NAT. This morning I've spent a few hours trying to set up Nginx from the VPS but there are many things that get enormously complicated, mainly with Peertube and Mastodon since, among other issues, I need to have access to a mount point with the data and in the end it leaves a pretty dirty configuration, complicated to maintain and not very efficient. And not only that, but Mastodon, by design, does not admit an inverse proxy outside of localhost, so on top of that I would have to modify internal code of the platform. It's a real headache and in reality, the benefits are few.

So finally I'm going to leave a simple NAT.

From now on, all traffic is going through OVH. Those who connect from Spain, you won't notice any difference. People coming from other countries and continents, yes they will see a much higher quality of service.

Regarding domains, the people at noip are smart and instead of generating a migration key for you automatically (like normal registrars do), you have to open a ticket and wait for them to respond with the key. So I guess this will be done next week.

[cobito]

I thought this was under control but it wasn't (that's what happens when you do things for the first time). This weekend there have been a lot of outages; my apologies for not having planned this well or for not giving timely notices. The issue is, in principle, resolved and from now on, it should go well and without outages.

In summary, OVH has an anti-DDoS system (not configurable or deactivatable) with which UDP packets are always suspicious and when a certain threshold is exceeded (I don't know if it's a data rate or a packet rate), this system blocks the IP from which the packets are coming. As my server was using UDP to connect with the VPS, when there was a bit more activity than normal, it would jump and throw all the services. I have simply had to switch to using the TCP protocol.

To learn, to lose (half a weekend).