Information & announcements

[cobito]

Version 4 of the test bench is in the "Release Candidate" phase (web only, the program as I said before will be left for later). If nothing goes wrong, it will be released this weekend.

In other news, subtitles for Peertube have started to be translated. The translation policy at the moment is:

• If the video is in English, it is translated into Spanish.
• If it is in a language other than English and compatible with the translation machinery (most major Western languages, Korean, Chinese, Japanese, Arabic, and Persian), it is translated into English.

When translations have progressed and the resource consumption of this has been better evaluated (I am having serious performance issues to which I have found a patch), it will be considered to translate everything into Spanish. The quality of the translations is generally quite good, mainly conditioned by the quality of the available transcription and the ability of the speaker to convey their ideas. That is to say, they are not perfect, sometimes mistakes are made with jargon (in this aspect, fewer errors than I expected) but at a level much more than decent.

The whole issue of translations will also condition decisions about acquiring new hardware in the future.

And about hardware, the second phase (NAS) and third phase (front-end server) of the hardware upgrade that was planned are on hold due to the rampant increase in prices, which is especially bleeding in mechanical hard drives and RAM.

[cobito]

Today the Peertube instance has been updated to version 8. Among other things, it comes with the ability to add collaborators to channels and a redesigned player.

[cobito]

Well, gentlemen, today marks the end of the quarter from an administrative standpoint. A lot has been done, but just as much has been left undone. What remains:

• Museum disc/file section (@vreyes1981 I haven't forgotten about this or your Micromanía discs). I tried to get it more or less done before Christmas, but I didn't have the time. This will be a high priority when tasks resume. Unfortunately, the lack of disk space (and the current situation in the hardware market) means that for now we will maintain the previous functionality even though the content organization has been completely restructured (this is almost finished but I want to finalize a few things before making it public). The idea is to have everything ready for when buying a new NAS is more viable than it is now and the deployment will be quick.
• Forum translation: the process has been running internally for a few days. This has been a bigger challenge than translating subtitles because the forum text is formatted in HTML and although the instructions to the model are clear and concise, sometimes it gets some HTML tags wrong (something that shouldn't be complicated to correct programmatically). Also (and this mistake probably occurred in the subtitles), it seems that the model wasn't being entirely clear about the task it had to do and the questions, it didn't always translate them literally, but rather translated the answer to the questions. If in any subtitle you see something like the definition of "free software" when the speaker said "but what is free software?", it's because of this lack of specificity on my part. The implementation on how to display the translations is outlined and is functional at an early testing level. Here, ChatGPT has messed things up for me several times and made me follow paths that led nowhere (I lost a couple of days doing nothing). The fault lies largely (apart from mine for delegating things I shouldn't have), in the poor documentation of NodeBB in some aspects although through much testing, I have managed to get a rough sketch that looks pretty good.

These are the two most priority things. Then there is an endless list of things that will come after and with which I won't bore you.

Development will resume in mid-January.

As every year, I will try to make the thread summarizing this year where the things that were done will come out.

[vreyes1981]

Hello @cobito, I just read your message.

¡¡No paasaa naa!! When it's possible, it's possible. This is done as a hobby and we're not obligated or paid for it, so don't worry, don't suffer. The important thing is that the saved material is in good hands.

That said, if we don't see each other these days, I wish you a merry Christmas.

[cobito]

This summer, a couple of Peertube users complained that they couldn't broadcast live because their connection was very slow. One was from Germany and another from Italy. I found it strange because the link is hundreds of Mbps symmetrical. After investigating a bit, I discovered that in certain cases, residential connections usually don't have good peering with other residential connections. This can happen because the routes are not good or because traffic from higher-level services is prioritized. I did some tests and, indeed, I could see that during peak hours (starting at 8 pm mainly), the bandwidth with one of the affected ISPs from here, didn't reach 1Mbps. With this situation in mind, I have realized that a residential connection is not a good option for maintaining the services that are maintained here. And more when there are strict bandwidth requirements. I was once doing tests with Cloudflare's Tunnel service. It's very easy to use, but there are limitations in terms of the protocols it supports (among which it doesn't support RTMP which is used for live broadcasts). In addition, the contract conditions do not make clear the traffic limiters, beyond responsible use. So I have looked for more complex options to configure but much more versatile and I have found OVHCloud's VPS. I have been doing some tests, and I think we are going to go there: they have installed Debian (the distro that the rest of the machines use), it has quite a lot of processing power and a sufficient amount of RAM and storage. In addition, they make it clear that the bandwidth is 400Mbps and the traffic is unlimited. All of this should be enough for the moment. The idea is that the Hardlimit data center will continue to be where it is but all traffic will go through OVH (server from France that was one of the few available when contracting the service). And with that, I hope that the problems related to bad routes/peerings disappear. In addition, I avoid exposing the public IP of my personal connection in the way it is exposed now. To be honest, I am surprised that my provider hasn't given me a warning after several DoS attacks that have been received. With this change, the domains' registrar will also move. Now we are using no-ip with its Enhanced DNS. Using the VPS it will no longer be necessary to have the dynamic DNS service, so it stops making sense to keep the domains in no-ip and they will also be moved to OVH. For the moment, the VPS will do the function exclusively of reverse proxy, although it is possible that the processing load of Peertube will end up being transferred because it turns out that CPU power is very cheap, but storage, RAM and GPU have astronomical prices. So the muscle will continue to be here. I say this only for informational purposes. The change will be made in the coming weeks and there will probably be service outages. Beyond that, the change should be transparent for you.

[_Neptunno_]

@cobito I have loved this post! Especially because, to be honest, if I read this a couple of years ago, I would have been completely lost in the technical part. Mind you, I'm not an expert now, but during this time at work I have been seeing things, I have researched on my own (also chats with Devops buddies who are great) and now everything sounds a lot more familiar.

Seeing your network diagram and how you have it set up, I am impressed with the hard work behind it. Analyzing my small homelab (where I have learned as much as I can by setting up a reverse proxy with NPM and managing my dynamic IP with DuckDNS), I realize that, although my system is very improvable, it's cool to see it work. But yours is on another level. I take my hat off to your work and the effort of maintaining the forum and all its services... what a job!!

You know, at first I didn't fully understand why you had ruled out Cloudflare (in my company we use it a lot, although I am not directly involved in its management). But after thinking about it, I realized that for your case it is unfeasible for two important reasons that you must have seen:

• The RTMP protocol: Cloudflare (in its free version) is optimized for web traffic. For PeerTube live streams you need RTMP and Cloudflare does not allow it in its free proxy; you would have to go to Enterprise solutions that cost a fortune.

• Streaming limitation: In their terms of service (policy 2.8 ) they make it clear that you cannot serve large amounts of video without a specific plan. As soon as they detect that you are moving terabytes of video through PeerTube, they would cut off the tap quickly.

Given this, I perfectly understand that you went for OVH, with whom we also work. We have some Proxmox nodes hosted there and even several MooseFS with their chunks (a beast where hundreds of disks with petabytes of information are stored). I tell you this because I like to share the things I discover, this world of systems and storage is crazy. By the way, we also have some servers at Voxility, now I can't remember certain configurations but we have several nodes in USA, Canada, Europe (UK, Germany and in another place I can't remember) and Singapore.
Anyway, we are moving many systems to AWS, but... this part is better known by Devops than my systems department jajaja

Anyway, sorry for the "lecture", I'm sure you don't know anything that I'm telling you jajaja, but I was excited to share that feeling of being able to understand such a technical post and seeing that I am learning a little more each day.
The truth is that I am super burned out at work for other reasons, but at least I get some knowledge and "fun"

Regards!!

P.D: If I let out any "technical nonsense" don't take it into account!!

[cobito]

Thanks @_Neptunno_

For me, having my homelab has always been fascinating... since my first 56kbps connection back in 2000, I was already tinkering with Counter-Strike servers, although with that bandwidth, it only allowed me (locally) and one other player.

Much later (it was actually just a few years later), when ADSL speeds started doubling, Jazztel implemented fastpath in their connections up to 20MBps (with uploads of about 1Mbps), I allowed myself to make a Counter Strike Source server for 16 players. And it was quite successful!

After that, I started hosting my personal blog on a Raspberry Pi. That's when I started learning Linux at an administrative level. And when I received Hardlimit (which was still on vBulletin), I fixed a Netbook (Atom N270 with 1GB of RAM) and it worked like a charm. Those were simpler times when only PHP and MySQL were fashionable

Little by little, I've been complicating my life until what there is now. And the truth is that, by doing so, you learn a lot of things, for example, until 6 months ago I had no idea what a peering agreement between providers was and I lived in the innocent world that my connection was (in relative terms by Mbps) equivalent to that of a data center

Continuing with the topic of VPS, CDNs and others, the problem with Cloudflare is that even with a paid contract (basic) they don't make clear the limitations of bandwidth or traffic. And, indeed, Peertube moves a few terabytes per month, which leaves me with doubt and uncertainty.

I like the VPS style better, because in the end I manage it like one more machine and in the case of OVH, they make all the characteristics and limitations totally clear. Plus, it has a very good price, all be said.

And returning to the migration, last night I was doing the first tests setting up a basic NAT. This morning I've spent a few hours trying to set up Nginx from the VPS but there are many things that get enormously complicated, mainly with Peertube and Mastodon since, among other issues, I need to have access to a mount point with the data and in the end it leaves a pretty dirty configuration, complicated to maintain and not very efficient. And not only that, but Mastodon, by design, does not admit an inverse proxy outside of localhost, so on top of that I would have to modify internal code of the platform. It's a real headache and in reality, the benefits are few.

So finally I'm going to leave a simple NAT.

From now on, all traffic is going through OVH. Those who connect from Spain, you won't notice any difference. People coming from other countries and continents, yes they will see a much higher quality of service.

Regarding domains, the people at noip are smart and instead of generating a migration key for you automatically (like normal registrars do), you have to open a ticket and wait for them to respond with the key. So I guess this will be done next week.

[cobito]

I thought this was under control but it wasn't (that's what happens when you do things for the first time). This weekend there have been a lot of outages; my apologies for not having planned this well or for not giving timely notices. The issue is, in principle, resolved and from now on, it should go well and without outages.

In summary, OVH has an anti-DDoS system (not configurable or deactivatable) with which UDP packets are always suspicious and when a certain threshold is exceeded (I don't know if it's a data rate or a packet rate), this system blocks the IP from which the packets are coming. As my server was using UDP to connect with the VPS, when there was a bit more activity than normal, it would jump and throw all the services. I have simply had to switch to using the TCP protocol.

To learn, to lose (half a weekend).

[cobito]

On the 14th and/or the 21st of this month, Hardlimit will be offline for maintenance. Several actions will be performed:

• Internal cleaning of all equipment.
• Peertube will be updated to version 8.1
• Minor hardware changes will be made to the NAS: change of case, power supply, SATA controller and fan. This change is made mainly for thermal and noise issues. An Inter-Tech IM-2 Expander case, a Pure Power 13 M 550W power supply and a Pure Wings 3 120mm fan have been chosen. As it is a motherboard with soldered micro and passive cooling, I hope that the only audible noise source will be the disks.
• The Redis server will be retired and reintegrated into the front server, but with a dedicated disk only for it. This is done for energy saving and noise issues. If this change impacts performance, a solution will be proposed after the summer. It is true that the 32GB of RAM will be a bit tight, but as long as the Ryzen 9 doesn't let us down, we're fine.
• The Peertube metadata disk is starting to give SMART warnings, so the content will be moved to the Mastodon disk (it has been seen that both have less activity than expected, so there should be no impact on performance). It is not critical because, in addition to having copies, it is information that can be regenerated, but the change will be made.

In other news, the end of the quarter is approaching. The first quarter of the year has been excessively focused on the museum due to the thorn in my side with the file browser. Fortunately for the museum (and the swamps), a rainy winter has made more hours available to us than initially programmed and this will be able to be practically closed except for cutting a few loose ends.

In this way, the second quarter of the year will be the quarter of the test bench.

There are several changes planned at the program level, to the point that version 2.0 will be released. The idea is to keep the current metrics intact for issues of backward compatibility with results and add a new test.

New features will also appear in the program, both proposed by you and of our own making. The idea is to release beta versions from which it will be possible to validate results and with which these new functions will be added incrementally. Once the version is closed, it will be signed to avoid warnings from Windows.

The details will be revealed launch by launch.

Finally, minor changes are expected in the museum (minimal priority) and some news in the forum related to integration with other platforms, mainly with Peertube, in addition to the promotion of its English version (of which, by the way, minor arrangements have been made).

[cobito]

Peertube has been updated to version 8.1.3 today. Also, this week Nodebb was updated to version 4.9.1. With this latest version, we have been experimenting with the federalization of the forum but it is very fragile so for now we will stay as we are.

Regarding the museum, it is now possible to download files individually, some CDROMs of Windows 95 and 98 have been added to expand the types of media we have and several improvements/corrections have been made to the search engine and file explorer. In addition, the indexing of extractable files has been completed. We have about 4.5 million files of which, just over half a million have been processed to extract their media (images, videos, midi, etc). This means that the files of all the magazine CDROMs are already processed and the extracted ones and the new media (Windows 95/98) are left. From now on, the development of the museum will take a back seat and will only receive minor changes and new content.

Regarding the network infrastructure, OVH has messed it up with billing issues and we have been without VPS for two or three days already (despite having paid until January 2027). I have a lot to say about this, but I will summarize it a lot: I have never seen a technical service so brutally incompetent, and I have no problem saying that because the negligence is of epic proportions. As soon as I have the opportunity (either because they return the money of the months not consumed or because the contracted services end), I will run away from there. So at the moment and until they solve the problem, all traffic is going directly through my ISP.