Information & announcements

[pos_yo]

@cobito
Although in AI, Nvidia is unbeatable, that's beyond any doubt, AMD's GPUs are doing a good job with ROCm and it's in the latest versions of Mesa by default and with official AMD support. They still have a lot of work to do, though.

[cobito]

I've been 100% into the AI topic. Life's a funny thing, someone has made an inquiry in English on the forum that's here.

In that particular thread there's a menu where you can select the language. The Spanish version is an automated translation generated with the tools I'm putting together for Peertube.

It's all very very green still and I wasn't planning on doing anything for the forum anytime soon, but this inquiry came up and I'll use it to test everything.

For now, it's manual. That means if you reply, it will take a few hours for the reply to appear in both languages. For now, it will be the only multilingual thread.

[cobito]

Today 3 new hard drives have been added to the front server:

• One to house Peertube auxiliary files (replacing a pretty crappy SSD drive) in addition to the instance itself; now the same setup is used for Hardlimit and Tubedu, which reduces the memory footprint and makes future updates easier.
• Another one to exclusively house Mastodon (the main drive was already full).
• And another one for webs and forum, which could also serve as a starting point for the new museum library.

Today's schedule also included updating the forum and other operations that I didn't have time for, so this week's schedule is moved by a day and tomorrow, the forum will be down in the morning. Also, Mastodon will be down for the rest of the day as well because files are still being moved from one site to another. EDITED: Mastodon is now up and running. To compensate for the all-day outage, it has been updated to 4.4.8, which is the latest version available (we were coming from 4.2.x).

[cobito]

Well, we are already on the latest version of Nodebb (forum). I have to give a big review to the CSS to clean it up and put back things that have changed. But in general, everything is pretty square.

[cobito]

This week we started working on the test site. Unlike the museum, the test site is a pretty active site, with lots of visitors and participation, so we won't be making changes on the fly like we did with the museum, so version 4 will come out once it's finished.

On this version, the changes will be essentially internal focused on the migration to MariaDB. There are also a lot of code hacks in the style, for example, having the Windows versions hardcoded (among many other things). In general, the code is a monstrosity in size. It's true that it's very well organized, well commented and with a fairly coherent structure. The exaggerated complexity comes mainly from using the DBMS that I created at the time, which is what I'm eliminating now.

All this cleaning will mean a before and after in the possibilities of displaying the information and adding new functionalities, but it will be to a greater extent transparent for the user.

Another relevant change (but I don't know if feasible), will be to generate the pages in real time: at the moment, every time someone sends a result, a script processes the entire database and generates a cache, which is the one used to display the information in the front-end. What I want is for everything to be fast enough so that this pre-calculation process is not necessary.

I hope to have all this finished in a couple of weeks.

[cobito]

The database of the test bank has already been migrated to MariaDB and the motherboard page (motherboard.php) has been reimplemented. Only everything else remains (another 10 pages). I am able to dedicate much less time to it than initially planned, but the invaluable help of ChatGPT has accelerated the process by about an order of magnitude. With a bit of luck, I hope to be able to have everything ready (or almost) this week because some of the more complex things have already been done.

That said, it is unlikely that new CPUs will be added or motherboards and memories will be updated from now until the new version goes into production.

On another note, the recoding season of the Peertube instance has begun. I think I have not announced this before but I have mentioned it to someone in private (I do not remember who). Anyway, for reasons of energy efficiency, during part of the spring, summer and part of the autumn, live streams are not recoded. This is done during the cold months because I use the heat of the machines as heating for the floor (although it may seem like a joke, the savings are much more important than one might imagine). This season, in addition, the transcription of all pending videos (several thousand) will be done and in a few days, the process of translating the subtitles will begin. Here, the new Ryzen 9 9900X will be able to show off all its muscle (and all its watts).

Let's see how this week goes...

[cobito]

Version 4 of the test bench is in the "Release Candidate" phase (web only, the program as I said before will be left for later). If nothing goes wrong, it will be released this weekend.

In other news, subtitles for Peertube have started to be translated. The translation policy at the moment is:

• If the video is in English, it is translated into Spanish.
• If it is in a language other than English and compatible with the translation machinery (most major Western languages, Korean, Chinese, Japanese, Arabic, and Persian), it is translated into English.

When translations have progressed and the resource consumption of this has been better evaluated (I am having serious performance issues to which I have found a patch), it will be considered to translate everything into Spanish. The quality of the translations is generally quite good, mainly conditioned by the quality of the available transcription and the ability of the speaker to convey their ideas. That is to say, they are not perfect, sometimes mistakes are made with jargon (in this aspect, fewer errors than I expected) but at a level much more than decent.

The whole issue of translations will also condition decisions about acquiring new hardware in the future.

And about hardware, the second phase (NAS) and third phase (front-end server) of the hardware upgrade that was planned are on hold due to the rampant increase in prices, which is especially bleeding in mechanical hard drives and RAM.

[cobito]

Today the Peertube instance has been updated to version 8. Among other things, it comes with the ability to add collaborators to channels and a redesigned player.

[cobito]

Well, gentlemen, today marks the end of the quarter from an administrative standpoint. A lot has been done, but just as much has been left undone. What remains:

• Museum disc/file section (@vreyes1981 I haven't forgotten about this or your Micromanía discs). I tried to get it more or less done before Christmas, but I didn't have the time. This will be a high priority when tasks resume. Unfortunately, the lack of disk space (and the current situation in the hardware market) means that for now we will maintain the previous functionality even though the content organization has been completely restructured (this is almost finished but I want to finalize a few things before making it public). The idea is to have everything ready for when buying a new NAS is more viable than it is now and the deployment will be quick.
• Forum translation: the process has been running internally for a few days. This has been a bigger challenge than translating subtitles because the forum text is formatted in HTML and although the instructions to the model are clear and concise, sometimes it gets some HTML tags wrong (something that shouldn't be complicated to correct programmatically). Also (and this mistake probably occurred in the subtitles), it seems that the model wasn't being entirely clear about the task it had to do and the questions, it didn't always translate them literally, but rather translated the answer to the questions. If in any subtitle you see something like the definition of "free software" when the speaker said "but what is free software?", it's because of this lack of specificity on my part. The implementation on how to display the translations is outlined and is functional at an early testing level. Here, ChatGPT has messed things up for me several times and made me follow paths that led nowhere (I lost a couple of days doing nothing). The fault lies largely (apart from mine for delegating things I shouldn't have), in the poor documentation of NodeBB in some aspects although through much testing, I have managed to get a rough sketch that looks pretty good.

These are the two most priority things. Then there is an endless list of things that will come after and with which I won't bore you.

Development will resume in mid-January.

As every year, I will try to make the thread summarizing this year where the things that were done will come out.

[vreyes1981]

Hello @cobito, I just read your message.

¡¡No paasaa naa!! When it's possible, it's possible. This is done as a hobby and we're not obligated or paid for it, so don't worry, don't suffer. The important thing is that the saved material is in good hands.

That said, if we don't see each other these days, I wish you a merry Christmas.

[cobito]

This summer, a couple of Peertube users complained that they couldn't stream live because their connection was very slow. One was from Germany and the other from Italy. It seemed strange to me because the connection is symmetric with hundreds of Mbps. After some research, I discovered that in certain cases, residential connections often don't have good peering with other residential connections. This can happen because the routes aren't optimal or because traffic for higher-tier services is prioritized. I ran some tests and, indeed, I could see that during peak hours (mainly after 8 PM), the bandwidth with one of the affected ISPs from here didn't even reach 1 Mbps.

With this situation in mind, I've realized that a residential connection isn't a good option for running the services we host here. Especially when there are strict bandwidth requirements.

At one point, I tested Cloudflare's Tunnel service. It's very easy to use, but there are limitations regarding the protocols it supports (among those it doesn't support is RTMP, which is used for live streams). Additionally, the terms of service don't clearly specify traffic limits, beyond requiring responsible use.

So, I looked for options that are more complex to set up but much more versatile, and I found OVHCloud's VPS. I've been running some tests, and I think we'll go with that option: they installed Debian (the distro used by the rest of the machines), it has plenty of processing power, and a sufficient amount of RAM and storage. Furthermore, they clearly state that the bandwidth is 400 Mbps and traffic is unlimited. All of this should be sufficient for now.

The idea is that the Hardlimit data center will remain where it is, but all traffic will route through OVH (a server in France, which was one of the few available when signing up for the service). With that, I hope the issues related to poor routing/peering will disappear. Additionally, it avoids exposing the public IP of my personal connection in the way it's currently exposed. Honestly, I'm surprised my provider hasn't given me a warning after the several DoS attacks we've received.

With this change, the domains will also be moved to a different registrar. We're currently using no-ip with their Enhanced DNS. Since using the VPS will make dynamic DNS unnecessary, it no longer makes sense to keep the domains with no-ip, so they will also be transferred to OVH.

For now, the VPS will function exclusively as a reverse proxy, though it's possible that some Peertube processing load will eventually be moved there, since CPU power turns out to be very cheap, while storage, RAM, and GPUs have astronomical prices. So, the heavy lifting will still remain here.

I'm sharing this purely for informational purposes. The transition will take place over the coming weeks and there will likely be some service interruptions. Beyond that, the change should be transparent to you.

[_Neptunno_]

@cobito I absolutely loved this post! Mainly because, to be honest, if I had read this a couple of years ago, I would have been completely lost in the technical parts. That said, I'm not an expert now either, but over time at work I've been picking things up, doing my own research (and chatting with DevOps coworkers who are real experts), and now everything makes much more sense to me.

Looking at your network diagram and how you've set everything up, I'm really impressed by the hard work behind it. Comparing it to my little home lab (where I've learned what I can by trial and error, setting up a reverse proxy with NPM and managing my dynamic IP with DuckDNS), I realize that even though my setup has plenty of room for improvement, it's still cool to see it working. But yours is on another level. Hats off to you for your work and the effort it takes to maintain the forum and all its services... what a massive undertaking!!

Actually, at first I didn't fully understand why you ruled out Cloudflare (we use it a lot at my company, though I'm not directly involved in managing it). But after giving it some thought, I realized it's simply not viable for your case for two major reasons you've probably already considered:

• The RTMP protocol: Cloudflare (in its free tier) is optimized for web traffic. For PeerTube live streams, you need RTMP, and Cloudflare doesn't support it on their free proxy; you'd have to go for Enterprise solutions that cost a fortune.

• Streaming limitations: In their terms of service (policy 2.8), they make it clear that you can't serve large amounts of video without a specific plan. The moment they detected you were moving terabytes of video through PeerTube, they'd cut it off quickly.

Given that, I completely understand why you went with OVH, which we also work with. We have some Proxmox nodes hosted there, and even several MooseFS clusters with their chunks (a real beast that stores hundreds of disks with petabytes of data). I'm sharing this because I enjoy talking about the things I'm discovering; the world of systems and storage is crazy. By the way, we also have some servers on Voxility. I don't remember certain configurations right now, but we have several nodes in the USA, Canada, Europe (UK, Germany, and another place I can't recall), and Singapore.
Anyway, we're moving a lot of systems to AWS, but honestly... the DevOps team knows that part better than my systems department, lol.

Anyway, sorry for the "wall of text," and I'm sure I'm not telling you anything you don't already know, lol, but I really wanted to share that feeling of being able to understand such a technical post and realizing I'm learning a little bit more every day.
Honestly, I'm super burned out at work because of other things, but at least I'm picking up some knowledge and having a bit of "fun"

Cheers!!

P.S.: If I drop any "technical nonsense," please don't hold it against me!!

[cobito]

Thanks @_Neptunno_

I've always been drawn to the idea of having my own homelab... Since my first 56kbps connection back around 2000, I was already tinkering with Counter-Strike servers, though with that bandwidth, it only allowed me (locally) and one other player to join.

Much later (actually just a few years after), when ADSL speeds began to double, Jazztel implemented Fastpath on their connections up to 20MBps (with uploads close to 1Mbps), I allowed myself to set up a Counter-Strike Source server for 16 players. And it was quite a success!

After that, I started hosting my personal blog on a Raspberry Pi. That's when I began learning Linux at an administrative level. And when I took over Hardlimit (which was still running on vBulletin), I got my hands on a Netbook (Atom N270 with 1GB of RAM) and it ran like a dream. Those were simpler times when only PHP and MySQL were the standard

Little by little, I've made things more complicated for myself until reaching what I have now. And the truth is, by doing so, you learn a ton about everything; for example, until 6 months ago, I had no idea what a peering agreement between providers was, and I lived in the innocent belief that my connection was (in relative Mbps terms) equivalent to a data center's

Continuing on the topic of VPS, CDNs, and the like, the problem with Cloudflare is that even with a paid (basic) plan, they don't clearly state the bandwidth or traffic limitations. And, indeed, Peertube moves quite a few terabytes a month, which leaves me with doubts and uncertainty.

I prefer the VPS approach, because in the end, I manage it like any other machine, and in OVH's case, they make all features and limitations completely clear. Plus, it has a very good price, all things considered.

And going back to the migration, last night I ran some initial tests setting up a basic NAT. This morning I spent a few hours trying to get Nginx running from the VPS, but there are many things that become incredibly complicated, mainly with Peertube and Mastodon since, among other issues, I need access to a mount point with the data, and in the end, it results in a pretty messy configuration that's hard to maintain and inefficient. And not only that, but Mastodon, by design, doesn't allow a reverse proxy outside of localhost, so on top of that, I'd have to modify the platform's internal code. Basically, it's a complete hassle and, in reality, the benefits are minimal.

So ultimately, I'm just going to stick with a simple NAT.

From now on, all traffic is routing through OVH. Those of you connecting from Spain won't notice any difference. People coming from other countries and continents, however, will see a significantly higher service quality.

Regarding the domains, the folks at No-IP are a bit of a pain; instead of automatically generating a migration key for you (like normal registrars do), you have to open a ticket and wait for them to reply with the key. So I guess this will happen next week.

[cobito]

I thought this was under control, but it wasn't (that’s what you get for doing things for the first time). There have been a lot of outages this weekend; my apologies for not planning this properly or giving timely notice. The issue is, in principle, resolved, and from now on, things should run smoothly without any further interruptions.

In short, OVH has an anti-DDoS system (which cannot be configured or disabled) that treats UDP packets as inherently suspicious. When a certain threshold is crossed (I’m not sure if it’s based on data rate or packet rate), the system blocks the source IP. Since my server was using UDP to connect to the VPS, any slight increase in normal activity would trigger it and take down all services. I """simply""" had to switch to using the TCP protocol.

It’s the price of learning (half a weekend, in this case).

[cobito]

On the 14th or 21st of this month, Hardlimit will be down for maintenance. Several tasks will be carried out:

• Internal cleaning of all equipment.
• Peertube will be updated to version 8.1.
• Minor hardware changes will be made to the NAS: replacement of the case, PSU, SATA controller, and fan. This change is primarily for thermal and noise reasons. We have chosen an Inter-Tech IM-2 Expander case, a 550W Pure Power 13 M PSU, and a Pure Wings 3 120mm fan. Since the board has a soldered CPU and passive cooling, I expect the only audible noise source to be the drives.
• The Redis server will be decommissioned and merged back into the frontend server, but with a dedicated disk just for it. This is done for power savings and noise reduction. If this change impacts performance, a solution will be proposed after summer. It's true that the 32GB of RAM will be a bit tight, but as long as the Ryzen 9 doesn't let us down, we'll be fine.
• The Peertube metadata disk is starting to show SMART warnings, so the content will be moved to the Mastodon disk (we've noticed both have less activity than expected, so there shouldn't be any performance impact). It's not critical since, in addition to having backups, it's information that can be regenerated, but the change will be made anyway.

On another note, the end of the quarter is approaching. The first quarter of the year was overly focused on the museum due to the persistent issue I had with the file browser. Fortunately for the museum (and the swamps), a rainy winter has given us more hours than initially planned, and it will be practically wrapped up, just needing to tie up a few loose ends.

As a result, the second quarter of the year will be the test bench quarter.

There are several planned changes at the program level, to the point where version 2.0 will be released. The idea is to keep the current metrics intact for backward compatibility with results and to add a new test.

New features will also appear in the program, both suggested by you and developed in-house. The plan is to release beta versions that will allow result validation, with these new features being added incrementally. Once the version is finalized, it will be signed to avoid Windows warnings.

Details will be revealed release by release.

Finally, minor changes are expected for the museum (lowest priority), along with some forum updates related to integration with other platforms, mainly Peertube, as well as promotion of its English version (which, by the way, has received minor fixes).

[cobito]

PeerTube has been updated to version 8.1.3 today. Additionally, NodeBB was updated to version 4.9.1 earlier this week. With this latest version, we've been experimenting with forum federation, but it's quite fragile, so for now we're sticking with the current setup.

Regarding the museum, you can now download files individually. We've added some Windows 95 and 98 CD-ROMs to expand our media library, and several improvements and fixes have been applied to the search and file explorer. Additionally, the indexing of extractable files is now complete. We have nearly 4.5 million files in total, of which just over half a million have been processed to extract their media (images, videos, MIDI, etc.). This means all the magazine CD-ROM files have already been processed, leaving only the extracted files and the newly added media (Windows 95/98). Going forward, development on the museum will take a back seat and will only receive minor updates and new content.

Regarding the network infrastructure, OVH has completely messed up with billing issues, and we've been without a VPS for the past two or three days (despite having paid until January 2027). I have a lot to say about this, but I'll keep it brief: I've never seen such brutally incompetent technical support, and I have zero hesitation in saying so because the negligence has been nothing short of epic. As soon as I get the chance (whether they refund me for the unused months or the contracted services simply expire), I will be leaving. So for now, and until they sort out this mess, all traffic is routing directly through my ISP.

[cobito]

We’re starting the spring quarter, and with it comes our first screw-up and first major task:

Yesterday I was dealing with some of the issues I’ll discuss below, and in the process, the forum database got corrupted. I had to fall back on a backup, and (here’s the screw-up), the most recent one was from March 18 (three weeks ago). Backups are supposed to run weekly, but between updates and time off, I missed the Redis one. As a crutch for my limited cognitive capacity, the forum actually runs on two different databases across two different DBMSs. That allowed me to recover the lost posts, but their timestamps are set to today. Furthermore, over the past three weeks, users registered to validate results in the testbed, and that data is unrecoverable.

And why was I tinkering with things yesterday? Well, because a bunch of scrapers felt the need to rip off all the content from the Peertube instance from hundreds of IP blocks originating mainly from Asia. We’ve finally been able to largely mitigate the attack, but due to the sheer volume of IPs, some requests are still slipping through. That’s causing everything to run slower than usual, particularly in the museum and the testbed. We’ll just have to hang in there for a bit until all the malicious IPs are banned (it’s a slow but automated process).

Right now, Hardlimit is under its most intense attack to date and will experience periodic outages.

In short, the server has been targeted by various attack attempts, not just Scraping/Crawling/Flood/DDoS, but also attempts to exploit zero-day vulnerabilities, privilege escalation, and the like (platforms like WordPress tend to be a sieve). However, these leeches are so aggressive that most attacks (though not all) are detected immediately. So far, they haven’t achieved anything (beyond taking down services by crashing the server) thanks to the web layer being properly isolated from the data layer. Still, monitoring the situation and applying fixes has been consuming way too much time. I hope that after implementing these latest measures, I can finally focus on more interesting things.

Essentially, IPs are being banned en masse, and I’ve been tweaking kernel parameters to avoid hitting swap space as much as possible. Right now, ZRAM is saving us from a much worse situation, and Crowdsec is saving our ass from leaving the server completely unusable.

On this front, there’s only so much we can do. We’re a small fish, the ocean is too vast, and AI has given sharp teeth even to the dumbest fish. These days, keeping internet-exposed machines secure is a real challenge. But for now, we’re holding our ground.

Setting aside the parable, the drama, and the whining, as I mentioned, I’m going to focus on the testbed, and more specifically, on the software itself. There’s a somewhat defined roadmap (more vague than defined) that, if completed, would take us to the second major version.

It’s been a long time since I last worked on it. The first step will be to dust off the development environment, which lives on a virtual machine, review the code, etc.

The first significant change I want to make is migrating from a multiprocess model to a multithreaded model. The reason I want to make this rather boring change is to simplify the testbed startup: currently, process synchronization is relatively fragile (in certain environments, the bm tends to fail) and delays the start of the tests. Additionally, this change will allow us to improve safeguards against result manipulation.

Overall, this will result in a more robust program, simpler internally (hopefully), and with identical behavior regarding the reproducibility of results across previous versions.

I’ll likely start on this next week and probably dedicate the entire week just to administrative tasks: reviewing the VM, getting familiar with the structure, code, etc.

That said, with each new feature (such as implementing the threading model), a beta version will be released. As was the case with the first version, anyone who participates and leaves feedback will be listed in the program’s acknowledgments window (if they aren’t already) and will have the option to change their name if they wish. Those who don’t participate but are already listed will remain there.

The goal is to release a new version roughly every two weeks, though this will depend heavily on various factors.

In parallel, I’ll be making minor tweaks to the museum, the translation system for both the forum and subtitles, and other small issues. And yes, damn cybersecurity will continue to hog my time.

[cobito]

A brief update:

The attack continues in bursts. It causes average loads of >150pu, bandwidth of >450Mbps, and hundreds of ffmpeg processes to process the requests. Disk I/O has been successfully reduced to a negligible amount, so despite the ridiculous numbers, the various configurations that have been applied, along with the fact that those requests are given minimum priority, allow the server to hold up and maintain a very good response speed during these bursts, which can last several hours.

We're in good shape on this front.

Regarding the testbed: migration to the threading model has begun. Test #1 and Test #2 have been completed. #3 (the most complex) and #4 remain. Additionally, in this batch of changes, all the hardware detection code and related components will be migrated to the program's new architecture. This means it's quite likely that the first beta will be released, at the earliest, in two weeks.

Once the program is in good shape, the rest of the changes should start arriving more quickly. Basically, the strategy is the same as with the museum, where it took a while for major changes to roll out, but once everything was in place, implementing improvements became much easier.

To wrap up: this weekend, a new controller was installed in the NAS. This saves me an external drive (which is now internal) and has allowed me to add individual activity LEDs, so I can check the load status at a glance. Beyond that, this change doesn't affect anything else.

[cobito]

Testing

• The migration to the threading model has been a failure. Tests #1, #2, and #4 are running fine, but #3 (compression) exhibits anomalous behavior in multithreaded mode (yielding results far below expectations). I've tried a wide range of approaches, but nothing works. I'm going to stop dedicating time to this process for now (though I might revisit it later).

• I'm focusing on tidying up the code by separating the main components to decouple them (currently, everything depends on everything else), but this has to be done with extreme caution to avoid altering the behavior. It's a slow process that won't introduce new features, but it's essential for maintainability. This means that, even though I'm not posting updates on this front, a significant number of hours are being (and continue to be) dedicated to it every week. The first beta will be released once we have something presentable. In short, 90% of the time resources are being focused on this.

Infrastructure

• As I already mentioned, OVH left us without a VPS quite some time ago. We're moving to Arsys, and currently all traffic is already being routed through a VPS. All that's left is to migrate the domains. Coincidentally, the day after we started the migration, OVH responded (after over a month) with a reply that leaves much to be desired in tone, but it does resolve the core issue (they are refunding me for a service I paid for but never received).

AI

• We've started using a new model for forum translation, which puts us at the very top in this area. It requires extreme resources and is reserved exclusively for new posts. Additionally, we've reserved a processing slot that allows for the retranslation of one post per day. Peertube continues to use the previous model.

Administration

• To wrap up, the forum goes down occasionally. I'm looking into it, but it's likely due to the heavy load we've been experiencing lately.

[cobito]

Infrastructure

The cause of the forum's outages has been identified: a faulty RAM module. This is one of those things that drives me up the wall, but on the upside, it's an 8GB DDR3 module, so the memory crisis probably won't affect me much here (at least I'm not seeing outrageous prices).

I'm looking for one of the exact same model, and in the meantime, services have been redistributed as best as possible. Until it's replaced, we'll try to make do with the 104GB of RAM that remain operational (and, by the way, they're running at practically 100% capacity).

Regarding this, I'm closely monitoring the machine in case it's not just that module or if the issue lies elsewhere (motherboard, socket, memory controller, or whatever).

Test Bench

Code refactoring is ongoing. If there aren't too many setbacks, the first beta with a few new small features should be out in one or two weeks.

Front Page

It's been pretty neglected in general. I've focused on new features (forum federation, an agent intern with video summaries, the RAM failure that took up a lot of my time, etc.). I think it's more worthwhile to spend time on these things than on the front page, so in the medium term, there will be few posts per week.

Forum

As I already mentioned, it's federated and can exchange content with the Peertube instance. I encourage you to visit the threads with the selected videos. There's one on lightweight desktop environments and another on video game news, where the topics are presented in a rigorous and well-explained manner.

Regarding the integration with federated threads, they've also implemented showing them in recent threads. This will make the content translate (at the moment, those threads are in limbo). However, it's not clear yet whether they will appear in unread threads when there are new posts (even if they're local). The issue doesn't seem to be philosophically or technically easy to resolve.

Security

Attacks continue in bursts, but they have slowed down. Optimizations have reached the point where the bottleneck is now bandwidth (>400Mbps serving only scalpers). I'll look for a way to minimize it without affecting other things as much as possible.

Additionally, as some of you may know, several major security vulnerabilities have been discovered in both the Linux kernel and other components. We're up to date here, but lately we've been seeing things we hadn't seen before, and as I mentioned in another update, we need to stay very vigilant.