Information & announcements

[cobito]

We’re starting the spring quarter, and with it comes our first screw-up and first major task:

Yesterday I was dealing with some of the issues I’ll discuss below, and in the process, the forum database got corrupted. I had to fall back on a backup, and (here’s the screw-up), the most recent one was from March 18 (three weeks ago). Backups are supposed to run weekly, but between updates and time off, I missed the Redis one. As a crutch for my limited cognitive capacity, the forum actually runs on two different databases across two different DBMSs. That allowed me to recover the lost posts, but their timestamps are set to today. Furthermore, over the past three weeks, users registered to validate results in the testbed, and that data is unrecoverable.

And why was I tinkering with things yesterday? Well, because a bunch of scrapers felt the need to rip off all the content from the Peertube instance from hundreds of IP blocks originating mainly from Asia. We’ve finally been able to largely mitigate the attack, but due to the sheer volume of IPs, some requests are still slipping through. That’s causing everything to run slower than usual, particularly in the museum and the testbed. We’ll just have to hang in there for a bit until all the malicious IPs are banned (it’s a slow but automated process).

Right now, Hardlimit is under its most intense attack to date and will experience periodic outages.

In short, the server has been targeted by various attack attempts, not just Scraping/Crawling/Flood/DDoS, but also attempts to exploit zero-day vulnerabilities, privilege escalation, and the like (platforms like WordPress tend to be a sieve). However, these leeches are so aggressive that most attacks (though not all) are detected immediately. So far, they haven’t achieved anything (beyond taking down services by crashing the server) thanks to the web layer being properly isolated from the data layer. Still, monitoring the situation and applying fixes has been consuming way too much time. I hope that after implementing these latest measures, I can finally focus on more interesting things.

Essentially, IPs are being banned en masse, and I’ve been tweaking kernel parameters to avoid hitting swap space as much as possible. Right now, ZRAM is saving us from a much worse situation, and Crowdsec is saving our ass from leaving the server completely unusable.

On this front, there’s only so much we can do. We’re a small fish, the ocean is too vast, and AI has given sharp teeth even to the dumbest fish. These days, keeping internet-exposed machines secure is a real challenge. But for now, we’re holding our ground.

Setting aside the parable, the drama, and the whining, as I mentioned, I’m going to focus on the testbed, and more specifically, on the software itself. There’s a somewhat defined roadmap (more vague than defined) that, if completed, would take us to the second major version.

It’s been a long time since I last worked on it. The first step will be to dust off the development environment, which lives on a virtual machine, review the code, etc.

The first significant change I want to make is migrating from a multiprocess model to a multithreaded model. The reason I want to make this rather boring change is to simplify the testbed startup: currently, process synchronization is relatively fragile (in certain environments, the bm tends to fail) and delays the start of the tests. Additionally, this change will allow us to improve safeguards against result manipulation.

Overall, this will result in a more robust program, simpler internally (hopefully), and with identical behavior regarding the reproducibility of results across previous versions.

I’ll likely start on this next week and probably dedicate the entire week just to administrative tasks: reviewing the VM, getting familiar with the structure, code, etc.

That said, with each new feature (such as implementing the threading model), a beta version will be released. As was the case with the first version, anyone who participates and leaves feedback will be listed in the program’s acknowledgments window (if they aren’t already) and will have the option to change their name if they wish. Those who don’t participate but are already listed will remain there.

The goal is to release a new version roughly every two weeks, though this will depend heavily on various factors.

In parallel, I’ll be making minor tweaks to the museum, the translation system for both the forum and subtitles, and other small issues. And yes, damn cybersecurity will continue to hog my time.

[cobito]

A brief update:

The attack continues in bursts. It causes average loads of >150pu, bandwidth of >450Mbps, and hundreds of ffmpeg processes to process the requests. Disk I/O has been successfully reduced to a negligible amount, so despite the ridiculous numbers, the various configurations that have been applied, along with the fact that those requests are given minimum priority, allow the server to hold up and maintain a very good response speed during these bursts, which can last several hours.

We're in good shape on this front.

Regarding the testbed: migration to the threading model has begun. Test #1 and Test #2 have been completed. #3 (the most complex) and #4 remain. Additionally, in this batch of changes, all the hardware detection code and related components will be migrated to the program's new architecture. This means it's quite likely that the first beta will be released, at the earliest, in two weeks.

Once the program is in good shape, the rest of the changes should start arriving more quickly. Basically, the strategy is the same as with the museum, where it took a while for major changes to roll out, but once everything was in place, implementing improvements became much easier.

To wrap up: this weekend, a new controller was installed in the NAS. This saves me an external drive (which is now internal) and has allowed me to add individual activity LEDs, so I can check the load status at a glance. Beyond that, this change doesn't affect anything else.

[cobito]

Testing

• The migration to the threading model has been a failure. Tests #1, #2, and #4 are running fine, but #3 (compression) exhibits anomalous behavior in multithreaded mode (yielding results far below expectations). I've tried a wide range of approaches, but nothing works. I'm going to stop dedicating time to this process for now (though I might revisit it later).

• I'm focusing on tidying up the code by separating the main components to decouple them (currently, everything depends on everything else), but this has to be done with extreme caution to avoid altering the behavior. It's a slow process that won't introduce new features, but it's essential for maintainability. This means that, even though I'm not posting updates on this front, a significant number of hours are being (and continue to be) dedicated to it every week. The first beta will be released once we have something presentable. In short, 90% of the time resources are being focused on this.

Infrastructure

• As I already mentioned, OVH left us without a VPS quite some time ago. We're moving to Arsys, and currently all traffic is already being routed through a VPS. All that's left is to migrate the domains. Coincidentally, the day after we started the migration, OVH responded (after over a month) with a reply that leaves much to be desired in tone, but it does resolve the core issue (they are refunding me for a service I paid for but never received).

AI

• We've started using a new model for forum translation, which puts us at the very top in this area. It requires extreme resources and is reserved exclusively for new posts. Additionally, we've reserved a processing slot that allows for the retranslation of one post per day. Peertube continues to use the previous model.

Administration

• To wrap up, the forum goes down occasionally. I'm looking into it, but it's likely due to the heavy load we've been experiencing lately.

[cobito]

Infrastructure

The cause of the forum's outages has been identified: a faulty RAM module. This is one of those things that drives me up the wall, but on the upside, it's an 8GB DDR3 module, so the memory crisis probably won't affect me much here (at least I'm not seeing outrageous prices).

I'm looking for one of the exact same model, and in the meantime, services have been redistributed as best as possible. Until it's replaced, we'll try to make do with the 104GB of RAM that remain operational (and, by the way, they're running at practically 100% capacity).

Regarding this, I'm closely monitoring the machine in case it's not just that module or if the issue lies elsewhere (motherboard, socket, memory controller, or whatever).

Test Bench

Code refactoring is ongoing. If there aren't too many setbacks, the first beta with a few new small features should be out in one or two weeks.

Front Page

It's been pretty neglected in general. I've focused on new features (forum federation, an agent intern with video summaries, the RAM failure that took up a lot of my time, etc.). I think it's more worthwhile to spend time on these things than on the front page, so in the medium term, there will be few posts per week.

Forum

As I already mentioned, it's federated and can exchange content with the Peertube instance. I encourage you to visit the threads with the selected videos. There's one on lightweight desktop environments and another on video game news, where the topics are presented in a rigorous and well-explained manner.

Regarding the integration with federated threads, they've also implemented showing them in recent threads. This will make the content translate (at the moment, those threads are in limbo). However, it's not clear yet whether they will appear in unread threads when there are new posts (even if they're local). The issue doesn't seem to be philosophically or technically easy to resolve.

Security

Attacks continue in bursts, but they have slowed down. Optimizations have reached the point where the bottleneck is now bandwidth (>400Mbps serving only scalpers). I'll look for a way to minimize it without affecting other things as much as possible.

Additionally, as some of you may know, several major security vulnerabilities have been discovered in both the Linux kernel and other components. We're up to date here, but lately we've been seeing things we hadn't seen before, and as I mentioned in another update, we need to stay very vigilant.

[cobito]

A few hours ago, a PeerTube security patch was released that also addresses another vulnerability discovered three days ago. The Hardlimit instance was updated both three days ago and a few hours ago.

The vulnerability allowed code injection, SQL injection, plugin installation, and the exploitation of processing tokens to manipulate videos published by users. Additionally, they detected a manipulation of the Google Analytics domain that was reporting traffic data to an illegitimate site (we do not use this).

The checks recommended by the platform developers have been performed, and no signs of penetration or manipulation have been detected.

The instance is now secure against these vulnerabilities, but all user sessions had to be reset, so you will need to log in again.

[cobito]

Today's tasks are essentially complete.

The web proxy component has been moved to the VPS. Following this, cache testing has been underway, and it now primarily caches content from the Peertube instances (videos and images), the Mastodon instance, and the museum. This should significantly improve the loading speed of certain files, particularly the museum's emulations, image loading, and video playback under higher traffic. Additionally, this is expected to reduce I/O activity on the NAS and the other drives.

From a security perspective, this is a major change as it isolates the internet-facing machine from everything else: nothing else has direct external contact except for the proxy-cache, which doesn't actually store any data.

Regarding software, the following updates have been made:

• Nodebb from 4.11 to 4.12 (improvements to the visibility of federated threads).
• Peertube from 8.1 to 8.2 (ability to transfer channels to other users, along with fixes for several important bugs).
• Mastodon from 4.4 to 4.5 (quoting other posts is now possible, provided the author allows it).

As for the hardware, XMP has been disabled on the Ryzen 9 9900X (dropping from 6400MT/s to 4800MT/s, a 25% reduction). This is done to lower the voltage from 1.4V to 1.1V and is purely a matter of thermals and longevity: if those RAM modules fail, we'll be facing a major issue. This will have a direct impact on the AI models. At the moment, while handling live tasks (processing them as they come in, without revisiting past ones), there is still spare capacity. Once the temperatures start to drop, we will switch back to the XMP profile.

[cobito]

The translator now handles federated threads. Until now, it wasn't possible to fetch them because NodeBB didn't expose any list that included these threads, but with the latest update, that has changed. As a result, you will no longer see the annoying "This post is being processed/translated..." message indefinitely. For now, it has been tested on some older threads (not all are included since we're running in "low power" mode), but it will be applied to all new ones.

Additionally, I've implemented a small utility I need for server management, which I'm sharing with you in case it comes in handy: visiting extra.hardlimit.com/ip will return your public IP as a plain text string, with no HTML formatting.

[cobito]

Quick update:

• It seems the Nodebb team actually did handle the embedded videos in cross-posts correctly. It was my fault for not enabling an option. Since yesterday, they've been showing up in new threads.
• We've brought on a new intern to process threads: someone named Smith. This is because if HIAL-9000 is both posting and processing the threads, it won't see its own threads during processing since they don't appear in its "unread" list (typical silly oversight that cost me almost an hour yesterday going over the scripts to figure out why HIAL-9000 wouldn't process its own threads ).

[pos_yo]

@cobito It's impressive everything you're implementing just "for fun". Great work, thanks!

[cobito]

Alright, last post of the term:

The end of the term is approaching, and with it, the holiday break, which will be longer than usual this year.

Benchmark

In the end, the benchmark has been left half-finished and will remain in beta for a few months. I want to keep adding features to it before finalizing it.

Basically, new features have been added to the interface along with secondary tools, but the Core remains unmodified despite receiving many changes that haven't been included. There are several decisions I still need to make.

The reason for not releasing more updates is that the program's code was (and to some extent still is) a genuine headache. It's much better now than before, but it's something that has been neglected all this time, unlike the rest of the pages, which have received some attention over the last 5 years. In fact, most of the Core's code was practically the same since the early stable versions (around 10 years ago?).

In short: many hours have been spent on invisible work, but it will help speed up the release of updates from now on.

It's been impossible for me to dedicate time to it in the last few days, so there probably won't be any more betas until after the summer.

Homepage

It's been the great neglected one. After experimenting with video summaries, I've seen that AI is superior to me in understanding, correcting, and condensing concepts. The homepage basically consists of writing a short news summary, which I've done manually until now. And honestly, after seeing that, I've developed an incredible amount of laziness to keep doing that task. It's like having a brand-new excavator with a full tank of fuel, sitting on its tracks, while a worn-out pickaxe and shovel wait for someone to pick them up to dig a trench. What a pain to dig a trench with a pickaxe and shovel! The downside of all this is that one of the homepage's purposes has been to show that there are people behind the scenes maintaining the site. Delegating the cognitive effort of this task to a machine will erase that "symbol" of activity in one stroke. On the other hand, I don't think it's necessary right now to show signs of activity just for the sake of it.

In general, the homepage has been something that hasn't motivated me much for a while, because it's hard to find real news (the world of personal computing is essentially dead), because sensationalism is the norm, and because sometimes you have to verify unlikely (but potentially true) information. It takes a lot of time and effort, and the value it brings is debatable. If a machine can do that job and even do it better than me, why not?

I still have to make the final decision, but it's highly likely that upon my return (sometime in September), the homepage will undergo a radical internal overhaul. It's the only page that hasn't received development hours in the last year, and since it's going to be redone, I think this is the way it will be approached.

Forum

I'm trying to inject some activity into it from the Peertube instance. It seems like the right place to publish text-based content, and in fact, I was considering migrating the posts from my old retro blog to the forum. I was also thinking about creating posts about thematic videos (The Computer Chronicles comes to mind right now) in a new retro subforum, since it's a branch with a lot of momentum at the moment.

In this regard, I'm still on uncharted territory. If social media and aggressive SEO ended the forum era over a decade ago, I'm pondering how a forum fits into the current AI era. Providing content as a starting point for discussions isn't complicated as long as there continues to be content generation on the Peertube instance. Being a Q&A hub remains a possibility for very specific and poorly documented topics. But attracting people to participate directly is a challenge.

We'll see if I can revive some old formulas...

Museum

It's been completely neglected this term. It remains the center of gravity for my interest because it's where I feel most comfortable experimenting without the fear of breaking things. But I can't leave the rest neglected, and besides, one also gets tired of always working on the same thing.

I'd like to shovel content in and push to the limit all the development I did a few months ago to see how it performs at scale, but the lack of storage prevents me from doing so. Still, I have some ideas to squeeze every last byte out of the 2TB allocated to this project.

Infrastructure

We have three urgent needs: more storage, more GPU processing power, and better energy efficiency. These are three machines that, as of today, cost an unaffordable fortune. During the next term, the acquisition of this hardware will depend exclusively on price trends, which puts us at the earliest in 2027.

Meanwhile, the hoarding frenzy by AI scalpers has greatly reduced to the point where, at the moment, we're running at nominal levels for both bandwidth and CPU usage. And to be honest, I'm genuinely curious to know the reasons behind this abrupt drop in activity compared to the momentum AI is currently experiencing, regardless of how beneficial this drop is for us. Knowing where personal computing is heading in the coming months directly depends on what the AI "industry" does, and given how extremely opaque that field is, having these discreet data points always helps a bit to understand the situation.

I don't have anything special planned for the next term: naturally, the benchmark program will be the priority. Essentially, I'll let myself be guided a bit more by whatever I feel like doing at the time (without neglecting anything and with some planning), and it will also depend on how feasible it is to purchase the planned hardware, since we're a bit limited in that regard at the moment.

If I can find some time, I might release a bit more until the first week of July, but not much.

Happy summer!

[cobito]

Last night, the hard drive on the Ryzen 9 9900X failed miserably. That left us without the forum or translations all day. It's been a disaster. I've spent about 10 hours getting everything back up and running, which involved finding a working hard drive, installing the system, and trying to recover the forum's database from the dead drive. It was impossible to recover the database, so I had to fall back on a backup from Saturday. That means any threads and crossposts created since then are lost.

For now, our bots (Smith and HIAL-9000) are offline. I need to get the entire AI infrastructure back online, which will be a slow process, so it'll be ready either today or tomorrow.

[cobito]

A PeerTube security update has been released and is already installed. It resolves high-severity security issues.

The details will be published in the coming days on the version tag.

[cobito]

We'll start getting things up and running little by little this term. The first thing will be to add to the database all the hardware that the staff has been validating on the test bench during the summer. I see quite a few processors, a good number of motherboards, and countless RAM models. I'll get to work on that in the coming days.

Software Update

The first major undertaking will be working on the servers. This will likely happen during the week of September 7th (the exact day will be specified later). Currently, all web requests pass through a caching proxy on a VPS. The initial idea was to reduce the load and bandwidth on the backend, but it hasn't worked as expected. On one hand, there are prolonged bursts of transfers between the backend and the VPS that saturate the connection (the exact opposite of what I intended). This is because when someone downloads a large file, the entire file is copied to the cache while being served. On the other hand, countless issues have arisen (mainly on the museum site) for reasons I still don't fully understand (it's my first time configuring Nginx's cache and there's something I'm not doing right).

That means we'll go back to the previous setup: using the VPS as a gateway to take advantage of its better connection routing, and that's it.

In parallel, the main server's OS will be updated (now acting as the backend). This operation is usually quite tedious and will likely take up almost a full day.

There are also pending updates for NodeBB (forum) and Mastodon.

Hardware Update

Regarding the current hardware, we'll be rotating hard drives. In recent months, a couple of drives have failed (one of them suddenly and disastrously). There's likely a methodological flaw on my part here: I've taken for granted that hard drives are permanent as long as operating conditions are adequate, and having backups has made me complacent. It's also true that the number of hard drives has multiplied over the past year: we used to just have the NAS and the system drive. Now, there are also dedicated drives for Mastodon resources, PeerTube, Redis, the museum, the web server, and the Ryzen 9900X. The probability of failure multiplies accordingly.

Anyway, after that, I was left without hard drives suitable for these tasks, so I've replenished the reserves (they aren't new, but have very little usage). I still need to figure out how to distribute the services and data, but that's irrelevant for now.

Regarding the hardware renewal campaign that was left unfinished last term, given the current situation, it won't be resumed anytime soon. A few months ago, I was hoping things would start normalizing around this time of year, and although there are signs of a cycle shift and some macroeconomic forces seem to be emerging that could help restore balance, the harsh reality is that it won't happen by 2026.

Development

I still don't know how much time I'll be able to dedicate to this over the coming months. I hope it'll be something similar to what I've been doing recently, but nothing is guaranteed.

Last term, the test bench project was shelved. Some beta versions were released, but issues arose when trying to reproduce results after making structural changes to the program (threading vs. process architecture). The initial idea is to keep the core intact for tests 1 through 4, add a new test (large memory) within a multithreaded framework, and gradually add features to the GUI. There are also pending details on the website (layout, graph aesthetics, and other minor issues).

In parallel, the homepage will be reimplemented to abandon the DBMS I originally created, which has proven a pain to maintain, slow, and insecure, in favor of migrating to the same structure used for the test bench and the museum. As for content, the idea is to experiment with AI (remember that at Hardlimit, only open-source models run locally are used). I want to run some tests to see how it works. If the results aren't as expected, we'll revert to the manual method. The idea is also to offer an English-translated version. All of this will require massive amounts of processing power, so that might be where the plan falls through. We'll see.

And to wrap this up, all the code related to forum translation and PeerTube subtitles will be refactored. This has always been done experimentally, in bits and pieces, throwing things together haphazardly. It has ended up resulting in a somewhat chaotic file and directory structure.

I also want to experiment with more flexible forum translations, meaning giving the model more freedom to split and merge sentences to make the translations sound more natural. Currently, it's instructed to respect the original text's structure, which sometimes leads to unnatural translations in the target language. It's a delicate topic (from the standpoint of preserving the fidelity of the information being conveyed), which is why I want to run some tests before implementing it.

The entire forum translation project will have higher priority than last term, as there's finally some activity (very little, but it's there), which leads me to believe that the number of English-speaking readers on the forum must also be increasing.

Anyway, there are more things on the back burner (promoting certain types of content on PeerTube, encouraging people to participate in the forum, pushing to use the Mastodon instance, etc.). But the current infrastructure calls for caution when pushing for even moderate growth in activity, especially on PeerTube. Since the ultimate goal is for everything to mutually reinforce each other, this has been put on hold for now.

[cobito]

September ended up being much more complicated than expected, but last week I managed to carve out a few hours to carry out the planned updates.

Going forward, I will no longer post software versions and other technical details here for security reasons:

• The operating system has been updated to the latest version.
• Mastodon and PeerTube are also on their latest versions (both were previously running versions with critical security vulnerabilities).
• NodeBB (the forum) was updated, but the latest version has a bug that prevents profile pictures from displaying for older users, so I had to roll back. The bug has been reported and already fixed, so I'll try upgrading again once the next version is released.

Other changes:

• The system hard drive has been replaced.
• Redis is back on the backend (it was previously running on the Ryzen 9).
• For now, we're operating without the VPS. This is still on my to-do list. I've dismantled the previous setup where Nginx and other services ran on the VPS while the rest were on the backend. The VPS will now be used exclusively as a bridge for connections.
• The NAS will now be used exclusively for PeerTube videos (videos only; everything else is on a different drive).

I won't have much time over the next few weeks, so all planning for this quarter will be delayed.