Information & announcements

[cobito]

A brief update:

The attack continues in bursts. It causes average loads of >150pu, bandwidth of >450Mbps, and hundreds of ffmpeg processes to process the requests. Disk I/O has been successfully reduced to a negligible amount, so despite the ridiculous numbers, the various configurations that have been applied, along with the fact that those requests are given minimum priority, allow the server to hold up and maintain a very good response speed during these bursts, which can last several hours.

We're in good shape on this front.

Regarding the testbed: migration to the threading model has begun. Test #1 and Test #2 have been completed. #3 (the most complex) and #4 remain. Additionally, in this batch of changes, all the hardware detection code and related components will be migrated to the program's new architecture. This means it's quite likely that the first beta will be released, at the earliest, in two weeks.

Once the program is in good shape, the rest of the changes should start arriving more quickly. Basically, the strategy is the same as with the museum, where it took a while for major changes to roll out, but once everything was in place, implementing improvements became much easier.

To wrap up: this weekend, a new controller was installed in the NAS. This saves me an external drive (which is now internal) and has allowed me to add individual activity LEDs, so I can check the load status at a glance. Beyond that, this change doesn't affect anything else.

[cobito]

Testing

• The migration to the threading model has been a failure. Tests #1, #2, and #4 are running fine, but #3 (compression) exhibits anomalous behavior in multithreaded mode (yielding results far below expectations). I've tried a wide range of approaches, but nothing works. I'm going to stop dedicating time to this process for now (though I might revisit it later).

• I'm focusing on tidying up the code by separating the main components to decouple them (currently, everything depends on everything else), but this has to be done with extreme caution to avoid altering the behavior. It's a slow process that won't introduce new features, but it's essential for maintainability. This means that, even though I'm not posting updates on this front, a significant number of hours are being (and continue to be) dedicated to it every week. The first beta will be released once we have something presentable. In short, 90% of the time resources are being focused on this.

Infrastructure

• As I already mentioned, OVH left us without a VPS quite some time ago. We're moving to Arsys, and currently all traffic is already being routed through a VPS. All that's left is to migrate the domains. Coincidentally, the day after we started the migration, OVH responded (after over a month) with a reply that leaves much to be desired in tone, but it does resolve the core issue (they are refunding me for a service I paid for but never received).

AI

• We've started using a new model for forum translation, which puts us at the very top in this area. It requires extreme resources and is reserved exclusively for new posts. Additionally, we've reserved a processing slot that allows for the retranslation of one post per day. Peertube continues to use the previous model.

Administration

• To wrap up, the forum goes down occasionally. I'm looking into it, but it's likely due to the heavy load we've been experiencing lately.

[cobito]

Infrastructure

The cause of the forum's outages has been identified: a faulty RAM module. This is one of those things that drives me up the wall, but on the upside, it's an 8GB DDR3 module, so the memory crisis probably won't affect me much here (at least I'm not seeing outrageous prices).

I'm looking for one of the exact same model, and in the meantime, services have been redistributed as best as possible. Until it's replaced, we'll try to make do with the 104GB of RAM that remain operational (and, by the way, they're running at practically 100% capacity).

Regarding this, I'm closely monitoring the machine in case it's not just that module or if the issue lies elsewhere (motherboard, socket, memory controller, or whatever).

Test Bench

Code refactoring is ongoing. If there aren't too many setbacks, the first beta with a few new small features should be out in one or two weeks.

Front Page

It's been pretty neglected in general. I've focused on new features (forum federation, an agent intern with video summaries, the RAM failure that took up a lot of my time, etc.). I think it's more worthwhile to spend time on these things than on the front page, so in the medium term, there will be few posts per week.

Forum

As I already mentioned, it's federated and can exchange content with the Peertube instance. I encourage you to visit the threads with the selected videos. There's one on lightweight desktop environments and another on video game news, where the topics are presented in a rigorous and well-explained manner.

Regarding the integration with federated threads, they've also implemented showing them in recent threads. This will make the content translate (at the moment, those threads are in limbo). However, it's not clear yet whether they will appear in unread threads when there are new posts (even if they're local). The issue doesn't seem to be philosophically or technically easy to resolve.

Security

Attacks continue in bursts, but they have slowed down. Optimizations have reached the point where the bottleneck is now bandwidth (>400Mbps serving only scalpers). I'll look for a way to minimize it without affecting other things as much as possible.

Additionally, as some of you may know, several major security vulnerabilities have been discovered in both the Linux kernel and other components. We're up to date here, but lately we've been seeing things we hadn't seen before, and as I mentioned in another update, we need to stay very vigilant.

[cobito]

A few hours ago, a PeerTube security patch was released that also addresses another vulnerability discovered three days ago. The Hardlimit instance was updated both three days ago and a few hours ago.

The vulnerability allowed code injection, SQL injection, plugin installation, and the exploitation of processing tokens to manipulate videos published by users. Additionally, they detected a manipulation of the Google Analytics domain that was reporting traffic data to an illegitimate site (we do not use this).

The checks recommended by the platform developers have been performed, and no signs of penetration or manipulation have been detected.

The instance is now secure against these vulnerabilities, but all user sessions had to be reset, so you will need to log in again.

[cobito]

Today's tasks are essentially complete.

The web proxy component has been moved to the VPS. Following this, cache testing has been underway, and it now primarily caches content from the Peertube instances (videos and images), the Mastodon instance, and the museum. This should significantly improve the loading speed of certain files, particularly the museum's emulations, image loading, and video playback under higher traffic. Additionally, this is expected to reduce I/O activity on the NAS and the other drives.

From a security perspective, this is a major change as it isolates the internet-facing machine from everything else: nothing else has direct external contact except for the proxy-cache, which doesn't actually store any data.

Regarding software, the following updates have been made:

• Nodebb from 4.11 to 4.12 (improvements to the visibility of federated threads).
• Peertube from 8.1 to 8.2 (ability to transfer channels to other users, along with fixes for several important bugs).
• Mastodon from 4.4 to 4.5 (quoting other posts is now possible, provided the author allows it).

As for the hardware, XMP has been disabled on the Ryzen 9 9900X (dropping from 6400MT/s to 4800MT/s, a 25% reduction). This is done to lower the voltage from 1.4V to 1.1V and is purely a matter of thermals and longevity: if those RAM modules fail, we'll be facing a major issue. This will have a direct impact on the AI models. At the moment, while handling live tasks (processing them as they come in, without revisiting past ones), there is still spare capacity. Once the temperatures start to drop, we will switch back to the XMP profile.

[cobito]

The translator now handles federated threads. Until now, it wasn't possible to fetch them because NodeBB didn't expose any list that included these threads, but with the latest update, that has changed. As a result, you will no longer see the annoying "This post is being processed/translated..." message indefinitely. For now, it has been tested on some older threads (not all are included since we're running in "low power" mode), but it will be applied to all new ones.

Additionally, I've implemented a small utility I need for server management, which I'm sharing with you in case it comes in handy: visiting extra.hardlimit.com/ip will return your public IP as a plain text string, with no HTML formatting.

[cobito]

Quick update:

• It seems the Nodebb team actually did handle the embedded videos in cross-posts correctly. It was my fault for not enabling an option. Since yesterday, they've been showing up in new threads.
• We've brought on a new intern to process threads: someone named Smith. This is because if HIAL-9000 is both posting and processing the threads, it won't see its own threads during processing since they don't appear in its "unread" list (typical silly oversight that cost me almost an hour yesterday going over the scripts to figure out why HIAL-9000 wouldn't process its own threads ).

[pos_yo]

@cobito It's impressive everything you're implementing just "for fun". Great work, thanks!

[cobito]

Alright, last post of the term:

The end of the term is approaching, and with it, the holiday break, which will be longer than usual this year.

Benchmark

In the end, the benchmark has been left half-finished and will remain in beta for a few months. I want to keep adding features to it before finalizing it.

Basically, new features have been added to the interface along with secondary tools, but the Core remains unmodified despite receiving many changes that haven't been included. There are several decisions I still need to make.

The reason for not releasing more updates is that the program's code was (and to some extent still is) a genuine headache. It's much better now than before, but it's something that has been neglected all this time, unlike the rest of the pages, which have received some attention over the last 5 years. In fact, most of the Core's code was practically the same since the early stable versions (around 10 years ago?).

In short: many hours have been spent on invisible work, but it will help speed up the release of updates from now on.

It's been impossible for me to dedicate time to it in the last few days, so there probably won't be any more betas until after the summer.

Homepage

It's been the great neglected one. After experimenting with video summaries, I've seen that AI is superior to me in understanding, correcting, and condensing concepts. The homepage basically consists of writing a short news summary, which I've done manually until now. And honestly, after seeing that, I've developed an incredible amount of laziness to keep doing that task. It's like having a brand-new excavator with a full tank of fuel, sitting on its tracks, while a worn-out pickaxe and shovel wait for someone to pick them up to dig a trench. What a pain to dig a trench with a pickaxe and shovel! The downside of all this is that one of the homepage's purposes has been to show that there are people behind the scenes maintaining the site. Delegating the cognitive effort of this task to a machine will erase that "symbol" of activity in one stroke. On the other hand, I don't think it's necessary right now to show signs of activity just for the sake of it.

In general, the homepage has been something that hasn't motivated me much for a while, because it's hard to find real news (the world of personal computing is essentially dead), because sensationalism is the norm, and because sometimes you have to verify unlikely (but potentially true) information. It takes a lot of time and effort, and the value it brings is debatable. If a machine can do that job and even do it better than me, why not?

I still have to make the final decision, but it's highly likely that upon my return (sometime in September), the homepage will undergo a radical internal overhaul. It's the only page that hasn't received development hours in the last year, and since it's going to be redone, I think this is the way it will be approached.

Forum

I'm trying to inject some activity into it from the Peertube instance. It seems like the right place to publish text-based content, and in fact, I was considering migrating the posts from my old retro blog to the forum. I was also thinking about creating posts about thematic videos (The Computer Chronicles comes to mind right now) in a new retro subforum, since it's a branch with a lot of momentum at the moment.

In this regard, I'm still on uncharted territory. If social media and aggressive SEO ended the forum era over a decade ago, I'm pondering how a forum fits into the current AI era. Providing content as a starting point for discussions isn't complicated as long as there continues to be content generation on the Peertube instance. Being a Q&A hub remains a possibility for very specific and poorly documented topics. But attracting people to participate directly is a challenge.

We'll see if I can revive some old formulas...

Museum

It's been completely neglected this term. It remains the center of gravity for my interest because it's where I feel most comfortable experimenting without the fear of breaking things. But I can't leave the rest neglected, and besides, one also gets tired of always working on the same thing.

I'd like to shovel content in and push to the limit all the development I did a few months ago to see how it performs at scale, but the lack of storage prevents me from doing so. Still, I have some ideas to squeeze every last byte out of the 2TB allocated to this project.

Infrastructure

We have three urgent needs: more storage, more GPU processing power, and better energy efficiency. These are three machines that, as of today, cost an unaffordable fortune. During the next term, the acquisition of this hardware will depend exclusively on price trends, which puts us at the earliest in 2027.

Meanwhile, the hoarding frenzy by AI scalpers has greatly reduced to the point where, at the moment, we're running at nominal levels for both bandwidth and CPU usage. And to be honest, I'm genuinely curious to know the reasons behind this abrupt drop in activity compared to the momentum AI is currently experiencing, regardless of how beneficial this drop is for us. Knowing where personal computing is heading in the coming months directly depends on what the AI "industry" does, and given how extremely opaque that field is, having these discreet data points always helps a bit to understand the situation.

I don't have anything special planned for the next term: naturally, the benchmark program will be the priority. Essentially, I'll let myself be guided a bit more by whatever I feel like doing at the time (without neglecting anything and with some planning), and it will also depend on how feasible it is to purchase the planned hardware, since we're a bit limited in that regard at the moment.

If I can find some time, I might release a bit more until the first week of July, but not much.

Happy summer!

[cobito]

Last night, the hard drive on the Ryzen 9 9900X failed miserably. That left us without the forum or translations all day. It's been a disaster. I've spent about 10 hours getting everything back up and running, which involved finding a working hard drive, installing the system, and trying to recover the forum's database from the dead drive. It was impossible to recover the database, so I had to fall back on a backup from Saturday. That means any threads and crossposts created since then are lost.

For now, our bots (Smith and HIAL-9000) are offline. I need to get the entire AI infrastructure back online, which will be a slow process, so it'll be ready either today or tomorrow.