Hardlimit Museum

[Sylver]

@cobito very interesting, quite a job. Thank you very much for the contribution, I am sure that many of us will appreciate this material a lot.

Best regards!

[cobito]

Thanks @Sylver. This episode seems to have had some audience so I'll keep looking for the next one.

[cobito]

New chapter: the hard drives of 1985. Very interesting the fear that existed at that time for these units due to their low reliability:

[cobito]

It is rumored that, in exchange for a handful of peanuts, a group of monkeys has been developing a file search engine for the museum from which you can search not only the name of the file but also the content of the files.

Rumors speak of an index that will contain hundreds of gigas distributed among millions of retro files and will be released sometime in February.

[cobito]

The retro file search beta is now available. The search engine not only searches for the file name, but also the content of the files. It also searches within containers and compressed files, including the content of said files recursively. In PE and COFF format binaries, the search is also carried out between the sections of the file. If the file contains any image or video, a thumbnail of the image/video will appear. In the advanced search you have all the available options. At the moment, the search is carried out between 979 GB spread across 5.5 million retro files. The search is slower than I would like, but we will work on that. If you find what you are looking for and it is on the CDROM or floppy disk of some magazine, you can download the disk to access the file. Remember that veterans can run the TDC games from the browser from the VIP area if what you are looking for is there. The indexing is taking longer than initially planned, so we are going to premiere a test version so that those who are interested can use it and suggest changes, report bugs and so on. For now you can search among all the CD-ROMs and floppy disks of PCManía and the games up to 1994 (inclusive) of the Total Dos Collection, which is practically all the DOS games that have existed up to that date. What interests me the most at the moment is that you make suggestions/critics about how the results are presented but if you see that the advanced search is messy or some kind of explanation is necessary, do not hesitate to comment on it. Of course, bugs and additional functionalities are also welcome. In short, any comment that makes the search engine more friendly and useful is appreciated. There is no link to the search engine from the museum's front page nor is the topic being moved by RRSS and such. For the moment, do not share links because the URL format could change. This is an exclusive version for forum readers. Every one or two weeks the index will be updated and I will put a message here with the update. You can access the search engine from [disabled] EDITED: As it seems that it is not being used much, I disable it to have more resources for the indexing. When the index is completely generated, the official launch will be made. In any case, if someone wants to test it, they can send me a pm to re-enable it.

[cobito]

This time, we go to 91 to delve into the exciting world of file compression, with options for all tastes. We can see Mr. PK showing off his PKZIP, hardware acceleration in file compression and digital video editing still in its infancy:

[Fassou]

I miss the ICE progress bars

[_Neptunno_]

@fassou I think I know what you mean but just in case... ICE progress bars?

Best regards!!

[Fassou]

@_neptunno_ A file compressor, which disappeared in favor of others that were more configurable and flexible like ARJ or LHA

Salu2!

[_Neptunno_]

@fassou thanks for the clarification, I didn't know that kind of compression format

ICE: Ergonomic Compression Interface.

Greetings!!

[cobito]

The "videoteca" section has been created in the museum's library, where the subtitled videos have been compiled along with an introduction.

As for the file search engine, it is almost finished with the TDC. After that, the CDROMs of all the magazines except PCManía will be left, and after that, the official launch will take place. For now, there are 2.1TB distributed in more than 9 million files.

[Fassou]

@cobito said in Museo Hardlimit:

For now, there are 2.1TB distributed in more than 9 million files.

9 million files...

[cobito]

I bring you another chapter, this time about artificial intelligence. In 1984 this technology was still in its infancy and the programs that came out were quite simple, but they were the first steps. It was based exclusively on logical tests and was very far from what is seen today. But that's where it all began. We see a couple of representative characters, like the creator of Lisp.

[cobito]

Well, I think the file search engine has reached a sufficiently mature level. In the end, it didn't turn out as I had imagined because of the enormous size it has reached.

It can be accessed from here.

Only the CD-ROMs of PCManía and Total DOS Collection have been indexed. I think it's an acceptable selection of programs, tools, drivers and games.

The search results are divided into these sections:
· Thumbnail: a thumbnail image of the file, if available.
· Location: places where the file was found. From here, you can navigate the directory tree using the museum's file explorer.
· Downloads: options to download the file. Normally, you will have to download a larger file (an iso, a zip, etc.) to get to the file you are looking for.
· Content: a snippet of the ascii content of the file. In binaries, you will often see meaningless content mixed with character strings.
· Other details: Size, MIME, date, specific details and an MD5 hash.

One thing to keep in mind is that the search time is quite high. It's a chore, but for now I can't do much to improve it.

If I see that the topic has moderate success, I will look for ways to expand the content and improve the speed.

[Sylver]

@cobito Many thanks for the piece of work, mayor

A question, the elevated search time ¿is it due to the storage medium used? ¿Or is it a more complex issue? To think of ideas that could help...

Best regards!

[cobito]

@sylver The main reason is that the server has been busier than usual for a few weeks now because of the video section.

The index is quite large and resides entirely in RAM. When there are no transcoders running, the wait time is acceptable, but when there are a couple of them, it becomes too slow. Looking at the behavior of the processes when there are one or the other, what I see is that there is a lack of bandwidth to memory.

I think this is one of the cases where DDR5 (despite its high latencies) would come in handy, but it will have to wait.

[Sylver]

@cobito I understand... Thank you very much for the explanation.

Maybe @jordiqui could donate one of his almighty servers to the cause...

Best regards!

[Yorus]

What a marvel!. I don't even want to imagine the amount of time it takes you to do all these things.

As for the necessary hardware, I imagine that at a certain point it is difficult for a single person to handle, whether it's the level of hardware needed, bandwidth, power consumption, noise...

[cobito]

Yes, in the end the hardware is limiting. I had to stop indexing due to lack of RAM but if I had more, the speed would be even worse, so with DDR3 in particular or a conventional machine in general, you can't scratch much more.

Thinking about it a bit when I've come across the limits during development and indexing, I think a "rig" of Raspberry Pis would be a viable option. Something like that would be relatively inexpensive, relatively simple and very scalable. But of course, there is always a significant initial outlay that has to be justified by intensive use of the tool.

For now, let's see how it's received because in the end it still has a niche audience.

[jordiqui]

@sylver Don't doubt it - If needed, I grant remote access and voila. I just read it, sorry for the delay.