MSX AcidTests

Emulators are now undeniably important development tools for the MSX community, and openMSX plays a major role on this scenario.

Also, keeping strict MSX cross-compatibility was/is always a problem, just like Internet browsers cross-compatibility.

The motivation

Testing software on a lot of different hardware is both time consuming, costly and ineffective. Akin, for a designer to test its pages on as many browsers as possible is both time consuming, costly and ineffective.

To make things worse, unfortunately it seems that the recently created community software is getting farther and farther of the MSX concept, by ignoring the BIOS completely and going to direct hardware access in a way that isn't compliant to the standard.

Comparing the MSX standard with the Internet HTML standard, the creation of the Web Acid tests helped to create a standardized way to assure that the interoperability between browsers could be quickly tested, exercising the specification to the maximum extend:
http://en.wikipedia.org/wiki/Acid1
http://en.wikipedia.org/wiki/Acid2
http://en.wikipedia.org/wiki/Acid3

The philosophy behind the web AcidTests is: "As with acid tests for gold which produce a quick and obvious assessment of the quality of a piece of metal, the web acid tests were designed to produce a clear indication of a browser's compliance to web standards."

The same philosophy can be applied to the MSX standard. I believe that the reason why many MSX programmers don't follow the coding recommendations is because it "just works" on their simple testing setup, exactly as it happened with the browsers before the AcidTests were created: programmers just tweaked their code to work under the limited testing environment they had.

As the MSX standard is all about expansion, the respect this same standard allows the hardware to evolve better and in a cheaper way, even in a hobbyist scenario like we have today. As the MSX community has very limited resources, there should be a way to assure that the most common software compatibility/interoperability errors are quickly detected.

For example, if someone was seeking to develop a new MSX motherboard these days, there would be two options for connecting new keyboard and mouses to it:

1) Very expensive hardware translations

Keyboard and mouse would need to be hardware-translated somehow. This can be done by either:

a) An FPGA
b) External microcontroller, like a PIC. Many solutions use one PIC for keyboard translation and another for the mouse.

Both solutions result in a very complex an expensive motherboard, with many separate buses going on and a lot of processing power being waste on parallel inaccessible CPUs. It's also very hard for a programmer to detect and fix bugs in an architecture like this.

2) Simply connecting a cheap i8042 clone directly to the MSX bus

The original IBM PC (the one that came before the PC XT) also used an i8255 for reading the keyboard, in a way very similar to the MSX. The MSX standard allows it to easily follow the same way: The keyboard-reading responsibility would be shifted from the i8255 to a new i8042 clone place on board. The BIOS would the be modified to receive keyboard scans from the i8042 and update the OLDKEY/NEWKEY tables accordingly.

This would be a much simpler and cheaper motherboard, allowing the design to focus the speed on the main processor.

How it began

I decided to create some "virtual" machines for openMSX, to help my compliance tests, and I'm sharing those machines here for other developers who might get interested on testing their software against the worst possible variations of the MSX standard. If your software champion to run on these bad-ass machines, it has a highly degree of probability of running on the plain-vanilla hardware out there.

What each machine does test?

Each "machine" will stresstest some specific aspect of the standard to assure that it's being respected. As for the web AcidTests, only well behaved software will run successfully. Bad players are expected to fail miserably.

The machines were grouped according to aspect and to a certain focus, to make debugging easier:

Available machines:
Acid1Test-slots: Slot layout test (aka "usual hardware in unusual slot disposition")
Acid2Test-hardware: Illegal direct hardware access (aka "unusual hardware in usual slot disposition")

Machines on drawing board:
Acid?Test-BIOS: Incorrect BIOS handling (aka "usual hardware, incorrect BIOS calling")
Acid?Test-interChip: Inter-chip timing Test (aka "usual hardware, incorrect I/O timing")
Acid?Test-speed: Speed test (aka "Turbo compliance")
Acid?Test-commonMistakes: Common Mistakes test (aka "everybody slips at those")

I'll publish each machine release and their respective status below:

Machine	Description	Status
Acid1Test-slots	Stress-test the slot accesses. Incorrect assumptions about memory/resources layout will fail to run on this machine. Software must also handle extended slots correctly to run.	Current version: v0.3 Completeness: 100% ROMs required: Sanyo PHC-70FD, CIEL ExpertTurbo, MSX-DOS v2.2
Acid2Test -hardware	This is certainly the hardest of all tests. It's aimed to make software that does illegal direct I/O to fail. Please note that legal VDP direct I/O will not fail. The MSX Technical Handbook, pg-336 clearly states that programs should not handle the hardware directly and the only direct I/O allowed is to the VDP, by using the ports indicated on addresses 0006h and 0007h of the BIOS. Pg-42 of this same book, on "Notes on I/O Address Assignments", mentions the reason: "Although I/O addresses are defined above, the software must not access those devices directly using the above ports. All I/O accesses must be done using BIOS calls, in order to make the software independent of hardware differences. MSX manufacturers may change some of the hardware from the standard MSX system and maintain software compatibility by rewriting BIOS. The hardware differences would thus be transparent to the software." In its current state of completeness, this machine can be used to test a lot of MSX software, but unfortunately it will not yet test the following features: - Software that uses the MemoryMapper - Software that do illegal direct sub-slot switching by accessing the FFFFh register	Current version: v0.2 Completeness: 70%. ROMs required: Sanyo PHC-70FD2, CIEL ExpertTurbo, standalone MSX-Audio v1.0 (32KB), MSX-DOS v2.2 The features below are missing because emulators lack support for it: - Not implemented: Expanded slot registers at nonstandard address - Not implemented: MemoryMapper registers on nonstandard I/O ports The following features are already implemented: - PPI8255 on nonstandard I/O port - V9958 at 88h I/O port - PSG on nonstandard I/O ports - RTC on nonstandard I/O port - PrinterPort at nonstandard I/O port - Built-in MSX-Music (YM2413) on nonstandard I/O ports - Built-in MSX-Audio (Y8950) at nonstandard I/O port - Built-in MSX-DOS2
Acid?Test-BIOS	Stress-test incorrect BIOS calls. It's aimed to catch programs that do direct calls to the routines instead of using the jumptables.	Current version: none Completeness: 0%
Acid?Test-InterChip	This one assures that the software has proper timing and don't rely blindly on software-loops for temporization. It assures that turbo machines will be supported correctly.	Current version: none Completeness: 0% Note: openMSX seems to be implementing this as helper scripts instead of a machine
Acid?Test-SpeedTest	This one assures that the software has proper timing and don't rely blindly on software-loops for temporization. It assures that turbo machines will be supported correctly.	Current version: none Completeness: 0% Note: Emulators lack support for creating a machine with the features below: a) Z80 running at 40MHz b) V9958 blitter working 100% faster c) The machine has a built-in SystemTimer, without S1990 (like the OCM) d) There's no way to disable the turbo, which is a common workaround used by programmers (if the turbo could be disabled this test would have no meaning at all: the idea is to test if the algorithms are ok) But some of it's features can be manually enabled on openMSX using any machine you want. I advice you to select a Sanyo Wavy 70FD, pop the openMSX console and type the following commands: set_z80_freq 40000000 set_z80_freq_locked off set cmdtiming broken After you finish debugging, don't forget to return the Z80 to its usual speed by typing: set_z80_freq_locked on
Acid?Test-commonMistakes	Machine or set of scripts to test for common programing mistakes as well as for known MSX quirks (as nobody would like to implement many of those quirks on the emulators), like: a) VDP register access with interrupts enabled b) Sending new blitter commands without checking if the previous one finished c) Using VDP_R#18 in combination with the blitter corrupts VRAM d) screensplits+VDP_R#18+sprites_disabled causes problems e) Overwriting the VDP_R#25 bit2 value (may cause VRAM corruption on some machines)	Current version: none Completeness: 0% No emulator has ready-to-use support for these yet. Some features can be achieved on openMSX by using watchpoints.

Stamps

If your software passes on the MSX Acid Tests, you can use the respective test stamp (on your website, on the game cover/manual, etc) for free, as a certification that it passes the test.

Big stamp (GIMP file, 350x350, can be scaled down to 64x64)
Small stamp (GIMP file, 224x224, can be scaled down to 32x32)

FAQ

Q: How do I install/use those machines?
A: To install, just unzip their files under the share/machines directory located under openMSX directory structure. You'll also need to provide the required ROMs under the share/systemroms directory. After that, on Windows, you must command openMSX-Catapult to rescan for installed machines so they will be shown on the machines list. On Mac OS-X, you just need to quit NekoLauncher and open it again. To use the machines, just select them as any other available machine and run the intented software.

Q: Why did you created those machines?
A: I created them for speeding up my debugging tests and assure a higher degree of compliance with the standard for my software. Then I decided to share those machines and the idea with the MSX community. They're free for you to use at your own risk. No warranty or support is assured at all.

Q: Why would anyone want to run their software on machines that don't exist?
A: Because they try to unite weird characteristics of a lot of machines in a single test. This way you don't have to own a lot of different hardware just to assure that your software runs on every possible configuration

Q: Is the Acid2Test-hardware machine really compliant to the MSX standard?!? I always thought that the MSX I/O addresses were standardized!
A: Yes, the MSX I/O addresses are standarized, but that info was meant only for hardware makers, not for software programmers. The clear intention of that information being disclosed was for hardware designers to avoid creating hardware with conflicting I/O ports. For software programmers, the MSX Technical Handbook clearly states that "Although I/O addresses are defined above, the software must not access those devices directly using the above ports. All I/O accesses must be done using BIOS calls, in order to make the software independent of hardware differences. MSX manufacturers may change some of the hardware from the standard MSX system and maintain software compatibility by rewriting BIOS. The hardware differences would thus be transparent to the software". This assertion also means that the Acid2Test complies with the standard by having modified I/O ports and the rewritten BIOS to maintain the software compatibility.

Q: Why some AcidTest machines requires so many ROMs?
A: That's because some AcidTest machines are a mosaic of other existing machines. Sometimes to make the test harder, sometimes because some parts of a specific existing machine allowed the test to be created more easily. Also the Acid2Test-hardware required all ROMs that would be dependent on the hardware modifications

Q: Is it mandatory to test software on those machines?
A: No, there isn't such a thing. Being able to make your software run on those AcidTest machines is more like an achievement. You certainly must have an excellent knowledge of the MSX standard for being able to achieve that.

Q: Why there are so many unfinished AcidTest machines?
A: Some are still on the drawing board, some have their specs completed, but emulators lack support for them to be implemented

Q: Can those machines be used with blueMSX?
A: Unfortunately, no. Those use an specific openMSX syntax. I'm not used to blueMSX as there's no Mac version, but the same concept probably can be applied to this emulator. If you want to port the AcidTest machines to blueMSX, it will be welcome.

Q: Don't you think that it's impossible to make good games that pass the Acid2Test-hardware? Doing I/O through the BIOS is just too slow!
A: Nearly all Konami games passes both Acid1Test and Acid2Test with flying colors, including Space Manbow (except, in this case, for a minor PSG music issue). This certainly proves that excelent action games can be developed without direct I/O access.