After having reimplemented most of my C# Z80 emulator in C, I’m now trying to figure out why every ZEXALL test is failing:

james@jameslaptop:~/Projects/Emulation/ZZ80E$ ./a.out z80/zexall_sdsc.sms
Z80 instruction exerciser

ld hl,(nnnn)................. CRC:350e5018 expected:5f972487
ld sp,(nnnn)................. CRC:45f3041a expected:7acea11b
ld (nnnn),hl................. CRC:c9f9ffd8 expected:a3608b47
...etc...

My first guess was that there was a bug in one of the opcodes used to calculate the CRC because I know for sure that the ld opcodes are fine.

However after going through the list of opcodes involved in the CRC with a toothpick, I still can’t find any glaring or even minor mistakes :( The problem with emulator debugging, especially at such an early stage, is that it’s hard to tell if you’re looking in the right place; the bug might be in a seemingly unrelated piece of code, the effects of which have simply cascaded down the chain.

It’s going to be a long night.

I’m happy to brag that in my first year of university, I have received the following grades (pending ratification):

Computer Architecture A
Software Development I A
Software Development II A
Introduction to Databases A-
Mathematical Methods 1 A
Patterns of Problem Solving A-
Basic Mathematical Techniques A
Computational Mathematics A

Not bad. :)

Well, this is something I’ve always wanted to learn more about; how operating systems work and how to write one of my own, and after a little light reading of Tanenbaum’s “Modern Operating System” I decided to give it a try. It’s pretty much the hardest thing I’ve ever attempted to do, and I haven’t even got an OS per se, just a program which can print out text to the screen after the BIOS has booted. Here’s a boring screenshot of it running in QEMU:

This Windows-killer took me the better half of a week to figure out, I’m sure I’ll have a fully functioning GUI, Nvdia drivers and a TCP stack by next Tuesday ;)

I’ve had to pick up a bit of x86 assembler on the way, with hopefully a lot more coming. A few of the functions look like this:

; Initial entry point
[extern kmain]
[global load]
load:
    mov esp, stack_end
    push eax
    push ebx
    call kmain
    cli

hang:
    hlt
    jmp hang

; Returns CPU details
[global get_cpuid]
get_cpuid:
    push ebp				; save the base pointer
    mov ebp, esp			; move the current stack pointer to ebp
    mov eax, [ebp+8]		; get first parameter (mode)
    mov ebx, [ebp+12]		; get second parameter (pointer to string)
    push ebx
    cpuid
    pop eax
    mov [eax], ebx
    mov [eax + 4], edx
    mov [eax + 8], ecx
    leave
    ret

[global is_in_protected_mode]
is_in_protected_mode:
	xor eax, eax
	smsw ax
	and ax, 1
	ret

call kmain transfers control over to a C function which inits the rest of the (insignificant) kernel.

void kmain(void * mbd, unsigned int magic)
{
	gfx_init();
    gfx_clear();

    if(magic != 0x2badb002) {
		oops("Invalid magic number: %d", magic);
    }

    kprintf("Booting ZoofOS...\n");

    char cpuid[20];
    get_cpuid(0, cpuid);

	int mode = is_in_protected_mode();

    kprintf("CPU is a %s, and is running in %s mode.\n", cpuid, (mode) ? "protected" : "real" );
}

The most interesting thing with kernel development is that you have to do everything from scratch, no standard libraries, no nothing.

ZoofOS isn’t much right now and never will be, but it will be an incredible learning experience.

Today I have started the momentous task of re-writing my Z80 emulator in C. It would have been possible to simply convert much of the C# code to C, but since I have learnt so much since I started writing it almost a year ago, I have decided to completely overhaul it.

Progress has been remarkably quick, with over a hundred of the most used opcodes implemented so far. One problem with the C# version was that as I wanted a built in disassembler and thought it would be cleaner, I had to have an individual method for each opcode which added the opcode’s mnemonic to the debug output. This time around, I have placed the vast majority of the opcode code in the switch statement, with common parts of code outsourced to inline functions. This has made the code a lot cleaner and manageable. To handle disassembly, I have created a Python script to convert a list of all the Z80 opcodes and their mnemonics to a C function which contains a huge switch statement to return the mnemonic. It might not be the fastest method, but it’s adequate for debugging.

This has been my first experience with C, and so far, it’s been a good one, I love the low level sexiness of it all. Would implementing a C standard library for Master System development using SDCC be too ambitious? :P

Well, I’ve just spent way more time than necessary debugging an issue I had with a TreeView not being correctly updated and freezing the whole application.

Unlike WPF (and also I think Winforms), Gtk# does not complain when a GUI object is accessed or modified from a separate thread, so I incorrectly assumed that the issue wasn’t related to cross thread safety. As it turns out, my assumption was wrong (they usually are ;) ).

In order to alter Gtk object from a separate thread, a method similar to the WPF Dispatcher can be used:

Gtk.Application.Invoke(delegate {
      randomGtkWindow.Title = "blah";
});

Now, if we don’t invoke on the original Gtk thread, sometimes it will work, sometimes it won’t, with quite disastrous results. From what I’ve experienced so far, generally no error is thrown and if it is, then it’ll be quite cryptic.

Maybe the Gtk# developers should cause Gtk objects to throw an exception when accessed in a different thread?

Well; after exclusivity using Windows for the past two years, I have decided to go back to Linux land for awhile and try out Kubuntu. I’ve used Linux on and off now for almost as long as I can remember (since Mandrake 6 or 7 I think, which Wikipedia tells me were around 2000), I had Fedora (up to version 4 or 5) as my primary OS for quite awhile before moving to Gentoo for probably a year, which was by far the best Linux OS I had used up until now.

I had honestly, never expected Linux to ever be ready for the desktop, yet maybe now after almost two decades I think it nearly is. After using Kubuntu for two days, I’m very impressed at how far it has come. Setup was a snap, KDE looks pretty (I used to only use GNOME), it feels solid and everything is nicely abstracted away from the command line, though obviously, all the flexibility and control which Linux provides is still available under the surface. I’m a geek and as such I don’t mind getting my hands dirty compiling sources, screwing with grub or rolling my own kernel, but sometimes it’s nice for things to just work without any hassle, especially when you have other things to focus on.

All it needs now is for Microsoft itself to port .NET to Linux and not Novell :P *ducks*

The matrix calculator has been updated to include support for finding the inverse and determinants for square matrices of arbitrary dimensions.

http://zoofware.com/zoofware-matrix-machine/

I’ve come across a rather interesting feature of IronPython and the DLR (forgive me if this is old news :P ) which I never previously knew about and would make an interesting addition to my planned Facebook XMPP client; the ability to execute embedded IronPython from any .NET language (even IronPython itself).

using System;
using System.Collections.Generic;
using IronPython.Hosting;
using Microsoft.Scripting.Hosting;

namespace ConsoleApplication3
{
	class Program
	{
		static void Main(string[] args)
		{
			Dictionary<int, int> bar = new Dictionary<int, int>();

			ScriptEngine engine = Python.CreateEngine();
			ScriptScope scope = engine.CreateScope();
			scope.SetVariable("bar", bar);

			string script = @"

from System import Console

foo = 'Hello from IronPython!'
Console.Title = foo

for i in range(0, 10):
	bar.Add(i, i**2)";

			engine.Execute(script, scope);
			Console.WriteLine(scope.GetVariable("foo"));

			foreach (KeyValuePair<int, int> kvp in bar)
			{
				Console.WriteLine("{0}^2 = {1}", kvp.Key, kvp.Value);
			}

			Console.ReadKey();
		}
	}
}

This is a great way to allow custom user defined scripts to alter a program without changing the actual code base. The embedded Python has complete access to the .NET library, likewise the calling C# has complete access to the embedded script.

I can’t see much practical use for this in a chat client, but I’m working on it :P

As the title suggests this is a quick guide to the authentication mechanism DIGEST-MD5 as used in XMPP. Digest authentication is used in other protocols also, but they are outside of the scope of this article.

Once an XMPP stream has been established and the DIGEST-MD5 SASL mechanism selected, the host should respond with a challenge that looks something like (when decoded from base-64):

realm="zoofware.com",nonce="0ddba11",qop="auth",charset=utf-8,algorithm=md5-sess

Now, the correct response would be:

username="jwb",realm="zoofware.com",nonce="0ddba11",cnonce="39931f36af066660f551
6142cbf02767",nc=00000001,qop=auth,digest-uri="xmpp/zoofware.com",charset=utf-8,
response=3353792a75fcb3337560e2a33041b3fa

Creating most of the response is incredibly simple, it’s simply a string composed of the following attributes:

  • username – The username part of the user’s JID (i.e jwb@zoofware.com)
  • realm – This should be the same realm as sent in the challenge, which should be the same as the domain part of the user’s JID.
  • nonce – Essentially an arbitrary string which is sent by the server in the challenge. It is for security.
  • cnonce – Another string, but this time generated by the client. It doesn’t matter what it is. Again for security.
  • nc – An incrementing counter to identify each request. Generally there is no need to change this from “00000001″.
  • qop – For XMPP, this will always be “auth”.
  • digest-uri – Simply the realm appended to the string “xmpp/”.
  • charset – The encoding to use.
  • The “response” directive is the hardest to generate; it is a hash based on several other hashes of passwords and so on and can be done in several steps. I have included my specific client C# code in the example.

    The following example uses the following directives:

    realm=”zoofware.com”
    username=”jwb”
    nonce=”0ddba11″
    password=”secret”
    cnonce=”randomz”
    digest-uri=”xmpp/zoofware.com”

    Step 1

    Create a string of the form "username:realm:password".
    Save this string as a byte array called A1.

    byte[] A1 = ASCIIEncoding.UTF8.GetBytes(string.Format("{0}:{1}:{2}", Xmpp.UserName, Xmpp.UserDomain, Xmpp.Password));

    Now compute the MD5 hash of A1 and keep the result in a byte array HA1_1.

    MD5 md5 = MD5.Create();
    byte[] HA1_1 = md5.ComputeHash(A1);

    .

    The result so far should be HA1_1 = afaab39bafa10040c5d165030c03e278.

    Step 2

    Create a string of the form ":nonce:cnonce" and save it as a byte array HA1_2.

    byte[] HA1_2 = ASCIIEncoding.UTF8.GetBytes(string.Format(":{0}:{1}", elements["nonce"], cnonce));

    .

    Create a byte array HA1 which has a length equal to the length of HA1_1 + HA1_2. Then copy the cotents of HA1_1 and HA1_2 into the new array.

    byte[] HA1 = new byte[HA1_2.Length + HA1_1.Length];
    HA1_1.CopyTo(HA1, 0);
    HA1_2.CopyTo(HA1, HA1_1.Length);

    Compute the MD5 of HA1 and call it HA1.

    HA1 = md5.ComputeHash(HA1);

    .

    The result so far should be HA1 = ef92f7f90f8bd0a364f01cfe9e9b0564.

    Step 3

    Create a byte array HA2 with the contents of the MD5 hash of the string “AUTHENTICATE:digest-uri”.

    byte[] HA2 = md5.ComputeHash(ASCIIEncoding.UTF8.GetBytes(string.Format("AUTHENTICATE:xmpp/{0}", Xmpp.UserDomain)));

    HA2 = c51d568aff0c3f543bdfbba2a6010a75.

    Step 4

    Now create two strings sHA1 and sHA2. Their contents need to be the base 16 representation of HA1 and HA2.

    Important: Do not just convert each byte to it’s ASCII letter. The string needs to be its literal value.

    For example, the byte array { 0x1a, 0x2e, 0x3f } would become the string 1a2e3f.

    public static string ConvertToHexString(byte[] inArray)
    {
         StringBuilder sb = new StringBuilder();
    
         foreach (byte part in inArray)
         {
             sb.AppendFormat("{0:x2}", part);
         }
    
         return sb.ToString();
    }

    Then compute the MD5 hash of the string “sHA1:nonce:nc:cnonce:qop:sHA2″. Save the result as the byte array hash.

    So far: hash = 6dd79d3dee313216e188bfc17b6254b5.

    byte[] b = ASCIIEncoding.ASCII.GetBytes(string.Format("{0}:{1}:{2}:{3}:{4}:{5}", ConvertToHexString(HA1), elements["nonce"], "00000001", cnonce, elements["qop"], ConvertToHexString(HA2)));
    
    byte[] hash = md5.ComputeHash(b);

    Final Step

    pheww!

    The base 16 representation of hash can now be used as the value for the response directive.

    Thus the final response will be:

    username="jwb",realm="zoofware.com",nonce="0ddba11",cnonce="randomz",nc=00000001
    ,qop=auth,digest-uri="xmpp/zoofware.com",charset=utf-8,response=6dd79d3dee313216
    e188bfc17b6254b5
    .

    Note: I’m not experienced with DIGEST-MD5 at all, so this solution my be incomplete for some situations. It does however work with the XMPP hosts I’ve tried it with.

    Further reading:

  • http://en.wikipedia.org/wiki/Digest_access_authentication
  • http://www.ietf.org/rfc/rfc2831.txt
  • http://web.archive.org/web/20050224191820/http://cataclysm.cx/wip/digest-md5-crash.html
  • http://www.charliedigital.com/CategoryView,category,XMPP.aspx
  • Over the weekend I’ve been busy writing an XMPP client from scratch. So far, I’ve got a basic console application connected to Facebook chat. No sending messages support yet, but the friends list, receiving messages and presence data are working quite well.

    Its also connected successfully to Google Chat or Talk or whatever it’s called using TSL.

    By far the hardest thing to implement was the digest authentication (DIGEST-MD5) mechanism used by Facebook (and many other hosts, Google supports PLAIN). The C# code for creating the response hash looks something like this:

    MD5 md5 = MD5.Create();
    
    byte[] a = ASCIIEncoding.ASCII.GetBytes(string.Format("{0}:{1}:{2}", Xmpp.UserName, Xmpp.UserDomain, Xmpp.Password));
    byte[] HA1_1 = md5.ComputeHash(a);
    byte[] HA1_2 = ASCIIEncoding.ASCII.GetBytes(string.Format(":{0}:{1}", elements["nonce"], cnonce));
    byte[] HA1 = new byte[HA1_2.Length + HA1_1.Length];
    
    HA1_1.CopyTo(HA1, 0);
    HA1_2.CopyTo(HA1, HA1_1.Length);
    
    HA1 = md5.ComputeHash(HA1);
    
    byte[] HA2 = Md5.ComputeHash(ASCIIEncoding.ASCII.GetBytes(string.Format("AUTHENTICATE:xmpp/{0}",xmpp.UserDomain)));
    
    byte[] b = ASCIIEncoding.ASCII.GetBytes(string.Format("{0}:{1}:{2}:{3}:{4}:{5}", ConvertToHexString(HA1), elements["nonce"], "00000001", cnonce, elements["qop"], ConvertToHexString(HA2)));
    
    byte[] hash = md5.ComputeHash(b);
    string res = ConvertToHexString(hash);

    And the code to convert an array of bytes to a base 16 string:

    public static string ConvertToHexString(byte[] inArray)
    {
    	StringBuilder sb = new StringBuilder();
    	foreach (byte part in inArray)
            {
            	sb.AppendFormat("{0:x2}", part);
            }
    
            return sb.ToString();
    }

    In the next few days I’ll write up a brief DIGEST-MD5 tutorial as I wasted quite a bit of time trying to piece it together from various resources and RFCs.