I got a reasonable amount of work done on the train, I still have a few bits and bobs to do to get it working though. I might get it compiling on the way home, we'll see.
So here's how it works...
Each socket has it's own WSAEVENT associated with it, and I request notification of socket read, write and close events. I have a thread for ever 63 clients, which sits in a WSAWaitForMultipleEvents on all the socket events associated with it, plus one signal event.
When I want to add a socket to the thread, remove a socket from the thread, or terminate the thread, I post an event in the threads message queue (Which is currently just a std::list), and signal that event. The thread picks up on the signalled event, and responds appropriately.
There's another thread for accepting and connecting sockets. It waits on up to 62 connecting sockets, plus the signal event, plus the listening socket's event. For adding or removing sockets and terminating, it behaves just like the other worker threads. When the listen socket is signalled, it allocates a new socket, sets it up, and then finds a thread to stuff it in, by looping through the list of worker threads and checking if there's room for it. If it can't find one, then it creates a new thread and adds it to the list.
After it's added the socket to a thread (and possible spawned a new thread), it sorts the thread list (The thread list is just a std::vector), so the next loop through to add a socket will choose a thread that has fewer sockets in it, which should help with load balancing a bit.
When CSocket::Release() is called, the socket manager posts an event telling the thread associated with the socket to remove the socket from its just. If the socket is disconnected, then the thread just flags the CSocket as disconnected, causing the client code to call CSocket::Release(). And finally, when the thread gets a signal to remove a socket, it removes it from its list, kills the socket, and sorts the thread list again.
If the thread list has more than 1 empty thread (I need to modify the code so it checks how many threads can optimally be used, so I don't end up with 20 threads with 1 socket in them), then the code will start culling dead threads.
That was an utterly rubbish explanation. I'll post the code once I get it all working so you can all shout at me about how terrible it is, how many bugs there are in it, and how I should just use RakNet or something.
The main reason I don't want to use another network library is because I want to be able to swap it out for different code if I need, and that would mean making a wrapper around the socket code. If I write it myself, I can just make sure I stick to the same interface and I'll be fine.
