At this point some of you may be thinking "ah, but we have Justin Fletchers EasySockets module, so why would we want to know about the complexities it hides from us?" Well Justins module is great for a programmer who wishes to knock up something quickly and easily and it does suffice for a large number of applications. However there are times when you need more control and also EasySocket's reliance on TCP (as opposed to offering UDP support) can cause problems for certain classes of application, most notably games. In this series I intend to cover both methods of communication while demonstrating some of the advantages and disadvantages of each approach.
The overall aim of this series is to provide a tutorial on the necessary features of network programming while building up a library to simplify development of network applications. The library will offer more flexibility than EasySockets but with the drawback that using it will require more effort from the programmer. Note that it will still be easier to use than the raw sockets interface as offered by various modules and libraries for RISC OS.
Before we get down to business I should make a few things clear. Firstly this series assumes that we will build our library in C as this allows us to make use of libraries such as Netlib. Netlib provides a berkley sockets style veneer over the RISC OS SWI calls. This is important as I do not have a networked RISC OS machine and thus must test code on a Windows box. However there are advantages to this, the must obvious being that the code will be relatively portable between different platforms. Indeed I hope to make the library compile on both RISC OS and Windows.
Secondly if possible I hope to provide the facility to compile the RISC OS version into a module in order to allow access from all languages. I have to admit though that I have no real experience in writing RISC OS modules.
Finally, depending on how successful these articles are and how much time I can spare it may be possible to investigate marshalling or "how to ensure your data can be read at the other end" and also examine a cross platform case study. I do have one in mind which would be very topical to the RISC OS market but I make no promises.
Enough of the preamble. The rest of this article is dedicated to giving an overview of network programming, discussing various issues such as addressing schemes and byte ordering before finishing with a section on what features we would like our simplified library to have. The following articles will discuss sockets on a more detailed level, implementing the library on top as we go.
Sockets Overview
This section provides an overview of the sockets network programming interface. More specifically it discusses the characteristics of the protocols of interest, how sockets differentiates between them, how network addresses are represented and miscellaneous (but important) issues such as byte ordering.
The sockets interface provides an abstraction over a raw network. Note it would not be correct to say that it provides an abstraction over an IP network as this may not be the case (Winsock 2 is not so much a protocol as an interface to a network service of somekind). You can think of a socket as a kind of file descriptor for networks and in this case we do specifically mean an IP network.
An IP network is a collection of endpoints each with a unique four byte address. Therefore, in order to send some data to a specific endpoint we must first know its address. There are several ways of discovering the address of an endpoint and I'll cover some of these as the series progresses. For now let's just assume we know the address of the endpoint we wish to converse with.
Knowing the address of an endpoint is only half of the story. Because an endpoint may be running multiple tasks each using a single network connection we require a method of selecting the task that we wish to transmit to. To support this sockets also has an abstraction known as a port. A port is a number between 0 and 65535 and is used to route incoming traffic to the correct task. One way of picturing this is to think of the IP address as the main telephone switchboard in a company and the port as the specific extension number.
There are a few issues concerning ports. The first is that any ports numbered less than 1024 are reservered for system usage. Secondly a large number of ports are also reserved for common services such as web servers, FTP and telnet. Ideally you should not use these port numbers and we will see that unless you are writing a server of somekind then you can let the system select an appropriate port number for your application.
Once we have an IP address and a port number we need to choose a network protocol which will do the job of transferring our data. Traditionally most people immediately think of TCP but there are a few choices available, each with its advantages and drawbacks.
Protocols
The most basic protocol is that of IP (Internet Protocol) itself. Although it is possible to transfer data directly using IP it is uncommon to do so. IP provides an unordered, unreliable service for packet delivery. In general the abstraction provided is rather low-level for our uses.
The next step up from IP is UDP (User Datagram Protocol). UDP offers a similar service to IP in that it is unconnected, unordered and unreliable. Normally it is used for single shot enquires where it is not critical that the recipient actually receives the message. Any form of reliability has to be built on top of the service which requires more effort on the part of the developer. However it is currently probably the best protocol for fast multiplayer games as we will see during the discussion of TCP.
TCP (or Transmission Control Protocol) is the most common means of transferring data across an IP network. it provides a connection based, fully ordered and reliable delivery service with congestion control. Because of this it requires the least effort to use but as with any high level abstraction you lose a certain amount of control over what goes on.
If you are simply writing an application which requires the reliable transmission of data between two endpoints you cannot go far wrong with TCP. However, there are several applications (most notably games) where TCP is a hindrance rather than a help.
The first problem is that of the totally ordered semantics. Because data is only received by the application in the order in which it was sent it becomes impossible to do any kind of out of order processing. While this may be seem to be an esoteric requirement it is actually very important in multimedia based applications and games. In multimedia applications most data has a useful lifetime. If the data is received after its life has expired then it must be discarded. However the fact that no other data can be received until after the late packet has arrived will probably cause later packets to arrive late and thus be useless. Alternatively a game like Quake will broadcast the game state at regular intervals. If the state does not arrive then you want to receive the next update as quickly as possible - again the data has an limited useful lifetime.
The second problem, again related to game and multimedia traffic, is TCPs congestion control. TCP can detect when a network becomes congested because packets are dropped. When this happens TCP halves its maximum transmission rate. There you are watching some streamed video and suddenly it crawls to a halt because the data is being received at a reduced rate. Congestion control has allowed the Internet to grow despite the fact that bandwidth is not increasing at the same rate as traffic. Nevertheless it can be a real problem for applications which require a minimum transmission and receive rate. In some cases it can be better to transmit a packet and have it lost in the network than block trying to get a single packet through.
The last protocol of interest is IP Multicast. Multicast allows an endpoint to address a group of endpoints using a single IP address. Endpoints can join a group and will then receive all traffic sent to it. This is an unreliable datagram based service similar to UDP. Obvious applications are the broadcasting of data (such as video) or distribution of data for group collaboration. At the moment support for multicast does not extend to all parts of the Internet. Also there are currently no standardised protocols for achieving reliable transmission because research has shown that building a single reliable multicast protocol that has good performance characteristics for all possible group based applications is pretty much impossible. Multicast transmission is still very much an open research topic and as such this series will not cover it further as its current applicability is slightly limited.
Miscellaneous issues
So we now know about IP addresses, ports and the protocols that are available to us. However there are still some other issues that we must be aware of. If you only intend to write network applications based entirely on RISC OS then some of these issues may not be relevant but it might still be useful to be aware of them. In general the problems arise because the Internet is a heterogeneous environment (all the machines are different and you have no idea what type of machine you will be talking to).
Byte Ordering
The first problem facing communicating tasks running on different architectures is that of byte ordering. Whether your processor stores data in big endian or little endian format will decide on how it interprets bytes within data such as integers. At worst all your data can come out reversed at the other end.
In order to overcome this the sockets library supports the notion of "network byte order." This is an independent byte ordering scheme which data is converted to before transmission and converted from on reception. Note this conversion must be carried out explicitly by the application as it is not done automatically. This just consists of calling certain provided functions.
Marshalling and Unmarshalling
Merely converting data into a generic byte ordering format is only part of the story. Data usually has to be transmitted in a particular format. Even if the two endpoints are the same type of machine there can be problems due to the fact that software developed using different compilers may store composite datatypes, such as structures, in different ways. This is a common problem in linux and the solution is to transmit the data in a predefined format. By this I mean that individual data items need to be clearly defined (for example an integer is 32 bits, a long is 64 bits) and then structure fields are marshalled / unmarshalled individually.
If enough people want me to cover this then I'll consider it, possibly by adding a "message" abstraction to the library which provides functions to make dealing with these problems easier.
RISC OS issues
Finally in this section it seems prudent to cover some RISC OS specific network programming issues. In my opinion sockets programming under RISC OS IS more difficult than on other platforms for two related reasons. Firstly there RISC OS has no multithreading capabilities. While it is true that there have been several add ons to try and add this, they remain just that - unsupported add ons. They usually have problems such as not being compatible with all compilers, for example the multithreading code in the Dreamscape library relies on certain features of the code generated by the Acorn Norcroft compiler and does not work with GCC. Multithreaded code makes sockets programming easier as you can create independent threads for receiving packets and then just let them block on a blocking socket. Incoming packets are then placed on a thread synchronised data structure such as a list, and received at the main applications leisure.
Instead RISC OS supports the Internet event which is generated when there is data to be received. Unfortunately we can only make use of this from ARM code or a module so I may not be covering this. There are workable alternatives which do not involve polling the sockets and thus wasting processor time but in general that are not as convenient.
Secondly we have co-operative and not pre-emptive multitasking. The downside of this is that without multithreading we cannot simply block an application on a socket even if it has nothing else to do. I do not have much experience in writing deskopt applications so I will not be covering these directly. However it should not be too difficult to apply the techniques covered for dealing with non-multithreaded applications to desktop programs.
Common Network Application Architectures
As we come to the end of this article it makes sense to discuss some common architectures for network applications. There are two main models, the most dominant being client-server but the alternative peer-to-peer model is becoming more applicable and as multicast is introduced throughout the Internet will probably eventually overtake client-server based applications.
Client-Server is exactly as described. A client, such as a web browser, connects to a server which provides some service. In order to do this the client must know both the servers IP address and the port number it is using. IP addresses can either be hard coded, specified by the user at runtime, or automatically looked up using a DNS server. The port number is usually published as "a well known port number." For example a http server will almost always reside on port 80. This model is also useful for applications that may not be networked based. APIs such as OpenGL adopt this approach with the API forming the server and the application being the client of the API. If you are writing a game with the intention of including network support then it is probably best to divide the code into input (client) and processing (server). Providing the server is accessed via an API rather than directly, adding networking support becomes a case of transparently modifying the code behind the server API. The client is unaware of where the server resides.
In peer-to-peer networking there is no server. Each member or "peer" can act as both client and server at the same time - there is no distinction. Any data is sent to all peers in the session. How this is acheived is dependent upon the implementation. For example ICQ requires the use of a broadcast server. Another application may use some for of multicast transmission.
The techniques described in this series are equally applicable to both client-server and peer-to-peer networking. It all depends on the architecture of your particular application.
Summing up
In this article I have given an overview of sockets programming and discussed some of the issues a developer must address when writing network based applications. I have also given a brief overview of the two dominant application architectures. In the next article we will examine the sockets API in more detail and start building a library which will provide a simplified, but powerful, interface to sockets programming.
