For a while I have been having a lot of trouble finding what to focus on and jumped between a few projects. I updated Colony to work with Argot versioning. I also rebuilt the software that Argot was originally built; a network monitoring solution that built on a virtual network of nodes for processing and filtering log messages at high speed. I contemplated polishing Argot 1.3; fixing up documentation, adding more examples and improving the web site. I've also thought about the BORED protocol (see older posts) and the redesign of the Colony communication protocol. This has lead to thinking about the model that Argot and Colony were based. The original model dates back over 15 years and hasn't changed much since then.
After a lot of heading off on different tangents, I think I'm now heading in the right direction. I'm looking at the full protocol stack from transport to application layer. I'm returning to the original model that Colony was based and re-establishing the fundamental concept of the personal browser. This will provide a more solid foundation from which to build Colony and the Argot Remote Type Negotiation system. It will also take on some of my learning's from investigating REST with the BORED (Binary Object REst Distributed) protocol.
In the numerous posts regarding BORED protocol, one of the most pertinent things I discovered is that designing protocols requires a very clear understanding of the vision for the protocol. The specific aims, tasks and model must be understood clearly from the start. For this reason its worth revisiting the history and thoughts behind Colony and Argot and understanding the model it's based upon.
Colony History
As I said before, the original model dates back over 15 years to the early 1990's from when I had just started computer science at University. Before I started at Uni I already had a fascination with protocols and communication. This interest was born from using Bulletin Board Systems (BBS's) as a teenager. Before starting Uni I had designed protocols and developed software for drawing vector graphics so two people could both draw on each others screens. What annoyed me from that early age about BBS's was the inability to do more than one thing. As I learnt more about communications I started to develop a model for distributed applications.
The model for Colony was based on a few simple constructs; zones, realms and nodes. These constructs still exist in Colony today, however, they don't integrate as cleanly into the design as I would like. I'll explain each of these constructs and their behaviour now.
A zone is a container with simple name/value pairs. A zone could be implemented as a HashMap, overlay a directory structure or proxy another zone from a remote system. It can contain simple values, objects or other zones. A zone provides the basis for naming and containment in the system.
A realm is an extension of a zone. A realm provides the security aspects of a zone or set of zones. The idea of a realm was that it be implemented as a process but include a security model to ensure applications only had a specific access to the underlying system.
A node provided the processing aspect of the design. A node contains a queue for receiving messages and a thread pool for processing the messages. A distributed application would consist of multiple nodes passing messages. The design was suited to network applications that would process a lot of data from different sources.
Applications would be designed with all their data contained in zones, realms and nodes which would allow real-time introspection of the data and how nodes were behaving.
The original design was based on a message processing architecture. Each node would be designed to perform a small and specific task and forward on the same or new message to another node. Applications could be configured by putting different nodes into a zone and wiring them up to perform a specific function. Messages could be passed between realms to create a distributed application. Since this original design a lot of things have changed about what is important in distributed applications.
Colony Architecture Revisited
Before launching into protocol design based on the original model of Colony, it's worth looking at the original design with over 15 years of experience. The original design was aimed at creating a BBS client/server system that allowed multiple application to communicate concurrently. It was expected that each application would be downloaded and installed and use the naming and communication system of Colony. Since that time things have changed considerably.
TCP/IP and the browser have become the key building block for the largest percentage of web communication. Many of the building blocks for distributed systems have been developed and are a lot better understood. Virtual Machines and scripting languages which leads to code on demand are now available. This all leads to a different architecture for Colony which mixes the old and new.
The revised aim and architecture of Colony is to build a “personal browser”. The Colony Personal Browser mixes concepts of Instant Messaging, Code on Demand, Distributed Computing, Security, Identity and Web Browsing to create a secure connected peer-to-peer browser environment. This environment should allow applications to create secure connections between both individuals and Colony servers offering services and/or web style documents. At a high level the components of the “personal browser” include:
Other than Identity all of these components make up a modern browser; so before moving on I'll explain what is different about each of these components.
Documents & Code on Demand – In a paper I wrote a number of years ago, I outlined a method of using Argot as a common data format for virtual machine byte code, document formatting and as a scripting language. All of these formats are compiled to a common byte code which is then targeted to a more specific virtual machine. Creating a common format for all data creates a very flexible development environment where different styles of programming languages can be built on the same virtual machine. It should also allow mixing different programming styles closely together allowing the best method of development to be used for each problem. You can read the evolvable programming languages paper here.
Security & Identity – The web of trust security model has been well documented and around for many years. However, after the initial interest in PGP its use has dwindled and it has not been accepted into the main stream. The aim of the personal browser is to wrap a set of user interfaces and processes around the web-of-trust model to make it easier and quicker for users to understand and use. You can read more about the web of trust at wikipedia.
Communications – The area of communications is obviously the most important aspect of the personal browser. Communicating between users and servers requires a hybrid peer-to-peer system. This style of communication requires a flexible transport layer that allows any peer to act as a client and initiate requests to other peers. This then requires a flexible naming system that allows peers to be found and communicated. The naming and routing aspect of the personal browser is probably the most difficult with the least defined solution. I'll discuss the protocol in more detail a little later.
Virtual Machine – The virtual machine and underlying execution environment must provide a sandbox environment that ensures the security of the underlying system. This is a well understood area and selecting the Google Chrome environment using V8 or Java Virtual Machine as a basis will both achieve the aims here.
User Interface – This is another area with very well understood and well developed code. Taking a browser layout engine that can be removed from an existing browser without being infected with HTML or CSS will provide a good clean user interface as a starting point.
Colony communication building blocks
This next section will examine some of the building blocks required as input into the next version of the Colony protocol.
Sand Boxes
The concept of realms can be compared to the sand boxes used by both Java and Google Chrome. Google Chrome is interesting in that it separates the browser into multiple Processes to improve security. Colony can use this same mechanism to separate each realm and provide limited access to hosted applications. This can be achieved by using a gateway system which provides firewall and routing mechanisms to data being sent and received from applications.
The gateway is able to handle connections to multiple hosts and direct messages between multiple Realms running on the same computer. Between the Gateway and Realms a system pipe or other communication mechanism can be used. For external communications IP based transport will be used.
SCTP – The right transport
One of the big challenges that I've been faced with over the years is building the right type of protocol over TCP. As TCP is a stream based protocol it is faced with a number of issues. A friend recently pointed me towards SCTP as a potentially better transport protocol. After reviewing some of the various introductory web sites on SCTP, I'm convinced. SCTP creates a much better base to build higher level protocols than both UDP or TCP. SCTP provides:
- Multiple Streams on a single connections
- Multi-homing
- Unordered delivery
- Reliable transmission
- etc
There's more information on SCTP available out there. There is also the issue that SCTP is only being introduced in Java 7 which is still in beta. Also, Java 7 when released will not support SCTP on Windows. There is third party implementations of SCTP for windows, however, the protocol is not yet supported by Microsoft. Given I'm developing Colony as a longer term solution I don't believe this will be a problem.
I will publish some prototype code soon which demonstrates TLS on SCTP using Java 7 which I developed on Open Solaris using VMWare.
TLS – Web of Trust
Another big issue in distributed applications is security and identity. To provide security the obvious answer is TLS which is a proven secure protocol. As mentioned earlier, the solution for identity will be the Web-of-trust model.
Ideally the digital certificate data used in the web-of-trust model will be encoded using Argot instead of ASN.1 to ensure consistency across the system. The Web-of-trust model will be developed after the main protocol.
Naming & Location
The URI has proved itself as one of the most flexible solutions to object naming and location. People understand it and find it easy to use. It is not perfect for this application as it binds the name to a specific host and IP address. This may not be a perfect mechanism to use, however, initially it provides the best choice for naming and location. I'll discuss why the URI is not perfect a little later into the design.
Another possible solution to naming and location would be based on a peer-to-peer naming system. This would fit better with the distributed nature of the system.
Protocol Stack Design
The main ingredients of Sand Boxes, SCTP, TLS/Web-of-trust and URI naming are all the ingredients required to design the lower levels of the protocol stack. I'll now examine how these parts would be best glued together in a protocol to meet the design.
My initial thought process for the protocol stack was that it would be obvious that TLS would sit over SCTP to provide secure communications between systems. However, after delving further into the design I realise that using the model where a gateway is the external communications end point it fails to provide true end to end security to the application. To provide end to end security the TLS connection must terminate at the realm and not at the gateway. To support this, a protocol gateway layer (Colony Routing Protocol) is required which acts as an application message router. The protocol stack then looks like:
In effect what is being created is a high level transport protocol that allows messages to be delivered directly to objects with security. This change in model has a number of advantages. It allows the gateway to act as a load balancer or facade onto a group of larger systems. It can also allow the gateway to proxy requests to other gateways. The gateway can also use other transport mechanisms to reach the final Realm destination. On a local computer it can use pipes, while on a remote system it could use SCTP, TCP or other transport. The protocol should also be light weight enough to send through to embedded devices.
As mentioned before, using SCTP encourages a message based system. The gateway router mechanism also suggests that a message based system would also be appropriate for this layer of the protocol stack. A secondary restriction is that TLS requires that each message packet be no longer than 16kb. This puts a restriction on the size of the overall SCTP packet to 16kb plus any header information.
It is expected that the end point location for a message will be any URL. The router will find the realm that contains the target object and direct the message to that realm. For example, a target of crl://some.server/target/realm/object will be received by the realm container at crl://some.server/target/realm.
Another advantage to this model is that while it offers a lot of flexibility, it does so without putting any restriction on the actual protocol used by the Realm or target object. Different objects can use different protocols on the same transport.
The Colony Routing Protocol data packet is likely to contain the following parts:
Preamble – header signifying the colony routing protocol.
Version – version of the protocol. Major & minor version details.
Headers – Additional header data. Does message require response, etc.
Target Location – URI of the target location.
Data – data to be delivered to the realm. Max. size 16kb.
Digital Signature – Optional signature for message.
An interesting element of this protocol design is that it does not dictate request/response semantics as used in REST/HTTP. A client can send a one way message to a target location. It is up to the semantics of the session layer protocol to decide if a response is required. The underlying system also puts no restriction on whether the client or server initiates the request.
To support server initiated requests requires that each client has a host and realm that will receive messages. As clients do not usually have a fully qualified host name it will require an alias is given to the client connection on the server. This is where a traditional HTTP browser paradigms do not provide enough flexibility and a peer based host naming system may be more appropriate.
Another requirement for the routing protocol is to allow the creation of routes between two realms/objects that are expected to last a long period. The advantage of this is that the target location and any digital signatures can be dropped allowing less overhead per packet. This would be especially useful for voice or video communication protocols. It would also allow some of the benefits of SCTP to be exposed to the upper layer protocols. This requires that the communication layer keep session state. Keeping session state is not ideal at this layer of the protocol stack, however, is inevitable to support this type of feature. The state should only be limited to link information state which is no different to the way a NAT router holds state.
The routing layer has many parallels with peer-to-peer naming and routing systems which have been documented previously. This area will need to be researched more thoroughly before locking in a specific solution.
Before finishing the subject of routing and naming it is important not to forget firewalls and NAT. This is the enemy of any peer to peer based system. There are various solutions to this problem which have been documented. A possible solution is to use Colony application servers which act as Internet routers and Proxies for clients behind firewalls. This is another area which will require further investigation.
Session (Discovery) Layer
The session layer will sit on the TLS layer to provide elements of a REST architectural style; specifically the client/server, stateless and uniform interface constraints. The aim of this layer is to provide a discovery, reflection and basic set of mechanisms to communicate with the target objects. The discovery aspect of the protocol layer will allow a client to discover what protocols can be used with the target object. The server could allow multiple presentation and application layer protocols to be provided, allowing a client to choose the most appropriate method to communicate.
The Uniform Interface would include at least the following types of messages:
GET - return the selected or default presentation of the target object. This may be a image or document file. It may also be code on demand.
META – return the associated protocols and presentations available for the target object. This could looking something like:
(protocol:cache,stream data-type:image/jpg)
or
(protocol:dynamic-argot-dictionary application:colony-vm)
DATA – send data to the target object using a selected protocol.
The packet structure for the stream includes the following:
message type – GET, META or DATA. Most likely encoded as a byte.
Protocol type – the protocol type being sent.
Data – data to be passed to the target object.
Depending on the protocol selected there may be multiple session, presentation and application layer options over this transport layer. These should be registered with the target object meta information. The client is able to select the most suitable method to communicate.
Application Level Protocols
The aim of this layered protocol design is to expose as many of the features of the underlying transport to the upper layers of the protocol stack and therefor distributed applications. It is likely that a number of application level protocols would be developed for the most common situations; the most obvious being a stream based solution for serving static files. This would allow both client or server side cache mechanisms to be included. Message queue based protocols are another obvious possible protocol. Smart agent or mobile code as already used by Colony is another possible application layer protocol.
As the design exposes the underlying SCTP packet structure to the application a combination of these protocols are possible. Using SCTP out-of-order and non-guaranteed delivery options also allows video or voice data protocols to be established.
Conclusion
This is very rough draft of the Colony Personal Browser model being developed and how the protocol would be designed to support the model. It builds on concepts from REST, uses SCTP and includes TLS security based on the web of trust model.
The design provides the following features:
Peer to Peer – Application protocols can direct messages to any object using a URL style format that users can easily understand.
User to User – Applications & Documents can be developed which connect users to users in the same way instant messenger applications work.
User to Server – Applications can be developed which connect directly to server applications in the same way the traditional browser operates.
Code & Data on demand – Using the virtual machine for both data and code creates a highly flexible environment to build interactive documents or applications.
Flexible – Application protocols have full access to the flexibility of the SCTP protocol; allowing to choose out of order delivery or non guaranteed delivery. This allows video and voice applications to be built.
Packet Based – Application protocols must adhere to sending a maximum data packet size of 16kb to adhere to TLS. To send larger data packets higher layer protocols must be established.
Short Messages – Messages shorter than 16kb can be sent directly to another object.
Established Links – Long conversations between peers can be created over established links.
Reflection – Objects can publish meta data to describe the available protocols or data presentation formats available.
Simplicity – A user is able to still add a simple text URL into a browser address bar and have a default data representation returned. This ensures the simplicity of web browsers is retained.
Complexity – An application developer can select the best protocol design for the interaction of client and server.
Security – TLS provides Realm to Realm security on all links.
Identity – Where a user chooses, their identity can be made known to the server they are connecting. This is perfect for peer to peer or applications where identity is required.
Obviously this is a rough outline and the details of all the components needs to be fleshed out. There are numerous parts to the design which is not complete; hopefully I've detailed enough to provide a clear understanding of the direction I'm taking the Colony Personal Browser. If you've made it this far please leave a comment. Do you think the design is good, think its got gaping holes, or just way too ambitious?
