Trillions - Climbing the right mountain

If you haven't already seen it, watch this great clip by the maya design group. It describes the problems that we are about to face with integrating the Internet of Things in the most succinct way I've seen to date.

Trillions from MAYAnMAYA on Vimeo.

A section of it resonated with me greatly; that is that we are not going to create a network of Trillions of devices using the technology we have today. We need to climb a new mountain of ideas that will enable Trillions of devices to work together.

Looking a little further past the clip, you can find the White Paper by Peter Lucus, a founder of the Maya Design group on the challenges faced by creating a Trillion node network. An interesting point he makes is that in terms of information exchange, there have only been two universally accepted units of data representation; the bit and the byte. It has been forty years and there has been no improvement on this. The White Paper also discusses some of the architectural requirements that any solution must meet to make the Trillon node network a reality; these include scalability, tractability and comprehensiveness.

Argot/XPL is our attempt at climbing the right mountain. Argot is designed around those architectural requirements identified by the Maya Design group, however, at a more practical level I believe the solution must meet a number of other technical requirements. These are:

1. Strict Internal Data Models – Each node/file in a Trillion node network must have an internal consistent data model of the types of data it contains or it can communicate. This data model describes the structure of each element it can communicate. Most distributed systems already have a strict data model; however the data model is often referenced and is not specific to the node of the network. XML Schema offers a shared data model which makes it difficult to split schemas and implement parts of a schema. A Strict Internal Data Model describes the data types that the individual node is capable of communicating.



2. Element Versioning – The Data Model must be versioned at each individual element or structure. This is one area where XML Schema and most other data modelling tools currently fail at providing the necessary functionality to meet the needs of a Trillion node network. As functionality improves and changes, the data model must be updated. XML Schema only allows very limited changes which retain backward compatibility. Sooner or later an XML Schema requires changes which break backward compatibility. When backward compatibility is at this stage all nodes in the network must be updated to support new Schemas. This already happens in large corporate systems that use XML Schemas heavily.



3. Partial Comparison – It must be possible to perform a partial comparison of the data model. The data model of a Trillion node network is likely to be made up of many different schemas. Devices will implement different amounts of functionality from different shared data models. It must be possible to allow two nodes to directly discover if devices can communicate and discover the method of communications.

An important aspect of these requirements for building a Trillion node network of heterogeneous devices is that agreement on the data format is not a core requirement. The format could be Argot, or any other suitable format supporting those requirements. The requirements for the Trillion node network is on the data models and the nodes in the network to be able to allow change through versioning, partial data model implementation, partial data model comparisons and finally discovery mechanisms to allow agreement on data formats to be established.

Peter Lucus finishes his white paper by suggesting that attribute/value pairs might be the next universally agreed unit of data representation. This is where I disagree, the next unit of universal data representation needs to be the data model. From this point of view, data modelling to support change is very immature. I've seen very little practical implementations that deal with change acceptably in a heterogeneous network.

In the end, I agree with the Maya Group. We need to start climbing the right mountain to create suitable solutions to the Internet of Things. I don't believe we will get there using REST and XML as they don't support change, and change is one of the few requirements I can truly be certain is required as the Internet of Things become a reality.

Argot submitted to IETF

A little while ago Esmond Pitt and I submitted the “Extensible Presentation Language (XPL) and Type Resolution Protocol (XPL/TRP)” to the IETF as part of the 6lowapp working group. XPL is the name we have selected, however, the technology is based on Argot which has been developed over the past six years. I'm incredibly pleased with the result, as the document now provides the most clear description of what Argot is all about. The document draft-ryanpitt-6lowapp-xpl-00 is published on the IETF web site.

The following exerts from the proposal provide a nice short and concise description of the XPL/Argot concept:

“XPL was created to combine (i) the extensibility provided by XML and XML Schema, (ii) the ability to describe and encode binary information like IDL+IOP and ASN.1, (iii) the ability to create remote procedure call services like CORBA, and (iv) to be inherently 'version-aware', to ensure that a change to one aspect of the system did not create cost across the whole network.

The XPL solution is to have each application in the network contain metadata about all the information it can communicate (send and receive). This allows each XPL application to negotiate directly with other peers the information they are able to exchange. It is this fundamental change of moving the metadata knowledge into the application or file, rather than keeping it externally, which is the most important aspect of XPL.

The XPL type system embedded (or notionally embedded) in each device is combined with methods for performing type comparisons with external systems. The type comparisons are designed to find the common definition type set shared by two XPL type systems.”

The proposal breaks the technology into two parts. The following descriptions are also from the proposal:

1. XPL, a powerful Extensible Presentation Language which is used both at protocol design time and by applications at runtime via very simple and small code libraries to communicate both primitives and compound types. XPL creates an internally consistent directed graph, which may be cyclic, to define complete versioned type systems to be stored with (or notionally with) an application or device.
2. XPL/TRP, a compact Type Resolution Protocol, in turn provides (i) dynamic negotiation of protocols and their constituent types between disparate devices; (ii) protocol versioning all the way down to the type level; and (iii) dynamic discovery of device capabilities and versions. An XPL device is able to describe and communicate its own application protocol. All these features are built-in to the protocol and intrinsic to its operation, rather than being extra-cost additions to it.

The XPL system is a departure from how most application protocols are designed. Protocols are normally designed and then the implementation is created from the design. XPL binds the design and implementation together so that they are interrelated. This has interesting consequences for versioning and detailed discovery. A device can be interrogated to discovery the structure of all the data that can be sent/received to it. Combined with scripting languages and other methods it would allow clients to be built automatically with zero code. XPL is the first time that you can implant a formal protocol description in any device or application down to the smallest of devices.

There's a lot of other interesting proposals that have been delivered to the IETF. In the application protocols area includes, XPL, Binary HTTP (chopan), Binary XML (exi), ZigBee Alliance requirements, and others have been submitted. The area of M2M (Machine 2 Machine) is going to continue to hot up over the next few years. There's still plenty more work to do, and it will take a while before a consensus is reached over the various protocols. Some interesting times are ahead!