Why REST Doesn’t Make Life More Rest-full

As I have observed before, IT as a field is highly driven by both fashion and received wisdom, and it can be difficult to challenge the commonly accepted position.

In the current world it is barely more politically acceptable to criticise the currently-dominant model of REST, Javascript and microservices than it is to audibly assess the figure of a female co-worker. I was seriously starting to think that I was in some age-defined Luddite minority of one in not being 100% convinced about the universal goodness of that model, but then I discovered an encouraging article by Pascal Chambon “REST is the new SOAP“, and realised that it’s not just me. I am not alone.

I don’t want to re-create that excellent article, and I recommend it to you, but it is maybe instructive to provide some additional examples of the failings Chambon calls out. I have certainly fallen foul of the quasi-religious belief that REST is somehow “better because it uses the right HTTP verbs”, and that as a result the “right verbs must be used”. On my last contract there was a lengthy argument because someone became convinced I was using the wrong ones. “You’re using POST to do a DELETE. That’s wrong.”

“No, we’re submitting a request to do a delete, if approved. At some later point, after the request has been reviewed and processed, this may or may not result in a low-level delete action, but the API is about the request submission. And anyway, you can’t submit a proper payload with a DELETE.”

“But you’re using a POST to do a DELETE…”

In the end I mollified him slightly by changing the URL of the API so that the tip wasn’t …/host, but …/host/request, but that did feel like the tail wagging the dog.

Generally REST promotes a fairly inflexible CRUD model, and by default without the ability to specify exactly which items are retrieved or updated. In a good design we may need a much richer set of operations. In either an RPC approach (as outlined in Chambon’s article), or a “remote object access” approach, such as one based on SOAP, we can flexibly tailor the operations precisely to the needs of the solution.

Here’s a good example. I need to “rename” an object, effectively changing its primary key. In the REST model, I have to choose one of the following:

• Add extra fields to the PUT payload to carry the “new” and “old” keys, and write both client- and server-side conditional code around their values, or an additional “operation” value
• Do a DELETE (with the old key) followed by a POST (with the new one), making sure that all the other data required to recreate the record is passed back for the POST, and write a host of additional code to handle cases like the DELETE succeeding but the POST failing, or the POST being treated as a new item, not just an update (because it’s not a PUT).
• Have a dedicated endpoint (e.g. …/object/rename) which accepts a POST operation with just the required data for the rename. That would probably be my favourite, but I can hear the REST purists screaming in the wind…

In a SOAP model, I can just have an explicit Rename(oldkey, newkey) operation on a service named for the underlying business object. Simples.

So Is SOAP The Old REST?

I’m comfortable with Chambon’s casting of REST as the supposed handsome hero who turns out to be a useless, treacherous bastard. I’m less comfortable with the casting of SOAP as the pantomime villain (boo hiss).

Now your mileage may vary, and Chambon obviously had some bad experiences, but in my own experience SOAP is a very strong and reliable technology which a lot of the time “just works”. I’ve worked in environments where systems developed in .Net, Oracle, Enterprise Java, a LAMP stack and Python cheerfully exchanged with each other using SOAP, across multiple physical locations, with relatively few complexities and usually just a couple of lines of code to access a full object model with formal schema and policy support.

In contrast, even if you navigate through the various different ways a REST service may work, inter-platform operation is by no means as simple as claimed. In just the past week I wasted about half a day trying to pass a body parameter between a Python client and a REST API presented by .Net. It should have worked. It didn’t. I converted the service to SOAP, and it worked almost first time. (Almost. It would have been even quicker if I’d remembered to RTFM…)

Notwithstanding the laudable attempts to fill the gap for REST, SOAP is still the only integration technology where every service has full machine and human readable documentation built in, and usually in a standard fashion. Get a copy of the WSDL (Web Service Definition Language) either from the service itself, or separately, and you know what it does, with what data, and, where it’s relevant to the client, how.

To extend the theatrical metaphor, in my world SOAP is the elderly retired hero who’s a bit pedantic and old-fashioned, maybe a bit slow on his feet, but actually saves the day.

It’s About the Architecture, Stupid

Ultimately it doesn’t actually matter whether your solution uses REST, SOAP, messages, distributed objects or CSV file transfers. Any can be made to work with sufficient attention to the architecture. All will fail in the presence of common antipatterns such as complex mixed data models, massive functional decomposition to too fine a level, or trying to make high-frequency chatty exchanges over higher-latency links.

Modern technologies attempt to hide a lot of technical complexity behind simple abstraction layers. While that’s an excellent approach overall, it does raise a risk that developers are unaware of how a poor design may cause underlying technical problems which will cause failure. For example while some low-level protocols are more tolerant than others, the naïve expectation that REST will work over any network regardless “because it is based on HTTP” is quite wrong. REST, SOAP and plain old web pages can all make good, efficient use of HTTP. REST, SOAP and plain old web pages will all fail if you insist on a unit of work being composed of vast numbers of separate small exchanges rather than a few larger ones. They will all fail if you insist on transferring large amounts of unfiltered data to the client, when that data should be pre-processed and filtered on the server. They will all fail if you insist on making every low-level exchange a network service when many of these should be direct in-process operations.

Likewise if you have a load of services, whether your own microservices or third party endpoints, and each service defines its own data structure which may be subject to change, and you try and directly consume and produce those proprietary data structures everywhere you need them, you are building yourself a world of pain. A core common data model with adapters for each format will serve you much better in the long run.

So Does Technology Choice Matter?

Ultimately no. For example, I have built an architecture with an underlying canonical data and adapter model but using REST for every exchange we controlled and it worked fine. Also in the real world whatever your primary choice you’ll probably have to deal with all the others as well. That shouldn’t scare you, but I have seen REST-obsessed developers run screaming from the room at the thought of having to use SOAP as well…

However, a good base choice will definitely make things easier. It’s instructive to think about a layered model of the things you have to define in a complex integration:

• Documentation
• Functionality
• Data structure and format
• Data encoding and transport
• Policies
• Service location and routing

SOAP is unique among the options in always providing built-in documentation for the service’s functions, data structures and policies. This is a major omission in the REST world, which is progressively being addressed by the Swagger / OpenAPI initiative and variants, but they will always be optional add-ons with variable coverage rather than a fundamental part of the model. For all other options, documentation is necessarily external to the service itself, and it may or may not be up to date and available to whoever needs it.

Functionality is discussed above and in Chambon’s article. Basically REST maps naturally to CRUD operations, and anything else is a bit of a bodge. SOAP and other RPC or distributed object models provide direct, explicit support for whatever functions are required by the business problem.

SOAP provides built-in definition and documentation of data structures and formatting, using XML Schema which means that the definition is machine and human readable, standardised, and uses namespaces and references to manage, for example, items with the same name but different uses and formats. Complexities such as optionality and alternative structures are readily defined. In addition a payload can be easily verified against the defined schema. Swagger optionally adds similar capabilities to the REST model, although without some discipline it’s easy for the implemented service to differ from the documented one, and it’s less easy to confirm that a given payload conforms. Both approaches focus on syntactic definition with semantic guidance optional and mainly through comments and examples.

In terms of encoding the data, the fashionable approach is JSON. The major benefits are that it’s simple, payloads are a bit smaller than the equivalent XML, and that it’s easy to parse into and generate from equivalent data structures in languages like Python.

However, I’m not a great follower of fashion. XML may be less trendy, but it offers a host of industrial-strength features which may be important in more complex use cases. It’s easy to unambiguously indicate the schema for each document and validate against it. If you have non-ASCII or binary data then their encoding is unambiguously defined. It’s easy to work separately with fragments of a larger document if you need to. Personally I also find XML easier to read and manually edit if I have to, but I accept that’s a bit subjective. One argument is that JSON is easier to render into a HTML page, but I’ve achieved much the same without any procedural code at all using XML with XSLT.

Of course, there’s no real need to have to choose. The best REST APIs I have worked with have the ability to generate equivalent JSON and XML from the same queries, and you choose which works best in a given context. Sadly this is again a bit too much for the REST purists, but a good solution when it works.

Beyond the functional definition of a service and its data, we also have to consider the non-functional behaviours, what are often referred to as “policies” in this context. How is the service secured? What encryption is applied to payloads and headers? What is the SLA, and what action should you take if it is exceeded? Is asynchronous or callback behaviour defined? How do I confirm I have all the required items in a set of exchanges, and what do I do about missing ones? What happens if a service fails, or raises an error?

In the early 2000s, when web services were a new concept, a lot of effort was invested in trying to establish standard ways to define these policies. The result was a set of extensions to SOAP known as the WS-* specifications: a set of rules to enable direct and potentially automated negotiation of all these aspects based on standardised information in the service WSDL and SOAP headers. The problem was that the standards quickly proliferated, and created the risk of making genuinely simple cases more complex than necessary. REST emerged as a simpler alternative, but with a KISS ethic which means ignoring the genuinely complex.

Chambon’s article touched on this in his discussion of error coding, but there are many other similar aspects. REST is a great solution for simple cases, but should not blind the developer to SOAP’s menu of standard, stronger solutions to more difficult problems.

A similar choice applies at the final level, that of locating and connecting service endpoints at runtime. For many cases we simply rely on network infrastructure and services like DNS and load balancing. However when this doesn’t meet more complex requirements then the alternatives are to construct or adopt a complex proprietary solution, or to embrace the extended standards in the WS-* space.

One technology choice is important. A professional modern Integrated Development Environment such as Visual Studio or Intellij Idea will do much of the “heavy lifting” of development, and does make work much quicker and less error-prone. I completely fail to understand why in 2018 some developers are still trying to do everything with vi and a Unix command line. When I was a schoolboy in the 1970s there was a saying “shouldn’t you have handed that in at the end of the war?”, referring to people still using or hoarding equipment issued in WW2. Anyone who is trying to do software development in the late 2010s with the software equivalent deserve what they get… It is a mistake to drive a solution from the constraints of your toolset.

Conclusions

The old chestnut that “to the man who only has a hammer, every problem looks like a nail” is nowhere more true than in software development. We seem to spend a great deal of effort trying to make every new software technique the complete solution to life, the universe, and everything, rather than accepting that it’s just another tool in the toolbox.

REST is a valid addition to the toolbox. Like it’s predecessors it has strengths and weaknesses. It’s a great way to solve a whole class of relatively simple web service requirements, but there are definite boundaries to that capability. When you reach those boundaries, be prepared to embrace some older, less-fashionable but ultimately more capable technologies. A religious approach will fail, whereas one based on an architectural viewpoint and an open assessment of all the valid options has a much greater chance of success.