Hello World, With Content Types

In the first two parts of this trail, we made a basic service and we enhanced it to accept a query parameter.

Both of these returned their responses as plain text. Plain text is not particularly useful: it doesn’t excel at presentation the way that HTML does, and it doesn’t offer structured data to a smart client the way JSON (or XML, or EDN, or a number of formats) would.

In fact, we should go a step further: the client should be in charge of whether they get human-presentable HTML, or code-focused JSON, using the very same endpoints.

It’s time to see how Pedestal handles content types and response bodies. We will also get our first taste of interceptors.

After reading this guide, you will be able to:

Like Hello World, this guide is for beginners who are new to Pedestal and may be new to Clojure. It doesn’t assume any prior experience with a Clojure-based web framework. You should be familiar with the basics of HTTP: URLs, response codes, and content types.

If you’ve already done some of those other things, you might want to skip ahead to Your First API to start building some logic and multiple routes.

If you like to jump straight in to the deep end, you might be interested in the Pedestal Crash Course which assumes you know quite a bit about Clojure and web frameworks.

This guide also assumes that you are in a Unix-like development environment, with Java installed. We’ve tested it on Mac OS X and Linux (any flavor) with great results. We haven’t yet tried it on the Windows Subsystem for Linux, but would love to hear from you if you’ve succeeded with it there.

We’ll take this in small steps. If you get stuck at any point in this guide, please submit an issue about this guide or, hop over to the Pedestal Users mailing list and raise your hand there. You can also get help from the #pedestal channel on the Clojurians Slack.

In this guide, we will build on the same hello.clj that we’ve built up over the last two guides. We will enhance it to return a JSON content body. Then we will add the ability to look at the client’s preferred content type and make a decision about what to return.

If you worked through Hello World, With Parameters then you already have all the files you need. If not, take a moment to grab the sources from the whole shebang in that guide. Feel free to browse the complete sources in the repository, but be warned that the file contains all the versions that we built up through the previous guide. You’ll need to navigate some magic comments to pare it down to just the final version.

Our first effort to deal with content types will be pretty manual. This is to show you what goes on under the hood before we use the handy built-in features that Pedestal offers. Don’t worry, this won’t hurt too much. It’ll also let us introduce the most important concept in Pedestal: Interceptors.

Back in Hello World, With Parameters we echoed an HTTP request back as the response. That was pretty useful for debugging, and it will come in handy for our next step. Since our /greet route is working so nicely, let’s make a general route to just echo requests back. Pop open hello.clj and change the routes definition to this:

You probably noticed that routes is created with a def and not with a defn. def creates a var and binds a value to that var. Under the jargon, that means it creates a new name in the namespace and attaches a value. defn is a macro that essentially expands into def with a fn, hence the name.

In practice, when you see a def, you should think "this is creating a value, and values don’t change." So it is with the routes. After changing the routes, you will need to restart your service.

When you restarted your service, you probably got a nasty message from Clojure like:

That is a very precise way to say we forgot to define echo before using it. Let’s do that now.

We’re going to define echo differently than our handler function hello. Instead of a simple handler function, we’re going to define an interceptor.

Interceptors are the basic unit of work in Pedestal.

At every step of the way between when a request arrives from the client, and when a response is sent back, there’s at least one interceptor. This includes:

An interceptor is essentially a simple map, but the values for the most important keys are not simple values (like strings or numbers), but functions.

This identifies two phases: enter and leave. Each interceptor may participate in the phase by providing corresponding callback functions in the :enter and :leave keys (respectively) ^[1].

Some interceptors only do work during the enter phase, some only during the leave phase, and some do work during both phases. An interceptor that omits the :enter or :leave key will just be skipped — it’s not an error.

But what does that work entail? It’s essentially a bucket brigade: each interceptor is passed the context map, is free to perform other work if necessary, and returns a modified context map, which is passed to the next interceptor.

Part of the context map is the interceptor queue, and the interceptor stack; the queue is a list of interceptors who have yet to be invoked during the enter phase. As interceptors are invoked^[2], they are also added to the interceptor stack, which is used during the leave phase.

Because it’s a stack, the leave phase invokes the interceptors in the opposite order from the enter phase:

Handler functions, like respond-hello in our example, are special cases. Pedestal can wrap a plain old Clojure function with an interceptor that takes the request map out of the context map, passes it to the function, and uses the return value of the function as the response map ^[3].

My advice is to build your value in the context with :enter functions, turn it into a response at the tail end of the interceptor queue, and then refine the response map with defaults, headers, cookies, and so on in :leave functions.

We’re ready to define echo as an interceptor:

We normally wouldn’t write this in such an expanded form, but I wanted to show all the pieces one by one. Ultimately, we’re just making a map with the keys :name and :enter.

We can try that interceptor out now. Bounce your service and use curl to exercise the /echo route:

This should look pretty familiar from the previous guide.

When Pedestal starts your service, it sets up some default interceptors for you. That includes an interceptor that does the routing. A router is just a specialized interceptor that looks at parts of the request, decides which interceptors to invoke next, and pushes those onto the interceptor queue.

That means any other interceptor can also modify the queue! This is one of the big benefits of using interceptors: the ability to make dynamic decisions during request handling.

Pedestal is agnostic as to what servlet container (such as Jetty 9) is used to accept requests and return responses. At the core of Pedestal is the Servlet Interceptor^[4], which is the bridge between the Servlet API (the domain of the servlet container) and Pedestal’s interceptors.

The servlet interceptor queues up several interceptors ahead of any provided by the service^[5], including the ring-response interceptor; part of this interceptor’s job is to determine a default response Content-Type header, if the actual response map omits it - this is deduced from the response map’s :body.

ring-response also uses the body type to determine how to stream the response to the client; for example, Clojure collections are printed as a string, which is then streamed. A File's contents are read and streamed as a byte stream.

ring-response does a reasonable job, but it’s not always right. You might notice that "text/html" doesn’t appear anywhere in that list. We can force that by setting a content type header in our response, like this:

Let’s see the result:

As expected. But, something about that doesn’t seem quite right. We’re completely ignoring HTTP content negotiation. The client might want JSON instead of HTML. Or it might want plain text. Or EDN. The trouble is that the HTTP content negotiation spec is a royal pain. Fortunately, Pedestal provides an interceptor to help.

io.pedestal.http.content-negotiation/negotiate-content does the job. If you look at the docs, though, you’ll see that is not an interceptor, but rather a function that returns an interceptor.

This is a common pattern when you need to include some state or customize the behavior of an interceptor. You pass arguments to a function which returns an interceptor that "closes over" those arguments. It returns a data structure that contains functions that carry those arguments around with them.

Let’s remove the content type header from ok and use this interceptor.

We are using a new namespace here, which will eventually let us write JSON data. That namespace comes from a library that we haven’t included before. So if you try to run this as is, you’ll get an error like this:

That is how Clojure tells you it is missing a library. By looking at the library’s project page, we see that the latest stable release is "2.4.0" (at least, it is when this guide is being written!). We can add the library in the dependencies part of our deps.edn file:

Any time you change the dependencies, you will definitely need to restart your service.

If you try this out, you’ll notice that absolutely nothing changed. That’s because the content negotiation interceptor handles the protocol, but it’s up to you to do something about the result. The interceptor runs the algorithm, then attaches the result to the request. That result is then available throughout the rest of processing. This is a great way to decompose your interceptors into small pieces that can be plugged together.

It’s up to our service code to return a different body format depending on the accepted content type. It probably won’t surprise you that this is a job for another interceptor!

Here’s our first stab at it.

Why does this new interceptor go at the start of the vector? Take look at the picture of the interceptors above. The first one in the vector is called first for the enter function but last for the leave function. We want this interceptor to get the last word on the response body so it goes at the top. Turn the queue sideways to write it as a vector, and the topmost interceptor is on the left. It only takes a little bit to get used to this.

Let’s try this out with curl. Restart your service and try sending in some curl requests with different "Accept" headers:

Notice the content type returned by our service was "text/html." Let’s try some others. Keep an eye on the "Content-Type" header in each response.

Notice when asking for the response in application/edn format, the string result was enclosed in quotes.

So coerce-body-interceptor works as intended. As usual, I’ve written it in a fairly "non-compact" style so it is easier to see how the parts work with Pedestal. I see a couple of things we can improve though.

First, there’s a straight-up bug. If a previous interceptor has already defined a content type, we should respect that and not overwrite the body or the headers. Second, we’re attaching a new headers map with only the "Content-Type" header. Any other headers attached by other interceptors will be lost. Finally, we could improve the testability by factoring out some of its logic into pure functions.

We’re also going to touch up the echo interceptor while we’re at it.

It is common style in Pedestal applications to make heavy use of the core functions like update and update-in. They are good at taking out a part of a nested data structure, applying a function to it, then reattaching the updated part. Let’s see how this interceptor would change if we refactor it that way.

We’re going to take one more step toward Clojure mastery. Do you see the coerce-body function? It looks inside the context. If there’s already a "Content-Type" header on the response, then it does nothing and returns the context without modification. On the other hand, if there is no content type assigned yet, it modifies the context by updating the response.

This is a really common pattern in functional languages. You want to make a series of changes to a data structure, where each change is conditional on some other logic. In an imperative language, this would look like a series of if statements whose bodies each mutate the object being built. In Clojure, they would look like a deeply nested if special form, where each if returns either a modified or unmodified version of the input. The trouble with deeply nested `if`s is that it’s too easy to get lost in the nesting.

This pattern is very common, and Clojure includes macros to help. The slightly tricky cond->footnote:">That’s read out loud as "cond arrow". Once you master it, you’ll really level up in Clojure skills. macro is what we need here. cond-> is similar to the usual threading macro ->, but instead of a just a series of expressions, it precedes each expression with a condition; the condition is evaluated first, and only if true is the value threaded through the expression.

In practice, it’s very straight forward:

If there were more clauses in the cond→, each clause would receive the value returned from the previous clauses.

We’re now getting a fair bit of code. This would be a good time to think about splitting into namespaces for different responsibilities. We’ll tackle that some other time. For now, let’s take a look at the whole thing. Spend some time making sure you understand how and when each line of code gets invoked.

In this guide, we built upon Hello World, With Parameters to add:

We also learned about interceptors and created a few.

The truth is that server-side applications don’t vend out HTML nearly as much as they once did. APIs are where it’s at. In the next tutorial we will make a REST style API to serve up "TO DO" lists.