Pedestal

Hello World, With Content Types

Welcome Back

In the first two parts of this trail, we made a very basic service and we enhanced it to accept a parameter.

Both of these returned their responses as plain text. Plain text is ugly, so we want to be able to return HTML. Rich clients don’t like HTML so much, so we also want to return JSON sometimes. It’s time to see how Pedestal handles content types and response bodies. We will also get our first taste of interceptors.

What You Will Learn

After reading this guide, you will be able to:

  • Add interceptors to routes.

  • Use functions to create interceptors.

  • Use the response map.

  • Transform responses into other content types.

Guide Assumptions

Like Hello World, this guide is for beginners who are new to Pedestal and may be new to Clojure. It doesn’t assume any prior experience with a Clojure-based web framework. You should be familiar with the basics of HTTP: URLs, response codes, and content types.

If you’ve already done some of those other things, you might want to skip ahead to Your First API to start building some logic and multiple routes.

If you like to jump straight in to the deep end, you might be interested in the Pedestal Crash Course which assumes you know quite a bit about Clojure and web frameworks.

This guide also assumes that you are in a Unix-like development environment, with Java installed. We’ve tested it on Mac OS X and Linux (any flavor) with great results. We haven’t yet tried it on the Windows Subsystem for Linux, but would love to hear from you if you’ve succeeded with it there.

Getting Help if You’re Stuck

We’ll take this in small steps. If you get stuck at any point in this guide, please submit an issue about this guide or hop over to the mailing list and raise your hand there.

Where We Are Going

In this guide, we will build on the same hello.clj that we’ve built up over the last two examples. We will enhance it to return a JSON content body. Then we will add the ability to look at the client’s preferred content type and make a decision about what to return.

Before We Begin

If you worked through Hello World, With Parameters, then you already have all the files you need. If not, take a moment to grab the sources from the whole shebang in that guide. Feel free to browse the complete sources in the repository, but be warned that the file contains all the versions that we built up through the previous guide. You’ll need to navigate some magic comments to pare it down to just the final version.

The Hard Way

Our first effort to deal with content types will be pretty manual. This is to show you what goes on under the hood before we use the handy built-in features that Pedestal offers. Don’t worry, this won’t hurt too much. It’ll also let us introduce the most important concept in Pedestal: Interceptors.

Back in Hello World, With Parameters, we echoed an HTTP request back as the response. That was pretty useful for debugging, and it will come in handy for our next step. Since our /greet route is working so nicely, let’s make a general route to just echo requests back. Pop open hello.clj and change the routes definition to this:

(def routes
  (route/expand-routes
   #{["/greet" :get respond-hello :route-name :greet]
     ["/echo"  :get echo]}))

You probably noticed that routes is created with a def not a defn. def creates a var and binds a value to that var. Under the jargon, that means it creates a new name in the namespace and attaches a value. defn is a macro that essentially expands into def with a fn, hence the name.

In practice, when you see a def, you should think "this is creating a value, and values don’t change." So it is with the routes. After changing the routes, you will need to restart your service. (This is purely a result of the way we’ve built things for this guide. There are techniques that let you automatically reload routes during development. We’ll get to those in Developing at the REPL.)

When you restarted your service, you probably got a nasty message from Clojure like:

clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to resolve symbol: echo in this context, compiling:(hello.clj:30:3)

That is a very precise way to say we forgot to define echo before using it. Let’s do that now.

We’re going to define echo differently than our handler function hello. Instead of a simple handler function, we’re going to define an Interceptor.

Interceptors

Interceptors are how Pedestal handles requests. Routing is done by interceptors. Parsing query parameters is done by interceptors. Even our hello handler function got wrapped up into an interceptor when we put it in the route table. An interceptor is a data structure with functions in it. Each function receives a context map and returns a context map, as shown in this diagram.

interceptors

That context map contains the request map, the response map, and a queue of interceptors that still need to be invoked.

An interceptors enter function is called on the way "down" the queue. Each enter function will be called in order. As interceptors are entered, they get pushed on to a stack to be called in reverse on the way back "up". Once the interceptor queue is empty, the leave functions are called from the stack. That means the leave functions are called in reverse order, as shown below.

interceptor stack

Handler functions, like our respond-hello, are special cases. Pedestal can wrap a plain old Clojure function with an interceptor that takes the request map out of the context map, passes it to the function, and uses the return value of the function as the response map. (That actually takes more words to explain in English than it does in code!)

Every interceptor can also have an error function, but we don’t need to deal with those just yet.

An Echo Interceptor

We’re ready to define echo as an interceptor:

(def echo
  {:name ::echo                                                                       (1)
   :enter (fn [context]                                                               (2)
            (let [request (:request context)                                          (3)
                  response (ok request)]                                              (4)
              (assoc context :response response)))})                                  (5)
1 Interceptor names help with debugging.
2 We’re making an enter function.
3 Take the request map out of the context map.
4 Make a response map out of it.
5 Attach the response to the context map, and return the new context map.

We normally wouldn’t write this in such an expanded form, but I wanted to show all the pieces one by one. Ultimately, we’re just making a map with the keys :name and :enter. When Pedestal sees this map in a route, it turns the map into an Interceptor record for you.

We can try that interceptor out now. Bounce your service and use curl to exercise the /echo route:

$ curl http://localhost:8890/echo
{:protocol "HTTP/1.1", :async-supported? true, :remote-addr "127.0.0.1", :servlet-response #object[org.eclipse.jetty.server.Response 0x78a74616 "HTTP/1.1 200 \nDate: Wed, 24 Aug 2016 13:24:53 GMT\r\nStrict-Transport-Security: max-age=31536000; includeSubdomains\r\nX-Frame-Options: DENY\r\nX-Content-Type-Options: nosniff\r\nX-XSS-Protection: 1; mode=block\r\nContent-Type: application/edn\r\n\r\n"], :servlet #object[io.pedestal.http.servlet.FnServlet 0x1d574b7c "io.pedestal.http.servlet.FnServlet@1d574b7c"], :headers {"user-agent" "curl/7.47.0", "accept" "*/*", "host" "localhost:8890"}, :server-port 8890, :servlet-request #object[org.eclipse.jetty.server.Request 0x7660b772 "Request(GET //localhost:8890/echo)@7660b772"], :path-info "/echo", :url-for #object[io.pedestal.http.route$url_for_routes$fn__11094 0x725f2d66 "io.pedestal.http.route$url_for_routes$fn__11094@725f2d66"], :uri "/echo", :server-name "localhost", :query-string nil, :path-params {}, :body #object[org.eclipse.jetty.server.HttpInputOverHTTP 0x375dbd49 "HttpInputOverHTTP@375dbd49[c=0,s=STREAM]"], :scheme :http, :request-method :get}

This should look pretty familiar from Hello World, With Parameters.

From Routes to Interceptors

When Pedestal starts your service, it sets up some default interceptors for you. That includes an interceptor that does the routing. A router is just a specialized interceptor that looks at parts of the request, decides which interceptors to invoke next, and pushes those onto the queue.

That means any other interceptor can also modify the queue! This is one of the big benefits of using interceptors: the ability to make dynamic decisions during request handling. That even includes the ability to bypass interceptors on the way back up.

Returning to Content Types

One of the most important default interceptors is the "servlet interceptor". It is both a servlet and an interceptor. This is how Pedestal interfaces with HTTP servers like Jetty and Tomcat. It is the first interceptor in the queue. On the way up through its leave function, it makes sure that every response has a content type. If you don’t set a content type header on your response, the servlet interceptor will pick a content type based on what kind of thing you return in the response body. This table shows the type mapping:

Table 1. Content Type Mapping
Object in :body Content Type

Byte array

application/octet-stream

String

text/plain

Clojure collection

application/edn

java.io.File

application/octet-stream

java.io.InputStream

application/octet-stream

java.nio.channels.ReadableByteChannel

application/octet-stream

java.nio.ByteBuffer

application/octet-stream

It does a pretty good job, but it’s not always right. You might notice that "text/html" doesn’t appear anywhere in that list. We can force that by setting a content type header in our response, like this:

src/hello.clj
(defn ok [body]
  {:status 200 :body body
   :headers {"Content-Type" "text/html"}})                                            (1)
1 Attach a header declaring the content type.

Let’s see the result:

$ curl -i http://localhost:8890/greet
HTTP/1.1 200 OK
Date: Wed, 24 Aug 2016 15:02:08 GMT
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)

Hello, world!

As expected. But, something about that doesn’t seem quite right. We’re completely ignoring HTTP content negotiation. The client might want JSON instead of HTML. Or it might want plain text. Or EDN. The trouble is that the HTTP content negotiation spec is a royally pain. Fortunately, Pedestal provides an interceptor to help.

io.pedestal.http.content-negotiation/negotiate-content does the job. If you look at the docs, though, you’ll see that is not an interceptor, but rather a function that returns an interceptor.

This is a common pattern when you need to include some state or customize the behavior of an interceptor. You pass arguments to a function which returns an interceptor that "closes over" those arguments. It returns a data structure that contains functions that carry those arguments around with them.

Let’s remove the content type header from ok and use this interceptor.

src/hello.clj
(ns hello
  (:require [clojure.data.json :as json]                                              (1)
            [io.pedestal.http :as http]
            [io.pedestal.http.route :as route]
            [io.pedestal.http.content-negotiation :as conneg]))                       (2)

(def unmentionables #{"YHWH" "Voldemort" "Mxyzptlk" "Rumplestiltskin" "曹操"})

(def supported-types ["text/html" "application/edn" "application/json" "text/plain"]) (3)

(def content-neg-intc (conneg/negotiate-content supported-types))

(def routes
  (route/expand-routes
   #{["/greet" :get [content-neg-intc respond-hello] :route-name :greet]              (4)
     ["/echo"  :get echo]}))
1 We’ll need this namespace soon to emit JSON responses.
2 This is the content negotiation namespace for Pedestal.
3 A short, picky list of content types we can emit.
4 Notice this route now has a vector of interceptors to invoke.

We are using a new namespace here, which will eventually let us write JSON data. That namespace comes from a library that we haven’t included before. So if you try to run this as is, you’ll get an error like this:

clojure.lang.Compiler$CompilerException: java.io.FileNotFoundException: Could not locate clojure/data/json__init.class or clojure/data/json.clj on classpath., compiling:(hello.clj:1:87)

That is how Clojure tells you it is missing a library. By looking at the library’s project page, we see that the latest stable release is "0.2.6" (at least, it is when this guide is being written!). We can add the library in the dependencies part of our build.boot file:

build.boot
(set-env!
 :resource-paths #{"src"}
 :dependencies   '[[io.pedestal/pedestal.service "0.5.1"]
                   [io.pedestal/pedestal.route   "0.5.1"]
                   [io.pedestal/pedestal.jetty   "0.5.1"]
                   [org.clojure/data.json        "0.2.6"]
                   [org.slf4j/slf4j-simple       "1.7.21"]])

Any time you change the dependencies, you will definitely need to restart your service.

If you try this out, you’ll notice that absolutely nothing changed. That’s because the content negotiation interceptor handles the protocol, but it’s up to you to do something about the result. The interceptor runs the algorithm, then attaches the result to the request. That result is then available throughout the rest of processing. This is a great way to decompose your interceptors into small pieces that can be plugged together.

It’s up to our service code to return a different body format depending on the accepted content type. It probably won’t surprise you that this is a job for another interceptor!

Here’s our first stab at it.

src/hello.clj
(def coerce-body
  {:name ::coerce-body
   :leave
   (fn [context]
     (let [accepted         (get-in context [:request :accept :field] "text/plain")   (1)
           response         (get context :response)
           body             (get response :body)                                      (2)
           coerced-body     (case accepted                                            (3)
                              "text/html"        body
                              "text/plain"       body
                              "application/edn"  (pr-str body)
                              "application/json" (json/write-str body))
           updated-response (assoc response                                           (4)
                                   :headers {"Content-Type" accepted}
                                   :body    coerced-body)]
       (assoc context :response updated-response)))})                                 (5)

(def routes
  (route/expand-routes
   #{["/greet" :get [coerce-body content-neg-intc respond-hello] :route-name :greet]  (6)
     ["/echo"  :get echo]}))
1 Get the result of the content negotiation interceptor. Use "text/plain" as a fallback in case no suitable match was found.
2 Get the current response out of the context map, get the current body out of the response. This must have been created by a previous interceptor’s enter or leave function.
3 Translate the body according to the chosen content type.
4 Create a new response by attaching headers and the coerced body.
5 Return a new context by attaching the updated response.
6 Tell Pedestal to put this interceptor at the head of the queue.

Why does this new interceptor go at the start of the vector? Take look at the picture of the interceptors above. The first one in the vector is called first for the enter function but last for the leave function. We want this interceptor to get the last word on the response body so it goes at the top. Turn the queue sideways to write it as a vector, and the topmost interceptor is on the left. It only takes a little bit to get used to this.

Let’s try this out with curl. Restart your service and try sending in some curl requests with different "Accept" headers:

$ curl -i http://localhost:8890/greet   # Don't specify any Accept header
HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 02:28:44 GMT
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/plain
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)

Hello, world!

Notice the content type returned by our service was "text/plain." Let’s try some others. Keep an eye on the "Content-Type" header in each response.

$ curl -i -H "Accept: text/html" http://localhost:8890/greet
HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 02:30:02 GMT
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)

Hello, world!

$ curl -i -H "Accept: application/edn" http://localhost:8890/greet
HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 02:30:24 GMT
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: application/edn
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)

"Hello, world!\n"

$ curl -i -H "Accept: application/xml, application/json" http://localhost:8890/greet
HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 02:30:58 GMT
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)

"Hello, world!\n"

Very nice.

Refactoring and Style

So our coerce-body interceptor works as intended. As usual, I’ve written it in a fairly "non-compact" style so it is easier to see how the parts work with Pedestal. I see a couple of things we can improve though.

First, there’s a straight-up bug. If a previous interceptor has already defined a content type, we should respect that and not overwrite the body or the headers. Second, we’re attaching a new headers map with only the "Content-Type" header. Any other headers attached by other interceptors will be lost. Finally, we could improve the testability by factoring out some of the logic into pure functions.

We’re also going to touch up the echo interceptor while we’re at it.

It is common style in Pedestal applications to make heavy use of the core functions like update and update-in. They are good at taking out a part of a nested data structure, applying a function to it, then reattaching the updated part. Let’s see how this interceptor would change if we refactor it that way. We’ll also use the slightly tricky cond→ macro. (That’s read out loud as "cond arrow." Once you master it, you will really level up in Clojure skills.)

src/hello.clj
(def echo
  {:name ::echo
   :enter #(assoc % :response (ok (:request %)))})

(def supported-types ["text/html" "application/edn" "application/json" "text/plain"])

(def content-neg-intc (conneg/negotiate-content supported-types))

(defn accepted-type
  [context]
  (get-in context [:request :accept :field] "text/plain"))

(defn transform-content
  [body content-type]
  (case content-type
    "text/html"        body
    "text/plain"       body
    "application/edn"  (pr-str body)
    "application/json" (json/write-str body)))

(defn coerce-to
  [response content-type]
  (-> response
      (update :body transform-content content-type)
      (assoc-in [:headers "Content-Type"] content-type)))

(def coerce-body
  {:name ::coerce-body
   :leave
   (fn [context]
     (cond-> context
       (nil? (get-in context [:response :body :headers "Content-Type"]))
       (update-in [:response] coerce-to (accepted-type context))))})

(def routes
  (route/expand-routes
   #{["/greet" :get [coerce-body content-neg-intc respond-hello] :route-name :greet]
     ["/echo"  :get echo]}))

This would be a fairly typical style for Pedestal code.

The Whole Shebang

We’re now getting a fair bit of code. This would be a good time to think about splitting into namespaces for different responsibilities. We’ll tackle that some other time. For now, let’s take a look at the whole thing. Spend some time making sure you understand how and when each line of code gets invoked.

src/hello.clj
(ns hello
  (:require [clojure.data.json :as json]                                              ;; <1>
            [io.pedestal.http :as http]
            [io.pedestal.http.route :as route]
            [io.pedestal.http.content-negotiation :as conneg]))                       ;; <2>

(def unmentionables #{"YHWH" "Voldemort" "Mxyzptlk" "Rumplestiltskin" "曹操"})

(defn ok [body]
  {:status 200 :body body})

(defn not-found []
  {:status 404 :body "Not found\n"})

(defn greeting-for [nm]
  (cond
    (unmentionables nm) nil
    (empty? nm)         "Hello, world!\n"
    :else               (str "Hello, " nm "\n")))

(defn respond-hello [request]
  (let [nm   (get-in request [:query-params :name])
        resp (greeting-for nm)]
    (if resp
      (ok resp)
      (not-found))))

(def echo
  {:name ::echo
   :enter #(assoc % :response (ok (:request %)))})

(def supported-types ["text/html" "application/edn" "application/json" "text/plain"])

(def content-neg-intc (conneg/negotiate-content supported-types))

(defn accepted-type
  [context]
  (get-in context [:request :accept :field] "text/plain"))

(defn transform-content
  [body content-type]
  (case content-type
    "text/html"        body
    "text/plain"       body
    "application/edn"  (pr-str body)
    "application/json" (json/write-str body)))

(defn coerce-to
  [response content-type]
  (-> response
      (update :body transform-content content-type)
      (assoc-in [:headers "Content-Type"] content-type)))

(def coerce-body
  {:name ::coerce-body
   :leave
   (fn [context]
     (cond-> context
       (nil? (get-in context [:response :body :headers "Content-Type"]))
       (update-in [:response] coerce-to (accepted-type context))))})

(def routes
  (route/expand-routes
   #{["/greet" :get [coerce-body content-neg-intc respond-hello] :route-name :greet]
     ["/echo"  :get echo]}))

(defn create-server []
  (http/create-server                                                                 ;; <1>
   {::http/routes routes                                                              ;; <2>
    ::http/type   :jetty                                                              ;; <3>
    ::http/port   8890}))                                                             ;; <4>

(defn start []
  (http/start (create-server)))                                                       ;; <5>

The Path So Far

In this guide, we built upon Hello World, With Parameters to add:

  • Rudimentary content negotiation.

  • Response body transforms.

We also learned about interceptors and created a few.

Where To Next?

The truth is that server-side applications don’t vend out HTML nearly as much as they once did. APIs are where it’s at. In the next guide we will make a REST style API to serve up wish lists.