Introducing The http-requests Library

I've recently put a lot of work into a new library I've named http-requests. I had previously written a project called jersey-request-builder, which is an HTTP client wrapper written on top of Jersey Client 1.x and implemented as a Grails plugin. Lately, I've been busy porting a bunch of Grails plugins from 2.x to 3.x, and I decided I wasn't very happy with how I had implemented the jersey-request-builder plugin and didn't want to put the effort into migrating it. In particular, I wanted to improve the following points.

  • The request builder code structure was fairly tightly coupled with Grails, and required some changes to make it work outside of that context.
  • It was even more tightly coupled with Spring Framework, and would not function as a standalone library.
  • Most of all, it was tightly coupled with Jersey Client, and other libraries could not be plugged in as the backend to the wrapper.

Those problems were my primary focus when I sat down and wrote the http-requests library. Now, the library is a set of standalone groovy JARs with no Grails or Spring Framework dependencies. It was written with modularity in mind, as Jersey Client 1.x, Jersey Client 2.x, and Apache HttpComponents Client are all supported as backends.

I also kept the things I liked best from the jersey-request-builder. In particular, most requests can be built with either a flat properties object or groovy DSL. Request entity converters can be plugged in to handle custom object types, and entity converters for common object types are built in. For more complicated requests, a powerful filter system is available, with several built-in filters to provide common functionality like request logging, content encoding, or different authentication schemes.

Additionally, since I'm a big Grails fan, there's a Grails 3.x plugin available that automatically handles loading the library, filters, and entity converters.

I've recently released version 0.1.7.BETA of the library. Previous to this version, there had been a lot of iteration on the codebase and the structure and interfaces were a bit volatile. With this release, all of the major things I wanted to bake into the library are complete and I expect the library to be quite a bit more stable. Assuming no issues are found with this release, I anticipate a non-beta release soon.

If you'd like to give the library a shot, its various JARs can be found on jCenter. For detailed instructions on how to install and use the library, the library is well documented.

This article is just intended to be an introduction of the library to the world and include some of my motivations behind it. I'll be posting more articles on how to use it, along with advanced use cases that are not covered in the documentation. As always, feedback is welcome, and issues will always be reviewed.

Simplifying Jersey Client with a builder-style syntax

I recently looked into an issue my friend was having with an HTTP call and its failure to correctly convert JSON. The conversion problem is partly due to the resource that he was hitting returning a newline before the actual JSON text. I found he was using HTTPBuilder, which provides a builder-style syntax to create requests. The syntax was easy to use, but the library itself is old and relies on old libraries.

I didn't feel comfortable with the library, so I looked into using the Jersey client to do requests since the JaxRS grails plugin we use provides it. Jersey itself works well, but the syntax can be a bit verbose and while you can chain functions together to avoid a bunch of lines of code, it must be done in the correct order due to the object types these functions return. To ease the transition to Jersey, I decided to write a builder-style class that hides the ugliness of Jersey client.

RequestBuilder

The RequestBuilder source can be found in this gist.

Using RequestBuilder

The RequestBuilder supports the GET, POST, PUT, and DELETE method, as well as query parameters, headers, and automatic JSON conversion. Each HTTP method also has a shortcut function that only requires a URL to hit.

Supported properties

  • uri - The request URI. This is the only required property.
  • query - A map of key/value pairs to use in the query string. These are escaped for you.
  • headers - A map of headers in key/value pairs.
  • form - A map of key/value pairs to use as an encoded form. This format is what web browsers use when posting web forms to a server. Note that if anything is set in here, the form will be used as the content of the request and body will be ignored.
  • accept - What content MIME-type to request back from the server.
  • body - Request body. Only useful with POST or PUT methods.
  • convertJson - Whether to automatically convert the response to a JSON map or list based on the content-type returned from the server. Defaults to true.
  • binaryResponse - If true, will not convert the response and instead return a byte array. Useful when you expect some binary data back from the server. Defaults to false.

Examples

// Get a person named Bud via query parameters
result = new RequestBuilder().get {  
    uri = 'http://example.com/person'
    query = [
        name: 'Bud',
        sex: 'Male'
    ]
    accept = 'application/json'
}

// Add a new person named John with an encoded form
result = new RequestBuilder().post {  
    uri = 'http://example.com/person'
    form = [
        name: 'John',
        sex: 'Male',
        age: 32
    ]
}

// Get a person named John via a RESTful call,
// using the shortcut function
result = new RequestBuilder().get('http://example.com/person/John')  

Error Handling

The response handler will look at the response status code, and return the content of the response with no exceptions thrown if the status code is in the 200 range. If anything else is encountered, a ResponseStatusException is thrown with both the status code and the content that would have been returned.

def result  
try {  
    // I'm expecting to get a 404 status back
    result = new RequestBuilder().get {
        uri = 'http://example.com/person/DoesNotExist'
    }
}
catch (ResponseStatusException e) {  
    println "Status code: ${e.status}"
    println "Response content: ${e.content}"
}

In Closing...

I believe this builder greatly simplifies making HTTP calls using Jersey. I know of the Apache Components library and how robust it is, but I used Jersey client merely because I was already using it. I hope this code helps someone, and just leave a comment if you have any questions or if I missed something.

vSphere 5 Home Lab Part 1: Introduction and Hardware

I've recently had aspirations to study and take the VCP 5 exam, and while reading material and doing sample tests is helpful, there's nothing like having your own lab environment to play around and tinker with. I didn't want to dump a whole lot of money into a powerful server with network storage, however.

With the release of vSphere 5, ESXi now makes it very easy to take virtualization to the next level by running ESXi as a VM guest. This allows you set up an entire virtualized infrastructure, including virtualized storage via iSCSI and NFS, and as many hypervisors as your hardware can support.

The goals for my lab setup were as follows:

  • Hardware must be cheap but high quality.
  • ESXi must support all the hardware, and visa versa.
  • Provide virtual, simulated network storage.
  • Support a clustered, balanced vSphere environment.

I know bullet item number 2 may seem obvious, but it never hurts to do a little research before buying hardware. It's always a good idea to check the supported hardware list on VMware's site, but a simple google search sufficed in order to check if the hardware I was thinking about buying would work.

Here's the hardware I ended up going with:

In order to support 64-bit guests, it is crucially important that the CPU support Intel VT, if you are using Intel processors. The case is about a square foot in size, so the physical footprint of the lab box is small enough to stick on a shelf or put in a corner. 16 gigs fills up both memory slots on the board, so I have plenty of memory to run several decent sized guests. The motherboard provides one 1Gb ethernet port, which is all we need for real network connectivity (we'll set up private virtual switches for features like vMotion that never touch a real network). The motherboard also provides a VGA output courtesy of the onboard graphics of the CPU, which saves me some money from not having to buy a small video card.

For hard drive space, I'm not doing anything fast or fancy. I will install the hypervisors, vCenter, and simulated storage on the 160GB drive, and make the 1TB drive my VM storage. I had the 160GB laying around the house, but had to buy the 1TB.

The cost for all this hardware came out to $540. I could have probably gotten the price down by going with lesser brand or refurbished hardware, but longevity is important to me and I trust these brands.

I won't go into details on how to put this hardware together, as building a system is quite easy. In my next article, I'll go into detail about how to prepare the hardware for ESXi, and how to install and configure ESXi.

JAX-RS and HashMaps

I started working with the Grails framework recently when I started working for my new company. We were writing an application that provided a RESTful service, and my team had opted to use JAX-RS to implement that service.

One of the frustrations I had while learning Grails had to do with some of the convenience Groovy provides. In particular, I ran into an issue where JAX-RS did not know how to handle maps, yet it handled lists (even with nested maps!) just fine.  That's certainly a workaround if I had to return a map, but it does not make sense to have an API call return a list of a map when only a single map makes sense.

The Problem

@GET
@Path("/api/test")
@Produces(["application/xml", "application/json"])
Response getTestCall() {  
    ok [name: "Bud", foo: "bar"]
}

JAX-RS uses annotations to set up resources.  The GET annotation tells the application that the function response to GET requests.  The Path annotation defines the url associated with the call.  Produces tells the application that the call will return either XML or JSON, depending on what the client requests.

The only line of code in the function is a great example of the grooviness that Groovy provides.  ok is a static method that is imported above this code that simply returns a Response object with the 200 OK HTTP status.  The function takes generic input (more on this soon) to use as the body for the response.  Groovy does not require you to use parenthesis for functions in some cases, as you can see in this line.  The notation after ok is what is used to create a new LinkedHashMap object.  Really convenient right?  So what happens when we hit this API call?

A message body writer for Java class java.util.LinkedHashMap, and Java type class java.util.LinkedHashMap, and MIME media type application/xml was not found

Out of the box, this code does not work.  Neither JAX-RS nor the Grails plugin include a message body writer that will convert a LinkedHashMap into XML or JSON.  If we were to add a set of brackets to the map, we have turned our map into a list containing a single map, and this works since a body writer that will convert a list object is included.  While this could be considered a viable workaround, a list implies multiple (possible) results, while for the specific call it does not make sense to return multiple items.

Click through to see the solution after the break.

The Solution

A message body writer is a component of JAX-RS that helps convert some input object type to the requested output type.  JAX-RS has several registered providers that do this.  It will look at each registered provider and determine if that specific provider will output the requested type to a string, and if it is able to convert the input object type.  So I merely had to write one of these providers.

The grails plugin makes it easy to register a new provider.  The documentation sets certain requirements for a new provider to magically work:

  • must be annotated with @javax.ws.rs.ext.Provider
  • must have a file name matching *Reader.groovy if the corresponding class implements javax.ws.rs.ext.MessageBodyReader
  • must have a file name matching *Writer.groovy if the corresponding class implements javax.ws.rs.ext.MessageBodyWriter

Also the file must be placed in a specific folder within your grails project: grails-app/providers (note that you will also need the directory structure defined by your package name, so grails-app/providers/com/budjb/util/jaxrs in my example). Simply drop the following code into HashMapWriter.groovy in that directory and the above call works as expected.

/**
 * Copyright 2012 Bud Byrd
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.budjb.util.jaxrs

import static org.grails.jaxrs.support.ConverterUtils.*  
import static org.grails.jaxrs.support.ProviderUtils.*  
import org.codehaus.groovy.grails.commons.GrailsApplication  
import grails.converters.JSON  
import grails.converters.XML  
import javax.ws.rs.Produces  
import javax.ws.rs.WebApplicationException;  
import javax.ws.rs.core.MediaType  
import javax.ws.rs.core.MultivaluedMap;  
import javax.ws.rs.ext.MessageBodyWriter  
import javax.ws.rs.ext.Provider  
import java.io.IOException;  
import java.io.OutputStream;  
import java.lang.reflect.Type  
import java.lang.annotation.Annotation

/**
 * JaxRS message writer supporting hash maps.
 *
 * @author Bud Byrd
 */
@Provider
@Produces(['text/xml', 'application/xml', 'text/json', 'application/json'])
class HashMapWriter implements MessageBodyWriter<Object> {  
    /**
     * Inject the grails application.
     */
    GrailsApplication grailsApplication

    /**
     * Returns the size of the converted text.
     * We return -1 because we won't be figuring it out.
     *
     * @return Size of the converted text.
     */
    public long getSize(Object t, Class type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        -1
    }

    /**
     * Determines if this writer supports the input and output types/objects.
     *
     * @return Whether this writer can do the conversion.
     */
    public boolean isWriteable(Class type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        if (!isXmlType(mediaType) && !isJsonType(mediaType)) {
            return false
        }

        HashMap.class.isAssignableFrom(type)
    }

    /**
     * Converts the input object to the requested output type.
     */
    public void writeTo(Object t, Class type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, OutputStream entityStream) {
        if (isXmlType(mediaType)) {
            def writer = new OutputStreamWriter(entityStream, getDefaultXMLEncoding(grailsApplication))
            def converter = new XML(t)
            converter.render(writer)
        }
        else {
            def writer = new OutputStreamWriter(entityStream, getDefaultJSONEncoding(grailsApplication))
            def converter = new JSON(t)
            converter.render(writer)
        }
    }
}

And that's it! Now you can return hash maps within a API service where it makes sense.