Server MVC and Solving the Wrong Problem

TL;DR MVC frameworks provide infrastructure to solve make believe problems. They complicate what should be simple.

Update (Feb 4, 2016): While writing this post focused on server-side MVC, I came across a somewhat related post, Why I No Longer Use MVC Frameworks, discussing problems with MVC on the client-side. It follows a slightly different direction and is well worth your time.

Background

In 2014, I presented a talk on F# on the Web at many user groups and conferences. One of my primary goals for the talk was to talk about F# adoption at Tachyus and show off just how easy it is to use F# for web development. After I presented the talk at CodeMash in January 2015, I put the talk on the shelf.

By January 2015, I had started contributing to Freya with Andrew Cherry and wanted to push the web machine-style approach, which I’ve found to be a far better solution. In the meantime, at Tachyus, we started using an in-house tool called Gluon, which generates a strongly-typed (and tightly coupled) TypeScript client for F# API definitions. These different approaches kept bringing me back to thinking how poorly the MVC paradigm fit server-side applications.

Model – View – Controller

With MVC, you are supposed to separate Models, Views, and Controllers. In most web MVC frameworks, models relate to some materialized version of your data store plus additional DDD or other machinery. Views are represented as HTML and, perhaps, ViewModels, though some like to lump ViewModels in with the Model piece of the puzzle. Controllers are almost always a replica of the RESTful Rails Controller pattern, a class containing methods the represent endpoints and return Views. For the purposes of our discussion, let’s assume that a View may be any serialization format: HTML, XML, JSON, etc.

Aside: I’m aware I just lost some of you. HTML as a serialization format? Yes, it’s just declarative markup and data, just the same as other XML, JSON, etc. formats. That it may be rendered by a browser into a user interface is not relevant, as many applications I’ve seen do use HTML as a data communications format b/c of its hypermedia support.

I think it’s worth noting that MVC started life as a UI pattern in Smalltalk. There was no server in sight. Stephen Walther wrote about The Evolution of MVC at the advent of ASP.NET MVC. If you are unfamiliar, please read it, and I’ll spare you my rendition here.

We’ve already uncovered that we have at least one rogue element floating around in our pattern: the ViewModel. “But wait; there’s more!” We can’t forget the real controller of our application: the Router. The Router is really like a Controller since it intercepts all incoming requests and dispatches to appropriate handlers. And this is really where it starts to break down. Note the terms I just used. I didn’t use “route to controllers, which then determine the appropriate methods to call.” I used entirely different terminology. If you break down most of these MVC frameworks, you will find something similar: a meta-programming mechanism for deconstructing all these pattern artifacts and turning them into handlers available for dispatch from the router. Don’t believe me? Look in your call stack the next time you debug one of these applications. You’ll see a glorious stack of meta-programming calls instead of the few, clean calls to your class-based implementation.

 

Router – Handler – Formatter

Let’s switch to the terms we actually use in describing what happens. I’ve listed them in the order they intercept a request, though we could switch them around and list them in the order in which they return a response. Isn’t that interesting? These pieces actually form a pipeline:

Request -> Router -> Handler -> Formatter -> Handler -> Router -> Response

This pipeline is listed in terms of handing off control. Most likely, your framework is tuned to let the Formatter write the formatted response directly back to the socket. However, many frameworks, such as ASP.NET Web API, let you return a value that the framework will later serialize and write to the socket. In other cases the framework may buffer the socket actions so it can intercept and manipulate headers, etc.

Router

As I was building the presentation I linked above, I started writing minimal implementations of web apps in F# for TodoBackend, a site showcasing server-side Todo application APIs. I wrote examples in ASP.NET Web API, a little DSL I wrote called Frank, and an implementation using only the Open Web Interface for .NET (OWIN). The OWIN implementation was intended to show the maximum LOC you would have to write. I was surprised that it was rather close to the other implementations, longer by only 30-40 LOC.

The OWIN implementation surprised and delighted me. I was amazed at how simple it was to write a for-purpose routing mechanism without a proper Router implementation. I had built a light wrapper around the ASP.NET Router for Frank, but the combination of F#’s Active Patterns and System.ServiceModel.UriTemplate provided a nice, explicit, and still comparatively light and powerful mechanism for composing a Router.

Aside: Please note I’m not stating we should absolutely throw away all routing libraries. I merely want to point out that 1) what we are doing in web applications is primarily routing to handlers and 2) routers are not all that complicated, so you shouldn’t think you _need_ a routing library.

FWIW, were I to recommend a routing library to F# web developers, I would suggest [Suave](https://suave.io/). I’ve really come to enjoy its flexibility.

Handler

We’ve covered the non-MVC part of MVC frameworks. What about the others? I’m not entirely certain about the Model aspect, to be fair. I think the ASP.NET MVC team took a good approach and left it out of the core, which is interesting in that they really provided a VC framework. Already the “MVC” pattern has broken down.

What about Controllers then? Controllers are typically classes with handler methods, as we said above. There’s that Handler word again. It’s worth noting that the core of ASP.NET contains an IHttpHandler interface that ASP.NET MVC builds on, as well. So let’s take a deeper look at a Handler, shall we?

The simple OWIN router I showed above calls simple functions. Here they are:

If you were to ignore all the machinery built into Controller base classes to support the meta-programming-heavy MVC pattern, you would find something like this at the bottom of the call stack. Ultimately, you need to pass in some things from the Request, look up some data, transform it to a representation, and return it with headers, etc. It’s all relatively simple, really.

However, there’s a trade-off to using simple functions: they are now easily re-used in different parts of a router. I don’t particularly find this a problem, but if you were to shove hundreds of these functions next to each other in a single file, you might run into a maintenance problem since the function signatures are almost all identical.

I think the class-based approach has some nice benefits, though, I do prefer the function-based approach. With a class, you can collect related things together and take advantage of compilers to prevent re-using the same method name with similar parameters. Unfortunately, almost all real MVC frameworks work around these potential safety measures with their meta-programming capabilities and use of attributes. Oh, well.

Aside: I’m sure some of you are wondering about the model binding support that turns query string and form parameters into typed arguments. I don’t really know where those fit in this, so I’m adding them to this aside. While model binding seems to work okay, I can’t think of an application where I didn’t run into problems with it. I found it easier and hardly more effort to manually deserialize query strings and payloads using simple functions. These are often quite easy to reuse across multiple handlers/controllers, and you again gain explicit control. YMMV.

Formatting

Formatting, or rather Content Negotiation, replaces the view layer. HTTP stipulates that the server returns a Representation of a Resource but not the Resource itself. In the case of a file, the Representation will likely be a copy. In the case of a dynamically generated response, the Handler serves as the Resource responds to a request using a formatter to render the requested output. While the most common response used to be HTML — and thus a View — many responses now return XML, JSON, or a myriad of other formats. Picking only one may be a pragmatic choice, but it’s really limiting in a framework. Note we have not even addressed the various special cases of JSON and XML formats that provide more meaning, links, etc. and don’t work with any one formatter.

Then again, you’ll find times when a single format and a general serializer is all you need. The example app I’ve described above serves only JSON and has serialize and deserialize functions that just call Newtonsoft.Json.JsonConvert:

What happened to all my talk of Content Negotiation, etc? It’s still there. Were I to realize I needed to support a client that wanted XML, I could add a pattern match on the Accept header and call different formatting functions. But why bother setting up all that infrastructure when I don’t need it? I’m not arguing for an ivory tower here. I just want to point out that HTTP supports this stuff, and MVC does not.

In short, formatting is an area that often severely limits a framework’s flexibility by choosing to make easy a few blessed formats and mostly forsaking the rest.

I think it’s also worth noting that writing a general-purpose serializer is difficult and requires a lot of work. Writing a bit of code to generate string formats from known types requires a bit of boilerplate but is rather trivial.

Conclusion

I want to make it clear I’m not trying to dump on all frameworks. While I do prefer composing applications from smaller libraries, I think frameworks can be very useful, especially when properly used to solve the problem for which the framework was designed. However, the MVC style frameworks fell off the rails (pun intended) long ago and suffers from a poor abstraction layer. A pattern or framework should provide abstractions that relate to their domain and solve common problems. Re-purposing a pattern name because it has some popularity is asking for trouble and will ultimately prove limiting. MVC is the reason so many people think of HTTP and REST as a form of CRUD over a network connection and have little to no understanding of the richness of the HTTP protocol.

MVC does not fit HTTP.

Image Credits: ASP.NET Web API Tracing (Preview) by Ron Cain on his blog

Advertisements

Web APIs vs RPC

I keep finding a lot of developers who like to model and build web apps using HTTP as a transport for RPC. I don’t understand this but I thought I would do my best to provide a framework for understanding why one might choose to use the HTTP protocol rather than ignoring it and layering RPC on top.

I’m not terribly surprised that many developers find it hard to want to follow HTTP. The current frameworks do little to help with the HTTP protocol. Most server-side frameworks give you some sort of MVC approach that only represents HTTP methods, URIs, and possibly content negotiation. I hesitate to include the concept of HTTP resources, as most of those frameworks prefer to ignore the strict concept and use something conceptually closer to a database table. Those brave, few frameworks that do provide additional support of HTTP typically leave the details to the developer. While this may appear to be a good thing, freedom to define the protocol semantics ignores the fact that the HTTP RFCs provide a well-defined processing model. Why don’t more frameworks provide the tools developers need to leverage the HTTP processing model?

Is it any wonder that naysayers might easily argue that HTTP-based applications look very similar to RPC, perhaps adding additional complexity to hide RPC under a thin veneer of URI templates and HTTP methods?

I imagine it’s a surprise to many people to read the abstract from the 1992 definition of HTTP:

HTTP is a protocol with the lightness and speed necessary for a distributed collaborative hypermedia information system. It is a generic stateless object-oriented protocol, which may be used for many similar tasks such as name servers, and distributed object-oriented systems, by extending the commands, or “methods”, used.

If this were more obvious, don’t you think developers — the majority of whom are in love with OOP — would want to embrace this protocol? The history of the RPC-over-HTTP trend appears to stem from the fact that HTTP, running on port 80, was typically a reliable port on which to send network traffic because most environments opened port 80 by default and infrastructure teams were a bit picky about opening up random ports for custom applications.

How might one approach HTTP-as-distributed-OO protocol? To start, I could imagine a client proxy to a server object. The client object looks and acts like the server object but makes all its calls over the network. Would you want to expose each of the object’s methods as HTTP method extensions? Well, you don’t really need to do so. The HTTP protocol defines POST for most purposes, and you can include any additional details such as the object’s method name, much as you might when using .send in Ruby. For those who favor a more type-safe approach and use a language like F#, you might find a sum type, or discriminated union, useful:

If this is starting to sound like SOAP and WS-*, you are not wrong. The thing is, SOAP was not terribly bad. SOAP and WS-* went wrong in layering additional protocol semantics on top of HTTP, which already has built-in support for many of those additional pieces. However, I’m not voting for a return to SOAP (though there are some aspects to tools like WCF that would be quite nice to have).

Wouldn’t it be nice to be able to declare method characteristics such as idempotency and safety and allow tooling to correctly negotiate that the appropriate method should use one of GET, PUT, or DELETE instead? What if you could define a graph of resources that know each other exist (by means of hypermedia) and that could correctly coordinate distributed workflows that would again be able to have correct client proxy representations driven through hypermedia?

This is still only scratching the surface of HTTP, of course. Do you really care about status codes? Headers? I bet you don’t. You may claim to care, but ultimately you probably care b/c most frameworks you use either do it wrong or entirely ignore. Consider a tool that would take this distributed OO style and then correctly negotiate representations for your client app, take care of conflicts, deal with cache invalidation, etc. Wouldn’t that be nice? Such tools exist. Take a look at webmachine and liberator. I’m working on one now for F#/.NET called Freya with Andrew Cherry. Granted, I don’t think any of those tools tackle the concept of a distributed OO framework, but that should be something we can build on top.

Taking this a bit further, wouldn’t it be nice to be able to generate both client and server boilerplate off of a central design document or even a media type definition? Swagger offers something like this. I think we can do more in this direction, and I’m excited to see where we land.

What do you think? Is this a lot of nonsense, or should we keep digging back into HTTP and attempting to leverage it more? I’m also interested in your thoughts as to whether other protocols, such as Web Sockets and Service Bus layers will eventually replace HTTP as our dominant protocol (or at least network port).

Special thanks to Darrel Miller for his review and feedback on this post.

New Names for Old Things

[This is the third in a series started long ago on the use of MVC for building web “applications”.]

I’m glad I’m only getting back to this series now. I’ve had an opportunity to build many more web applications and have a much better appreciation for the poor terminology used to define web applications. For starters, this MV? business silly. We’ll get to that.

I know I’m a bit of an extremist in some things. Specifically, I like things to mean what they mean. When we abuse terms, we don’t communicate well. REST. There, I said it. I feel better. Stop using the term. Most people have a wrong idea of what it means b/c of all the silliness that has been done in its name. I don’t claim to know exactly myself. I don’t think it’s possible to rescue the term from the abuses heaped upon it. There, you see? I’m an extremist.

Now that we’ve covered that, on to MVC. I’m not sure who decided this was an accurate description for what happens on the server-side of the web, but it’s just flat wrong. As noted previously, HTTP uses a functional interface. It’s an IO-bound Request -> Response function. Can you use patterns on either side to help maintainability? Certainly! Just don’t confuse things. Let’s start with Views.

Views

What is a view?

The [view or viewport] is responsible for mapping graphics onto a device.
A viewport typically has a one to one correspondence with a display surface
and knows how to render to it. A viewport attaches to a model and renders
its contents to the display surface. In addition, when the model changes,
the viewport automatically redraws the affected part of the image to reflect
those changes. […] there can be multiple viewports onto the same model and
each of these viewports can render the contents of the model to a different
display surface.

If a view was merely a serialization of a model, this would make sense for building web applications. Unfortunately, there’s a problem. The definition suggests that the view automatically updates whenever the model changes. How do you do that with HTTP? HTTP doesn’t define any mechanism for hooking up observation of a server model. Before you say JavaScript, consider first the current use of View, or even UI. People commonly mean HTML. HTML is not a UI. HTML is a serialization format. The client (normally a browser) must interpret that HTML. Many of you will remember when that wasn’t so standard.

Can we achieve MVC today? Possibly. You might be able to leverage web sockets to reach across a client/server architecture such as that presented by HTTP. However, you are more likely to find that “MVC” on the server is just limiting. You are typically better off building a sort of restricted data access service, a.k.a Web API (subtle hint). There’s really no point in trying to enrich a serialization format to make it work more like true MVC across the client and server.

Controllers

This is no different than routing. Instead of calling your Router a Controller, you split them up. However, most frameworks really just use the router as a top level dispatcher and the controller as a lower-level dispatcher. Otherwise, I’d say web frameworks stay a lot closer to the original meaning than a lot of the other MV? patterns. (Hence the ?, of course.)

Models

This really is the crux. HTML is a model. I noted this last time. It’s just a serialization of data you want displayed. It happens to be a lot richer, but it’s still just a data model. HTML is a great way to bootstrap an application that otherwise uses JavaScript as a model serialization format. If you want to disagree, ask why HTML5 removes the presentation elements. Why has layout and style moved to CSS? CSS and the browser define the actual rendering. In a no-script web application, you don’t have to build a view. You get it for free.

Conclusion

So what? Am I just ranting that I don’t like how people abuse terms? Possibly. However, I think this goes deeper. When you allow the slippery slope, you get caught on it, as well. It’s inevitable. The bigger, lurking danger is that we start to confuse useful patterns and use them in the wrong places. Many people use MVC frameworks today to build web APIs. However, that’s not MVC. So if you then switch to a desktop app to write MVC applications, you are either confused or delighted to find that it’s so much richer.

I don’t know what I would call what we build for the web; I know I wouldn’t call it MVC. In my experiments with Frank, I’ve found that writing simple functions and nesting them with rules makes a very easy mechanism for building web APIs. I think that would essentially just be a Command pattern. Simple, elegant, and very easy to understand. YMMV.

The Functional Nature of Web API

Since my last post on the functional nature of the web, I’ve given two presentations describing this idea in more detail. Unfortunately, circumstances have conspired against me such that I have been unable to build any good sample apps. That’s about to change, as I’ve started working on screencasts on Web API that I’ll be posting here and on fpish.net as I complete them.

As a preview, In this post I’ll describe the inherent functional nature of Web API. You have looked into Web API at all, you’ve probably seen the Contact Manager sample application, originally created by the infamous “2C”. (Kudos to Dan Roth on the Web API team for keeping it alive!) The controller-based goodness baked into Web API hides a lot of the functional nature, so I will understand if you think I’m a little crazy for my statement above.

See? Not so much? Okay, how about in F#?

type MyMessageHandler() =
    inherit System.Net.Http.HttpMessageHandler()
    override x.SendAsync(request, cancellationToken) =
        (* Implementation ... *)

Eh, let’s try again.

val app : HttpRequestMessage -> Async<HttpResponseMessage>
let app request = async { return response }

let makeHandler app =
    { new System.Net.Http.HttpMessageHandler() with
        override x.SendAsync(request, cancellationToken) =
            Async.StartAsTask <| app request }

Here I use several unique features of F# to emphasize the point. The type signature should be familiar from my last post. I then create a handler using F#’s object initialization syntax, which creates an instance from a base class using the app from above. I think I’ve made my point that, at its heart, the HttpMessageHandler is functional in its core.

Okay, let’s translate back to C#-land. We don’t have object initialization syntax, but we can create pure, functional approaches for C#. Also, we’ll use DelegatingHandler, as that’s the actual type you’ll need to inherit in order to use it in Web API. (I apologize now for making your eyes bleed with the type signatures.)

Why is this core?

I’ve mentioned this is core, but if you’ve only used Web API with controllers, you may never have interacted with HttpMessageHandlers. HttpMessageHandlers are at the root of everything. HttpClient, HttpServer, and everything else that uses these types all inherit HttpMessageHandler. If you look deep into Web API, you find this consistency of HttpMessageHandler at the core. What does that mean? It means “it’s [functions] all the way down.”

A quick example

You can see this in action in the AspNetWebApi.Conneg sample application.

Okay, so now we have our “function.” How would you hook this up? That’s also quite simple. HttpApplication now has a GlobalConfiguration courtesy System.Web.Http. This provides you access to an instance of HttpConfiguration, which provides you all the extensibility points for managing your Web API. The Self Host option let’s you create your own. Attaching a handler is dead simple:

The point

By no means am I trying to convince you to give up all the ease of built-in model binding and conneg. I only want to drive home the point that the core of Web API is really just functions. As I mentioned previously, HTTP exhibits a very strong functional side. Web API really addresses this head on and provides a very solid core for building web applicaitons, whether using a more MVC-style or functions directly.

Try it out. For really simple applications, you are likely to find that these are all you need. You may also appreciate that you can implement your own routing and other mechanisms directly within a composed set of functions. We’ll explore this more as we continue in this series, so stay tuned.

HTTP and Functional Programming

I made what some might consider a bold claim on Twitter this past Friday, that HTTP holds to the functional programming paradigm. For the purposes of demonstration, I’ll be using F# as a means of declaring function and type signatures, though you can use any language in a functional style. I’ll further use type names from Microsoft’s new Web API library. (Actually, I’ll be sticking to the System.Net.Http types, which are already generally available in the HttpClient NuGet package and will be part of .NET 4.5.)

The Basics

At the bare minimum, HTTP supports an interface in which a client submits an HttpRequestMessage and responds with an HttpResponseMessage. This contract can be represented as a function with the following signature:

HttpRequestMessage -> HttpResponseMessage

I could try to just leave it here and say, “See, it’s a function!” but that would be cheating a bit. Nevertheless, the essential contract for HTTP is a simple function. That’s at least something.

Summary

Let’s dive a bit deeper. What precisely might one mean by “functional programming paradigm”? Typically, you’ll see some variation on the following list (which is by no means exhaustive):

  • Declarative – what not how
  • Pure functions – a function with no side-effects such as I/O or mutating global state
  • First-class and higher-order functions – functions can be created and passed as parameters just like other values
  • Referential transparency – an expression can be replaced with its value without changing program behavior
  • Memoization – a performance enhancing technique made possible by referential transparency

Aligning the above items with ideas found in HTTP, we get:

  • Declarative -> HttpRequestMessage headers
  • Pure functions -> Safe, idempotent HTTP methods such GET and HEAD
  • First-class and higher-order functions -> Hypermedia controls and content negotiation
  • Referential transparency and Memoization -> Cache control mechanisms

Discussion

Declarative HTTP

This is the easiest to demonstrate. Request headers are nothing other than a means to declare what you want (e.g. Request Line) and set expectations (e.g. Accept). The entire HttpRequestMessage exists as a means of expressing intent. As a client, you have absolutely no chance of instructing the server on how to process your request. Even RPC-style calls must abide by this constraint, at least as it concerns the request message.

Pure Functions

Some may argue this point due to the prevalence of non-conforming web applications that allow side effects on safe, idempotent methods. Nevertheless, HTTP as it is defined specifies that GET and HEAD methods should be safe. This is quite important for other attributes of HTTP, such as the ability to cache representations, which I’ll discuss a bit further down.

First-class and Higher-order Functions

I will acknowledge that this is my weakest point. However, the fact that both the client and server are able to communicate “callable” options appears very close to the very mechanism used when supplying callbacks or continuations in functional programming. It’s close enough for me. You are free to disagree.

Referential Transparency and Memoization

Referential transparency, which is supported by the presence of pure functions (see above), allows us to safely support caching, known as memoization in functional programming. In functional programming languages, memoization allows us to offset the cost of immutability by calling a function once and then re-using the original value rather than continuously invoking the function over and over. It’s especially useful for expensive operations. In much the same way, HTTP provides the ability to cache representations at intermediaries to speed the process of returning a response.

What About Non-Safe-Idempotent and Non-Idempotent Methods?

Very few functional languages are considered pure, Haskell arguably the most popular. F#, OCaml, Lisp, Scala, JavaScript and others all support mutation. Truly, you would be unable to do very much without the ability to change state somewhere. Thus, these other methods are both useful and in no way against the idea that HTTP supports the functional style. Much as even Haskell can eventually persist data in mutable storage such as a database, HTTP supports controlled mutation (albeit without explicit monads; count your blessings).

Conclusion

I think the above arguments support my original idea pretty well, but I’m curious what you think. Also, I certainly am using this as a means of drawing attention to a project I’ve been working and churning on, related to both F# and Web API. In upcoming posts, I’ll introduce Frank, which is finally approaching a fairly stable api.

Web Architecture Done Right

I’ll go ahead and confess that a single right way to design for the web doesn’t exist. If someone wants you to believe otherwise, they are just wrong. That said, I do think that you’ll have a hard time going wrong by starting with one simple rule: start with a web api.

Why should you start with a web api rather than just building a web site/app that has HTML and JavaScript all working together? Frankly it’s because you can’t properly decouple the api from the serialization format well enough. When you are designing an api for the web, you (should be) thinking in terms of resources and representations. That’s representations in the plural form. You may not know exactly what forms you’ll need, but you should consider that you will eventually have many. When you start with a web site/app with HTML in mind, you’ve coupled yourself to a single format, and extracting that out later could (will) be difficult. Don’t just take my word for it. Mike Amundsen has an excellent post on the right way to think about these things.

There are certainly some instances where this may be overkill, but I hate rewriting software unless it really is a prototype or just an exploratory attempt to get something up. In those cases, go for the quick and dirty. If you are working on something you want to last a long time, however, you owe it to yourself to consider the evolvability of your app by focusing on api design. You’ll then be able to take advantage of a number of client options. Certainly, supporting the growing number of clients is one of the biggest challenges continuing to face developers as we head into 2012.

Let’s suppose you agree with me on this point. How do you go about building a really solid api design? I don’t think I could articulate it better than Darrel Miller already has. His goals for good apis are suitable both for internal teams and external customers. Who wouldn’t love gaining visibility b/c a customer was able to accelerate their business by using your api in an unforeseen way that drives additional business for your own company? How nice is it to knock out not just one project but several b/c you are able to leverage existing platforms for new projects? We’re doing that at Logos, in large part because we moved to MVC and took a more service-oriented approach to building our apps. The number of new projects has grown tremendously, but we are also able to respond much more quickly b/c the services are ready for consumption.

I’ll be continuing to discuss this topic in future posts. In the meantime, check out Mike’s book,
Building Hypermedia APIs with HTML5 and Node. It uses HTML5 and Node to illustrate, but the concepts are excellent and portable to other platforms. I also highly recommend REST in Practice as an excellent resource for understanding the fullness of what HTTP offers for building apis. Enjoy!