Onboarding API Sequence Diagram

State Transitions through Sequence Diagrams

This post is my contribution to F# Advent 2018. For years I’ve contributed here and there to a large number of projects, so it is hard to pick a topic. I decided to choose something that cuts across all my various hobby projects through the years and in which I recently found inspiration and practical value when designing software systems, specifically those portions of software systems that want to expose and/or enforce a correct sequence of user actions.

Motivation

I’ve built many systems where the user interaction design was either missing or delayed, both of which led to user frustration or confusion as to how to correctly interact with the software. I’ve also experienced this from both perspectives as a producer and consumer of software library APIs. Most people think it easier to expose all functionality to a user and provide documentation as to how the software should be used. In many cases, this is a pretty good trade-off. Other cases — especially those where a specific sequence of steps should be followed — can lead to misuse, bugs, and lots of frustration.

Many of these issues can be solved by simply thinking through and documenting the desired flow and then implementing that flow within the software. The latter part is often easier said than done, at least in my previous experience.

While at Open FSharp this year, I read a tweet by Mike Amundsen linking to his talk from RESTFest:

I didn’t get to see or hear the talk, but the slides and linked source code clued me in on the concept, which I found very compelling. Here’s my take:

Use sequence diagrams to design resource state transitions, i.e. workflows.

Mike’s slides show a slightly different direction, using the sequence diagram to identify resources, not states, and the transitions between them (though I could be misinterpreting his slides). I found that using each actor line as a state within a single resource made more sense for what I’ve needed to do at work, and the model is much simpler than writing a full-fledged Open API spec. Mike’s slides indicate he’s thinking along these lines, as well as the potential for generating the full specification, documentation, etc. from a simple sequence diagram.

Design

For this post, I’ll stick with Mike’s example so as to remain consistent and progress the conversation. The example is that of an onboarding workflow. I’m going to identify this as an onboarding resource. The first step, then, is to identify the states the onboarding resource may have at any given stage:

home
WIP
customerData
accountData
finalizeWIP
cancelWIP

We then need to identify all the transitions from state to state that we want to support. This is where we immediately diverge from what most API formats allow you to specify and where you can immediately find value from this process. (Note: the funny syntax for the arrows is the convention used by Web Sequence Diagrams (WSD) for specifying sequence diagrams in text format.)

home->+WIP:
WIP->+customerData:
customerData-->-WIP:
WIP->+accountData:
accountData-->-WIP:
WIP-->+finalizeWIP:
finalizeWIP->-home:
WIP-->+cancelWIP:
cancelWIP->-home:

The snippet above shows, for example, that we cannot navigate directly from home to accountData or WIP to home or customerData to finalizeWIP. There are specific transitions that are allowed from each state. Most libraries and frameworks for building applications give you the ability to specify all possible methods, not a means of limiting actions from a given state. This is sufficient to get our first glimpse of a visual representation.

However, we don’t currently have a means of instructing the resource when to move from state to state. We can do this by specifying messages for each transition.

home->+WIP: startOnboarding
WIP->+customerData: collectCustomerData
customerData-->-WIP: saveToWIP
WIP->+accountData: collectAccountData
accountData-->-WIP:saveToWIP
WIP-->+finalizeWIP:completeOnboarding
finalizeWIP->-home:goHome
WIP-->+cancelWIP:abandonOnboarding
cancelWIP->-home:goHome

With these additions, we can see the messages provided to transition from state to state.

Adding parameters provides further context and something we could conceivably call to produce results. The following follows Mike’s formatting from his slides.

home->+WIP: startOnboarding(identifier)
WIP->+customerData: collectCustomerData(identifier,name,email)
customerData-->-WIP: saveToWIP(identifier,name,email)
WIP->+accountData: collectAccountData(identifier,region,discount)
accountData-->-WIP:saveToWIP(identifier,region,discount)
WIP-->+finalizeWIP:completeOnboarding(identifier)
finalizeWIP->-home:goHome
WIP-->+cancelWIP:abandonOnboarding(identifier)
cancelWIP->-home:goHome

Running this specification through the WSD tool produces the following visual rendering, which I find very informative.

Onboarding API Sequence Diagram
Web Sequence Diagram

Implementation

You can get what I’ve presented thus far by reading through Mike’s slides. When I first read through them and visited his code repository, I was hoping to see an implementation of some of the additional directions he proposed. However, to date the repository contains only a CLI for generating the visuals above from the text specifications. With the intent of trying to drive this idea further, I ported the libraries to F#.

In order to show the capabilities of this approach, I planned on implementing several tools:

Unfortunately, life happens and I have not made as much progress as I would like. (Fortunately, I haven’t made too much progress, or you would be settling in for a very long post indeed.) I’m pleased to show a rough proof-of-concept using F# agents that I think demonstrates the utility of this approach nicely.

/// Defines a transition from one state to another state based on a message.
type Transition<'State, 'Message> =
    { FromState : 'State
      Message : 'Message
      ToState : 'State }

/// A resource-oriented agent that transitions the state based on messages received.
type Agent<'State,'Message when 'State : comparison and 'Message : comparison> =
    new : identifier:System.Uri * initState:'State, transitions:Transition<'State, 'Message> list * comparer:('Message * 'Message -> bool) -> Agent<'State,'Message>
    /// Returns the identifier for the agent.
    member Identifier : System.Uri
    /// Retrieves the current state and allowed state transitions.
    member Get : unit -> 'State * Transition<'State,'Message> list
    /// Posts a message to transition from the current state to another state.
    member Post : message:'Message -> unit
    /// Registers a handler to perform a side-effect, e.g. save a value, on a state transition.
    member Subscribe : transition:Transition<'State,'Message> * handler:('Message -> unit) -> unit

With these definitions, we can translate the WSD syntax into types and values.

type State =
    | State of name:string

type Message =
    | Message of name:string * data:string

let createAgent initState =
    let transitions = [
        // home->+WIP: startOnboarding(identifier)
        { FromState = State "home"
          ToState = State "WIP"
          Message = Message("startOnboarding", "identifier") }
        // WIP->+customerData: collectCustomerData(identifier,name,email)
        { FromState = State "WIP"
          ToState = State "customerData"
          Message = Message("collectCustomerData", "identifier,name,email") }
        // customerData-->-WIP: saveToWIP(identifier,name,email)
        { FromState = State "customerData"
          ToState = State "WIP"
          Message = Message("saveToWIP", "identifier,name,email") }
        // WIP->+accountData: collectAccountData(identifier,region,discount)
        { FromState = State "WIP"
          ToState = State "accountData"
          Message = Message("collectAccountData", "identifier,region,discount") }
        // accountData-->-WIP:saveToWIP(identifier,region,discount)
        { FromState = State "accountData"
          ToState = State "WIP"
          Message = Message("saveToWIP", "identifier,region,discount") }
        // WIP-->+finalizeWIP:completeOnboarding(identifier)
        { FromState = State "WIP"
          ToState = State "finalizeWIP"
          Message = Message("completeOnboarding", "identifier") }
        // finalizeWIP->-home:goHome
        { FromState = State "finalizeWIP"
          ToState = State "home"
          Message = Message("goHome", "") }
        // WIP-->+cancelWIP:abandonOnboarding(identifier)
        { FromState = State "WIP"
          ToState = State "cancelWIP"
          Message = Message("abandonOnboarding", "identifier") }
        // cancelWIP->-home:goHome
        { FromState = State "cancelWIP"
          ToState = State "home"
          Message = Message("goHome", "") }
    ]
    Agent(Uri "urn:agent:1", initState, transitions, function (Message(expected,_)), (Message(actual,_)) -> expected = actual)

The F# translation is a bit more verbose than the WSD text format, but the connection can clearly be seen. It’s important to note that the Message type splits the message name and parameters. I’ve kept the parameters as-is for now, but we could easily extend this to use the query parameter portion of URI Templates to specify an expected schema for the parameters against which to validate and extract arguments.

Testing

With these transitions defined, what do we expect to happen? In most apps and APIs, you have access to every supported method or interaction right away. However, we don’t want to expose everything; we want to expose a workflow and restrict user actions. Will the Agent provide the correct response for a given state?

The following tests verify that an Agent in the home state represents that it is in the home state and can only transition to the WIP state with the startOnboarding message.

test "agent starts in 'home' state" {
    let expected = State "home"
    let agent = createAgent (State "home")
    let actual, _ = agent.Get()
    Expect.equal actual expected "Should have been able to transition only to WIP."
}

test "agent can transition to 'WIP' from 'home'" {
    let expected = [
        { FromState = State "home"
          ToState = State "WIP"
          Message = Message("startOnboarding", "identifier") }
    ]
    let agent = createAgent (State "home")
    let _, actual = agent.Get()
    Expect.equal actual expected "Should have been able to transition only to WIP."
}

So far, so good. What happens if we transition to the WIP state?

test "agent transitions to 'WIP' after receiving a message of 'startOnboarding'" {
    let expected =
        State "WIP", [
            { FromState = State "WIP"
              ToState = State "customerData"
              Message = Message("collectCustomerData", "identifier,name,email") }
            { FromState = State "WIP"
              ToState = State "accountData"
              Message = Message("collectAccountData", "identifier,region,discount") }
            { FromState = State "WIP"
              ToState = State "finalizeWIP"
              Message = Message("completeOnboarding", "identifier") }
            { FromState = State "WIP"
              ToState = State "cancelWIP"
              Message = Message("abandonOnboarding", "identifier") }
        ]
    let agent = createAgent (State "home")
    agent.Post(Message("startOnboarding", ""))
    let actual = agent.Get()
    Expect.equal actual expected "Should transition to WIP state with 4 transitions."
}

The Agent represents that it is in the WIP state and can transition from WIP to four other states, just as we specified in our WSD spec. Here are a few more tests for good measure.

test "agent transitions to 'finalizeWIP' after receiving a message of 'completeOnboarding'" {
    let expected =
        State "finalizeWIP", [
            { FromState = State "finalizeWIP"
              ToState = State "home"
              Message = Message("goHome", "") }
        ]
    let agent = createAgent (State "WIP")
    agent.Post(Message("completeOnboarding", ""))
    let actual = agent.Get()
    Expect.equal actual expected "Should transition to finalizeWIP state with 1 transition to home."
}

test "agent transitions to 'home' from 'finalizeWIP' after receiving a message of 'goHome'" {
    let expected =
        State "home", [
            { FromState = State "home"
              ToState = State "WIP"
              Message = Message("startOnboarding", "identifier") }
        ]
    let agent = createAgent (State "finalizeWIP")
    agent.Post(Message("goHome", ""))
    let actual = agent.Get()
    Expect.equal actual expected "Should transition to home state with 1 transition to WIP."
}

test "agent transitions to 'cancelWIP' after receiving a message of 'abandonOnboarding'" {
    let expected =
        State "cancelWIP", [
            { FromState = State "cancelWIP"
              ToState = State "home"
              Message = Message("goHome", "") }
        ]
    let agent = createAgent (State "WIP")
    agent.Post(Message("abandonOnboarding", ""))
    let actual = agent.Get()
    Expect.equal actual expected "Should transition to cancelWIP state with 1 transition to home."
}

test "agent transitions to 'home' from 'cancelWIP' after receiving a message of 'goHome'" {
    let expected =
        State "home", [
            { FromState = State "home"
              ToState = State "WIP"
              Message = Message("startOnboarding", "identifier") }
        ]
    let agent = createAgent (State "cancelWIP")
    agent.Post(Message("goHome", ""))
    let actual = agent.Get()
    Expect.equal actual expected "Should transition to home state with 1 transition to WIP."
}

All the states return the expected representations.

Starting test execution, please wait...

Total tests: 8. Passed: 8. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.3494 Seconds

Conclusion

There are clearly a lot more directions in which we could take this. I’m very interested in writing the parser and generating a representation like this, as well as creating some similar implementations for things like Freya, Service Fabric, Azure Functions, etc. I was also surprised at how few lines were required for this implementation (Agent.fs is 73 lines, including white space). I’ve tried doing similar things in the past only to give up because I was overcomplicating the schema format, implementation, or something else.

I hope you’ll give the Sequence Diagram approach a shot, and I would love to know how it does or doesn’t work for you.

You can find all the code for this post at https://github.com/panesofglass/wsd-gen/tree/wsd-agents. Thanks for reading, and Merry Christmas!


Advertisements

Experiments with Modern Browser JavaScript Support

I recently stumbled across The Case for React-like Web Components by Andrew Rabon and was surprised to learn that browsers are supporting quite a bit of the new JavaScript and related HTML features. I found this quite timely as I’ve been struggling to update my company’s front-end build. (A bit on that later.)

I created a little repository to play around with ES6 module loading, Web Components, and a few other things, such as Proxy objects (especially with respect to Web Workers). I’m quite impressed with what’s possible with browser-only support. It seems there are others working out how to do something similar with React, as well.

You may be wondering why care about all this? We use React and TypeScript at Tachyus, and we currently use webpack to compile, bundle, and minify all our TypeScript on my project. Webpack is a great tool, but it’s configuration has been difficult to manage over time. With the release of webpack 4 a few months ago, I thought it would be great to see how minimal I could get our configuration. Unfortunately for our webpack configuration, we use handsontable wrapped in a React component. The work-around for using handsontable with webpack isn’t so bad, but wrapping it in React requires a loader that does not yet appear to work with the latest webpack.

These customizations also appear to block use of the react-scripts package, which provides a pre-configured webpack build for working with React. This is the package used by the handy create-react-app scaffolding tool. There’s also a TypeScript-specific set of scripts for working with React + TypeScript. If you don’t have a lot of customizations or are just starting out, I recommend using these tools. They are terrific.

The last thing I tried was using the fork-ts-checker-webpack-plugin. When used in conjunction with the ts-loader in transpileOnly mode, this plugin runs the type checker in a separate process, which dramatically increases build times. However, transpileOnly failed to load some of our scripts. I’m guessing we have something wrong with our paths, but I haven’t figured it out.

In any case, I am longing for the simplicity of simple tag loading, and it seems it’s close at hand. I suspect that using the browser module loading with HTTP/2 would actually solve most of the problems except for compiling the TypeScript and JSX.

I’d love to hear if you have tried moving from a build tool to ES6 module loading and how that’s worked out for you.

Response to “Open source in the F# community, part 1”

I just got back from the outstanding Open FSharp conference where I got to reconnect with and meet a bunch of people I’ve followed or with whom I’ve worked on various OSS projects. I’m energized, excited to contribute, etc. Yet, in the midst of the conference, the first tweets eventually leading up to Henrik Feldt‘s post appeared. I was and continue to be really sad about what he’s going through, and it took the edge off my enthusiasm (but only a little). Henrik has been a huge contributor and work horse in the community and created some incredibly valuable libraries. He’s also great to work with in OSS, and I am going to miss his presence.

However, I cannot agree with Henrik’s conclusions about the community. I do recognize he’s expressing some valid emotional turmoil. I have no problem with the abundance of options the community explores and think it’s a sign of a healthy community. I also think the catalyst to his decision is mostly based on misunderstanding, but I reserve the right to change my mind; I’m basing this exclusively on Twitter and a blog post and haven’t actually spoken to Henrik. I have spoken to several others in the past who saw their projects more or less hijacked by Microsoft. Henrik mentioned several of these: MonoRail, OpenWrap, etc. The project leads faced animosity and/or apathy from Microsoft as their projects were replaced and ideas taken without recognition of the work they had done or much inclusion in the process.

Microsoft

It’s important to remember, though, that wasn’t the Microsoft of today. Also, not all parts of Microsoft worked that way. I had the pleasure of working on an advisory board that helped Microsoft eventually release ASP.NET Web API. It was a lot of fun, and the Microsoft people involved were very conscious that what they were doing could ultimately replace some other frameworks and wanted to make sure everyone with similar offerings was able to be heard and offer input. I find the Microsoft of today works, generally, more like that, and I’ve often had very good interactions with them.

In the midst of that advisory project, I started working on OWIN with a bunch of the other advisory team members and a few people from Microsoft who wanted to be involved because they liked the idea. Most people don’t realize OWIN was fully conceptualized and developed by the community. I’ve presented on the evolution of OWIN in the past (original / more recent), and I always find people are surprised by this. Another surprise is how we landed on a Func<IDictionary<string, object>, Task> as the primary abstraction. (The short answer is that all framework maintainers are picky about syntax and wouldn’t budge and also didn’t want a common dependency). Microsoft fully embraced it and gave it life, which was very exciting. We, the community, successfully engaged Microsoft and saw them adopt a substantial change to their technology platform.

Then things began unraveling. Issues with IAppBuilder created a strain that led to misunderstandings and hurt feelings. The close ties we had formed working together on OWIN frayed, and Microsoft eventually became quiet even as we formed the first OWIN management team post-release. The ideas for ASP.NET Core were forming around this time, and those of us who were working closely on OWIN assumed ASP.NET Core would use OWIN as the core abstraction. It wasn’t, and there is a very good reason for this: performance.

Microsoft wants a fast platform for web projects. Their customers want a fast platform. Their customers, on the whole, don’t care about abstractions that let them compose things in sometimes bizarre ways, e.g. Nancy + Web API + Auth0 + enter-framework-here. OWIN, for all its good of no common dependencies and very simple and open protocol, also involves a lot of boxing and unboxing. There’s no type checking. It’s really not ideal. And I find no fault in Microsoft choosing a different direction. They followed a lot of the same style used in OWIN, and they made sure to support OWIN, which I still think is fair and fantastic. It would certainly be nice had they made sure all their newer security middleware “just worked” on OWIN, as well, but even their old security middleware often failed to adhere correctly to the OWIN spec, working only on the Microsoft.Owin libraries. To their further credit, they introduced Assembly Neutral Interfaces in early versions of ASP.NET Core to avoid having shared dependencies, but these eventually had to be scrapped.

I think Microsoft has and continues to improve its relationship with its community. The F# team, for example, is simply fantastic. I’ve enjoyed working with the ASP.NET team. Sure, there are some groups that have had more problematic relationships, but on the whole, I give Microsoft an A.

.NET Community

I mentioned some friction amongst the community members working on OWIN above. Much like that project, I’ve found the same sort of friction in the F# community while trying to find a common abstraction layer for web projects. In short, it didn’t happen. Here’s my own analysis, and I agree reserve the right to change my mind. I think people have very limited time and resources as it is, and they are doing their best to try things and share them. In some cases, e.g. Henrik, those solutions pick up steam and become popular. In the case of Suave, the syntax became more popular than the whole library, and some wanted to have the best of both great syntax and a fast server with extensions supported by a broader community. As a member of the other side of the spectrum, someone who has shared a lot of stuff but found much lower uptake, I don’t have a problem with that. I’ve even abandoned projects in order to help out with others I thought had more uptake. Maybe I’m an anomaly. (I doubt it.) I recall being asked on occasion to abandon my stuff and contribute to a framework so that there would be only one. I didn’t, in part b/c I don’t like the notion of there being only one option, but also because I enjoyed following my own experiments at that time and didn’t like the design of that other library. (I changed my mind later.)

That’s a long way of saying I don’t think we need to have only one solution. This isn’t Highlander. I realize there’s value in having a single, blessed solution as it’s easier for newcomers. However, in that case we should all just work on whatever Microsoft is doing. So I don’t buy it. I also appreciate that different people want different things, and as this is all OSS, while I often find some differences silly or frustrating (GitLink “porting” SourceLink in C# b/c they didn’t like F#), I have a hard time with getting mad at people for rebuilding tooling to meet their needs.

No grade here, just an ask: please, please, please ask maintainers about changes or use of their libraries. Since the catalyst here involves Giraffe and Suave, had Dustin first asked Henrik about changes to allow him to run the Suave syntax directly on Kestrel, perhaps there would be no issue here. Most likely, Henrik would not have wanted to make the move, but I never found him fundamentally opposed to separating Suave’s framework from the server. It just wasn’t a priority for him. I looked into it once but didn’t have the time to do it. That may not have satisfied everything Dustin wanted, but at least they would have had the conversation. Maybe they did. I haven’t spoken with Dustin either; I’m generalizing.

It always helps to ask first.

Example: Freya is the product of a few different frameworks, created b/c Andrew Cherry reached out to me. (Thanks to Marcus Griep for keeping it alive!)

Conclusions

I feel for Henrik. I’m sure I would feel as frustrated were I in his position. Actually, I don’t know if I could fully understand. I don’t support an eighth (probably a lot less) of the libraries he’s created and maintains. I never had the opportunity to pour so much into a library only to see a nearly exact replica succeed, and in the same language and runtime, no less. I’m very sorry to lose such a valuable and prolific member of the community from OSS work.

Yet I understand why Giraffe exists, and I agree with Isaac Abraham that choice is important. I don’t use Suave at work right now b/c I ran into trouble migrating onto the Suave server. It’s not insurmountable; I just didn’t have the time to work out the kinks, and it isn’t critical to our software. I’d love to use the Suave syntax, and having the ability to migrate pieces and when time permits would have been welcome.

I’m also full of hope for the future of F#. Open FSharp was full of great presentations, conversations, and workshops. Many new people are entering the community and bringing their enthusiasm and ideas. We celebrated a new batch of Community for F# Heroes, and we had a great turnout in terms of nominations and voting. If that’s a good metric, then I’m seeing a lot of positive momentum in the F# community right now.

The future is bright.

 

.NET Fatigue

For those that missed it, Ali Kheyrollahi wrote a post describing his “commentary on the status of .NET” in which he concludes that .NET is a platform for small- to mid-size projects that represent a declining market that will soon die and argues for .NET to pivot. His tweet triggered a very long discussion and at least one blog post in agreement by Rob Ashton, which triggered another long Twitter dialogue. The subsequent discussions center a lot around .NET Core, ASP.NET Core, and all the craziness around the meanings (if any) of beta, RC, RTM, and whether breaking API changes are allowed in each of these various stages. I think it’s safe to say the general feeling outside Redmond is that .NET Core 1.0 RTM is really .NET Core 1.0 alpha 1, and developers are back to waiting for the third release committing to new platforms. (That’s certainly not an absolute.)

The root of this battle, however, appears to stem from bitterness across several boundaries:

  • Microsoft vs OSS (which has arguable improved by leaps and bounds)
  • Enterprise vs ___? (really just the poorly-named “dark matter” vs not)
  • Web vs desktop
  • Windows vs cross-platform/Docker
  • Azure vs on-prem/other cloud
  • C# vs VB/F#/other
  • TFS + VS + NuGet vs alternative tool chain
  • fill-in-the-blank, e.g. SharePoint vs WordPress

Historically, Microsoft has set the tone, and most of their customers follow suit. Microsoft provided a lot of excellent tools to build applications quickly, and this was seen as a good thing. However, other platforms have stolen the leader role, leaving .NET to copy, borrow, and generally play catch-up. There are a few exceptions, but as can be seen above, most of those are typically given second fiddle and not promoted as well as they should.

I have been quite disappointed over the last year and a half to see how Microsoft managed their transition to OSS. With the exception of F#, a lot of the discussions around the new world of .NET Core have appeared to be a lot less about finding out what customers wanted and instead continuing to tell customers what they want. (This really only applies to a vocal minority and not to the team as a whole.) With the recent announcements about .NET Core 1.0 RTM, that seems to have been rectified to a large degree in that .NET Core will now have a significantly better chance to allow migration from (now) netstandard to netcore. I think this is one of the better moves Microsoft has made in the last few years.

Will this ultimately right the ship and ensure everyone currently using .NET remains aboard? I don’t know. Much like the recent post on JavaScript Fatigue, I think a lot of .NET developers are feeling .NET Fatigue and looking for something else. .NET Core was the future hope, and like it or not, the continuous breaking changes have broken the hopes of many. I gave up on caring at least six months ago. There are enough missing APIs that I can’t move to .NET Core any time soon, if ever.

In the F# community, several are looking for ways to cross-compile to other platforms. Fable compiles F# to JavaScript via Babel, and Dave Thomas did a bit of work to show how one might compile to Elixir. This isn’t restricted to F#, of course. Several projects exist to transform C# to JavaScript, and Xamarin has been transforming .NET code to iOS and Android apps for years.

This leads me to wonder whether the question behind the creation of .NET Core, which I think was something like “Can we make a tiny .NET that can run everywhere?” was the right question. Remember, .NET was created b/c Sun sued Microsoft for extending Java. A lot has changed since then. Java is now supported on Azure! While I think the CLR is arguably better than the JVM, I wonder whether a better solution might not have been to contribute toward the JVM and work toward merging the JVM and CLR. Perhaps Erlang/BEAM is the right target? WebAssembly all the things?

All of these are non-trivial. I’m not convinced any are good ideas. However, what if the right answer is somewhere along the lines of VS Code (or even VS with its Python, TypeScript, etc. tools), an editor for far more than Microsoft-owned platforms and languages.

Is .NET needed anymore?

What if C#, F#, and VB compilers were extended to target additional platforms, e.g. JVM, Erlang, etc? Would you care?

What if you could use TypeScript to build Windows applications rather than C#? Would you care? (For what it’s worth, I would rather write TypeScript than C#, so this is not a made up question.)

What if Visual Studio and VS Code focused on providing excellent tooling for traditionally CLI-driven tools in the JVM ecosystem and integrated seamlessly across the new Linux support coming in Windows 10? Would that impact you?

Is .NET Core a hope for the future, or yet another Silverlight- or Metro-like fad? Can it really succeed where Java failed? (NOTE: Erlang and node.js succeeded in cross-platform support, so it is possible.)

React: What WPF Should Have Been

I’ve been learning and using React, and I like it’s general approach to building UIs. I really like the unidirectional data flow, though I’m still on the fence about virtual DOM. I get the benefits and really like that it makes testing really easy. However, I am not sold on its being unquestionably better than directly manipulating the DOM or Google’s Incremental DOM. Nevertheless, React, Om, Elm, and others have proven it a viable and fast approach.

One of the best advantages of the aforementioned frameworks is they all allow you to compose custom, reusable elements or components building off of existing HTML elements. (I think WebComponents allows something similar, though I have not personally liked its direction.) Another advantage is that you can declare all these things within JavaScript or your transpiler of choice.

I liked many similar attributes of WPF and XAML when I worked on desktop software. At that time, I had just taken a job doing .NET consulting and was still learning .NET. I had mostly tinkered with VB6, PHP, Ruby, Python, SVG, and XForms to build desktop and web applications. I really enjoyed adding SVG and XForms into web apps to make them richer, though I never quite mastered either and, at the time, both required plugins that didn’t often play well together. XAML offered a chance to do something similar with desktop applications, and I enjoyed it a lot.

However, I worried that Microsoft had opted to build their own markup language when so many, similar languages already existed and offered a nice composition story. Also, I recall thinking that the previous strategy of walled gardens appeared ready to collapse. Fast forward a few years, and Web Standards were alive and well in the form of HTML5 (with SVG baked in) as Chrome and Firefox began picking up market share, and XAML was isolated to the Microsoft platform. A few other vendors had similar markup language efforts, all of which remained isolated to their own platforms.

Imagine, if you will, a different scenario. Had Microsoft been prescient enough to see the revival and adoption of HTML and SVG and built WPF on top of these, do you think things would have gone differently? XAML had features of XForms without the complexity. XForms didn’t survive, so it is possible that a XAML built on HTML and SVG with some of the XForms-like features could perhaps have been the baseline for HTML5. Microsoft developers would have a single UI platform to write web and desktop applications.

One can dream.

The one thing Microsoft didn’t solve was the unidirectional data flow. I’m not sure they would have solved that. Data binding was and still is a popular approach. I dislike it b/c I’ve so often been bitten by refactoring things and those refactorings not making it into the UI markup templates. However, XAML didn’t require the markup file; you could/can also create those elements in your programming language. I first encountered the unidirectional style in a sample F# WPF application. There is no JSX equivalent, but the unidirectional data flow, immutable values, and all the rest are certainly possible. Yet WPF is still primarily known more for its data-binding abilities and preferred MVVM than for its flexibility in allowing different styles of development.

I can imagine an alternative reality in which React was just a JavaScript implementation of something realized first by Microsoft in .NET. Unfortunately, that didn’t happen. Nevertheless, one could write a set of elements for WPF to mimic HTML and SVG and then layer on a version of React, were one of a mind to do so. Even if not, you may still find it useful to give F# a try in order to mimic the same unidirectional data flow you can find in React. You can see an example in the recent Community for F# recording with Phil Trelford on F# on the Desktop.

Server MVC and Solving the Wrong Problem

TL;DR MVC frameworks provide infrastructure to solve make believe problems. They complicate what should be simple.

Update (Feb 4, 2016): While writing this post focused on server-side MVC, I came across a somewhat related post, Why I No Longer Use MVC Frameworks, discussing problems with MVC on the client-side. It follows a slightly different direction and is well worth your time.

Background

In 2014, I presented a talk on F# on the Web at many user groups and conferences. One of my primary goals for the talk was to talk about F# adoption at Tachyus and show off just how easy it is to use F# for web development. After I presented the talk at CodeMash in January 2015, I put the talk on the shelf.

By January 2015, I had started contributing to Freya with Andrew Cherry and wanted to push the web machine-style approach, which I’ve found to be a far better solution. In the meantime, at Tachyus, we started using an in-house tool called Gluon, which generates a strongly-typed (and tightly coupled) TypeScript client for F# API definitions. These different approaches kept bringing me back to thinking how poorly the MVC paradigm fit server-side applications.

Model – View – Controller

With MVC, you are supposed to separate Models, Views, and Controllers. In most web MVC frameworks, models relate to some materialized version of your data store plus additional DDD or other machinery. Views are represented as HTML and, perhaps, ViewModels, though some like to lump ViewModels in with the Model piece of the puzzle. Controllers are almost always a replica of the RESTful Rails Controller pattern, a class containing methods the represent endpoints and return Views. For the purposes of our discussion, let’s assume that a View may be any serialization format: HTML, XML, JSON, etc.

Aside: I’m aware I just lost some of you. HTML as a serialization format? Yes, it’s just declarative markup and data, just the same as other XML, JSON, etc. formats. That it may be rendered by a browser into a user interface is not relevant, as many applications I’ve seen do use HTML as a data communications format b/c of its hypermedia support.

I think it’s worth noting that MVC started life as a UI pattern in Smalltalk. There was no server in sight. Stephen Walther wrote about The Evolution of MVC at the advent of ASP.NET MVC. If you are unfamiliar, please read it, and I’ll spare you my rendition here.

We’ve already uncovered that we have at least one rogue element floating around in our pattern: the ViewModel. “But wait; there’s more!” We can’t forget the real controller of our application: the Router. The Router is really like a Controller since it intercepts all incoming requests and dispatches to appropriate handlers. And this is really where it starts to break down. Note the terms I just used. I didn’t use “route to controllers, which then determine the appropriate methods to call.” I used entirely different terminology. If you break down most of these MVC frameworks, you will find something similar: a meta-programming mechanism for deconstructing all these pattern artifacts and turning them into handlers available for dispatch from the router. Don’t believe me? Look in your call stack the next time you debug one of these applications. You’ll see a glorious stack of meta-programming calls instead of the few, clean calls to your class-based implementation.

 

Router – Handler – Formatter

Let’s switch to the terms we actually use in describing what happens. I’ve listed them in the order they intercept a request, though we could switch them around and list them in the order in which they return a response. Isn’t that interesting? These pieces actually form a pipeline:

Request -> Router -> Handler -> Formatter -> Handler -> Router -> Response

This pipeline is listed in terms of handing off control. Most likely, your framework is tuned to let the Formatter write the formatted response directly back to the socket. However, many frameworks, such as ASP.NET Web API, let you return a value that the framework will later serialize and write to the socket. In other cases the framework may buffer the socket actions so it can intercept and manipulate headers, etc.

Router

As I was building the presentation I linked above, I started writing minimal implementations of web apps in F# for TodoBackend, a site showcasing server-side Todo application APIs. I wrote examples in ASP.NET Web API, a little DSL I wrote called Frank, and an implementation using only the Open Web Interface for .NET (OWIN). The OWIN implementation was intended to show the maximum LOC you would have to write. I was surprised that it was rather close to the other implementations, longer by only 30-40 LOC.

The OWIN implementation surprised and delighted me. I was amazed at how simple it was to write a for-purpose routing mechanism without a proper Router implementation. I had built a light wrapper around the ASP.NET Router for Frank, but the combination of F#’s Active Patterns and System.ServiceModel.UriTemplate provided a nice, explicit, and still comparatively light and powerful mechanism for composing a Router.

Aside: Please note I’m not stating we should absolutely throw away all routing libraries. I merely want to point out that 1) what we are doing in web applications is primarily routing to handlers and 2) routers are not all that complicated, so you shouldn’t think you _need_ a routing library.

FWIW, were I to recommend a routing library to F# web developers, I would suggest [Suave](https://suave.io/). I’ve really come to enjoy its flexibility.

Handler

We’ve covered the non-MVC part of MVC frameworks. What about the others? I’m not entirely certain about the Model aspect, to be fair. I think the ASP.NET MVC team took a good approach and left it out of the core, which is interesting in that they really provided a VC framework. Already the “MVC” pattern has broken down.

What about Controllers then? Controllers are typically classes with handler methods, as we said above. There’s that Handler word again. It’s worth noting that the core of ASP.NET contains an IHttpHandler interface that ASP.NET MVC builds on, as well. So let’s take a deeper look at a Handler, shall we?

The simple OWIN router I showed above calls simple functions. Here they are:

If you were to ignore all the machinery built into Controller base classes to support the meta-programming-heavy MVC pattern, you would find something like this at the bottom of the call stack. Ultimately, you need to pass in some things from the Request, look up some data, transform it to a representation, and return it with headers, etc. It’s all relatively simple, really.

However, there’s a trade-off to using simple functions: they are now easily re-used in different parts of a router. I don’t particularly find this a problem, but if you were to shove hundreds of these functions next to each other in a single file, you might run into a maintenance problem since the function signatures are almost all identical.

I think the class-based approach has some nice benefits, though, I do prefer the function-based approach. With a class, you can collect related things together and take advantage of compilers to prevent re-using the same method name with similar parameters. Unfortunately, almost all real MVC frameworks work around these potential safety measures with their meta-programming capabilities and use of attributes. Oh, well.

Aside: I’m sure some of you are wondering about the model binding support that turns query string and form parameters into typed arguments. I don’t really know where those fit in this, so I’m adding them to this aside. While model binding seems to work okay, I can’t think of an application where I didn’t run into problems with it. I found it easier and hardly more effort to manually deserialize query strings and payloads using simple functions. These are often quite easy to reuse across multiple handlers/controllers, and you again gain explicit control. YMMV.

Formatting

Formatting, or rather Content Negotiation, replaces the view layer. HTTP stipulates that the server returns a Representation of a Resource but not the Resource itself. In the case of a file, the Representation will likely be a copy. In the case of a dynamically generated response, the Handler serves as the Resource responds to a request using a formatter to render the requested output. While the most common response used to be HTML — and thus a View — many responses now return XML, JSON, or a myriad of other formats. Picking only one may be a pragmatic choice, but it’s really limiting in a framework. Note we have not even addressed the various special cases of JSON and XML formats that provide more meaning, links, etc. and don’t work with any one formatter.

Then again, you’ll find times when a single format and a general serializer is all you need. The example app I’ve described above serves only JSON and has serialize and deserialize functions that just call Newtonsoft.Json.JsonConvert:

What happened to all my talk of Content Negotiation, etc? It’s still there. Were I to realize I needed to support a client that wanted XML, I could add a pattern match on the Accept header and call different formatting functions. But why bother setting up all that infrastructure when I don’t need it? I’m not arguing for an ivory tower here. I just want to point out that HTTP supports this stuff, and MVC does not.

In short, formatting is an area that often severely limits a framework’s flexibility by choosing to make easy a few blessed formats and mostly forsaking the rest.

I think it’s also worth noting that writing a general-purpose serializer is difficult and requires a lot of work. Writing a bit of code to generate string formats from known types requires a bit of boilerplate but is rather trivial.

Conclusion

I want to make it clear I’m not trying to dump on all frameworks. While I do prefer composing applications from smaller libraries, I think frameworks can be very useful, especially when properly used to solve the problem for which the framework was designed. However, the MVC style frameworks fell off the rails (pun intended) long ago and suffers from a poor abstraction layer. A pattern or framework should provide abstractions that relate to their domain and solve common problems. Re-purposing a pattern name because it has some popularity is asking for trouble and will ultimately prove limiting. MVC is the reason so many people think of HTTP and REST as a form of CRUD over a network connection and have little to no understanding of the richness of the HTTP protocol.

MVC does not fit HTTP.

Image Credits: ASP.NET Web API Tracing (Preview) by Ron Cain on his blog

Demand Driven Architecture or REST or Linked Data?

I recently listened to David Nolen‘s talk from QCon London conference from back in July called Demand Driven Architecture. Before continuing, you should have a listen.

Ready?

I really like a lot of things Mr. Nolen has done and really enjoy most of his talks and posts. I was less enthused with this one. I think my main hang up was his mis-representation of REST and resources. I get the feeling he equates resources with data stores. If you watched the video and then skimmed that Wikipedia page, you will quickly see that the notion of “joining” two resources is nonsensical. I think Mr. Nolen is really referring to that “pragmatic” definition that means POX + HTTP methods, which really would correlate well to data stores.

Real REST drives application state, so you would not need to join to anything else if using REST. His criticism of performance also misses the mark, for if you are using REST then you should also be carefully planning out and leveraging a caching strategy. REST isn’t appropriate for every application, but not for the reasons Mr. Nolen so casually dismisses it. Think about it this way: if REST is not suitable for performance or mobile devices, then you must also agree that all websites (not apps, just sites like Amazon.com) fail on mobile devices. That’s just absurd.

I can’t imagine anyone is surprised when Mr. Nolen mentions SQL. He’s just mentioned joins, and what else would a developer associate with that term? If you have hung around the web long enough, you may have heard of SPARQL, which is a query language for linked data. Linked Data and SPARQL never seem to have caught on, but they address at least part of the problem Mr. Nolen presents. A big part of Linked Data’s failing is lack of tooling and painful implementation (mostly related to RDF). Perhaps new tools like BrightstarDB will help turn things around.

Interestingly, Mr. Nolen’s solution, SQL, is strong at ad hoc querying but not so great at scalability. If the web, linked data, etc. were already addressed by SQL, then those technologies would not exist. I really don’t get the need for mis-representation and indirection here. (On a related note, Fabian Pascal recently posted on the “Conflation & Logical-Physical Confusion” surrounding mis-representing SQL as the relational model, which ties in well to this post.)

The only thing presented here is an alternative to REST where the client specifies dependencies for the server to fulfill. This is a flip of the REST style where the server drives application state and provides contracts via media types to a client-driven approach. This is a perfectly valid approach, and interestingly one where Linked Data could excel. Tools such as Datomic, GraphQL + Relay, and Falcor certainly look interesting and appear to work well for very large projects.

I have no doubt that any of these techniques, done well, provides excellent results. Tooling will likely determine the winner, for better or worse.