Response to “Open source in the F# community, part 1”

I just got back from the outstanding Open FSharp conference where I got to reconnect with and meet a bunch of people I’ve followed or with whom I’ve worked on various OSS projects. I’m energized, excited to contribute, etc. Yet, in the midst of the conference, the first tweets eventually leading up to Henrik Feldt‘s post appeared. I was and continue to be really sad about what he’s going through, and it took the edge off my enthusiasm (but only a little). Henrik has been a huge contributor and work horse in the community and created some incredibly valuable libraries. He’s also great to work with in OSS, and I am going to miss his presence.

However, I cannot agree with Henrik’s conclusions about the community. I do recognize he’s expressing some valid emotional turmoil. I have no problem with the abundance of options the community explores and think it’s a sign of a healthy community. I also think the catalyst to his decision is mostly based on misunderstanding, but I reserve the right to change my mind; I’m basing this exclusively on Twitter and a blog post and haven’t actually spoken to Henrik. I have spoken to several others in the past who saw their projects more or less hijacked by Microsoft. Henrik mentioned several of these: MonoRail, OpenWrap, etc. The project leads faced animosity and/or apathy from Microsoft as their projects were replaced and ideas taken without recognition of the work they had done or much inclusion in the process.

Microsoft

It’s important to remember, though, that wasn’t the Microsoft of today. Also, not all parts of Microsoft worked that way. I had the pleasure of working on an advisory board that helped Microsoft eventually release ASP.NET Web API. It was a lot of fun, and the Microsoft people involved were very conscious that what they were doing could ultimately replace some other frameworks and wanted to make sure everyone with similar offerings was able to be heard and offer input. I find the Microsoft of today works, generally, more like that, and I’ve often had very good interactions with them.

In the midst of that advisory project, I started working on OWIN with a bunch of the other advisory team members and a few people from Microsoft who wanted to be involved because they liked the idea. Most people don’t realize OWIN was fully conceptualized and developed by the community. I’ve presented on the evolution of OWIN in the past (original / more recent), and I always find people are surprised by this. Another surprise is how we landed on a Func<IDictionary<string, object>, Task> as the primary abstraction. (The short answer is that all framework maintainers are picky about syntax and wouldn’t budge and also didn’t want a common dependency). Microsoft fully embraced it and gave it life, which was very exciting. We, the community, successfully engaged Microsoft and saw them adopt a substantial change to their technology platform.

Then things began unraveling. Issues with IAppBuilder created a strain that led to misunderstandings and hurt feelings. The close ties we had formed working together on OWIN frayed, and Microsoft eventually became quiet even as we formed the first OWIN management team post-release. The ideas for ASP.NET Core were forming around this time, and those of us who were working closely on OWIN assumed ASP.NET Core would use OWIN as the core abstraction. It wasn’t, and there is a very good reason for this: performance.

Microsoft wants a fast platform for web projects. Their customers want a fast platform. Their customers, on the whole, don’t care about abstractions that let them compose things in sometimes bizarre ways, e.g. Nancy + Web API + Auth0 + enter-framework-here. OWIN, for all its good of no common dependencies and very simple and open protocol, also involves a lot of boxing and unboxing. There’s no type checking. It’s really not ideal. And I find no fault in Microsoft choosing a different direction. They followed a lot of the same style used in OWIN, and they made sure to support OWIN, which I still think is fair and fantastic. It would certainly be nice had they made sure all their newer security middleware “just worked” on OWIN, as well, but even their old security middleware often failed to adhere correctly to the OWIN spec, working only on the Microsoft.Owin libraries. To their further credit, they introduced Assembly Neutral Interfaces in early versions of ASP.NET Core to avoid having shared dependencies, but these eventually had to be scrapped.

I think Microsoft has and continues to improve its relationship with its community. The F# team, for example, is simply fantastic. I’ve enjoyed working with the ASP.NET team. Sure, there are some groups that have had more problematic relationships, but on the whole, I give Microsoft an A.

.NET Community

I mentioned some friction amongst the community members working on OWIN above. Much like that project, I’ve found the same sort of friction in the F# community while trying to find a common abstraction layer for web projects. In short, it didn’t happen. Here’s my own analysis, and I agree reserve the right to change my mind. I think people have very limited time and resources as it is, and they are doing their best to try things and share them. In some cases, e.g. Henrik, those solutions pick up steam and become popular. In the case of Suave, the syntax became more popular than the whole library, and some wanted to have the best of both great syntax and a fast server with extensions supported by a broader community. As a member of the other side of the spectrum, someone who has shared a lot of stuff but found much lower uptake, I don’t have a problem with that. I’ve even abandoned projects in order to help out with others I thought had more uptake. Maybe I’m an anomaly. (I doubt it.) I recall being asked on occasion to abandon my stuff and contribute to a framework so that there would be only one. I didn’t, in part b/c I don’t like the notion of there being only one option, but also because I enjoyed following my own experiments at that time and didn’t like the design of that other library. (I changed my mind later.)

That’s a long way of saying I don’t think we need to have only one solution. This isn’t Highlander. I realize there’s value in having a single, blessed solution as it’s easier for newcomers. However, in that case we should all just work on whatever Microsoft is doing. So I don’t buy it. I also appreciate that different people want different things, and as this is all OSS, while I often find some differences silly or frustrating (GitLink “porting” SourceLink in C# b/c they didn’t like F#), I have a hard time with getting mad at people for rebuilding tooling to meet their needs.

No grade here, just an ask: please, please, please ask maintainers about changes or use of their libraries. Since the catalyst here involves Giraffe and Suave, had Dustin first asked Henrik about changes to allow him to run the Suave syntax directly on Kestrel, perhaps there would be no issue here. Most likely, Henrik would not have wanted to make the move, but I never found him fundamentally opposed to separating Suave’s framework from the server. It just wasn’t a priority for him. I looked into it once but didn’t have the time to do it. That may not have satisfied everything Dustin wanted, but at least they would have had the conversation. Maybe they did. I haven’t spoken with Dustin either; I’m generalizing.

It always helps to ask first.

Example: Freya is the product of a few different frameworks, created b/c Andrew Cherry reached out to me. (Thanks to Marcus Griep for keeping it alive!)

Conclusions

I feel for Henrik. I’m sure I would feel as frustrated were I in his position. Actually, I don’t know if I could fully understand. I don’t support an eighth (probably a lot less) of the libraries he’s created and maintains. I never had the opportunity to pour so much into a library only to see a nearly exact replica succeed, and in the same language and runtime, no less. I’m very sorry to lose such a valuable and prolific member of the community from OSS work.

Yet I understand why Giraffe exists, and I agree with Isaac Abraham that choice is important. I don’t use Suave at work right now b/c I ran into trouble migrating onto the Suave server. It’s not insurmountable; I just didn’t have the time to work out the kinks, and it isn’t critical to our software. I’d love to use the Suave syntax, and having the ability to migrate pieces and when time permits would have been welcome.

I’m also full of hope for the future of F#. Open FSharp was full of great presentations, conversations, and workshops. Many new people are entering the community and bringing their enthusiasm and ideas. We celebrated a new batch of Community for F# Heroes, and we had a great turnout in terms of nominations and voting. If that’s a good metric, then I’m seeing a lot of positive momentum in the F# community right now.

The future is bright.

 

Advertisements

Server MVC and Solving the Wrong Problem

TL;DR MVC frameworks provide infrastructure to solve make believe problems. They complicate what should be simple.

Update (Feb 4, 2016): While writing this post focused on server-side MVC, I came across a somewhat related post, Why I No Longer Use MVC Frameworks, discussing problems with MVC on the client-side. It follows a slightly different direction and is well worth your time.

Background

In 2014, I presented a talk on F# on the Web at many user groups and conferences. One of my primary goals for the talk was to talk about F# adoption at Tachyus and show off just how easy it is to use F# for web development. After I presented the talk at CodeMash in January 2015, I put the talk on the shelf.

By January 2015, I had started contributing to Freya with Andrew Cherry and wanted to push the web machine-style approach, which I’ve found to be a far better solution. In the meantime, at Tachyus, we started using an in-house tool called Gluon, which generates a strongly-typed (and tightly coupled) TypeScript client for F# API definitions. These different approaches kept bringing me back to thinking how poorly the MVC paradigm fit server-side applications.

Model – View – Controller

With MVC, you are supposed to separate Models, Views, and Controllers. In most web MVC frameworks, models relate to some materialized version of your data store plus additional DDD or other machinery. Views are represented as HTML and, perhaps, ViewModels, though some like to lump ViewModels in with the Model piece of the puzzle. Controllers are almost always a replica of the RESTful Rails Controller pattern, a class containing methods the represent endpoints and return Views. For the purposes of our discussion, let’s assume that a View may be any serialization format: HTML, XML, JSON, etc.

Aside: I’m aware I just lost some of you. HTML as a serialization format? Yes, it’s just declarative markup and data, just the same as other XML, JSON, etc. formats. That it may be rendered by a browser into a user interface is not relevant, as many applications I’ve seen do use HTML as a data communications format b/c of its hypermedia support.

I think it’s worth noting that MVC started life as a UI pattern in Smalltalk. There was no server in sight. Stephen Walther wrote about The Evolution of MVC at the advent of ASP.NET MVC. If you are unfamiliar, please read it, and I’ll spare you my rendition here.

We’ve already uncovered that we have at least one rogue element floating around in our pattern: the ViewModel. “But wait; there’s more!” We can’t forget the real controller of our application: the Router. The Router is really like a Controller since it intercepts all incoming requests and dispatches to appropriate handlers. And this is really where it starts to break down. Note the terms I just used. I didn’t use “route to controllers, which then determine the appropriate methods to call.” I used entirely different terminology. If you break down most of these MVC frameworks, you will find something similar: a meta-programming mechanism for deconstructing all these pattern artifacts and turning them into handlers available for dispatch from the router. Don’t believe me? Look in your call stack the next time you debug one of these applications. You’ll see a glorious stack of meta-programming calls instead of the few, clean calls to your class-based implementation.

 

Router – Handler – Formatter

Let’s switch to the terms we actually use in describing what happens. I’ve listed them in the order they intercept a request, though we could switch them around and list them in the order in which they return a response. Isn’t that interesting? These pieces actually form a pipeline:

Request -> Router -> Handler -> Formatter -> Handler -> Router -> Response

This pipeline is listed in terms of handing off control. Most likely, your framework is tuned to let the Formatter write the formatted response directly back to the socket. However, many frameworks, such as ASP.NET Web API, let you return a value that the framework will later serialize and write to the socket. In other cases the framework may buffer the socket actions so it can intercept and manipulate headers, etc.

Router

As I was building the presentation I linked above, I started writing minimal implementations of web apps in F# for TodoBackend, a site showcasing server-side Todo application APIs. I wrote examples in ASP.NET Web API, a little DSL I wrote called Frank, and an implementation using only the Open Web Interface for .NET (OWIN). The OWIN implementation was intended to show the maximum LOC you would have to write. I was surprised that it was rather close to the other implementations, longer by only 30-40 LOC.

The OWIN implementation surprised and delighted me. I was amazed at how simple it was to write a for-purpose routing mechanism without a proper Router implementation. I had built a light wrapper around the ASP.NET Router for Frank, but the combination of F#’s Active Patterns and System.ServiceModel.UriTemplate provided a nice, explicit, and still comparatively light and powerful mechanism for composing a Router.

Aside: Please note I’m not stating we should absolutely throw away all routing libraries. I merely want to point out that 1) what we are doing in web applications is primarily routing to handlers and 2) routers are not all that complicated, so you shouldn’t think you _need_ a routing library.

FWIW, were I to recommend a routing library to F# web developers, I would suggest [Suave](https://suave.io/). I’ve really come to enjoy its flexibility.

Handler

We’ve covered the non-MVC part of MVC frameworks. What about the others? I’m not entirely certain about the Model aspect, to be fair. I think the ASP.NET MVC team took a good approach and left it out of the core, which is interesting in that they really provided a VC framework. Already the “MVC” pattern has broken down.

What about Controllers then? Controllers are typically classes with handler methods, as we said above. There’s that Handler word again. It’s worth noting that the core of ASP.NET contains an IHttpHandler interface that ASP.NET MVC builds on, as well. So let’s take a deeper look at a Handler, shall we?

The simple OWIN router I showed above calls simple functions. Here they are:

If you were to ignore all the machinery built into Controller base classes to support the meta-programming-heavy MVC pattern, you would find something like this at the bottom of the call stack. Ultimately, you need to pass in some things from the Request, look up some data, transform it to a representation, and return it with headers, etc. It’s all relatively simple, really.

However, there’s a trade-off to using simple functions: they are now easily re-used in different parts of a router. I don’t particularly find this a problem, but if you were to shove hundreds of these functions next to each other in a single file, you might run into a maintenance problem since the function signatures are almost all identical.

I think the class-based approach has some nice benefits, though, I do prefer the function-based approach. With a class, you can collect related things together and take advantage of compilers to prevent re-using the same method name with similar parameters. Unfortunately, almost all real MVC frameworks work around these potential safety measures with their meta-programming capabilities and use of attributes. Oh, well.

Aside: I’m sure some of you are wondering about the model binding support that turns query string and form parameters into typed arguments. I don’t really know where those fit in this, so I’m adding them to this aside. While model binding seems to work okay, I can’t think of an application where I didn’t run into problems with it. I found it easier and hardly more effort to manually deserialize query strings and payloads using simple functions. These are often quite easy to reuse across multiple handlers/controllers, and you again gain explicit control. YMMV.

Formatting

Formatting, or rather Content Negotiation, replaces the view layer. HTTP stipulates that the server returns a Representation of a Resource but not the Resource itself. In the case of a file, the Representation will likely be a copy. In the case of a dynamically generated response, the Handler serves as the Resource responds to a request using a formatter to render the requested output. While the most common response used to be HTML — and thus a View — many responses now return XML, JSON, or a myriad of other formats. Picking only one may be a pragmatic choice, but it’s really limiting in a framework. Note we have not even addressed the various special cases of JSON and XML formats that provide more meaning, links, etc. and don’t work with any one formatter.

Then again, you’ll find times when a single format and a general serializer is all you need. The example app I’ve described above serves only JSON and has serialize and deserialize functions that just call Newtonsoft.Json.JsonConvert:

What happened to all my talk of Content Negotiation, etc? It’s still there. Were I to realize I needed to support a client that wanted XML, I could add a pattern match on the Accept header and call different formatting functions. But why bother setting up all that infrastructure when I don’t need it? I’m not arguing for an ivory tower here. I just want to point out that HTTP supports this stuff, and MVC does not.

In short, formatting is an area that often severely limits a framework’s flexibility by choosing to make easy a few blessed formats and mostly forsaking the rest.

I think it’s also worth noting that writing a general-purpose serializer is difficult and requires a lot of work. Writing a bit of code to generate string formats from known types requires a bit of boilerplate but is rather trivial.

Conclusion

I want to make it clear I’m not trying to dump on all frameworks. While I do prefer composing applications from smaller libraries, I think frameworks can be very useful, especially when properly used to solve the problem for which the framework was designed. However, the MVC style frameworks fell off the rails (pun intended) long ago and suffers from a poor abstraction layer. A pattern or framework should provide abstractions that relate to their domain and solve common problems. Re-purposing a pattern name because it has some popularity is asking for trouble and will ultimately prove limiting. MVC is the reason so many people think of HTTP and REST as a form of CRUD over a network connection and have little to no understanding of the richness of the HTTP protocol.

MVC does not fit HTTP.

Image Credits: ASP.NET Web API Tracing (Preview) by Ron Cain on his blog

Community for F# Activity

My consistency with running the Community for F# over the last year or so has been lacking, to say the least. I’ve spread myself too thin working on various open source projects, mostly relating to pushing OWIN. Now that work is (mostly) done, I plan on re-focusing on driving the Community for F# early next year. I’ve already started trying to line up speakers for the first six months. Unfortunately, we’ll miss December and possibly even January, as Google changed their +Pages to My Business and dropped Hangouts support, so far as I can tell. I need to sort that out.

As exciting as that is to some of you, I am most excited about the new web application that is currently under development. It looks like a slightly broken version of the site you can currently find at http://c4fsharp.net/, but if you look through the issues, you will find a lot of exciting new features coming to you in 2016. Here’s a short list:

  1. Video feed – pull all F#-related videos from the YouTube and Vimeo channels into a searchable list
  2. User groups and Events – move the work Reed Copsey built to list F# user groups and events into a proper data store with an admin interface for people to add more; also, possibly pull events and groups from meetup.com and lanyrd.com
  3. Searchable dojo list – similar to the above for the list of dojos
  4. Community wishlist – allow users to submit and promote the things they would really like to see added to F# or its ecosystem; essentially an improved UserVoice
  5. Community ratings – add ratings or at least a “Like” button to every piece of content
  6. A fresh re-design of the site to better accommodate the content

Possibly the most exciting news is that this web app refresh will become a showcase for F# web development. I’ve started building the new web application with Suave and WebSharper, and I have plans to add Freya. I also started writing a GitBook that will explain how the site was developed and the decisions made in selecting each implementation, as well as why other technologies were not selected. I hope this book provides some guidance to those looking to build web apps with F# and struggling to understand how to proceed.

If you are interested in helping with any of these things, please reach out. Community for F# is driven by a handful of very busy volunteers, and we could definitely use more help. Not only do I have big dreams for improving the web app, but we need help managing the Twitter account, finding and scheduling speakers and hangouts, managing the much neglected LinkedIn page, and more.

I would like to especially thank Mathias Brandewinder and Reed Copsey for their help in expanding and pushing the Community for F# forward. I would also like to thank the presenters who have agreed to do Live Meetings and Hangouts with us.

Lastly, thank you to all the local user group organizers and presenters! We have been honored this year by those that allowed us to stream their content, especially the Portland F# Meetup Group, the San Francisco F# User Group, and the F# Seattle User Group. If you would like to partner with the Community for F# to stream your group’s meetings, please reach out.

Thanks for a terrific 2015, and here’s to an exciting 2016!

Bundling and Minification with Web Essentials

Pta.Build.WebEssentialsBundleTask A few months ago, the Tachyus web application used a C# + F# web application approach to separate the front-end HTML, CSS, and JavaScript from the back-end F# ASP.NET Web API application. With this configuration, we introduced Web Essentials to bundle and minify our CSS and JavaScript at build time within Visual Studio. To simplify deployments and better group related and shared code, we decided to merge the back-end with another Web API application, which used the F# MVC 5 project template. We originally tried using CORS, which worked great in almost every environment. Unfortunately, the one environment in which we ran into trouble was our staging/production environment. Since we are building an internal-only API, we haven’t spent the extra effort to make our APIs evolvable; therefore, we decided to just merge all three projects together into the F# web project. This worked rather well, except for Web Essentials. We abandoned Web Essentials, as well as any form of bundling or minification at that time.

Fast forward to last week: we again split the front-end application out into a separate, C# project. We did this for several reasons:

  • We ran into trouble trying to remote debug the F# web project
  • The project had grown to the point that it was quite large, and it made sense to separate the two for maintenance
  • It’s common even in other languages to separate the front-end into a separate folder or project as different teams are often responsible for the different apps, which is not quite our case but close
  • We wanted to clean up our Angular code so that we had less Angular spread throughout and more standard JavaScript; for this we wanted to use bundling again

I was able to add Web Essentials back into the solution since we were again using a C# project. However, this had its own challenges, specifically in the form of communication to the rest of the team that they would need to install Web Essentials in order for their updates to take effect.

Fortunately, my colleague Anton Tayanovskyy recently found and pointed me toward the WebEssentialsBundleTask, which is a MSBuild task that will run the Web Essentials transformations at runtime depending on the build configuration, i.e. Debug or Release. This tool provides explicit script references for Debug builds and a bundled (and minified, if desired) version for Release builds. It seems to only require the presence of a Web Essentials-style .bundle file to work, so I would expect this to work equally well with a F#-only solution, though I have yet to try that. The WebEssentialsBundleTask has its own issues, though. It will modify your index.html file whenever it runs, so you must make sure to revert changes you don’t want to keep. We rarely change our index.html file since nearly everything is built in the form of Angular directives or templates. Nevertheless, you should consider the cost to your own project.

You may wonder why we didn’t just build a simple FAKE task. After all, whitespace in JavaScript is relatively meaningless, so a very simple concat + remove could probably get the job done, especially since we use ; where applicable. I definitely considered this option, as well as creating new FAKE tasks built around a node.exe using tools like grunt.js or gulp.js. In the end, these all seemed like overkill with the availability of Web Essentials, at least until we had evaluated whether WE would work for our purposes. We are still evaluating. What are you using? Did you find this helpful?

F# Web Stack Group

I’ve recently seen an increase in the number of those interested in F# programming for web development. I don’t think this is a recent phenomenon; I just haven’t had the privilege of running into many of them previously. Nevertheless, we have decided to band together to provide a set of useful, well-designed libraries for web development in F#.

If you are interested in learning more, please join us in the F# Web Stack Google Group. I’ve added a list of current projects and efforts in the welcome message so that you can gain your bearings quickly. Current topics include merging various projects into a cohesive stack and planning one or more type providers to build out server and client boilerplate from API documentation.

Video

WebSharper UI Improvements

The WebSharper project has been making significant strides of late in the realm of building composable and reactive user interfaces, especially for the purpose of building SPA-style applications. You can find documentation and demos for WebSharper.UI.Next on its new site hosted on GitHub. Team members have also been blogging about how to build UIs with the new tools on the WebSharper blog.

Anton Tayanovskyy also joined Community for F# a month or so ago to describe the concepts behind the design of WebSharper.UI.Next and how it is different from some other popular approaches, including the virtual DOM approach used in Facebook’s React library. You can find the recording on YouTube.