Customising eWater Source (Part 3): Introducing Veneer

This is third and final article in a series on Customising eWater Source. The first article looked at the various motivations for customising Source, before exploring plugins and the Source Function Manager. Plugins are the most established customisation option and there is a lot of power available to the plugin developer. However, for many problems the plugin approach is not the best or most efficient option. The Function Manager is the most commonly used customisation option in Source and is well tailored to its niche of implementing custom operating rules and other aspects of behaviour modelling. The second article looked at two very different approaches to scripting Source: (1) using the Source external interface and (2) manipulating the running software components.

Within their respective niches, the Function Manager and the two scripting options present compelling alternatives to developing C# plugins for Source. However none of the alternatives covered previously are ideal for developing custom user interfaces for Source: to date, that has been the purview of plugins. Unfortunately, plugin development can be slow and tedious.

I’ll conclude this article and the series by exploring the respective niches of the various customisation options. When should you use a plugin and when should you use an alternative?

However, first I want to cover one more way to customise Source.

The bulk of this article is devoted to introducing Veneer, a new way to create custom front ends for Source using web technologies. Developing with Veneer is fast and the results can be very impressive.

Customising eWater Source (Part 2): Scripting Source

This is the second of a three part series of posts on customising Source. The first gave an overview of some of the options and went into detail on two of those options: 1) the traditional approach of building a plugin DLL in C#, and 2) the newer and very popular approach of building ‘Functions’ within Source itself.

Functions provide a great alternative to plugins for implementing bespoke model algorithms, but there are many other customisation situations not covered by Functions. In the first article, I gave my view on various problems with the plugin development model and suggested limiting our use of plugins to a small set of Source customisation problems. For that to work, we need alternatives, and in this post we explore the options provided by scripting environments.

I distinguish between two styles of scripting Source:

1. Scripting the Source command line tool, which provides a means to run models, manipulating inputs and parameters and subsequently retrieving results; as opposed to
2. Writing scripts, in a .NET scripting environment, that allows direct manipulation of Source models.

Scripting the command line tool (which I call ‘external scripting’) is more broadly understood at this point in time than ‘internal scripting’, which I believe is another very powerful technique to have in your tool belt. I’ll cover both here, with examples, and give pointers for finding out more.

However, first I want to clarify what I mean by ‘scripting’.

Customising eWater Source (Part 1): Thinking Beyond Plugins

Plugins seemed like a great idea in 2004.

That’s when the plugin concept was built into the then fledgling E2 product (that would go on to become eWater Source). In the scheme of things, very little has changed about the way plugins are built from that original version in 2004¹. Plugins are still created as .NET assemblies and almost exclusively in C#². In my mind, the most significant change is improved support for data persistence in plugins.

In many respects it is a Good Thing that the plugin mechanism has been so stable and is becoming more popular with Source users.

However, in many cases there are better options than building a plugin: Faster (cheaper!) and more flexible options.

This is the first of a three part series of articles exploring those options.

In this article, I’ll cover off on plugins and the Function Manager. Plugins are very powerful and in some cases a plugin is still the only option to meet a customisation need. However plugins suffer from a development model that is better suited to ‘big’ software development rather than the small to medium size initiatives that a customisation project usually requires. Of the alternatives to plugins, the Function Manager³ is the best understood by the Source community. The Function Manager is used by many users to provide custom rules and behaviours in simulations. The Function Manager fills a particular niche, but doesn’t cover many situations where plugins are commonly used: for example, in reporting.

In the second part of this series, I’ll explore some of the options for scripting with Source, including the common approach of writing scripts against the standard issue Source command line, as well as the less well understood options for scripting the internals of Source using a native .NET language. Many automation problems are well served by writing against the Source command line, and it can be quite simple to automate jobs when the command line is used through a library that handles the low level details. Scripting the command line is limited for the same reason that it is simple: The command line exposes a deliberately restricted set of operations, such as running a scenario and modifying meta-parameters. Scripting using a native .NET language, on the other hand, allows direct querying and manipulation of any aspect of the modelling system, and is a good option for automating tasks that change the configuration of a Source model, such as bulk manipulation of parameters.

In the third and final article in this series, I’ll introduce Veneer, a new technology, developed by Flow Matters, that provides an alternate customisation option based on standard web technologies. As its name suggests, Veneer is about putting a lightweight layer over an underlying Source model. Veneer itself is a Source plugin, but a plugin that, in many situations, cuts out the need for writing a new plugin. Rather, the customisation happens in HTML and Javascript, providing a very rapid turnaround on changes and the ability to draw upon a great many useful libraries from the web world. Veneer also illustrates a further point about customising Source: Our options are not set in stone and it is possible for anyone to create an entirely new customisation platform for Source.

Throughout this series we’ll look at what the various customisation options are best suited to and we’ll recap that in the end with a handy decision tree!

Now, before we go further, lets review the motivations for customising Source.

groovyFEWS - Pragmatic Scripting of FEWS configurations with Groovy

I’ve really enjoyed working with Delft-FEWS over the past 12 months or so, but it didn’t take long before I sought out an alternative to building the configuration in XML. That search led me to the Groovy language, which I’d not used before.

Don’t get me wrong, the XML configuration system for FEWS is incredibly flexible and on the whole its very well documented. Once you’re comfortable with editing XML (either in a text editor or a suitable GUI editor), you can build a sophisticated operational system, tailored to your needs. You will build a lot of XML configuration, and there is some complexity there, with a lot of inter-relationships between different configuration files. That said, once you’ve got the concepts in place, you can advance quite quickly.

The two main shortcomings that I saw were that (1) some aspects of the configuration seemed very tedious to develop in XML (whether by hand or with a GUI) and (2) there were various forms of duplication in the configuration, which is bad for maintainability. My FEWS XML configuration files didn’t obey the DRY principle – Don’t Repeat Yourself – and I wanted to fix that.

I was after a solution that would let me develop the configuration with the repetitive and tedious aspects automated. At the same time, I wanted the solution to be a natural fit for the way FEWS is configured. Put another way, I didn’t want my whizz-bang automation approach to complicate the development of the bulk of the system, where there isn’t much in the way of duplication.

After planting the issue in my subconscious, I was rewarded with a recollection of Neal Ford’s excellent book, The Productive Programmer. Among many other pearls, Ford’s book first exposed me to the XML authoring capabilities in dynamic languages, such as Ruby and Groovy.

In the end, Groovy ticked all the boxes and it’s now my preferred way to author FEWS configurations. I’ve released a simple tool for working with FEWS in Groovy. The tool is designed to be easy to pick up and use with an existing, XML based, configuration. Used well, it can make your FEWS configuration much more adaptable and easy to extend.

Source and FEWS (Together At Last?)

Delft-FEWS is a great system for bringing together different simulation models in an operational context. FEWS has been widely applied in Europe, Asia and North America and, increasingly, in Australia. eWater Source is a flexible simulation model for water resource systems, with the ability to model a wide range of runoff, supply, allocation and demand situations. Source has a significant installed user base in Australia and a growing user group in Asia and elsewhere.

I was first exposed to FEWS in late-2005 when I was working on the, then nascent, Source project. I was struck then by the apparent complementary nature of the two systems and the idea of a Source model running in an operational context inside FEWS was very appealing. In a case of better late than never, I’ve had several recent opportunities to integrate Source and FEWS.

SourceRy - Simple Potions for Complex Source Scenarios From R

SourceRy is a collection of R modules for managing lots of Source model runs: Setting up the runs, running them in parallel with available computing resources and post-processing the results.

Along the way, SourceRy allows for a powerful recasting of the scenario concept in Source.

Welcome

Hello and welcome!

Before we get underway, a few words of introduction: for me and for this forum.