How to organise code in the large scale

BOSS_Great_Wall.jpg
Figure 1. The large scale structure of the universe - what should the large scale structure of code look like?

John R Spray

Last update: 2022-01-09

This online book is a work in progress. Feedback is welcome. Please send to johnspray274<at>gmail<dot>com


Introduction

Intuitively, good quality software would allow you to read and understand any one part of the code without also having to read and understand any other part. I call this zero coupling. We are taught the meme loose coupling and high cohesion. Unfortunately, this implies that zero coupling is not achievable. It is said that a zero coupled system would not do anything. But this is only because we confuse design-time coupling with run-time communications. It is entirely possible for parts of a system to have zero knowledge about one another at design-time and still communicate at run-time. We define the word coupling to mean design-time coupling throughout this book. That’s the coupling that matters. By design-time, we mean that any time you are reading, writing or understanding code in one module, how much do you have to know about code inside other modules? Modules in conventional code tend to have a lot of coupling in the form of collaboration. ALA converts that collaborative coupling into cohesion contained in a new module. Here’s how:

ALA is a set of three architectural constraints:

  • The only unit of code is an abstraction.

  • The only relationship allowed is a dependency on an abstraction that is significantly more abstract.

  • All abstractions must be small - as a guide, less then about 500 LOC.

Many patterns and properties emerge from these three constraints. These are explored throughout this book. But first let’s elaborate on these constraints.

The only unit of code is an abstraction

abstracttree

Abstractions are fundamental to the constraints. It’s important that we think in terms of abstractions rather than modules or classes, because abstractions are zero coupled with one another at design-time.

An abstraction is a learnable idea or concept. Once learned it becomes part of your language.

In conventional methods, a system is broken into smaller pieces that we call modules, components or classes. These modules, components or classes tend to have a fixed arrangement with one-another (although an individual piece may be substitutable). This fixed arrangement leads to implicit collaboration with one another to form the system. They are like jigsaw pieces. Specific pieces fit together in a specific way to make a specific picture. The system is not explicit, it is just the collaboration of all the individual parts.

In contrast, abstractions are like lego pieces. These pieces are general and can be composed in an infinite variety of ways. A specific arrangement is an explicit artefact in itself. The parts in a specific arrangment are not collaborating. What makes the system is the particular compositon of them. It can be varied without changing the abstractions it uses.

Abstractions have no knowledge of one another, nor the system that uses them.

All implicit collaboration between modules in conventional architecture becomes explicit code inside a new abstraction in a higher layer in ALA.

This first constraint means that the ALA equivalent of a module is an abstraction. The distinction is crucial. Modules that are not good abstractions only provide encapsulation. The term encapsulation means hiding details at compile-time, but not necessarily at design-time. Abstractions are what our brains use, so only abstractions hide details at design-time.

Abstractions are the only mechanism that provides design-time information hiding. When David Parnas coined the term information hiding he meant at design-time. Unfortunately there is a popular meme that information hiding means encapsulation, which only hide at compile-time. The same idea of design-time information hiding has other names such as Alistair Cockburn’s protected variations, and Robert Martin’s version of the OCP (open closed principle).

In the absence of direct computer language support for abstractions, ALA generally implements abstractions as source files (like modules in C). The file usually contains one class, but may contain a small number of classes, interfaces, enums, delegates, typedefs, functions, etc.

Inside an abstraction, there are no dependency rules, or any organisational rules for that matter. It is all considered cohesive code. So internally, abstractions are small balls of mud.

Charles Krueger pointed out that abstraction and reuse are two sides of the same coin. More abstract means more reusable. More abstract also tends to mean more stable because a more abstract idea is drawn from more examples, and is therefore a more ubiquitous, more fundamental idea. The abstraction 'squareroot' has been stable for thousands of years despite the fact that new instances of its use occur every day.

Software libraries tend to be good abstractions because they were written without knowledge of what will use them. If we conceive of and write abstractions in our applications similarly, without knowledge of the code that will use them, we will also get good abstractions. However, we also need to build useful abstractions. This will require some knowledge of what sorts of applications we are wanting to build. We call that the domain, and the abstractions are called domain specific abstractions.

The only legal relationship is a dependency on a more abstract abstraction

The second constraint is that dependencies must be on abstractions that are significantly more abstract (than the abstraction whose implementation has the dependency).

When we use a more abstract abstraction, for example, when a function that calculates standard deviation uses a function that calculates squareroot, the dependency is only on the concept of squareroot. As a concept, squareroot is stable and learnable. So a dependency on a more abstract abstraction is a dependency on a concept that is relatively stable. Because of this stability, there is zero coupling between the code that implements standard deviation and the code that implements squareroot.

For this to work, the abstraction being used must be significantly more abstract than the one whose implementation code uses it.

Communication between instances of peer abstractions

collaboration.jpg
Figure 2. Collaboration between abstractions is illegal

Seemingly, communication is needed between peer abstractions to make a system work. However, in ALA, a dependency from one abstraction on a peer is illegal. Even a dependency on the interface of another is illegal. For example, if the code implementing standard deviation were to, say, output the result directly to a peer module like a display, it would destroy the standard deviation abstraction. Even if the dependency is on the interface of the display abstraction, the code in standard deviation would still be coupled to a concept that is not more abstract. Standard deviation would no longer be an abstraction, no longer easily learnable as a concept, and it would no longer be reusable without dragging display with it.

Furthermore, over time, the fixed arrangement between the standard deviation function and the display function is likely to lead to the display function providing specifically what the standard deviation function needs, and vice versa. So both functions would be specific parts of a specific system. They would be collaborating modules.

So in ALA, abstractions do not communicate. Their instances do. The instances only communicate at run-time but for that to happen something else needs to connect them.

In ALA, there must be a more specific abstraction in a higher layer to represent the system consisting of an instance of a standard deviation abstraction connected to an instance of a display abstraction. This higher layer abstraction calls the standard deviation function and then calls the display function, or arranges for this to happen in some way. The higher layer abstraction can be thought of as representing the user story.

Classes with ports

In ALA we wont be restricting ourselves to pure functions. We will use classes as well. That’s because in ALA, when we need state, we consider abstraction to be more important than referential transparency (invariant behaviour over time.) Essentially we are using zero design-time coupling through use of abstractions as the tool to get analysability instead of removing time. Sometimes separating state from functions breaks otherwise good abstractions. An example would be if our standard deviation abstraction receives a stream of data and produces a running standard deviation result. In ALA, such an abstraction has it’s state (SumX, SumXX and N) stored with it, not passed into it. In other words, if state together with some methods makes a good abstraction, then we don’t break it.

Another example of the need for a class rather than a function might be our display abstraction. A display probably needs some configuration, for example to tell it how many decimal places to display. That kind of configuration is usually set once when the abstraction is instantiated, so the state should be stored inside the display abstraction.

ALA is actually highly object oriented. But it’s object oriented in a more disciplined way than conventional object orientation. That’s because peer classes may not have associations, period. In other words, an object’s communication with another object at run-time may not be implemented by one class having knowledge of another at design-time. Not even on an interface of a peer class.

There must be a more specific abstraction in a higher layer that instantiates two classes and arranges for them to communicate. This is more than dependency injection. Dependency injection, where one class knows about the interface belonging to another peer class, or a set of substitutable peer classes, is illegal in ALA.

Now we could have the more specific abstraction in the higher layer handle the communication itself by simply calling methods on each of the classes. For the example above, the user story abstraction in the higher layer could call a standard deviation method to get the result, and then call a display method to display it. But that would mean the user story abstraction would be handling the data itself, and driving the execution model. We don’t particularly like that. We prefer the user story abstraction’s job is just to compose the instances of the two peer abstractions, in a declarative style.

To do this, we build the classes with ports. We then simply wire the instances together by their ports. For example, we could make a standard deviation class with an output Dataflow port and a display class with an input Dataflow port. Instances of them can then be wired together by the user story abstraction in a higher layer.

It is important to understand that these ports must use a significantly more abstract interface that comes from an even lower layer. We call these more abstract interfaces programming paradigms. For example, the Dataflow programming paradigm can be used anywhere we need to pipe data in or out of an abstraction.

Classes can have multiple ports. Ports can use different programming paradigms, not just data flow. For example, the display class could also have a UI port. The UI programming paradigm would be about laying out the user interface. A wiring of UI ports means one part of the UI is displayed within another part. Instances of display could be put inside an instance of a panel. Other types of programming paradigms are invented as we need them in order to meaningfully express requirements. So now, while the standard deviation and display abstractions know nothing about each other, their instances can communicate directly at run-time.

All dependencies are knowledge dependencies

A dependency on an abstraction in a lower layer means you need to understand the concept the abstraction represents at design-time. We call it a knowledge dependency. You could not understand the code that calculates standard deviation without first knowing about the concept of squareroot. However, to understand the code that calculates standard deviation does not require knowledge of the concept of a display, so that’s why that dependency is illegal.

Going one layer up, to understand the code in our example user story abstraction requires knowledge of both the concept of standard deviation and the concept of display. It is therefore fine for the user story abstraction to have dependencies on both and compose a standard deviation and a display.

Abstractions that are more abstract become part of your language. We take this for granted when we use library abstractions, such as squareroot. In ALA, all lower layers make up the language for the next higher, more specific layer until we are able to express the application.

Good and bad dependencies

This is an alternative way of viewing ALAs dependency constraints.

dependencyimage.png

Conventional code contains both good and bad dependencies. Good dependencies are when we use abstractions, such as when we use a library class.

There will typically be very high numbers of bad dependencies in every application. These bad dependencies are usually put there for different modules of a program to inter-communicate. They are also used to break a large module up into smaller modules - hierarchical decomposition. These smaller pieces may start with a stated functionality, but they are pieces specific to the module they are in, and therefore more specific than the module.

Bad dependencies make the dependency diagram look like the one on the right. My partner says these look like two ferris wheels engaged in a mating act. Even if we manage to avoid cyclic dependences, there are still typically many bad dependencies.

In ALA, all bad dependencies are illegal. Normally in conventional code we don’t distinguish between good and bad dependencies. We consider them all to be necessary if the system is to work. But it turns out that systems can be built using only good dependencies.

Good dependencies are not just good - they are really good. We want as many of them as possible, because then we are reusing our abstractions. Bad dependencies are not just bad - they are really bad. They cause a growing tangled network of complexity.

Since bad dependencies are illegal in ALA, how do parts of the system communicate? Each bad dependency becomes a line of code completely inside a higher layer abstraction. That line of code connects two instances of the abstractions. There it is cohesive with other lines of code that connect other instances of abstractions to make a cohesive user story.

If you think about it, circular dependencies come about because there is circular communications. Circular communications is perfectly valid. When you stop representing communications with dependencies, and start representing them with wiring code, circular wiring makes perfect sense. When you use only good dependencies, the whole dependency management problem just goes away.

Nearly all relationships in the UML class diagram are bad dependencies. The only legal UML class relationship is composition (filled diamond arrow), and then only if the class being used is more abstract (in a lower layer). Now when you use abstractions you always refer to it by name, not use an arrow on a diagram. You wouldn’t draw a line on a diagram to a library function such as squareroot - you would just use squareroot by name as if it’s part of your language. Similarly, you wouldn’t draw composition relationships on a UML diagram of an ALA application. So, it turns out that if you drew a UML class diagram of an ALA application, you would just get disconnected boxes sitting in space. They would be arranged in layers, as described below, but there would be no lines at all. The UML class diagram is redundant in ALA. In fact the UML class diagram, by encouraging relationships between would-be peer abstractions, has probably caused more damage to software than any other software engineering meme. UML class diagrams are evil.

Metrics

Unfortunately metrics do not understand abstractions. For example, CBO (Coupling of objects), which, despite its name, is the total number of classes with dependencies to or from a given class, does not distinguish between good and bad dependencies.

ALA recognises the importance of using abstractions and composability in dealing with complex systems. If metrics don’t also take into account this importance, they cannot work well.

A problem when calculating CBO is whether or not to include library classes. Intuitively we know that library classes are abstractions. We see them as part of our programming language. So we feel they should not be included in the metric. The same should apply to all good dependencies - those that are dependencies on the knowledge of a concept in a lower layer. CBO should count only bad dependencies.

If the metric first identified all potential abstractions, and then identified their abstraction levels relative to each other, then it could just count the bad dependencies. CBO would then be an extremely useful metric.

Emerging layers

Layers.png
Figure 3. Abstraction layers

Because of the constraint that an abstraction that is depended on must be significantly more abstract, abstractions form layers. This gives the architecture its name: abstraction layered architecture. We give the layers names that reflect the types of abstractions that tend to go in them - application layer, domain abstractions layer, programming paradigms layer.

Each layer becomes a folder and a namespace in the implementation. This makes it very easy to know how to arrange our source files. The folders for each layer are not nested.

For large applications, another layer called the features layer or user stories layer comes into the picture between the application and domain abstraction layers. A specific application abstraction in the top layer then just composes a set of features or user stories needed in that specific application, and sets up any communication that may be needed between them.

There is no 'hierarchical' structure in ALA. In other words, abstractions cannot be contained within abstractions. There is no analog of a sub-module or sub-component - no such thing as a sub-abstraction. Abstraction layers replace hierarchical containment. This is because lower layer abstractions must be visible for reuse, not hidden inside an encapsulation hierarchy.

Once you have learned the set of concepts of the abstractions available in lower layers, it is easy to read and understand code in higher layers that use instances of them. Reading a line of code that uses an abstraction by name is like reading any other line of code. A good abstraction is when we don’t need to follow the indirection and go and read the code that implements the abstraction. For example, when we see a use of squareroot, if our brain has already learned the concept of squareroot once, we can concentrate on the code that is using the squareroot. We do not have to change contect to read any code. Squareroot is just part of our language. ALA is about achieving that level of readability for every single dependency in the entire application.

All abstractions must be small

smalltree.jpg

The final constraints prevents us from conforming to the first two constraints by simply putting everything into one big ball of mud abstraction. That’s obviously not desirable, so we need this constraint to force us to create abstractions from which we can compose applications.

Abstractions are internally highly cohesive, which means that all code inside them is inter-related. Internally they are a small ball of mud. If that highly interrelated code is to be understood, it needs to be small.

A rule of thumb is around 100 to 500 lines of code. If the code is in diagram form (which will often be the case for reasons we will explain later), we should limit the size to 100-500 nodes plus edges.

If an abstraction contains more than 500 lines, it is starting to get over the brain limit for other programmers to understand. If abstractions average less than 100 lines of code, we will likely have more abstractions than we need to, and burden ourselves with an unnecessarily high number of them to learn. The sweet spot, which relates to the size of our brains, is somewhere in-between.

Other emergent patterns

ropespiral.jpg

ALA emerges many other patterns and properties. Many of them are already known about in software engineering, which is not surprising - such things as DSLs (Domain Specific Languages), Dependency Injection, Composite and Decorator patterns, monad like composition, etc. However, there are often subtle but important differences. For example the observer pattern (publish/subscribe) can be used to achieve calls going up the layers at run-time, but may not be used between abstractions in the same layer. The subscribers’s reference to the publisher would be an illegal dependency. A line of code in an abstraction in a higher layer must do the subscribing, which is effectively what the wiring pattern described below is.

Inheritance is not used in ALA. We only use composition because inheritance breaks abstractions.

Wiring pattern

WiringPattern.png
Figure 4. Wiring pattern frequently used in ALA

I use the wiring pattern shown on the right frequently to build ALA applications.

Abstraction A in the application layer has legal dependencies on abstractions B and C in the domain abstractions layer. B and C have dependencies on D in the programming paradigms layer.

D is a programming paradigm interface. This interface is not owned by B or C. It is its own abstraction representing a compositional idea, such as Dataflow or UI layout.

B and C have ports of type D. B’s port is implemented as a field of type D, and C’s port is implemented by implementing D’s interface.

A can create an instance of B and an instance of C and wire them together because they have compatible ports. Wiring causes the instance of C, cast as the interface, to be assigned to the field in B.

While abstraction B and abstraction C know nothing about each other, the instances of B and C can communicate with each other at run-time because they both know about the interface D.

Going forward

When the idea first occurred that zero coupling could be achieved through the stated constraints, we did not know what it would be like, or even if it was possible, to write working systems. Our experience since then is that it has always been possible, and gets easier with experience. Coming up with the abstractions is sometimes obvious and sometimes not so obvious. The most difficult part is breaking bad habits. Your programming language, some conventional patterns such as MVC like patterns, and architectural styles such as conventional UI/business logic/database layering are memes that will conspire against you.

In time there will be tools to help enforce the ALA constraints. As Robert Martin points out, an indication of a good abstraction can be the number of times it is used, but ultimately this may require AI. When that is achieved, it will be possible to automatically refactor legacy code into ALA code. ---

A tiny example

Every chapter has an example, and we do the same here in the introduction. Unlike most pedagogical sized examples, these examples progressively become non-trivial. Yet because of ALA’s power, they remain uncomplicated and easy to understand.)

Requirement: Make a switch control a light.
diagram switch light

The diagram above is not documentation. Nor is it a high-level architectural view of the solution. It is the solution. It is the implementation for the user story abstraction. It contains all the detail needed for an executable user story. The diagram is literally compiled. Let’s do that now by hand:

Application layer

Here is the diagram converted to text form:

System.cs
var system = new Switch().WireTo(new Light());
system.Run();

Given this code, it is not difficult for any programmer to write the necessary Switch and Light domain abstractions as classes, together with a Dataflow programming paradigm abstraction as an interface to make that code execute. We will do that in a moment.

In conventional code, we would likely have broken the system up into two modules, one for the switch and one for the light. The switch might directly call a method in the interface for the light, or vice versa. In ALA you can’t do that. The concepts of Switch and Light, already handed to us as abstractions in the words of the requirements, must remain as abstractions. At design-time, they can’t know about one another and they can’t know about the specific system. As abstractions they can’t communiate with one another. But at run-time, instances of them, after having been wired together, can communicate with one another.

The abstract concept of dataflow is invented to allow instances of Switch and Light and many other things to communicate at run-time. Dataflow is an even more abstract abstraction. It resides in the programming paradigms layer. It is a stream of data without end. It is not a specific interface of either Switch or Light. It allows Switches and Lights to be wired arbitrarily to anything using the same dataflow concept.

The Switch-Light system is also an abstraction, albeit a more specific one than Switch or Light. Multiple instances of it could be used in a building. Its purpose is to know about the system comprising a light and a switch connected together. It knows about Switch, Light and Dataflow as abstract concepts but doesn’t know anything about their implementation details.

The system diagram is a direct, formal restatement of the requirements. So the diagram is three things in one: The formal statement of requirements, the high level architecture, and the executable. One source of truth for all three. Conventional software engineering usually has three different documents for these, which then must be kept synced.

When the program is this small, it looks like we just created four abstractions when two modules would have done. However, by creating an application layer abstraction to represent the system level knowledge, and a programming paradigm layer abstraction to represent the concept of Dataflow, we make several powerful improvements to the code that are important as the application scales up in size:

  1. There is now an explicit place that cohesively implements the requirement instead of having the requirement’s implementation distributed in an obviscated way inside Switch and Light. What is loose coupling between modules in conventional code becomes cohesive code completely contained inside a higher (more specific) abstraction in ALA. Changes to the requirements are made in this one place.

  2. The code inside all the abstractions: System, Switch, Light and Dataflow are zero-coupled with one another. Every abstraction’s internal code is readable in isolation.

  3. Being abstractions, Switch and Light are reusable. They are reusable in the same application or in other applications in the domain. Dataflow is even more reusable.

  4. Programming paradigms provide the meaning of connecting together two instances of domain abstractions. We can use multiple different programming paradigms in the one top level design. ALA is said to be polyglot in programming paradigms. This makes it very expressive.

  5. Switch and light do not need a shared understanding for the interpretation of data (a shared language). We think that shared languages are necessary because people need a shared language to communicate. So we tend to create modules that work in the same way. These are sometimes referred to as contracts. If Switch and Light were modules, they would need to agree on something like true means on and false means off. In ALA, this type of coupling is also gone. The knowledge about interpretation of data is wholly contained inside the user story abstraction in the higher layer. This may seem trivial in this simple case, but it becomes enormously significant in larger programs and distributed programs.

  6. If the instances of Switch and Light are deployed to different physical locations, the Switch-Light system is still a single cohesive abstraction describing that system. Normally, conventional programs are decomposed first into modules across physical locations, but in ALA you always compose first by user stories, regardless of the physical deployment of its constituent domain abstraction instances. This will also be highly significant as the system scales up in size.

  7. Switch and Light are easily testable with unit tests.

  8. Testing the system abstraction is exactly acceptance testing. In ALA, you always test with dependencies in place, but you mock the ports. Just as you would not mock out a dependency such as squareroot, you do not mock any dependencies in ALA, because all dependencies are knowledge dependencies. Testing the system abstraction means testing that an instance of the switch and Light system works.

  9. The writing of the Switch and Light abstractions can be handed off to different individuals or teams because, as abstractions, they know nothing about each other, and they know nothing about the System. In fact abstractions will be better quality if the teams do not collaborate with each other so that the abstractions themselves do not collaborate. (Corollary of Conway’s law.)

  10. In terms of methodology, instead of decomposing the Switch-Light system into parts, we compose it from abstractions. This point may seem subtle, but it is profoundly important. The conventional methodology of dividing a system into smaller parts until the parts are small enough to implement is arguably the prevalent approach in traditional software engineering. However it tends to result in parts that are more specific than the system (can’t be reused for anything) and more seriously inter-collaborating parts that have implicit knowledge of each other. The meme divide and conquer should be replaced with invent and compose

Here is skeleton code of the two domain abstractions.

Domain abstractions layer

Switch.cs
// domain abstraction
class Switch
{
    // port
    private IDataFlow<bool> output;

    // called from internal code (not shown) when it detects a hardware change
    private void SwitchChange(bool newState)
    {
        output.Send(newState);
    }
}
Light.cs
// domain abstraction
class Light : IDataFlow<bool>
{
    // port
    IDataFlow<bool>.Send(bool data)
    {
        if (data) // turn on the light
        else // turn off the light
    }
}

Each of these abstractions implements a port, which allows instances of them to be wired using the programming paradigm, DataFlow.

You may be wondering why the Switch’s output port is private. That’s because we wan’t the public interface of Switch to just be the 'configuration interface' used by the system abstraction when it instantiates a Switch. The WireTo method is designed to be able to wire the private port.

Here is our programming paradigms layer which contains the DataFlow abstraction:

Programming Paradigms layer

DataFlow.cs
// Programming paradigm: DataFlow
interface IDataFlow<T>
{
    void Send(T data);
}

In ALA, we frequently use the wiring pattern, as depicted by the diagram in Figure 4, which consists of instantiating domain abstractions and wiring them together by ports that use an even more abstract interface. The wiring pattern is quite ubiquitous, and therefore comes from a foundation layer that resides below the Programming Paradigms layer:

Foundation layer

Wiring.cs
public static object WireTo(this object a, object b)
{
    // using reflection:
    // 1. Find a private field in object "a" that matches in type an interface implemented by object "b".
    // 2. Cast and assign object "b" to that field in object a.
    // 3. Return object "a".
}

Note: A basic implementation of WireTo is listed in Chapter Two. You can get the source for a WireTo extension method from one of the Github repositories for the example projects in several following chapters.

Note: ALA does not require the use of reflection. I like to use reflection because it allows me to use an extension method to get a WireTo operator implemented for all domain abstractions. It allows me to make specifying the port name optional. It also allows me to make the port fields in the class private so they do not look like part of the public configuration interface to the layer above. And it spares me from writing setter methods for every port.

Instead of using a WireTo function, and if you are generating wiring code from the diagram automatically using a tool, and you make the port fields public, you could generate wiring code like the following:

Application.cs
new Switch().output = (IDataFlow<bool>) new Light();

Now that we know how to express requirements by composition of domain abstractions, let’s quickly demonstrate the ease of maintenance of our application:

Requirement: Add a sensor to turn on the light when the switch is on and it is dark. And give a feedback indication:

(For these small examples, we will manually generate code from the diagrams.)

diagram switch light sensor

Here is the diagram converted to text form:

Application.cs
var andGate = new AndGate();
new Main
    .WireTo(new Switch()
        .WireTo(andGate
            .WireTo(new Light())
            .WireTo(new Indicator())))
    .WireTo(new Sensor(threshold:0.5)
        .WireTo(andGate))
    .Run();

We just invented some new domain abstractions: AndGate and Sensor, again directly implied by the requirements.

One of the domain abstraction instances has a configuration of 0.5. This is a threshold for expressing the requirement clause "is dark".

Notice that this application is easier to write in this way than it would be in conventional C code. This is because the programming paradigm we are using, dataflow, suits the expression of these requirements. Most C code to do even such a simple requirement as this would likely already be messy in the way it handles run-time execution.

The astute reader will notice that the AndGate can’t implement IDataFlow<bool> twice for its two inputs. In later projects, we will show how we work around this unnecessary language constraint.

You may also notice that the fanout from the output of the AndGate to both the Light and the Indicator won’t work because an output port can be wired only once. We show how this implementation problem is solved in later projects as well.

Now that we have some reusable domain abstractions and programming paradigms, let’s quickly write another trivial application:

Requirement: Turn on the light from a tick item in the Tools menu of a PC application, and give an indication in the status bar when the light is on.

diagram menu light status

Here is the diagram converted to text form:

Application.cs
var indicator = new Indicator( {"Light is off", "Light is On"} );
new MainWindow()
    .WireTo(new Menu())
        .WireTo(new Menu("Tools")
            .WireTo(new TickBox(label:"Light")
                .WireTo(new Light())
                .WireTo(indicator)
            )
        )
    )
    .WireTo(new StatusBar()
        .WireTo(indicator) // put the indicator on the UI
    )
    .Run();

Here we are introducing some graphical UI, so we invented another programming paradigm for "UI layout". It is used between all the UI elements: MainWindow, Menu, Tickbox, StatusBar, Indicator. Wiring things together using that programming paradigm means things are arranged inside things on the UI.

Notice how ALA is a polyglot programming paradigm. We use two programming paradigms, UI layout and dataflow, in the same diagram. Notice also that we don’t separate UI from business logic and data models as we do in conventional architectural laering patterns. These are highly cohesive things from the perspective of user stories. Instead we separate the implementations of the domain abstractions. It is still easy to swap out, for example, the UI implementation. The diagram above could be implemented as a web application or a desktop application by swapping between two sets of UI domain abstraction implementations.

Let’s do an application to browse for and display a (dynamic content) CSV file on a grid, filtered by a user specified name, and sorted by names. The CSV file has headings that will display in the grid.

diagram csv to grid

The wiring between the MeniItem and the OpenFileBrowser uses an Event-driven programming paradigm.

The wiring between the CSVReadWriter, Filter, Sort and Grid uses a programming paradigm that allows dynamic row and columns of data to flow. The Grid abstraction is able to pull rows of data as needed.

Here is the diagram converted to text form:

Application.cs
var grid = new Grid();
var csvReaderWriter = new CsvReaderWriter
var filter = new Filter() {Column = "Name"};
new MainWindow()
    .WireTo(new Menu()
        .WireIn(new Menu("File"))
        .WireIn(new MenuItem("Open"))
        .WireIn(new OpenFileBrowser(extensions = {"csv"} ))
        .WireIn(csvReaderWriter)
        .WireIn(filter)
        .WireIn(new Sort() { Column="Name" })
        .WireIn(grid) { Column="Name" }
    )
    .WireTo(new TextBox(Label="Filter by name")
        .WireTo(filter)
    )
    .WireTo(grid)
    .Run();
Note: if you already have monads in your programming library for things like sorting and filtering (such as LINQ or reactive extensions), then instead of creating new domain abstractons for Filter and Sort, you could just create a domain abstraction called _query_ and use LINQ or reacitve extensions code to configure the query instance.
Note: WireTo() returns its first operand (this). WireIn() is the same as WireTo() except that it returns its second operand. These operators support the fluent coding style being used in this hand compiled code so that we do't have to think of names for every instance of an abstraction.

The methodology we have been following is that you write the application code (diagram) first (or part of it), just focusing on expressing the requirements. This causes you to invent domain abstractions and programming paradigms. Then you come up with an execution model that will make the programming paradigms execute. For example, the two interfaces listed below might be what you would come up with for the Event and PullTable programming paradigms:

Event.cs
// Programming paradigm: Event driven
interface IEvent
{
    void Execute();
}
PullTable.cs
// Programming paradigm: TableDataFlow
interface IPullTable : IEnumerable<List<string><T>>
{
    List<string> GetHeaderLabels();
}

This interface handles dynamic data (types unknown at compile-time) by crudely using strings. In a later project we will show a more sophisticated IPullTable interface that uses static typing for fields known at compile-time and an ExpandoObject for dynamic fields. We will also do a push version.

Both Filter and Sort will have both input and output ports of type IPullTable. The List in the IEnumerable represents one row of the table.

Don’t worry, we won’t be creating new programming paradigm abstractions at this rate for long. In fact we already have most of the ones we will use in all our example projects in this book.

Notice how in the above examples, we have used software engineering patterns we already know about, just in a different way. There is DSL (Domain Specific Language), Dependency Injection (which is what the WireTo operator does), Event driven programming, XAML-like UI layout (without the XML), RX (Reactive programming), monad-like wiring up, and the fluent style.

Notice how the application diagram in each case is both a direct representation of the requirements and the executable. It is executable because unlike conventional higher level or architecture diagrams, ALA application diagrams do not leave out details from the requirements because. They are a complete expression of them. Instead, they leave out all details of implementation, which are taken care of by the domain and programming paradigm abstractions.

To the extent that requirements are cohesive, so the code that expresses them should be. For example, we do not try to separate the specific UI, I/O, business logic, persistent data etc into different modules because they are highly cohesive for a given requirement. Most other architectural patterns, such as MVC, do separate in this way, which creates coupling. Instead we reduce the problem in a different way - through the use of domain abstractions which provide reusable aspects of implementation. All knowledge and details from the requirements end up in the application layer, but that’s all that goes there.

Note: A graph-like structure is showing up in these small applications because each requirement in itself contains a graph of relationships. ALA embraces this and makes it explicit, which is why the requirements are best expressed as diagrams. In conventional code, the graph structure is still there, but it is typically obfuscated as many intermodule relationships, making it hard to see. Worse still, the natural cycles in the graph of relationships in the requirements, would cause circular dependencies amoung the modules. Conventionally, some of these dependencies are broken by using pulling data half the time rather than pushing, or worse still, using indirections such as the observer pattern. These removals of dependencies don’t make the relationships themselves go away, they just make them harder to see.

Although the fluent style is a nice way to hand-compile these small diagrams, code like this with indenting and nested brackets does not scale up well for large diagrams. (It is still better than the tangled web of dependencies they would form in conventional modular code though.) But we can do better. We will not be hand-writing code like this for large applications - we will automatically generate the code from the diagrams.

Finally, the wiring pattern used in the examples above is only one possible way of meeting the fundamental ALA constraints. For example, ALA can also be applied to functional programming using monads as domain abstractions. But we will use explicit objects with explicit classes and the wiring pattern shown above in most of our examples. The wiring itself is implemented by a field in the objects, but that is field is immutable. ALA may appear to be synonymous with this wiring pattern, but actually ALA is just the three fundamental constraints stated at the beginning of this introduction.


This online book is a work in progress. ALA is a research in progress. Please don’t hesitate to provide feedback to the e-mail address given at the top.

I would like to acknowledge the help of Roopak Sinha at AUT (Auckland University of Technology) for his significant academic contributions and ideas for ALA, and the contributions of many students who have implemented ALA projects, and helped refine the ALA methodology.

1. Chapter one - What problem does it solve?

If you have already experienced difficult to maintain code, big balls of mud, spaghetti code, or the CRAP principle, you can probably skip this chapter.

However, the example at the end is pretty cool - it starts with the type of typical C code that I see most students write, and then refactors, it step by step, into ALA compliant code - you should take a look at that.

The problem that ALA solves can be seen as any one of the following perspectives.

1.1. The Big Ball of Mud

dung.jpg
Figure 5. The big ball of mud antipattern

Brian Foote and Joseph Yoder popularized this term in their 1997 paper. It describes the default architecture when no other architecture is used. A similar term is spaghetti code. I think it describes the architecture of most software even when so-called architectural styles, such as layering, are used.

ALA is an in-the-large strategy to organise code. It provides the constraints needed for the code structure to not degenerate into a big ball of mud. As the software life cycle continues, retaining the organisation becomes easier rather than harder.

1.2. Simplify down the overwhelming set of architectural styles, patterns, and principles

There are many traditional architectural styles, patterns, principles and paradigms. The problem of structuring software code to meet quality attributes involves mastering an overwhelming number of them. Here are some examples:

  • loose coupling and high cohesion, information hiding, separation of concerns

  • DSLs, aspects, model driven, MVC, inversion of control, functional programming, UML Class diagrams, sequence diagrams, activity diagram, state diagram.

  • Views, Styles, Patterns, Tactics, Models, ADL’s, ADD, SAAM, ATAM, 4+1, Decomposition

  • CBD/CBSE, Components & Connectors, Pipes & Filters, n-tier, Client/Server, Plug-in, Microservices, Monolithic, Contracts, Message Bus

  • Modules, Components, Layers, Classes, Objects, Abstraction, Granularity

  • Semantic coupling, Syntax coupling, Temporal coupling, existence coupling, Good and bad dependencies, Collaboration

  • Interfaces, Polymorphism, Encapsulation, Contracts, Interface Intent

  • Execution models, Event-Driven, Multithreaded, Mainloop, Data-driven, Concurrency, Reactor pattern, Race condition, Deadlock, Priority Inversion, Reactive

  • Principles: SRP, OCP, LSP, ISP, DIP; MVC, MVP, etc

  • Design Patterns: GOF patterns, GRASP patterns, Layers, Whole-Part, Observer, Strategy, Factory method, Wrapper, Composite, Decorator, Dependency Injection, Callbacks, Chain of Responsibility, etc

  • Expressiveness, Fluency, DDD, Coding guidelines, Comments

  • Programming Paradigms, Imperative, Declarative, Object oriented design, Activity-flow, Work-flow, Dataflow, Function blocks, Synchronous, State machine, GUI layout, Navigation-flow, Data Schema, Functional, Immutable objects, FRP, RX, Monads, AOP, Polyglot-Programming Paradigms

  • Messaging: Push, Pull, Synchronous, Asynchronous, Shared memory, Signals & Slots

  • Memory management, Heap, Persistence, Databases, ORMs

  • Waterfall, Agile, Use cases, User stories, TDD, BDD, MDSD

Mastering all these topics takes a lifetime. Even if you can, juggling them all and being able to use the right ones at the right time is extremely taxing on any developer. Add to that the mastering of technologies and tools, keeping to agile sprint deadlines, and commitment to your team and management, it is an almost impossible task. 'Working code' tends to be what the team is judged on, especially by project managers or product owners who have no direct interest in architecture or even the Definition of Done. They don’t want to know about the rather negative sounding term, "technical debt".

Most texts will tell you that these are all tools and that you need to use the right tools for each job. It all depends, they say, on the particular system, and its particular functional and non-functional requirements. In most cases they end up being used in an ad-hoc manner that doesn’t work well. In some cases their use is actually harmful.

Being a pre-worked recipe of the aforementioned styles and patterns, ALA probably contains no truly novel ideas. Every aspect of what ALA does can be found already done by someone. However the combination that ALA uses is as far as I know unique.

Some ingredients from the above list are accentuated in importance more than you might expect (such as abstraction). Some are relatively neutral. The biggest surprise for me during the conception process of ALA was that some well-established software engineering memes seemed to be in conflict. Eventually I concluded that they were in-fact bad (such as UML class diagrams). We will discuss these in detail in subsequent chapters.

Like any good recipe, the ingredients work together to form a whole that is greater than the sum of the parts. The resulting code quality is significantly ahead of what the individual memes do by themselves. It continues to surprise me just how effective, and enjoyable, the simplified view is.

1.3. An optimal solution for quality attributes

ALA is an optimal solution for these quality atributes:

  • Readability

  • Complexity

  • Maintainability

  • Testability

  • Understandability

  • Modifiability

  • Extensibility

  • Dependability

  • Reusability

It is independent of any specific domain, so it is a general reference architecture. By optimal, I mean that it makes these qualities as good as they can be.

If other non-functional requirements are also important, ALA provides a good starting point.

  • Performance

  • Availability

  • Scalability

  • Portability

  • Distributability

  • Security

  • Usability

  • Fault-tolerance

Even if the ALA structure must be compromised in places for other qualities, it is still better to start with these quality attributes optimised and deviate from them as necessary. As it happens, the maintainability resulting from ALA frequently makes other quality attributes easier to achieve as well. For example, in an ALA application it is often easy to make performance optimizations in the execution model that don’t affect the application code. For example, an application first written to run on a single processor can more easily be distributed to multiple processors. Or, you can port an application by swapping out domain abstractions without changing the application code.

1.3.1. Readability

close_up_code.jpg
Figure 6. One big program

Modules don’t necessarily make pieces of code that are readable in isolation.

ALA code is readable, not because of style, convention, comments or documentation, but because any one piece of code appears to you as a separate uncoupled little program that is readable in complete isolation.

1.3.2. Complexity

There is a meme in the software industry that says that the complexity of software must be some function of its size. This need not be so. With proper use of abstraction it is possible to have complexity that is constant regardless of program size. ALA makes use of this.

complexity curve

This is a qualitative graph comparing the complexity of an ALA application with that of a big ball of mud and an average loosely coupled application. This is further explained later here.

1.3.3. Maintainability

The maintainability effort over time should qualitatively follow the green curve in the graph below because as software artefacts are written, their reuse should reduce the effort required for other user stories. Product owners seem to have an innate sense that we manage to organise our code such that this happens. That is why they get so frustrated when things seem to take longer and longer over time, and they often ask us "haven’t we done this before". In practice, too often we follow the red curve. Maintenance eventually gets so difficult that we want to throw it away and start again. We reason we can do better. My experience is that we don’t do better when we rewrite. We just create another mess. It is just a psychological bias on the part of the developer caused by a combination of a) the Dunning Kruger effect and b) the fact that it is easier to read our own recently written code than someone else’s.

If we apply all the well known styles and principles, the best we seem to be typically manage is the orange curve, which comes from the COCOMO models, and which still has maintenance effort continuously increasing.

When we did an experimental re-write of a legacy application using ALA, and measured its maintainability attribute, it comes out as improving over time by several different measures.

effort curve

ALA is based on the theoretical architectural constraints needed to follow the green curve.

1.3.4. Testability

In ALA all code is testable. ALA makes it clear when to mock and when to test with dependencies in place. All dependencies are left in place, because all dependencies are design-time or knowledge dependencies.

Therefore, when testing the application layer abstractions, they are tested with their domain abstraction dependencies. In other words, testing the application is acceptance testing.

Testing domain abstractions is easy with units tests because abstractions are zero-coupled. Mocks objects are wired to ports.

1.4. Structure hidden inside the modules

The problem in most large code bases is that the system structure, the in-the-large structure, is not explicit. It is distributed inside the modules themselves. Collaboration between modules is implicitly hidden inside them. Finding this structure, even for a single user story can be time consuming. I have often spent a whole day doing that, doing countless all-files searches following function calls or method calls of the user story through many modules just to end up changing one line of code. Many developers I have spoken to can identify with this experience.

It can get a lot worse as the system gets larger. In a seemingly bizarre twist, the more loosely coupled you make the elements, the harder it gets to trace a user story because of the indirections. Some people conclude that loose coupling and being able to trace through a user-story are naturally in conflict.

I call this situation SMITA (Structure Missing in the Action). The internal structure is sometimes drawn as a model - high-level documentation of the hidden structure. But such models are a secondary source of truth.

ALA completely eliminates this problem and this conflict. The structure is explicitly coded in one place, without any indirections. Yet the abstractions are zero-coupled.

1.5. The CRAP cycle

Typical bright young engineers come out of university knowing C++ or Java (or other C*, low-level, imperative, language that mimics the silicon), and are confident that, because the language is Turing-complete, if they string together enough statements, they can accomplish anything. At first they can. There hardly seems a need for a software architect to be involved. And besides, we are told that a design can emerge through incremental refactoring.

Cynefin.jpg
Figure 7. Code can quickly follow the Cynefin quadrants into the complex

As the program gets larger, things get a little more complicated, but the young developer’s brain is still up to the task, not realizing he has already surpassed anyone else’s ability to read his code. He is still able to get more features working. One day parts of the 'transition'. It becomes somewhere you don’t want to go. On the Cynfin diagram, it has transitioned from the complicated quadrant to the complex quadrant. And now it is trapped there. It is too complex for refactoring.

The incremental effort to maintain starts to eat away and eventually exceed the incremental increase in value. This now negative return causes the codebase itself to eventually lose value, until it is no longer an asset to the business.

It has transitioned to chaos. It will be abandoned. When a new bright young engineer who knows C* arrives, he looks at the legacy codebase and is convinced that he can do better. And the cycle repeats. This is the CRAP cycle (Create, Repair, Abandon, rePlace). ALA is the only method I know that can prevent the CRAP cycle.

1.5.1. A short history of ALA

From early on in my career, I experienced the CRAP cycle, not so much rewriting applications, but trying to avoid the mess when writing new ones. When starting from a blank piece of paper, I would research all the architectural styles and principles. I would come across things like 'loose coupling', and I remember asking myself, yes but how does one accomplish that? Each time I would still fail.

I started searching for a pre-worked, generally applicable, 'template architecture' that would tell me what the organisation of the code should look like for any program. I searched for such a thing many times over a long career and never found one. Some would say that this is because the highest level structure depends on project specific requirements.

Finally, near the end of my career, I have that template meta-structure that’s applicable to all programs. The turning point was when I noticed two (accidental) successes in parts of two projects. These successes were only noticed years later, 15 years in one case and 5 years in the other. They had each undergone considerable maintenance during that time. But their simplicity had never degraded and their maintenance had always been straightforward. It was like being at a rubbish dump and noticing two pieces of metal that had never rusted. "That’s weird", you think to yourself. "What is going on here?"

One of them had the same functionality as another piece of software that I had written years earlier. That software was the worst I had ever written. It was truly a big ball of mud, and maintenance had become completely impossible, causing the whole product to be abandoned. So it wasn’t what the software did that made the difference between good and bad. It was how it was done.

Analysing the common properties of those two code bases, gave clues that eventually resulted in a theoretical understanding of how to deal with complex systems. This meta-structure is what I now call Abstraction Layered Architecture.

Subsequently, I ran some experiments to see if the maintainability and non-complexity could be predictably reproduced. These experiments, which have worked spectacularly well so far, are discussed as a project at the end of every chapter.

1.6. Example project - Thermometer

In this example project, we will first do conventional C code using functions, then refactor it into abstraction layers, and finally improve on that using classes.

Functions have an execution model we are already familiar with, making this first example easier to understand. However, keep in mind that, for whole programs, this execution model does not usually make a good programming paradigm. An emergent property of ALA is its support of multiple and diverse programming paradigms including your own. We do this to improve expressiveness of the requirements.

Nevertheless, functional composition is a passable programming paradigm for a tiny, dedicated embedded program in a micro-controller such as our thermometer. Let’s have a look at some typical code:

1.6.1. Bad code

configurations.h
 #define BATCHSIZE 100
main.c
 #include "configurations.h"
 void main()
 {
    int temperatures[BATCHSIZE];
    ConfigureTemperaturesAdc();
    while (1)
    {
        GetTemperaturesFromAdc(temperatures); // gets a batch of readings at a time
        ProcessTemperatures(tempertures)
    }
 }
process.c
 void ProcessTemperatures(int adcs[])
 {
    float temperature;
    for (i = 0; i<BATCHSIZE; i++) {
        temperature = (adcs[i] + 4) * 8.3; // convert adc to celcius
        temperature = SmoothTemperature(temperature);
        ResampleTemperature(temperature);
    }
 }
Resample.c
 void ResampleTemperature(float temperature)
 {
    static int counter = 0;
    counter++;
    if (counter==15)
    {
        DisplayTemperature(temperature);
        counter = 0;
    }
 }
smooth.c
 // smooth the reading before displaying
 float SmoothTemperature(float temperature)
 {
    static filtered = 0;
    filtered = filtered*9/10 + temperature/10;
    return filtered;
 }
adc.c
 #include "configurations.h"
 void ConfigureTemperaturesAdc()
 {
    // configure ADC channel 2 to do DMA BATCHSIZE values at a time
 }
 float GetTemperaturesFromAdc(int temperatures[])
 {
    for (i = 0; i<BATCHSIZE; i++) {
        temperature[i] = ReadAdcChannel(2);  // pseudocode here for the adc read
    }
 }

At first this code wont look that bad, but that’s only because the whole program is so small. It looks modular, but you still have to read all of it to understand any part of it. That’s possible for small programs, but of course that strategy won’t scale up.

As we are taught to do, different responsibilities of the thermometer implementation have been separated out into smaller pieces with smaller responsibilities, although ProcessTemperatures appears to have three responsibilities. The problem is that all the pieces are in some way collaborating to make a thermometer. They are all coupled in some way, both explicitly or implicitly. That’s why we have to read all the code to understand the thermometer. Scale this up to 5000 lines of code, and we will have a big mess.

We are going to refactor the program using the ALA strategy:

  • every piece of knowledge about 'being a thermometer' will be in one function

  • that 'Thermometer' function will be at the top

  • that function will do no real work itself

  • how to do more abstract things will be put into other functions

  • those functions will not know anything about temperature or thermometer

  • The top layer function will compose the abstract functions it needs to build a thermometer

1.6.2. Toward ALA code

application.c
 #define BATCHSIZE 100
 void main()
 {
    int adcs[DMABATCHSIZE];
    float temperatureCelcius;
    float smoothedTemperatureCelcius;
    while (1)
    {
        GetAdcReadings(adcs, 2, DMABATCHSIZE);  // channel=2
        for (i = 0; i<BATCHSIZE; i++) {
            temperatureInCelcius = OffsetAndScale(adc, offset=4, slope=8.3);
            smoothedTemperatureCelcius = Filter(temperatureCelcius, 10);
            if (SampleEvery(15))
            {
                Display(FloatToString(smoothedTemperatureCelcius, "#.#"));
            );
        }
    }
 }
offsetandscale.c - (domain abstraction)
 // offset and scale a value
 void OffsetAndScale(float data, float offset, float scale)
 {
    return (data + offset) * scale;
 }
filter.c - (domain abstraction)
 // IIR 1st order filter, higher filterstrength is lower cutoff frequency
 float Filter(float input, int strength)
 {
    static float filtered = 0.0;
    filtered = (filtered * (strength-1) + input) / strength
    return filtered;
 }
resample.c - (domain abstraction)
 // Returns true every n times it is called
 bool SampleEvery(int n)
 {
    static counter = 0;
    counter++;
    if (counter>=n)
    {
       counter = 0;
       rv = true;
    }
    else
    {
       rv =  false;
    }
    return rv;
 }

The code now begins to be arranged into two abstraction layers, the application layer and the domain abstractions layer. The application is now the only function that knows about being a thermometer. (It is still doing some logic work - the 'for loop' and 'if statement', which we will address soon.)

All the other functions are now more abstract - they know nothing about thermometers - GetAdcReadings, OffsetAndScale, SampleEvery, Filter, FloatToString, and Display. Notice that the word 'thermometer' has been removed from their names, and none of them contain constants or any other references that are to do with a thermometer or temperature.

These abstract functions give you six things:

  1. Abstract functions are way easier to learn and remember what they do

  2. Abstract functions give design-time encapsulation i.e. zero coupling.

  3. Abstract functions can be understood by themselves

  4. Abstract function interfaces are way more stable - as stable as the concept of the abstraction itself

  5. Abstract functions are reusable

  6. Abstract functions are testable

  7. As a consequence of 1., the application function can also now be understood by itself

Now let’s go one more step and create an abstraction to do what that for loop does: This may seem like a retrograde step, but we need to understand this mechanism to move to our final goal of expressing the requirements through pure composition of abstractions. We want to move the 'for loop' out into its own abstraction, but we don’t want to move the code that’s inside it. We accomplish this by putting the code inside it into another function and passing that function to the for loop function:

1.6.3. Further toward ALA code

application.c
 #define DMABATCHSIZE 100
 void main()
 {
    int adcs[DMABATCHSIZE];
    float temperatureCelcius;
    float smoothedTemperatureCelcius;
    ConfigureAdc(2, DMABATCHSIZE)
    while (1)
    {
        GetAdcReadings(adcs, 2, DMABATCHSIZE);  // channel=2
        foreach(adcs, func1);
    }
 }
 void func1(float adc)
 {
    temperatureInCelcius = OffsetAndScale(adc, offset=4, slope=8.3);
    smoothedTemperatureCelcius = Filter(temperatureCelcius, 10);
    if (SampleEvery(15))
    {
        Display(FloatToString(smoothedTemperatureCelcius, "#.#"));
    );
 }
foreach.c
 void foreach(int values[], void (*f)(int))
 {
    for (i = 0; i<sizeof(values)/sizeof(*values); i++) {
        (*f)(values[i]);
    }
 }

"func1" is not an abstraction - you cannot give it a name and learn a simple concept of what it does. That’s why I gave it a non-descript name. The content of func1 is cohesively just part of the thermometer application. The name func1 only serves as a symbolic connection within cohesive code - nothing more than a wiring between two points in the program. In this case func1 is immediately below where it is used in the same small file. But as a program grows, these symbolic wirings are always hard to follow. You would need to resort to text searches to find these connections. These types of connections can be numerous and unstructured in larger programs, and the best way to deal with them is diagrams. A line on a diagram is like a symbolic connection between two points, but it’s anonymous and easy to follow. However, this particular one can be dealt with in text form. So let’s go ahead and remove it by using an anonymous function directly as the second parameter of foreach:

application.c
 #define DMABATCHSIZE 100
 void main()
 {
    int adcs[DMABATCHSIZE];
    float temperatureCelcius;
    float smoothedTemperatureCelcius;
    ConfigureAdc(2, DMABATCHSIZE)
    while (1)
    {
        GetAdcReadings(adcs, 2, DMABATCHSIZE);  // channel=2
        foreach(adcs, (adc)=>{
            temperatureInCelcius = OffsetAndScale(adc, offset=4, slope=8.3);
            smoothedTemperatureCelcius = Filter(temperatureCelcius, 10);
            if (SampleEvery(15))
            {
                Display(FloatToString(smoothedTemperatureCelcius, "#.#"));
            );
        });
    }
 }

It uses the lambda syntax '()⇒{}', which if you are not already familiar with, is worth getting used to. It’s a function without a name, so think of the ⇒ as being instead of the name of the function, the round brackets as the parameters, and the curly braces as the body of the function.

The next thing we want to do is get rid of the while loop, get rid of the indenting, and stop handling the data that is being passed from one function to another. None of them have anything to do with a thermometer. All those intermediate holding variables: adcs, temperatureCelcius, etc are all just symbolic connections. They are too much work when we just want to compose our thermometer from abstractions.

The while loop and all the indenting are there only because we have 'execution flow' tied in with our composition of abstractions. Basically we want to make control of execution flow another abstraction so that the thermometer can be built by just composing abstractions rather than writing executing code.

To do this we will first show how its done using monads. If you don’t know about monads just skip the section as we don’t need this step to understand our final goal. But for those who do understand monads, it is interesting to visit this step to see why the functional programming guys invented them. Then in the following step we will go to ordinary classes with ports instead of monads.

1.6.4. Brief detour: composing with monads

application.c
 void main()
 {
    program = new ADC(channel=2, batchSize=100)
    .foreach()
    .OffsetAndScale(offset=4, slope=8.3)
    .Filter(strength=10)
    .SampleEvery(15)
    .NumberToString(format="#.#")
    .Display();

    program.Run();
 }

Monads have allowed us to separate execution flow from composition flow. The composition flow is now a pure dataflow paradigm. Data will flow from the ADC to the display, so that is directly represented by the composition. How it executes is separated out, and we will go into how that works shortly. Let’s first understand the 'composition' and why this is so important.

Even if you don’t understand how the monads work, you can see that syntactically the program is now very nice because all it does is compose instances of abstractions, and configure them with constants to be a thermometer. The composition is not declarative - it is dataflow, because dataflow suits how to describe the thermometer. If we let go of how it executes and just trust that the dataflow from one instance of an abstraction to the next works, the program becomes highly readable.

We are using the word 'composition' here to mean the things we are joining together in adjacent lines of code. It can also mean joining boxes with lines in a diagram. Think of a composition as analogous to the adjacent notes in a music score, which are always played successively. If the lines of code are statements or function calls, we are composing things for imperative execution by the CPU. If the lines of code are data processors, we are composing things for successive processing of data. The output of one passes directly to the input of the next.

If we are stuck with thinking in terms of imperative execution flow (the only way of thinking in the C language) we will need to try hard to let that go, and realize that in ALA, 'composition' can be any programming paradigm you want.

Also notice that the first statement just builds the program. Then the second statement sets it running. This two stage aspect of monads is common in the programming paradigms we will use in ALA. It is because the underlying execution flow is not the same as the flow of the programming paradigm. We first wire it up, and then we tell the wired up structure to 'execute'.

There is a second important difference from the while loop version. The while loop version handled the data itself. Each function returned the data which was stored in a local, otherwise useless, variable and then passed into the next function. The monad code doesn’t do that. Instead, it creates and wires together objects which will, at run-time, send the data directly from one to another via an interface. This does not mean that the abstractions themselves know anything about each other - they are still zero coupled. But the application now doesn’t have to deal with the mechanics of dataflow. It just has to compose abstractions.

Lastly, here’s how monads actually execute - the execution model. Don’t worry if this doesn’t make sense.

Each function in the program statement (the function after each dot) executes once at the start. They are not executed when the program is running. Each of these functions first instantiates an object (using new), and secondly wires that object to the previous object.

The functions wire the objects together using an abstract interface. Common interfaces used for monads are IEnumerable or IObservable. These interfaces support iteration of data, by returning an IEnmerator or IObserver. If using the IEnumerator interface, there is a simple method in the interface that pulls data from the previous object. If using the IObseravble interface, there is a simple method in the interface that pushes data to the next object. So IEnumerable/IEnumerator and IObservable/IObserver as abstractions are pretty much just the concept of dataflow, the same abstract concept we will use in the ALA version.

1.6.5. Composing with plain objects

Here is the same program as above, but we are using plain classes with ports instead of monads. We use the 'new' keyword explicitly to create the instances of abstractions, and explicitly wire them together using a wiring function. It’s a little less succinct than the monad version, but the idea of "objects with ports that you wire together like electronic components" is easier to understand, and more versatile. It is necessary for developers to be able to write new domain abstractions, so this needs to be easy.

application.c
 void main()
 {
    program = new ADC(channel=2,batchSize=100)
        .WireIn(new Foreach())
        .wireIn(new OffsetAndScale(offset=4, slope=8.3))
        .wireIn(new Filter(strength=10))
        .wireIn(new SampleEvery(15))
        .WireIn(new NumberToString(format="#.#")
        .wireIn(new Display());

    program.Run();
 }

The wireIn method is doing dependency injection.

The WireIn method returns the new object, so it is possible to string WireIns together. This is called fluent syntax.

1.6.6. Using multiple programming paradigms:

Monads are generally not versatile enough to handle multiple ports of different programming paradigms, which we will want in ALA programs. Abstractions usually only support dataflow. But what if we want to also compose the UI, or event-driven? What if we want to compose transitions between states of a state machine? In ALA, we are able to do all this in the one application, in the same way - using whatever programming paradigms are the best way to express the requirements.

Some instances of abstractions will need to take part in multiple paradigms, such as both UI and dataflow. When we boil down the description of our requirements to pure composition, our composition will often be a graph of relationships. And when you have a graph, your composition is best described by a diagram.

To illustrate this let’s add some UI to our thermometer:

ThermometerDiagram.png
Figure 8. Thermometer application complete with UI

The diagram has both UI composition and dataflow composition. For the UI part of the composition, the lines obviously don’t mean dataflow - they mean 'display inside'. So now different lines in our diagram have different meanings. Here is how that diagram is represented as text.

application.c
 void main()
 {
    FloatField temperature;

    program = new ADC(channel=2)
        .WireIn(new Foreach())
        .wireIn(new OffsetAndScale(offset=4, slope=8.3))
        .wireIn(new Filter(10))
        .wireIn(new SampleEvery(100))
        .WireIn(new NumberToString()
        .wireIn(temperature = new FloatField());

    mainwindow = new Window()
       .wireTo(new Label("Temperture:"))
       .WireTo(temperature);

    mainwindow.Run();
 }

The text of the ALA thermometer has a symbolic connection for one of the wirings, "temperature". This is ok in this small program, but doing that won’t scale up. That is why we consider the diagram to be the source, and this text version is generated from it.

Looking once again at the diagram, you can see that ALA has allowed us to keep all cohesive knowledge about a thermometer together, and quite succinctly. It contains all the details needed to describe a thermometer, but does so in terms of domain abstractions that are not specific to a thermometer at all. There is no implementation in the application code. All implementation is done by domain abstractions. If you can see that point in the example code, then you are pretty much understanding ALA.

Once we have this diagram, it is easy to conceive how we might add features. For example, we could add two radio buttons into the UI, and wire then to a switcher abstraction that switches the data path between two instances of OffsetAndScale to change between Celcius and Faranheit.

2. Chapter two - The structure

In this section we describe the structure of ALA without trying to explain too much about why it looks that way. An analogy is if we described the physical anatomy of the human body without explaining how it works. How ALA works will be covered in chapter three, and why it works in chapter six.

The organisation of this chapter (and all chapters) is to use different perspectives. We all have different prior knowledge on which we build new knowledge, so we will each have a different best way to understand things. Use the perspective that makes the most sense to you. Because of the use of perspectives, there will often be some repetition of ideas between the major sections.

2.1. Software elements, relations among them

Some definitions of software architecture see it in terms of software elements and the relations among them. Indeed the first two fundamental constraints of ALA define what elements are and what their relations are.

In this first perspective of ALA, we look at these two fundamental constraints in terms of code that lets us see how everything works right down to the C# executable level. We show all the code in all the layers so there are no gaps.

2.1.1. The only unit of code is an abstraction

The only unit of code in ALA is an abstraction. Not module, not class, not function. An abstraction is more than a module, class or function in that it must be a 'generalized conceptual idea'. It must be learnable as a concept. Many modules, classes or functions in traditional code will not be good abstractions, especially if they have dependencies on peer modules, classes or functions.

Abstraction goes hand in hand with reuse. Krueger said that abstraction and reuse are two sides of the same coin. Abstractions tend to be stable, as stable as their conceptual idea. When we have a dependency on an abstraction, it is a dependency on a conceptual idea.

Here is an example abstraction which smooths data arriving on its input and outputs it at a slower rate.

DomainAbstractions\LowPassFilter.cs
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary> (6)
    /// ALA Domain abstraction.
    /// Has one DataFlow input port and one DataFlow output port, both type double.
    /// Smooths the incoming data and outputs it at a lower frequency.
    /// The strength parameter sets the degree of filtering
    /// (cutoff frequency relative to input frequency).
    /// and also sets the lower rate of output.
    /// e.g. if Strength is set to 10, then there is one output for every 10 input
    /// datas received.
    /// You need to understand the programming paradigm, IDataFlow,
    /// to understand this code.
    /// </summary>
    class LowPassFilter : IDataFlow<double> // input port (1)
    {
        public LowPassFilter(int strength, double initialOutput = 0.0) (3)
        {
            this.strength = strength;
            this.lastOutput = initialState;
        }

        private IDataFlow<double> output;  // output port (2)

        private int strength;
        private double lastOutput = 0.0; (4)
        private int resampleCounter = 0; (4)

        void IDataFlow<double>.Push(double data) (5)
        {
            lastOutput = (data + strength * lastOutput) / (strength+1);
            if (resampleCounter==0)
            {
                resampleCounter += strength;
                output?.Push(lastOutput);
            }
            resampleCounter--;
        }
    }
}

The abstraction smooths an incoming data stream, and slow its down.

The abstraction has two ports:

1 Input port, the implemented interface
2 output port, simple private field of type interface

The input port is the IDataFlow<double> interface that is implemented by the class. Dataflow is a programming paradigm which is a stream of data without end. Incoming data arrives via the Push method in this interface. We never know where the data comes from. The output port is the private field named "output". This field will be set pointing at an IDataFlow<double> input of an instance of another abstraction somewhere. We never know where the data goes to. How the output field gets set is described later.

Because the abstraction uses ports, it has no knowledge of other peer abstractions. If it did, it would not be an abstraction. It is self-understandable at design-time, except for the even more abstract concepts on which it depends. In this case that is DataFlow and the concept of Wiring, which will wire instances of this abstraction to instances of other abstractions with compatible ports. These are described more fully shortly.

3 The public interface is only used by the layer above. It is only for creating and configuring instances of the abstraction. The abstraction has two configurations parameters. One is optional. Optional configurations can also be implemented with setters.
4 The abstraction keeps some internal state. This goes somewhat contrary to functional programming where we want all state to be passed in. In ALA a good abstraction trumps being stateless. As an abstraction that is naturally stateful, a filter should hide its state internally.
5 The implementation of the input interface does the work of the filter. Note that port interfaces are always implemented explicitly. (If implemented implicitly, they would be part of the public API of the class, which we don’t wan’t. We only want the port accessed via the interface.)

Internally, abstractions are cohesive (which means sticking together) or highly interconnected code collaborating for a single purpose.

In ALA, abstractions are implemented as files. The name of the file is the name of the abstraction. The code that implements an abstraction needs no internal organisation. We do not care about dependencies or any inter-relationships inside an abstraction. In fact the more cohesive everything is the better, as long as it is small. It will often be implemented as a single class or function, but may consist of a small number of items such as classes, functions, interfaces, enums, delegates, etc, as long as the total is small.

6 When an abstraction is created, it is critically important to add comments that make it learnable by explaining the concept it provides, its ports, its configurations and an example of its use. In Visual Studio, I copy these comments before the constructor because that allows them to pop up in the IDE when hovering on the constructor name where instances are created with a new keyword.

Abstractions are the only mechanism that provides design-time information hiding. When David Parnas coined the term information hiding he meant at design-time. Unfortunately there is a popular meme that information hiding means encapsulation. The same idea of design-time information hiding has other names such as Alistair Cockburn’s protected variations, and Robert Martin’s version of the OCP (open closed principle).

A module or class can encapsulate information at compile-time but does not necessarily hide information at design-time. In the case of our LowPassFilter, the fact that it is just a low pass filter means that its internals cannot be coupled or collaborating with other modules. It’s bounded by its conceptual idea. If it were a module instead, it might primarily perform the filter function for a specific application, but knowing which application it is part would cause it to take on ancilliary functions, such as offset and scale to output the units needed by the application. Or it might grab its input knowing where it comes from, or it might send it’s output to two different places. It might connect to the applications setting menu for its settings. All these things can be encapsulated, but they are all causing design-time coupling.

Meaning of abstraction

Unfortunately, there are now two distinct meanings for the meme 'higher level of abstraction' in common usage in software engineering. We need to take a moment to understand the difference. In ALA, abstraction means the original dictionary meaning:

Etymology: abstract literally means draw away [a common idea or concept from specific examples]

Miriam Webster: relating to or involving general ideas or qualities rather than specific people, objects or actions.

The other meaning of 'higher level of abstraction' used in the software engineering community appears to be 'further away from the domain of the computer and closer to the problem domain'. For example, layers are often shown building up from the hardware. They can also build up from the database, or a physical communication medium, such as the layers of the OSI communications model. In this meaning, the application is considered the most abstract. For example, a 3-tier system or a communication stack uses this type of layering. The perception is that because we no longer have to deal with computer domain details such as data storage, communications protocols, hardware, etc, we must be more abstract. The problem with this is that the problem domain also deals with details. These details come from the real world and are described by detailed requirements. The modules that contain these details are no more abstract than those in the various domains of computing.

Conventional layering tends to use this second meaning of 'abstract'; layers are said to be more abstract as you go up. ALA layers use the original meaning of the word 'abstract' and layers get more abstract as you go down.

The layers are not the same either. To convert conventional layers to ALA, you generally just tip them on their side so that they are not layers but independent disconnected abstractions. Each of them knows about details of something but they no longer directly connect to each other in either direction. On their own they will do nothing. The layer above, whose job is to know the details of a specific application or system, composes instances of them by instantiating them, configuring them, and wiring them together. Conventional layers can be many. ALA layers are few.

A final note about abstractions versus the SRP (single responsibility principle). The SRP is not really the best way to think about abstractions. It is better to think about what details an abstraction implementation knows about. It can be a specific user story, a specific feature, a type of UI element, a type of database, a protocol, a hardware device, etc. It will contain all the cohesive knowledge about that thing. In doing so, it may have multiple responsibilities. For example an abstraction that knows about a protocol or a hardware device may have responsibility for both input and output. It may have responsibility for configuring the hardware device and getting data. A filter abstraction may both smooth the input data and resample the input, because resampling is cohesive with smoothing.

We will look further at what 'abstraction' means in chapter six.

2.1.2. Abstraction internal structure

Internally an abstraction is cohesive. This means that every line of code is related to, and collaborates with, every other line of code to. It has no structure. It is a small ball of mud.

It should be more cohesive than most conventional code, simply because it is implementing an abstraction. An abstraction doesn’t necessarily have a single purpose. An abstraction that represents a serial UART device type may have several responsibilities to do with the device, such as configuring, reading and writing. But it is still all cohesive code.

We shouldn’t think of an abstraction as being composed functions, variables, structs, methods, enums, delegates, statements, lines of code or the like. These are all syntactical components of the abstraction, but not semantic components. None of them have meaning on their own related to the concept of the abstraction. Only when we take all of lines of code together do we have meaning. So from a semantic point of view, we can’t break an abstraction into smaller parts.

The external structure and internal structure with respect to abstractions are at opposite extremes. One is zero coupled (no structure at all), and the other is fully coupled (highly cohesive).

This idea that there is no structure an abstraction is important because we are used to structural hierarchies. In ALA, if we want to know the structure at a larger scale than abstractions, say a whole user story or even the whole system, well they are inside abstractions of their own. Those abstractions are in the higher layers, but they are still just small abstractions. They implement a whole user story or a whole system, yet remain small because they simply compose instances of abstractions from lower layers.

2.1.3. ALA uses only one relationship type

This is the second of the three fundamental constraints. ALA uses a single type of relationship - a dependency on an abstraction that is more abstract than the one whose implementation uses it.

diagram 05a

In terms of our previous example, LowPassFilter is Abstraction A and IDataFlow is Abstraction concept B.

Notice how in the diagram the relationship arrow comes from inside A. This is significant. It is the code that implements A that requires knowledge of abstraction concept B. We never actually draw lines when using abstractions, so you wont see this type of line in ALA diagrams. We only use it here while describing this one relationship that we are allowed to use.

B must be more abstract than A. "More abstract" means more general, not specific to A, and have a greater scope of reuse than A.

Because B is more abstract, it is more stable. ALA therefore automatically enforces the Stable Dependencies Principle.

The relationship means that, to read and understand the code inside A, you must know the abstraction concept B - not how the insides of abstraction B work. The word "abstraction" implies that it should be learnable in a short time and easy to retain. When we get an abstraction concept is sometimes described as a moment of insight.

Here are some more legal dependencies, this time from the application layer to the domain abstractions layer.

Application\Thermometer.cs
    new AnalogInput(channel: 2)
    .WireIn(new LowPassFilter(strength: 10))
    .WireIn(new OffsetAndScale(offset: -31, scale: 0.2))
    .WireIn(new Display(label: "Temperature"));

Legal dependencies from a Thermometer user story abstraction on abstractions AnalogInput, LowPassFilter, OffsetAndScale, Display and WireIn. The user story takes input from an analog to digital converter, filters them to remove noise, scales them to be in degrees, and displays them.

This type of dependency has zero coupling. Because of the way our brains have evolved to understand a complex world in terms of abstractions, abstractions are the only mechanism that hide knowledge at design-time. The dependency has zero coupling between all codes using the abstraction and the code that implements the abstraction.

This type of dependency also tells us explicitly what other knowledge we need in order to understand code. For example, if abstraction A is standard deviation and abstraction B is square root, then to understand the code that implements standard deviation requires knowledge of the concept of square root. That’s why we sometimes refer to it as a knowledge dependency. Such a dependency always applies at design-time, as well as compile-time and run-time. Some knowledge dependencies may be implicit and so apply at design-time only. For example a dependency may be on a convention, or it may be simply the knowledge of ALA. You need knowledge of those things to fully understand the code. We want to be explicit and clear about knowledge dependencies for every bit of code inside every abstraction, so we endeavour to always state implicit dependencies in comments.

Architectures generally work by applying constraints that restrict the infinite variety of ways code could be organised. The ALA constraints seem severe at first, but after some practice they are exceedingly helpful in guiding the design. There is essentially one right way to organise the code. Sometimes the abstractions come easily and sometimes they require sustained inventive thought processes, especially in a new domain. In most domains, I usually start with the UI description in the requirements as they most readily reveal their abstractions. Then it becomes apparent that dataflows must be connected to these instances of UI abstraction elements. Data transformation and state abstractions will then become apparent next.

ALA compared with the UML class diagram

All UML relationships except one are illegal in ALA, and that one is highly restricted to being a composition on a more abstract class. Such a relationship is always represented in code by just referring to the abstraction by name. You do not draw a line on a UML class diagram. For example, you would never use a library abstraction such as regex by drawing a line on a diagram to a box representing the regex class. You would just use the regex abstraction by name.

Furthermore, such a use of an abstraction by name is inside the class. It’s part of the class’s internal implementation. It makes no sense to show the relationship at the zoomed out scale outside the class with a filled diamond line coming from the border.

Class diagrams are evil. They have done more damage to software architecture than any other meme.

If a UML class diagram were drawn of an ALA application, there would be no lines at all, just boxes in space arranged in layers. This makes sense, because classes are used to implement abstractions, and abstractions have zero coupling with one another.

ALAClassDiagram.drawio.png
Figure 9. UML class diagrams for ALA have no relation lines.

The diagram is not useful. We will never use it again. That’s not to say we won’t use diagrams. Diagrams can be used in ALA to represent the internals of an abstraction. This is often done in the layer representing the application or a feature or user story. In ALA terms, it shows instances of (lower layer) abstractions wired together. In implementation terms it is a (static) UML object diagram.

Following are the UML relationships you cannot use, and what you do instead.

  • Associations: A conventional program will typically have many, many bad dependencies referred to as associations in UML. Most are there because data, messages, events, execution flow, etc, need to get from one place to another in the program. These are simply illegal in ALA. The whole difficult concept of dependency management vanishes. But where do these relationships go? How can the program still work? How is it possible for messages and events to travel around at run-time? The short answer is that all these associations become a line of code inside an abstraction in the layer above. Or they become a line on a diagram composing two instances of abstractions. Such lines are not relationships or dependencies - they are inside an abstraction, no different to a line of code that say calls one function and passes the result to another function. In ALA, you cannot use associations between classes. Instead you create objects in a higher level abstraction and then wire them together. The abstractions themselves will not know where their data comes from, nor where it goes.

    Note that dependency injection or otherwise passing an object into another object doesn’t remove an association relationship between the classes. It only changes the relationship from composition to association, neither of which is allowed between peer classes. In other words, in ALA you are not allowed to know about a class in the same layer, not even its interface. Not even a base class. Instead you use ports that use a much more abstract interface called a programming paradigm interface from a lower layer.

  • Composition: Although the knowledge dependency relationship used in ALA can be implemented as a UML composition relationship (directed arrow with filled diamond), the ALA knowledge dependency is more constrained. It must be instantiating a class in a lower, more abstract, layer. This means that you cannot break a module up into pieces arbitrarily. Instead, you must invent abstractions. The composition relationship in ALA can go down by one or more layers, but never within a layer.

  • Inheritance: ALA doesn’t need or use inheritance. It would break the abstraction of the (more abstract) base class in the lower layer. Instead we always use composition. Inheritance is often used to provides a 'calling up the layers at run-time' mechanism by its use of virtual functions. In ALA, we do this with ordinary observer pattern (events in C#), or by passing in a method as a configuration (usually anonymously or as a lambda expression), or with the strategy pattern.

  • Packages: ALA does not use hierarchies or nesting. In other words, abstractions cannot be contained by other abstractions. Abstractions are never private. The reason they are never private is simple. An abstraction that is depended on should be more abstract than the abstraction using it. A more abstract abstraction needs to be public so it can be reused. ALA uses abstraction layers instead of encapsulation hierarchies. In ALA, packages would only be used as a distribution mechanism, not as part of the architecture for information hiding.

    The word package means container. Packages are usually just a container of abstractions such as a library. We should not consider knowledge dependencies to be on the package because we can’t generally learn a package. We should consider knowledge dependencies to be on the individual abstractions inside the package.

    Let’s consider the situation where a conventional package is a good abstraction in itself. Because it was implemented as a package, it’s internal implementation is large (Facade pattern). Let’s say our conventional package hides a lot of complex implementation and contains abstractions that we are not interested in using in the rest of our application. For example it could be a compiler that we can invoke from our application. The thing is, if the compiler abstraction is written using ALA, it will use lots of useful abstractions for the domain of compilers and parsing. We still want those abstractions to be public for reuse. It’s just that we don’t want them particularly visible to the rest of our application, which is in a different domain. To solve this problem we should still make the abstractions used by the compiler abstraction public, but put them into a different DomainAbstractions folder and namespace. When we do this, we will want the DomainAbstractions folders to be qualified with the name of the domain, such as CompilerDomainAbstractions.

  • Namespaces: While not part of the UML, we can discuss namespaces here in case you think of them in some way similar to packeages. In ALA, namespaces are used for the layers. For example we use namespaces such as Application, DomainAbstractions and ProgrammingParadigms. This allows unrelated abstractions in different layers to have the same name. The files that implement abstractions are put inside folders that have the same names as the namespaces.

    Note that unlike packages, namespaces are not encapsulations. Namespaces only make names unique. One 3rd party tool I used to generate dependency graphs showed dependencies on namespaces as if namespaces were abstractions. This gave a completely misleading view of the true nature of the dependencies in the code. I had to write a custom query for the tool to show the actual dependencies on the abstractions inside the namespaces.

2.2. Abstraction layers

Because the target of a dependency must be more abstract, abstractions arrange themselves in discrete layers. This is what gives the architecture its name: Abstraction Layered Architecture.

Only a small number of layers are needed. Consider that we can construct the human body with just six layers: Atoms, Molecules, Proteins, Cells, Organs, Body. Probably need another two or three to build the human brain from neurons. Small applications generally use four. The layers are given standard names that describe their level of abstraction:

Layers.png
Figure 10. Abstraction layers

These layers are not fixed by ALA. But we tend to return to these ones in our experience so far. Following is discussion of each layer together with example code to see how everything works.

Application layer

In describing example layers, we start with example code. This code will build into a complete running application so no holes are left in understanding the code level mechanics of how everything works. The accompanying bullet points then explain the high level theory of why the code is organised the way it is. The code is available here: https://github.com/johnspray74/Thermometer

Application\Thermometer.cs
using DomainAbstractions;
using ProgrammingParadigms;
using Foundation;

namespace Application
{
    class Thermometer
    {
        public static void Main()
        {
            Console.WriteLine("Wiring application");

            private ADCSimulator adc;

            adc = new ADCSimulator(channel: 2, period: 1000) { simulatedLevel = 400 }; (1)
            adc.WireIn(new LowPassFilter(strength: 10)) (2) (5) (6)
                .WireIn(new OffsetAndScale(offset: -200, scale: 0.2)) (3)
                .WireIn(new DisplayNumeric<double>(label: "Temperature") { units = "C"} ); (4)

            Console.WriteLine("Running application");
            adc.Run(); (7)

            Console.WriteLine("press any key to stop");
            Console.ReadKey();
        }
    }
}

To understand the code, you need to have knowledge of the abstraction concepts on which it depends. These are:

1 ADCSimulator - domain abstraction simulates an analog to digital converter hardware peripheral. Has a single output port of type IDataFlow<int>
2 LowPassFilter - domain abstraction - we already met this at the beginning of this chapter
3 OffsetAndScale - domain abstraction - has a single input port and a single output port, both IDataFlow<double>. Adds a constant and Multiplies by another constant to transform data like a straight line on an x-y graph.
4 DisplayNumeric - domain abstraction - has one input port of type IDataFlow<double>. Displays the value on the console with label and optional units.
5 WireIn - foundation abstraction - wires compatible ports of instances of abstractions by setting the private field in the first object that matches the interface implemented by the second object.
6 These wirings are using the Dataflow programming paradigm. Dataflow is used by the ports of the domain abstractions and allows their instances to push data from one to the next at runtime if they are wired together. For dataflow programming, we default to pushing data through the system (from ADC to display). We use pulling when there is good reason, usually for performance.
7 The adc, which is the source of the data that gets pushed through the system, needs to be told to start running.

Once you have knowledge of these abstractions, notice that the application code is readabable by itself. It holds all knowledge about thermometers. That knowledge sits at the abstraction level of the requirements. It is highly cohesive - every line works with every other line to make a thermometer. It does none of the work itself - it just assembles and configures the needed worker objects.

The application layer is three things in one: The architecture design, the expression of requirements, and the executable. In conventional software development, these are three separate artefacts.

Execution typically occurs in two phases (similar to some monads). In the first phase the application wires together instances of abstractions. In the second phase the network of instances executes (which is what the finalizing call to Run starts).

Domain abstractions layer

At the beginning of this chapter we had an example of a domain abstraction, LowPassFilter. Here is another example:

DomainAbstractions\OffsetAndScale.cs
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary> (1)
    /// ALA domain abstraction
    /// Has one input port of type IDataflow and one output port of type IDataflow
    /// (both type double)
    /// Performs y = m(x+c) like operation where x is the input and y is the output
    /// If visualized as a straight line on an x,y graph, -c is the x axis intercept
    /// and m is the slope.
    /// You need to understand the programming paradigm abstraction, IDataFlow,
    /// to understand this code.
    /// </summary>
    class OffsetAndScale : IDataFlow<double> // input (2)
    {
        public OffsetAndScale(double offset, double scale) (4)
        {
            this.offset = offset;
            this.scale = scale;
        }

        private double offset;
        private double scale;


        private IDataFlow<double> output; (3)

        void IDataFlow<double>.Push(double data) (5)
        {
            output.Push((data + offset) * scale);
        }
    }
}

Notes on the code:

1 Important to comment what the abstraction concept is.
2 The input port is the implemented interface.

In the application code in the previous section, an instance of LowPassFilter was wired to an instance of OffsetAndScale. When wired, the output field of the LowPassFilter is set to the OffsetAndScale object, cast as the appropriate interface, in this case IDataFlow<double>.

3 The output port output is private so that it does not appear as a configuration to the layer above. It is set by WireIn or WireTo using reflection.
4 It has two compulsory configuration parameters.
5 When data is pushed into the input, it is transformed and pushed out of the output port.

For completeness, here are the other two domain abstractions that we used in the Thermometer application example:

DomainAbstractions\DisplayNumber.cs
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary>
    /// ALA Domain Abstraction
    /// Ouptuts incoming data to the console with a preceding label and optional units.
    /// Has one input port of type IDataFlow which can take int, float, double
    /// The label must be passed in the constructor.
    /// The units property may be used to set the units.
    /// fixPoint Property sets the number of decimal places.
    /// You need to understand the programming paradigm abstraction, IDataFlow,
    /// to understand this code.
    /// </summary>
    class DisplayNumeric<T> : IDataFlow<T>
    {
        public DisplayNumeric(string label)
        {
            this.label = label;
        }

        public int fixPoints { get; set; } = 0;

        private string label;
        public string units { get; set; }


        void IDataFlow<T>.Push(T data)
        {
            double d = (double)Convert.ChangeType(data, typeof(double));
            Console.WriteLine($"{label}: { d.ToString($"F{fixPoints}") } {units}");
        }
    }
}
DomainAbstractions\ADCSimulator.cs
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary>
    /// ALA Domain Abstraction.
    /// Simulate a 10-bit ADC (analog to digital converter).
    /// Normally an ADC is a hardware peripheral, but here we just do a software
    /// simulation of one to use as a source of data for example applications.
    /// A real ADC driver would have properties for setting the channel and period.
    /// You would create one instance of this driver for each ADC channel.
    /// It would output raw data in adc counts.
    /// Since it is a 10 bit ADC, the adc counts are in the range 0-1023.
    /// We retain the channel although it is not used by the simulated version.
    /// The simulated version has two simulation properties, one to set the simulated
    /// ADC reading.
    /// and one to set the level of noise in the simulated readings.
    /// You need to understand the programming paradigm abstraction, IDataFlow,
    /// to understand this code.
    /// </summary>
    class ADCSimulator
    {
        public ADCSimulator(int channel, int period = 100)
        {
            this.channel = channel;
            this.period = period;
        }

        private int channel;  // unused on simulated ADC
        private int period;   // milliseconds
        public int simulatedLevel { get; set; } = 512; // 0 to 1023
        public int simulatedNoise { get; set; } = 0; // 0 to 1023

        private IDataFlow<int> output;

        public void Run()
        {
            RunAsyncCatch();
        }

        public async Task RunAsyncCatch()
        {
            // because we are the outermost async method, if we let exceptions go,
            // they will be lost
            try
            {
                await RunAsync();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }
        }

        Random randomNumberGenerator = new Random();

        public async Task RunAsync()
        {
            while (true)
            {
                // add a bit of noise to the adc readings
                int data = simulatedLevel + randomNumberGenerator.Next(simulatedNoise)
                            - simulatedNoise/2;
                if (data < 0) data = 0;
                if (data > 1023) data = 1023;
                output.Push(data);
                // throw new Exception("exception test");
                await Task.Delay(period);
            }
        }
    }
}

As can be seen in the above examples, the domain abstractions layer contains concepts that can be composed into applications. These are typically building blocks for I/O, data transformations, and persistent state, but many other types of abstractions are possible.

Some of these types of things may also be found in general language libraries but domain abstractions are more specific to the types of applications we want to express using them. They are specific to a domain, making them more expressive, but less reusable than general purpose language abstractions. They are still reusable both within a single application and by other applications in the same domain.

Another difference between ALA domain abstractions and typical library abstractions is the use of ports. This supports building functionality by simply composing instances of abstractions. While the composing code controls who will communicate with whom, it does not actually handle the data at run-time. The data moves directly between instances via the wired ports.

A further difference between ALA domain abstractions and typical library abstractions is that when domain abstractions are composed, the meaning of composition can be a programming paradigm other than imperative. For example the application above uses a dataflow programming paradigm. Imperative is not usually a good programming paradigm for the expression of requirements, but it’s all your basic language gives you (unless you are using language integrated monads, which also give you dataflow programming).

The domain abstractions layer and programming paradigms layer together are like a DSL (Domain Specific Language). We can think of domain abstractions as composable domain language elements and the programming paradigms as providing the grammar.

It is an internal DSL because it uses the underlying language (WireTo method, and new). This allows you to easily fall back on the greater flexibility of the underlying language when you need to. For example, you can pass lambda expressions as configuration properties to a domain abstraction. Or, you can pass a whole object of a class that you write in the application layer (which is the strategy pattern).

Conventional libraries generally contain good abstractions. What makes them good abstractions is that their designers don’t know anything about the specific applications that will use them. Writing domain abstractions is best done in the same way. After the need for an abstraction is decided, pretend you don’t know anything about the application, and are writing something to be useful, reusable and learnable as a new concept. Then they should not just be specific parts of a specific application.

As we said, abstractions know nothing of each other at design-time, yet can still communicate directly with one another at run-time. It is the responsibility of the code in the layer above that composes them to know the meaning of the data. It knows the meaning of the data even though it does not handle the data. For example, our application above knows that data going from the ADC to the LowPassFilter is raw adc values representing temperature, and what the values mean. It knows that the data passing between the LowPassFilter and the OffsetAndScale is sufficiently smoothed and slowed down for a stable display. It knows that the data passing from the OffsetAndScale to the display is in degrees celsius. All these pieces of knowledge are cohesive in the design of the thermometer, and so belong together inside the Thermometer abstraction.

Programming paradigms layer

For the Thermometer example application above, we wired four instances of domain abstractions. That wiring used a single programming paradigm, Dataflow. Here is the interface for the dataflow programming paradigm:

IDataFlow.cs
namespace ProgrammingParadigms
{
    interface IDataFlow<T>
    {
        void Push(T data);
    }
}

Programming paradigm interfaces are often this simple. Another example programming paradigm is synchronous event driven. The corresponding interface might be:

IEvent.cs
namespace ProgrammingParadigms
{
    interface IEvent { void Execute(); }
}

Here is the interface for another common programming paradigm, the UI layout. In this programming paradigm, a parent UI element is wired to its contained child elements.

IUI.cs
namespace ProgrammingParadigms
{
    interface IUI { UIElement GetUIElement(); }
}

The interface returns the .NET WPF element of the child. This allows domain abstractions to construct the UI using underlying WPF elements. In effect, UI domain abstractions are wired together in a similar way to XAML. Both have descriptive, tree structured syntax, but ALA is using the underlying C# language whereas XAML is using XML.

You can conceive other programming paradigms to give different meanings to composition of domain abstractions in such a way as to allow the easiest representation of typical requirements. For example, for implementing game scoring, I used a programming paradigm called ConsistsOf. For tennis, I used it to express that a match consists of a sets, a set consists of a games, and a game consists of a points. See the example project at the end of chapter four which uses it to implement tennis and bowling. Being able to compose instances of domain abstractions together with meanings that you define in the programming paradigms layer is powerful.

ALA is polyglot in programming paradigms. An application typically mixes a small set of different paradigms. Each provides a different meaning when the application wires two instances of domain abstractions together. Examples might be Dataflow, UI Layout, Event driven, State machine transition, Data schema entity relationship. Some may have variations such as pushing or pulling, or synchronous and asynchronous.

Programming paradigms control the way the application actually executes. Execution models such as synchronous vs asynchronous, push vs pull, and fan-out and fan-in wiring are discussed in Chapter four.

Programming paradigms provide the rules for the way instances of domain abstractions can be composed by the application. They are the grammar of the DSL.

Programming paradigms provide the mechanism of direct communication between instances of domain abstractions. They do this without the abstractions themselves knowing anything about each other.

Through the use of programming paradigms, domain abstractions know nothing about each other at design-time, yet instances of them can communicate at run-time. Of course we could achieve this by making the application handle the run-time communications. The common example would be an application that calls one function which returns a result, and then calls another function passing the result to it. This would almost comply with ALA. The problem is we don’t really want the application to be concerned with run-time communications. The application really just wants to concentrate on representing user stories by composing instances of domain abstractions. So we use a layer below the domain abstractions called programming paradigms. The programming paradigms allow domain abstractions to have compatible ports, which in turn allows them to communicate directly with one another at run-time. The design of the communications is in the Application layer, but the execution is in the domain abstractions and programming paradigms layers.

Foundation layer

The foundation layer contains code used to support ALA programs in general.

A common pattern I use is a WireIn() and WireTo() extension methods in the foundation layer. The application layer uses them to wire together instances of Domain Abstractions using ports.

WireIn() and WireTo() are dependency injection methods.

WireTo(), uses reflection. It’s not essential to use reflection for ALA. You could use dependency injection setters in every domain abstraction instead. You would need one setter per port on the left abstraction. You wouldn’t use constructor dependency injection because sometimes wiring a port is optional. I prefer using the WireTo extension method because it allows domain abstractions to not need all these setters.

Here is minimal code for the WireTo method.

Wiring.cs
namespace Foundation
{
    public static class Wiring
    {
        /// WireTo is an extension method on the type object.
        /// Wires instances of classes that have ports by matching interfaces.
        /// Port name can be optionally provided for the A side.
        /// If object A has a private field of type interface,
        /// and object B implements that interface,
        /// and the private field is not yet assigned,
        /// assigns B to the field in A.
        /// Uses reflection.
        /// Returns the left object for fluent style programming.
        public static T WireTo<T>(this T A, object B, string APortName = null)
        {
            // achieve the following via reflection
            // A.field = B;
            // if 1) field is private
            //    2) field type matches one of the implemented interfaces of B
            //    3) field is not yet assigned

            if (A == null) throw new ArgumentException("A is null "); (4)
            if (B == null) throw new ArgumentException("B is null ");

            bool wired = false;
            var BType = B.GetType(); (1)
            var AfieldInfos = A.GetType().GetFields(System.Reflection.BindingFlags.NonPublic
                | System.Reflection.BindingFlags.Instance)
                .Where(f => (APortName == null || f.Name == APortName)) // match portname if any
                .Where(f => f.GetValue(A) == null) // not yet assigned
            var BinterfaceTypes = BType.GetInterfaces().ToList(); // ToList to do the reflection once

            foreach (var AfieldInfo in AfieldInfos) (2)
            {
                var BimplementedInterface = BinterfaceTypes
                    .FirstOrDefault(interfaceType => AfieldInfo.FieldType == interfaceType);
                if (BimplementedInterface != null)  // there is a matching interface
                {
                    AfieldInfo.SetValue(A, B);  // do the wiring (3)
                    wired = true;
                    break;
                }
            }

            if (!wired) // throw exception (4)
            {
                var AinstanceName = A.GetType().GetProperties()
                    .FirstOrDefault(f => f.Name == "InstanceName")?.GetValue(A);
                var BinstanceName = B.GetType().GetProperties()
                    .FirstOrDefault(f => f.Name == "InstanceName")?.GetValue(B);

                if (APortName != null)
                {
                    // a specific port was specified - see if the port was already wired
                    var AfieldInfo = AfieldInfos.FirstOrDefault();
                    if (AfieldInfo?.GetValue(A) != null)
                        throw new Exception($"Port already wired  {A.GetType().Name}[{AinstanceName}].{APortName} to {BType.Name}[{BinstanceName}]"
                    );
                }
                throw new Exception($"Failed to wire {A.GetType().Name}[{AinstanceName}].\"{APortName}\" to {BType.Name}[{BinstanceName}]");
            }
            return A;
        }



        /// Same as WireTo, but returns the right object instead of the left object
        public static object WireIn<T>(this T A, object B, string APortName = null)
        {
            WireTo(A, B, APortName);
            return B;
        }
    }
}
1 It first gets an IEnumerable of all the private fields in class A. If a port name is passed in, it must match. Then it gets a list of all the interfaces of class B.
2 It iterates through the fields to find one that matches any of the interfaces of B.
3 It sets the field in A pointing to B, cast as the interface type.
4 When there are errors in wiring code, it would be nice to get errors at compile-time. The WireTo extension method can’t do that, but it does throw exceptions at wiring time when the application first starts. Since in ALA all wiring is generally done at this time, at least you wont have potential exceptions later during normal run-time.

Four different exceptions may be thrown. 1) Object A (left object being wired) is null. 2) Object B (right object being wired) is null. 3) A specific A side port was specified, but it is already wired. 4) No matching A side port was found.

Once again, slightly more complete code is available here: https://github.com/johnspray74/Thermometer

Extra layer for larger applications

If a single abstraction is used for the application, then as more and more user stories are added into it, it will eventually get too large for the ALA size constraint. Meanwhile, domain abstractions and programming paradigms are stable and do not generally grow larger with overall program size. They may increase in number, but it is the application that will go over the 500 line complexity limit.

ALA will need to be applied to the large application abstraction by adding a new layer below it. The requirements are likely already written in terms of abstractions which we call features or user stories or use cases. We can use these abstractions as the basis for the new layer. We call the layer "Features" or "UserStories". The application abstraction becomes a composition of features or user stories.

Let’s extend the Thermometer example code we used above to have a new feature for measuring load. At the same time, let’s introduce a features layer with the two features: temperature and loadcell.

I have deliberately retained a need for communication between the two features to show how features can also have ports and be wired together.

Application layer
Application\Application.cs
using Features;
using Foundation;

namespace Application
{
    class Application
    {
        /// <summary>
        /// Instantiate two features: a temperature readout and a loadcel readout.
        /// Also wire the Temperature to the Loadcell for temperature compensation
        /// </summary>
        public static void Main()
        {
            Console.WriteLine("Wiring application features");

            var temperature = new Temperature(); (1)
            var load = new LoadCell(); (1)

            temperature.WireTo(load); // for temperature compensation (2)

            Console.WriteLine("Running application");
            Console.WriteLine("press any key to stop");
            temperature.Run();
            load.Run();
            Console.ReadKey();
        }
    }

}
1 The code instantiates two features for this particular application.
2 The code wires together the feature to get temperatures sent to Loadcell at run-time. Feature abstractions can have ports. A common example of wiring between features would be to wire a feature instance’s menu items to a main menu feature.
Features layer

The Features layer contains independent features or user story abstractions.

Each feature creates instances of domain abstractions, configures the instances with feature specific details, and connects them together as needed to express the feature or user story.

Here is the Thermometer application rewritten to be a Temperature feature:

Features\Temperature.cs
using DomainAbstractions;
using ProgrammingParadigms;
using Foundation;

namespace Features
{
    /// <summary>
    /// Feature to coninuously measure temperature and periodically display it
    /// in degrees C on the console.
    /// Has an output port that outputs the temperature.
    /// </summary>
    class Temperature
    {
        private IDataFlow<double> output; // temperature in celcius (1)

        private ADCSimulator adc;

        public Temperature()
        {
            const int adcLevel = 400;  // 40 C
            adc = new ADCSimulator(channel: 2, period: 1000) { simulatedLevel = adcLevel, simulatedNoise = 100 };
            adc.WireIn(new ChangeType<int, double>()) (2)
                .WireIn(new LowPassFilter(strength: 10, initialState: adcLevel))
                .WireIn(new OffsetAndScale(offset: -200, scale: 0.2)) // 200 adc counts is 0 C, 300 adc counts is 20 C
                .WireIn(new DataFlowFanout<double>()) (3)
                .WireTo(new DisplayNumeric<double>(label: "Temperature") { units = "C"} ) (5)
                .WireTo(new DataFlowExternalPort<double>((d) => output?.Push(d))); (4)
        }


        public void Run()
        {
            adc.Run();
        }
    }
}
1 The feature has an output port for temperature

The wiring itself is the same as it was in the Thermometer application except that three extra objects are used to make the dataflow wiring work. Their classes come from the Dataflow programming paradigm abstraction. These classes are:

2 ChangeType: allows Dataflow ports of one type to be wired to Dataflow ports of a different type. In this case, the output of ADCSimlator is int and the input of LowPassFilter is double.
3 DataFlowFanout: A normal output port can only be wired once. DataFlowFanout allows you to wire to multiple places. In this case we wanted to wire the output of OffsetAndScale to both a DisplayNumeric and an external port.
4 DataFlowExternalPort. We want to wire the output of the DataFlowFanout to the external port output. You might think you could just write .WireTo(output). The reason this doesn’t work is that when the Temperature constructor runs, the code on the outside is instantiating a Temperature. That same external code would not have wired our output port yet. It’s value will be null at this time. Therefore we instead wire to a simple class that takes a function as its configuration. For the function we pass in a lambda expression that will push the data via the output port.
5 Note that WireTo is used to wire DataFlowFanout to multiple places. WireIn wires things in a chain.

Here is the other feature used by our example application, the Loadcell.

Features\Loadcell.cs
using DomainAbstractions;
using ProgrammingParadigms;
using Foundation;

namespace Features
{
    /// <summary>
    /// Class:
    /// Feature to coninuously measure a load from a load cell and display it in kg on the console.
    /// Displays with one decimal place.
    /// Has temperature compensation for better accuracy (optionally feed temperature into the input port in degress C)
    /// </summary>
    class LoadCell : IDataFlow<double> // input for temperature compensation (1)
    {
        private ADCSimulator adc;
        private DataFlowInitializer<double> defaultTemperature;
        private OffsetAndScale offsetAndScaleTemperature;

        /// <summary>
        /// Constructor:
        /// Feature to coninuously measure a load from a load cell and display it in kg /// on the console.
        /// Displays with one decimal place.
        /// Has temperature compensation for better accuracy (optionally feed temperature
        /// into the input port in degress C)
        /// </summary>
        public LoadCell()
        {
            // Wire an adc to an OffsetAndScale to an Add to a DislayNumeric.
            adc = new ADCSimulator(channel: 3, period: 500) { simulatedLevel = 200, simulatedNoise = 0 }; (2)
            var add = new Add(); (4)
            adc.WireIn(new ChangeType<int, double>())
                .WireIn(new OffsetAndScale(offset: 0, scale: 0.5)) (2)
                //.WireIn(new DataFlowDebugOutput<double>((s)=> System.Diagnostics.Debug.WriteLine(s))) (3)
                // .WireIn(new DataFlowDebugOutput<double>(Console.WriteLine)) (3)
                .WireIn(add)
                .WireTo(new DisplayNumeric<double>(label: "Load") { fixPoints = 1, units = "kg" } ); (2)

            // Wire the inut port for temperature to another OffsetAndScale to the other input of the Add.
            defaultTemperature = new DataFlowInitializer<double>(); (6)
            offsetAndScaleTemperature = new OffsetAndScale(offset: -20, scale: -0.1); // compensate -0.1 kg/C from 20 C (2)
            defaultTemperature.WireIn(offsetAndScaleTemperature) (7)
                .WireIn(new DataFlowConvert<double, Double2>((d)=>new Double2(d))) (5)
                .WireIn(add);

        }



        void IDataFlow<double>.Push(double data)
        {
            ((IDataFlow<double>)offsetAndScaleTemperature).Push(data); (7)
        }


        public void Run()
        {
            defaultTemperature.Push(20);  // in case no temperture is connected to the input port, set it to 20 C
            adc.Run();
        }
    }
}
1 This time the feature has an input port for temperature, which is the implemented interface.
2 Notice the reuse of several domain abstractions in this feature. The DisplayNumber abstraction is configured to display one decimal place.
3 Debugging a dataflow can be done by inserting an object (decorator pattern) that outputs the values in the stream.
4 A new domain abstraction called Add is used. It has two IDataflow<double> inputs and a IDataFflow<double> output. We assign the Add to a local variable so that we can wire the second input later. Note that a C# class cannot implement the same interface twice (even though there is no reason why not). There are several ways we have used to work around this limitation. The one used here is to make one of the ports a Double2, a struct containing a double. This allows to have a double with a different type.
5 To wire to the Double2 input port, we convert from double to Double2 using a DataFlowConvert<double, Double2>() abstraction. This abstraction can do any transformation on Dataflow, so is analogous to the Select() or Map() functions used in query languages.
6 DataFlow initializer is a domain abstraction that can be used to initialize inputs of a dataflow in case no input arrives in time at run-time. In this case one of the two inputs to Add may not arrive if the temperature input port is not connected to anything. We therefore want to initialize it with a default temperature of 20 C.
7 Note that it is not a problem to do fan-in wiring. Both defaultTemperature and the input port are connected to the input port of offsetAndScaleTemperature.

For completeness, here is the code for the Add domain abstraction. Note that this abstraction is doing more than what a single + operator would. It is adding two dataflows.

DomainAbstraction\Add.cs
using System;
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary>
    /// ALA domain abstraction to add two numeric dataflows.
    /// Currently only supports doubles.
    /// Two input ports are implemented interfaces.
    /// One output port called "output".
    /// Both inputs must receive at least one data before output begins.
    /// Thereafter output occurs when either input receives data.
    /// One of the inputs is type Double2, which is a struct containing a double.
    /// This is a work around for can't implement the same interface twice.
    /// When wiring to the Double2 port, do it via an instance of DataFlowConvert like this:
    /// .WireIn(new DataFlowConvert<double, Double2>((d)=>new Double2(d))).WireIn(new Add());
    /// You need to understand the programming paradigm abstraction, IDataFlow,
    /// to understand this code.
    /// </summary>
    class Add : IDataFlow<double>, IDataFlow<Double2> (1)
    {
        private IDataFlow<double> output; (2)

        private double? operand1; (3)
        private double? operand2; (3)

        void IDataFlow<double>.Push(double data) (4)
        {
            operand1 = data;
            if (operand2.HasValue)
            {
                output.Push(operand1.Value + operand2.Value);
            }
        }

        void IDataFlow<Double2>.Push(Double2 data) (5)
        {
            operand2 = data.Value;
            if (operand1.HasValue)
            {
                output.Push(operand1.Value + operand2.Value);
            }
        }
    }


    /// <summary>
    /// Wrap a double in a struct.
    /// We do this only to get a different type of double to effectively get multple inputs
    /// for the "Add" class because C# wont allow implementing the same interface
    /// twice (it should though).
    /// </summary>
    struct Double2
    {
        public Double2(double value) { this.value = value; }
        private readonly double value;
        public double Value { get { return value; } }
        public override string ToString() => $"{value}";
    }
1 Two input ports
2 One output port
3 For storing the last value received on each input. They are nullables so that we know when we have had input.
4 Implement the first input port
5 Implement the second input port

2.3. Folders, files, classes, objects, interfaces, functions

This is the second perspective on ALA. In this perspective we look at the implementation language elements and see how they are used in ALA applications.

2.3.1. folders and namespaces

If you see an ALA application, you will find three to five folders that correspond with abstraction layers (described in the previous section). For example:

  • Application

  • Features

  • DomainAbstractions

  • ProgrammingParadigms

  • Foundation

Knowledge dependencies only go down these layers. So to understand the code inside files in the higher layers, you need to have knowledge of what all the files in lower layers do. There are no dependencies between files in any folder.

Namespaces exactly correspond with these folder names. Therefore we have namespaces called "Application", "DomainAbstractions", "ProgrammingParadigms", etc. This tells us which layer an abstraction comes from, and which folder it resides in.

Namespaces also avoid naming conflicts between layers. They are not useful beyond that. Unfortunately, there is no convenient way of telling the compiler or the IDE to not 'see' classes, interfaces etc in other files in the same namespace or folder.

2.3.2. Files

Abstractions are implemented as files. Abstractions are often implemented as a single class, function, or interface, but sometimes an abstraction consists of a small cohesive group of them, with things like delegates, enums, or even variables. Internal to an abstraction, they interconnect with each other unconstrained. There are no dependency rules inside a file. The only constraint ALA makes is that the total must be small - of the order of 200-500 lines of code, or under the brain size limit. This applies to all abstractions, including the ones that make up the application itself in the top layer.

In time I hope languages catch up and give us visibility support for ALA abstraction. This would probably involve a new construct called Abstraction{} to group the elements of an abstraction. It does not need a name. This construct replaces the use of a file that we are currently using as a stand-in. Anything public inside the Abstraction is only visible to code in higher layer abstractions, i.e. abstractions in higher namespaces. It is not visible in your own namespace, nor to those of lower layers. The compiler would need to know the namespace layering order. If we had this, we would have compiler checking for illegal dependencies.

2.3.3. Objects

In ALA, abstractions are usually a single class. Instances of such abstractions are objects. It is the objects that get wired togther by their ports. Classes are the design artefacts that know nothing about one another. Objects are the run-time artefacts that communicate with one another at run-time.

2.3.4. Interfaces

Classes have a 'main interface', the constructors, and any public methods and properties. A class can also implement other interfaces. In ALA, a class’s 'main interface' (it’s constructors and public methods and properties) are only used to instantiate and configure the class from a higher layer. It is never used to actually use the class to do its work. This is effectively the ISP (interface segregation principle). The client who instantiates a class object is different from the classes whose objects will interact with it, so different interfaces are used.

Only the higher layer with knowledge of the system has the relevant knowledge of what should be instantiated, how it should be configured, and how the instantiated objects should be composed together to make a system.

The 'main interface' of a class is 'owned' by the class and is specific to the class. This may sound like stating the obvious, since it is there to allow instantiation and configuration of said class. The thing is that no other interface implemented or required by the class can be 'owned' by the class. No other interface can be specific to the class. The class may not provide an interface designed specifically for it, not may it require an interface designed specifically for it. In other words, all other interfaces must be abstract and in a lower layer.

The idea that classes may not own any interface except the 'main interface' is critically important. If a class were to own another interface that is used for the class to do its work, then that interface would inherently have in its design knowledge about the class. This is true whether the interface is a provided interface (for other classes to use) or a required interface (for other classes to implement). Required interfaces are common for example in 'clean architecture'. They are illegal in ALA.

The inherent knowledge about the class contained in 'owned' interfaces will cause coupling. A class using an interface provided by another class will have design knowledge of what that other class provides at the same abstraction level as itself. It will be written according to what is being provided. There will be a fixed arrangement between the two classes. Over time, this fixed arrangement will cause a blurring of their respective responsibilities.

A class implementing an interface that is required by another class will have a similar problem. It will have design knowledge of what that other class requires at the same abstraction level as itself. It will be written according to what is required. There will be a fixed arrangement between the two classes. Over time, this fixed arrangement will cause a blurring of their respective responsibilities.

Therefore, classes in ALA do not have association relationships. Instead they just have fields of the type of these more abstract interfaces or they implement these more abstract interfaces. We call both of these ports.

The abstract interfaces that we put in lower layers are obviously have to be be general. It can be hard to see how this could work, but it does. For example, interfaces often implement a set of four methods for CRUD operations (Create, Read, Update, Delete). The very existence of this acronym suggests an abstract interface.

2.3.5. Composition with objects

An abstract interface in a lower layer makes it much easier to have multiple classes implement or require them. Objects of different classes can then be composed together in arbitrary ways, giving us the powerful principle of compositionality. (The meaning of a complex expression is determined by the meanings of the constituent expressions and the rules used to combine them.)

Abstract interfaces suggest general meanings for the ways we compose objects. They end up looking a lot like programming paradigms, which is why we call the layer ProgrammingParadigms.

Any given class will typically implement/accept more than one of these abstract interfaces. These are the called ports. When using dataflow they are I/O ports. We do not think of the objects that get wired to these ports as clients. The word client is best used for classes in a higher layer (that use the main interface). The classes of the objects to which an object is wired are just called peers.

2.3.6. Composition with functions

ALA can be applied to functional programming too. Abstractions are then obviously functions, and the same ALA relationship restriction applies - a function may only call a significantly more abstract function. The functions then form layers.

Where one function may have called a peer function in conventional code, now a higher layer function that has the system specific knowledge is needed to call the first function and then call the second function, in effect composing with functions. Parameters and return values are effectively port. If the first function called the second function in the middle rather than at the end, the second function will now need to be passed into it. The function parameter is also a port.

A higher layer function may call a series of lower layer functions, passing data from one to the next. We don’t often need the data in the higher level function. All we are trying to do is compose functions. It clutters up the code something awful when we have to handle data without needing to use that data. So that’s why we prefer to compose with objects with ports.

Monads also allow composition of functions without this cluttering. But they only support a dataflow type of programming paradigm. We want to compose using multiple ports on our abstractions using multiple programming paradigms. Objects with ports are a more straightforward way to think about this. Monads are objects under the covers, and this is part of the reason they are hard to understand (although eventually you get used to it). By composing with objects directly, it is clearer what is going on.

2.3.7. Readme file

There should be a readme file in the root folder that points to this website (or equivalent documentation) about ALA. In ALA, we are explicit about what knowledge is needed before a given piece of code can be understood (knowledge dependencies). To understand an ALA application, you need a basic understanding of ALA (from this chapter). So that’s why there should be a readme file pointing here.

2.4. Executable expression of requirements

This is the third perspective of ALA. It is essentially the perspective of a DSL (domain specific language).

Writing software is re-expressing requirements in a different language. If that language is general purpose, we end up using a lot of symbols to express those requirements - many more than we would use in English. This is because in English we would use, or even invent terms in the domain to help us to be expressive and succinct. I’m not talking about customers or orders. I’m talking about general concepts for the UI, of data storage, or of certain events implied in our user stories.

Furthermore, if we are re-expressing requirements in a language that is essentially imperative (executes step by step instructions in computer time) it’s going to be super awkward. Things like UI layout or asynchronous events don’t map directly to imperative style. It’s going to require a lot of cleverness to express them every signle time.

Furthermore, let’s say it takes 100 lines of English to state the requirements and 10000 lines of code to implement the requirements. Potentially all 10000 lines of code know about the requirements in some way. Each also knows about some computing detail like how to store data, how to do input/output, or how to schedule what its doing in real time. The details of requirements and the details of computing are mixed together. The expression of requirements is extremely verbose in such a design.

ALA separates out the expression of requirements from computing details. It does this by first identifying the types of relationships inherent in requirements. These are not imperative calls in computer time. They are things like dataflows, events, and UI layouts. We invent programming paradigms for these. Then we invent types of things implied in the requirements with these relations. Things like UI elements, data transformations, data stores, and transactions. Now we have a domain specifc language. We compose instances of the thing types together using instances of our relationship types. The resulting expression of requirements is direct and succinct.

In ALA the thing types are called domain abstractions. The relations are called programming paradigms. Each domain abstraction hides one generic piece of computing implementation. Each programming paradigm defines what the composition of two domain abstractions means. Programming paradigms hide an execution model for how the relationship will work in terms of underlying imperative execution.

It is the top layer (or top two layers for larger applications) that describe all the details in the requirements (and nothing but the details of requirements.) All details of actual computing work go are the implementations inside the domain abstractions and programming paradigms.

The amount of code that describes requirements is typically about 3-10% of the entire application. When requirements change, you only need to understand this 3-10%.

The percentage of code expressing requirements does depend on how many requirements there are. Because ALA emphasises the use of abstractions, and abstractions are reusable, an application with many requirements may have so much reuse that the percentage of code expressing requirements goes higher.

The expression of requirements in the top layer is executable. This could be compared with BDD (behavioural driven design) which is also expresses the requirements and is executable. But BDD only executes the tests. ALA goes one step further to make the expressed requirements the executable solution.

The executable description of requirements in the top layer is also the architecture or the design. (I do not make a distinction between architecture and design.) There is no separate artefact or documentation of the 'architecture', no model, no other "high level" design document. The one artefact expresses requirements, expresses the architectural design, and is the executable. So one source of truth for everything.

2.4.1. Polyglot programming paradigms

In this perspective of ALA, we view it as a vehicle for multi-paradigm programming.

Many higher level programming models are based on a single programming paradigm. Examples are the dataflow model, event-driven, actor (message passing) model, or ladder logic.

A given paradigm makes it easy to solve some problems but not others. Having a single programming paradigm makes the programming model pure and simple. But it’s just plain awkward for certain aspects of typical problems.

So ALA takes the approach that not only will we be able to compose using multiple programming paradigms, we do it it with the same wiring operators. This allows them to be easily intermixed in the same user story or feature, or in the same application diagram. Furthermore, creating and implementing a new programming paradigm is straightforward.

In the end, we want to attain a certain level of expressiveness of composition. If we are too expressive we wont have enough versatility to vary our applications in the domain. If we don’t have enough expressiveness, we will have to compose too many low level elements to get anything done.

Some examples of programming paradigms that we use frequently are UI layouts, dataflow, schema relationships, state transitions.

Each programming paradigm usually results in a type of port for the domain abstractions. Instances of two abstractions can then be wired by their compatible ports. The programming paradigm provides the meaning of that composition, and provides the execution model for that meaning to be carried out.

There are other types of programming paradigms that don’t need to use ports as well. For example, say you want a programming paradigm for style. You would create a Style concept abstraction in the programming paradigms folder. Then every UI domain abstraction would get its style properties from this abstraction. Then the application layer configures the style abstraction for a specific application, and all UI instances would take on that style. You would only use this method rather than ports if every ports would be connected to a single instance of something. If things are connected to one instance, that indicates that the instance itself can be an abstraction, and simply be put down a layer for everyone to access.

Some will disagree with the last paragraph as it effectively makes the style object a global. That’s not great even if it is a good abstraction. Indeed if you want to say test a UI domain abstraction with styles, and do these tests in parallel, the global wont work. Or there may be leftover state in the global between different tests. Or we may want to override the style on one UI instance. If we don’t want to use global instance of style, then we go back to ports. We then create an instance of style and wire every instance of every UI domain abstraction to this instance. To make such wiring easier, I have WireMany operator. This extension method will look for a compatible port on every instance of every domain abstraction.

2.5. Diagrams

In this perspective, we view ALA as a vehicle for diagram oriented design.

We don’t have to use diagrams in ALA. It only comes about because requirements typically contain a whole network of relationships. For example, UI elements have spacial relationships with one another. They have relationships with data. Data has relationships with storage (state which is expected to persist across user stories). Stored data has it’s own inter-relationships. All have relationships with real-time events.

In conventional code, this network of relationships results in a network of dependencies across the modules of the code. These types of dependencies are used for run-time communications. Inverting these types of dependencies doesn’t help. It’s still a dependency that’s only there for run-time communications. We don’t like circular dependencies, but communications are often naturally circular. So we introduce still more indirections, obscuring the natural network even further.

The result is a big ball of mud. It consists of thousands of symbolic references. 'All files' searches are needed to find these references and unravel the network.

ALA elliiminates this network of dependencies and replaces them with ordinary lines of code that instantiate abstractions and wire them together. That code is placed inside the top application abstraction (or into a set of feature or user story abstractions). The code is cohesive because it represents in one place the network of relationships that make up a feature user story.

In this wiring code, the network nature of the inter-relationships within a user story between instances of domain abstractions becomes obvious. It can become clear that the best way to express it is a diagram. ALA therefore uses diagrams quite often. These diagrams are, more or less, like static UML object diagrams. So in ALA, we throw away the UML class diagrams (relationships between abstractions), and use UML object diagrams instead (relationships between instances).

2.5.1. Diagrams vs text

In the trivial examples of composition that we already gave, we either used fluent style text for the wiring code drectly, or we manually translated a diagram into fluent style wiring code. You may wonder, why bother with diagrams? They require a tool.

There is a bad meme in the software industry that diagrams and text are equivalent for representing software, They are far from equivalent. They each have different strengths and weaknesses. Using the wrong one will significantly increase the difficulty of reading it.

The only reason I convert diagrams to fluent style text is because our examples are small and they have a mostly linear or shallow tree topology. I manually translate the diagrams to readable text to show how the diagrams execute.

Linear diagrams and shallow trees can better represented in text. Deep trees become hard to read because it results in too much indenting. For larger diagrams with arbitrary cross connections, using text requires a lot of 'symbolic connections' or labels to represent what would be anonymous lines on the diagram. These symbolic wirings make the code much harder to read as their number increases beyond a few connections. If you find yourself doing "all files searches" when reading code, you finding those connections, painstakingly one at a time. If you use a diagram, you just follow the lines.

An advantage of expressing the network in diagram form is that you don’t have to give names to instances. You can leave them all anonymous if you want to. Sometimes you will give them names anyway as documentation. For example, if you have two grids implied in your requirements, you will want to give them names so you know which is which in the diagram.

Avoiding diagrams is like an electronics engineer avoiding a schematic, or an architect avoioding drawings. In conventional code, a network of relationships implied by the requirements is still in there. If we use text to represent it, it is much harder to read, even if we put it in one cohesive place. But what we actually do is even worse. We distribute that text throughout our modules, making it difficult to see. That is why conventional code typically becomes a mass of dependencies resembling a big ball of mud. Using such tricks as dependency inversion, indirections, or container style dependency injection makes the situation even worse. The network of relationships is still there, but now it is even more difficult to see.

Sometimes programming with diagrams is called model driven software development. I prefer not to use the word 'model'. In the real world, models leave out details. Software models tend to leave out details too. ALA diagrams do not leave out details. All details from the requirements are represented, for example in the form of configuration of the instances. That no details are left out is why the diagram is also the executable.

The diagram is stored in the application layer folder. When a diagram is used for the internals of a feature or user story, it resides in the respective layer folder. When diagrams become large, they need two tools. One tool allows you to draw the diagram, and the other generates wiring code automatically. The generated code does not use fluent style - it is just a list of instantiations followed by a list of wirings between them. The wiring code generated from the diagram lives in a subfolder from where the diagram is, because it is not source code.

If manually generating code from a diagram, the diagram should always be changed first, then the code. There should be a readme explaining exactly what the generated code should look like.

Automatically generated code does not need to be readable except to the extent of finding where it doesn’t accurately reflect the diagram.

2.6. Composition vs decomposition methodologies

In this perspective, we look at software design methodology. Conventional wisdom is a decomposition approach. You decompose a system into modules or components. Those modules are further decomposed into submodules and so on. By contrast, ALA is a composition approach. It composes the system from instances of abstractions. Those abstractions are composed from instances of even more abstract abstractions. The difference is important as it results in a completely difefrent structure.

In the next chapter we will discuss in detail why ALA uses a 'composition' approach rather than a 'decomposition' approach. Here we describe the two different structures that result from these two approaches.

In the conventional approach, components tend to get more specific than the system because they are specific parts of it. It is a bit like jigsaw pieces to a jigsaw picture. The pieces are not reusable. The picture is not a separate entity - it is just the set of pieces, which have a rigid arrangement with each other. The picture cannot change without changing the pieces.

In the ALA approach, abstractions used to compose a system must be more abstract than the system. It is a bit like lego pieces to a specific lego creation. The pieces are reusable. The lego creation is a separate entity in itself - it is more than the set of pieces. The lego creation can change without changing the pieces.

2.6.1. Encapsulation hierarchy vs layers

Because a decomposed system tends to create modules that are specific to the system, these modules tend to be not reusable. They may be replaceable with modules that have the same interfaces, but not actually reusable. We tend to encapsulate such modules inside the system. Similarly with submodules, we encapsulate them inside their modules. This creates an encapsulation hierarchy. It is sometimes likened to a map in which we can zoom in for greater detail.

This actually doesn’t work for hiding information at design time. Because the modules are specific and not abstractions, you will always have to zoom in for the details of the inner modules in order to understand the system. Encapsulation makes as much sense as hiding the picture on every individual piece of a jigsaw puzzle unless you zoom into it, then trying to see the big picture.

The encapsulation may help to unclutter the IDE namespace at the system level, but it doesn’t reduce how much you have to go inside the encapsulations to understand the system.

If the modules and submodules are abstractions, and those abstractions are more abstract than the modules that use them, then we don’t have to zoom in. We can understand a system in terms of the abstractions it uses.

Abstractions are reusable. So we explicitly do not want to encapsulate them inside something that uses them. We need them to be public for reuse. Instead of encapsulating them, we use abstraction layers.

2.6.2. Primary separation

Decomposition tends to break up a system first according to these types of criteria:

  • locations of physical machines or processors (e.g. tiers, services)

  • computing problems (e.g. UI, business logic, data storage)

  • business structure (Conway’s law)

A system decomposed in this way will make features or user stories span the modules. This is bad. It forces us to create dependencies for communications within a feature or user story.

In ALA, features or user stories are obvious abstractions given to us by the requirements. As such we keep them together, even if they cross over these other boundaries. For example, if UI, business logic, and storage span three different machines, there is nothing stopping us coding or drawing a single diagram containing all the elements of the UI, business logic and storage for a user story. The elements will be instances of abstractions deployed on different machines, but that is a deployment detail. As long as the internal lines in the diagram represent asynchronous communications, the feature or user story will still work when deployed. Deployment time abstractions can insert the necessary middleware. No specific interfaces are needed between the elements on different machines because they are instances of abstractions that already have compatible asynchronous-ready ports. (We cover asynchronous ports in detail in chapter four.)

The way the resulting code is organised will be completely different from a convention decomposition. The relationships that exist between conventional modules will disappear. They become cohesive lines of code inside a new abstraction representing the system. In fact that’s all the system abstraction will need to do.

contrasting the two structures
Slide8.jpg
Figure 11. Decomposition into elements and their relations

The figure shows five conventional modules (or components) and their relations (as interactions). Study almost any piece of software, and this is what you will find (even if it supposedly adheres to the so-called layering pattern).

The structure generally can be viewed as 'clumping'. Like galaxies, certain areas have higher cohesion, and so go inside boxes. Other areas are more loosely coupled, and so are represented by lines between the boxes. The difference between high cohesion and loose coupling is only quantitative.

Software health in this type of architecture is effectively management of the resulting coupling between the cohesive clumps. Allocate code to boxes in such a way as to minimize coupling. This coupling management has two conflicting forces. One is the need to have interactions to make the modules work as a system. The other is to minimize the interactions to keep the modules as loosely coupled as possible. As maintenance proceeds, the number of interactions inevitably increases, and the interfaces get wider. Cohesion will reduce, and coupling will increase over time.

Various architectural styles are aimed at managing this conflict. Most notably:

  • Layering pattern (break circular dependencies and replace them with indirections which are even worse.)

  • Try to avoid both high fan-in and high fan-out on a single module

  • Try to avoid dependencies on unstable interfaces

  • MVC type patterns

Note that none of this 'dependency management' actually avoids design-time coupling. There will always be 'implicit coupling' in both directions between modules of a decomposed system, regardless of the dependencies. This is because the modules are the opposite of abstractions - specific parts designed to interact or collaborate to make a system. For example, a function of a decomposed system will tend to be written to do what its caller requires even if there is no explicit compile-time dependency on its caller. So circular coupling may be avoided at compile-time, but will still be present at design-time. That is why in the diagram above, couplings are drawn from the insides of each of the modules in both directions. This indicates that the code inside the modules has some inherent collaboration with the code inside other modules. To the compiler or a dependency graphing tool, the lines may appear to be layered, but this is not telling you the whole story of the design-time coupling.

The compose approach

When you use abstractions instead of modules, there is qualitative difference in how the structure is built. There are no interactions, collaboration, or coupling between abstractions:

Slide9.jpg
Figure 12. Abstraction do not interact

The word 'modules' has been changed to the word 'abstractions'. All the dependencies are gone. And with them all their problems. You no longer have to worry about dependencies and all their management. The implicit coupling that we talked about earlier is also gone. The 'clumping' structure has become isolated boxes. Loose coupling has become zero coupling.

The obvious question now is how can the system work? Where do all the interactions between elements that we had before go? The answer is they become normal code, completely contained inside one additional abstraction. This code composes instances of the abstractions to make a system:

Slide10.jpg
Figure 13. Abstractions and composition of their instances

The code inside the new system abstraction does not involve dependencies between abstractions. It uses dependencies on abstractions. It’s code that instantiates abstractions and wires them together via their (even more abstract) ports. Since interactions between the instances are implemented without dependencies between the abstractions, circular wiring is fine. In fact we should embrace it, because that is how the system works.

This instantiation and wiring code is cohesive. It is the code that has knowledge of the specific system. None of the code inside the abstractions knows about the specific system, only this new code.

We put the abstractions, A, B, C, D and E into a layer. The system abstraction goes in the layer above.

Software engineering should not be about managing coupling.

It should be about inventing abstractions.

2.7. cf Component based development

In this perspective, we compare ALA with components, component based software engineering (component based development), components and connectors.

When you read the intentions for components, they are meant to be reusable. Since reuse and abstraction go hand in hand, it should follow that components are abstractions. Furthermore, just as we do in ALA, they have ports to supposedly allow them to have run-time communications with one another without breaking them as abstractions.

That’s the intentions. In practice, all the component diagrams I have seen fall far short of this ideal. The components themselves appear to be specific pieces of a specific system. Although they have ports to allow reuse, they are too specific to the system they are designed for to be reusable abstractions. There maybe exceptions of course, but components lack a fundamental rule that constrains components to be more abstract than the systems they are used in.

The UML component diagram uses lollipops to represent the ports. At first this seems great because it looks like you should then be able to wire them up in arbitrary ways. But, at the port level, all the example component diagrams I have seen use interfaces that are specific to one or other of the connected components. In other words the components have a fixed arrangement with each other. This in turn encourages them to collaborate and have implicit coupling with each other. It is the jigsaw analogy.

In ALA, you must have compositonality. This means the abstractions have no fixed arrangement. You have the capability to compose instances of abstractions in an infinite variety of ways. It is the lego analogy.

Component architecture does have one thing - the ability to sunstitute one component for another with the same interfaces.

Components allow hierarchical composition by having sub-components, but I am not clear on what that means. Does it mean component instances or component types? Some implementations I have seen allow you to configure the visibility of a component type. This means that component type can be completely contained inside another component type. This type of hierarchy is illegal in ALA because used components must be more abstract and therefore must be defined outside where they are public for reuse. Of course using instances of a component inside another component is the whole point of how we build up a system in either ALA or component driven development.

2.7.1. Components and connectors

One implementation model for components is so called components and connectors. The mechanics of components and connectors is that the lines drawn between components are connector objects. They contain a value, which is the 'data on the wire'. Thinking of the wire as being a variable with a value is quite a useful programming paradigm. It is also a relatively efficient execution model. The variable itself is a shared variable that isn’t globally visible. Only the two instances of components that are wired together can ever see it. Senders need only set the value of the variable, and receivers need only read the variable.

When two compatible instances of components are composed or wired together using this programming paradigm, the implied connector object is created automatically and wired inbetween them.

The instances of the components must be active objects (somehow execute by themselves).

Let’s see how to create such a programming paradigm for ALA applications:

Connector.cs
namespace ProgrammingParadigms
{
    class Connector<T> : IOutput<T>, IInput<T> (1)
    {
        T data { get; set; } = default(T);
        T IOutput<T>.data { get => data; set => data = value; }
        T IInput<T>.data { get => data; }
    }


    public interface IOutput<T> (2)
    {
        T data { get; set; }
    }

    public interface IInput<T> (3)
    {
        T data { get; }
    }


    public static class StaticMethods
    {
        public static void Wire<T>(ref IOutput<T> Aport, ref IInput<T> Bport) (4)
        {
            Connector<T> connector = new Connector<T>();
            Aport = connector;
            Bport = connector;
        }
    }
}
1 The connector type itself. Instances are to be wired between two instances of domain abstractions.
2 One domain abstraction must have a port implemented as a field of the IOutput interface.
3 One domain abstraction must have a port implemented as a field of the IInput interface.
4 A method for wiring two instances of domain abstractions creates the connector for you, and then wires the two instances to it.

Let’s create two domain abstractions to demonstrate the use of this programming paradigm. First a domain abstraction with an output port using this programming paradigm.

NaturaNumbersTenPerSecond.cs
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary>
    /// ALA Domain Abstraction
    /// Demo class to send data via a connector
    /// </summary>
    class NaturaNumbersTenPerSecond
    {
        public IOutput<int> output; (1)


        public void Run() (3)
        {
            RunAsyncCatch();
        }


        public async Task RunAsyncCatch()
        {
            // because we are the outermost async method, if we let exceptions go,
            // they will be lost
            try
            {
                await RunAsync();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }


        public async Task RunAsync()
        {
            int data = 0;
            while (true)
            {
                data++;
                output.data = data; (2)
                await Task.Delay(100);
            }
        }
    }
}
1 The output port
2 Outputting data to the output port
3 This domain abstraction is active, so we need a Run method to start it running.

And another domain abstraction with an input port using this programming paradigm.

ConsoleOutputEverySecond.cs
using ProgrammingParadigms;

namespace DomainAbstractions
{
    /// <summary>
    /// ALA Domain Abstraction
    /// Demo class to send data via a connector
    /// </summary>
    class ConsoleOutputEverySecond
    {
        public IInput<int> input; (1)


        public void Run()
        {
            RunAsyncCatch();
        }


        public async Task RunAsyncCatch()
        {
            // because we are the outermost async method, if we let exceptions go,
            // they will be lost
            try
            {
                await RunAsync();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }


        public async Task RunAsync()
        {
            while (true)
            {
                Console.WriteLine(input.data); (2)
                await Task.Delay(1000);
            }
        }
    }
}
1 The input port
2 Inputting data from the input port

And finally an application to wire together instances of these domain abstractions:

Application.cs

using DomainAbstractions;
using static ProgrammingParadigms.StaticMethods;

namespace Application
{
    class Application
    {
        /// <summary>
        /// Application to demonstrate two active components running at different rates
        /// communicating using a connector.
        /// </summary>
        public static void Main()
        {
            NaturaNumbersTenPerSecond numbers = new NaturaNumbersTenPerSecond();
            ConsoleOutputEverySecond console = new ConsoleOutputEverySecond();
            Wire(ref numbers.output, ref console.input);

            numbers.Run();
            console.Run();
            Console.ReadKey();
        }
    }

}

2.8. Real world analogies

2.8.1. Atoms and molecules

Here are two atom abstractions:

Oxygen atom Hydrogen atom

Instances can be composed to make a molecule:

Water molecule

If water was implemented in the same way we typically write software, there would be no water molecule per se; the oxygen atom would be modified to instantiate hydrogen atoms and interact with them. Even if dependency injection is used to avoid the instantiating, it is still unlikely that a water abstraction would be invented to do that, and there would still be the problem of the oxygen atom interacting with hydrogen’s specific interface. The oxygen module still ends up with some implicit knowledge of hydrogen. And hydrogen probably ends up with some implicit knowledge of oxygen in providing what it needs.

This implicit knowledge is represented by the following diagram. The relationship is shown coming from the inner parts of the modules to represent implicit knowledge of each other.

diagram o h

While oxygen and hydrogen are modules, they are not abstractions because oxygen is implicitly tied to hydrogen and vice-versa. They can’t be used as building blocks for any other molecules.

To keep oxygen as abstract as it is in the real world, an interface must be conceived that is even more abstract than oxygen or hydrogen. In the molecule world this is called a polar bond. It is one of the programming paradigms of molecules. Its execution model at run-time is the sharing of an electron.

The corresponding software would look like this:

Slide15.jpg
Figure 14.

The water molecule has a "uses instances of" relationship with the two atoms, and the atoms have a "uses instance of" relationship with the even more abstract polar bond abstraction. Polar bond is an example of an 'abstract interface'.

2.8.2. Lego

The second real world analogy is Lego. Shown in the image below is the same three layers we had above for molecules, atoms and bonds.

Slide16.jpg
Figure 15.

The domain abstractions are the various Lego pieces, instances of which can be assembled together to make specific creations. Lego pieces themselves have instances of an abstract interface, which is the stud and tube. This is a programming paradigm. There is a second abstract interface, the axle and hole. These programming paradigms have an 'execution model' at run-time. The execution of the stud and tube programming paradigm is to hold structural integrity. The execution of the axle and hole programming paradigm is rotation.

Conventional code analogy

If Lego is a good analogy for ALA, then what would be a good analogy for conventional code?

It’s an upside down jigsaw puzzle.

jigsaw.png
Figure 16. Conventional code is built like a jigsaw puzzle

The pieces are modules, and the interlocking shapes are the interfaces. The picture is the application or system.

Like the interlocking shapes, interfaces tend to be specific to pairs of modules. They may nominally belong to one module or the other, but the complimentary one bends to that interface, and vice versa. They have one rigid structure for how they fit together. The fixed relationship between modules tends to cause collaboration and coupling between them, and this tends to get worse over time as new features or user stories are added.

The jigsaw puzzle is upside down because there is no view of the complete picture. You are allowed to pick up one piece at a time and look at the part of the picture that’s on the other side. This is the equivalent of opening one module and reading the code inside it. By doing this repeatedly for many adjacent modules, you can start to get an idea of how part of the bigger system works. But, you have to keep the pieces of picture in your head, because there is no explicit view of it.

A jigsaw is all in one layer. The big picture, interfaces and pieces all exist in that one layer as a single information entity. ALA has at least three layers of information. Firstly, the general ways that pieces can be combined is an information entity in the lower layer. The building block types exist the next higher layyer. And particular arrangment of instances of building blocks is a separate information entity in a higher layer.

Essentially ALA, like Lego, has the property of compositionality. Conventional code modules, like jigsaw pieces, generally do not.

2.8.3. Electronic schematic

The third real world analogy comes from electronics. The abstractions are electronic parts, instances of which can be composed as a schematic diagram:

Slide17.jpg
Figure 17.

In this domain, we have at least two abstract interfaces as programming paradigms, one for digital logic signals and one for analog signals. Their execution model at run-time is continuous-time voltage levels.

2.8.4. A clock

Our forth and final real world analogy is a clock. In this diagram, we show the process of composition of abstractions to make a new abstraction. The process is a circle because instances of the new abstraction can themselves be used to make still more specific abstractions. Each time around the circle adds one layer to the abstraction layering.

Slide18.jpg
Figure 18.

Let’s go round the circle once. We start with abstract parts such as cog wheels and hands. Instances of these have abstract interfaces as programming paradigms. Their execution models allow them to interact at run-time, such as spinning on axles and meshing teeth. The next step is to instantiate some of these abstractions and configure them. For example, configure the size and number of teeth of the cog wheels. Next comes the composition step, where they are assembled. Finally we have a new abstraction, the clock. Instances of clocks can in turn be used to compose other things such as scheduling things during your day. Because you have now created the abstraction clock you don’t have to think about cog wheels when thinking about how to meet someone at a certain time.

There are many other instances of this pattern in the real world, and in nature. In fact everything appears to be actually composed in this way. At least that’s the way we understand and make sense of the world - in terms of abstractions, which are in composition layers.


2.9. Example Project - Coffee machine

Robert Martin posed an interesting pedagogical sized embedded system problem about a coffee maker in his book “Agile Software Development: Principles, Patterns and Practices”. The original chapter can be found by searching for “Heuristics and Coffee”.

Although I agreed with Martin that his students' 'object oriented' solutions were hideous, I did not like his worked solution either. Although he had claimed to use abstractions, they were actually collaborating modules, just with abstract sounding names. So you had to read all the code to understand any of it. I wanted to know if ALA would tame the problem into a solution as succinct as the requirements. Because this is the first ever ALA project I did, some of the more refined ALA conventions, patterns and methods are not used here. But the fundamental constrains are met, and the result is spectacularly simple compared to Martin’s solution.

Martin’s worked solution to this problem uses decomposition into three modules that collaborate or interact with one another. The ALA solution follows the opposite philosophy. It has three abstractions. They come from the specification - a button with an indicator light, a warmer plate, and a boiler. They do not collaborate or interact with one another. As domain abstractions, they also know nothing about the coffee machine. The coffee machine is then constructed (as another abstraction in the top layer) that makes use of the three domain abstractions.

This example uses different execution models from the UI and dataflow ones that we use a lot in other examples. Here we use some simple, yet quite interesting electronic-signal-like execution models that use a simple main-loop polling type implementation, just as Robert Martin’s original solution also had.

Reading an ALA application requires first knowing the pre-requisite knowledge you need from lower layer abstractions. So before presenting the application, let’s first familiarise ourselves with the abstractions we need from the domain layer, and the Programming Paradigms layer.

2.9.1. Domain abstractions layer

Here are the three domain abstractions:

Coffee Maker Domain Components.vsd.jpg
Figure 19. Coffee maker domain abstractions

Take a moment to look at these three abstractions:

 — The UI has a lamp you can control, and a push button which outputs an event (should have been two separate abstractions).

 — There is a WarmerPlate. It tells you whether or not a container is on the warmer plate, and whether or not it is empty. It controls its own heater.

 — There is a Boiler. It can be turned on or off. It will tell you when it is empty of water. And you can stop water flow instantly with a steam release valve. It will turn its own heater off if it runs out of water, or the valve is opened.

That’s all there is to know about the three domain abstractions.

2.9.2. Programming Paradigms layer

We have three programming paradigms

 — live dataflow (works like an electronic circuit)

 — events

 — simple state machine

The API for the Programming Paradigms layer is described in the key on the right of the diagram below. It gives you all the knowledge from this layer to be able to read the diagram. So, for example, a solid line is a dataflow; the rounded box is state with the states enumerated inside it.

The details of how to turn the diagram into code is explained in a project document, also provided in the Programming Paradigms layer.

2.9.3. Application layer

Now that we have understood the knowledge dependencies in all lower layers, we can read the diagram that resides in the top layer, the application layer:

CoffeeMaker Dataflow diagram
Figure 20. Coffee maker solution

The diagram to the left is the application itself. Instances of the three domain abstractions, UI, Boiler and Warmer plate are shown as boxes.

Follow me now as we go through the user stories by looking at the lines on the diagram:

  • When the UI push button is pressed, we set the state to Brewing, provided the Boiler is not empty and the pot is on the Warmerplate. (On the diagram, it looks like a NOT operator is missing on the Empty signal from the boiler to the AND gate.)

  • When the state is brewing, it turns on the boiler, and coffee making starts.

  • If someone takes the pot off, the valve is opened to momentarily release pressure from the boiling water, which stops the water flow.

  • When the boiler becomes empty, the state is set to Brewed. When the state is Brewed, the light in the UI is on.

  • When the coffee pot is replaced empty, the state goes back to the idle state where we began.

That’s all there is to reading this application. The code for the coffee machine can be read and understood in about one minute. Compare that with reading other solutions to the coffee machine problem.

Note that the paragraph above is pretty much a restatement of the requirements in English. It could have been the requirements. The amount of information in the English form (or the diagram form) is about the same, thus the Domain Abstractions gave us the correct level of expressiveness. Further confirmation of this is if the level of expressiveness allows us to modify it.

For example, say a requirement was added that a coin device was to enable the machine to be used. The coin device is an abstraction that provides an output when a coin is given, and has a reset input. Looking at the diagram, and being able to reason about its operation so easily, you can see that the coin device’s output would intercept the Pushbutton using another instance of an AND gate. And to reset the coin device, you could use the boiler empty output event.

2.9.4. Execution

To make it actually execute, we apply the manual procedure documented in “Execution models.doc”. This document is in the Programming Paradigms layer. It will generate these 6 lines of code:

if (userInterface.Button && warmerPlate.PotOnPlate && !boiler.Empty) { state = Brewing; } userInterface.Button = false;
boiler.OpenSteamReleaseValve = !warmerPlate.PotOnPlate;
boiler.On = state==Brewing;
if (boiler.Empty && !prevBoilerEmpty) { state = Brewed; } prevBoilerEmpty = boiler.Empty;
if (warmerPlate.PotEmpty && !prevPotEmpty) { state = Idle; } prevPotEmpty = warmerPlate.PotEmpty;
userInterface.LightOn = state==Brewed;

There is a one-to-one correspondence between the lines in the diagram and the lines in the code.

As you can see, the execution model is a simple one. The 6 lines of code are continually executed in a loop. This execution model is effective and appropriate for this small application.

The 6 lines of code can be built into a complete program shown below:

 #ifndef _COFFEE_MAKER_H_
 #define _COFFEE_MAKER_H_
 // Coffee Maker domain abstraction
 #include "CoffeeMakerAPI.h"  // original hardware abstraction supplied by hardware engineers
 // Knowledge dependencies :
 // "PolledDataFlowProgrammingParadigm.doc" -- explains how to hand compile a dataflow diagram of this type to C code
 // Following are 3 Domain abstractions that the application has knowledge dependencies on



 #include "UserInterface.h"
 #include "Boiler.h"
 #include "WarmerPlate.h"



 class CoffeeMaker
 {
 private:
    enum {Idle, Brewing, Brewed} state;
    Boiler boiler;
    UserInterface userInterface;
    WarmerPlate warmerPlate;
    bool prevBoilerEmpty, prevPotEmpty;
    void _Poll();
 public:
    CoffeeMaker()
        : state(Idle), prevBoilerEmpty(boiler.Empty), prevPotEmpty(warmerPlate.PotEmpty)
    {}
    void Poll();
 };
 #endif //_COFFEE_MAKER_H_
 // CoffeeMaker.c
 // This is not source code, it is code hand compiled from the CoffeeMaker application diagram
 #include "CoffeeMaker.h"

 void CoffeeMaker::_Poll() (1)
 {
    if (userInterface.Button && warmerPlate.PotOnPlate && !boiler.Empty) { state = Brewing; } userInterface.Button = false;
    boiler.OpenSteamReleaseValve = !warmerPlate.PotOnPlate;
    boiler.On = state==Brewing;
    if (boiler.Empty && !prevBoilerEmpty) { state = Brewed; } prevBoilerEmpty = boiler.Empty;
    if (warmerPlate.PotEmpty && !prevPotEmpty) { state = Idle; } prevPotEmpty = warmerPlate.PotEmpty;
    userInterface.LightOn = state==Brewed;
 }



 void CoffeeMaker::Poll()
 {
    // get inputs processed
    userInterface.Poll();
    boiler.Poll();
    warmerPlate.Poll();
    // run application
    _Poll();
    // get outputs processed
    userInterface.Poll();
    boiler.Poll();
 }
1 The 6 lines of code appear in the "CoffeeMaker::_Poll()" function.

If you are using a diagram as we are in this solution, you always change the diagram first when the requirements change. It provides the expressiveness needed to see the application’s requirements represented in a clear, concise and cohesive way. There the logic can be ‘reasoned’ with. It is not documentation, it is the source code representation of the requirements, and executable, both important aspects of ALA.

The next step is to implement the three abstractions. These are straightforward using the same execution model as was used for the application, so are not shown here.

The resulting application passes all of Martin’s original acceptance tests plus a number of additional tests of behaviour gleaned from his original text.

3. Chapter three - Why the structure works

In the previous chapter we described what the structure, the anatomy, of ALA looks like as if we were dissecting a dead body. We see where things are but we don’t yet understand why they are there. In this chapter we explain why that structure works. Why does this way of organising code result in software that meets those non-functional requirements we listed in Chapter one?

The organisation of this chapter (and all chapters) is to use different perspectives. We all have different prior knowledge on which we build new knowledge, so we will each have a different best way to understand things. Use the perspective that makes the most sense to you. Because of the use of perspectives, there will often be some repetition of ideas between the major sections.

3.1. A thought experiment

Imagine you are reading the following function, abc123, and trying to understand it:

float abc123(float[])
{
    ...
    b = xyz789(a)
    ...
}
float xyz789(float)
{
    ....
    // complicated code
    ....
}

You don’t know what abc123 nor xyz789 do. They may as well be called fubar (fubar stands for ****ed up beyond all recognition), so you follow the indirection, an inconvenience at the least because you are really just wanting to understand abc123. You have to mentally stack where you were in the code of abc123, including everything you understand about it so far.

You begin reading the code in xyz789. It only has about 20 lines but it is complicated. You need to use the code in abc123 to try to unravel what xyz789 might be providing to it. A comment mentions that it uses a CORDIC algorithm and gives a reference. But before following that indirection as well, you note that both abc123 and xyz789 have the following properties:

  • they are modules

  • apparently loosely coupled

  • have a simple interface

  • use encapsulation of internals

  • use no external variables

  • have no side effects

  • hide information

  • probably separate two concerns

  • are small

  • follow coding guidelines

  • have comments

Despite having all these great properties, still we are forced to read both functions to understand the code in either of them. They are effectively fully coupled - understanding any of the code involves understanding all of the code.

Now we make a small change:

float StandardDeviation(float[])
{
    ...
    b = Sqrt(a)
    ...
}
float Sqrt(float)
{
    // complicated code
}

Suddenly understandability is absolutely transformed. All we did was make the two functions abstractions. Now we don’t have to read the complicated code inside xyz123 at all. The code inside each of the two functions goes from highly mutually coupled to zero coupled.

All those other attributes that we listed above seemingly made no difference. The quality attribute that really mattered was abstraction. The others are still good to have, but they are completely insufficient. The abstraction property is the one that our brains have evolved to use.

The quality of abstraction is subjective. Software engineers must invent good quality abstractions. No compiler or tool can yet check that quality.

  • In the downward direction, coupling goes to zero because the standard deviation function need only know the concept of the squareroot abstraction.

  • In the upward direction, coupling goes to zero because squareroot is more abstract and therefore can’t know anything about the more specific Standard deviation abstraction that happens to use it.

There are other benefits too:

  • Abstraction and stability go hand in hand. The Sqrt abstraction is as stable as the concept of squareroot. That’s a concept that’s been stable for thousands of years. All dependencies in an ALA program go in the direction of the more stable.

  • Abstraction and reuse go hand in hand (as pointed out by Krueger). The more abstract an abstraction is the more reusable. Code reuse in ALA programs increases markedly.

The complicated code inside SQRT no longer matters. It is completely isolated by the abstraction. If your brain already knows the SQRT concept (I had to choose one that everyone knows), there is no need to follow the indirection when reading the code inside StandardDeviation. The reader just continues reading the next line of code after the Sqrt invocation as if Sqrt is just like any other line of code in their base language. That’s what abstraction is.

With this new understanding, we will now define the word dependency to be compile-time relationships, and coupling to be the design-time. One is what the compiler sees, the other is what our brain sees.

Using these definitions, you can have coupling without dependencies (sometimes called implicit coupling). The reverse is also true - it is possible to have dependencies without coupling. ALA makes use of this by simply making a constraint that all dependencies must be on abstractions. When you do that, every artefact (abstraction) in the program is zero-coupled with every other.

Doing this isn’t always easy because unfortunately there are many established architectural methods, patterns and styles that break this constraint. On the other hand, applying this constrain emerges some patterns that we will immediately recognise. DSLs and dependency injection are two examples. We will also emerge some less well known ones that are none-the-less not novel. There already exists an "abstract interactions" pattern, for example, which uses interfaces that are more abstract than the modules using it.

There are two situations that commonly cause coupling in conventional code:

  1. In the above example, imagine that def456 is just a source or destination for messages (in the same abstraction layer as abc123) such as a display. Then abc123 cannot be an abstraction because it cannot be reused without dragging def456 with it. If abc123 is an abstraction, it cannot know (or care) where the data comes from or goes to. To fix this, def456 must be passed into abc123 by something else above both of them. In other words, they must be composed. This can be passing in a function, passing in an object (dependency injection), or other mechanism such as function composition, monad composition, or the WireTo operator that we will use a lot in our ALA example projects.

    In conventional code, if abc123 calls directly def456, then the connection relationship between abc123 and def456 is hidden inside abc123. In ALA that relationship has to be an explicit line of code (inside another abstraction) in the layer above that composes the two instances. There, it will be cohesive with other similar relationships that work together in a collaborative way to make the application.

    Often these collected together wirings form a graph, making diagrams rather than code an even better way to describe the application.

  2. In conventional code, if xyz789 provides a part of the implementation of abc123, it will be more specific than abc123. Sometimes such a function or class is called a helper or submodule because xyz789 could only ever be used by abc123. In ALA xyz789 needs to be significantly more abstract than abc123 or it will be highly coupled to it. If xyz789 is put inside abc123 the complexity inside abc123 is still that of both of abc123 and xyz789 together.

    This is contrary to what we are taught. We are taught to "divide and conquer" or to separate out the responsibilities. If we do this arbitrarily, we will end up with specific pieces (such as UI and business logic) which are highly coupled with each other, and with the specific application. We need to work hard to separate only by finding abstractions - potentially reusable artefacts. Then we configure instances of those abstractions for each specific use by passing the application specific details into them.

In summary, ALA’s starting premise is a constraint. The constraint is that you can only use one type of dependency - a dependency on an abstraction that is significantly more abstract than the one using it. This results in zero coupling throughout the abstractions of the entire program.

3.2. Abstractions

3.2.1. Design-time encapsulation

Abstractions are the human brain’s version of encapsulation.

The maintainability quality attribute is often thought of in terms of ripple effects of change. I don’t think that is quite the right way to look at it. I have often had to make changes across a number of modules in poorly written code. The changes themselves just don’t take that long. The problem I see is the time you have to spend understanding enough of the system to know where to make a change, even if it ends up being just one line of code in one place. To make that small change with confidence that it wont break anything can take a long time understanding the collaboration between modules. You may have had to understand a lot of code to figure that out. You have to understand all the code that is potentially coupled to that one line of code, which is essentially the complexity.

Unlike encapsulation which works at compile-time, abstractions hide complexity at design-time. They give boundaries to how far you have to read code to understand it.

3.2.2. Abstractions and Instances

Software architecture should contain two concepts for its elements equivalent to abstractions and instances.

If you are going to have abstraction, it makes sense that you would have instances. An instance is nothing more than the use of an abstraction by referring to its name. If your abstraction is a pure function, then an instance is just using the function, or getting a reference to it.

If your abstraction is a class, and if that class contains data, then you need to instantiate the class so that each instance has its own data. Object oriented languages of course already have these two concepts as classes and objects.

Many discussions on software architecture seem to combine them into one term, such as modules or components. These terms may implicitly contain the separate concepts of abstractions and instances, or they may be intended to only one instance. Not having explicit terms, like class and object, will inevitably lead to confusion. In ALA the terms we use are abstractions and instances.

The problem is that when we become vague about the difference between abstraction and instance, we will then create dependencies between abstractions, such as to get or put data. If you create dependencies between peer abstractions, they are no longer abstractions. Instead you need to wire the instances. If we don’t have two separate and clear terms for abstractions and instances, we will end up with no abstractions. Many architectural styles appear to have this problem.

A common example of the problem is the UML, which already has the separate concepts of classes and objects. The UML class diagram encourages us to create associations between classes, destroying them as abstractions. The most important potential idea that OOP brought us was the idea of classes as reusable abstractions, and objects as their instances. It never happened in part because of the UML class diagram, and the very harmful habit of putting dependencies between abstractions instead of wiring instances.

The quality of an abstraction’s concept or idea is important. It is the existence of the concept that allows the brain to learn it and not have to know how it’s implemented each time it comes across it. It is the concept of the abstraction that blocks coupling. ALA sometimes requires effort over several days to conceive good abstractions, especially for the first application in a new domain.

3.3. Zero coupling and higher cohesion

3.3.1. Zero coupling

Larry Constantine, who coined the expression "loose coupling and high cohesion" wrote: "The key question is: How much of one module must be known in order to understand another module?". That is the essence of the most useful way to think about coupling in software.

ALA has mutual zero coupling between the code inside (or the code that implements) all abstractions. This is the case both horizontally between peers in the same layer, and vertically up or down the layers.

In software design we are only interested in design-time coupling. This means that to understand one piece of code, how much do we need to know about other pieces of code? It is about knowledge, which is about design-time. This is the coupling that matters. We will use the word coupling to refer to design-time coupling.

Here is a typical definition of Loose coupling from the internet: "Loose coupling refers to minimal dependencies between modules, accomplished through strict, narrow, stable interfaces at the boundaries."

This definition of loose coupling differs from ALAs in two respects.

Firstly, we are not minimizing dependencies. We are elliminating bad dependencies and maximizing good dependencies.

Secondly, it’s not just about using interfaces. A lot of design-time coupling is what I call collaboration coupling. An interfaces between two modules that have a fixed arrangement with each other may hide some details, but it doesn’t stop fundamental collaboration between the two modules. When modules have a fixed arrangement with each other, they are highly likely to be collaborating, and this collaboration will tend to increase during maintenance.

Larry Constantine’s key question about coupling (quoted above) is what is important. His reasoning came from identifying how to reduce complexity, the time taken to understand software, and reducing the incidence of bugs. Some definitions of coupling are in terms of the ripple effects of change. This is part of it, but even if a change ends up being made in one place, the time taken to understand all the collaborating code can be severe.

Wikipedia defines coupling as "the degree of interdependence between software modules". It doesn’t really distinguish between design-time, compile-time or run-time coupling, and the given formula for coupling seems to reflect compile-time. I prefer to think of coupling as a design-time property. It is about knowledge of internals of a module. The use of abstractions instead of modules allows us to achieve zero coupling.

Consider the principle of compositionality. As stated in Wikipedia, "In semantics, mathematical logic and related disciplines, the principle of compositionality is the principle that the meaning of a complex expression is determined by the meanings of its constituent expressions and the rules used to combine them." In ALA we use domain abstractions as the 'constituent expressions', and programming paradigms as the 'rules used to combine them'. We have the objective that all code conforms to the principle of compositionality. We can then define coupling as anything that compromises this principle.

To understand any one part of the code should involve understanding only that one part of the code, and the abstractions it uses.

Unfortunately there is a meme in the software engineering industry that there must be some coupling between 'modules'. The argument goes that if the system is to do anything it must have some coupling between its parts. We therefore hear of "loose coupling" as being the ideal. Using the definition of coupling given above, this is completely incorrect. Because of this meme, in conventional code we have developed a habit of using dependencies to implement communications. We are settling for design-time coupling to achieve run-time communication connections between different parts of a system. This is not necessary. Part of the problem is that the same word, coupling, is being used for both design-time coupling and connections or wiring used for communications.

In our A & B example above, the code inside B knows nothing of A. The code inside A, while it knows about the concept of the abstraction B, knows nothing about the code that is inside B. So it’s not like we don’t know how to do zero-coupling. ALA is basically a constraint to force us to always do zero coupling.

For example, in conventional code, if function Switch calls function Light, the code inside Switch is coupled with Light. If the light’s abstraction level is about the same as that of the Switch, then the abstraction of Switch is destroyed. When you reuse it you have to know the internal code brings in a Light. To understand the system (a Switch connected to a light), you have to go inside the Switch:

switch light bad

If instead, an abstraction, System, has code inside it like Light(Switch()), then Switch remains a good abstraction whose internal code is now only concerned with how a switch works. The code inside all three abstractions is now zero coupled. Understanding the system no longer requires looking inside Switch.

diagram collaboration A B C


diagram collaboration B C invis

A similar argument applies if Switch and Light are classes. In conventional code they will commonly have an association relationship. Even if Light is injected into Switch by a higher entity called System, Switch still knows the specific interface of a light (LightOn(), LightOff()). This interface is not abstract enough to prevent Switch knowing about Light, and Switch knowing about the System. If you instead have a class System that has code like new Switch().WireTo(new Light()) using a generic interface then all three abstractions are zero coupled.

ALA never uses coupling for connections or wiring between parts of a system. A larger system typically consists of many connections. These connection are typically cohesive, and belong in one place. In conventional code they tend to be distributed and buried inside the modules. A smell is that you are doing 'all files' searches to unravel them

3.3.2. Cohesion

"Collaboration becomes cohesion".

In ALA, collaboration between modules becomes cohesion inside a new abstraction. A call from one module to another becomes a line of code. Calloboration between a group of modules for a single purpose, such as a user story, becomes several cohesive lines of code.

Cohesion also increases in a different way. An abstraction is closely aligned with the single responsibility principle. We can think of abstraction as a single concept principle. Using abstractions increases the cohesion of the code that implements the abstraction.

ALA provides no structure for the internals of an abstraction because the code is cohesive. The internals of an abstraction could be described as a small ball of mud, which is why they should be small. There is no such thing as a sub-abstraction. Instead the code is composed of instances of abstractions from lower layers. So layers replace hierarchical encapsulation. Zero coupling and high cohesion limits ripple effects of change, whether in higher layers or lower ones. A ripple stops at an abstraction concept because of the inherent stability of the concept itself.

What does happen though is that abstractions can be improved as abstractions. Often you can generalize an abstraction to make it more reusable by adding a configuration that has a default behaviour, so it doesn’t affect other uses of the abstraction (convention over configuration).

In our experience, the most common type of change that still affects multiple abstractions are changes to conventions. Conventions in the ways abstractions are commented, and their code laid out are effectively abstractions in themselves that live in the bottom layer. So when they change, it makes sense that all abstractions that depend on them change. These conventions will mature in time. Besides, while these types of changes may require a lot of editing, they don’t require simultaneous understanding of multiple modules, which is where the real problem with coupling lies.

3.4. Good versus bad dependencies

For the purposes of this book, a dependency is when some code symbolically refers to a class, interface or function or other artefact in a separate piece of code. This covers everything from dependencies on classes, interfaces, modules or components, to dependencies on libraries or packages.

A dependency can be on something inside a class or interface, usually a method or property. Even if using an object reference, there is still a dependency if there is a reference to something named inside the class or interface.

We can distinguish between good and bad dependencies. Good dependencies are design-time dependencies. These are dependencies on the knowledge you must have to even understand a given piece of code. I will often refer to this type as a "knowledge dependency" or "use of an abstraction". It is also sometimes called "semantic coupling". This type of dependency effectively adds to the language you use to write code. Here is a diagram shwoing a good dependency.

GoodDependency1

A bad dependency is one that is there to facilitate run-time communications between two modules or components. Here is a diagram representation.

BadDependency1

Another type of bad dependency is when a module uses a submodule that is a specific part of it. An example is a 'helper' class. The submodule is often thought of as being logically contained inside its parent module.

BadDependency2

Dependencies on more abstract abstractions are good.

Dependencies for communciations between peers are bad, as are dependencies on submodules.

A simple example of a communication dependency is a module that calculates the average then calls a display module to display the result. To understand the code that calculates the average requires no knowledge about displays, nor even where the result will be sent. So it is a bad dependency.

BadDependency3

The intention of the fixed arrangment between Average and Display was to measure rainfall. To do that, an instance of an Average module needs to be connected to an instance of Display module at run-time, but you don’t need a dependency to achieve that.

GoodDependency2

A simple example of knowledge dependencies occurs in an abstraction that measures rainfall. To understand the rainfall code, you must understand the concepts of average and of display. The rainfall abstraction can use the abstractions for averaging and displaying.

We typically find both good and bad dependencies in conventional code. A typical modular program is full of bad dependencies. But whether a knowledge dependency or a communication dependency, they all look syntactically the same - a function call or a 'new' keyword, etc. We are not gnerally taught how to distinguish between them. We lump them together when we talk about dependency management, loose coupling, layering, fan-in, fan-out, circular dependencies or dependency inversion. Dependency graphing tools cannot distinguish between them because identiying good dependencies would require understanding abstraction levels.

Good and bad dependencies are not just good and bad. They are really good and really bad.

A knowledge dependency is good because it’s only a dependency on an abstract concept, on something stable and learnable. We want more of them, because then we are reusing our abstractions. Then the more dependencies you have on such an abstraction, the more abstract it is.

So it’s doubly important that we are able to tell good dependencies from bad.

It’s entirely possible to build a system using only good dependencies.

In ALA we eliminate ALL bad dependencies.

Bad dependencies destroy abstractions. They cause explicit and implicit coupling. They obscure the structure of the application by distributing that structure throughout its modules.

When we remove all bad dependencies, they become normal lines of code, cohesive with one another, inside a more specific abstraction in a higher layer. Each line of code composes instances of the abstractions which would otherwise have had the bad dependency.

Consider the diagram below. It’s the conventional modular way to write a rainfall meter. An ADC reading is averaged, converted, accumulated, and displayed. The middle three modules have bad dependencies which they use to make function calls to pull data in and push data out.

dependency diagram

There are four bad dependencies, two from Conv and one each from Avg and Accu.

Now consider this diagram, which uses only good dependencies.

dependency diagram 1

There are five good dependencies from the Application to the five Abstractions. There are also five good dependencies from the Abstraction on Dataflow. We never draw arrows on a diagram for good dependencies. Instead we just refer to the abstractions by name.

Connections between the instances of teh abstractions are completely described inside the Application. There it is cohesive code that knows about how to build a rain meter.

The code in the application abstraction could look something like this if using functions (although you would likely use some temporary variables in practice):

    Disp(Accu(Conv(Avg(ADC()))));

It might look something like this if using classes:

    new ADC().WireIn(new Avg()).WireIn(new Conv()).WireIn(new Accu()).WireIn(new Disp());

How this code is done is not what’s important. How syntactically succinct this code is is not important. What’s important is where it is. We want the code that cohesively and fully expresses a rain meter to be in one place.

The lower-case letters used in the top layer of the diagram represent instances of the respective abstractions. (In UML they would be underlined.)

You never draw arrows for knowledge dependencies - you only ever refer to abstractions by name. (Just as you would never draw an arrow to a box representing the squareroot function - you would just use Sqrt by its name.)

In common programming languages, the communication dependencies in the first diagram and the knowledge dependencies in the second diagram could both be syntactically written in the same form, either new A() or just a function call, A(). The only difference is in where those new keywords or function calls are.

The application abstraction can move the data between the instances of ADC, Avg, etc itself, as we did in the first code example, however strictly speaking that pollutes it with details of how to move data that actually belongs in the programming paradigms layer. We much prefer the application code just does the composing - just specifies who connects to whom, but is not involved with how data actually flows. That’s why in most of the examples, we compose with classes that have ports rather than functions. In the second code example, the dataflow programming paradigm would be implemented with an execution model that knows how to actually move data. The application only knows that it is composing a flow of data.

The interface used to connect the instances is called Dataflow. It’s important that this interface is abstract. It is two layers down. It is not an interface specific to any one of the domain abstractions, ADC, Avg, etc. This is the abstract interactions pattern. Other domain abstractions can either implement it or accept it, or both.

3.4.1. Comparison of good versus bad dependencies.

Table 1. Comparison of two approaches
Run-time dependencies version Knowledge dependencies version

Knowledge about the specific application is spread through all modules.

Knowledge about the specific application is only in one place. The abstractions know nothing of each other or the specific application.

The class or function names A, B, D and E will relate to what they do (which is fine). For example, they may be the specific hardware chips used in the case of drivers. The calling module must know these names, creating a fixed arrangement between the modules. The modules are only loosely coupled.

No abstractions refer to the names of peer abstractions. There is no fixed arrangement between abstractions. The abstractions are zero coupled. The code that knows that a particular hardware chip is used in this application is where it belongs, in the application abstraction.

Since there is a fixed arrangement, responsibilities can be blurred. For example, it may be unclear whether to add something to B or C.

With no relations between abstractions, responsibilities are clear. Something to be added clearly belongs in one or other of the abstractions, or in a new abstraction that may be wired in between the two.

The fixed dependency from C to B will encourage implicit coupling. B can make assumptions about details inside B resulting in collaborative coupling.

C cannot make any assumptions about some details of B. It cannot have collaborative coupling with B

Although there is no dependency, for example from B to C, the fixed arrangement is likely, over time, to make B implicitly collaborate with C (do what C requires), resulting in collaborative coupling.

No implicit coupling can develop over time because there is no fixed relationship between them. B cannot collaborate with C (do what C specifically requires).

The arrangement between A, B, C, D and E is not obvious in the code. It is buried inside of B, C and D. All must be read to find the application’s dataflow structure

The arrangement between instances of A, B, C, D and E is explicitly coded in one place. The dataflow between them is cohesive information that belongs in one place.

Only A and E can potentially be abstractions.

All of A, B, C, D and E are abstractions.

Arbitrarily, only the two ends of the dataflow chain can be reused independently .

All of A, B, C, D and E are independently reusable.

Difficult to insert another module between, say, B and C.

Easy to insert a new instance of some operator between B and C, etc.

If the observer pattern is used (in the mistaken belief that it reduces the coupling), it only mirrors the same problems. For example B would now have a dependency on C when it registers. But because it adds indirection, the observer pattern makes the program even harder to understand.

If the observer pattern is used (as the means to implement the wiring between the instances), the receivers do not do the registering, the application does (not strictly the observer pattern). The abstractions themselves don’t get more difficult to understand because, being abstractions, they only have knowledge as far as their interfaces anyway. The application does not get harder to understand either, because the arrangement of the instances is still explicit and in one place.

If dependency injection is used with automatic wiring, the arrangement is still somewhat fixed, but is now even more obscure. All classes can still be collaborating with one another. A smell that this is happening is that over time the interfaces, IA, IB, ID and IE change as the requirements of the system change.

If dependency injection is used, the application does the wiring explicitly. It is the only place that should know who will talk to whom at run-time for this specific application. There are no specific interfaces between pairs of modules to change over time, because they all just use a stable abstract interface.

Each module has its own interface. But they are all doing essentially the same thing, getting data.

Uses a single more abstract interface called Dataflow.

The arrangement between the modules cannot easily be changed, both because the wiring code is buried inside the modules and because the interfaces are essentially specific to pairs of modules.

The composition can easily be changed. Instances of the abstractions can be re-wired in any combination. New abstraction instances can be inserted.

There is no diagram of the arrangement between A, B, C, D, E, or if there is, it is likely a high level overview, lacking in detail, and a second source of truth that gets out of date.

There is a diagram that shows the arrangement of the instances of A, B, C, D and E. It is the one source of truth. It includes all details about the specific application. It is executable.

During code creation, run-time dependencies are easily introduced, and never seem too terrible at the time as they get the immediate job done. But when they accumulate to hundreds or even thousands of them, as they do in most typical applications, that’s when the system, as described on the left side of the table, just appears as a monolithic big ball of mud.

3.4.2. Free lunch?

When you are comparing the left and right sides of the table above, you may be wondering, where did the free lunch come from? Where did the runtime dependencies go? Is this some kind of magic? Or how can the program work without them? Or haven’t I just moved them somewhere else? No there are no tricks. The answer is that we have been taught to do programming in a very bad way. The knowledge that ADC will talk to Avg, etc at run-time is there, but it is now contained within an abstraction, not a dependency between modules. If you really want to find a disadvantage, then it is the need for the abstractions. It only works as well as the quality of the abstractions. Effectively we have traded the need for dependency management, and all the complexity that bad dependencies cause, with the need to create good abstractions. Creating good abstractions is a skill that does take time to get used to.

Just to recap the only dependencies we have used are good design-time or knowledge dependencies:

  1. The application should and must 'know' at design-time what domain abstractions it needs to compose to make a rain meter application.

  2. The domain abstractions should and must know at design-time what programming paradigm they need - the abstract interfaces to use for their input and output ports.

3.4.3. Stable dependencies principle

A dependency on an abstractions is a dependency on the concept or idea of that abstraction. A concept or idea is generally stable. So dependencies are toward the more stable.

Even if the implementation details inside an abstraction are complicated or change, the abstraction concept itself be stable. The application example above is really just depending on the idea of an ADC or the idea of a Display. If the details inside change it doesn’t matter. For example, if the ADC silicon is changed, the ADC abstraction implementation can also change. But the application is still just using an ADC as it’s means to get input.

ALA therefore naturally conforms with the Stable Dependencies Principle (depend in the direction of stability). The SAP is mostly used in relation to packages, but ALA does not use hierarchical encapsulations. Here we are applying it at the level of the abstractions themselves.

3.4.4. Dependency fan-in and fan-out

One of the guidelines sometimes used for dependencies in conventional code is that a class that has high fan-in should not have high fan-out (also called afferent and efferent coupling). Another is that modules higher in the layers should have low fan-in and those lower in the hierarchy have low fan-out.

The argument goes that a class with high fan-in should have high stability but one with high fan-out would have low stability (presumably because dependencies are thought to be things that cause changes to propagate).

In ALA, dependencies are on abstractions. Furthermore the abstractions are increasingly abstract as you go down the layers, and therefore increasingly stable. Therefore the conventional fan-in and fan-out recommendations are reversed. In ALA, it is perfectly fine, in fact really good to have both high fan-in and high fan-out. It simply means that the abstractions are useful and are getting reused.

If we are talking about dependencies in a conventional modular system that are used for communication between modules in the system, of course ALA says we want zero fan-in and zero fan-out, because such dependencies are illegal anyway.

In chapter four we will also talk about fan-in and fan-out. Note that the fan-in and fan-out discussed in chapter four is different. In this chapter fan-in and fan-out is talking about dependencies on abstractions between layers. In chapter four we are talking about fan-in and fan-out in the wiring.

3.4.5. Circular dependencies

Of course in ALA, with only knowledge dependencies present in the system, and the dependencies needing to go toward more abstract abstractions, you obviously cannot have circular knowledge dependencies. Nor would that even make sense. (Recursion appears to require circular knowledge dependencies but actually doesn’t. We will visit that in the last chapter.)

Since there are no run-time dependencies, the issue of circular dependencies with them does not arise at all. What might have been circular dependencies in conventional code becomes circular wiring of instances of abstractions inside a user story abstraction in the application layer. Such circular wiring is quite valid, and very common. The potential issues with the execution models are discussed in chapter four.

In conventional software design, run-time communication channels between modules are frequently implemented with dependencies. Then we realize these dependencies are a problem and so we add a rule that we don’t like circular dependencies. This is an attempt to mitigate the problem by forcing the modules to have a sort of arbitrary layered structure. That structure does not actually exist in the nature of peer modules themselves. (Many modules will actually have a similar level of abstraction, for example views, business logic and data.) The forced arbitrary layering structure becomes its own nuisance.

So then what happens is circular dependencies are most often avoided by using pushing in one direction and pulling in the other. (Pushing means a function or method call with a parameter, pulling means a function or method call returning a value). This is sometimes actually convenient, and other times a real nuisance. Whether we push or pull should be able to depend on performance or other considerations (which end wants to initiate the communications, which depends on when the source changes, or when we want to receive new a the data, or how often the source changes, or on latency, etc), not on an arbitrary layering of modules.

So, when we do want to push or pull in the reverse direction of the allowed dependency, we end up creating an indirection, such as a callback, virtual function call, or observer pattern (publish-subscribe). This indirection further obscures the already obscure communication flows through the system.

ALA simply eliminates all this nonsense. In ALA, communication flows:

  • are explicit

  • can be in both directions

  • each set of cohesive flows are contained in one place

  • allowed to be push, pull, or asynchronous on a port by port basis

  • don’t use dependencies at all

  • use indirection in the correct way, which is that when you are reading code inside an abstraction, you don’t know, and shouldn’t know, where your inputs and outputs are wired to.

That concludes our discussion on why the ALA structure works from the point of view of good and bad dependencies.

3.4.6. Knowledge dependencies are on all layers below

Sometimes layers are used incorrectly as partitions or really just modules. We would be better off to just tip all such layering models on their side. Because of this mistake, there is a meme that we should only have dependencies on the immediate layer below. For ALA layers this is incorrect.

When we write our programs using only knowledge dependencies, the knowledge needed to understand a piece of code can come from all the layers below.

For example, to understand this application layer code:

    new ADC().WireIn(new Avg()).WireIn(new Conv()).WireIn(new Accu()).WireIn(new Disp());

You need to know all of these things from lower layers:

  1. Understand what the domain abstractions, ADC, Avg, Disp, etc do.

  2. Understand the dataflow programming paradigm. When you compose these particular domain abstractions, you are composing a flow of data from left to right.

  3. Understand that the WireTo operator, which comes from the Libraries layer, is what you use to do composition.

  4. Understand your general purpose programming language, which sits below the Libraries layer.

  5. Understand ALA which is an abstraction that sits below the programming language layer.

All of these knowledge dependencies should be explicit. This means that the application folder should contain a readme file explaining all these knowledge dependencies, and link to information about them.

It’s nt necessarily the case that all lower layer knowledge is needed to understand something. The application is itself an abstraction. There can be many instances of it being used by different users. These users don’t need to understand all the abstractions in all the layers, only the application abstraction by itself.

3.5. Executable expression of requirements

We have previously discussed this aspect of ALA in terms of structure. It is the top layer. And we have used this aspect as the starting point in the method to develop the example projects. But why does the succinct description of requirements in that top layer work?

In conventional software development, we typically break a user story (or feature or functional requirement) up into different implementation responsibilities. For example, layers like GUI, business logic and database, or a pattern such as MVC (Model, View, Controller). But a user story or feature actually starts out as cohesive knowledge n the requirements. And its not a huge amount of cohesive knowledge, so it doesn’t need breaking up. Cohesive knowledge, knowledge that is by its nature highly coupled within itself should be kept together. All we need to do to keep it together is find a way to describe it so that it is executable. Don’t try to do any implementation, just get it described in a concise and complete form. If you can do that, the chances are you will be able to find a way to make it execute.

In ALA we want to find a way to express the user story with about the same level of expressiveness as when the user story was explained in English by the product owner. The language he used would have contained domain specific terms to enable him to explain it concisely. The same thing ought to be possible in the code. Anything that does not come directly from the requirements and starts to look like implementation detail is separated out. It comes out into abstractions. These abstractions typically contain knowledge of how user stories in general are implemented - how things can be displayed, how things can be saved, how data can be processed.

Many times, abstractions that know how to implement useful things for expressing user stories are not only reusable for user stories, but can be reusable for other applications. In other words, they are domain level abstractions. A typical user story might be composed of several of them, some to implement the user story’s UI, some to implement the user story’s business, and some to implement the user story’s saving of data. A user story instantiates the abstractions, configures them with the specific knowledge from the requirement, and then wires them together.

Most maintenance is probably changing, adding or fixing user stories or features. When those features are described entirely in one place instead of distributed through a lot of modules, you have a direct understanding of how the user story is represented by code, and therefore of how to change it or fix it.

Of course application code makes heavy use, in fact is entirely composed of, instances of domain abstractions. When fixing a bug, it quickly becomes clear if the application code itself doesn’t represent the requirements as intended, or one of the abstractions is not doing its job properly. Again the maintenance is easy.

3.5.1. The meaning of composition

Expressing user stories as a composition of domain abstractions, as discussed in the previous section, is all well and good, but it doesn’t work without defining what composion means. That’s where programming paradigm abstractions come in.

For example, many applications have displayed values or outputs that need to be updated 'live'. In conventional code, we often write imperative code to implement this live behaviour. The code repeatedly gets data from its source(s), does some manipulation on that data, and updates the output. This kind of code uses lots of bad dependencies. we really should have a programming paradigm for it. In ALA you think of it as a dataflow. Wiring together domain abstractions represents data flowing automatically. This programming paradigm is not new, of course, it appears in Unix’s pipes and filters, functional programming, binding GUI display elements to a data source, LINQ, Reactive Extensions, Labview, function blocks, ladder logic, and countless other programming systems. Dataflows are often used on distributed systems because implementation over literal wires is naturally a dataflow. But the paradigm is just as applicable inside monolithic systems. What ALA does is make it easy and natural to implement dataflow yourself every time it is the best way to express requirements. We should never be writing imperative code when dataflow is what best expresses what we want to do. ALA makes it easy wire a linear chain of data manipulations. But it also makes it easy to wire up an arbitrary network of dataflows.

The same idea applies to the event-driven programming paradigm. It is common these days for GUI elements such as buttons, menu items, etc to have event-driven output ports. But then we often just wire them to imperative methods with a dependency. In ALA you create input ports as well. For example all popup window abstractions such as file browsers, wizards, settings, navigable pages, etc have input ports. The main window has a close input port. Long running tasks that need to be told when to start have an input port. Then you can use the event-driven programming paradigm for composing instances of these types of domain abstractions.

Another programming paradigm is building the UI. Building the UI by composing abstractions is common using conventional libraries these days. The meaning of composition is containing one UI element inside another. The composed UI structure is a tree. For example XAML does this using XML. I do not like the use of XML for this. What ALA brings is doing all composition in a consistent way. Wiring for the UI is done in the same way as the composition wiring for dataflow, or event-driven, or any other programming paradigm you care to use. That way a user story is fully and cohesively expressed inside its own abstraction.

3.5.2. Requirements are what’s left when you factor out all implementation details

This is another way of thinking that comes to the same solution. As we know from the previous section, ALA requires you build your entire application factoring out all pieces of computing work into domain abstractions and programming paradigm abstractions. So what does the application that’s left in the top layer look like? Well if everything abstract has been factored out, what remains must be details specific only to this application. Essentially these details equate with the requirements.

The application code becomes a formal re-expression of the requirements. There will be some information there that wasn’t explicitly stated in the requirements, but they were requirements all the same. For example, it may not have been stated in the requirements that a number displayed on the UI should not change its value too frequently - it should be slow enough for a human to read successive values. A consequence of that requirement is that it should not contain noise that has a frequency higher than the display update rate. So the application will end up with an instance of a re-sampler abstraction and an instance of a filter abstraction wired into its dataflow before the display. The application will specify the re-sampling rate, and the filter bandwidth.

3.5.3. DSL - Domain Specific Languages

ALA’s succinct expression of requirements discussed above is obviously a form of DSL (Domain Specific Language). Under the broader definition of a DSL, The domain abstractions and programming paradigms layers are a DSL. But ALA is not just a DSL. ALA is fundamentally about organising all code into small abstractions that are in layers that are increasing abstract. This constrains the organisation of code much more than simply implementing a DSL.

ALA does not pursue the idea of an external DSL (a new syntax), nor even the syntactic elegance of DSLs. It doesn’t try to move application development away from the developer to a requirements team as some DSLs can do. For example, you don’t get a new language such as XAML to express UI structure. In fact, expressing the UI structure in ALA moves away from XML back to code. If moving away from code, ALA uses diagrams because they are more flexible and much more readable than XML.

Seen as a DSL, in ALA you wire together plain old objects or functions while conforming to a grammar. The grammar comes from the 3rd layer programming paradigms and from which classes use which programming paradigm for ports. This grammar defines the rules for their composition.

3.6. Diagrams vs text

The fundamental rules of ALA don’t prescribe the use of diagrams. But diagrams often emerge. So why do we often use a diagram instead of text in the application (top) layer of an ALA application?

It’s because in any non-trivial program, there is structure inherent in the requirements that forms a graph. If you have UI that graph is a tree - still representable with indented text. But the UI must have connections. (These particular connections are often called bindings.) They need connections with data. They need connections with event handlers. These connections must be done symbolically if using text. The connections go further. There are connections to business logic and to some form of persistent data model, and from there to real databases or files. There are arbitrary connections for navigating around different pats of the UI. If text, most of these connections must be done symbolically. On the way, they may need to connect arbitrarily with things that process, reduce, or combine. There may be states involved, with arbitrary transitions needed between those states. There may be activities that have to happen in a prescribed time sequence, which by itself is representable as a linear instructions in text. But there are often loops or alternative routes through the sequence, which is representable as indented text. But then there is always some connection between the activities and some data or the outside world. If text, these connections must generally be done symbolically.

All these connections are inherent in the requirements. Like or not, they form a graph. And this graph structure is somewhere in your code.

As we said, in text from, this graph needs to use at least some symbolic connections. That is, we can represent some of the graph with indenting and judicious use of anonymous functions or classes, but in general we will need to represent many of the connections by using names of variables, functions or objects.

This is bad enough. In fact this is already really, really bad compared with how the electronics guys do things.

But it gets much worse. In most conventional code, we take all these symbolic connections and distribute them evenly through the files/modules/classes/functions. Now the graph is totally obfuscated. The graph is highly cohesive. Why do we make it harder for ourselves by breaking it up?

But it gets much worse. Graphs have circles in them. There is nothing wrong with that, it’s inherent in the connections in the requirements. But circles are at odds with dependency rules. So now what we do is break the cyclic dependencies using principles like dependency inversion or observer pattern. The connections don’t go away. We just further obfuscated them. These connections are now done at run-time by code written somewhere else. This is the so called indirection problem.

What a mess we have got into!

ALA tells us how to fix this entire mess. It’s really quite simple. ALA breaks up your application by factoring out abstractions. When you have done that to the maximum extent, what’s left behind is nothing but the specifics of the requirements, including that (highly coherent) graph.

Now you can choose to go ahead and represent that graph in text in one place, using many symbolic connections, and you would already be way, way better off than how we write conventional code. But even better is to do what the electronics guys do, and just build the tools to handle the graphs as diagrams properly.

3.6.1. Diagrams and text are not equivalent

Diagrams and text are sometimes thought of as equivalent, as if they have a duality like waves and particles in physics. It is said to be a matter of personal preference which you use, and since graphical tools are hard to produce, why not use text? I do not agree with this. From the point of view of how our brain’s work best, they are very different, and each is powerful at its own job.

Consider an electronics engineer who uses a schematic diagram. Ask him to design a circuit using text and he will just laugh at you. Electronics naturally has a network structure that is best viewed and reasoned about as a diagram. If you turn a diagram into a textual list of nodes and connections, the brain can no longer work with it directly. It is constantly interrupted to search for symbolic references when it should be free to just reason about the design.

Most software naturally has an arbitrary network structure. Think about whenever you are working with legacy code - how often to you need to do "all files searches" or "find all references". And even those are foiled by indirections. Try designing or reasoning about a non-trivial state machine without using a diagram.

Text can readily be used to compose elements in a linear chain or sequence. It is excellent for telling stories. White space is the normal connector between the elements. Sometimes periods or other symbols are used instead. Text can also handle shallow tree structures, simply by using indenting. Compilers may use brackets, usually () or {}. Interestingly, the brackets work for the compiler, but not for the brain. The brain doesn’t see them, it just sees the indenting. So I personally don’t agree that Python’s significant indenting is a mistake as many do.

When the tree gets deep, the indenting is too deep for our brains to follow. So text is only suitable for linear structures and shallow trees. Structured programming and XAML are examples of tree structured code represented successfully in text.

Text becomes troublesome when there are arbitrary connections across the structure forming a graph. It must be done with matching names, labels or identifiers. Most imperative programs are actually not a tree structure because of the variables. They must be done with labels. Local variables in a small scope are not a problem. It only requires an editor that highlights all of them. For large scopes we end up spending too much time finding and trying to remember the connections, resorting to many all-files searches. It is a cumbersome way to try to reason about what is usually a reasonably simple structure when viewed as a diagram.

(When we talk about labels, we are talking about labels that are used for connecting two or more points. These labels are not abstractions. References to the names of abstractions are absolutely fine, and we don’t draw lines for them even if we are using a diagram. We always just use abstractions by their name.)

When we need to compose instances of abstractions in an arbitrary network structure, our brains work much better using a diagram. The brain can readily see and follow the lines between the instances of the abstractions. Unlike with text labels, the lines are anonymous, as they should be. When symbolic connections are used, the symbols themselves need an encapsulation scope. Lines don’t need encapsulation. They quietly connect two point without any other code seeing a symbol.

Generally lines connect only two points or ports, but sometimes may connect three or more. To understand all places connected by a symbolic connection requires an all files search. To understand all places connected by lines, the brain just follows the lines instead. The spacial positioning of elements is also something the brain readily remembers. So, diagrams can qualitatively do things that text simply cannot.

If lines connect a high number of ports, it starts to smell as if a new abstraction may be waiting to be discovered.

ALA does not require a diagram per se. It only requires abstraction layering, and it’s quite possible for a user story to just consist of a linear sequence of abstracted operations. For example, a sequence of movements by a robot or a "Pipes and Filters" sequence of operations on data. However, ALA is a polyglot programming paradigm because user stories will generally combine multiple programming paradigms: UI, event-flows, dataflows, state machines, data schemas, etc. These aspects of a user story tend to be naturally interrelated (inherent in the requirements), which is what causes the resulting relationships among its instances of abstractions to be a graph. Diagrams, then, embrace the bringing together of all these different interrelationships of a user story in one cohesive place.

3.6.2. No XML as code

If dependency injection is used to implement the wiring, I prefer not to use XML to specify the application. Firstly XML is not very readable. Secondly it only handles tree structures well, not networks, and it becomes more unreadable if the tree is deep. If you must use text for specifying wiring, use normal code. Represent the diagram as a tree as much as possible, and use indenting to represnet that tree. Any nodes that need cross connections should be saved in local variables. The cross connections can then be wired using the variables. You will see this done in many of the examples.

You are still better off with this code in one place than having it distributed inside your modules. But if a graph structure is inherent in the requirements, there is really no substitute for the readability of diagrams. In time there will be good tool support for ALA applications.

3.6.3. Diagramming tools

The ALA design process (which is describing your requirements and inventing the needed abstractions as you go) is an intense intellectual activity, especially the first time in a new domain. As well as expressing your user stories, you are inventing abstractions. You are inventing a set of domain abstractions and programming paradigms that will allow you to express all user stories with a finite number of them. It requires all your focus. I have found that hand drawing the diagram on paper is not good. The diagram quickly gets into a mess which requires redrawing, and that interrupts your design flow. I have found that a diagramming tool that constantly needs you to control the layout, such as Visio, is also not good.

So until there is a better tool, I have been using Xmind because as a mind-mapping tool, it is designed to not get in your way as you are creating. It lays itself out as a tree structure, and then allows cross connections on the tree to be added using a key short-cut at the source and a mouse click at the destination node. It has its limitations, however I use some simple conventions to get around these. For example, I use '<' and '>' to represent input and output ports.

Furthermore, the tree structure allows easy hand translation of the diagram into indented, fluent style code.

While Xmind allows you to be creative in the beginning (I couldn’t imagine doing without it), it is far from ideal once the abstractions have matured, and you are just churning out user stories.

We use a simple tool that takes Xmind files and generates the code automatically.

And even more recently, we have in progress a purpose built graphical IDE for ALA. But it is not complete.

See the end of this chapter for an example project using Xmind.

Thoughts on the essentials of a diagramming tool.

It would have the low driving overhead of a mind mapping tool. As with a mind-mapping tool, you control the logical layout, and the tool does the actual spacial positioning. It would primarily use keypresses, but allow mouse clicks where it makes sense, for example, to specify the destination of a 'cross connection'. The tool would route the cross conenction for you.

A tree topology can be done with simple key presses. The tree would capture the primary relationships between instances, on their main ports.

You can make mutiple trees for different user stories that are disconnected logically, but for the purpose of automatic layout, are connected to the main tree (just an invisible line).

Abstractions are defined in a separate panel as stand-alone boxes with ports. Once a new abstraction is  defined, it can be instantiated in the diagram by its abstraction name with auto completion. Boxes represent these instances of abstractions with the ports still lablled around their boundary.

The abstractions are fully inegrated with the classes in the code. This is in both directions. So for any existing classes, the IDE shows them with their port, and fully supports the entry of constructor arguments and properties.

In the other direction, if you create a new abstraction in the tool. You can specify its ports and their types and names. You can specify the constructor arguments and properties and their default values. It will create/modify a template for that class.cs.

The tool's purpose is to aid creativity in the ALA process of representing a user story, inventing new abstractions as you go. Of course the tool would also automatically generate the wiring code.

In my experience, a low overhead drawing tool is essential during the iteration zero design phase and during subsequent maintenance.

3.7. Composition, not decomposition

In this perspective, we look at ALA as the antithesis of the prevalent decomposition methodolgy.

The conventional technique for tackling system complexity is "divide and conquer".

Consider this phrase, which has been used as the definition of software architecture:

"decomposition of a system into elements and their relations".

Notice the word 'their', which I have italicised to emphasis that the relations are inferred to be between the decomposed elements. It suggests that the decomposed elements know something about each other, that they collaborate to create the whole. In ALA we think about building the system in a completely different way. Here is how to reword the meme for ALA:

"composition of a system using instances of abstractions".

This seemingly subtle shift in thinking leads to a qualitative difference in the resulting structure.

First let’s understand what we mean by composition through a few examples:

  • When we compose musical notes, we create a tune. The structure is linear. The execution is sequential like activity flow below.

  • When we write code in a general purpose programming language, we are composing statements. Statements are low level (fine grained) elements and only support a single programming paradigm, which we could describe as 'imperative', but by composing enough instances of them we can create a program. The structure is a linear or a tree.

  • In functional programming, we are composing with functions, so the elements are higher level things that you create. But the programming paradigm is still imperative (unless you use monads). The structure is either linear or a tree.

  • When programming with monads, we are composing functions. The programming paradigm has changed from imperative to dataflow. The structure is usually linear, but sometimes it is a reverse tree (two dataflows can be combined). (ALA is compared with monads in detail later in this chapter.)

  • When programming using the UML class diagram, we are composing with classes directly (not objects). The programming paradigms are associations and inheritance.

  • When programming using the UML activity diagram, we are composing activities to be done in a set order. The structure is a graph, because you can branch, recombine and loop back arbitrarily. Activity diagrams are not imperative (like the old style flow diagrams). The CPU is not necessarily dedicated to each activity being done. Activities may take an arbitrarily long time without the system blocking.

  • When programming with XAML, we are composing UI elements. The programming paradigm is UI layout (what goes inside what and in what order). The structure is a tree.

Let’s list the different properties present in these types of composition:

  • Low-level or high-level - Sometimes we are composing fine-grained general elements and we need a lot of them. Sometimes we are composing 'higher level' more specific elements, and we need relatively few of them.

    Note that sometimes people think of these higher level elements as more abstract. They are actually less abstract. For example, a class that handles complex numbers is less abstract than the fundamental float type. Complex numbers are a more specific case because its only useful when you need complex numbers in your solution. But when you do need complex numbers, then they are obviously more expressive than using pairs of floats everywhere. This means that you need to compose less abstractions to build your solution.

  • There is only one meaning of a composition relationship in each case. It can be one of imperative, dataflow, UI layout etc.

  • Linear/Tree/Network: The structure built by the composition relationships can be linear, a tree structure or a general graph or network.

  • Syntax: The syntax for the composition of two joined elements can be using spaces, dots, or lines on a diagram. We can use various types of bracketing or indenting for the text form of tree structures.

In ALA, we are setting up to use composition to create user stories or features. We want the composition to have the following properties:

  • Composing more course grained expressive elements by letting them be specialized to your domain.

  • Allow use of many programming paradigms (meaning of composition)

  • Allow linear, tree or graph structures.

  • Allow the programmer to add new programming paradigms with new meaning if that’s the best way to express typical requirements.

  • Uses the same syntax for all the different composition relationships.

ALA can therefore be thought of as a 'generalised compose from abstractions' methodology.

3.8. No Data coupling

The term data coupling here doesn’t mean that one module communicates data with another. It means that the two modules agree on the meaning of that data.

The actual communication of data at run-time is not a problem. The sharing of knowledge on how to interpret that data is.

In conventional programming, data coupling is considered unavoidable. There is a misconception meme that two modules have to share the knowledge of the meaning of data if they are to communicate at run-time. Even if you have an understanding of ALA, you will still be trapped by this misconception unless you know about it. This will cause you to write modules in the conventional way and they will have coupling.

The misconception is especially rife if the two modules run in different locations. It seems a self-evident truth that the two modules must share some kind of language if they are to communicate, just as people do.

Let’s use an example. There is a temperature sensor on a Mars rover. The temperature is to be displayed at a ground station on Earth.

In conventional programming, to implement this user story, one module resides in the Mars rover and one module resides in the ground station. These two modules must agree on the meaning of data. For example, it is an integer number of tenths of degrees C (Celsius).

Obviously a lot of other system parts are involved in transporting the data from the sensor module to the display module. These are referred to as middleware. It is common to containerise the data so that none of the middleware needs to know its meaning. But the two end points at least seemingly must have shared knowledge.

In ALA, the meaning of the communication is completely contained inside another abstraction. That abstraction is the only one that knows about the user story, so it’s the only one that needs to know the meaning of the data going from Mars to Earth.

Here is the user story implementation.

class AmbientTempertureUserStory {
    new TemperatureSensor()
        .WireIn(new OffsetAndScale(4.3, 712))
        .WireIn(new Display("#.#"))
}

The meaning of the temperature data does not need to be known outside of this small abstraction. It does not need to be known by the sensor itself, or the display, or anything in-between. The meaning only needs to be known by the engineer who wants the sensor on the rover and wants to see what it says on the display, and so wites the above code.

Now if he were to change the units of temperature were to change, only this user story abstraction would change. Just change the OffsetAndScale configuration, and change the way the display is formatted.

It doesn’t even matter if software needs to interpret the data. For example, let’s add an alarm that goes off at 50 C:

    new TempertureSensor()  // unit is celcius
        .WireIn(new OffsetAndScale(4.3, 712))
        .WireTo(new Display("#.#"))
        .WireTo(new Alarm(500));

The interpretation of the data is still contained inside the user story abstraction. Everything about that temperature is here as cohesive code.

3.8.1. deployment

The user story code spans physical locations. So how do those instances of abstractions get deployed?

Inside the user story abstraction, we can annotate the three instances with their physical locations. An abstraction that knows about the concept of physical view would already have been configured to know about the three physical locations. The physical view engine takes care of deploying the instances of abstractions for the user story to the correct locations, configuring them, and it takes care of actually connecting both ends through middleware. It also knows how to take care of version compatibility, and updating versions at different times at different locations.

3.8.2. modules written by different parties

What happens if one end of the user story is written by a team that has no communication with the team who does the user story. They just provide an API. In this case the team responsible for the user story itself will write an adapter for the API that also knows about the common programming paradigm abstraction. We can still have a separate abstraction to repesent the whole user story, and keep the data coupling contained in the adapter.

The idea of no data coupling of course relies on the common programming paradigm. It relies on the teams who write the domain abstractions (or the adapters) all using that programming paradigm. And it relies on having a spearate team responsible for the user story, and all teams agreeing to use ALA and the common programming paradigm.

But there is an organisational problem in the form of Conway’s law.

Conway's law: Any organisation that designs a system will produce a design whose structure is a copy of the organisation's communication structure.

It is unlikely that there would be a dedicated team in an organisation to write all the code for a specific user story if the user story spans different locations, or code written at different times. These are likely to fall to entirely different departments who both expect the two parts of the system to communicate between an agreed API and contract. In this siutaion there cannot be a separate abstraction for the user story, because there is no department for it.

So there will be collaboration in the code at each end in the form of data coupling. There will need to be be contracts. The contracts will describe all the implicit coupling. The contract will be a second source of truth, which must be kept updated.

In this situation it is still possible to mitigate the effects of coupling somewhat. Let’s say the display end has been written by the 3rd party, but is written in such a way that it accepts self describing data according to a standard. Effectively this is just making it more abstract. Without changing the display end, the user story can be implemented from scratch by sending to the display the self describing data. The display then knows how how to receive the label and display format (which can be sent once) as well as the numeric data. The display knows how to create a space for displaying the data. This is how browsers work.

It is common, for example, for a 3rd party to provide a sensor and publish the data on an MQTT server as self describing data. Say the other team is writing an application to use this data, not only display it, but interpret the data as well. They will subscribe to the topic. They will write code that is coupled with design knowledge provided by the 3rd party about the MQTT topic.

However, if the 3rd party is selling you abstract sensors that you install yourself and selling you the MQTT communication infrastructure, then you could be provided with a more abstract 'configuration API' from the 3rd party. You would then write a domain abstraction that knows about that configuration API. Whenever you want to do a new user story, you can use an instance of that 'device configuration' abstraction. You can fully configure the MQTT topic, and its data format, then subscribe to it and process it. Everything specific to the user story is now cohesively contained inside a single abstraction once again.

3.8.3. 3rd party library abstractions

All the above applies when teams are supplying peer modules for a system. The modules have a similar level of abstraction. If the 3rd party is providing something more abstract like a library, we can choose to be directly dependent on it, if it is abstract enough to be considered part of the language we want to write user stories in. The canonical example is a relational database with the abstraction being SQL.

The common problem here is that if the abstraction comes from a 3rd party, we are making ourselves dependent not only on the abstraction, which is ok, but on the provisioning of the implementation. This may be okay when we choose to depend on, for example, the windows or MacOS operating system, but is dubious for a database. (Actually its not ok for Windows either, but being able to swap out windows is considered too hard). So it’s become good practice to allow swapping out of the database. And since SQL is not quite as abstract as it should be between vendors, it means we don’t want to be dependent on SQL either.

Clean architecture suggests to do this by using interfaces specific to the user stories, and then writing adapters for every interface to SQL.

In ALA, you probably already have a 'tabular dataflow' type of programming paradigm. All abstractions that deal with tabular data already use ports of that type.

It is a matter of writing one adapter that is configured with the schema. The adapter then generates the appropriate SQL queries.

We can write a Query domain abstraction that takes a linq query as a parameter. This one abstraction allows us to use LINQ’s From, Select, SelectMany, Sort, Where, Join etc. The query abstraction is then has ports to make it wirable into a user story using ALA. Since Linq is already compliant with ALA from the point of view of composing data manipulation abstractions, there is no reason not to use it directly in this way.

TBD Write a query abstraction that takes a LINQ query as a parameter and has ITabularDataflow<T> ports. Shouldn’t be too hard so full code can go here.

TBD Write an abstract adapter for SQL Lite database with a special port that is used directly by the Query abstraction.

3.9. Composability and Compositionality

We have used the word compose a lot so far in describing ALA. The term Composability means the ability to create an infinite variety of things by composing instances of a finite number of things.

Composability is a very important property for dealing with complexity. The Principle of Compositionality states: In mathematics, semantics, and philosophy of language, the principle of compositionality is the principle that the meaning of a complex expression is determined by the meanings of its constituent expressions and the rules used to combine them.

Brian Beckman, who does the best explanation of monads I have found in "Don’t fear the monad" says that composability is the way to deal with complexity.

Jules Hedges says of this property "I claim that compositionality is extremely delicate, and that it is so powerful that it is worth going to extreme lengths to achieve it."

In software engineering, it is described by a pattern called "Abstract Interactions" or "Configurable Modularity" by Raoul de Campo and Nate Edwards - the ability to reuse independent components by changing their interconnections but not their internals. It is said that this characterises all successful reuse systems.

ALA has these properties by using domain abstractions with ports. The ports are instances of programming paradigms. The domain abstractions are the constituent expressions, and the programming paradigms are the rules used to combine them.

There are other software systems that have composability, usually using the dataflow programming paradigm, such as RX (Reactive Extensions), or more generally monads. Most composability systems are restricted to a single paradigm. In ALA, to achieve the correct level of expressiveness of requirements, multiple different programming paradigms are needed.

3.10. ALA compared with Object oriented programming

Let’s start with Brian Will’s explanation of why object oriented programming is crap from his Youtube channel:

I am in agreement with Brian in that trying to associate all code with data causes inappropriate fragmentation of the code, encourages a model of collaborating agents, and creates a dependency hell.

The idea of encapsulation is only partially realized because objects effectively know about the existence of another object’s state and collaborate with that state. They reach into each other’s data indirectly.

Also the UML class diagram encourages relationships directly between classes, which should be uncoupled abstractions. It encourages mutable data. And it encourages a horrendous model of agents interacting with each others data in a multithreaded environment. To solve this, Brian advocates a return to procedural programming and provides several examples which demonstrate that procedural programming is better.

Although ALA uses objects, it is not object oriented. You don’t try to model everything with objects and you can’t create associations between classes. It uses objects as a language feature, not a design philosophy. Objects are used in ALA for the following four reasons.

  1. Objects store references to other objects to which they are wired. A form of dependency injection is used to receive the references to the other objects.

  2. Domain abstractions, being reusable entities, often need configuring. The object stores its own configuration data passed in the constructor or via setters.

  3. Some abstractions need state to work internally. For example an abstraction that implements a low pass filter for a dataflow needs to keep some kind of historical value or values. It is inherent in the nature of the abstraction that it has state.

  4. There is usually some state data that doesn’t belong with any code. In ALA we will often create a special domain abstraction called 'state' that acts as a source or destination for dataflows.

3.10.1. Dependency injection

The dependency injection pattern was introduced as an attempt to clean up the dependency mess created by OOP. It came too late to make the famous GOF patterns book. The authors wish they had included it instead of singleton. But dependency injection alone does not solve OOPs problems.

Previously we mentioned the use of dependency injection in ALA by using the wiring pattern to wire up instances of abstractions by their ports. The way this dependency injection is done is quite different to container based dependency injection.

Container based dependency injection works by matching interface types. The interfaces are implemented by one class, and required by another. The matching of this interface type is the implicit wiring of the two classes. There is no place where you can see the wiring explicitly. This is really bad. It is very difficult to trace the flow of a user story through the classes.

Now a class may be substitutable with another class that implements or provides the same interfaces. That’s why there is a container. You instantiate an object of the class you want to wire in, and put it into the container. But this is a far cry from general composability.

In ALA interfaces do not belong to one or other of the classes being wired. They are more abstract and represent a compositional concept which we call a programming paradigm. When a domain abstraction uses one of these abstract interfaces, either implementing it or using it, we call it a port. The abstraction has no implicit fixed arrangement with other abstractions. A separate abstraction in a higher layer is needed to specify how instances of these abstractions with ports should be composed.

Note that ALA is extremely polymorphic. That’s because from the point of view inside an abstraction with ports, you do not know where that port will be wired to. Despite the use of polymorphism for wiring up everything, there is none of the usual disadvantage of indirection. In fact it is way easier to trace through a proram or system. From inside an abstraction, the abstraction doesn’t need or want to know where it is wired. It’s completely self-readable up to the port. And if you are trying to trace a dataflow through the program or system, well that’s explicit and all in one place for any given user story.

3.11. From procedural programming to ALA

Let’s get back to procedural programming that Brain Will advocates instead of object oriented programming. How does ALA relate to procedural programming?

Starting from pure procedural programming, we will make five step changes to get to ALA. In this progression, you will see that we introduce objects but not object oriented programming per se.

  1. To begin with, you can apply ALA directly to procedural programming style. Abstractions are implemented as groups of procedures. You must structure the code so that you only call procedures in an abstraction that is significantly more abstract. You will have user story abstractions in the top layer, and domain abstractions in a second layer. Procedures that directly code a given user story are put together to form a user story abstraction. Procedures that are cohesive in the domain layer, such as configure/read/write sets, are grouped together as abstractions. Such abstractions could be implemented as a code source file or a static class.

  2. Once you have abstractions, you can of course reuse them. Abstractions often need configuring. Configuring requires storing some data. We can put the configuration data inside the abstraction, and provide constructor parameters or setters. Since the abstraction now contains some data, the abstraction needs to be implemented as a class so that it can be instantiated with each instance having it own configuration. So already we have objects.

    For example, a filter abstraction needs configuring with a cutoff frequency and a stop band rejection. If the abstraction consisted only of a function, then that configuration data would need to be passed in every time the function is used. That would be awkward. It would also mix the dataflow parameters of the function with the configuration parameters, breaking the Interface Segregation Principle. By using objects, ALA can configure an abstraction once, and then the contained function can be used may times. This separation of configuration and function use is important for abstractions - the configuration is done at instantiation time of the abstraction, whereas the function can be used many times once it is running.

  3. In procedural programming, the user story will frequently call one procedure, get some data returned by that procedure and then pass it straight to another. This handling of data is not really something the abstraction should need to do. It should just compose procedures. But if we make the abstractions classes, then we can wire the instances so that at run-time data flows directly from one instance to another. Control of execution becomes the responsibility of dataflow programming paradigms. Now we have two reasons to use objects.

  4. In procedural programming, you will tend to have some state variables, which are practically globals. We won’t call them globals because we wouldn’t access them directly from any procedures. We would instead always pass them into the procedures as required. This would create extra parameters for our procedures. Some procedures will need extra parameters even though they don’t know what the data is. They are just passing it through to other procedures they call. All this extra passing around is a real pain.

    Sometimes an abstraction is by its very nature stateful. Some of the data may really belong with a given procedure internally, so the pain seems to make no sense.

    For example, a running average abstraction needs to hold past values. In procedural programming, the array of past values is kept in the top layer and is passed into the function every time. This completely breaks the abstraction. In ALA, we keep the state and code together to preserve the abstraction. This is what objects were meant for.

    In a multithreaded environments, it would be prudent for only one thread to be using each instance of such objects.

    For user story abstractions, there is probably one instance per application, so the class could be static. But once again we need objects.

  5. Lastly, there may still be state data that does not belong to a specific abstraction. This will be sitting around in a top layer looking like a global. In object oriented programming, this is the type of data we would stuff into a class anyway, and then have almost pointless accessor methods. The other classes then have harmful dependencies on these data classes. Furthermore, the dataflows through such a network of objects is completely obfuscated.

    In ALA, what we do is create a domain abstraction for such state. This abstraction has dataflow input/output ports. Instances of the abstraction can then be a source/destination of data. These are wired into data flows in the same way as any other dataflow domain abstraction. We can create instances of it for each item of state data needed by the application. Such state objects are not globals, nor do they need to be passed around. Other domain abstractions do not even know about them. Instead they are wired to them by the user story abstractions using dataflow ports. This is another legitimate reason for using objects.

    If the type system is dynamic, a state abstraction could hold any complex data structure, and the user stories it is wired to can use the data in a dynamic way. Only the application layer would know the actual structure of the data at design-time. Or it may be completely dynamic until run-time.

    If the type system is static, and we want to group data together in a single instance of a state abstraction, The application layer can use an explicit or implicit struct type. If explicit, the state abstraction will be a generic, and the struct type is passed to it at compile-time. User stories that are wired to the state instance will also have the struct type passed to them. The other way to do it is type inferencing if your language supports this. The source of the dataflow is given a type at compile-time, and the rest of the dataflow gets their types from type inference.

Through the five steps above, we have transformed procedural code into ALA code. We have used objects, but we did not use object oriented design. The resulting ALA version has these properties:

  • No class knows of the existence of state in any other class. If a UML diagram was drawn, it have no association relationships between peer domain abstration classes.

  • Despite the fact that we use objects, the ALA constraints avoid most of the problems of conventional object oriented programming. For example, both the configuration data, and the wiring data stored in an instance can be immutable. Only instances of abstractions that contain state data are mutable, and this is clear from very nature of the abstraction.

3.12. How ALA fixes object oriented programming

Unconstrained or overdone object oriented programming causes serious issues:

  1. Data gets associated with code, whether it makes a good abstraction or not.

  2. Code ends up 'reaching in' via class associations to get or manipulate data in other classes, creating coupling.

  3. Multithreading issues like race conditions and deadlocks.

ALA solves these issues by

  1. Only putting data with code when it makes a good abstraction. Other data goes into 'data' abstractions.

  2. Associations between abstractions are illegal.

  3. Asynchronous dataflow programming paradigms ensure that any given instance of an abstraction only executes on one thread. Where local synchronous communication is used, groups of instances always run on the same thread.

3.13. ALA Compared with functional programming

3.13.1. ALA can be applied to pure functional programming

The fundamental ALA constraints could be applied directly to pure functional programming (not using monads). This would require that the functions be abstractions, and that functions would only call or use functions that are significantly more abstract than themselves.

But the functions would not be great abstractions for two reasons.

1) Functions expose their inputs and outputs to the layer above, but the layer above is not interested in the data itself, only in the abstract concept of what the function does. It just wants to compose the functions with other functions, not deal with the run-time data.

2) Functions, or sets of functions, that naturally associate strongly with some state must have their state passed into and back out of them every time they are used. The layer above needs to handle this state data for the function. If the composing functions are intermediate level, they themselves will need to receive and return the state from high functions.

+ Functional programming in one way encourages abstractions by not allowing side effects. However, this also destroys any abstractions that would otherwise be highly coherent with their own state.

+ This problem gets even worse when there are several layers of functions. The middle layer functions end up with extra parameters that don’t have anything to do with them, just so they can pass state data through to even lower functions. This makes these intermediate functions not great abstractions in themselves.

Both ALA and monads solve these problems.

3.13.2. ALA uses state

The ALA constraints can be used for either object oriented or purely functional programming. Either way, using abstractions that have zero coupling with one another changes how the code is organised for the better.

In pure functional programming, if the data and the functions are good abstractions, then the structure is already ALA compliant. Both the data and the functions are in the domain abstractions layer. Then user story functions in a higher layer passes the data to the functions.

However, it is unlikely that pure functions that require state to be passed into them are also good abstractions. An example would be an averaging filter for a stream of data. Passing the running average to the function every timer there is new input data breaks an otherwise good abstraction. There are many situations when insistence on referential transparency makes a computation more awkward and less precise, and it is this loss of abstraction that causes this.

In fairness pure functional prgramming uses monads to partially solve this problem. ALA is actually somewhat analogous to monads. Monads are a mechanism that allow the creation of abstractions that can be wired together, similar to domain abstractions. Indeed ALA user stories that wire instances of domain abstractions use pure functional code. A full explanation of monads for programmers who know nothing about them, but are familiar with imperative programming in C# is in chapter six. They are compared in detail with ALA. Essentially domain abstractions together with programming paradigms are a more versatile, powerful but conceptually simpler analog of monads.

ALA prioritizes abstraction over referential transparency as its method to achieve analysability. This sometimes results in pure functions and more often results in objects with mutable state. When using objects though, ALA’s big difference from conventional object oriented programming is that the classes must be much better abstractions (because they can have no associations with one another).

The use of state in conventional object oriented programming is considered a problem for two reasons.

  1. Class methods lack the advantages of referential transparency. Referential transparency essentially works by removing time as factor when analysing or understanding a section of code.

  2. At runtime, methods called on different threads can cause issues such as race conditions, deadlocks, and priority inversions.

For the first problem, instead of removing time as an internal mechanic of the way classes work, ALA removes all coupling with other classes. So the class is locally analysable even if it contains state. This allows many computational problems that naturally involve time or ordered operations to be expressed in a straightforward way, without descending into the quagmire conventional unrestrained imperative programming.

Many systems or subsystems are inherently state machines anyway - they are driven by events and need to change their behaviour according to their history. Programs for such systems ultimately have mutable state. Functional programming tends to separate that state out causing a lot of design-time coupling between functons and their data. Object oriented programming on the other hand tends to encapsulate it with methods too much, causing a lot of coupling with other classes that indirectly use the data. ALA solves both these problems.

State is a fundamental aspect of computation. I define a 'computation' as some state and a function:

input + state --> state + output.

We can associate the state in two ways:

(input + state) --> (state + output)
input ( + state --> state +) output

The first form is the pure functional form. The second form is the object oriented form.

In ALA we can use either form depending on where the abstractions are. When there is some state that is used by multiple user stories, then it would be put in its own abstraction. For example, a blackboard pattern or a game score could be an abstraction itself. This abstraction could then be wired by user story abstractions in a higher layer to various abstractions implemented as pure functions.

In ALA we choose between these two philosophies on a case by case basis. When it makes sense to put state with methods as a class, and that makes a good abstraction, then there is no resulting coupling of that state with other classes. On the other hand, when it makes sense to keep state separate, we can put it in its own separate abstraction (usually as a class with simple accessor methods). Either way, what you then must not do is create direct associations between these classes. That will break them as abstractions. Instead, top layer user stories wire up insteances of those classes using ports.

In other words, the problem with object oriented programming is not that objects contain state per-se. It is that most classes are not good abstractions of a computation. We allow other objects to 'reach' into them or couple with them at design-time. They know there is state there, even if that 'reaching in' is via methods. This coupling is what makes conventional programs with state so hard to reason about.

In summary, ALA enforces abstraction instead of referential transparency.

Multithreaded programming with state

The second problem with object oriented programming is multi-threaded environments. This is true even if we are using abstractions. That’s because multi-threaded issues happen at run-time and abstractions only deal with design-time coupling. At rn-time, instances of abstractions communicate in ALA just as much as they do in conventional object oriented programs. The abstractions themselves may not know anything about each other, but their instances are wired to communicate.

If the programming paradigms used for the wiring use asynchronous method calls, there is not problem with multi-threading. But if the programming paradigms used for the wiring use synchronous method calls, there will be issues such as race conditions, deadlocks or priority inversions.

Effectively this is a dependency of all domain abstractions on an understanding that if synchronous method calling is used in the programming paradigm, two instances should not be wired if they run on different threads.

This is fine under the ALA constraints. Domain abstractions that contain state have a knowledge dependency that programming paradigms that use synchronous must be used only for instances that run on one thread. The user stories in the layer above similarly have this knowledge dependency. With this knowledge dependency, domain abstractions can be written without regard to thread safety, and user stories must be written only use synchronous programming paradigms for wiring instances of domain abstractions that run on the same thread.

You could argue that this knowledge used in the user stories is a violation of ALA in that the user story, as an abstraction expressing a requirement, should not also have to be concerned with this knowledge. This is a valid criticism.

From the point of view of the domain abstractions, you can write them so that they can be used for either asynchronous or synchronous. A port that uses a programming paradigm that is asynchronous can be used either asynchronously or synchronously. The user story can configure whether the programming paradigm instance uses synchronous or asynchronous at design time. User stories can then follow the GALS principle (Globally Asynchronous, locally synchronous) applied to threads.

Writing domain abstractions to use asynchronous programming paradigms is more difficult than ones that use synchronous programming paradigms. The difficulty comes when the port is used with an outgoing method call that returns a value. Such programming paradigms might be simple pull style data transfers, request/response type, or perhaps programming paradigms that return an error.

Using push style programming paradigms by default for data transfers mitigates this problem in most cases. That’s why in ALA I recommend push style or reactive programming by default.

But when something must be returned, and we want to use an asynchronous port, this forces the domain abstraction to be a state machine. By far the easiest way to do this is to use async/await. This allows the domain abstraction to be written in the synchronous style, save for the addition of async and await keywords everywhere in the code. The compiler turns the code into the required state machine.

If async/await is not available, but tasks or future objects are available, then you can use a bind function to chain the task objects. This will allow you to chain operations using dot syntax, which is just as elegant as a chain of synchronous calls. This is effectivly a monad. Monads are fully explained in chapter six.

If you can’t even use the task monad, then the domain abstractions must be written in the style of a state machine.

Turning an outgoing port method call into a state machine means moving all the code that is after the method call into a separate call back function. Any local variables in the containing method become state that persists until the callback function is called. Writing code in this way can be straight-forward or awkward. If it’s awkward, at least it is confined to the internals of an abstraction.

Remember all this is only necessary when you want to write domain abstractions whose instances can be wired to run on different threads (or over a network).

3.13.3. ALA compared with monads

In chapter six the first section explains monads for people familar with imperative programming. Here we just give a comparison of ALA and monads, assuming prior familiarity with monads. We will show how ALA does a similar job to monads, but in a more powerful, more versatile and more straight-forward way.

When we use the term monad library in this section, we are referring to a whole library based on monads. For the most common monad, the monad based on IEnumerable, this library consists of the familiar functions such as Select, Where and Order found in LINQ.

Strictly speaking the monad itself only has the Bind function (which is SelectMany in the case of the IEnumerable monad), not all the other map and reduce functions you find in the library as well.

Monads compose functions, ALA composes objects

The fundamental difference between ALA and monads is that ALA composes objects whereas monads compose functions. Monads often use objects under the covers, but in the application code you are composing functions.

With a monad library, your application code can combine functions in two different ways.

  1. Compose more generic functions that essentially compose lambda functions. These more generic functions include Bind (SelectMany), Map (Select), Reduce (Aggregate), Filter (Where), Join, and GroupBy. Each takes a lambda function. The application code is composing both the generic functions and the specific lambda functions that are passed to them.

  2. Compose more specific functions such as Repeat, Cast, Sum, First, Concat, Order. These functions generally work by themselves without also composing other functions passed to them.

Whether we consider the more generic functions, the specific functions, or the lambda functions, most all the functions just have two 'ports', an input and an output. The input is the first parameter and the output is the return value. (Some functions such as Join, Zip, Concat, take two inputs.) ALA’s domain abstractions, on the other hand, can have many ports for their inputs and outputs. These ports can use different programming paradigms, not just a specific data flow such as IEnumerable. This makes ALA abstractions more general.

Furthermore, most of the monad library functions are designed to work with finite sequences. Sum, for example, requires the input sequence to complete. ALA ports can, and often are just a stream of data for the lifetime of the application. When data is in batches, such as from a file or database query, the ports use programming paradigms that handle batches of data. Such a programming paradigm is a stream of sequences (see IObserverPush in chapter six for an example). The ports remain wired for the lifetime of the application. For monad libraries, the composed expression is usually rewired for each use (although an IEnumerable can be saved in a variable and then reused). This is especially true for the reactive extensions, in which most functions stop working once OnCompleted or OnError occurs.

So the mental model of how monad libraries and ALA domain abstractions are composed is different. Monad library expressions are a declarative version of a finite loop such as a forloop. It builds an object structure for lazy execution. But these expressions are wired in isolation and then executed within the context of conventional imperative code.

In ALA, on the other hand, the entire user story is built declaratively because the ports use programming paradigms that declaratively support everything needed to express a user story. The entire object structure representing the user story, indeed the entire application, is wired once, and then set running for the lifetime of the application.

Having said that, WPF (Windows Presentation Foundation) provides a declarative way to build the UI. So when WPF is used with bindings to LINQ expressions, more of the user story can be expressed declaratively. ALA just makes the idea of declarative programming generalized to the point where you always build the entire application declaratively.

To use an analogy, ALA’s domain abstractions are somewhat analogous to wiring integrated circuits in electronics. Like domain abstractions, integrated circuits can have multiple ports (pins) of different types to suit what the overall abstraction of the part. Monad libraries are more like discrete electronic components such as resistors and capacitors, components that just support analog data with a single input and output.

Monad versus ALA syntax

In ALA application layer code, you are wiring up instances of abstractions that have multiple ports in an arbitrary network (that expresses a whole cohesive user story).

In monad library application layer code, you are generally chaining functions, and only wiring up a dataflow. So the monad library code can look more succinct for the specific part that involves the chain of functions:

monad version

source.Filter(x => x>0).Select(x => x+1)

ALA version

source.WireIn(new Filter(x => x>=0)).WireIn(new Select(x => x+1)

However, we wouldn’t normally create ALA domain abstractions to do the same job as monads if we already have a monad library. Instead, we would use the monads, and their more direct syntax to configure an ALA domain abstraction. Chapter six has full details of how to this.

More generally, in ALA the usual pattern is to explicitly instantiate the domain abstractions using new and then explicitly wire their ports using WireTo or WireIn. We could create extension methods to combine these two operations, but we prefer to keep it explicit because conceptually we are wiring up a network like wiring up components of a schematic diagram. Extension methods that wire abstraction directly using dot syntax are only elegant for building linear chains.

If the application code is generated from a diagram, it doesn’t make any difference, but if you are hand writing the wiring code, the ALA syntax is longer. But it’s more obvious what is happening.

Monad library code usually builds large object structures full of delegate objects, closure objects and other 'under the covers' objects. This 'under the covers' structure makes monads difficult to understand, trace and debug, compared to ALA object structures.

Deferred Monads vs wiring classes with ports

Monad Bind functions can call the composed function immediately to return an actual value, or they can return an object that can be used later to get the value. For example, monad library functions that return an IEnumerable or IObservable are the deferred form. They can also work using either pull or push. It is the deferred/push form of monads that most closely compares with ALA, so that’s the type of monad we will concentrate on in this section.

The deferred versions of monads are more difficult to understand in terms of equivalent imperative code because relatively simple expressions create large object structures containing delegates, closure, or other objects under the covers. You can’t really see any of them. This also makes them difficult to debug. Example diagrams of these structures are shown in Chapter six.

ALA is easier to understand because it uses plain objects. You instantiate and wire up these objects yourself.

The only thing that is peculiar to ALA over conventional objected oriented programming is that the ALA objects must use ports. You are not allowed to make a class have an association with another peer class. You are not even allowed to make an association with a same layer interface.

Ports can be thought of as dependency injection with two constraints.

  1. The interfaces must be significantly more abstract. These interfaces are called programming paradigms because they give meaning to the wiring.

  2. The dependency injection wiring must be explicit. It must be specified in code in a cohesive user story abstraction in a higher layer. The wiring cannot be done by using a dependency injection container or relying on matching interfaces. Because port interfaces are more abstract and can be used by many disparate classes, there is an infinite number of ways that objects could be wired together.

Another difference about ALA dependency injection is that I like to use reflection and implement a single WireTo extension method for wiring every type of port. That way I don’t need constructor injection or setter injection methods for every port of every domain abstraction class.

That’s pretty much all there is to understanding the wiring of ALA classes.

3.13.4. Why prefer deferred/push monads?

We mentioned that deferred/push monads are closest to ALA. For the case of IObservable, these are the reactive extensions library. Deferred/push monads have two useful properties.

  1. The first property comes from being deferred. Often people write a deferred query using monads, and then immediately run the query. The deferred nature of the monads is not really utilized because the query is used immediately. Usually it needs to be run immediately because even though the query itself is a declarative description of a dataflow, it is used in an imperative context.

    Now imagine constructing a deferred query, and then wiring it into still more declarative code that describes the entire user story. It is the deferred property of the query code that allows us to build the query along with the UI, the schema, the business logic, etc, all declaratively. Once the whole program is built, it can be set running.

    Building a query from deferred monads once with SelectMany, Select etc, and then reusing it many times as the program runs means that the monads must be designed to be used more than once. The monads based on IEnumerable are fine in this respect because it has a Reset method that can be used to re-run the IEnumerable. However monads based on the IObservable interface are not designed to be able to run more than once. Once that OnCompleted or OnError method is called, the object structure wont run ever again. It has to be reconstructed by re-running the SelectMany and Selects, which will re-run the Subscribe method calls. I think this design in reactive extension is rather limiting. You can have so called 'hot' observables, but that just means they never call OnCompleted or OnError.

    So when I use IObservable monads as part of a user story, I need rebuild the query in imperative code each time it is used. With ALA domain abstractons, I can build the query once and wire it into the rest of the user story that also uses declarative code.

  2. The other advantage of deferred/push monads is the fact that they use push. In ALA we design dataflow or event programming paradigms to use push by default, and only use pull if there is some good reason to use pull.

    The difference is that push can be either synchronous or asynchronous, whereas pull can only be synchronous (unless you use future objects.)

3.14. Dependency graphs for ALA vs conventional code

Our example for this chapter compares dependency graphs for a legacy application and one with the same functionality written in ALA.

The legacy application had been maintained for approximately 20 years, so as might be expected, maintenance had become difficult. In fact there were certain new requirements we could not do because of the prohibitive effort. Normally I wouldn’t ever re-write an application. But I wanted to run a research experiment to see how ALA would work on it. I had intern students available, and it would give us an opportunity to compare metrics of the two code bases.

The original application has around 70 KLOC. Rather than look at any of the details of the application itself, we present here dependency graphs generated by Ndepend for the legacy application and the new ALA application.

3.14.1. Legacy application dependency graphs

One of the core tenets of ALA (as discussed in an earlier section) is "Composition using layers" instead of "Hierarchical decomposition using encapsulation". Unfortunately Ndepend is designed with the assumption that the application should be built using the latter approach. It likes to present a decomposition structure, starting with assemblies (packages) at the outermost level, then namespaces, and then classes. I’m not sure why it considers namespaces a viable encapsulation mechanism because they don’t provide encapsulation.

Namespaces provide no useful decomposition structure. They do not make abstractions in themselves, nor do they implement a facade pattern or an aggregate root type of pattern with even logical encapsulation. Any classes inside each namespace can have unconstrained direct relationships with any classes in any other namespace.

So Ndepend out of the box gives us a false picture, because it omits all dependencies that go in or out of namespaces. To really get an idea of what the class dependency graph looks like, I configured Ndepend to use a query that gives me all the classes in all the namespaces. Here is what the legacy application truly looks like:

classes-in-all-namespaces.png
Figure 21. Legacy application - all classes in all namespaces

This graph is very large. Right click on it, and open it in a new tab in your browser, so you can zoom in to see the dependencies in the background. It is truly frightening. Ndepend had no chance to find the dependency layers. There may be vague onion type layers going outwards from the middle. It makes readily visible why continued maintenance on this application is so difficult. You have to read a lot of code to find even a tiny part of this hidden structure.

The developer who maintains the application tells me this is a fair reflection of the complexity that he has to deal with.

To be fair, some of the dependencies in this diagram are 'good' dependencies (as described in an earlier section on good and bad dependencies). For example, the box near south-east called ScpProtocolManager has a lot of dependencies coming into it, which means it is possibly used a lot and therefore is potentially good abstraction. Ndepend does not know about the concept of good and bad dependencies, but if it did I would have it just display the bad ones.

3.14.2. New ALA application dependency graphs

Here is the equivalent Ndepend generated class dependency graph for the new ALA version of the application. This graph has the classes from all namespaces.

classes-in-all-namespaces.png
Figure 22. New ALA application - classes in all namespaces

You can see the three ALA layers which are vertical and go from left to right. Only the Application sits in the top layer. The DomainAbstractions layer contains the next two columns of classes and a few from the next column. And the ProgrammingParadigms layer contains the rest on the right. Actually there were a couple of bad dependencies present when this graph was generated which have since been fixed. (There should be no dependency between Panel and OptionBox, nor between Wizard and WizardItem.) With these removed, the graph would form into the three abstraction layers.

The newly rewritten application is a work in progress at this point. However, as features are added, this is all the dependencies you will ever see. The Application already uses most of the domain abstractions we will ever need, and the domain abstractions already use the programming paradigm interfaces they need. There are a few DomainAbstractions to be added, but this is essentially what the class dependency graph will stay looking like.

As of this writing, the new ALA version of the application is still a research project, but so far everything has gone smoothly with two weeks spent doing the description of the requirements as a diagram, and 18 months so far spent writing the domain abstractions. So far there are no issues getting it to actually execute. It is expected that we will actually commercialize the project soon and replace the old application.

3.14.3. The application’s diagram

As we said in this chapter, diagrams can be an important aspect of ALA when the user story naturally contains a network of relationships among its instances of abstractions. In this application this is the case. There are UI relationships between elements of the UI. There are dataflow relationships between UI elements, data processing elements, and data sources. There are event-flows from UI to wizards and between wizards and the SaveFileBrowser. and there are minor dataflows such as a the filepath from the file browser to the csvFileReaderWriter.

Here is a sample section from the application diagram that shows all the relationships that implement the user story:

DatalinkApplication.xmind.png
Figure 23. Xmind being used to design an application

This diagram was drawn using Xmind. It shows a single user story. There is a UI with a menu item or a tool bar to start the user story. It then displays a browse dialogue to specify the location of the file. When the filepath has been selected, it gets data off a device on a COM port, using a protocol, and writes it to a CSV file. The data is also routed to be shown on a grid on the UI.

The user story diagram makes use of four different programming paradigms (which become four different interface types). Firstly there is the UI structure consisting of the window with its menubar, grid etc arranged inside it. Secondly, there is an event connection for when the menu is clicked which opens the browse dialog. Thirdly a dataflow connection carries the output of the browse dialog, a string containing the selected filepath, to the CSVFileReaderWriter. Another dataflow connection carries characters between the COM port and the SCPProtocol and another carries SCPcommands from the SessionDataSCP. The forth programming paradigm is a table dataflow that carries dynamic columns and rows of data from the SessionDataSCP object to the grid object in the UI and to the CSVFileReaderWriter.

Having drawn the diagram to represent the user story, we need to make the diagram execute. When we started this particular project we had no tool for automatically generating the code from the diagram, but during the project, one of the interns wrote a tool to do this. It parsed the Json output from Xmind and generated C# wiring code equivalent to what we will show below.

However, at first we were hand generating code, and it is instructive to know what this hand generated code looks like, just so we know how the diagram actually executes.

When we were hand generating the code, it was important that the code was readable from the point of view of seeing how it corresponds exactly with the diagram. (It wasn’t important that the code was readable from the point of view of seeing how the user story works - that was the job of the diagram.) We had various conventions to support the one to one matching of diagram and code. One of these conventions was to indent the code to exactly mirror the tree structures in the diagram. Another was that whenever a new instance of an abstraction instantiated, all its ports would be wired immediately, and they would be wired in the order they were declared in the abstraction. This implies a depth first wiring strategy, analogous to walking the diagram tree depth first. Any ports with cross connections (the red lines in the diagram) would also be wired to their destinations at the time the abstraction were instantiated. If the destination instance did not already exist it would be pre-instantiated.

Using these conventions, it is a simple matter to hand generate the code below from the diagram.

using System;
using System.Windows.Media;
using DomainAbstractions;
using Wiring;


namespace Application
{
    class Application
    {
        private MainWindow mainWindow = new MainWindow("App Name") { Icon = "XYZCompanyIcon"};

        [STAThread]
        public static void Main()
        {
            new Application().Initialize().mainWindow.Run();
        }

        private Application Initialize()
        {
            return this;
        }

        private Application()
        {
            var getInfoWizard = new Wizard("Get information off device") { SecondTitle = "What information do you want to get off the device?" };
            Grid DataGrid;
            var sessionDataSCP = new SessionDataSCP();
            var csvFileReaderWriter = new CSVFileReaderWriter();

            mainWindow
            // UI
                .WireTo(new Vertical()
                    .WireTo(new Menubar()
                        // XR3000
                        .WireTo(new Menu("File")
                            .WireTo(new MenuItem("Get information off device") { Icon = "GetDeviceIcon.png", ToolTip = "Get session data or LifeData or favourites from the device\nto save to a file or send to the cloud" }
                                .WireTo(getInfoWizard)
                            )
                            .WireTo(new MenuItem("Put information onto device") { Icon = "PutDeviceIcon.png" })
                            .WireTo(new MenuItem("Exit") { Icon = "ExitIcon.png" })
                        )
                        .WireTo(new Menu("Tools"))
                        .WireTo(new Menu("Help"))
                    )
                    .WireTo(new Toolbar()
                        // XR3000
                        .WireTo(new Tool("GetDeviceIcon.png") { ToolTip = "Get information off device" }
                            .WireTo(getInfoWizard)
                        )
                        .WireTo(new Tool("PutDeviceIcon.png") { ToolTip = "Put information onto device" })
                        .WireTo(new Tool("DeleteDeviceIcon.png") { ToolTip = "Delete information off device" })
                    )
                    .WireTo(new Horizontal()
                        .WireTo(new Grid() { InstanceName = "Sessions" })
                        .WireTo((DataGrid = new Grid() { InstanceName = "DataGrid" })
                            .WireFrom(sessionDataSCP)
                        )
                    )
                    .WireTo(new Statusbar()
                        .WireTo(new Text() { Color = Brushes.Green }
                            .WireFrom(new LiteralString("Connected to device"))
                        )
                    )
                );


            getInfoWizard
                .WireTo(new WizardItem("Get selected session files") { Icon = "IconSession.png", Checked = true }
                    .WireTo(new Wizard("Select destination") { SecondTitle = "What do you want to do with the session files?", ShowBackButton = true }
                        .WireTo(new WizardItem("Save selected sessions as files on the PC") { Icon = "SessionDocumentIcon.png", Checked = true }
                            .WireTo(new SaveFileBrowser("Select location to save data") { Icon = "SaveIcon.png", InitialPath = "%ProgramData%\XYZCompany"}
                                .WireTo(csvFileReaderWriter)
                            )
                        )
                        .WireTo(new WizardItem("Send records to NAIT") { Icon = "NAIT.png" })
                        .WireTo(new WizardItem("Send sessions to NLIS") { Icon = "NLIS.png" })
                    )
                    .WireTo(getInfoWizard)
                )
                .WireTo(new WizardItem("Get Lifedata"));

            var comPorts =
                new ComPortAdapter()
                    .WireTo(new SCPProtocol()
                        .WireTo(new SessionDataSCP()
                            .WireTo(DataGrid)
                            .WireTo(csvFileReaderWriter)
                        )

                    );

        }
    }
}

We used a 'diagram first' rule to keep the diagram and code in sync. Change the diagram first, then change the wiring code.

As of this writing, a graphical IDE is being developed for these types of ALA applications.

4. Chapter four - Programming paradigms and their execution models.

4.1. Introduction to programming paradigms

ALA fundamentally begins with the premise of using abstractions to achieve zero coupling at design-time. Zero coupling is preserved if relations between abstractions are always in the direction of greater abstraction. Thus abstraction layers emerge, with each layer significantly more abstract than the one above. It is interesting to observe how these layers seem to emerge typical usage patterns, which in turn give rise to their names: Application layer, User stories layer, Domain abstractions layer, Programming Paradigms layer, and so on.

JacquardLoom.jpg
Figure 24. Jacquard loom as a programming paradigm for combining weave pattern rows

The layer below the domain abstractions is really interesting in this respect. When we compose or wire two or more instances of domain abstractions together, we need that to have a meaning. Here are some common examples:

  • Imperative ((sequential activities in computer time)

  • event driven

  • dataflow

  • UI layout

  • activity flow (sequential activities in real time)

  • state machine transition

  • state machine substate

  • data schema

These are all quite abstract concepts. They have great potential for reuse. We call them programming paradigms because each is a different way of thinking about programming. Each gives different meaning to composition. We call the layer they go in programming paradigms.

It is an essential part of ALA that we can use multiple programming paradigms in the same user story. In ALA, user stories (or features) are cohesive abstractions. To completely describe a user story, common programming paradigms needed may be UI layout, dataflow, activity and data schema. This use of multiple programming paradigm is referred to as polyglot programming paradigms.

The programming paradigms layer may also contain other abstractions useful for building domain abstractions, but are not used as relationships between instances of abstractions. Examples are the concepts of 'Persistence' or 'Styles'. In this chapter we will be concentrating on programming paradigms used to compose instances of domain abstractions, and showing the ways they actually execute, which is called their execution model.

Here is Peter Van Roy’s taxonomy of programming paradigms which gives us an idea of what the term "programming paradigm" means in general. Peter Von Roy is an advocate for multiple programming paradigms in a computer language, which is why the language Oz appears all over the diagram. In ALA we invent and implement our own programming paradigms, without relying on them being built into the underlying programming language.

TaxonomyProgrammingParadigms.png
Figure 25. Taxonomy Programming Paradigms - cited from Van Roy

In ALA "programming paradigms" are used to compose instances of domain abstractions. The programming paradigm provides "the meaning of composition".

Programming paradigms are by no means limited to the ones discussed in this chapter. Custom ones can be invented as needed (when they allow better expression of the requirements). We do this with the example at the end of this chapter for scoring games such as Bowling or Tennis. We use a 'ConsistsOf' programming paradigm which allows us to express that a match consists of sets, a set consists of games, and so on.

4.2. Introduction to execution models

Programming paradigms and execution models are closely related but not precisely the same thing. Programming paradigms are the meaning of composition. Execution models are how we make that meaning actually work. Essentially, the execution model is how the underlying imperative programming paradigm of the CPU is going implement the new programming paradigm.

Some programming paradigms can have simple execution models. They can be implemented with an interface with one method. The simplest example of this is the synchronous event programming paradigm:

IEvent.cs
namespace ProgrammingParadigms
{
    interface IEvent
    {
        void Send();
    }
}

Such an interface looks like it adds nothing, but it transforms programming to a paradigm of composing instances of abstractions in an infinite variety of possible ways.

Ports use instances of these interfaces. Essentially the interface allows any output port using this paradigm to be wired to any input port using this paradigm.

Other programming paradigms may require an engine or framework. Possible ways these execution models could work is discussed under each programming paradigm in this chapter.

4.3. Coding execution models

Here we just show simple code examples for two execution models for the event driven programming paradigm to get a grounding at the code level.

4.3.1. Synchronous events

The interface for synchronous events was given just above.

The difference between this and your everyday common imperative function call or method call is only the indirection. The sender doesn’t know who it’s calling. (In conventional code, indirection creates problems, but in ALA these problems do not exist, so all communications use this type of indirection. This is discussed further later.)

First let’s create two dummy domain abstractions with ports using the synchronous event driven programming paradigm. The two domain abstractions will use and implement this interface respectively:

A.cs
using System;
using ProgrammingParadigms;

namespace DomainAbstractions
{
    class A
    {
        private IEvent output; (1)

        public void start()
        {
            Console.WriteLine("1");
            output?.Send();
            Console.WriteLine("3");
        }
    }
}
1 The output port is a private field of type interface
B.cs
using System;
using ProgrammingParadigms;

namespace DomainAbstractions
{
    class B : IEvent (2)
    {
        // input port
        void IEvent.Send()
        {
            Console.WriteLine("2");
        }
    }
}
1 The input port is an implemented interface

Now we can write an application that wires an instance of A to an instance of B.

Application.cs
using System;
using DomainAbstractions;
using ProgrammingParadigms;
using Foundation;

namespace Application
{
    class Application
    {
        static void Main(string[] args)
        {
            var program = new A().WireTo(new B()); (1)
            program.start();
        }
    }
}
1 The meat of the application wires an instance of class A to an instance of class B.

The output of the program is "123".

The Main function instantiates one instance of each of our domain abstractions, and wires them together. (If you have not seen the WireTo abstraction before, it is an extension method that uses reflection to search in class A for a private variable with a type that is an interface. It then sets it pointing to the instance of B if B implements the interface. WireTo is not central to the current discussion, the IEvent interface is. WireTo is discussed in more detail in the example projects of chapters two and three.)

Notice just how abstract IEvent is. It’s highly reusable. It’s not specific to any domain abstraction or the application. It just knows how to transmit/receive an event. Because it is so abstract, it is stable. The more domain abstractions that depend on it the better, as that will allow them to be wired together in arbitrary ways, which gives us composability.

The IEvent interface can be compared with the observer pattern (publish/subscribe) which also claims to achieve decoupling. However the observer pattern only reverses the dependency of a normal method call. Instead of the sender knowing about the receiver, the receiver knows about the sender (when it registers for the event). If the sender and receivers are peers in the same layer, the observer pattern does not solve the problem. The IEvent interface decouples in both directions. The job of 'subscribing' is moved to the application layer, because only the application should have the knowledge of what should be wired with what.

4.3.2. Asynchronous events (the event loop)

In the above example, we used the word event, but implemented it in a specific way (a synchronous method call). The terms event and event driven may have overloaded meanings. To some it may mean asynchronous or it may mean observer pattern (an event is a public thing you can subscribe to), or it may mean both.

In ALA the term means neither of these. As a programming paradigm it simply means that we think of programming as reacting to what happens instead of prescribing what will happen next - a reactive rather than prescriptive programming style. They can be either synchronous or asynchronous. They are never public - the layer above always wires them up from point to point explicitly. Events can be wired to fan-in or fan-out.

We discuss the meaning of synchronous and asynchronous in more depth later, but here we just want to see how asynchronous can be implemented at the code level. Synchronous and asynchronous are two different execution models for the same programming paradigm.

To implement the asynchronous execution model, conventional code may use an event loop that works something like this: the originator of the event calls a Send method on an EventLoop object. It passes a reference to a function or method of another object that it wants to send the event to. The Send method in EventLoop creates an object that represents the event and puts it into a queue. The Send method then returns. The main loop resides in this EventLoop object. It loops taking events from the queue one at a time and calls the referenced function or method. This is sometimes called the reactor pattern, but its actually a simplified version of reactor so we will call it simply an event loop.

For ALA, the only difference is that the sender can not specify the receiver function and object.

Here is the application layer code:

Application.cs
using System;
using DomainAbstractions;
using ProgrammingParadigms;
using Foundation;

namespace Application
{
    class Application
    {
        static void Main(string[] args)
        {
            // instantiate an asynchronous execution model
            var eventLoop = new EventLoop();

            // Wire using the asynchronous execution model
            var program = new A().WireTo(new B(), eventLoop); (1)
            program.Start();

            eventLoop.Start();
        }
    }
}
1 The meat of the application wires an instance of class A to an instance of class B.

The difference with our previous synchronous application is that we first spin up an asynchronous execution engine object called eventLoop. The WireTo is used in the same way except that we pass in the execution model.

Here are the A and B dummy domain abstractions again. They are identical to the ones we used for the synchronous version above.

A.cs
using System;
using ProgrammingParadigms;

namespace DomainAbstractions
{
    class A
    {
        private IEvent output;

        public void Start()
        {
            Console.WriteLine("1");
            output?.Send();
            Console.WriteLine("3");
        }
    }
}
B.cs
using System;
using ProgrammingParadigms;

namespace DomainAbstractions
{
    class B : IEvent
    {
        // input port
        void IEvent.Send()
        {
            Console.WriteLine("2");
        }
    }
}

When this program runs, it will print "132" instead of the "123" that the synchronous version did. At the domain abstraction level, we need to not care whether it is "123", or "132". If we do care, then we need to use a different programming paradigm.

Now let’s have a look at the programming paradigm abstraction to see how it works.

AsynchronousEvent.cs
using System.Collections.Generic;
using Foundation;

namespace ProgrammingParadigms
{
    public interface IEvent (1)
    {
        void Send();
    }


    static class EventLoopExtensionMethod (2)
    {
        public static T WireTo<T>(this T A, object B, EventLoop engine, string APortName = null)
        {
            engine.WireTo(A, B, APortName);
            return A;
        }
    }




    class EventLoop
    {

        private List<IEvent> queue = new List<IEvent>(); (3)

        public void WireTo(object A, object B, string APortName) (4)
        {
            A.WireTo(new Intermediary(this, (IEvent)B), APortName);
        }

        public void Start()
        {

            while (!Console.KeyAvailable)
            {
                if (queue.Count > 0)
                {
                    IEvent receiver = queue[0];
                    queue.RemoveAt(0);
                    receiver?.Send();
                }
            }
        }


        private class Intermediary : IEvent
        {
            private IEvent receiver;
            private EventLoop outerClass; // needed to access our outer class instance

            public Intermediary(EventLoop outerClass, IEvent receiver)
            {
                this.receiver = receiver;
                this.outerClass = outerClass;
            }

            void IEvent.Send() (5)
            {
                outerClass.queue.Add(receiver);
            }
        }
    }
}
1 The abstraction begins with the interface itself, which is unchanged from the synchronous version.
2 Overload of the WireTo extension method. We want an extension method so that we can wire things up using the same fluent syntax as the standard WireTo. This WireTo overload differs from the usual WireTo by the extra parameter for passing in the AysnchronousEventLoop instance. The method simply defers to the WireTo method in the EventLoop class.

This mechanism of overloading the WireTo method can be used by any programming paradigm.

1 The EventLoop class keeps a list of events waiting for execution.
2 Unlike for the synchronous case, the sender’s port is not wired directly to the receiver’s port. An intermediary object is wired in-between. The class for the intermediary object is inside the EventLoop class as we don’t want it to be a public part of the EventLoop abstraction.

The WireTo method instantiates an intermediary object, stores the receiver object cast as the interface into it (which is effectively wiring the intermediary to the receiver), and then calls the standard WireTo in the Foundation layer to wire the sender to the intermediary object.

1 When the sender calls Send on its output port, the intermediary object intercepts the synchronous call. The intermediary object queues the call in the EventLoop class and immediately returns. It actually queues the reference to the interface of the receiver.
2 The EventLoop class has a loop that takes the references to receiver objects out of the queue one at a time, and calls the IEvent’s Send method in the receiver.

In this example we have put the main loop inside the execution model for simplicity. We would not normally do this because we may have several different programming paradigms each with their own main loops. So we could have the main loop in the Foundation layer, and the different execution models would register a Poll method on it. Alternatively we could make the loop function an async function that awaits on an awaitable queue implementation.

As usual in ALA, we do not try to decouple anything inside the AsynchronousEvent.c abstraction. Everything in it cohesively works together.

The propose of the examples is to show that we can create our own programming paradigms and that their implementation can be simple.

4.4. Execution model properties

Now that we have the idea of what we mean by programming paradigms and execution models at the code level, we next need to discuss some general properties of execution models, such as direct vs indirect, synchronous vs asynchronous, push vs pull, etc. we will refer to these properties when discussing specific execution models later.

In conventional imperative code, the execution model is inherently synchronous in the use of the function or method call. In ALA we have free choices for execution models. Also in conventional code, one of the forces is managing dependencies. This can influence the execution model. For example we might pull for a dependency reason even though we would rather push for a performance reason. In ALA, wiring does not involve dependencies, so we are free to focus on other design choices with respect to execution models.

In this section we will try to clarify what these design choices are for programming paradigms that mean communication. We will note the forces on these design choices.

4.4.1. Sideways vs down vs up communications

In conventional code, communications generally follow dependencies directly. If we try to think in terms of layers, with dependencies always going down the layers, these communications always go either up or down. So we may not be used to thinking of sideways communications. Or if we do allow sideways dependencies within a layer, we may not be used to thinking about sideways communications and up/down communications as different things.

In ALA, we need to think of them differently. Let’s refer to sideways communications as wired-communications, and up/down communications as abstraction-use-communications.

A common example of abstraction-use communication is when you configure an instance of an abstraction by passing parameters to the constructor, or by calling setters. Another example is calling a squareroot function in your math library. A common example of upward communication using abstraction-use-communication is executing a lambda expression that has previously been passed in to an instance of an abstraction during its configuration. Upward calls are always indirect in some way, such as the mentioned lambda expression, passing in an anonymous or named function, observer pattern (publish subscribe), callback, or strategy pattern. We don’t use virtual functions in ALA for up calling because we don’t need or want to use inheritance.

A common example of wired-communications is when an instance of an abstracton sends something out on a port. It arrives at the input port of another instance of an abstraction to which it was wired by the layer above.

In all the following discussions of programming paradigms, we will be talking about wired-communications unless noted otherwise. Note that we use the word communications to cover for both events and dataflow types of programming paradigms. Another common term is message.

4.4.2. Indirect function calling

Sideways communications in ALA is always indirect. The sender never names the receiver or the function or method in that receiver. Conversely, receivers never register themselves to a sender, or to a public event. Global event names are never used. Communications always follows the wiring put in place by the layer above.

In conventional code, there is a downside associated with indirection, which is tha it becomes harder to follow the flow of execution through the modules for a given user story. That downside does not exist for ALA. In fact it is the opposite - it is easier to trace the flow of calls through the system. This is because user stories are expressed in one place cohesively. You see all the explicit wiring of a user story abstraction in one place instead of tracing it through multiple modules. Only if an abstraction it uses does something unexpected do you need to drop down inside the abstraction, and enter a different self-contained self-cohesive set of code.

When reading code inside an abstraction, it is in the very nature of abstractions that they know nothing of the outside world. They do not need or want to know where events come from or go to externally. Indirection is used so that flow can lift out of the internals of an abstraction to the more specific wiring code in a layer above.

Even synchronous function calls are always indirect. At run-time, the inside of one abstraction synchronously calls a method inside another abstraction under the direction of the wiring in the layer above. But at design-tie, it has no knowledge of what that other abstraction or method is. Whether the run-time execution model is synchronous or asynchronous, push or pull, with fan-out or fan-in, the wiring model between instances of abstractions with ports is always indirect.

Even if the communications is asynchronous, the caller does not send the event to a particular destination, nor does it give the event a global name so that receivers can register to it. Both patterns would involve a bad dependency. Instead it still only goes as far as its own output port.

Conventional code will often use an interface or the observer pattern (publish-subscribe) (or C# events) to invert a dependency. If the two modules were peers in the same layer, inverting the dependency by adding an indirection only makes the program even more difficult to follow. ALA does not need to use the dependency inversion principle or the observer pattern for peer to peer communications because there is no dependency. In other words ALA completely sidesteps the dependency inversion principle and the observer pattern for all communications between peers.

ALA generally uses dependency injection directed by explicit wiring.

Having said that ALA does not use the observer pattern (or any other form of the receiver subscribing to senders in the same layer), the observer pattern is sometimes used within a programming paradigm interface. Consider a programming paradigm where communications is needed in both directions. In the same direction as the wiring, it is usually implemented as a simple method call. The way interfaces work in our programming languages, the A end uses the interface and the B end implements it. The asymmetry is a shame. If we want a method call in the other direction, we use the observer pattern inside the interface. The publisher, the B end, implements the observer pattern. The subscriber, the A end, subscribes to it. The difference from the standard observer pattern is that the subscriber does not know the publisher. It is only subscribing to it indirectly via the interface.

If a dependency were going up from one abstraction layer to a higher one, then of course we invert the dependency. But a dependency from a more abstract abstraction to a more specific one doesn’t make sense in the first place and so this situation never occurs. The dependency inversion principle is already built into the ALA constraints, so you never need to invert dependencies later.

4.4.3. Push vs pull

If we are using standard synchronous function calls or method calls as the execution model, we have a choice between push and pull. In other words, does the sender of an event or data initiate the call, or does the receiver?

Push

Send(data);

Pull

data = Receive();

In conventional code, the decision as to whether to use push or pull is often dictated by the need to control the direction of dependencies. To change a pull to a push without reversing the dependency would require indirection or the observer pattern. Similarly, to change a push to a pull without reversing the dependency would require an indirection. So usually we use the one that allows us to use a simple function call with the dependency in the desired direction.

With ALA, most run-time communications take place within a layer, and there are no dependendencies between the abstractions involved. Instances of abstractions are wired using interfaces that represent programming paradigms:

IDataflow.cs
namespace ProgrammingParadigms
{
    interface IDataflow<T>
    {
        void Push(T data);
    }
}
IDataflow.cs
namespace ProgrammingParadigms
{
    interface IDataflowPull<T>
    {
        T Pull();
    }
}

Because there are no dependency constraints, we are free to choose between push and pull. Usually it would be for performance reasons. If the source data changes infrequently we would use push. If source data changes frequently, and the receiver only needs the value infrequently, we could choose to use pull.

An example of pull is getting data from a database. Pulling is the only choice that makes sense because any particular data is needed so infrequently. And pushing is the only sensible choice for putting data into a database. For this reason, in conventional code, the dependency is almost always towards the database. This is not the desired direction. Clean architecture reverses this dependency. But we don't want the reverse dependency either. So clean architecture will use a set of adapters that have dependencies both on the business logic interfaces and the database. ALA uses no dependencies on the business logic. In effect it will use a single adapter with dependencies on both a programming paradigm interface and the database.

It would be nice if you could choose between push and pull at wiring time. In other words, we design domain abstraction ports to handle both push or pull, and you choose push or pull when wiring instances in the application. For example a signal filter could support both push and pull. If not we might need two version of the filter. Unfortunately it increases the amount of code inside the abstractions. So we usually write abstraction ports to use either push or pull.

To allow optimal composability of abstractions, I use push ports by default so that most ports can be wired directly. Push also works quite naturally for events. It means that the initiator of an event pushes it as soon as it happens. The opposite is possible: receivers poll the source when they are interested to know if an event has occurred.

For dataflows, push means that the data 'flows' whenever it changes. This works better performance-wise if the data does not change too frequently. It works well when all data must be processed. It is ok when all the data does not need to be processed, and only the latest data is important. Push is usually more efficient than periodically polling for data.

A final factor in the preference to use push by default is that push ports can be wired for either synchronous or asynchronous execution models without changing the domain abstractions (discussed above in the section on synchronous vs asynchronous). To allow this for pull ports requires the pull end to be written for an asynchronous execution model, which can be awkward. This aspect is discussed more fully in the section on the request/response programming paradigm later.

For all the above reasons we use push ports by default, and pull ports when we have to. It is analogous to using RX (reactive extensions).

Remember we are talking about 1-way communcations. In a later section we discuss programming paradigms that use 2-way communications.

Wiring incompatible push & pull ports

It is possible to wire together instances of domain abstractions that have incompatible ports with respect to push and pull, provided the communications becomes asynchronous. A send port that uses push can be wired to a receive port that uses pull. And a send port that uses pull can be wired to a receive port that uses push. This can even be done automatically, so that the user story doing the wiring does not need to worry about it.

For the case of a push send port being wired to a pull receive port, the wiring system detects this situation and wires in an intermediary object which is an instance of a simple buffer abstraction. If the paradigm is simple events, the abstraction stores a flag for whether or not the event has been sent. When the receiver pulls the event, it clears the flag.

For the case of a pull send port being wired to a push receive port, the wiring system detects this situation and wires in an intermediary object which is an instance of a simple polling abstraction. This instance is configured with a default polling rate. It polls the sender periodically to see if the event has occurred, and then calls the receiver if it has. For dataflow, it calls the sender periodically, and then calls the receiver at least once and thereafter whenever the data changes.

A situation where a sender may want to have a pull port is a driver that gets data from the outside world. The driver doesn’t want the responsibility of controlling when the external read takes place. So it will use a pull port so it reads at a time determined by the user story. The user story will either configure the polling rate of the intermediary or configure an active object somewhere that will pull the data when needed.

Another situation to use pull is where the sender is completely passive or lazy. For example, it doesn’t want to execute a computationally expensive routine until the output is needed.

Another situation where a pull port makes sense is an abstraction with many inputs. We want the abstraction to react when a specific port receives data or an event. If we don’t want to buffer the data coming in on other inputs internally in the abstraction, we can just make them pull ports. If they need to be wired to push ports, then intermediary buffer objects would be wired in.

When a sender with a push port is wired to a receiver with a pull port using a buffer intermediary object, a situation can arise where the sender produces data faster than the receiver consumes it. In some cases this wont matter. In other cases the user story has the knowledge of how to resolve the situation. It can wire in an averager or filter abstraction. If the receiver must process all the data, and the sender produces data only in bursts, the user story can wire in a FIFO abstraction to smooth out the rate of data. The Fifo can have a reverse flow control channel that tells the source when to stop and start so the fio doesn’t overflow. If none of these solutions work, the user story can wire in a load splitter to multiple receivers.

If pull ports are quite common, we may then want 'pull' versions of some domain abstractions. For example, we may need a filter abstraction to have a pull variant.

In summary, I use push ports for domain abstractions by default. In situations where this doesn’t suit I can still use pull ports. When incompatible ports need to be wired, then a variety of intermediary objects can be wired in to solve the issues without having to change the sender or receiver abstractions.

4.4.4. Fan-in, fan-out wiring

In chapter three, we used the terms fan-in and fan-out in relation to dependencies down layers. Here the terms fan-in and fan-out are used for something completely different. Here we are talking about wiring.

Fan-out means that an output port of one instance of an abstraction is wired to many instances. Fan-in means many instances are wired to a single input port. It depends on what makes sense for each particular programming paradigm.

Fan-out implementation

Some programming paradigms support fan-out out of the box. An example is the UI programming paradigm. Many UI domain abstractions have a list port for child UI elements. The WireTo can wire directly from this port to multiple instances of other UI elements.

Most output ports of domain abstractions for other programming paradigms do not use a list for their output ports, so they do not directly support fanout. This is because they are usually wired one point. If they used a list, then the domain abstraction internal code would need to use a for loop to output to every instance in the list. We can still do fanout using an intermediary object. This intermediary object simply contains the needed for loop. An example of such an intermediary for the Dataflow programming paradigm is:

IDataFlow.cs
/// <summary>
/// DataFlowFanout has multiple uses:
/// 1) Allows fanout from an output port
/// 2) If the runtime order of fanout dataflow execution is important, DataFlowFanout instances can be chained using the Last port, making the order explicit.
/// 3) Allows an abstraction to have multiple input ports of the same type. (A C# class can implement a given type of interface only once.)
/// --------------------------------------------------------------------------------
/// Ports:
/// 1. IDataFlow<T> implemented interface: incoming data port
/// 2. List<IDataFlow<T>> fanoutList: output port that can be wired to many places
/// 3. IDataFlow<T> last: output port that will output after the fanoutList and IDataFlow_B data changed event.
/// 4. IDataFlow_B<T> implemented interface: ouput port but is wired opposite way from normal.
/// </summary>

public class DataFlowFanout<T> : IDataFlow<T>, IDataFlowPull<T>, IDataFlow_R<T>  (1) (2) (3)
// input, pull output, push output
{
    // properties
    public string InstanceName = "";

    // ports
    private List<IDataFlow<T>> fanoutList = new List<IDataFlow<T>>(); (4)
    // ouptut port that supports multiple wiring

    private IDataFlow<T> Last; (5)
    // output port that outputs after all other outputs to allow controlling order of execution through chaining instances of these connectors.


    // IDataFlow<T> implementation (input) ---------------------------------
    void IDataFlow<T>.Push(T data) (6) (7)
    {
        this.data = data; // buffer the data in case its needed by the pull output
        foreach (var f in fanoutList) f.Push(data);
        push_R?.Invoke(data);
        Last?.Push(data); (5)
    }

    // IDataFlowPull<T> implementation ---------------------------------
    private T data = default;  // used to buffer data for later pull on the output port
    T IDataFlowPull<T>.Pull() { return data; } (7)

    // IDataFlow_R<T> implementation ---------------------------------
    // make explicit so it's not visible without using the interface
    private event PushDelegate<T> push_R;
    event PushDelegate<T> IDataFlow_R<T>.Push { add { push_R += value; } remove { push_R -= value; } } (7)
}
1 IDataFlow<T> is the input port
2 IDataFlowPull is an output port (purpose discussed later)
3 IDataFlow_R is an output port (purpose discussed later)
4 Output port that’s a list to support fan-out. WireTo will wire it any number of times.
5 Output port called Last (purpose discussed later)
6 Implementation of the input port. When data arrives at the input, it outputs the data directly to all the different output ports, including to every destination in the fanout output port list.
7 All implemented interfaces are implemented explicitly in C# (not implicitly). There are two reasons for this in ALA: 1) We only want the interface’s method/event to be visible through a reference to the interface, not the public interface of the class. The public interface of the class is for the layer above to create and configure objects of the class. It generally has no need to access the ports of the class at run-time, and if it did we would want to cast to the interface to make that clear. 2) If there were two interfaces using the same method name or same event delegate, we will want to implement them separately.
Fan-out ordering

The need for fan-out in the wiring is common for many programming paradigms. The order of the synchronous calls to the different fan-out destinations may or may not be significant. Only the layer above doing the wiring knows if the order is significant. Sometimes it is sufficient for the order to be defined as the order they are wired in, or ’down’ in a diagram. The UI fanout works this way to control top to bottom or left to right UI layouts. This is a satisfactory way to define order in a UI.

For events or Dataflows, this is not considered explicit enough. Where order matters, we should use ”Activity Flow” (exactly analogous to UML activity diagrams) to control ordering. The order can be controlled by using a chain of DataFlowFanout instances. DataFlowFanout has a port called Last which facilitate this chaining. Last is invoked after all other output ports.

1 The Last port can be seen in the DataFlowFanout listing given above.

4.4.5. Work around for multiple inputs of the same type

C# and other languages don’t allow an interface to be implemented more than once. Sine we use interfaces as ports in ALA, this can be a serious limitation.

For example, consider implementing an AND gate with 4 inputs all IDataFlow<bool>.

AndGate.cs
    public class AndGate : IDataFlow<bool>, IDataFlow<bool>, IDataFlow<bool>, IDataFlow<bool>
    {
    }

Implementing IDataFlow<bool> more than once like that gives a compiler error.

It’s a valid thing to do however. I can only assume that outside of ALA, no one seems to have needed it. In fact the whole concept of ports should be part of all object oriented languages. Only then would OOP realize it’s potential for reuse. (ALA is really just OOP done right.)

If the C# language allowed the same interface to be implemented multiple times, the only syntactical difference would be that the implementations would be given names:

AndGate.cs
    // We want to do this, but can't in C#
    public class AndGate : IDataFlow<bool> Input1, IDataFlow<bool> Input2, IDataFlow<bool> Input3, IDataFlow<bool> Input4
    {
        void Input1.Push(bool data)
        {
        }
    }

You would be able to set a reference to the object’s interface using this name instead of casting to the interface type.

// We want to do this, but can't in C#
var ag = new AndGate();
IDataFlow<bool> referenceToInput1 = ag.Input1;

Java almost allows this to be done using method references. But it only works when there is one method in the interface.

We already used a work-around for this limitation of C# in the Add domain abstraction in chapter 2. In that work-around we created a Double2 type which was a simple struct containing a double. That allowed us to implement both IDataFlow<double> and IDataFlow<Double2>. But it’s not a general solution.

A more general work-around for this limitation of C# is to use interface fields instead of interface implementations and reverse the wiring.

AndGate.cs
    IDataFlow_R<bool> Input1;
    IDataFlow_R<bool> Input2;
    IDataFlow_R<bool> Input3;
    IDataFlow_R<bool> Input4;

We append an "_R" to the name of the interface to indicate it is a 'reversed wired' interface. Here is the interface:

IDataFlow.cs
    public delegate void PushDelegate<T>(T data);

    public interface IDataFlow_R<T>
    {
        event PushDelegate<T> Push(T data);
    }

The receiver registers an event handler method to the event in the interface:

AndGate.cs
    private void Input1Initialize() (1)
    {
        Input1.Push += PushHandler1;
    }

    private void PushHandler1<T>(T data) (2)
    {
        ...
    }
1 After the WireTo operator has wired a port, it looks for a method named <Portname>Intialize and calls it. This method is useful if the port’s interface has a C# event. It can be used to register a method to the event.
2 Method called for incoming data on port Input1.

To complete the workaround we need an intermediary object. Both the sender and receiver are wired to this object. It implements both IDataFlow<T> and IDataFlow_R<T>. The class for this object resides inside the IDataFlow programming paradigm abstraction:

IDataFlow.cs
public class DataFlowIntermediary<T> : IDataFlow<T>, IDataFlow_R<T> // input, output (1)
{
    void IDataFlow<T>.Push(T data) (2)
    {
        push?.Invoke(data);
    }

    // IDataFlow_R<T> implementation ---------------------------------
    private event PushDelegate<T> push; (3)
    event PushDelegate<T> IDataFlow_R<T>.Push { add { push += value; } remove { push -= value; } } (4)
}
1 Unlike normal output ports, this output port is an implemented interface.
2 When data arrives on the input port it outputs it directly to the output port.
3 The output port interface has a C# event, which needs to be implemented.
4 The interface implemented explicitly so that the event is only accessible via a reference to the interface.

The above code is also added to the DataFlowFanout class listed above. See note 3 in that listing. This allows the DataFlowFanout intermediary object to be used for the purpose of this workaround among its other uses.

A problem with this workaround is that you need to wire in the reverse direction to the flow of data. So if data is to flow from A to B, we would need to write:

var intermediary = new DataFlowFanout();
new A().WireTo(intermediary);
new B().WireTo(intermediary);

This is unintuitive at the wiring level.

We would prefer to write like we do normally:

new A().WireTo(new(B));

We can write an override of WireTo in the programming paradigm abstraction and register it with the Foundation WireTo.

The override WireTo would look for a field interface in A that matches a field interface in B by name with a _R suffix.

TBD write the override WireTo.

4.4.6. Wiring arbitrary execution models

To accomplish wiring, the application, feature or user story abstraction’s code makes calls to the WireTo method, passing in the two object/ports to be wired. The WireTo method, by default, wires the two objects by assigning the second object to a private field in the first object, provided the interface matches. This default behaviour sets up a direct connection between two communicating objects.

For arbitrary execution models, we don’t always want direct connection between connected objects. We may want an intermediate object to be automatically wired in, or other special behaviours. For example if the two objects being wired are in different locations, we will want to automatically wire in the necessary middleware intermediary objects. Intermediary objects are commonly needed in ALA. We have previously used them for several different purposes, such as asynchronous communications, pull communications, etc.

In the asynchronous programming code earlier in this chapter, we used an override of the WireTo method that had an extra parameter. But what if there is not extra parameter. Then the WireTo method that resides in the ALA foundation layer is the one that will be called. It can’t know anything about programming paradigms or execution models in higher layers. But it can know in an abstract way about allowing itself to be overridden.

The WireTo method in the foundation layer can support a list of registered override functions. It calls every override function in the list. If all return false, then it does its default behaviour.

The foundation WireTo can first do the reflection work. It can create lists of potential field and implemented interface ports in both the A and B objects. Then it can pass these lists to the override functions.

TBD Modify WireTo to support run-time overridding. Use it to implement a null decorator intermediary on the synchronous programming paradigm. Then use it to implement wiring of a push port to a pull port and a synchronousmiddleware for wiring objects in different locations.

4.4.7. Diamond pattern glitches

Consider a wiring topology of an application in which wiring diverges from a single instance of an abstraction, and then converges to a single instance of an abstraction. The two paths will be executed at sightly different times. So one input of the end instance will get data from the common source before the other. During the time between the two, the inputs may be in an invalid state. This is what we mean by a glitch.

Glitches also happen in conventional code where they are a cross cutting concern. They also even happen in electronic circuits.

In ALA, they are a concern within a single abstraction, either the application abstraction, or a feature or user story abstraction. This is where the diamond topology of the wiring is apparent and the problem can be easily understood.

Abstractions may have a minor inputs which it expects to get data first and major input that triggers operation. In such a scenario, the application can control the order of execution in the wiring so that the major input gets its data last.

One solution is to provide an trigger event port on abstractions that have multiple inputs. The application must trigger the port once all inputs are valid.

It is a future topic of research to automatically detect glitches on abstractions with multiple inputs, and potentially to automatically resolve diamond wiring glitches.

4.4.8. Circular wiring

In ALA, it is no problem to have circular data paths. Note that by circular, we are referring to wiring inside an abstraction, not dependencies between layers. Circular wiring naturally occurs in feedback systems, just as it does in electronics. It is nice to be able to represent such feedback systems directly in the wiring.

In conventional code, circular data paths may need a pull or an indirection to avoid circular dependencies. ALA does not have this problem. Circular wiring is as natural as it is in electronics.

A programming paradigm’s execution model needs to consider circular wiring. For example, circular wiring using all synchronous programming paradigm will result in an infinite loop at run-time, just as it does in conventional code. It easy to solve however. It can be as simple as an abstraction instance placed in the circuit that does an asynchronous call, or an abstraction instance that does a delay. This effectively causes a return to the main loop where the circuit can be called again. The main loop can process higher priority tasks first. It is no problem for such a circuit to repeat forever.

Alternatively, we can implement programming paradigms utilizing existing rigorous execution models, such as the discrete time execution models used in function blocks or clocked-synchronous execution models. The continuous time execution model underlying Functional Reactive Programming will automatically flags such loops.

While circular data loops can occur in conventional code as well (recursion), they are more likely in ALA because ALA is likely to have dataflow abstractions which can easily be wired as a circuit. However, in ALA it is usually explicit and clear in the wiring diagram or code. The Calculator project in chapter two contained Dataflow loop circuits.

4.4.9. Synchronous vs asynchronous

Although we already did simple coding examples for synchronous and asynchronous execution models at the start of this chapter, the design choice between synchronous and asynchronous needs deeper considerations.

Synchronous communication is like asking someone a question. You stop your life and wait, albeit for a brief time. You don’t resume your life until you get the answer or a nod. Asynchronous communication is like sending an e-mail.

Synchronous means that the calling code resumes execution after the callee has finished processing the communication.

There are reasons why you may want to use synchronous communications. The communication may cause a side effect, which we want to be sure is completed before continuing execution.

If the receiver will take a long time to execute, which can be for many reasons such as a long running algorithm, receiver not ready, external IO, a deliberate delay, etc, then a synchronous call will do what is referred to as blocking. Blocking means the thread will stop and wait. If the blocked thread needs to do something else in the meantime, this blocking will be a problem in one way or another.

In ALA we prefer single threaded solutions. Multi-threaded programming should only be used for performance reasons e.g. meeting a challenging latency or throughput requirement. A single threaded system will use run to completion, so in that respect is commonly referred to as cooperative. Being cooperative sounds like it doesn’t comply with ALA’s zero coupling. To some extent this is true, but the requirement to keep all routines short (non-blocking) can be thought of as an abstract requirement from a lower layer rather than relative coupling between domain abstractions. All higher abstractions need to know about this. Usually if nothing in an application blocks, the latencies needed for an application to respond to a human in reasonable time (which is the most common soft deadline requirement) will be acceptable. Using a single thread when things take time, or things need to happen in real time requires asynchronous communications.

Asynchronous means that the sender instance’s call returns before the callee has finished processing the communication. It will usually be before the callee even receives the communication.

Asynchronous calls can be implemented in several different ways. What they all have in common is that the caller makes a synchronous call that starts the communication or starts the callee’s execution in some way. The caller will then resume executing the next line of code pretty much immediately.

In ALA, as with the synchronous case, the caller does not know where it is sending the communication and the callee does not know where it came from. Where synchronous and asynchronous communications differ is only in when the call returns.

Note that here we are discussing the fundamental case of one way communication. We will consider two way communication programming paradigms later.

With one way communication, we have the option to decide at wiring time whether to use synchronous or asynchronous, provided the sender doesn’t care whether it resumes processing before or after the receiver gets or processes the message.

Some common ways of implementing asynchronous calls are:

  1. The sender can make a synchronous call on the receiver, which just initiates an on-going activity and returns. It can be starting I/O, starting a timer, changing a state, etc.

  2. The sender can make a synchronous call that just sets a flag, which is later polled by the main loop which then calls the receiver code.

    In ALA this is easily implemented using an intermediary object that is wired between the caller and callee. See "Wiring arbitrary execution models" below. The intermediate object’s class resides inside the programming paradigm abstraction. It contains the flag. Within the programming paradigm abstraction, all the intermediary objects are put on a list. The main loop simply polls every object on the list. When the poll method in the object sees that the flag is set, it clears it and calls the callee.

  3. The sender can make a synchronous call which is turned into an object which goes into a queue. The main loop takes these objects from the queue and calls the receiver code. In terms of run-time execution this is the same as the simple version of the reactor pattern or simply 'event loop'. Example code for this method was given above.

  4. The sender can make a synchronous call which puts an object into the receiver’s queue on a different thread, process or processor.

  5. If the language has async/await, the sender can call a method marked with the async keyword (without using await itself). The call returns immediately the first time the receiver awaits.

Other mechanisms are possible. Note that all of these mechanisms describe how the sender’s synchronous call returns before the receiver completes.

Remember that in all these implementation examples, we are talking about fundamental one-way communication - an event or pushing some data. Two-way communications gets more complicated, and is discussed below.

All the asynchronous programming paradigm execution models discussed above use pushing. Analogous pulling asynchronous communications are also possible. For an asynchronous pull, the receiver makes a synchronous call which returns a previously calculated result without waiting for the sender to calculate it. It returns the last result available from the sender, or a value from a FIFO, etc. The sender will calculate new values in its own time.

Asynchronous communications has inherent concurrency. This simply means that tasks of different features or user stories or channels or whatever can be executing in an interleaved fashion. That’s why we are using it. The concurrency is at a courser grained level compared with pre-emptive multitasking. There can still be a need to lock any resources that can be in an invalid state for a time, or to think in terms of transactions.

4.4.10. Wiring incompatible synchronous/asynchronous ports

Generally ALA can use both asynchronous and synchronous execution models in its programming paradigms. It does not have rules for when to use one or the other. The design choices remain more or less the same as in non-ALA applications according to real-time factors discussed above.

However, ALA is all about abstractions and zero coupling at design-time. It would be good if the abstraction didn’t need to know whether the external communications beyond their ports is going to be asynchronous or synchronous. We would like to decide that when we wire instances of them up. It is therefore desirable that domain abstraction ports that generate events and ones that listen to events can be wired for either synchronous or asynchronous execution. That way, for example, they can be wired synchronously by default for best efficiency, but asynchronously if they are in different locations, or if the recipient will take a long time.

One directional case

A sender port that is strictly one way can be coded to be synchronous and still be used asynchronously. The receiver can be either synchronous if the operation is quick, or asynchronous if the operation takes time. Either way the call returns quickly so that the sender is never blocked.

If it is strictly one way, we are not interested in the function call return value or its return timing. By strictly it means that the sender is zero-coupled with the reactions to the communication. It doesn’t care if it executes before our own next line of code or after.

In the example code at the beginning of this chapter, the domain abstractions did not change when we did the asynchronous version. But the order of output of system did. One was "123", and the other was "132". The application has knowledge of this order, but not the domain abstractions themselves.

If a certain domain abstraction needs to make an assumption that the next line of code executes after the call must execute after the effects of the call, then that abstraction knows something about the outside world. It isno longer an abstraction. It is probably orchestrating a side effect of some kind. It would need to be written differently and not use one-way communWhatever that orchestration is, it needs to be factored out into a higher layer where it will become cohesive code.

Two-directional case

The two-directional, synchronous, case is familiar to us because it can be implemented with the common and elegant function call mechanism of the CPU.

Although a 2-way communication port can be implemented as a function call in the execution sense, in ALA it is always indirect. The function is always in an interface. The requester always has a reference to the reresponder, cast as the interface. The reference is always determined and set by the wiring in a higher layer. The interface itself always comes from a lower layer and is always more abstract, representing the request/response programming paradigm.

The subroutine call instruction can be thought of in this way: it passes both the request message and the CPU resource to the responder, and receives both the response message and the CPU resource back to the requester when done.

This allows the lines of code that are to be executed following the request/response completion to be written immediately following the call (direct style). We are so used to this that we take it for granted. But its actually a clever and elegant mechanism provided by the subroutine call instruction. Because of the convenience of this mechanism, the synchronous function call dominates as the default way to implement request/response in conventional code.

But the synchronous function call causes problems as soon as the function takes real time. For example, the responder may need to wait for input/output. Or, it may be in a different location or processor. Or it may have to delay. It will block the thread. Unlike the more fundamental one-way cases discussed earlier, if we want to use the CPU to do other work in the meantime during a real-time 2-way communication, life gets tricky in one way or another.

Unlike the one-direction case, a port cannot support both synchronous and asynchronous. Here are two example interfaces for synchronous and asynchronous respectively. For the asynchronous one, we have used callbacks because they are easy to understand, but there are other better mechanisms as will be discussed shortly.

IRequestResponse.cs
namespace ProgrammingParadigms
{
    interface IRequestResponse<T,R>
    {
        R Request(T data);
    }
}
IRequestResponseAsync.cs
namespace ProgrammingParadigms
{
    public delegate void CallbackDelegate<R>(R data);

    interface IRequestResponseAsync<T,R>
    {
        R Request(T data, CallbackDelegate callback); (1)
    }
}
1 For the asyncronous version of the interface, the request passes an additional parameter, the function to be called on completion.

Given that for 2-way communications, the interfaces for synchronous and asynchronous are different, you cannot directly wire a synchronous port to an asynchronous one or vice versa.

Ideally we would like to be able to wire instances of domain abstractions together without regard to whether the ports are synchronous or asynchronous. And we would like to be able to wire synchronous ports with asynchronous wiring inbetween when we want to (for when they are on different processors.)

The only way to get this type of compatibility is for all senders to be asynchronous by nature. Asynchronous senders can work with either synchronous or asynchronous destinations. They can also work with asynchronous wiring (or synchronous wiring, provided the destination is synchronous).

Unfortunately, making senders asynchronous by nature means not using the function call mechanism.

A domain abstraction with an asynchronous output port needs to have a callback function:

Sender.cs
namespace DomainAbstractions
{
    public class Sender
    {
        private IRequestResponseAsync<string,string> output;

        public void DoSomething()
        {
            output.Request("message", Callback);
        }

        public void Callback(string returnMessage)
        {
            Console.WriteLine(returnMessage);
            // next operation
        }
    }
}

Of course, such a sender port can be wired directly to an instance of any domain abstraction implementing IRequestResponseAsync.

But the sender can also be wired to any domain abstraction implementing IRequestResponse (via a small intermediary object). The sender doesn’t care whether the callback is called back asynchronously or synchronously in the outgoing output.Request() call. Similarly if we had used a Task or Promise or async/await, it doesn’t care if the Task or Promise already in the complete state when it is returned.

Here is the intermediary object that needs to used when wiring an asynchronous port to a synchronous port:

IRequestResponse.cs
public class RequestResponseAsyncToSyncIntermediary<T,R> : IRequestResponseAsync<T>, // input
{
    private IRequestResponse<T,R> output;

    void IRequestResponseAsync<T,R>.Request(T data, CallbackDelegate<R> callback)
    {
        R returnValue = output.Request(data);
        callback(returnValue);
    }
}

We can’t wire a sender with a synchronous port to an asynchronous destination. If we did, the call would return immediately without a result.

In summary, to have domain abstractions with two-way ports zero-coupled with respect to synchronous/asynchronous communications, the senders need to be asynchronous by nature.

Receivers with asynchronous ports can behave synchronously, but not the other way around.

If instances of any two abstractions are connected within the same processor they can both behave synchronously from a performance point of view. If instances are on different processors, asynchronous middleware can be easily wired in.

Making sender 2-way ports asynchronous

Unfortunately, if you make all your domain abstractions that have 2-way requester ports asynchronous so that they are compatible with either asynchronous or synchronous responders, they must be written in the 'coding style' of asynchronous. While never impossible, this can be seriously awkward.

Mechanisms for asynchronous (2-way) calls include

  • using two separate one-directional calls, one in each direction (This is harder in conventional code, because you need to avoid circular dependencies. It is easy in ALA but requires two wirings. Intuitively a bi-directional port should need only one wiring.)

  • callbacks

  • coroutines or protothreads using Duff’s device

  • a promise or task object that will later have the result

  • continuations

  • async/await

  • a state machine (a complete event is sent back to the machine)

We will cover most of these below, but first we need to know about direct programming style.

Direct programming style

The problem with some of the mechanisms for asynchronous coding is that they don’t allow direct programming style. Direct style is when you can do successive operations with successive statements in a with simple syntax. For example, consider the following direct style synchronous code (which will block the thread):

    RobotForward(7);
    Delay(1000);
    RobotTurnRight(90);

Using callbacks, it gets unwieldy:

    void Step1() {RobotForward(7, Step2);}
    void Step2() {Delay(1000, Step3);}
    void Step3() {RobotTurnRight(90, null);}

And with anonymous callbacks, even more unwieldy because of increasing indenting at each step:

    RobotForward(7,
        ()=>Delay(1000,
            ()=>RobotTurnRight(90)
        )
    );

That’s why some of the mechanisms listed above go to great lengths to allow direct programming style.

But even if you settle for callbacks or a state machine, at least it only affects code that is written inside a single domain abstraction where it is contained.

Prescriptive and reactive styles

Callbacks or state machines have the advantage of not committing to prescriptive style. Prescriptive style means that we know what we expect to happen next. That’s why we want to use direct style so we can put what we expect to do next in the following statement.

But if something different may happen, then we want reactive style. We want to react to whatever events may happen in the meantime. In general we want to retain the flexibility to be reactive because during maintenance we learn about less likely scenarios.

Reactive style means we can easily add handling of unforeseen events to the code. There will almost always be a need to handle timeouts in abstractions because we don’t know to whom the ports will be wired. If they are wired asynchronously across an unreliable network, or to an external device, a timeout will likely be needed. Or, if something arrives on a different port while we are waiting for an asynchronous function, we will want to handle that. And we may want to abort the asynchronous communication. Callbacks and state machine handle these kinds of situations easily and naturally because the CPU is not stuck at one point in the code.

What we really want is the direct style of a multithreaded solution, and the reactive style of callbacks or state machines. They are not necessarily mutually exclusive.

Asynchronous execution models

What asynchronous execution models all have in common is they use a synchronous call for the forward direction that always returns immediately, and possibly without a result. It must return all the way back to the main loop so that the thread can do other work. The response comes back later in some other way.

There are several ways to handle the response:

async/await

If you have async/await available in your language, it is by far the best way to write asynchronous style code:

    await RobotForward(7);
    await Delay(1000);
    await RobotTurnRight(90);

If the Task object returned by any of the function calls is not complete, the CPU returns (from the containing function) at that point so it can do other things in the meantime. When the task is complete, the CPU magically returns to the point of the await to resume execution.

await gives you the benefits of direct style, needing only the addition of the keyword await on every asynchronous call (and the addition of the async keyword on the containing function).

await also gives you the benefits of reactive style. While the code waits for the response to an asynchronous function call, other code in the abstraction can still react to other incoming or internal events. If the waiting asynchronous function call needs to be cancelled, this can be done using a cancellation token. The await will release and you can use exceptions to change the course of the prescriptive part of the code.

async/await keywords must be put on every function in the call stack back to main. Apart from that, the direct style code looks syntactically the same as a synchronous function calls. But under the covers it is not - the compiler transforms the code into a state machine.

When an asynchronous call (using the await keyword) executes synchronously at the responder end, the task object that is returned by the call has a completed status and a return value already, and so awaiting on it simply causes execution to continue immediately with the next statement as if it was a synchronous call.

When an asynchronous call executes asynchronously at the responder end, the task object that is returned does not have a return value and a completed status. The requester async function returns immediately at the point of the await without executing the statements following the await. When the task object status changes to complete, the statements following the await then magically resume with the functions’s context all restored.

The code following the await is actually compiled as callback function, but the syntax is such that it looks like direct style. It’s the best of both worlds, however its confusing when you are new to it, because functions marked with async do not behave like normal functions.

Async/await is the best addition to programming languages since objects.

State machine

Consider if the requester is better written as a state machine. If the requester is mostly reacting to events anyway, it might be best viewed as a state machine. The requester sends an event out the port and puts itself in a state for handling a response event. This solution is more flexible because it can also handle any other events that might happen in the meantime, or even instead of the response, such as a timeout. The response comes back on the port as an event for the state machine.

If the requester is not so much reacting to events but prescribing the order that things happen, then a state machine will be awkward, especially if the requesting function is nested in loops of other functions. In this case we want the direct style (that looks syntactically like a synchronous function call). Direct coding style allows the code that follows the request call to go immediately after it rather than in a different function.

Coroutines or protothreads.

In C code there are mechanisms such as coroutines and protothreads that use macros that make the code style direct. Under the covers the macros make switch statements that work as a state machine.

Callbacks

The requester can pass a callback function reference to the responder. When the responder has processed the communication it calls the callback function.

This can be a workable, albeit not entirely elegant, solution. The function containing the call to the asynchronous port is split up into two smaller functions, which is not great if direct style code would express the solution better. Also local variables or parameters that would have been in the original function now end up as globals to be shared by the multiple functions. You can’t put callback functions in a loop or another statement or inside another function, so such structures have to be split up also, and effectively made to work as a state machine.

The request call will be at the very end of the function that contains it. This is so that it returns to the main loop when the request call immediately returns (tail call). The callback function immediately follows this function so that the flow is still relatively clear.

Finally, the callback function could be passed by the request call as an anonymous function. However this involves much nesting of brackets and indenting for successive callback functions. This is called triangle hell. If there is more than one such request/response in a row, these nestings will quickly become unreadable. I find named functions following each other is clearer.

Tasks, Futures, Promises

Without going into the detailed differences between futures and promises (the terms get mixed up anyway), this approach is more modern than callback functions.

The requester makes a synchronous call on the receiver which immediately returns with an object known as a future. The future object will have the result in it in the future. You can save a reference to the object, do something else in the meantime, and check it periodically.

The future can contain a continuation function, which is essentially just our previous callback function idea.

The future may contain a continuation function reference which gets called when the result is ready.

Pairs of ports

Finally, request/response could be implemented asynchronously by having pairs of ports on each of the requester and responder and having two wirings, one to carry the request and one to carry the response. Both can be synchronous pushes in themselves, but the overall wiring is request/response.

Doing function calls in both directions is usually avoided in conventional programming because it would involve circular dependencies. But in ALA its just wiring, so it is quite feasible.

Sometimes, it turns out that what would be request/response function calls in conventional code are really best written without request/response at all.

Let’s have a look at an example:

void main()
{
    while (true)
    {
        data = Scale();
        Display(data);
        delay(1000);
    }
}


float scale()
{
    data = Filter;
    return = data*0.55 + 23.2;
}


float Filter()
{
    static float state = 0;
    data = Adc(channel=2);
    state = data*0.12 + state*0.88;
    return state;
}


float Adc(int channel)
{
    ...
}


void Display(float data)
{
    ...
}

The function main requests data from the adc at intervals via two functions which processes the data during the return trip. Main then pushes it to a display.

The functions main, scale, filter and adc are chained using request/response implemented as function calls.

(The scale and filter functions being chained may look strange to some because they are so obviously abstractions. But add a few more application specific details to them and I have seen plenty of conventional code that chains function or method calls through multiple modules or classes like this.)

The main function is not abstract. Not like the ideas of adc conversion, filtering, scaling or displaying. It’s code that’s deciding when to read the ADC and then passing the processed result to the display. In other words, it’s specific to the application. Also, in the chain of function calls, the chaining itself is specific to the application.

So let’s get closer to ALA by pulling out the application specific bits into an abstraction in the application layer.

void main()
{
    while (true)
    {
        data = Adc();
        data = Filter(data);
        data = Scale(data);
        Display(data);
        delay(1000);
    }
}

It’s almost ALA compliant, but the application is handling data a lot at run-time. Handling data is not an application specific detail. It’s a very common implementation detail, so its done at the wrong abstraction level. The passing of data from abstraction to abstraction at run-time is the idea of dataflow, and it’s quite abstract so it should go into a layer below the domain abstractions.

Also the loop is a common implementation detail that doesn’t belong in the application abstraction. We wnt the application to just be a composition of the 'ideas' of adc, filter, scale, display and clock. Something more like this:

void main()
{
    new Clock(1000)
    .WireTo(new(Adc(channel=2))
    .WireTo(new Filter(0.88))
    .WireTo(new Scale(0.55, 23.2))
    .WireTo(new Display());
}

That’s our target code. Let’s see how to get there from the while loop code.

First let’s switch to diagram form. Lets use the request/response programming paradigm used by the original code so that it closely mimics the function calling execution model of the main loop version.

diagram clock adc filter scale display1

We’ve put the main loop into a new domain abstraction called Main. It pulls data from its request/response port and pushes it out on its output port at regular intervals. The execution model is working the same way as the conventional code.

The Main domain abstraction is not a great abstraction because it assumes all possible applications are just going to pump data.

Lets fix that:

diagram clock adc filter scale display2

We have introduced a new domain abstraction called a 'Pump' that pulls data from a request/response port and then pushes it out of an output port. The pump has an input event port to tell it when to do it. Pump will also be a temporary abstraction, but lets run with it for now.

Note that the arrow between the clock and the pump is using the event programming paradigm. The arrows between the Pump, the Scale, the Filter and the Adc are the request/response programming paradigm. The arrow between Pump and Display is Dataflow (which pushes data).

Because the Adc takes real time, the pump, scaler, filter and ADC must all now have asynchronous request/response ports. So they must all be written in asynchronous style. But, if we look at the diagram, we can wonder if we really need to use request/response. Is it a left over artefact of the conventional code?

We can see that we can lose some of the request/response ports simply by moving the Pump.

diagram clock adc filter scale display3

Now the Filter and Scale abstraction uses simple push ports.

Now let’s take this one step further. The Adc abstraction is more versatile if the event that starts it does not have to come from the same place where the output goes. In other words, the Adc would be a better abstraction if it had a pair of ports, an event input called start, and a push dataflow port called output.

diagram clock adc filter scale display4

Now we don’t need the Pump. It was there just to make the request/response execution model work and wasn’t providing anything useful. The application just wires the clocked event source directly to the ADC.

Now we have exactly what we wanted when we wrote that earlier code that just composed ideas. The composition now seems natural and elegant. The idea of splitting a request/response port into two separate ports has actually lead to better abstractions and a better solution overall.

It also makes sense to split a request/response port when the requesting end is already a state machine. Waiting for the response becomes just another state, and the response becomes just another event wired back to the state machine machine.

For receiving the response, the requester has an input port and a function that implements the interface of that port. If that function makes a further request, the stack will have two returns pending, one for the original request and one for the 2nd request. Some systems use 'tail optimization' for this situation to stop the stack accumulating calls. Because request calls occur at the end of a function, tail optimisation converts the instruction from a call to a jump.

The request/response pattern is common so we prefer to implement it as a single port on each of the requester and responder with a single wiring.

In summary, all these techniques allow us to write asynchronous requesters, which allows us to avoid using multithreading.

However there is still danger associated with these asynchronous mechinams compared with synchronous function calls. The CPU is freed up to do other work while a request that takes real time is being processed. This is still concurrency, its just not fine grained concurrency that multithreading has. During concurrency, a shared state somewhere can be changed when you don’t expect it. For example, if the requester is performing a transaction such as the canonical debit one account and credit another, the requester that was written using normal synchronous calls is safe without locking the two accounts. This is because synchronous calls effectively lock everything by hogging the CPU resource until they complete. The asynchronous version has to be worried about what else might happen between two successive request/response calls. We call this type of non-splitable operation a transaction. Transactions still need explcit locking of resources that need to be kept in an internally consistent state. This needs to happen at the user story level becasue it is the user story that understands transactions. We can deal with this type of locking by using the "Arbitration programming paradigm", which is described later.

Multithreading

The conventional solution for function calls that take real time is to use multithreading.

At first this seems elegant as it keeps the same direct style syntax used for function calls that are non-blocking. This has the advantage that the code in the requester is written in almost the same way whether or not the instances it will be wired to will block. There is useful design-time decoupling resulting from that - the requester does not have to know what it will be wired to. It also appears to abstract concurrency, allowing other tasks to execute while the thread is blocked.

In ALA, every instance of an abstraction containing a prescriptive routine that could potentially block would need its own thread. But unfortunately threads do not remain confined within abstractions. They have far reaching effects as they call out into other abstractions. And abstractions that need to do work while waiting on a blocking call will themselves need multiple threads.

Because instances of abstractions do not know to whom they will be wired, they would need to assume that incoming function calls could be on a different thread. This would cause the multithreading model to have collaborative coupling between abstractions to have sufficient locking without causing deadlocks. This is the same problem for conventional classes as well, but its worse in ALA because abstraction internals must have zero design-time coupling with one another. They cannot collaborate on locking.

If a single thread is capable of doing all the work, I don’t recommend multithreading for solving the problem of function calls that take real time, even if unwieldy callbacks are the only alternative. Using callbacks to implement asynchronous ports is at least contained inside an abstraction.

Using non-preemptive multi-threading avoids race condition and deadlock problems by not requiring locks. All non-blocking sections of routines will run uninterrupted.

Once a multithreading is available, it tends to be the solution for every concurrency problem. That tends to commit code to prescriptive style even when a state machine would be better. (Prescriptive style as opposed to reactive style was discussed above.)

If we want to abort a blocked synchronous function call, (in the same way that we can abort an await with a CompletionToken,) we could have a second method in the programming paradigm interface called Cancel(). When the interface is implemented, the Cancel function (which has to run on a different thread) must release the block at the point where it is blocked, and cause it to return. It would return with a cancelled result so that the calling thread can follow a different flow. I have not tried this programming paradigm as yet.

Of course multithreading is still a solution for throughput types of performance issues. Multithreading is discussed further in a later section.

4.4.11. Priorities

Synchronous communications are deterministic. They prescribe the order in which everything happens. Furthermore, they effectively put a system wide lock on everything until the entire function calling tree completes. Nothing else can happen anywhere until it finishes.

Asynchronous communications, on the other hand, is inherently less deterministic. The non-determinism is made necessary by the external system, things like: real-time I/O, external networks, or by the need to improve performance.

During asynchronous communications, the functions can be executed in the order in which they are scheduled (using a simple queue) by default. This is what we did in the sample code at the start of this chapter. If this ordering scheme is used, then from the point of view of an asynchronous call tree, the natural order of execution is different from the synchronous function call tree. A synchronous function call tree will be depth first, whereas an asynchronous function call tree will be width first.

During the execution of a call tree, other call trees may be executing in parallel. This does not mean parallel in the fine grained sense of multithreading. It means parallel in the course grained sense that between the execution of asynchronous functions, other functions of other call trees may execute.

One consequence of asynchronous communications is that if any resource, including any object, is left in an invalid state between the running of two asynchronous functions, it must be locked. The need for locking is much less common than in a multithreaded situation. How locking can be accomplished without introducing coupling into the abstractions by using an arbitration programming paradigm is discussed later. Locking will change the order that functions execute.

The order of execution of asynchronous functions, can also be explicitly changed using priorities. Priorities are usually used to explicitly improve performance by doing more urgent things first.

Because the order of execution is outside the control of the abstractions involved, domain abstractions should not care about when it’s one-way asynchronous communications are executed. If the priority system were to reverse the order of execution of every asynchronous function in the system, a domain abstraction results should be the same (except for its performance). If the order does matter, the order needs to be explicit in some way. For example, a domain abstraction could use a 2-way communications port so that it gets a communication back when something is complete and it can move onto the next step. Another example is to use an activity programming paradigm (UML activity diagram). Abstractions have start input port and a finished output port. The application wires instances of them in a sequence.

Priorities are generally a system wide concern, so the application abstraction (or feature or user story abstractions) are the only ones that have the knowledge to know how to set priorities.

In conventional modular systems, priorities are usually a cross-cutting concern, but in ALA they are cohesive with the wiring code, which is already in one place for a given feature or user story. The application may need to. prioritize the features and user stories.

TBD Show example implementation code for priorities. Add an optional priority parameter to the WireTo of the asynchronous programming paradigm abstraction. The appplication can use priority numbers such as 0,1,2. We need a default priority so that WireTo can be called without specifying a priority. The application would configure the default, for example to 1.

The final requirement is that applications can still use the asynchronous programming paradigm without using priorities at all. There would be a default default priority level of 0.

A priority abstraction could be created in the domain abstractions layer. It would contain a dictionary for priority levels. You would not use an enum for priority level in this abstractions because the levels are specific to an application. The application configures the dictionary with level names such as Low, Middle, High, that associate with numeric priorities.

Also, we need to consider if domain abstractions may ever want to use priorities internally. If so we need to do it in such a way that they do not have a dependency on a priority abstraction because if they are used without priorities, we don’t want to have to include the priority abstraction.

4.4.12. Busy resources

When a resource that takes is used asynchronously, more than one user may try to use it at the same time. For example a transaction on a database may involve several asynchronous function calls, and have multiple users. It would need to be locked for the duration of the transaction. Or an ADC converter that takes time to do a conversion may be used by multiple users. It would be busy to new requests while it is performing a conversion. If the resource is busy, the communication to the resource will need to be queued until it is ready.

The reactor pattern can handle this situation. It can check if the receiver is busy before giving it the communication. The dispatcher wont remove asynchronous function calls from the queue unless the destination resource is ready for it.

If a simple event loop is used, a solution to this problem, is an intermediary object that is wired in front of the resource. It keeps its own queue of event objects. When the resource signals that it is free, it takes the first event from the queue and sends it to the resource via the main event loop. That way only one event at a time can be in the event loop’s queue.

4.5. Example Programming paradigms

In the previous section of this chapter, we discussed many aspects of execution models in general. Many were applicable to both event-driven and 1-way dataflow programming paradigms.

Nest we will look at some particular programming paradigms and see how their execution models might work. It is not an exhaustive list. There are no doubt many other possibilities waiting to be invented that have new meanings for the composition of abstractions, and allow succinct expression of requirements.

4.6. Request/response

A common type of 2-way communication is request/response. This programming paradigm is fundamentally an orchestration of two one-way messages, but we are used to thinking of it as a fundamental communication pattern in its own right. That’s because it’s implemented so easily with a common function call. Earlier in the chapter we observed that if requesters were asynchronous by nature, they would have wiring compatibility with either synchronous or asynchronous receivers. And we discussed ways of writing requesters to be asynchronous.

A request carries two types of implicit information. Firstly, since they are wired point to point, a request is implicitly a command. It doesn’t need any command name or any explicit data specifying a command. Secondly a request and a response implicitly carry timing information. The time that they occur is in itself information.

Examples of request/response:

  • The requester needs to know when it’s completed (before it continues with the next line of code).

  • The requester needs to know a success or failure status of a command.

  • The requester needs to request latest information (pull) (e.g. from an I/O port).

  • The requester needs to request lazy information (information not calculated until its needed).

  • The requester needs to request specific information e.g from a database.

4.6.1. Wiring incompatible request/response ports

As discussed in earlier sections, synchronous and asynchronous 2-way communications have different advantages. There is a principle, GALS, that suggests that we use synchronous locally (within a processor) and asynchronous globally across processors. I think this is too simplistic. There are reasons other than cross-processor communications that cause certain communications to take real time, such as IO or delays. These communications should be asynchronous, and then all the ones that might be wired to them need to be asynchronous as well. Nevertheless there may be some communications in the average application which needs the advantages of synchronous communications.

The request/response ports of domain abstractions may end up a mixture of synchronous and asynchronous.

If the requester is asynchronous and the responder is synchronous, there is little problem in connecting them using an intermediary object. When the requester calls the intermediary, the intermediary in turn calls the responder which returns immediately. The intermediary then places the result in the task or future object, or calls the requester back if it uses a callback.

If the requester is synchronous and the responder is asynchronous, it would be possible to create an intermediary adapter, but it will block the requester’s thread, which probably isn’t what we want. The requester would need its own thread (or its requester), which, as I said earlier I don’t recommend as the way to solve this problem. So they are essentially incompatible. The requester code would need to change to asynchronous, as described by one of the methods above.

4.7. Event-driven programming paradigm

We now return to the 'Event driven' programming paradigm. At the beginning of this chapter we showed both synchronous and asynchronous code examples of this paradigm, both of which used the IEvent interface.

'Event' is an overloaded term in software engineering. Sometimes it means asynchronous, as in using an event loop. Sometimes it means indirect, as in C# events. Sometimes it means both. Earlier in this chapter we clarified these two independent notions. We discussed that in ALA, communications between abstractions within a layer are always indirect and explicit. We also discussed that they may be either synchronous or asynchronous. And we discussed 1-way and 2-way communications.

The interpretation of event-driven that I use is asynchronous and 1-way. Of course it’s always indirect and explicit in ALA.

Note that this interpretation is different from the C# language version of events. C# events are synchronous (they get delivered and processed before the function returns). C# events also directly support fanout. C# receivers are usually registered by the receiver itself (observer or publish/subscribe pattern). In ALA of course, events must be wired by a layer above.

In my interpretation of the event-driven programming paradigm, output ports can only be wired point to point. You would use a fanout intermediary object to achieve wiring an event to multiple destinations.

Even though my interpretation of even-driven is asynchronous, the output ports use a function or method call. This is fine because they are 1-way communications. The function gets the event on its way and returns immediately. The return itself carries no information.

When an event is taken from the event queue and dispatched to the receiver, we call it a task. The task is just the execution of a function or method, (which is different from a C# task object). A task must always runs to completion quickly. No task should take real time to execute (spin loop, or block).

4.7.1. Events with parameters

Another section of this chapter discusses the dataflow programming paradigm. Dataflow can be similar to event-driven with a parameter when it pushes data. However Dataflow has variants where can be synchronous, can be pull rather than push, and can send a whole table of data in batches. For this reason Dataflow and event-driven are considered different programming paradigms.

4.7.2. Reactive vs prescriptive programming

Event-driven programming is a reactive style in that it contrasts with the prescriptive or orchestrated style of the imperative or activity programming paradigms. In event-driven, the system is idle until something happens, and then things react to it, possibly changing some state, possibly generating more events, completion events, or timeout events. Event driven systems like to use interrupt routines to get events from the outside into the system. The interrupt routine puts the event directly into the main loop event queue.

In a reactive system, we don’t know what will happen next, in either the outside world or what code will execute next. It is less deterministic. Reacting to an event often changes some stored state. This state may change the way we will react to subsequent events. In other words, event-driven often goes hand in hand with state machines.

Event-driven programming is generally not thought of as a request/response type of paradigm. There can be a response, but it would be thought of as a completely separate message that needs its own point to point wiring. We don’t need synchronous communications because there is no response associated with an event (in the same wiring).

ALA is polyglot with respect to programming paradigms, so there is no reason to try to make an entire system either event-driven or prescriptive. Both can be mixed for maximum expressibility of the requirements.

When there are no forces favouring reactive or prescriptive, I generally default to reactive. This is because reactive systems are more versatile in maintenance. A prescriptive style becomes awkward when an unforeseen event needs to be handled in the middle of a prescriptive routine. The flow of the routine becomes more complicated.

4.7.3. Properties of event-driven designs

  • Event-driven design easily accommodates events happening externally to the system at unpredictable times. We may be busy processing a previous event when a new events occurs. We typically have an interrupt put the event into the asynchronous event queue. When we are ready to process the event, we may still want to process higher priority events first.

  • Long running tasks such as a heavy algorithm or updating a large display may cause issues with latency for other events. They need to be split into a series of tasks, usually at the outer loop. The loop state needs to be coded manually as a state machine. The C# 'yield return' keyword will tell the compiler to do this for you.

  • Event driven systems need a Timer abstraction to be provided in the programming paradigms layer. The Timer can be asked to issue an event at a future time. It can be asked to issue events at regular intervals.

  • Wiring in ALA may be circular. There is no problem with this from a dependency point of view. Since event-driven is asynchronous there are no issues execution wise either. If they were synchronous, there would be recursion and an infinite loop. Events may flow around the circle continuously. If there are no delays around the circle, the main loop will be constantly busy processing the events as fast as it can.

    Events in a loop should not fan out. Events in a loop that reproduce more events will overload the event queue.

  • The reactor pattern can be used for when the receiver is not ready. The reactor pattern is an event loop which will check if the receiver is in a ready state before dispatching any events to it.

  • A developer used to a synchronous function calling style may expect what looks like a synchronous function call inside a domain abstraction to fully process the event before returning. The port itself will show that it uses an IEvent interface. Inside the IEvent abstraction it can explain that it is an asynchronous programming paradigm. However, where the code actually sends the event, it will only have output.Send(); The choice of the word Send rather than Execute is to indicate it’s only sending the event not executing it.

  • Because the event-driven programming paradigm is asynchronous, senders and receivers can be on different processors or different locations. The decision about where instances of domain abstractions run can even be after the application or user stories abstractions are written. This means that within the architectural 4+1 views framework, the physical view can be changed independently of the logical view.

4.7.4. Global event names

Some conventional event-driven systems use global event names for inter-communication between modules. Each receiver names the events it is interested in, effectively a variation of the observer or publish subscribe pattern. They do this by registering to global event or signal names. This is considered relatively decoupled by its proponents, because senders and receivers don’t know directly about each other, only about global events names. It is illegal in ALA because most events will not abstract enough to be named and become globals. They will tend to be specific to pairs of modules that need to communicate.

Event names then essentially become symbolic wirings. Symbolic wiring is difficult to follow because you have to search for where the names appear throughout the entire code.

By effectively collaborating on symbol names, abstractions are coupled with each other still. It’s a rigid system because modules could not be rewired in a different way without changing them.

In ALA we use point to point wiring instead, or should I say port to port. Wiring is brought out to a coherent place. Because the wiring is point to point, the events are anonymous. You don’t have to name the lines on a diagram.

Having said that, it is possible to have an event that is abstract enough to go into a layer below. Such an event would need to be used by many many domain abstractions so that it is truly more abstract. If only a few domain abstractions need to use an event, then they should still use ports and be all wired up.

If you do create a global abstract event, it would be so ubiquitous that you never want to use the domain abstractions without it. They will have a dependency on it after all.

I can’t think of an example of such an event. Perhaps an event called initialize. It is generated after the wiring code has executed but before an application is set running. Domain abstractions use it to do initialization that needs the wiring in place.

Another example may be a closing event, giving domain abstraction instances a chance to persist their context data before the application closes down.

4.8. Dataflow

A dataflow model is a model in which wired instances in the program (or connected boxes on a diagram) are a path of data without being a path of execution-flow. The execution flow is like in another dimension relative to the data flow - it may go all over the place.

A stream of data flows between the connected components. Each component processes data at its inputs and sends it out of its outputs.

Each input and output can be operated in either push or pull mode. Usually the system prescribes all pull (LINQ), all push (RX), all inputs pull and outputs push (active objects with queues) or all outputs pull and inputs push (active connectors). In ALA we can use a mix of these different mechanism when we define the programming paradigm interfaces.

The network can be circular provided some kind of execution semantic finishes the underlying CPU execution at some point (see synchronous programming below).

The dataflow paradigm raises the question of type compatibility and type safety. Ideally the types used by the components are either parameterised and specified by the application at each connection or determined through type inference.

4.8.1. IDataFlow<T>

I frequently use dataflow execution models.

Here is one variation which works well:

TBD

This variation has these properties:

  • On a diagram, the line (wire) represents a variable that holds the value.

  • Fan-out - one output can connect to multiple inputs. All inputs read the same output variable.

  • Fan-in - multiple outputs cannot connect to one input.

  • Each output is implemented by a single memory variable whose scope is effectively all the places connected by the line (wire).

  • Receivers can get an event when the value changes

  • Receivers can read and re-read their inputs at any time.

  • Operator don’t need to have an output variable, they can pass the get through and recalculate every time instead.

Here is the version I use most often.

TBD

Note that domain abstractions may not collaborate on a specific type for T. A pair of domain abstraction may not, for example, share a DTO (data transfer object) class as that would then be an interface specific to one or other of those classes. T must be more abstract and come from a lower layer, so is often a primitive type from the programming language. T may be passed in by the application, which always knows types of data moving through the system.

Type inferencing is desirable. For example, an instance of a DataStore<T> abstraction could be configured by the application to have some specific fields. Ideally this is the only time the application specifies the fields. The application wires it to a select abstraction that removes one field and then to a join abstraction that adds one field. From there it is wired to a form abstraction that displays the fields. Ideally the form, select and join abstractions do not also have to be configured by the application to know the types of their ports. Instead they are able to infer the type as an anonymous class as it goes from port to port at compile-time.

4.8.2. ITable

This interface moves a whole table of data at once. The table has rows and columns. The columns are determined at runtime by the source.

Run-time types can also be used. For example, the fields in an instance of a table abstraction may not be fully known at compile-time. This is especially true if the table abstraction provides persistence, or, for example, if the data source is a CSV file with unknown fields. In this case a ITable programming paradigm would transfer type information at run-time as well as the data itself.

TBD implementation examples

4.8.3. Glitches

All systems can have glitches when data flows are pushed in a diamond pattern. The diamond pattern occurs when an output is wired to two or more places, and then the outputs of those places eventually come back together. If they never come together, even both seen by a human, then we generally don’t care what order everything is executed in. But when they come together, the first input that arrives with new data will cause processing, and use old data on the other inputs. This unplanned combination of potentially inconsistent data processed together is a glitch. It even happens in electronic circuits.

The following composition of dataflow operators is meant to calculate (X+1)*(X+2)

diagram 25

When X changes, there can be a glitch, a short period of time, in which the output is (Cnew+1)*(Cold+2).

In imperative programming, this problem is up to the developer to manage. He will usually arrange the order of execution and arrange for a single function or method to be called at the place where the data-paths come back together. As he does this, he is introducing a lot of non-obvious coupling indisde the modules of the system, which is one of the big problems with imperative programming.

When we have composability, we don’t know inside the abstractions how data will propagate outside, and how it will arrive at its inputs. We want to execute whenever any of our inputs change, because as far as we know it may be the only change that might happen. So we really want the execution model to take care of eliminating glitches automatically for us.

This is a work in progress for the IDataFlow execution model described above. In the meantime, as a work-around I take care of it at the application level using a pattern. When I know dataflows will re-merge in a potentially inconsistent manner, I wire in an instance of an abstraction called 'Order' between the output and all its destination inputs. This instance of order is configured to explicitly control the order that the output date stream events are executed in. Then I will use a second abstraction called 'EventBlock' at the end of all data paths except one, the one that executes last.

diagram 26

By default multiple IDataFlows wired to a single output are executed in the order that they are wired anyway. On the diagram, they are drawn top to bottom in that order. This improves the determinism but is a little too implicit for my liking, so that is why I use the order abstraction.

4.8.4. Live dataflow

As used in the coffee-maker example earlier, this paradigm simulates electronic circuits instead of using the concept of discrete messages. Semantically the inputs have the values of the outputs they are wired to at all times. This type of flow is readily implemented with shared memory variables.

FRP (Functional Reactive Programming) also is effectively a live dataflow execution model.

4.8.5. Synchronous dataflow

The use of the word synchronous here is different from its use in the discussion of synchronous/asynchronous events above. Here it means a master system clock clocks the data around the system on regular ticks. At each tick, every instance latches its own inputs and then processes them and places the results on their outputs. Data progresses through one operator per tick, so takes more time to get through the system from inputs to outputs. The result is a more deterministic and mathematically analysable system.

The execution timing and the timing of outputs occurs at a predictable tick time, albeit on a slower time scale than an asynchronous system. All timings are lifted into the normal design space.

Glitches that could occur in an asynchronous system (discussed earlier) are eliminated at the level of single clock ticks. A fast glitch could not occur. A glitch would occur when different data paths had different lengths, and would last for at least one tick duration. Controlling glitches is therefore lifted into the normal design space.

4.9. Activity-flow

The name Activity-flow comes from the UML activity diagram. Activities that are wired together execute in order. One starts when the previous one finishes. The activity itself may take a long time to complete (without blocking the CPU). Activity flows can split, run concurrently and recombine.

Activity-flow contrasts with event-driven. Where event-driven is reactive, activity-flow is prescriptive. It orchestrates what will happen rather than reacting to what might happen.

Activity-flow is not the same as the old flow diagrams. Flow diagrams were for the imperative programming paradigm where the flow was the flow of the CPU. Activity flow can have delays and other time discontinuities as it syncs with what’s happening in the outside world.

Activity-flow’s execution model can be the same as event driven. Each domain abstraction has a start input port and a done output port. The 'done' port of one instance of a domain abstraction can be wired to the 'start' port of the next. The ports are just event ports and can be wired for synchronous or asynchronous execution.

If the Activity-flow is a linear sequence, we can consider wiring the instances using text. However activity-flow abstractions will often need other wiring (using other programming paradigms) to UI or other input/output. C The domain abstractions may have request/response ports for their I/O. These may be synchronous or asynchronous depending on the design factors discussed earlier. It may wish to poll something external at regular intervals to see if it’s complete, so it may register on a timer for regular events. (The timer is an abstraction in the programming paradigms layer, which is typically wired to the event-loop abstraction for asynchronous execution).

The domain abstractions may internally use an asynchronous execution model, such as for a delay.

4.9.1. Structured activity flow wiring using text (experimental)

This is a thought experiment at this stage. The experiment is to see if we can do structured programming for activity flow. Remember activity flow is instances of domain abstractions, each of which generally has a start port and a done port.

The idea is to mimic imperative structured programming. Structural programming is what got rid of the goto and introduce block structured statements such as while and if. It is generally laid out with indenting that exactly matches the nested structure of braces. Your brain sees the indenting but the compiler sees the curly braces. (Except for Python which makes the compiler use what the brain sees).

In this program, we will string together some instances of domain abstractions and include a loop and a conditional. The indenting structure is the same as for the imperative version.

Remember this code is not executing the activity flow, it is just wiring it all up for later execution.

TBD need the corrsponding diagram here to show what this code is trying to do

ActivityFlow.cs
program = new A();
    program.
    .WireIn(new B())
    .WireIn(
        Loop (
            new C()
            .WireIn(
                If (new D(),
                    new E(),
                    new F()
                )
            )
            .WireIn(new G())
            ,
            new H();
        )
    )
    .WireIn(new I())

First remember that WireIn returns its second parameter to support this fluent style. A is the first activity. A’s done port is wired to B’s start port, so B is the second activity. Everything else is in a loop. The 'Loop' function takes two parameters, one is another flow and one is the looping condition, which in this case is H. B gets wired to C. 'If' is a function that takes three parameters, a condition, which in this case is D, and two flows. C gets wired to D. The 'If' function expects D to have two done ports, called donetrue and donefalse. It wires donetrue to E. It wires donefalse to F. 'If' wires the done ports of both E and F to a null activity instance to recombine the flow. The null instance is returned by 'If'. The null instance is wired to G. G gets wired to H. The 'Loop' function expects H to have two exit ports calls done and loop. 'Loop' wires H’s loop port to C, and returns H. H is wired to I.

This code looks okay, however, as is often the problem with text based representations of relationships, most of the instances will probably need additional wiring to other things as well. If this is the case, and the requirements implicitly contains a graph structure rather than a tree structure, then a diagram wll be the best way to represent it.

4.10. Work-flow

Persisted Activity-flow. This includes long running activities within a business process such as an insurance claim.

4.11. IIterator

This dataflow interface allows moving a finite number of data values at once. It does so without having to save all the values anywhere in the stream, so has an efficient execution model that moves one data value at a time through the whole network.

This is the ALA equivalent of both IEnumerator and IObserver as used by monads. ALA uses the WireTo extension method that it already has to do the Bind operation. So the IIterator interface is wired in the same consistent way as all the other paradigm interfaces. There is no need for IEnumerable and IObservable type interfaces to support Also unlike monads, multiple arbirary interfaces can be wired between two objects with a single wiring operation.

IIterator has two variants that handle push and pull execution models. Either the A object can push data to the B object, or the A object can pull data from the B object.

TBD implementation examples

4.12. UI layout

This programming paradigm is used for laying out a graphical user interface. A relationship means put the target instance of a UI element inside the first instance of a UI element. The order of the fanout of relationships sets the order that the elements appear. For most UI domain abstractions, UI elements default to going go vertically downwards.

I use two domain abstractions called vertical and horizontal to control whether they are layed out vertically or horizontally.

Here is the interface for use by domain abstractions that will use .NET’s WPF class library for the implementation.

IUI.cs
using System.Windows;

namespace ProgrammingParadigms
{
    /// <summary>
    /// Hierarchical containment structure of the UI
    /// </summary>
    public interface IUI
    {
        UIElement GetWPFElement();
    }
}

This programming paradigm is similar to XAML. It doesn’t use XML syntax, it uses wiring code or diagrams the same as all other programming paradigms. Binding to data in XAML is done using dataflow ports on domain abstractions. Unlike XML, the entire application is built the same way.

4.13. UI navigation flow

TBD

4.14. Data schema

TBD

4.15. Locking resources

Even in a single threaded system, we still have concurrency at a course grained level. We want to allow our one thread to do other tasks whenever something else is waiting. Or, whenever an asynchronous communication occurs, we may choose to do previously queued tasks, or higher priority tasks, before processing the latest one. We can call the concurrent sets of tasks an activity.

We may have a resource or external device that can be be used by multiple activities. There is a set of tasks that need to complete on the resource without interrupton by other activities. This is called a transaction. Examples of resources that can have transactions are a database or an external device such as a robot arm. Several queries or movements may be involved in the transaction.

We need a locking mechanism for the resource. I recommend an arbitration programming paradigm. At the application level, we need to specify which instances of domain abstractions that perform transactions need to collaborate by locking or waiting for a given resource.

Every domain abstractions that performs a transaction on a given resource has a port of this programming paradigm. All instances using a given resource are wired to a single instance of an arbitrator abstraction. Effectively this wiring specifies the collaboration that must occur between the instances. This collaboration is done at the abstraction level of the system, where it belongs, not inside the abstractions.

The ALAExample project at www.github.com/johnspray74 has an example of this. The IArbitrator interface is considered a programming paradigm. It contains an async method for locking the resource. This method can be awaited on until the resource is free. A second method releases the resource, which would allow another activity waiting to proceed.

The arbitrator abstraction could be given the ability to detect deadlocks and even break deadlocks.

4.16. State machines

To get used to how different these programming paradigms can be, let’s go now to something completely different - state machines. We wont be going into understanding them at the code level because we want to support hierarchical state machines, and the code for that is a little bit non-trivial, but we do want to get an understanding of how state machines are just another programming paradigm that allows us to wire together instance of abstractions. The meaning of the wiring is different than what it was for the event programming paradigm.

I assume a basic understanding of what state machines are.

FSM-generic.png
Figure 26. State machine execution model

At first it can be difficult to express the solution to a requirements problem as a state machine, even when the state machine is a suitable way to solve the problem. It takes some getting used to the first time. But it only takes a little bit of practice to begin to master it.

I once had to express a set of user stories that involved different things that could happen from the outside, either through the UI or other inputs. I knew these were the kind of user stories that were nicely expressed by a state machine, but I had no idea where to start. I only knew that the previously written C code to do the job was a big mess that could no longer be maintained. But I started drawing the state machine, first on paper and then in Visio, and everything started to fall into place very nicely. Before I knew it I had represented what used to be 5000 lines of C code by a single A3 sized state machine diagram. This diagram so well represented the user stories that it was easy to maintain for years to come. This experience was a big factor in the final conception of ALA.

Here is the diagram.

BigStateMachine.pdf.jpg
Figure 27. My first significant state machine for a real embedded device

Notice that the diagram makes heavy use of hierarchical states (boxes inside boxes). These turn out to be important in most of my state machines.

State machine diagrams are drawn in their own unique way. The boxes of the diagram are instances of the abstraction "State". The lines on a state machine diagram are actually instances of another abstraction, "Transition". Out of interest, to relate a state machine diagram to a more conventional ALA wiring diagram, you would replace all the lines on the state machine with boxes representing instances of Transition. The event, guard and actions that associate with a transition then go inside the transition box to configure it. Lines would then wire the transition box to its source state instance and destination state instance. Hierarchy is drawn on the state machine by boxes inside boxes, but in the conventional ALA wiring diagram, the boxes would be drawn outside with lines showing the tree structure. This analogous to the tree structured wiring we have used in previous examples for expressing UIs, which are actually 'contains' relationships.

The graphical tool being developed will allow the drawing of hierarchical state machines. It will internally transform it to conventional wiring of instances of states and transitions. Interfaces called something like ITransitionSource, ITransitionDestination and IHiercharical would be used to make it execute. It is a simple matter to write code inside the state and transition abstractions to make them execute that would be adequately efficient for most purposes.

How to make hierarchical state machine execute in an optimally efficient way is a non-trivial problem, but I have worked out the templates for what the C code should look like. Generating this code is a topic for another web page.

4.17. Imperative

Much conventional code is written using the so called imperative programming paradigm. This paradigm has the same execution model of the underlying CPU hardware. Imperative means sequential execution flow of instructions or statements in computer time.

Imperative is seldom a good programming paradigm for expressing whole user stories. Even though we call our imperative languages high level languages, its actually quite a low level programming paradigm. However it is efficient because it executes almost directly on the hardware. Imperative highly prescriptive. We can code applications in it directly or we can use it to build other programming paradigms.

Function or method calls go to a named destination, and are synchronous (pass the CPU to the called function for execution, and pass it back to the caller on completion.

The imperative programming paradigm is wonderful for writing algorithms that are not tied to real-time. However, in modern software, that is a tiny fraction of what programs do. We will seldom use the 'imperative' programming paradigm in ALA.

Imperative can be structured to comply with ALA constraints, almost. The user story simply makes function calls or method calls to the domain abstractions in the layer below. The problem is that the user story ends up controlling the execution flow, and it handles the data at runtime. The data it receives from one domain abstraction will be passed to the next domain abstraction. This is not really a responsibility we want to put on the user story. We want to factor out execution flow and data. We want the user story to be just about composing instances of domain abstractions.

4.18. Multithreading

Compared with ALA, modular programming will look like a big pool of mud. Multithreaded programming will look like a big pool of boiling mud.

In the section about request/response, we briefly considered using multithreading to solve the problem when the request/response is implemented as a synchronous function call, but it takes time and the call blocks.

In this section we discuss briefly why we avoid using multithreading to solve that particular problem, and discuss what problems might justify using multithreading.

TBD WIP

Because threads block, we must put everything that needs to be concurrent on different threads. Whether it’s a conventional architecture or an ALA architecture this leads to coupling throughout the system. Modules may tend to be based on threads rather than a more logical separation. Furthermore, different parts of the system have to collaborate by locking accesses to shared state. There is a misconception that shared state is caused by globals. This is incorrect. Shared state occurs all the time in object oriented programs. Any objects accessed from different threads are shared state even if all state in an object is private. So if a UI object gets work done by a different thread so that the UI remain responsive, then the result will come back to the UI objects on a different thread unless this is carefully avoided. By default most objects are not thread-safe. Missing locks will lead to race conditions. As locks are added, there is even more blocking occurring. This can reduce performance, increase non-determinism, or require even more threads. Too much locking can lead to deadlocks or priority inversions. These issues will hide and appear rarely.

Unless it is required for latency or other performance throughput reasons that can’t be solved on a single thread, I don’t recommend going into the quagmire of pre-emptive multithreading. Even if another thread is needed for a specific performance case, I still recommend putting the majority of code in one thread despite any difficulties that entails (as discussed below).

4.19. Agent based programming

Note that there is a different programming style of multithreading that doesn’t use shared state. It is called agent based programming. In this style, we think of every thread as effectively being on a different processor. They can only communicate with one another with messages. Every thread has a single input queue. All communications are asynchronous. Synchronous calls between agents is not possible, so there is no shared state.

The thread’s main loop does nothing other than take events from the input queue one at a time, process them, and asynchronously sends events to other such threads. This execution model is a completely different thing. It is called the agent model or producer/consumer. It is safe because there is no shared state and locks are not required. If there is a 'shared' resource, one thread can be assigned to resource. This model does not solve the problem of how to do synchronous request-response calls that block. It is not even the prescriptive programming style that we are trying to achieve with request/response. Every thread is already transformed into a reactive style. Such an execution model is equivalent to a single threaded system where all calls are asynchronous. Like an all asynchronous execution model, performance can be improved by assigning certain abstraction instances to their own processors.

4.19.1. State machine vs multithreading concurrency styles

A bigger problem with callback functions or futures is that if the requester call is inside structured statements such as a loop or if statement, or has been called from another function, all the code right back to main() needs to be rewritten like a state machine. It must keep state variables to remember what would normally be implicit in the program counter state, and manually store any other stack based state that the compiler would normally handle for the execution flow through the program. If the code is a simple function called directly from main, this can be done fairly easily. Each time the function is called, it reads the state, which is usually a function pointer, and dispatches to it. Callbacks or continuations go to their own functions.

+ An advantage of this style of programming is that it easily handles all time discontinuities - things that would otherwise block a thread. It allows reacting to unexpected events much more easily. And it allows longish routines to yield by simply returning part way through, say inside a loop, to reduce the latency of any other concurrent tasks waiting to execute. When the main loop calls back, it can use the state variables to resume processing where it left off.

+ The great disadvantage of this style of programming is when the program is more prescriptive than reactive. There is a fixed sequence of things that will happen, and we want to express that as normal sequential lines of code, even though certain operations will block. For example, we are moving a large amount of data. Exceptions to the prescribed sequence are rare. I find that async/await or co-routines are the best solutions for this situation. If they are not available, then a cooperative (non-preemptive) thread could be considered to solve the one situation.

+ One of the most common requirements for concurrency is responding to user input. For this we may specify a soft deadline of 0.1s. This means that all state machine, callbacks, or other run-to-completion routines should execute in less than 0.1s. This not difficult to do because the vast majority will execute very quickly. What I sometimes do is put in a system timer to measure the longest running routine. It’s usually updating a large display.

+ What I see happening in most traditional systems is that once an RTOS is included in the system, it is considered to be the solution to all concurrency in the system. But probably 99% of concurrency in most systems can be done on a single thread. Most tasks may have priorities, but will wait until the CPU resource gets to them. So what I do is avoid using threads except for when the specific case of performance can’t be solved in any other way. So, in my entire career in embedded systems, I have never ended up having to use a second thread, even when I have an RTOS already in the system at my disposal. Short interrupt routines have handled all situations with hard real time latency requirements. The state machine programming style has better suited the reactive nature of most embedded systems.

+ Remember you can only have one highest priority thread. If you are really in a situation where you have one or more hard real time deadlines that can’t be done in interrupts, then you should probably consider putting in multiple MCUs rather than trying to do, for example, rate-monotonic analysis.

+ Of course, if your system has multiple CPU cores, then you probably have a performance requirement that will need multiple threads to make use of them.

Some people, when confronted with a problem, think "I know, I’ll use regular expressions." Now they have two problems. - Jaimie Zawinski

Some people, when confronted with a problem, think "I know, I’ll use threads." Now they have ten problems. - Bill Schindler

4.19.2. Agents on a single thread

TBD

4.20. Example project - Ten-pin bowling

The full source code for the bowling application can be viewed or downloaded from here: https://github.com/johnspray74/GameScoring

The ten-pin bowling problem is a common coding kata. Usually the problem presented is just to return the total score, but in this example we will tackle the more complicated problem of keeping the score required for a real scorecard, which means we need to keep all the individual frame ball scores. We can afford to do this even for a pedagogical sized example because ALA can provide a simple enough solution.

bowling scorecard2

The ALA method starts by "describing the requirements in terms of abstractions that you invent". When we start describing the requirements of ten-pin bowling, we immediately find that "a game consists of multiple frames", and a "frame consists of multiple balls". Let’s invent an abstraction to express that. Let’s call it a "Frame". Instances of Frame can be wired together by a "ConsistsOf" relationship. So let’s invent an abstract interface to represent that, and call it 'IConsistsOf'.

Here is the diagram of what we have so far.

diagram bowling 1

This is the first time we are using a diagram for an ALA application, so le’s go through the conventions used.

The name in the top of the boxes is the abstraction name. The name just beneath that is the name of an instance of the abstraction. For the bowling application above, we are using two instances of the Frame abstraction, one called "game" and one called "frame". Below the abstraction name and instance name go any configuration information of the instance.

The Frame abstraction is configured with a lambda function to tell it when it is finished. The Frame abstraction works like this - when its last child is complete it will create a new one. It will stop doing that when the lambda expression is true. It will tell its parent it is complete when both the lambda expression is true and its last child Frame is complete.

The end of the chain is terminated with a leaf abstraction that also implements the 'IConsistsof' interface called 'SinglePlay'. It represents the most indivisible play of a game, which in bowling is one throw. Its job is to record the number of pins downed.

The concept in the Frame abstraction is that at run-time it will form a composite pattern. As each down-stream child frame completes, a Frame will copy it to start a new one. This will form a tree structure. The "game" instance will end up with 10 "frames", and each frame instance will end up with 1, 2 or 3 SinglePlays.

Note, in reference to the ALA layers, this diagram sits entirely in the top layer, the Application layer. The boxes are instances of abstractions that come from the second layer, the Domain Abstractions layer. The arrows are instances of the programming paradigm, 'InConsistsOf', which comes from the third layer, the ProgrammingParadigms layer.

This diagram will score 10 frames of ten-pin bowling but does not yet handle strikes and spares. So let’s do some 'maintenance' of our application. Because the application so far consists of simple abstractions, which are inherently stable, maintenance should be possible without changing these abstractions.

The way a ten-pin bowling scorecard works, bonuses are scored in a different way for the first 9 frames than for the last frame. In the first nine frames, the bonus ball scores come from following frames, and just appear added to the frame’s total. They do no appear as explicit throws. In the last frame, they are shown as explicit throws on the scorecard. That is why there are up to 3 throws in that last frame.

To handle the different last frame, we just need to modify the completion lambda expression to this.

frameNum<9 && (balls==2 || pins==10) // completion condition for frames 1..9
|| (balls==2 && pins<10 || balls==3) // completion condition for frame 10

To handle bonuses for the first 9 frames, we introduce a new abstraction. Let’s call it Bonuses. Although we are inventing it first for the game of ten-pin bowling, it is important to think of it as a general purpose, potentially reusable abstraction.

What the Bonus abstraction does is, after its child frame completes, it continues adding plays to the score until its own lambda function returns true.

The completed ten-pin bowling scorer is this:

diagram bowling 2

Note that the "game" instance (the left box of the diagram) implements IConsistsOf. This is where the outside world interfaces to this scoring engine. During a game, the number of pins knocked down by each throw is sent to this IConsistsOf interface. To get the score out, we would call a GetScore method in this interface. The hard architectural work is done. We have invented abstractions to make it easy to express requirements. We have a diagram that describes the requirements. And the diagram is executable. All we have to do is put some implementation code inside those abstractions and the application will actually execute.

First let’s turn the diagram into equivalent code. At the moment, there are no automated tools for converting such diagrams to code. But it is a simple matter to do it manually. We get the code below:

private IConsistsOf game = new Frame("game")
    .setIsFrameCompleteLambda((gameNumber, frames, score) => frames==10)
    .WireTo(new Bonus("bonus")
        .setIsBonusesCompleteLambda((plays, score) => score<10 || plays==3)
        .WireTo(new Frame("frame")
            .setIsFrameCompleteLambda((frameNumber, balls, pins) => frameNumber<9 && (balls==2 || pins[0]==10) || (balls==2 && pins[0]<10 || balls == 3))
            .WireTo(new SinglePlay("SinglePlay")
    )));

All we have done is use the 'new' keyword for every box in the diagram. We have made the constructor take the instance name as a string. (This name is not used except to identify instances during debugging.) We use a method called "WireTo" for every line in the diagram. More on that in a minute. And we pass any optional configuration into the instances using setter methods. The WireTo method and the configuration setter methods all return the 'this' pointer, which allows us to write this code in fluent style. If you are not familiar with fluent style it is just making methods return the this reference, or another object, so that you can chain together method calls using dot operators.

Not all ALA applications will be put together using the method in the previous paragraph, but I have found it a fairly good way to do it for most of them, so we will see this same method used for other example projects to come.

So far, this has been a fairly top-down, waterfall-like approach. We have something that describes all the details of the requirements, but we haven’t considered implementation at all. Past experience tells us this may lead us into dangerous territory. Will the devil be in the details? Will the design have to change once we start implementing the abstractions? The first few times I did this, I was unsure. I was not even sure it could actually be made to work. The reason it does work is because of the way we have handled details. Firstly all details from requirements are in the diagram. The diagram is not an overview of the structure. It is the actual application. All other details, implementation details, are inside abstractions, where they are hidden even at design-time. Being inside abstractions isolates them from affecting anything else. So, it should now be a simple matter of writing classes for those three abstractions and the whole thing will come to life. Implementing the three abstractions turns out to be straightforward.

First, design some methods for the IConsistOf interface that we think we will need to make the execution model work:

    public interface IConsistsOf
    {
        void Ball(int score);
        bool IsComplete();
        int GetScore();
        int GetnPlays();
        IConsistsOf GetCopy(int frameNumber);
        List<IConsistsOf> GetSubFrames();
    }

The first four methods are fairly obvious. The Ball method receives the score on a play. The Complete, GetScore and GetnPlays methods return the state of the sub-part of the game. The GetCopy method asks the object to return a copy of itself (prototype pattern). When a child frame completes, we will call this to get another one. The GetSubFrames method is there to allow getting the scores from all the individual parts of the game as required.

The SinglePlay and Bonus abstractions are very straightforward.

So let’s code the Frame abstraction. Firstly, Frame both implements and accepts IConsistsOf. A field is needed to accept an IConsistsOf. The WireTo method will set this field:

// Frame.cs
private IConsistsOf downstream;

Frame has one 'state' variable which is the list of subframes. This is the composite pattern we referred to earlier, and what ends up forming the tree.

// Frame.cs

private List<IConsistsOf> subFrames;
private readonly Func<int, int, int, bool> isFrameComplete;
private readonly int frameNumber = 0;

The second variable is the lambda expression that is a configuration passed to us by the application. It would be readonly (immutable) except that I wanted to use a setter method to pass it in, not the constructor, to indicate it is optional.

The third variable is the frameNumber, also immutable. It allows frame objects to know which child they are to their parent - e.g. 1st frame, 2nd frame etc. This value is passed to the lambda expression in case it wants to use it. For example, the lambda expression for a bowling frame needs to know if it is the last frame.

The methods of the IConsistsOf interface are now straightforward to write. Let’s go over a few of them to get the idea. Here is the most complicated of them, the Ball method:

public void Ball(int player, int score)
{
    // 1. Check if our frame is complete, and do nothing
    // 2. See if our last subframe is complete, if so, start a new subframe
    // 3. Pass the ball score to all subframes

    if (IsComplete()) return;

    if (subFrames.Count==0 || subFrames.Last().IsComplete())
    {
        subFrames.Add(downstream.GetCopy(subFrames.Count));
    }

    foreach (IConsistsOf s in subFrames)
    {
        s.Ball(player, score);
    }
}

It looks to see if the last child frame has completed, and if so starts a new child frame. Then it just passes on the ball score to all the child objects. Any that have completed will ignore it.

The IsComplete method checks two things: 1) that the last child object is complete and 2) that the lambda expression says we are complete:

private bool IsComplete()
{
    if (subFrames.Count == 0) return false; // no plays yet
    return (subFrames.Last().IsComplete()) &&
        (isLambdaComplete == null ||
         isLambdaComplete(frameNumber, GetnPlays(), GetScore()));
}

GetScore simply gets the sum of the scores of all the child objects:

private int GetScore()
{
    return subFrames.Select(sf => sf.GetScore()).Sum();
}

The GetCopy method must make a copy of ourself. This is where the prototype pattern is used. This involves making a copy of our child as well. We will be given a new frameNumber by our parent.

IConsistsOf GetCopy(int frameNumber)
{
    var gf = new Frame(frameNumber);
    gf.objectName = this.objectName;
    gf.subFrames = new List<IConsistsOf>();
    gf.downstream = downstream.GetCopy(0);
    gf.isLambdaComplete = this.isLambdaComplete;
    return gf as IConsistsOf;
}

The few remaining methods of the IConsistOf interface are trivial. The implementation of IConsistsOf for the other two abstractions, SinglePlay and Bonuses, is similarly straightforward. Note that whereas Frame uses the composite pattern, Bonuses uses the decorator pattern. It implements and requires the IConsistsOf interface. The SinglePlay abstraction, being a leaf abstraction, only implements the IConsistsOf interface.

One method we haven’t discussed is the wireTo method that we used extensively in the application code to wire together instances of our domain abstractions. The wireTo method for Frame is shown below:

public Frame WireTo(IConsistsOf c)
{
    downstream = c;
    return this;
}

This method does not need to be implemented in every domain abstraction. I use an extension method for WireTo. The WireTo extension method uses reflection to find the local variable to assign to.

The WireTo method will turn out to be useful in many ALA designs. Remember in ALA we "express requirements by composing instances of abstractions". If the 'instances' of 'abstractions' are implemented as 'objects' of 'classes', then we will use the wireTo method. If the 'instances' of 'abstractions' are 'invocations' of 'functions', as we did in the example project in Chapter One, we wont use WireTo obviously. In the coffeemaker example to come, 'instances' of 'abstractions' are 'references' to 'modules' because a given application would only have one of each abstraction.

The wireTo method returns 'this', which is what allows the fluent coding style used in the application code. The configuration setter methods also return the this reference so that they too can be used in the fluent style.

Here is the full code for the Frame abstraction (with comments removed as we just explained everything above):

// Frame.c
using System;
using System.Collections.Generic;
using System.Linq;
using GameScoring.ProgrammingParadigms;
using System.Text;

namespace GameScoring.DomainAbstractions
{

    public class Frame : IConsistsOf
    {
        private Func<int, int, int[], bool> isLambdaComplete;
        private readonly int frameNumber = 0;
        private IConsistsOf downstream;
        private string objectName;
        private List<IConsistsOf> subFrames = new List<IConsistsOf>();


        public Frame(string name)
        {
            objectName = name;
        }




        public Frame(int frameNumber)
        {
            this.frameNumber = frameNumber;
        }



        // Configuration setters follow.

        public Frame setIsFrameCompleteLambda(Func<int, int, int[], bool> lambda)
        {
            isLambdaComplete = lambda;
            return this;
        }





        // Methods to implement the IConsistsOf interface follow


        public void Ball(int player, int score)
        {
            if (IsComplete()) return;

            if (subFrames.Count==0 || subFrames.Last().IsComplete())
            {
                subFrames.Add(downstream.GetCopy(subFrames.Count));
            }

            foreach (IConsistsOf s in subFrames)
            {
                s.Ball(player, score);
            }
        }




        public bool IsComplete()
        {
            if (subFrames.Count == 0) return false;
            return (subFrames.Last().IsComplete()) &&
                (isLambdaComplete == null ||
                 isLambdaComplete(frameNumber, GetnPlays(), GetScore()));
        }




        public int GetnPlays()
        {
            return subFrames.Count();
        }




        public int[] GetScore()
        {
            return subFrames.Select(sf => sf.GetScore()).Sum();
        }



        List<IConsistsOf> IConsistsOf.GetSubFrames()
        {
            return subFrames;
        }




        IConsistsOf IConsistsOf.GetCopy(int frameNumber)
        {
            var gf = new Frame(frameNumber);
            gf.objectName = this.objectName;
            gf.subFrames = new List<IConsistsOf>();
            gf.downstream = downstream.GetCopy(0);
            gf.isLambdaComplete = this.isLambdaComplete;
            return gf as IConsistsOf;
        }

    }
}

4.21. Example project - Tennis

Now let’s modify the bowling application to score tennis. If the bowling game hadn’t been implemented using ALA, you probably wouldn’t contemplate doing this. But ALA excels for maintainability, and I want to show that off by changing Bowling to Tennis. The Frame and IConsistsOf abstractions look like they could be pretty handy for Tennis. A match consists of sets, which consists of games, which consists of SinglePlays.

We will need to make a small generalization to the Frame abstraction first. This will allow it to keep score for two players. We just change the type of the score from int to int[]. The Ball method will be generalised to take a player parameter to indicate which player won a play. A generalization of an abstraction to make it more reusable is a common operation in ALA.

The only other thing we will need to do is invent a new abstraction to convert a score such as 6,4 into a score like 1,0, because, for example, the winner of a game takes one point into the set score. This new abstraction is called WinnerTakesPoint (WTP in the diagram).

Here is the tennis scoring game:

tennis1

The diagram expresses all the details of the requirements of tennis except the tiebreak.

Here is the diagram’s corresponding code:

private IConsistsOf match = new Frame()
    .setIsFrameCompleteLambda((matchNumber, nSets, score) => score.Max()==3)
    .WireTo(new WinnerTakesPoint()
        .WireTo(new Frame()
            .setIsFrameCompleteLambda((setNumber, nGames, score) => score.Max()>=6 && Math.Abs(score[0]-score[1])>=2)
            .WireTo(new WinnerTakesPoint()
                .WireTo(new Frame()
                    .setIsFrameCompleteLambda((gameNumber, nBalls, score) => score.Max()>=4 && Math.Abs(score[0]-score[1])>=2)
                    .WireTo(new SinglePlay()))))));

The new WinnerTakesPoint abstraction is easy to write. It is a decorator that implements and requires the IConsistsOf interface. Most methods pass through except the GetScore, which returns 0,0 until the down-stream object completes, then it returns either 1,0 or 0,1 depending on which player has the higher score.

And just like that, the tennis application will now execute. The frame abstraction we invented for bowling is already done.

4.21.1. Add tiebreak

Now let’s switch our attention back to another example of maintenance. Let’s add the tiebreak feature. Another instance of Frame will score the tiebreak quite nicely. However we will need an abstraction that can switch us from playing the set to the tie break. Let’s call it Switch, and give it a lambda function to configure it with when to switch from one subframe tree to another. Switch simply returns the sum of scores of its two subtrees. Here then is the full description of the rules of tennis:

tennis2

And here is the code version of that diagram. This application passes an exhaustive set of tests for the scoring of tennis.

private IConsistsOf match = new Frame("match")
    .setIsFrameCompleteLambda((matchNumber, nSets, score) => score.Max()==3)
    .WireTo(new WinnerTakesPoint("winnerOfSet")
        .WireTo(new Switch("switch")
            .setSwitchLambda((setNumber, nGames, score) => (setNumber<4 && score[0]==6 && score[1]==6))
            .WireTo(new Frame("set")
                .setIsFrameCompleteLambda((setNumber, nGames, score) => score.Max()>=6 && Math.Abs(score[0]-score[1])>=2)
                .WireTo(new WinnerTakesPoint("winnerOfGame")
                    .WireTo(new Frame("game")
                        .setIsFrameCompleteLambda((gameNumber, nBalls, score) => score.Max()>=4 && Math.Abs(score[0]-score[1])>=2)
                        .WireTo(new SinglePlay("singlePlayGame"))
                    )
                )
            )
            .WireTo(new WinnerTakesPoint("winnerOfTieBreak")
                .WireTo(new Frame("tiebreak")
                    .setIsFrameCompleteLambda((setNumber, nBalls, score) => score.Max()==7)
                    .WireTo(new SinglePlay("singlePlayTiebreak"))
            )
        )
    )
);

And just like that we have a full featured executable tennis scoring engine.

4.21.2. Final notes

Notice that I have added string names to the instances of Frame and other objects. This is not required to make the program function, but generally is a good habit to get into in ALA. It is because in ALA we typically use multiple instances of abstractions in different parts of the program. The names give us a way of identifying the different instances during any debugging. Using them I can Console.Writeline debugging information along with the object’s name.

Around 8 lines of code express the rules of ten-pin bowling and around 15 lines of code express the rules of tennis. That sounds about right for the inherent complexity of the two games. The two rule descriptions actually execute and pass a large battery of tests.

The domain abstractions are zero-coupled with one another, and are each straightforward to write by just implementing the methods of the IConsistOf interface according to what the abstraction does. The abstractions are simple and stable. So no part of the program is more complex than its own local part.

The domain abstractions are reusable in the domain of game scoring. And, my experience was that as the details inside the abstractions were implemented, the application design didn’t have to change.

Why two example applications? The reason for doing two applications in this example is two-fold.

  1. To show the decreasing maintenance effort. The Tennis game was done easily because it reused domain building blocks we had already created for bowling.

  2. To emphasis where all the details of the requirements end up. The only difference between the bowling and tennis applications is the two diagrams, which are translated into two code files: bowling.cs and tennis.cs of 8 lines and 15 lines respectively. These two files completely express the detailed requirements of their respective games. No other source files have any knowledge of these specific games. Furthermore, Bowling.cs and Tennis.cs do not do anything other than express requirements. All implementation to actually make it execute is hidden in domain abstractions and programming paradigm abstractions.

Here is a link to the code on Github: GameScoring code

5. Chapter five - Methodology

5.1. The first few strokes

As a software engineer contemplating a new project, I have often asked myself "Where do I start?" This also happens with legacy code, when contemplating the direction that refactorings should take. "If this software were structured optimally well, what would it look like?"

Christopher Alexander, the creator of the idea of design patterns in building architecture, said, "As any designer will tell you, it is the first steps in a design process which count for the most. The first few strokes which create the form, carry within them the destiny of the rest". This has been my experience too.

In Agile, where architecture is meant to emerge, this wisdom has been lost. ALA restores that wisdom to software development, and gives the software architect the exact process to follow for that little piece of up-front design. No more than one sprint is required to do this architectural work, regardless of the size of the project.

Furthermore, once this architectural work is done, the Agile process works significantly better thereafter.

My experience over several projects so far is that the initial architecture does not need to change as the development proceeds if good quality abstraction are invented.

There is one aspect of ALA that is hard to master - the invention of appropriate abstractions in a new domain. The reason why the "standard deviation uses squareroot" example seems easy is that the squareroot abstraction was already invented, and we already know it. In ALA you will need to invent your own domain level abstractions. In other engineering disciplines, new abstractions come along only every few years, or hundreds of years sometimes. In software engineering, we have to do it every day in the first two weeks of a project in a new domain, and probably every iteration after that for a few iterations. But all whom I have taught how to do this have found it worth the effort, and all get much better at doing it. Working in the resulting zero-coupled code becomes a joy.

5.2. Agility

ALA is inherently optimally agile (except for an Iteration zero which we will discuss next). By optimally agile, we mean that the amount of code that changes to change functional requirements is the minimum it can be. The amount of code that actuall depends on requirements is a small percentage of the total. This differs from the average application where most code depends on the requirements.

ALA achieves this at its first level of separation of concerns. This first separation is to separate code that will just describe requirements from code that will implementation the abstractions needed to describe the requirements. This implementation code never has knowledge of any requirements, so it generally doesn’t change when requirements change. Only the code that describes requirements needs to change, and that code is optimally minimal.

5.2.1. Iteration zero

When a new project begins, the only new information we have is the requirements. Any design decisions that don’t depend on the specific requirements could already have been made beforehand. It is those decisions that form the ALA reference architecture. Therefore, when we get the requirements, that is our immediate and total focus. We may not know all of them, but we will only need a sample to build start inventing our domain abstractions.

Looking at the available new information as a whole first instead of taking it a bit at a time during the project’s sprints will make a huge difference to the quality of the abstractions we invent, and eventually the quality of the application’s architecture.

The process in the first iteration takes requirements one by one, and represents them, in all their detail. Domain abstractions will be invented as you go, and they will have parameters or properties that will handle those details from requirements.

For the first green field application, you spend a maximum of one sprint. After that you do need to find out if your design works. So you may not get through all the known requirements. That does not matter.

To know whether knowledge from your design goes in the application layer or the domain abstractions layers, you consider what the scope of that knowledge is. Is the knowledge specific to this one requirement in the one application, or is it potentially reusable in the same or other applications? A softkey label is clearly specific and goes in the application. The concept of softkeys is clearly a domain abstraction.

The output of the first sprint does not implement any of the invented abstractions, but it does include all details of the requirements that are looked at. In so doing, you design the first approximation of a DSL. The DSL may be refined later as more requirements are looked at.

Each abstraction will eventually be implemented as a class, but initially we just record the names of the abstractions, and a short explanation that provides the insight into what the abstraction does.

By the end of the first sprint the requirements will have become easier and easier to represent, as the set of abstractions will have taken shape. Sometimes you will generalise an abstraction further to enable it to be useful in more parts of the application.

By keeping moving through the requirements at a much faster pace than in normal development (say one feature per hour instead of one per week), we can keep representing them in a coherent way, revising abstraction ideas we have already invented. Ideally, we will end up with a set of domain abstractions that can be wired together in different ways to represent the requirement space - the space of all possible requirements. That space, which we call the domain, will grow slightly as time goes on and it accommodates a growing family of products, but we don’t want it to grow beyond that. We don’t want to invent ways of implementing things we will never do.

On the other hand, we do want to invent quite a few abstractions during this first iterations so that we end up with a coherent set of them that will compose together in an infinite variety of ways.

The output of sprint zero is usually a diagram showing the wiring of instances of abstractions, together with configuration information for those instances.

It doesn’t matter if some of the requirements are wrong. Chances are they are still useful for scoping out the domain space. What we are actually producing in this phase are the necessary abstractions for the Domain Abstractions layer. If the requirements change later, it will be trivially simple to change them as only the Application layer wiring should change.

Once this process has started to become easy, which should happen within the first sprint, the burning question in our minds will become “Will all this actually work?” We have to trust that there will be a way to write those abstractions to make the whole system execute.

5.2.2. How to invent Domain Abstractions

The most difficult part for people new to ALA is the skill of inventing the domain abstractions. You can fully understand the theory of ALA and why we need to invent abstractions, and still find it tricky to actually invent them. That’s because in this step in the process, we have to literally become inventors who are adding to the state of the art within your domain.

So the biggest insight I can offer here is don’t expect it to come without effort. Put your inventors hat on. Deliberate about the problem over a long period of time. Leave it overnight. Come back to it afresh in a week. Be patient.

I sometimes find that a solution will pop out of nowhere while reading about something completely unrelated. Once the human mind has been deliberating for some time, it remembers the whole context of the problem, and seems to carry on pattern matching for potential solutions in the background. I am reminded of the story of the Dam Busters. The British wanted to bomb the German dams to flood the industrial valleys below them. They worked out that they could deliver the bombs to the face of dam by skipping them across the surface of the lake. But they couldn’t get their bomber low enough to the water at night to get them to skip, because they had no way of knowing their height above the water.

Then one of the pilots was attending a play. He saw the spotlights pointing at the actor from different angles. And suddenly he had the solution. By mounting spotlights at the nose and the tail of the aircraft at a certain angle, they would know when they were at the correct height when the two spots came together at the surface of the water.

In my experience, there is always a solution, and it is always worth the effort. Just as good inventions are like gold, good abstractions are like gold in your application.

This section offers some practical tips on where to start.

  • You start by simply expressing requirements. You are inventing domain abstractions to allow you to express the requirements succinctly, but including all their details.

  • Draw a diagram that describes the requirements. You will draw boxes with lines. The boxes are your invented domain abstractions. The lines are your invented programming paradigms.

  • Look at the English version of the requirements for words that recur. These are candidates for abstractions.

  • Your diagram should have about the same amount of information as the English requirements. It will be succinct, even though it will include lots of detail such as text strings that come from requirements. Anything else that even begins to look like implementation or how it will work or execute is the beginning of an idea for an abstraction. Don’t worry about them. Just concentrate on making a description of requirements.

  • Always keep in mind that reusability and abstraction are two sides of the same coin. Your invented abstractions are anything that is potentially reusable.

  • Start with the user interface. Sketch multiple parts of the UI to make it more concrete. Then it is relatively easy to invent abstractions for your UI. They are things that will recur in different parts of the UI that you will always want to be done the same way. We can draw inspiration from the many widgets we already find in UI frameworks. ALA will often have equivalent ones that are just a little more specialised to your domain. Buttons may have a consistent size and style in your domain. Or your domain may need Softkeys, which are not usually part of a UI framework.

    ALA UI abstractions will usually be composed together in a tree structure representing the containment structure. This is similar to the tree structure of XAML, but in ALA we will usually do it as a diagram that is just one part of the entire application diagram.

  • Once you have designed some UI, you will then want to connect the UI elements that display data to some data sources. These data source are candidates for abstractions. For example, a data source that represents a disk file can be an abstraction that handles a disk file format. You will start to have dataflows between instances of data sources, be they UI elements or other source/destinations.

  • Between your data end points, data may need to be transformed, aggregated, filtered, sorted, validated or transacted (transacted means either all of it or none of it). All of these are great candidates for domain abstractions.

  • Sometimes a data source or destination will involve a protocol. A protocol is a domain abstraction. One abstraction should be invented to know about the protocol so that no other abstraction needs to know anything about the protocol. Sometimes there are protocols on top of protocols. For example, on top of a serial data stream protocol such as line terminated text, you may have another protocol that specifies the expected content of the first line.

    This same idea applies to file formats. A file format, such as a CSV file becomes an abstraction. If there is further formatting expected on top of the basic CSV format, such as a header row, that becomes a second abstraction.

    As with all abstractions, these abstractions know all about the details of something, e.g. a protocol or a format, and become the only thing in the entire application that does know about it.

    As a result these types of abstractions will usually handle the data going in both directions - sending and receiving, reading and writing.

  • If you have hardware devices, each will become an abstraction. For example, an ADC device will become a domain abstraction. The abstraction will know all the detail about the device (everything that is in the datasheet for the device). No other part of the program will know these details.

    If you have an ADC device that has an SPI interface, that will become two abstractions, one that knows all the details of the device, and one that knows all the details of the basic SPI interface.

  • Sometimes a section of requirements will seem like it should become a 'module' or 'function' - for example to parse a string. Try to turn the module or function into something more generally useful. Even if you still end up having only one instance of it for now, by separating that module or function into a general part and a configuration part (that has the information for it’s specific use in the one part of your application), you will make the general part easier to know what it does - simply because it is more abstract. In time you will often generalise it further and start to have instances of it elsewhere.

  • If all else fails, just start writing code that implements the requirements in the old fashioned way, not worrying about how messy it gets. When it is functioning as you want, then refactor it to ALA as follows. This is actually quite a straightforward process.

    Any code that has details that come from requirements, move that to the application layer, leaving behind generalised functions or classes that have parameters or configuration properties that the application will pass in.

    At first the generalised classes may have only one instance each. Look for ones that have similar functionality and combine them. The difference between them become further configuration properties, sometimes in the form of lambda expressions.

    Now refactor the generalised classes or functions so that they do not call each other directly.

    • In the case of functions, this step will involve adding new 'wiring' parameters that themselves take functions. The application will pass in the function that it needs to be wired to. These wiring parameters will usually either be to pull the input or push the output. The wiring function signatures should be even more generalised to allow Lego-like composition of functions by the application. They become interfaces in the programming paradigms layer. Section 1.11 has a worked example of refactoring of functions to ALA.

    • In the case of classes, this step will involve adding dependency injection setters. These will be used by the application to specify what class and instance it will talk to. This step removes all uses of the "new" keyword from the class (except ones that are instantiating classes in a lower abstraction layer such as your framework).

      If you have any uses of the observer pattern (publish/subscribe pattern), move the code that does the actual registering or subscribing up to the application. Provide a dependency injection setter for it to use.

      The dependency injection setters should all take generalised interfaces as thier parameter to allow Lego-like composition of class instances by the application.

Your classes and functions are now proper abstractions because they don’t know anything about the outside world, including who they are wired to, making them reusable.

5.2.3. 2nd Sprint

In the second sprint we start converging on normal Agile. You pick one feature to implement first. Agile would say it should be the most essential feature to the MVP (minimum viable product), but this can be tempered by the need to choose one that requires a fewer number of abstractions to be implemented. Next, you design the interfaces that will allow those abstractions to plug together according to the wiring you already have from the first sprint. What messages will these paradigm interfaces need to pass at runtime between the unknown clients to make them work?

It may take several sprints to produce the first working feature, depending on the number of abstractions it uses.

At first this sounds as if it might be just the waterfall method reincarnated. Do an overall design, document it or model it, and then write lots of code before everything suddenly starts working. But the design we created in iteration zero is very different from what a normal waterfall would produce, and is resilient to the sorts of problems waterfall creates. Instead of a ‘high level’ design of how the implementation will work which is lacking in detail, the design is a representation of requirements, in full detail. The design is not a model. It is executable.

There is one more important thing that the design phase in Iteration Zero does. While it deliberately doesn’t address any implementation, it does turn the remainder of the implementation into abstractions, and those abstractions are zero coupled. To convert from executable to actually executing, it only remains to implement these now completely separate parts. You can give these abstractions to any developer to write. Together the developers will also easily be able to figure out the paradigm interface methods needed to make them work, and the execution models to take care of the execution flow through them with adequate performance.

Often when a project is split into two phases, the first phase turns out to be waste. The devil is in the details so to speak. This happens because the implementation details in phase two are coupled back to and affect the design in phase one. As learnings take place during implementation, the design must change. In ALA the output from phase one is primarily abstractions, which are inherently stable and therefore hide details that can’t affect the overall design. If the abstractions are good, phase two will typically have little effect on the work done in phase one.

Once the first feature is working, several abstractions will have been implemented. The second feature will take less time because some of the abstractions are already done. In ALA velocity increases as time goes on and keeps increasing until new features only involve instantiating, configuring and wiring domain abstractions in new ways. This velocity acceleration is the complete opposite of what happens in monolithic code.

5.2.4. Later sprints

Imagine going into a sprint planning meeting with a Product Owner, a small team of developers, and a mature ALA domain that already has all the common domain abstractions done. As the Product Owner explains the requirements, one of the team members writes them down directly as they would be represented in terms of the domain abstractions. Another team member watches and remembers any lost details without slowing the product owner down. A third member implements the acceptance tests in similar fashion, and a fourth provides him with test data. It would be nice to have a tool that compiles the diagram into the equivalent wiring code. With such a tool, the team could have it executing by the end of the meeting. At the end of the planning meeting the development team say to the product owner "Is this what you had in mind?". The team can get immediate feedback from the Product Owner that the requirements have been interpreted correctly.

Of course, the planning meeting itself would only produce 'normal' functionality. Usually it is up to the development team, not the Product Owner, to uncover all the abnormal scenarios that can happen, and that is usually where most of the work in a software system goes. Having said that, in a mature domain, the validation of data already has decorator abstractions ready to go.

effort curve

The graph shows the effort per user story against months into a green-field project. The left axis is arbitrary - the shape of the curves is what is important. For a big ball of mud, experience tells us that the effort increases dramatically and can asymptote at around 2 years as our brains can no longer handle the complexity, and the project must be abandoned.

The COCOMO model, which is an average of industry experience, has a power relationship with program size, with an exponent of around 1.05 to 1.2. I have used the mid point, 1.1, for this graph. The model appears to imply that getting lower than 1.0 is a barrier, but there is no reason to believe this is the case. Reuse can make the power become less than 1. The range of 1.05 and 1.2 probably results from some reuse mitigating some ever increasing complexity.

ALA takes advantage of the fact that zero-coupled abstractions can keep complexity relatively constant and drastically increase reuse. A spectacular fall in effort per user story is thus possible.

5.3. Technical debt

Technical debt is real. Convincing your non-technical manager that it is real is not realistic, especially when you tell him that this is based on estimates. He already knows about your 'estimates'. Even if you could measure it, and offer your manager a payback period for a one-day investment in refactoring, there is a problem. He is measured on this sprints performance. Or he is measured on some other criteria such as this years budgeted sales. Technical debt is tomorrow’s problem, and that is not his problem, not ever.

A better answer would be to not generate technical debt in the first place. In Agile we are taught that the architecture depends on the what we learn as we write the code. The Definition of Done is supposed to tell us to clean it up as we learn. But it is also possible that we will learn something that requires previous sprint’s of work to be cleaned up. A refactoring could need changes to the structure at the largest granularity level.

So here is a better solution. What if, as far as possible, we know a meta architecture that all software programs should follow. It is a big scale meta-architecture, so it gets things into the right places in the big scale.

So now you just get it right in the first place. Isn’t that better?

Now some will argue that they do that already. Their large scale structure is layering, or it is MVC or it something else. I have been told that this doesn’t work because we fail to actually keep to the prescribed architecture. But think it is because up till now, these architectural patterns have not worked very well.

5.4. Folder structure

[tree,file="folderstructure.png"]
root
|--Application1
|  `--application.cpp
|--Application2
|  `--application.cpp
|--DomainAbstractions
|  |--abstraction1.cpp
|  |--abstraction1.h
|  |--abstraction2.cpp
|  `--abstraction2.h
`--ProgrammingParadigms
   |--Paradigm1.h
   `--Paradigm2.h

This is a suggested folder structure for ALA. Because ALA does not use decomposition, you don’t end up with components that are contained by the applications, so there are no subfolders under the application. Instead, you end up with Domain Abstractions outside the application, so they go in their own folder in a flat structure.

Similarly, the Programming Paradigms code is not contained by an application, or even by the domain, so would not be contained by the domain’s projects folder.

5.5. Convention over configuration

When the application create an instance of an abstraction, most of the configuration of that abstraction should have defaults. In ALA, setters allow optional configuration, reducing the amount of information that would otherwise be required in the application to fully configure each abstraction. Any configuration that we wish to enforce goes into the constructor.

There is a counter argument that says that all configuration should be explicit so that nothing can be forgotten. ALA prefers optional configuration because we want the application to just express the requirements. Also optional configuration allows abstractions to default to their simplest form, making them easier to learn.

5.6. Knowledge prerequisites.

When other programmer are doing maintenance on your code, you should make sure they have the knowledge they need. They will need knowledge of ALA. They will need to know about the programming paradigms used. They will need to know about the domain abstractions, and the insight of what each one does. And then they should know that the application diagram is the source code. It is up to you that every develop that follows will know all this.

5.6.1. Intellisence

After they have modified the diagram, the maintaining developers will need to manually modify the corresponding code. Here they will see instances of abstractions being used all over the place, either 'new' keywords or function calls. If we have done our job with knowledge prerequisites, they will have been introduced to these abstractions. However, it doesn’t hurt to have brief reminders of what they are pop up when the mouse is hovered over them. So put in triple slash comments (or equivalent) describe the abstraction succinctly, with the intention of it being a reminder to someone who has already met the abstraction. Put a full explanation in the remarks and examples sections.

The class name after a new keyword is actually the constructor name, so you must duplicate the summary section there. Often in ALA, the class name is not referred to at all in the application.

5.7. Two roles

ALA requires two roles. Both can be done by the same person, but always he should be wearing only one hat at a time. There is the role of the architect, and the role of the developer.

The role of the architect is harder than that of the developer, that’s why we have the role. Expect it to be hard. Perhaps, surprisingly, the architect’s main job is to focus on the requirements, and the developers main job is to implement the abstractions (which know nothing of the requirements). In describing the requirements, the architect invents domain abstractions.

The architect also has a role in helping the developer to design the interface’s methods. In other words, how at runtime the system will be made to work.

This aspect of ALA can also be difficult at times. I have sometimes got stuck for a day or so trying to figure out how the interfaces should work, while still keeping them more abstract than the domain abstractions. The ALA constraints are that these interfaces should work between any two domain abstractions for which it may be meaningful if they are composed together in an application. However, the problem has always been solvable, and once solved, it always seems to have a certain elegance, as if you have created a myriad of possibilities at once. Implementation of the interfaces by the relevant domain abstractions becomes easy. Development then proceeds surprisingly quickly.


5.8. Example project - Calculator

This project was originally done in a hurry for an ALA workshop. Apart from being a cool example of the use of ALA, the calculator itself is cool. This calculator is in Github, as a work in progress here: https://github.com/johnspray74/ReactiveCalculator

The original development for the workshop was done in about a day, so here we tell the story of that development.

When I was first asked to do the workshop, I needed to think of a suitable pedagogical sized project. It was suggested to do a calculator. Ok, I thought, if we have the domain abstractions already in place before the workshop, we should be able to write a calculator application (top layer) during the workshop.

When we think of a calculator application, we usually imagine a user interface that mimics a handheld calculator. It has a one line display and a keypad on the screen. We certainly could have built that calculator (primarily using a state machine programming paradigm). But that problem has already been solved by Miro Samek. Besides, I hate those simulated handheld calculators. I think they are a stupid way to do calculations on a computer. Such a calculator would be cumbersome.

As an aside, I once loved HP calculators. The first programmable anything I ever owned was an HP65 calculator. I have owned many top end models at one time or another. But as their displays got larger, I became more and more disappointed with how they used that display real-estate. They just used it as a stack.

The calculator I wanted would show the expression you had entered so you can check it. It could show it in algebraic or even textbook form, even though you had entered it as RPN. You could re-edit the expression. You could label your expression result and then you could use the result in another expression. When you changed something, all results would be updated like in Excel. This is what the HP prime should have been.

So I drew this sketch of what I wanted a calculator to be:

Calculator3.png
Figure 28. Calculator requirements

Here is a screenshot of the working calculator as it was two half-days later:

CalculatorScreenshot.png
Figure 29. Calculator screenshot

The first step in the design of the calculator was to express the requirements (UI plus behaviours), inventing any needed abstractions to do so. Here is the actual first sketch I drew:

Calculator2Rows-HandDrawing.png
Figure 30. Calculator (2 Rows only shown)

As we shall see, this diagram is practically executable code. The calculator is practically done. We don’t know if the invented domain abstractions will work yet, but let’s go through how this calculator works anyway.

5.8.1. How the calculator works

First notice how the entire calculator is there. Every detail for this particular calculator is represented in the diagram. All those details cohesively describe the calculator. This is an important aspect of ALA designs. All the UI and all the dataflows to make a working calculator are in this one diagram. What is not here is the details we left to the domain abstractions. None of these abstractions is specific to a calculator. They can be used for all sorts of things. Even the Formula abstraction would be useful in any application where a calculation needs to be changed at run-time. For example an insurance application may need configurable calculations.

The left side shows two instances of the abstraction, Horizontal. These arrange their children horizontally in the UI. To the left of those (not shown) is an instance of Vertical, which arranges the two Horizontals vertically. And to the left of that, also not shown, is an instance of MainWindow.

Each Horizontal has 3 instances of TextBox and one Text. The TextBoxs allow you to enter a string. The abstraction, Text, can only display a string. I see at this point, I hadn’t put in the TextBox for the description.

We can ignore the use of labels for a moment and just enter a formula containing constants into the first row TextBox. Let’s say we type in "2+1". The string "2+1" goes along the dataflow wire to the StringFormat instance on its port 0. The StringFormat is configured with the format string "({1}⇒{0})". StringFormat uses this format string in the same way as an interpolated string in C#. The {0} is substituted with the string coming in on port 0. The {1} is substituted with the string coming in on port 1. Since we have no string on port 1, the output from the StringFormat will be "()⇒2+1". This is simply a lambda expression with no parameters. This string is fed into the instance of Formula. Formula is an abstraction that knows how to evaluate a lambda expression. Actually it will accept just a formula string (such as "2+1") as well. We can ignore the other input of Formula for the moment. Formula will evaluate "()⇒2+1" and produce the number 3 on its output. This output is a dataflow of type double. This number is fed to an instance of NumberToString, and from there it goes to the instance of Text that knows how to display a string.

Now let’s follow the use of labels in the calculator. Let’s put the labels "a" and "b" into the TextBoxs for labels on the two rows. "a" and "b" are fed to the Concat instance. Concat’s input port can have any number of string dataflows wired into it. In this diagram it has only two. What Concat does is concatenate all its inputs adding a separator. In this case the separator is configured to be a comma. The output of the Concat is "a,b". The concatenated list of labels is fed into port 1 of both StringFormat instances.

Now let’s put the formula "a*3" into the 2nd row of the calculator. The output of the StringFormat for that row will be "(a,b)⇒a*3". That lambda expression will be fed to the Formula instance, which will evaluate it, using the value on its first input port for the value of 'a'. The output will appear on the corresponding Text in the 2nd row.

So that’s all there is to understanding how the calculator works. At this point it takes a leap of faith that the abstractions can all be made to work, and that the two programming paradigms used, the UI layout and the dataflow, can be made to work. Not withstanding that, all the information required in the design of the calculator is captured.

At this point I drew little drawings of all the invented abstractions. Actually I reused TextBox, Text, Vertical, Horizontal and FormatString from a previous project. And I had already made the UI and dataflow programming paradigm interfaces in previous projects, so I reused them as well.

CalculatorDomainAbstractionsHandDrawings.png
Figure 31. Calculator Domain Abstractions

I see I forgot to draw Concat. I had to write that one. Here is its template as it was in my Xmind templates diagram.

CalculatorStringConcatTemplate.png
Figure 32. StringConcat template

The two ports are both drawn on the right hand side unfortunately - a limitation of using Xmind as the drawing tool. Drawing the templates makes the abstractions clearer for their implementation.

5.8.2. Implementing the domain abstractions

Abstractions are generally trivial to implement because they are zero coupled with anything. They are like tiny stand-alone programs. Here is the full code for StringConcat.

SringConcat.cs
using ProgrammingParadigms;
using System;
using System.Collections.Generic;
using System.Linq;

namespace DomainAbstractions
{
    /// <summary>
    /// Outputs the input strings concatenated together
    /// Whenever an input strings changes, a new output is pushed.
    /// ---------------------------------------------------------------------------------------------------------
    /// Ports:
    /// 1. List<IDataFlowB<string>> inputs: inputs (indefinite number of string inputs)
    /// 2. IDataFlow<string> output: output
    /// </summary>



    public class StringConcat
    {
        // Properties ---------------------------------------------------------------
        public string InstanceName { get; set; } = "Default";
        public string Separator { private get; set; } = "";

        // Ports ---------------------------------------------------------------
        private List<IDataFlowB<string>> inputs;
        private IDataFlow<string> output;


        /// <summary>
        /// Outputs a boolean value of true when all of its inputs are true. Null inputs are treated as false.
        /// </summary>
        public StringConcat() { }

        // This function is called immediately after each time the inputs port is wired to something
        private void inputsPostWiringInitialize()
        {
            inputs.Last().DataChanged += () =>
            {
                var result = "";
                bool first = true;
                foreach (IDataFlowB<string> input in inputs)
                {
                    if (!first) result += Separator;
                    first = false;
                    result += input.Data;
                }
                output.Data = result;
            };
        }

    }
}

The code in Concat is straightforward if you know C#, except for a few conventions which are to do with the use of ALA (which I was already proficient with):

  1. We put a property "InstanceName" in every domain abstraction. It’s not required, but the reason is because abstractions get reused. So you are likely to end up with multiple instances of an abstraction all over your application. If you name the instances, it makes debugging a lot easier because you can see it in the debugger and know which instance you are in.

  2. All the IO for the abstraction is in a section called "Ports". Usually an abstraction would have some ports that are private fields of the type of an interface, and some that are an implemented interface. It just so happens that StringConcat has no implemented interface ports.

  3. The two ports are private, and yet they get wired by the application code to other objects. This may seem a little bit magic. The reason they are private is to indicate they are not for use by anything else. The application will use a method called WireTo() to achieve the wiring. WireTo is an extension method on all objects. It uses reflection to find and assign to these "port" fields.

  4. Notice that the "inputs" port uses IDataFlowB (not IDataflow). The B on IDataflowB indicates a tricky workaround for a limitation in the C# language. What we would have liked to do is implement IDatFlow. But we would have needed to implement IDataFlow multiple times. You can’t do that in C# (although there is no reason why not in theory, and hopefully all languages will have this feature to support the concept of ports one day).

  5. Notice that the method, inputsPostWiringInitialize, is private and apparently unused. When the WireTo operation wires a port "xyz" it looks for a private method called xyzPostWiringInitialze and invokes it immediately. This gives us the opportunity to set up handlers on any events that may be in the interface. In this case, the IDataFlowB interface has a DataChanged event (that tells us when there is new data on the inputs).

  6. It doesn’t make sense to use a StringConcat without wiring its output to something. So the line "output.Data = result" will throw an exception if the application has not wired it. Often times, abstractions have ports that are optional to wire, in which case we would use "outputs?.Data = result"

For reference, here is the IDataFlow interface, which lives in the ProgrammingParadigms folder, and is used by most of the domain abstractions including the StringConcat abstraction:

IDataFlow.cs
namespace ProgrammingParadigms
{
    public interface IDataFlow<T>
    {
        T Data { get; set; }
    }
}

As you can see, this interface is simple, but its importance in giving us a dataflow programming paradigm is huge. It allows objects to be wired together so that data can flow from object to object without the abstractions knowing anything about each other. Note that IDataflow uses a 'push' execution model. This means that the source always initiates the transfer of data on its output (by using 'set'). The IDataFlowB interface is a little more complicated, so we wont go into how it works just now. But it achieves exactly the same purpose of the dataflow programming paradigm, just in a different way that allows more than one input port of the same type. .

The other domain abstraction I needed to write for the first time was Formula. Here is the template as used in Xmind:

CalculatorFormulaTemplate.png
Figure 33. Formula template

Once again, one of the input ports is shown on the right when we would prefer it to be on left.

Formula can be configured with an optional C# lambda expression when it is instantiated by the application, for example:

Application.cs
new Formula() {Lambda = (x,y) => x+y; }

If used in this way, the formula is fixed at design-time. But its real power comes from the fact that it can take a formula as a string at run-time. Formula has an input dataflow port called "formula".

The Formula abstraction has to parse the formula string and then evaluate it. I used Roslyn to do this in a few lines of code. These lines of code took me a few hours to figure out however:

Formula.cs
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;
using static System.Math;

namespace DomainAbstractions
{
    using LambdaType = Func<double, double, double, double, double, double, double>;

    public class Formula
    {
        public LambdaType Lambda { private get; set; }


        // Other code omitted from here


        private async void Compile(string formula)
        {
            var options = ScriptOptions.Default;
            options = options.AddImports("System.Math");
            try
            {
                Lambda = await CSharpScript.EvaluateAsync<LambdaType>(formula, options);
            }
            catch (CompilationErrorException e)
            {
                Lambda = null;
            }
        }
    }
}

As you can see, currently the Formula abstraction can only handle formulas that use a maximum of six parameters. The calculator application can use any number - it’s just that any one formula is limited to using only six.

The rest of the code in the Formula abstraction is mostly dealing with this requirement of exactly six parameters used by LambdaType. But that’s all internal to the abstraction. An instance of the Formula abstraction can handle any number of operands from zero to many. The Formula abstraction is also tolerant of the string on the formula input port being either just a formula such as "2*(3+1)" (implying it’s not using any operand inputs), or a proper lambda expression such as "(x,y,z)⇒x*(y+z).

Now that we have our needed domain abstractions, let’s return to the application layer, and see if we can get this calculator running.

5.8.3. Hand wiring the application code from the diagram

First here is the startup code for a wired ALA application. It is very simple:

Application.cs
namespace Application
{
    public class Application
    {
        private MainWindow mainWindow;

        [STAThread]
        public static void Main()
        {
            Application app = new Application();
            app.mainWindow.Run();
        }

        private Application()
        {
            // mainWindow = WireHelloWorld();
            mainWindow = Calculator2RowHandWired();
        }
    }
}

Here is an ALA Hello World application

Application.cs
private MainWindow WireHelloWorld()
{
    return new MainWindow().WireTo(new Text("Hello World"));
}

Ok, now we are ready to hand wire the hand drawn calculator diagram shown above:

Application.cs
private MainWindow Calculator2RowHandWired()
{
    // To understand this code, you need the wiring diagram of the two row calculator

    // First instantiate instances of abstractions we need to give names to. The rest can be anonymous.
    StringConcat stringConcat = new StringConcat() { Separator = "," };
    DataFlowConnector<string> stringConcatConnector = new DataFlowConnector<string>(); // Connectors are needed when there is fan-out or fan-in in the diagram
    stringConcat.WireTo(stringConcatConnector, "output");
    Formula[] formulas = { new Formula(), new Formula() }; // instantiate both the formulas up-front because we need to cross wire them


    MainWindow mainWindow = new MainWindow("Calculator")
        .WireTo(new Vertical()
            .WireTo(WireRow(stringConcat, stringConcatConnector, formulas[0], formulas))
            .WireTo(WireRow(stringConcat, stringConcatConnector, formulas[1], formulas))
            );
    return mainWindow;
}


private Horizontal WireRow(StringConcat stringConcat, DataFlowConnector<string> stringConcatConnector, Formula formula, Formula[] formulas)
{
    // To understand this code, you need the wiring diagram of the two row calculator

    // first instantiate objects we need to give names to.  The rest can be anonymous.
    Text result = new Text();

    // Wire up a calculator row
    Horizontal row = new Horizontal()
        .WireTo(new TextBox()
            .WireTo(new DataFlowConnector<string>()
                .WireFrom(stringConcat, "inputs")
            )
        )
        .WireTo(new TextBox()
            .WireTo(new StringFormat<string>("({1})=>{0}")
                .WireTo(stringConcatConnector, "inputs")
                .WireTo(formula
                    .WireTo(new DataFlowConnector<double>()
                        .WireFrom(formulas[0], "operands")
                        .WireFrom(formulas[1], "operands")
                        .WireTo(new NumberToString()
                            .WireTo(result)
                        )
                    )
                )
            )
        )
        .WireTo(result)
        .WireTo(new TextBox());
    return row;
}

Commentary on this wiring code

  • The code is written in "fluent style". This is possible because the WireTo extension method returns its 'this' parameter, allowing you to use .WireTo() multiple times on an instance.

  • The parts of the diagram that have a tree structure can be wired by using more .WireTos immediately inside the brackets of another WireTo. This is what causes the indented code.

  • The previous two points allow much of the wiring to be done without having to think up names for the instances. The instances are anonymous just as they were on the diagram.

  • Where the diagram has cross wires that formed a cycle, we need to give instances names so that we can complete all the wiring to them.

  • Sometimes the WireTo method is given the port name of the first object. These are used when WireTo may get the wrong port if it were left to use the port types alone. (If an abstraction has multiple ports of the same type, WireTo doesn’t know which port to use.)

  • Most abstractions have ports that can only be wired to one place, and ports that can only be wired from one place. (The UI abstractions are exceptions, for example, Horizontal can be wired to multiple children.) The hand drawn wiring diagram has several places where the wiring either 'fans out' from a port or 'fans in' to a port. Unfortunately C# does not support the concept of ports, so we improvise to make them work in normal C# code. The way this improvisation works for fan-in and fan-out is to use a connector.

    • For fan-out you wire the output port to a new Connector. Then you can wire the connector to multiple places.

    • For fan-in, the abstraction, instead of implementing the port, uses a list field of a type like IDataFlowB. The 'B" on the end means the flow of data is reversed from IDataFlow. Now to wire such an inputs port to an output port also requires a connector (a second use for connectors). When wiring an IDataFlowB, you wire from the input to the connector (the opposite way to the direction of dataflow unfortunately).

  • As a convenience, you can use WireFrom. It reverses the two operands being wired compared to WireTo.

Handwritten wiring code like the above can be managed for small applications. It is easy to get it wrong though. The code is not readable from the point of view of understanding the application. That’s what the diagram is for. So every time you make a change to the requirements, you need to do it on the diagram first, then update the hand-generated code. You are still better off doing this than not using ALA. If not using ALA, the relationships of the cohesive, explicit ALA diagram still exist, but they become obscurred and distributed inside your modules.

But we don’t have to hand generate code. At the time of writing a hand written graphical IDE is under development.

Here is the calculator row part of the diagram, which was successfully used to auto-generate code:

CalculatorRow-GALADE.png
Figure 34. CalculatorRow abstraction internal wiring

Before we had that tool, we used Xmind to do diagrams from which we could generated code. This is the approach described in the next section.

5.8.4. Automatic code generation from the diagram

Here is the diagram of the 2-row calculator as it was originally entered into Xmind. You can click on the image to get a bigger version.

Calculator2Rows.png
Figure 35. Calculator drawn in Xmind (two rows version)

Xmind is not the perfect tool to do this, but it has one huge advantage - it lays itself out. This is so important that it’s why we use it (until our new tool is ready). We will put up with the disadvantages, and the necessary conventions until then. Those conventions are documented in the wiki section of the project in Github here: https://github.com/johnspray74/ReactiveCalculator/wiki

Entering the hand-drawn version of the diagram is a simple matter of copying and pasting the Xmind templates for the abstractions to the right place in the diagram. This connects most of the needed wiring from port to port. Xmind supports tree structured diagrams, so any cross tree wiring was done by using the red lines, which are quick to put in with a shortcut key.

The Xmind version of the diagram is pretty much identical to the hand drawn version. The colored boxes are instances of abstractions. All the other nodes attached around those colored boxes are the ports. The < and > signs in the ports are significant, and tell the automatic code generator which way to wire the instances. The asterisks are also significant, and tell the code generator that many wires can be wired to the one port.

The diagram done, it’s time to generate the code. A tool called XmindParser does this. It can be downloaded from the Github project main page. Here is the tool in use to generate a calculator from the diagram.

CalculatorXmindParserTool.png
Figure 36. XmindParser tool in use

The tool can put the generated code into your Application.cs file if you give it special markers like the one below.

Application.cs
private void Calculator2Rows()
{
    // BEGIN AUTO-GENERATED INSTANTIATIONS FOR Calculator2Rows.xmind
    // END AUTO-GENERATED INSTANTIATIONS FOR Calculator2Rows.xmind

    // BEGIN AUTO-GENERATED WIRING FOR Calculator2Rows.xmind
    // END AUTO-GENERATED WIRING FOR Calculator2Rows.xmind
}

The markers contain the name of the Xmind diagram they get code from. This allows several diagrams to be used for one application.

I usually put the markers inside a function so that all the instantiated objects are private to the function.

Here is the code again with three lines of generated code shown in each section.

Application.cs
private void Calculator2Rows()
{
    // BEGIN AUTO-GENERATED INSTANTIATIONS FOR Calculator2Rows.xmind
    Formula Formula1 = new Formula() { InstanceName = "Formula1" };
    Formula Formula2 = new Formula() { InstanceName = "Formula2" };
    Horizontal id_24914ab245484fe1b70af8020ca2e831 = new Horizontal() { InstanceName = "Default" };
    // END AUTO-GENERATED INSTANTIATIONS FOR Calculator2Rows.xmind

    // BEGIN AUTO-GENERATED WIRING FOR Calculator2Rows.xmind
    mainWindow.WireTo(id_b02d2caea938499b997b9bfcb80fb0e9, "iuiStructure");
    id_b02d2caea938499b997b9bfcb80fb0e9.WireTo(id_24914ab245484fe1b70af8020ca2e831, "children");
    Formula1.WireTo(dfc1, "result");
    // END AUTO-GENERATED WIRING FOR Calculator2Rows.xmind

}

Completing the diagram had taken another morning of work. But the calculator was now working and I was ready for the workshop.

I made a diagram with six rows, but it was getting pretty large, and the duplication was pretty clumsy.

5.8.5. Calculator with 10 rows

The 6-row calculator is powerful compared to any normal calculator, but still wouldn’t do jobs like the one shown here which has 11 rows:

CalculatorNRowsScreenshot.png
Figure 37. Later version of the calculator to support an unlimited number of rows

So it was time to do some maintenance, and solve that problem of the repetition in the diagram at the same time. During this maintenance, none of the existing abstractions changed. Their ports stayed the same. Their internals were improved a little in some cases but nothing significant. Formula was modified internally so that it could handle more than six operand inputs, although any one formula can still only use six of them.

There were two major changes though. One was to put the repeated wiring for a calculator row inside its own abstraction. This abstraction is called "CalculatorRow". This abstraction is less abstract than the Domain abstractions it uses, but more abstract than the application, which will use it multiple times. Other version of the calculator alos reused it. So it goes into its own layer between the two. This new layer is called "Requirements Abstractions". We don’t make new layers lightly, but we had had the experience in a larger project that the diagram got too large. We needed to factor out some of it as 'Feature level abstractions'. These abstractions needed a new layer between the Application layer and the Domain abstractions layer. So I was reasonably happy to make use of this new layer in the calculator to reduce a now quite large diagram.

Here is the Xmind template for the CalculatorRow abstraction showing its ports:

CalculatorRowTemplate.png
Figure 38. CalculatorRow abstraction

The implementation of CalculatorRow was done with a diagram using the new Graphical ALA tool which was shown earlier.

When you implement an abstraction by an internal diagram, there needs to be some extra code to wire from the ports (shown in the template above) to the internal wiring. I found this code quite tricky the first time and it took me a while. But there is a pattern to it. The new tool will be able to automatically generate this code as well, but for now I did it by hand.

I tested the completed CalculatorRow abstraction by making an application that uses it twice:

Calculator2ARows.png
Figure 39. Calculator drawn in Xmind using the new CalculatorRow requirements abstraction

Because this test calculator has only two rows we can directly compare it with the 2-row calculator above and see how the two rows are now represented by two instances of this new CalculatorRow abstraction. However we are still not there to building a calculator with 10 rows. If we were to instantiate CalculatorRow 10 times, it would need 100 wirings in Xmind to connect all ten results to every CalculatorRow’s operands input. Having thought the 36 wirings of the 6 row calculator was nuts, there was no way I was going to do 100 of them by hand.

So the next step was to invent a domain abstraction called 'Multiple'.

What 'Multiple' does is you instantiate it in your application and configure it with the number you want. In this case N:10. Multiple has a port called factory which uses a new interface called IFactory. Then what you do is add a small Factory class inside any abstraction that you want multiple instances of. In this case we want multiple instances of CalculatorRow, so I added a small class to that abstraction called CalculatorRowFactory. CalculatorRowFactory implements IFactory (which resides in the Programming Paradigms layer). Multiple can now create many instances of the abstraction that is wired to it. But these instances now have to be wired into the rest of the wiring in the application diagram as needed. To accomplish that, multiple is configured with two methods. These methods, which are part of the application, know how to wire the new instances into the rest of the wiring.

Here is the Xmind template for Multiple:

CalculatorMultipleTemplate.png
Figure 40. CalculatorRow requirements abstraction

Using the new Multiple abstraction, we can now build a 10-row calculator. Here is the new diagram:

Calculator10Rows.png
Figure 41. Calculator drawn in Xmind using the new Multiple domain abstraction to create as many rows as we like

You can see inside the instance of Multiple that it is configured to make 10 of whatever is attached to its factory port. The CalculatorRow abstraction is shown next to the CalculatorRowFactory so you can see what it makes. You can see inside Multiple the two methods that Multiple calls when it makes a new instance. These methods are used to wire the CalculatorRow into the rest of the application wiring.

Getting to this point had taken another two Saturday mornings of work. It was mainly spent on thinking out the patterns for how the internal wiring inside CalculatorRow should get wired to CalculatorRows own border ports. Also in the initial attempt I had allowed temporal coupling to creep in between abstractions. It mattered whether the application wiring was done first. And if you let coupling creep in bugs will happen. The temporal coupling was resolved by making CalculatorRow not care whether or not the external wiring is done when it is instantiated.

5.8.6. Calculator with N rows

Wouldn’t it be cool if the calculator started with say 4 rows, and had a button for adding additional rows? You will notice that the Multiple abstraction template above has a port called addRow. It takes an IEvent. If it receives an event on that port at run-time, it will create another row and call the lambda wiring functions to get that row wired in.

Here is the calculator with the button added to the UI. This is the calculator from which the screen shot shown above was taken.

CalculatorNRows.png
Figure 42. Later version of the calculator to support an unlimited number of rows

I wanted the Button to be at the bottom, so I added another instance of Vertical to act as the wiring point for new row to be attached to the UI. The output of the Button is another programming paradigm interface called an IEvent.

Here is the source code of IEvent:

IEvent.cs
namespace ProgrammingParadigms
{
    public interface IEvent
    {
        void Execute();
    }
}

Again, considering that this interface enables a whole programming paradigm, it is extremely simple and abstract, consisting of only a synchronous function call. It allows any objects that can produce events to be wired to any object that can receive an event. In the CalculatorNRows example, it allows a Button to be Wired to a Multiple. When the Button is pressed, Multiple creates a new row.

There is one major Programming Paradigm that we have not explained yet. It is the one that allows UI instances to be wired together. The meaning of wiring two UI objects together is that one contains the other. Here is the IUI interface that implements this programming paradigm.

IUI.cs
using System.Windows;

namespace ProgrammingParadigms
{
    public interface IUI
    {
        UIElement GetWPFElement();
    }
}

The interface simply allows a containing UI instance to get the WPF (Windows Presentation Foundation) element from the contained UI instance it is wired to.

Making Multiple able to add rows to the calculator at runtime took yet another Saturday morning. But it was worth it to solve the challenge of learning how to change the wiring at run-time. The wiring diagram now statically describes how it dynamically changes itself.

That concludes the story of the development of the calculator using ALA. The full source code, and all the diagrams for every version along the way are in the Github project at https://github.com/johnspray74/ReactiveCalculator

6. Chapter six - ALA compared with:

In this chapter, our perspective is to compare ALA with existing programming paradigms, principles, styles, and patterns.

The idea is to understand ALA in terms of similarities and differences with something you may already understand.

A programming paradigm, principle, style, or pattern generally takes a long time for the average developer to get used to and master. It then generally only makes an incremental improvement to software quality, if any. ALA is a reference architecture that combines the best elements of these existing programming paradigms, principles, styles, and patterns into one coherent idea.

Some programming paradigms, principles, styles, or patterns are just bad. The most prominent examples are the UML class diagram, the idea of 'decomposition' of a system (which includes all patterns like MVC), the idea that indirection is necessarily hard to trace, the idea of loose coupling, the idea of dependency management without distinguishing between good and bad dependencies, and the idea of layering patterns (sometimes called stacks) where the layers are driven by communication dependencies.

In this chapter we look at each of these good and bad patterns or styles in detail.

6.1. Monads

In chapter 3, we did a brief comparison of ALA and monads in which we assumed prior familiarity with monads. In this section we assume no familiarity with monads. So we start by explaining what monads are. I promise this will be the easiest explanation of monads you will find on the internet. I’ll be using the approach that monads are a refactoring pattern. We’ll first do some imperative code, and then do the equivalent refactored code. We will do the refactoring in several stages. We will point out the multiple advantages of the refactored code along the way. At the last stage the refactored code qualifies under as a monad.

Often abstract concepts in programming can be explained in terms of how they are syntactical sugar for a longer, imperative form of the code. For example, lambda expressions can be explained as syntactical sugar for passing in a named function. This is hard to do for monads, especially 'deferred' monads, because there is code generated under the hood. So most explanations of monads do not attempt to explain them in terms of equivalent imperative code. They go for an abstract or mathematical explanation instead. This makes these explanations very difficult to understand, especially for developers who are not so much mathematicians, but do understand imperative object oriented code just fine. I am going to show that code generated behind the scenes as well.

Monads are motivated by three problems in functional programming:

  1. Top layer code that calls functions must handle data for the functions. For example, data returned by one function is often passed to the next. This often involves creating local variables to hold data temporarily. But really we want to think of the top layer as just 'composing' functions. That’s a more declarative approach - just composing things together. When we compose functions we want the output of one to be piped to the input of the next without us having to do it explicitly. Looked at in a different way, the functions' interfaces for input/output (the parameter and the return value) are exposed to the top layer when they don’t need to be. We would rather think of this input and output as separate interfaces or ports. When the top layer composes two functions, these ports are wired up so that at runtime data is passed directly from one function to the next.

  2. Most programs use state because is often the best way to express a computation. This is especially true when the events coming into the system are asynchronous, for example coming from the outside world.

    In pure functional programming, this state ends up essentially in the top layer. It is passed into the pure functions. Returned values are stored back in the state variable. The state structure itself may be immutable, so that if another thread has a reference to it, it does not see changing data. But the stateful reference to the structure must be stored in place while the system waits for the next external event.

    Passing state into functions that essentially own the state is a responsibility that the top layer should not have. Functions that would otherwise be good abstractions if they were self-contained with their state get broken by exposing their private parts.

    Instead good abstractions should stay. The problem of their mutable state should then be handled by treating each instance (or a local group of instances) as a unit running on a single thread. These groups can then only communicate with one another asynchronously.tt

  3. Many times in functional programming when composing functions, the return value cannot be fed directly into the next function. Some common code is needed between functional calls. For example, if the function could have an error, if or try statements would be needed after every function call to check for the error.

    This type of common code should also not be the responsibility of the top layer. The top layer should just be about composing functions. This common code, which can take many forms, should be refactored out.

Monads allow the top layer to just compose the functions. The monad takes care of passing data from function to function, allowing the composed function to have state without the top layer code handling or knowing about the state, and doing any common code that needs doing after function calls. The top layer code remains pure functional code.

To accomplish all this, monads (usually) use objects under the covers. These objects can be delegates (object/function references), closures (objects of compiler generated classes that capture local scope variables), or specific under the hood classes. These objects are wired together to build a structure that can be executed in much the same way as ALA wires together domain abstraction objects to build a program that can be executed.

The motivating problems described above will still seem quite vague to the reader without code examples. So that is what we will do in the following few sections.

Most of the code snippets in this section are demonstrated by small executable projects on Github here:

The project names are IEnumerableMonad, ContinuationMonad, and MaybeMonad.

6.1.1. Composing functions

Let’s start by composing three functions in familiar imperative style.

In these examples, we will feed the number 42 into the first function, just to make it easy for you to see the source of the composed function chain:

int result = function3(function2(function1(42)));

Composing functions in this way is not scalable because of the increasing levels of nested brackets every time we want to add a new link to the chain. Also we cannot add statements such as if statements to control the flow if there is an error.

To get a scalable version, we can write it like this instead:

int r1 = function1(42);
int r2 = function2(r1);
int result = function3(r2);

This is representative of the basic style that most imperative code follows. There may be some logic statements thrown in amongst the function calls, but essentially most imperative code follows that style.

This form has the disadvantage of creating temporary local variables, which makes the entire scope more complicated because anywhere in the scope can potentially use any variable. The variable can be immutable, but that doesn’t help much.

Let’s try to get to a style where we don’t need to handle that data ourselves and we are just declaring what functions we want to compose. Lets try using a new function called 'Compose' to compose the functions:

int Compose(int x, Func<int,int> f)
{
    return (f(x));
}

So now we can compose the functions like this:

int result = Compose(Compose(Compose(42, function1), function2), function3);

Well there doesn’t seem like there is any advantage to that. We got our nested brackets back, plus it looks even more complicated than our original imperative code. But let’s persevere just a little longer with this idea before we give up on it, because we are about to uncover a number of very cool advantages.

First let’s make the Compose function an extension method by adding a this keyword:

static class ExtensionMethods
{
    public static int Compose(this int x, Func<int,int> f)
    {
        return f(x);
    }
}

Now we can compose the three functions using this syntax:

int result = 42
    .Compose(function1)
    .Compose(function2)
    .Compose(function3);

We just solved the nested brackets problem. And we no longer need local variables. That looks useful. We are actually just composing the functions. This syntax is called fluent syntax. The chain of functions is now scalable - we can easily add more functions to the chain. Fluent syntax is our first advantage of using a Compose function.

The three functions are chained in a way that it looks like a declarative dataflow programming paradigm instead of an imperative programming paradigm. We are just specifying what functions we want to compose in what order instead of imperatively calling them. The code now describes a flow of data more than a flow of execution. That’s a good thing. This dataflow programming paradigm is our second advantage of using a Compose function. This advantage will turn out to be huge. In our user story code we just declare what functions we want to compose to make a dataflow. The compose function itself then takes care of how everything actually executes.

The Compose function takes a function pointer or delegate as its parameter. If the function is only ever used once, we can make it anonymous and put the function code directly into the Compose call:

int result = 42
    .Compose(delegate(int x){return x+1;)
    .Compose(delegate(int x){return x*10+1;})
    .Compose(delegate(int x){return 1/x;});

Doing it with delegates like that is somewhat verbose, so the next step is to change the syntax to lambda expressions.

int result = 42
    .Compose(x => x+1)
    .Compose(x => x*10+1)
    .Compose(x => 1/x);

So that’s our third advantage of using a Compose function. We can use lambda expressions right in the parameter of the Compose function instead of creating separate named functions.

Named functions are good if the function is a good abstraction in a lower layer. But if a function is specific to a user story (only used once, ever), it is not an abstraction. So the name becomes just a symbolic wiring between two points in the code. Symbolic wiring is bad. It’s indirection without an abstraction. We would have to use an editor to search for the other point to find the wiring. Lambda expressions solve this problem because they are anonymous functions.

You can see that although this last form is just a refactoring of the original imperative code, it now looks even more like a dataflow programming paradigm. By dataflow, we mean that we are thinking of it in terms of piping data from lambda expression to lambda expression rather than imperatively execution of statements.

In the example so far, we know that under the covers of the Compose function, the execution flow follows the dataflow. Since we are just describing a dataflow with our top level code, it is possible for the execution flow in the compose function to work in a completely independent way. We can for example, implement deferred execution, where the Compose function builds an executable structure which can be run later. Later we will do even more powerful examples of an independent execution model such as asynchronous execution. This separation of how it executes from the declarative composition of the functions as a dataflow is our forth advantage of using a Compose function.

6.1.2. Deferred execution

The Compose function we had in the previous section evaluated the functions immediately and returned a result directly. If we write a deferred version, the Compose function will return a new function that represents the entire computation, but can be called later. Here is a deferred version of the Compose function:

static class ExtensionMethods
{
    static Func<int> Compose(this Func<int> source, Func<int,int> function)
    {
        return () => function(source());
    }
}

The () ⇒ syntax is a lambda expression for a function that takes no parameters. Compose returns a function that calls the source function and then calls the function.

What the Compose function returns is actually an object structure made up of delegates and closure objects created by the compiler:

ComposeClosure.drawio.png
Figure 43. Object diagram of the structure returned by the deferred version of the Compose function

Note that in the imperative world we would say we are really passing in two pointers to functions, and returning a pointer to a function, but in the functional world this is said to be just composing functions.

The closure object has two fields, which are delegates to the two functions. A delegate is a pointer to an object together with a method in that object. A closure is an object made from a compiler generated class with a single method and one or more fields which are references to variables in the local scope.

Now that we have a deferred version of our Compose function, we can use it like this:

Func<int> composedFunction = 42.ToFunc<int>
    .Compose(x => x+1)
    .Compose(x => x*10+1)
    .Compose(x => 1000/x);

This doesn’t call any of the lambda expressions immediately. That can be done at any time later with:

int result = composedFunction();

Note that we used another extension method called ToFunc to get a function that returns 42, our starting value. We needed this function because Compose takes a function as its first parameter, so we needed a function that returns 42 to start the chain.

Deferred function composition generally returns a surprisingly large object structure containing delegate objects and closure objects. Here is what the object structure for the composedFunction above looks like:

ThreeComposedFunctionsClosureDiagram.drawio.png
Figure 44. Object diagram of expression composing three functions using deferred Compose function

We will generally go for a deferred version of a Compose function because then you have the option of using it immediately, or using it to build a larger program to be executed later. Deferred execution is our fifth advantage of using a compose function. Even if you run it immediately, the way a deferred composed function runs can be more efficient with use of memory.

We are not finished yet. There is an even bigger advantage of using a compose function, which we will do next.

6.1.3. Composing functions that need logic between them

Sometimes we need to put some common logic between function calls such as to check for any errors that have occurred. We can refactor this common logic into the Compose function. This refactoring can’t even be contemplated in the imperative version.

First we’ll give four examples of imperative code. In each case, the functions we are composing are not returning a simple value that can be fed directly to the next function. They are returning a variety of different things, but in every case we need a little bit of extra code to handle what the function returns and then feed it to the next function.

Example 1

Composition of functions that can fail.

In this first example, we may need to allow for the fact that functions can throw an exception, or return null, or a Maybe object or even -1. For example the function may contain a divide by zero. In imperative code, we would commonly have to add if statements or try statements so that we don’t call the rest of the functions in the chain in case something goes wrong.

In C code, returning -1 is often used for this purpose, so let’s use that for our first example because it’s so simple. Here is the imperative code:

// procedural composition of functions that can return -1 or null

int result1 = function1(42);
if (result1 != -1)
{
    int result2 = function2(result1)
    if (result2 != -1)
    {
        int result3 = function3(result2);
        if (result3 != -1)
        {
            DoSomething(result3);
        }
    }
}
// fall through means something returned -1

Note that, even though we are using intermediate variables, every composed function involves nested brackets for the if statements, which is really not scalable.

Example 2

Composition of functions that return many values.

We may have functions that return many values, such as an array, a list, an IEnumerable or an IObservable. We then want to feed all the individual values into the next function, and then recombine the results. In imperative code, we do this with nested for statements. For example, the function may be given customers one at a time and returns a list of their orders, which we want to join back into a single list of orders.

// procedural composition of functions that return a list

var results1 = function1(42);
List<int> combinedList1 = new List<int>;
foreach(result1 in results1)
{
    var results2 = function2(result1)
    List<int> combinedList2 = new List<int>;
    foreach (result2 in results2)
    {
        var results3 = function3(result2)
        combinedList2.Append(results3);
    }
    combinedList1.Append(combinedList2);
}
List<int> result = combinedList1;

Again note the extra levels of brackets and indenting for every foreach.

Example 3

Composition of functions that return a Future, Promise or Task object.

The functions that we want to compose may contain delays, or they may wait for input or output. So each function, instead of returning the result, may return a Task, future or promise object. The glue code between function calls needs to wait for the future object to have the result before calling the next function:

static void ComposedFunction()
{
    function1(1)
    .ContinueWith(task1 =>
    {
        function2(task1.Result)
        .ContinueWith(task2 =>
        {
            Console.WriteLine($"Final result is {task2.Result}.");
        });
    });
}

ContinueWith itself returns a Task, but these are discarded as we are only interested in continuing from the Task that is returned by function1 and function2.

Again notice the nasty indenting for every function we want to chain. In this case we can eliminate the indenting by using Unwrap() like this:

static void ComposedFunction()
{
    function1(1)
    .ContinueWith(task => function2(task.Result))
    .Unwrap()
    .ContinueWith(task =>
    {
        Console.WriteLine($"Final result is {task.Result}.");
    });
}

Note that both versions require lambda expressions (for example, the lambda expression starting with 'task1 ⇒" ). In the first implementation above, the lambda expression is an Action. In the second implementation the lambda expression is a function. So they are two different overloads of ContinueWith. In the second implementation, the lambda function returns the type returned by the function, which is a Task<T>. So ContinueWith returns Task<Task<T>>. The Unwrap discards the outer Task.

Of course, async/await also simplifies this particular example, but I want to show how monads can also do it.

Example 4

Composition of functions that return angles.

There can be many other motivating examples. In fact we can do almost anything we like between the function calls as long as we are always doing the same thing. Let’s do one more example just to show that we can do something fairly arbitrary. Let’s say we always want to do modulo 360 arithmetic. And let’s throw in a rotation counter as well:

// procedural composition of functions that can return angles

int rotations = 0;
int result1 = function1(42)
rotations += result1 / 360;
result1 = result1 mod 360;
int result2 = function2(result1)
rotations += result2 / 360;
result2 = result2 mod 360;
int result3 = function3(result2)
rotations += result3 / 360;
result3 = result3 mod 360;

In all these above examples, we would like to be able to just compose the three functions in a declarative way like we were before, and have the common execution code inbetween refactored into the Compose function.

6.1.4. The monad refactorings

This is our sixth advantage of using a Compose function. This one is huge. The refactoring is called the monad pattern. The Compose function is usually called Bind. It also goes by other names such as ==>, flatmap, and SelectMany. From now on we will use the name Bind instead of Compose that we always used before.

Let’s now do the Bind function for each of our imperative examples above.

We will do immediate versions of Bind first because they are simpler.

MinusOne monad

Composition of functions that can fail by returning -1.

Here is top layer code that composes functions that may return -1.

Application layer code

int result = 42
    .Bind(x => x+1)
    .Bind(x = x*10+1)
    .Bind(x => x==0 ? -1 : 1/x);

Compare this with the imperative version we had previously that had to use nested if statements.

It’s almost the same as when we used Compose except that now the lambda expressions are allowed to return -1. If any one of them does, the result will be -1. The last lambda expression is an example of one which can return -1.

Here is the Bind function:

Monad layer code

static class ExtensionMethods
{

    public static int Bind(this int source, Func<int, int> function)
    {
        return source == -1 ? -1 : function(source);
    }
}

You can see that if any function in the chain returns -1, the rest of the functions are skipped and the final result is -1.

That is all there is to our first monad Bind function.

IMaybe monad

Composition of functions that can fail by returning IMaybe<T> or Nullable<T>.

Using minus one to represent a no value is not used outside the C world, and has limited use with only positive integers. The more general solution is the IMaybe<T> monad:

The IMaybe version is similar to the -1 version. However Bind in this case requires an IMaybe and returns an IMaybe, and the functions that we compose together also return an IMaybe. Here is example top layer code composing functions that return IMaybe.

Application layer code

IMaybe<double> combinedFunctions = 42.ToMaybe()
    .Bind(x => new Something<int>(x+1))
    .Bind(x => new Something<int>(x*10+1))
    .Bind(x => x==0 ? new Nothing<double>() : new Something<double>((double)1/x) );

Notice that we need to convert the starting value, 42, to an IMaybe so that Bind can be used on it. That’s the reason for the ToMaybe extension method. To be a monad, we need to supply this function. In monad land, this function is sometimes called unit or return.

The IMaybe interface itself consists of two getters, one called HasValue() that returns a bool to find out if a value is there, and the other called Value to get the actual value out if there is one.

public interface IMaybe<T>
{
    bool HasValue { get; }
    T Value { get; }
}

You would normally use HasValue first and only if it returns true would you use Value. HasValue is analogous to the MoveNext method in the IEnumerator interface, which you also have to call first before retrieving a value. We will need two classes that implement IMaybe, one to represent a nothing, and one to represent something:

Monad layer code

public class Nothing<T> : IMaybe<T>
{
    bool IMaybe<T>.HasValue { get => false; }
    T IMaybe<T>.Value { get { throw new Exception("No value"); } }
}


public class Something<T> : IMaybe<T>
{
    private T value;

    public Something(T value) { this.value = value; }

    bool IMaybe<T>.HasValue { get => true; }
    T IMaybe<T>.Value { get => value; }
}

The Bind function takes an IMaybe as a parameter and returns an IMaybe. It uses its input IMaybe<T> to see if there is a value present or not. If there is nothing it doesn’t even call the function. It just returns a new IMaybe<U> implemented by a nothing object. If there is a value, it gets the value and passes it to the function, then Bind returns the IMaybe returned by the function.

Monad layer code

static class ExtensionMethods
{
    public static IMaybe<T> ToMaybe<T>(this T value)
    {
        return new Something<T>(value);
    }


    public static IMaybe<U> Bind<T, U>(this IMaybe<T> source, Func<T, IMaybe<U>> function)
    {
        return source.HasValue ? function(source.Value) : new Nothing<U>();
    }
}

Monads in general consist of three things: an interface, a Bind function and a way to create an object that implements the interface. In this case they are IMaybe, Bind, and ToMaybe.

List monad

Composition of functions that return many values, in this case a list.

Here is example top layer code that composes functions that return a list:

Application layer code

var result = List<int> result = new List<int>(){ 0 }
    .Bind(function1)
    .Bind(function2)
    .Bind(function3);

The functions each return a list. So as we Bind each new functoin, the number of items in the list multiplies up. Here is an application using function that are lambda expressions that each return a list of three items:

var result = new List<int> { 0 }
    .Bind(x => new List<int> { x * 10 + 1, x * 10 + 2, x * 10 + 3 })
    .Bind(x => new List<int> { x * 10 + 1, x * 10 + 2, x * 10 + 3 })
    .Bind(x => new List<int> { x * 10 + 1, x * 10 + 2, x * 10 + 3 });

In this case Bind will receive a list as its input. It will feed all the values one by one to the function. Each call of the function will return a new list. Bind will then join all the lists together and return the combined list. Because we compose three functions, and each returns a list of three items, the result list at the end will contain 27 items. The output is:

ConsoleOutputListMonad.png
Figure 45. Output of three Bind functions in a row

Here is the Bind function for the List monad:

static class ExtensionMethods
{
    public static List<U> Bind<T, U>(this List<T> source, Func<T, List<U>> function)
    {
        List<U> output = new List<U>();
        foreach (T t in source)
        {
            var List<U> functionOutput = function(t);
            output.AddRange(functionOutput);
        }
        return output;
    }
}}

Let’s say the List<T> input were a list of students. Bind uses a for loop to get all the students one at a time. It passes each student to the function. Each call of the function returns a List<U>. Let’s say this is a list of courses for the student. The bind function then joins all the separate course lists together to make a single list of courses of type List<U>, which it returns.

Note that the monad itself is designed to compose functions that return lists. It then has effectively a list of lists. It then flattens the lists. Often we will want to just do a one-to-one mapping of the values in a list, or we will even want to aggregate the values in the list down to a single value such as Sum. Methods to do these are usually supplied along with the monad, but the monad itself is the Bind function is the one that composes functions that return a list, and then flattens all the lists.

Mod360 monad

Composition of functions that return angles.

This is not strictly speaking a monad because the function doesn’t return the same interface as the Bind function uses for its input and output. That’s because in this case the function didn’t need to know anything about the rotations. However it still shows how the monad pattern can refactor arbitrary common code between composed functions.

Here is top layer code to compose function that return degrees. The second value in the Tuple is the number of rotations, which we initialize to 0.

Application layer code

Tuple<int,int> result = new Tuple(42,0)
    .Bind(function1)
    .Bind(function2)
    .Bind(function3);

Here is the Bind function:

public static Tuple<int,int> Bind<T, U>(this Tuple<int,int> source, Func<int, int> function)
{
    int result = function(source.Item1);  // call the function
    return new Tuple<int,int> (
        result mod 360,   // normalize the angle
        source.item2 + result/360);   // count rotations
}

This time Bind takes a Tuple and returns a Tuple. The Tuple contains the angle between 0 and 359 and the rotations. Bind will do the mod 360 on the result returned by the function, and add any rotations. It returns a new Tuple with those two values.

Note that it was easy to get the starting 42 value into the Tuple needed by the Bind function by simply using 'new Tuple(42,0)'. So in this case we didn’t need something like a ToTuple extension method.

Those were the immediate versions of the monads. Let’s now have a look at the deferred versions of these monads. We want to understand the deferred version because they are much closer to how ALA works.

6.1.5. Deferred monads

If the monad is an immediate (eager) type, the value returned by the monad chain is the actual result. But if the monad is a deferred type, the value returned by the monad chain is an object structure that you can use to get the value. You might do things like the following to force the actual value out.

if (result!=-1) { use result }               // -1 monad
if (result.hasValue) { result.value }        // maybe monad
result.ToList()                              // IEnumerable
foreach (var value in result) {...}          // IEnumerable
result.Subscribe((x)=>{....})                // IObservable
result.ContinueWith(result => result.Result) // task
await result                                 // task
result.Item0, result.Item1                   // tuple
MinusOne monad (deferred version) (pull version)

Composition of functions that can fail by returning -1.

For the deferred version of the MinusOne monad, we use Func<int> instead of an integer as the interface. The Bind function takes a Func<int> and returns a Func<int>:

Here is top layer code that composes functions that can return -1:

Func<int> CombinedFunction =
    42.Bind(x => x+1).Bind(x = x*10+1).Bind(x => x==0 ? -1 : 1/x);
}

The Bind implementation doesn’t call the function, it returns another function that can do that later:

Pull version

namespace Monad.MinusOne
{
    public static class ExtensionMethod
    {
        public static Func<int> ToMinusOne(this int source)
        {
            return () => source;
        }

        public static Func<int> Bind(this Func<int> source, Func<int, int> function)
        {
            return () =>
            {
                int value = source();
                return value == -1 ? -1 : function(value);
            };
        }
    }
}

The lambda functions are turned into closure objects by the compiler. The returned object structure looks like the diagram below.

MinusOneDeferredPullMonadDiagram.drawio.png
Figure 46. Object diagram of expression using deferred/pull version of MinusOne monad

This structure is exactly the same as the one we showed above for the Compose function. The only difference is that for the three closures that are created by the Bind function, the closure method contains the common code, that is it checks for -1 from the source before calling the next function.

We got a little lucky with the implementation of the deferred/pull MinusOne monad. That is that we were able to use Func<int> as the interface that Bind takes and returns instead of using an actual interface with a function in it. That allowed us to use closures to implement the Bind and ToMinusOne functions, just as we did for the Compose function. From now on we won’t be able to do that because the monads will be using an actual interface.

MinusOne monad (deferred version) (push version)

Composition of functions that can fail by returning -1.

With deferred monads, we can do either pull versions or push versions.

For the pull version, we keep a reference to the last object in the structure. We call a function in that object when we want the result. That call pulls the data through the chain of objects.

For a push version, we still keep a reference to the last object in the structure, but when we call a function in that object, it just calls functions through to the first object in the chain, which then pushes data through the chain.

We will do both types so that we can properly understand the nuances of each. The push versions will be more comparable with default ALA programming paradigms. Here is the application code for the push version.

// deferred monad composition of functions that might return -1

IMinusOneObservable<int> result = 42.ToMinusOne()
    .Bind(x => x+1)
    .Bind(x = x*10+1)
    .Bind(x => x==0 ? -1 : 1/x);
}

The interface that Bind takes and returns is IMinusOneObserver. Here it is:

public interface IMinusOneObservable
{
    void Subscribe(IMinusOneObserver observer);
}

All this interface does is give Bind a way to wire up another interface. And yes these two interfaces are exactly analogous to the IObservable and IObserver interfaces in reactive extensions.

public interface IMinusOneObserver
{
    void Push(int value);
}

The IMinusOneObserve interface is wired in the same direction as the pushing (although we could have chosen to use C# events instead). Since this interface is wired in the same directon as the dataflow, destinations implement the interface and sources will have a field of the type of this interface.

Bind can’t be defined on the IMinusOneObsever interface because it’s the wrong way around. Bind therefore uses the IMunusOneObservable interface that goes in the opposite direction of the dataflow.

For the push version we don’t have the luck we had in the pull version that allowed us to implement it with closures because we had to use the IMinusOneObserver interface. The Bind function will instead use an explicit class, which we will call MinusOne. Here is that class, together with the ToMinusOne and Bind extension methods:

namespace Monad.MinusOne
{
    static class ExtensionMethods
    {
        public static IMinusOneObservable ToMinusOneMonad(this int value) (5)
        {
            return new MinusOneStart(value);
        }

        public static IMinusOneObservable Bind(this IMinusOneObservable source, Func<int, int> function) (1)
        {
            MinusOne minusOne = new MinusOne(function);
            source.Subscribe(minusOne);
            return minusOne;
        }
    }




    class MinusOne : IMinusOneObservable, IMinusOneObserver (2)
    {
        private IMinusOneObserver observer; (3)

        private Func<int, int> function;

        public MinusOne(Func<int, int> function) (4)
        {
            this.function = function;
        }

        void IMinusOneObserver.Push(int value) (5)
        {
            if (value == -1)
            {
                observer.Push(-1);
            }
            else
            {
                observer.Push(function(value));
            }
        }

        void IMinusOneObservable.Subscribe(IMinusOneObserver observer)
        {
            this.observer = observer;
        }
    }




    class MinusOneStart : IMinusOneObservable (6)
    {
        private int value;
        private IMinusOneObserver observer;


        public MinusOneStart(int value) { this.value = value; }

        void IMinusOneObservable.Subscribe(IMinusOneObserver observer)
        {
            this.observer = observer;
        }

        public void Run()
        {
            observer.Push(value);
        }
    }
}
1 The Bind method just instantiates a class to do the work. The Bind function also wires up the IMinusOneObserver interface using the Subscribe method.
2 IMinusOneObservable is implemented by data sources. IMinusOneObserver is implemented by data destinations. Our MinusOne class, as part of a chain of operations, is both a source and a destination, so it implements both.
3 Once wired, the only reference between the objects is the reference from source to destination in the field called observer in the MinusOne class.
4 The constructor just needs to store the function. we are composing.
5 The Push method is the only part that runs when the monad object structure runs.
6 The last thing to note is the usual method we need to get the 42 into the monad type so that we can start using Bind. In this case the monad type is IMinusOneObservable, so there needs to be a class that implements IMinusOneObservable. That class is MinusOneStart. The ToMinusOne extension method simply needs to instantiate this class.

Here is the object diagram of the resulting structure:

MinusOneDeferredPushMonadDiagram.drawio.png
Figure 47. Object diagram of expression using deferred/push version of MinusOne monad

You can see that the three delegate-closure pairs we had in the pull version are replaced with an object of class MinusOne. The three objects are wired together in the direction of the data flow (left to right) using the IMinusOneObserver interface. The IMinusOneObservable was only used by the Bind function to effect the wiring of IMinusOneObserver. It is unused when the structure runs. The IMinusOneObservable interface at the end can be used to wire to an output object that implements IMinusOneObserver.

The 42 is stored in the object of the MinusOneStart class. This class has a run function which is used to start the structure executing. We start it from the source end because it is a push monad we are using. (This differs from the reactive extensions, which starts executing on Subscribe, so execution is actually initiated from the destination end.) In ALAs push programming paradigms, we usually initiate dataflow at the source end.

You can start to see the ALA pattern to this structure. It is instantiating objects and wiring them together to build a structure to run later. IMinusOneObserver is the equivalent of the ALA programming paradigm.

All the deferred monads we do from now on have this same structure. The push ones will be wired in the direction of dataflow, left to right, like this one is. The pull ones will be wired in the opposite direction of the dataflow, right to left. As I said, we were just lucky that the deferred pull version of the MinusOne monad that we did above was able to be implemented with closures because the monad type was Func<int> instead of a real interface. We will always need an explcit class from now on.

Next well do a deferred pull monad that uses a real interface<T>, the IMaybe<T> monad.

IMaybe monad (deferred version) (pull version)

Composition of functions that can fail by returning IMaybe<T> or Nullable<T>.

Here is top layer code to use the deferred/pull implementation of the maybe monad.

IMaybe<double> combinedFunctions = 42.ToMaybe()
    .Bind(x => new MaybeSomething<int>(x+1))
    .Bind(x => new MaybeSomething<int>(x*10+1))
    .Bind(x => x==0 ? new MaybeNothing<double>() : new MaybeSomething<double>((double)1/x) );

It looks the same as the immediate version. But it returns an IMaybe that’s implements a large object structure instead of returning one of the two concrete IMaybe value objects.

First we define the IMaybe interface, which is the same as for the immediate version above. The MaybeNothing and MaybeSomething classes are also the same as before.

    public interface IMaybe<T>
    {
        bool HasValue { get; }
        T Value { get; }
    }

    public class MaybeSomething<T> : IMaybe<T>
    {
        T value;

        public MaybeSomething(T value) { this.value = value; }

        bool IMaybe<T>.HasValue { get => true; }
        T IMaybe<T>.Value { get => value; }
    }



    public class MaybeNothing<T> : IMaybe<T>
    {
        bool IMaybe<T>.HasValue { get => false; }
        T IMaybe<T>.Value { get { throw new Exception("No value"); } }
    }

The Bind function is different as it must build a structure that can be run later. It instantiates a class that implements IMaybe, which will do all the work at runtime.

namespace Monad.MaybeDeferredPull
{
    static class ExtensionMethods
    {
        public static IMaybe<T> ToMaybe<T>(this T value)
        {
            return new MaybeSomething<T>(value);
        }

        public static IMaybe<U> Bind<T, U>(this IMaybe<T> source, Func<T, IMaybe<U>> function)
        {
            return new Maybe<T, U>(source, function);
        }
    }



    class Maybe<T, U> : IMaybe<U>
    {
        // implement the constructor, which receives the Action function
        private Func<T, IMaybe<U>> function;
        private IMaybe<T> source;
        private IMaybe<U> result;

        public Maybe(IMaybe<T> source, Func<T, IMaybe<U>> function) { this.source = source; this.function = function; }

        bool IMaybe<U>.HasValue
        { get
            {
                if (result == null)
                {
                    if (source.HasValue)
                    {
                        result = function(source.Value);
                    }
                    else
                    {
                        return false;
                    }
                }
                return result.HasValue;
            }
        }

        U IMaybe<U>.Value
        {
            get
            {
                if (result == null)
                {
                     result = function(source.Value);  // will throw exception if no value
                }
                return result.Value; // will throw exception if no value
            }
        }
    }
}

The code that runs later in the Maybe class is the HasValue and Value getters. They do all the work.

Bind creates objects of the class Maybe and chains them together. This diagram shows the resulting structure from our little bit of application code:

MaybeDeferredPullMonadDiagram.drawio.png
Figure 48. Object diagram of expression using deferred/pull version of IMaybe monad

Because this is a pull implementation of the monad, the references go in the opposite direction of the dataflow - from destination to source or from right to left. When you want to run the combined function, you pull the value from the right end.

IMaybe monad (push version)

Composition of functions that can fail by returning IMaybe<T> or Nullable<T>.

Now the push version of the deferred IMaybe monad. Here is the top layer code, which in this case returns a IMaybeObservable.

IMaybeObservable<int> result = 42.ToMaybe()
    .Bind(function1)
    .Bind(function2)
    .Bind(function3);

I’ve purposely left the lambda expressions out for now. Well get back to them in a minute.

As with the deferred push version of the MinusOne monad, we need two interfaces, IMaybeObservable<T> that Bind takes and returns, and IMaybeObserver for doing the actual pushing of data at runtime.

Here are the two interfaces:

    public interface IMaybeObservable<T>
    {
        void Subscribe(IMaybeObserver<T> observer);
    }
    public interface IMaybeObserver<T>
    {
        void NoValue();
        void Value(T value);
    }

The Bind function uses the IMaybeObservable to wire the IMaybeObserver interface in the opposite direction.

Now we consider the type that the functions that you compose should return. Normally with monads, this is the same interface that Bind takes and returns. So that would be IMaybeObservable. IMaybeObservable will certainly work, but the functions will be a little complicated. They will have the form: Func<T, IMaybeObservable<U>>. They would have to create an object implementing the IMaybeObservable interface to return. That interface then has a Subscribe method called on it, which gives the object an IMaybeObserver. Then the object can finally push out its result by pushing it via the IMaybeObserver.

It would be just so much simpler if the functions were passed the IMaybeObserver directly. If we did that, the functions would have the form Action<T, IMaybeObserver<U>>. Now when the functions run, they don’t need to create an object to return. Instead they just directly push the result out via the IMaybeObserver<U>> interface. This kind of makes sense because it’s a push monad.

Here is the application layer code with lambda expressions:

IMaybeObservable<double> combinedFunctions = 42.ToMaybe()
    .Bind((x,ob) => ob.Value(x+1))
    .Bind((x,ob) => ob.Value(x*10+1))
    .Bind((x,ob) => { if (x==0) ob.NoValue(); else ob.Value((double)1/x); } );

So remember when reading the monad implementation below, the 'functions' that you compose in the application layer are really Actions that take a T and a IMaybeObserver<U> and don’t return a value.

namespace Monad.MaybeDeferredPush
{
    static class ExtensionMethods
    {
        public static IMaybeObservable<T> ToMaybe<T>(this T value)
        {
            return new MaybeStart<T>(value);
        }

        public static IMaybeObservable<U> Bind<T, U>(this IMaybeObservable<T> source, Action<T, IMaybeObserver<U>> action) (1)
        {
            var maybe = new Maybe<T, U>(action);
            source.Subscribe(maybe);
            return maybe;
        }
    }





    class Maybe<T, U> : IMaybeObserver<T>, IMaybeObservable<U> (2)
    {
        private Action<T, IMaybeObserver<U>> action;

        public Maybe(Action<T, IMaybeObserver<U>> action) { this.action = action; }


        private List<IMaybeObserver<U>> subscribers = new List<IMaybeObserver<U>>(); (3)

        void IMaybeObservable<U>.Subscribe(IMaybeObserver<U> observer)
        {
            subscribers.Add(observer);
        }


        void IMaybeObserver<T>.NoValue()
        {
            foreach (var subscriber in subscribers)
            {
                subscriber.NoValue();
            }
        }

        void IMaybeObserver<T>.Value(T value)
        {
            action(value, new ActionObserver<T, U>(this));
        }


        private class ActionObserver<T, U> : IMaybeObserver<U> (4)
        {
            private Maybe<T, U> outer;
            public ActionObserver(Maybe<T, U> outer) { this.outer = outer; }

            void IMaybeObserver<U>.NoValue()
            {
                foreach (var subscriber in outer.subscribers)
                {
                    subscriber.NoValue();
                }
            }

            void IMaybeObserver<U>.Value(U value)
            {
                foreach (var subscriber in outer.subscribers)
                {
                    subscriber.Value(value);
                }
            }
        }
    }




    class MaybeStart<T> : IMaybeObservable<T>
    {
        private T value;
        public ToMaybe(T value) { this.value = value; }

        private List<IMaybeObserver<T>> subscribers = new List<IMaybeObserver<T>>();
        void IMaybeObservabe<T>.Subscribe(IMaybeObserver<T> subscriber)
        {
            subscribers.Add(subscriber);
        }

        public void Run()
        {
            foreach (var subscriber in subscribers)
            {
                subscriber.Value(value);
            }
        }
    }
1 As you can see, the Bind function just creates an object of the Maybe class to do all the work at runtime. The Bind function is defined on the IMaybeObservable interface and returns that same interface. It composes Actions rather than functions. These actions take an IMaybeObserver.
2 The Maybe class implements both IMaybeObservable and IMaybeObserver. IMaybeObservable is only iused by Bind to call Subscribe. IMaybeObserver is the one that is wired (in the same directions as the dataflow) to be used at runtime to push the data through.
3 The wiring of Maybe supports fanout or multiple subscribers (just like the observer pattern). We didn’t do this for the MinusOne deferred push monad just to keep it simpler. But we will do it for all deferred push style monads from now on. It is normal for push monads to support fan out, in other words many observers can be listening to the same data that is pushed. It is another advantage of push style monads over pull style monads.
4 You will notice an inner class called ActionObserver inside the Maybe class. At runtime, the Maybe class will need to call the action, and it needs an object that implements IMaybeObserver to pass to that action. That’s what ActionObsserver is for.

Here is an object diagram of the complete expression.

MaybeDeferredPushMonadDiagram.drawio.png
Figure 49. Object diagram of expression using deferred/push version of IMaybe monad

You can see that the references between the objects, which use IMaybeObserver, go in the same direction as the dataflow. IMaybeObservable is only used for wiring the structure up.

The structure starts executing when the Run method in the MaybeStart object on the left is called. The application needs to keep a reference to this object so it can start the program.

So far we have done deferred pull and deferred push implementations of the MinusOne and Maybe monads. Let’s do a couple more examples of deferred monads to get more used to the pattern:

IEnumerable monad

Composition of functions that return many values, in this case an IEnumerable.

The IEnumerable monad is the deferred version of the list monad we did earlier. The IEnumerable monad is the most commonly used monad, and is what LINQ is based on.

The Bind function for the IEnumerable monad is called SelectMany in C#. SelectMany is not used as often as Select. Select takes a simpler function that returns U instead of IEnumerable<U>, so it doesn’t expand the number of items, it just does a one-to-one mapping. While Select is used more often, it is the SelectMany function that makes it a Monad. Here in our example application we will use three SelectManys in a row. Each will expand in number by 3, so we will end up with an IEnumerable with 27 items in the end.

Here is example top layer code that composes functions that return IEnumerable

IEnumerable<int> result = 42.ToEnumerable()
    .SelectMany(function1)
    .SelectMany(function2)
    .SelectMany(function3);

You may remember that we used lambda expressions in the immediate example above that returned literally lists something like this:

var result = new[] { 0 }
    .Bind(x => new[] { x * 10 + 1, x * 10 + 2, x * 10 + 3 })
    .Bind(x => new[] { x * 10 + 1, x * 10 + 2, x * 10 + 3 })
    .Bind(x => new[] { x * 10 + 1, x * 10 + 2, x * 10 + 3 });

While this will run fine when using the IEnumerable version of Bind, it’s not really in the style of a deferred monad to create memory hungry arrays. So let’s write functions that will do the same job in a lazy way:

private static IEnumerable<int> MutiplyBy10AndAdd1Then2Then3(int x)
{
    yield return x * 10 + 1;
    yield return x * 10 + 2;
    yield return x * 10 + 3;
}

The yield return keyword causes the compiler to generate an IEnumerable object, which it returns. The IEnumerable object contains a state machine where each state executes code till it hits the next yield return statement.

Let’s just reuse that function three times in our composed function:

static void Application()
{
    var program = new[] { 0 }
    .Bind(MutiplyBy10AndAdd1Then2Then3)
    .Bind(MutiplyBy10AndAdd1Then2Then3)
    .Bind(MutiplyBy10AndAdd1Then2Then3);

    var result = program.ToList();  // now run the program
    Console.WriteLine($"Final result is {result.Select(x => x.ToString()).Join(" ")}");
}

The Bind function (SelectMany) for this type of monad takes an IEnumerable<T> and returns an IEnumerable<U>. The Bind function doesn’t use a for loop immediately as that would defeat the laziness. Instead the bind function uses an object that keeps state. Let’s call this object the output IEnumerable. The output IEnumerable knows how to use the source IEnumerable<T> to get the first value, which it gives to the function. The function returns an IEnumerable<U> which we will call the function IEnumerable. The output IEnumerable then knows how to get the values from the function IEnumerable<U> and return them one at a time. When it has exhausted all of them, the output IEnumerable<U> then gets the next value from the source IEnumerable<T>, and gives that to the function. The function again returns an IEnumerable<U>. This process continues until the source and function IEnumerables are both exhausted.

In C#, the Bind function is really easy to write because the compiler can build an IEnumerable for you using the yield return syntax:

namespace Monad.Enumerable
{
    static class ExtensionMethods
    {
        public static IEnumerable<U> Bind<T, U>(this IEnumerable<T> source, Func<T, IEnumerable<U>> function)
        {
            foreach (var t in source)
            {
                var enumerator = function(t);
                foreach (var u in enumerator)
                {
                    yield return u;
                }
            }
        }
    }
}

Note that the code in the function does not run when this Bind function runs. The compiler sees the yield return and builds an object containing a state machine that implements IEnumerable<U>, and returns that.

Since our purpose is to show how the Bind function is a refactoring of imperative code, here is a version that doesn’t cheat by using the yield return syntax:

static class ExtensionMethods
{
    public static IEnumerable<U> Bind<T, U>(this IEnumerable<T> source, Func<T, IEnumerable<U>> function)
    {
        return new EnumerableMonad<T, U>(source, function);
    }
}

All Bind does is instantiate the class and return it. The class gets passed the source IEnumerable and the function. The class implements IEnumerable<U> for its output, which means it must be able to return an object implementing IEnumerator. The easiest way to do that is have the class implement IEnumerator<U> as well. Then the IEmumerable can just return 'this'.

class EnumerableMonad<T, U> : IEnumerator<U>, IEnumerable<U>
{
    private readonly IEnumerable<T> source;
    private readonly Func<T, IEnumerable<U>> function;

    public EnumerableMonad(IEnumerable<T> source, Func<T, IEnumerable<U>> function)
        { this.source = source; this.function = function; } (1)

    private IEnumerator<T> sourceEnumerator = null;

    IEnumerator<U> IEnumerable<U>.GetEnumerator()
    {
        sourceEnumerator = source.GetEnumerator();
        return (IEnumerator<U>)this;
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        sourceEnumerator = source.GetEnumerator();
        return this;
    }


    private IEnumerator<U> functionEnumerator = null;

    U IEnumerator<U>.Current => functionEnumerator.Current;

    object IEnumerator.Current => throw new NotImplementedException();

    void IDisposable.Dispose() { }

    bool IEnumerator.MoveNext() (2)
    {
        while (true)
        {
            if (functionEnumerator != null)
            {
                if (functionEnumerator.MoveNext())
                {
                    return true;
                }
            }

            if (sourceEnumerator.MoveNext())
            {
                functionEnumerator =
                    function(sourceEnumerator.Current).GetEnumerator();
            }
            else
            {
                return false;
            }
        }
    }

    void IEnumerator.Reset()
    {
        functionEnumerator = null;
        sourceEnumerator.Reset();
    }
}
1 The constructor is passed both the sourceIEnumerable and the function. It saves both of them in local variables.
2 The IEnumerator MoveNext method does all the work of the class at runtime. It is called by the next object in the chain. It gets the first element from the source, and feeds it to the function. Then it stores the Enumerator it gets from the function so it can use it in subsequent calls. Then it gets the first element from the function’s Enumerator and returns it. A while loop is necessary because when the Enumerator that is returned by the function runs out, it needs to go back and get the next element from the source and pass that to the function.

The class is completely lazy, so it doesn’t even get the source IEnumerator from the source IEnumerable until the first call of MoveNext.

The two fields, sourceEnumerator, and functionEnumerator are the state. The first can have a state of null, which is the state before we got the first value.

The object diagram for the program again shows three objects wired in a chain from right to left:

IEnumerableDeferredPullMonadDiagram.drawio.png
Figure 50. IEnumerable Deferred Pull Monad Object Diagram

Bind just wires the IEnumerable interface. The IEnumerable GetEnumerator method then effectively wires the IEnumerator interface (in the same direction). So you might wonder if the IEnumerable interface could be considered redundant. We not just make Bind wire up the IEnumerator interfaces and dispensed with IEnumerable altogether? That would work, but I guess the reason IEnumerable exists is because IEnumerator is already implemented by many underlying library collections. When writing a new class that will support foreach, we need only provide a GetEnumerator method that simply returns the underlying collection instead of implementing the whole IEnumerator interface. However in our class above, this didn’t help because we had to implement the whole IEnumerator interface because we were recombining multiple collections.

IObservable monad

Composition of functions that return many values, in this case an IObservable.

The IObservable monad is the 'push' version of the IEnumerable monad, sort of. Once the flow of data begins, it is indeed pushed (source to destination). The data is pushed using the IObserver interface. But often it is the destination that initiates the transfer. The destination uses the Subscribe method in the IObservable monad to register to observe the data. But often this Subscribing is also what initiates the transfer in the source. Once a transfer is completed, another transfer can usually be started by unsibscribing and resubscribing. When used in this way, IObservable is sort of a pull programming paradigm for data transfer initiations.

Some writers equate IObservable with "asynchronous". However, a pushing interface like IObserver can be either synchronous or asynchronous. Data flows from the source object by calling a method in the IObserver interface called OnNext. That method can execute synchronously all the way to the destination end of the chain, or it can return at any point along the chain, and the data flow can resume from that point at a later time, which is what we refer to as asynchronous.

Pull communications can’t be asynchronous or broken up in time, at least not in a straight forward way. It either requires blocking the thread (we don’t want to go there) or using a Callback, or using a Task or future object (which we cover later). The IEnumerator interface, being a pull interface, can only work synchronously. With IEnumerator, the destination end pulls data by calling a method. The function must execute synchronously all the way to the source otherwise it would return without a result.

The ability of a push style programming to be either synchronous or asynchronous is a good reason to default to using it. It is the reason ALA defaults to using push. Sometimes there are good reasons to use pull, but where it doesn’t matter, we prefer push. So it is worth covering the IObservable monad, even though IEnumerable monads tend to be more common. IObservable is the closest for comparison with the common ALA programming paradigms.

Unlike the IEnumerable/IEnumerator pair of interfaces which go in the same direction, the IObservable/IObserver interfaces go in opposite directions. The IObservable interface goes from destination to source whereas the IObserver interface goes from source to destination.

In the context of monads, the IObservable interface, being in the direction of destination to source, is the one that is used by BInd. IObserable is then used to wire and initiate the IObserver interface in the opposite direction.

In the context of ALA, it is a disadvantage to combine the 'wiring' and the 'start transfer' in the same Subscribe method call. In ALA we keep these two things separate because we want the code for these two things to be in two separate places. The wiring code represents a user story and so goes in a user story abstraction. We wire up entire program first and then set them running. The starting of a data transfer is an run-time event. It originates, for example, from a button domain abstraction. The two pieces of code should be in two separate abstractions. However, because this is an IObservable monad implementation example and not ALA, the Subscribe method will do both the wiring and initiating the data transfer.

Another thing we will do, like we did for the deferred/push version of the Maybe monad, is compose Actions instead of Funcs. When an Action is called at runtime, it will be passed an object for its output that implements IObserver. The action will use the IObserver to output directly instead of returning a function that has to create an IObservable in order to do its output. This greatly simplifies the code in the Actions, which is what we want because these Actions are application code. If we composed Funcs that return IObservables, we would need to make every Func more complicated. Instead the Bind function will take on extra work. It needs to create an IObservable object to pass to the action.

If you look at the SelectMany in the reactive extensions library for C#, you will see that it takes a Func. But there are two overloads. In one, the Func returns an IObservable object as expected. For the other, it returns an IEnumerable. It’s a shame that the second overload doesn’t take an Action that takes an IObserver. That would have truly simplified things the actions. Anyway that’s what we will do in our example here.

Here is an action to use in our example applicaton:

static void MutiplyBy10AndAdd1Then2Then3(int x, IObserver<int> observer)
{
    observer.OnNext(x * 10 + 1);
    observer.OnNext(x * 10 + 2);
    observer.OnNext(x * 10 + 3);
    observer.OnCompleted();
}

It takes a single integer as input and outputs a steam of three integers. The output goes to the IObserver that is also passed to teh Action.

Here is our test application.

static void Application()
{
Observable.Create<int>(
    observer => {
        observer.OnNext(0);
        observer.OnCompleted();
        return Disposable.Empty;
    })
    .Bind<int,int>(MutiplyBy10AndAdd1Then2Then3)
    .Bind<int,int>(MutiplyBy10AndAdd1Then2Then3)
    .Bind<int,int>(MutiplyBy10AndAdd1Then2Then3)
    .Subscribe((x) => Console.Write($"{x} "),
                (ex) => Console.Write($"Exception {ex}"),
                () => Console.Write("Complete")
                );
}

We start with a single integer with value zero, and then compose the action three times. Finally we send the output to the Console.

Now lets write the Monad’s bind function. As usual, C# (in this case the reactive extension library) provides us with a shortcut way to implement Bind by using Observable.Create and Observer.Create. This shortcut method obscures the way the Bind function is a refactoring of the imperative code, which is our purpose. However, for reference, here is the shortcut version first:

static class ExtensionMethods
{

    public static IObservable<U> Bind<T, U>(this IObservable<T> source, Action<T, IObserver<U>> action)
    {
        return Observable.Create<U>(outputObserver => (1)
        {
            source.Subscribe( (2)
                x => { action(x, Observer.Create( (3)
                        value => outputObserver.OnNext(value), (4)
                        ex => outputObserver.OnError(ex), (4)
                        () => { } (4)
                    ));
                }, (5)
                ex => outputObserver.OnError(ex), (3)
                () => outputObserver.OnCompleted() (3)
            );
            return Disposable.Empty;
        });
    }

If you find this version hard to read, just skip forward to the next version.

1 Bind returns an IObservable, so the first thing we do is create a new IObservable to be returned.

The Observable.Create method in the reactive extension library will create an object that implements IObservable. You pass it a Subscribe function. It does nothing more than create an object that implements IObservable, and uses the Subscribe method you gave it as the implementation of the IObservable. In this case we pass in a lambda (anonymous function) as the Subscribe method.

Remember a Subscribe method is passed an IObserver, so that’s the 'outputObserver' part of the lambda expression. The lambda expression takes up the entire rest of the code starting from 'outputObserver ⇒'.

2 When the Subscribe lambda expression gets called at runtime, it must subscribe to the source.
3 In subscribing to the source, we supply three functions for the source to call, OnNext, OnError and OnCompleted. The OnError and OnCompleted are routed directly to the outputObserver. The OnNext is routed to the action.
4 The action must in turn be given an observer for it to output to. Observer.Create creates an object that implements IObserver. You provide the three functions, OnNext, OnError, and OnCompleted that the IObserver interface needs.

If the action outputs data it is passed directly to the outputObserver. If the action outputs an error, it too is passed directly to the outputObserver. But if the action outputs OnCompleted, it is discarded. This is ecause the monad must combine the streams from multiple calls of the action into a single stream.

You may think we do not need the extra observer. Why not just pass outputObserver to the action like this:?

x => action(x, outputObserver);

That would indeed correctly pass the multiple outputs of the action to the outputObserver. However, the action may call OnCompleted at the end of each of its sequences. If it does we need to intercept it and remove it because otherwise it will terminate the outputObservable sequence prematurely. This removal of the OnCompleted from the function’s output is effectively what 'flattens' the output.

Removing the OnCompleted call is the reason we use Observer.Create().

Now we do a verion that does not use either Observable.Create or Observer.Create. Although the code is longer, this will be easier to understand since our purpose is to show how we can refactor the original imperative code. This shows more clearly that the Bind function works by instantiating an object that will do all the work at runtime, and then simply wiring that object to the previous one.

public static IObservable<U> Bind<T, U>(this IObservable<T> source, Action<T, IObserver<U>> action)
{
    return new Observable<T, U>(source, action);
}

The bind function simply instantiates an object from an explicit class called Observer. This class is listed below.

private class Observable<T, U> : IObserver<T>, IObservable<U> (1)
{
    private readonly IObservable<T> source;
    private readonly Action<T, IObserver<U>> action;

    public Observable(IObservable<T> source, Action<T, IObserver<U>> action) { this.source = source; this.action = action; } (2)


    private IObserver<U> output;
    private InnerObserver<U> innerObserver;

    IDisposable IObservable<U>.Subscribe(IObserver<U> observer) (3)
    {
        output = observer;
        innerObserver = new InnerObserver<U>(output);
        source.Subscribe(this);
        return Disposable.Empty;
    }

    void IObserver<T>.OnCompleted() (4)
    {
        output.OnCompleted();
    }

    void IObserver<T>.OnError(Exception ex) (4)
    {
        output.OnError(ex);
    }

    void IObserver<T>.OnNext(T value) (5)
    {
        action(value, innerObserver);
    }

    // Observer that simply interceps OnCompleted
    private class InnerObserver<U> : IObserver<U> (6)
    {
        public Observable(IObserver<U> output) { this.output = output; }

        IObserver<U> output;

        void IObserver<U>.OnCompleted() { } // discard

        void IObserver<U>.OnError(Exception ex) { output.OnError(ex); }

        void IObserver<U>.OnNext(U value) { output.OnNext(value); }
    }
}
1 The objects of this class implement both IObserver and IObservable. IObserver allows the object to be used to subscribe to the source. IObservable allows the next object in the chain to subscribe to it.
2 The class’s constructor stores the source and the action.
3 The class’s Subscribe method saves the output observer. It also Subscribes this object to the source, which usually starts the transfer of data.
4 The OnCompleted and OnError methods, (which are called by the source) simply pass through to the output observer.
5 The OnNext method, (which is called by the source) calls the action, and passes it the InnerObserver object to output to. The InnerObserver passes OnNext and OnError through to the output, but discards any OnCompleted produced by the action. This discarding of OnCompleted from the action is what joins all the sequences produced by the calls to the action together.
6 The InnerObserver’s only function is to remove OnCompleted calls from the action getting to the output so that the sequences get joined. (Note: We could have used Observer.Create instead of having the InnerObserver class. However, we would have had to use Observer.Create in the OnNext method to get a new instance to pass to the action every time. This is because the observer object created by Observer.Create will stop working when it gets a OnCompleted.) The explicit InnerObserver class makes it a little clearer what is going on.
Task monad

Composition of functions that return a Future, Promise or Task object.

With this monad, we will be able to compose functions that return a Task object, which is an object that represents a value it will get in the future.

We did not do an immediate version of this monad earlier because the functions don’t return a value immediately so we can’t. So this is the first time we will do this monad.

We did the imperative code that called the functions one after the other earlier in this section. You may remember that we attached a continuation action to Task objects returned by each function. In the first imperative version, each continuation had another level of nesting, and in the second version, an Unwrap was required. Also, if you look at the version on Github, the ContinueWith requires an additional parameter to cause everything to run on one thread.

For the Task monad, we simply factor out all this logic into a Bind function, which is really easy to do.

The application code then uses the Bind function to compose functions declaratively in the same way as any other monad:

// monad composition of functions that return a future

Task<int> CombinedFunction =
    42.ToTask().Bind(function1).Bind(function2).Bind(function3);

The difference from other application code we have done is that the interface that Bind takes and returns is Task<T>. The starting value has to be converted to a Task<T> first, which is the purpose of the ToTask extension method.

Once again there is a way of using the compiler to cheat to implement the Bind function:

public static async Task<U> Bind<T, U>(this Task<T> source, Func<T, Task<U>> function)
{
    return await function(await source);
}

The async/await feature is indeed powerful, but our purpose is to see how Bind is a refactoring of the original imperative code. So here is the version that uses ContinueWith instead of async/await.

public static Task<U> Bind<T, U>(this Task<T> source, Func<T, Task<U>> function)
{
    var tcs = new TaskCompletionSource<U>();
    source.ContinueWith(
        (t) => function(t.Result).ContinueWith(
            (t) => tcs.SetResult(t.Result)
        )
    );
    return tcs.Task;
}

The Bind function is passed a Task<T> and immediately creates a new Task<U> via TaskCompletionSource, which is returned. A closure object is created for the first lambda expression and a delegate object is created to call that. The ContinueWith attaches the delegate to the source Task<T> as a (callback) Action. The Task<> that is returned by ContinueWith is discarded.

When the source Task<T> produces a result, the first lambda expression will run. When it does, it receives the Task<T> and passes the result from it to the function. The function immediately returns a Task<U> (a different Task<U> from the one created earlier). That Task<U> is attached to a second continuation lambda expression. When the Task<U> produces a result, the second lambda is called. It puts the result into the tcs.

The Bind function can also be written using Unwrap, which eliminates the need for the TaskContinuation source:

public static Task<U> Bind<T, U>(this Task<T> source, Func<T, Task<U>> function)
{
    source.ContinueWith((t) => function(t.Result)).Unwrap();
}

When the lambda expression runs, it returns the Task<U> that is returned by the function, so the ContinueWith itself returns a Task<Task<U>>. The Unwrap discards the outer Task<>.

The async/await version generally runs everything on the same thread by default, which is great, but this is not the case for the ContinueWith version unfortunately. The example code on Github https://github.com/johnspray74/ContinuationMonad shows a console application that passes a TaskScheduler.FromCurrentSynchronizationContext() parameter to the ContinueWiths so that everything runs on the Console UI thread. That thread is never blocked.

The functions that can be composed using this Bind function must return synchronously with a Task object, but can take as long as they want to put a value into the Task. In the sample application on Github, we use one function with a delay, and one that does I/O. Another case is a function that will do CPU bound work on another processor.

Here are the two examle functions we can compose:

    private static Task<int> function1(int x)
    {
        return Task.Delay(3000).ContinueWith(_ => x + 2);
    }
private static Task<int> function2(int x)
{
    Console.WriteLine($"Value is {x}. Please enter a number to be added.");
    string line = null;
    return Task.Factory.StartNew(() => line = Console.ReadLine())
    .ContinueWith(_ => x + int.Parse(line));
}

The Task monad consists of the Task<T> type, the Bind function and the ToTask function. It is used to compose asynchronous functions. It is inherently a deferred/push monad.

Mod360 monad

Finally, let’s do a deferred version of the mod360 monad that we used as an example of refactoring arbitrary code. You’ll remember that we had imperative code that was doing mod 360 after every function call. We already did a simple immediate version of the monad. Let’s skip the deferred/pull version and go straight to the deferred/push version.

Here is a suitable interface for the monad:

interface IMod360Observer
{
    void Push(Tuple<int,int> value);
}

Item0 in the Tuple is the angle, and Item1 in the tuple is the rotations.

And we will need a second interface for the Bind function to use:

interface IMod360Observable
{
    void Subscribe(IMod360Observer observer);
}

Here is the application example code using the monad:

Application layer code

var program = 42.ToMod360();
program.Bind(function1).Bind(function2).Bind(function3);

program.Run()

Here is the Bind function and ToMod360 function. Both use explicit classes to do the actual work.

Monad layer code

static class ExtensionMethods
{
    public static IMod360Observable ToMod360(this int value)
    {
        return new Mod360Start(value);
    }

    public static IMod360Observable Bind(this IMod360Observable source, Func<int,int> function)
    {
        var mod360 = new Mod360(function);
        source.Subscribe(mod360);
        return mod360;
    }
}

The Bind function just instantiates a Mod360 class, configures it with the function being composed, and wires it to the previous object using the Subscribe method of its observable interface. The Subscribe method effects wiring in the opposite direction using the observer interface, which is needed because it is a push monad.

The class that does the work for the Bind function is below. It implements IMod360Observer for use by the previous object, and IMod360Observable for use by the next object.

class Mod360 : IMod360Observer, IMod360Observable
{
    private Func<int,int> function;

    public Mod360(Func<int,int> function) { this.function = function; }


    private List<IMod360Observer> subscribers = new List<IMod360Observer>();

    void IMod360Observable.Subscribe(IMod360Observer observer)
    {
        subscribers.Add(observer);
    }


    void IMod360Observer.Push(Tuple<int,int> value)
    {
        int functonResult = function(value.Item1);
        Tuple<int,int> result = new Tuple<int,int> (
                functionResult mod 360,   // normalize the angle
                value.Item2 + functonResult/360) // count rotations
            );
        foreach (var subscriber in outer.subscribers)
        {
            subscriber.Push(result);
        }

    }
}

The Observer.Push function does all the work at runtime. It first calls the composed function, and then creates a result Tuple from the source Tuple and the Tuple that is returned by the function.

This is the class used by ToMod360, which is straightforward.

class Mod360Start : IMod360Observable
{
    private int value;
    public Mod360Start(int value) { this.value = value; }

    private List<IMod360Observer> subscribers = new List<IMod360Observer>();
    void IMod360Observabe<T>.Subscribe(IMod360Observer<T> subscriber)
    {
        subscribers.Add(subscriber);
    }

    public void Run()
    {
        foreach (var subscriber in subscribers)
        {
            subscriber.Push(new Tuple<int,int> {value,0});
        }
    }
}

Note that previously with the IObservable monad, we used two interfaces IObservable and IObserver. The Subscribe method in the IObservable interface is what starts the data being pushed from the source.

In this Mod360 monad, we have deliberately gone to a purely push paradigm. Calling the Subscribe method from the destination end does not intiate the dataflow. Instead we keep a reference to the source, and have a Run method in the source. This makes an object structure that is more purely a push system, because the initaition of the dataflow is not done by a pull call from the destination end. This is much closer to how ALA works for its default programming paradigms.

That completes our four examples of refactoring imperative code using the monad refactoring pattern. We are now in a position to understand the general monad refactoring pattern for composition of functions.

6.1.6. The monad pattern

In the examples of Bind above, the type that Bind takes and returns for chaining is generally a class or interface. A class is like an interface with only one implementation, so we are generally going to think of it as an interface. We did have one example where it was an integer, and one where it was a Func, but these too can be thought of an interface in a broad sense.

The interface can be anything we want for the refactored code to communicate along the chain. It can be an actual interface, such as IEnumerable<T>, or IMaybe<T>, or it can be a class such as Task<T>. Or it can be a complex interface that we write to get any common information we want through the chain.

Bind always takes this interface and returns the same interface. You can therefore chain Bind calls together using fluent syntax.

The interface is usually generic, so takes a type as a parameter, e.g. IEnumerable<T>. The Bind function takes an Interface<T> and returns an Interface<U>. So the generic type can change as it goes along the chain.

The pattern is about composing functions. These functions gnerally take a T and return an Interface<U>.

Here is an application that composes three functions using a Bind function:

var I4 = source.Bind(function1).Bind(function2).Bind(function3);

When composing functions like this, you can’t explicitly see the type of the interface that’s being used. I sometimes insert a decorator to write the type to the console:

var I4 = source.Type().Bind(function1).Type().Bind(function2).Type().Bind(function3).Type();
public static T Type<T>(this T source) { Console.Writeline(typeof(T)); }

Here is pseudo code showing the actual types:

Interface<T> I1 = source;
Interface<U> I2 = I1.Bind(func<T, Interface<U>>);
Interface<V> I3 = I2.Bind(func<U, Interface<V>>);
Interface<W> I4 = I3.Bind(func<V, Interface<W>>);

As you can see, while Bind always takes an interface and returns the same interface, the generic type may change along the way. In our examples above we didn’t change the type much, but remember that you can.

Here is a diagram of the general monad pattern.

MonadPattern.png
Figure 51. The monad pattern

As you can see, monads are a 2-layer pattern. The two layers correspond roughly with ALA’s application and programming paradigms layers. The code that uses Bind to compose functions, and the lambda functions themselves are in the application layer. The Bind function and the Interface<T> are in the programming paradigms layer. Often monads come with a set of more specialized functions such as Sort, Filter or Sum. These would go in the equivalent of the domain abstractions layer. These functions either use Bind, or do the equivalent logic as Bind themselves.

In the higher layer you have the functions that you are composing to build a specific application. In the lower layer, the Bind function contains the common refactored code. Everything is more abstract and more reusable in the lower layer.

The functions that are being composed take a T and return an Interface<U>. It is tempting to think that the Bind function simply returns the Interface<U> that is returned by the function, because they have the same type. But that is not usually the case. Bind usually creates a new object that implements Interface<U>, and then combines information from both the input Interface<T> and the output of the function to provide the output Interface<U>. That’s what the diagram is trying to convey.

In many explanations of monads, they call the interface the monad type, or a wrapped type, or a container type, or a type in a box, or an amplified type, or just the notation M T. I don’t think any of these forms are helpful in understaning monads. The wrapped, container and box terms don’t work well for deferred monads, which don’t actually contain a value. They contain a means of getting a value. For example, the deferred version of a list is IEnumerable. If our function returns an IEnumerable, that’s not really a container or box.

The term amplified just introduces another seemingly abstract concept which is unnecessary. And the term Monad type or the notation M T seems a bit circular - let’s not explain monads in terms of monads. So I prefer to think of the thing that the Bind function takes and returns as simly an interface. It sometimes has one implementation, such as Task or List, but often it has more than one implementation such as IMaybe or IEnumerable. Usually the Maybe monad uses IMaybe with two implementations, one for when there is a value and one for when there is no value.

So generally I just think of it as Interface<T>.

The monad pattern requires three things: * an Interface<T> * a constructor or method for making ordinary values of type T into an object that implements Interface<T> * a Bind function that takes an Interface<T>, returns an Interface<U>, and is passed a function of the form Func<T, Interface<U>>.

The constructor or method for getting ordinary values into Interface<T> form is required to get started at the beginning of a chain. At the end of the chain, we can always get values back out because the Interface<T> always provides that.

Bind can pipe any extra information or capability we want through the interface. We could, for the sake of a silly example, pipe through an audio stream if we really wanted to. The bind function would take care it.

SelectMany vs Select

The LINQ opertors such as Select and SelectMany use IEnumerable as their composition interface - they take an IEnumerable and return an IEnumerable. This allows them to be composed in chains using dot operators.

Select is like Map. It takes a function that maps inputs to outputs in one to one correspondence. Aggregating operators such as Sum produce a single output from many inputs. SelectMany is the opposite - it produces many outputs from a single input.

Select is probably the most common operator used in LINQ statements. So why is SelectMany the fundamental Bind operator and not Select?

It’s because SelectMany is the one that strictly fits the monad pattern as shown in the previous section. For a monad, the function being composed also returns the composition interface. SelectMany is the one that does that. Select only takes a function that returns a single value.

So while we sometimes think of LINQ as being monads, strictly speaking only SelectMany is part of the IEnumerable monad.

Summary of monad benefits.

Monads allow us to simply compose functions declaratively in the top layer to implement a user story. How everything executes is handled by the Bind function in a more abstract lower layer.

The declarative code in the top layer is a different programming paradigm from imperative. It’s called dataflow, because we are directly composing a flow of data, irrespective of how the underlying execution will work.

Monads make it possible for the application code to concentrate on expressing user stories, and not be concerned with execution details.

Monads take care of passing data from function to function within Bind, without the application layer code needing to handle it.

The execution code in the Bind function can handle many different cases of logic that would otherwise have been messy imperative code between function calls.

We can compose as many functions as we like in chains of arbitrary length without any nesting of brackets or indenting.

Monads make it possible for application code itself to be pure functional code, even though the structure of connected objects that is built is not.

The application code examples that use the deferred versions of Bind look much the same as the immediate versions. That’s because at the application level, we are just declaratively composing functions.

We prefer to implement deferred versions of Bind because then we have the option of executing them straight away as if it was immediate, or use them as part of a larger program for later execution.

Deferred monads make it possible to completely separate all code that expresses user stories from code that implements computing details.

6.2. ALA compared to monads

Now that we have an understanding of monads, and deferred/push monads in particular, we are in a position to compare them with ALA.

In chapter three, we compared ALA and monads.

The points were:

  • In the application layer, monads compose functions whereas ALA composes objects with ports.

  • Composing with monads is a dataflow programming paradigm, whereas composing objects with ports is a multi programming paradigm.

  • Composing monads creates mostly a chain structure whereas composing objects with ports creates an arbitrary network structure. Monads can be networked as when two streams are merged, but in practice most functions have a single input and single output port.

  • Both deferred monads and ALA build a structure of objects which is subsequently executed in a second phase. This separates declarative application code from computing code.

  • Both monads and ALA use pure functional code for the application code in the top layer. In this respect ALA and monads achieve the same job by putting the dirty computational work inside a pre-written Bind function or class. This dirty work can include private state and I/O side effects.

  • ALA’s domain abstraction objects are more versatile than functions because they can more naturally have many ports, and these ports can use different programming paradigms. This allows for abstractions suitable for composing all aspects of user stories, such as UI, schema, business rules, etc.

    For example, you can have a single domain abstraction with a UI port (to be attached somewhere in the UI) multiple event driven ports (for mouse clicks) and a dataflow port (for binding to a data source).

  • Dataflow ports can each use either push or pull as appropriate in each particular case, whereas monads tend encourage you to use only one type or the other as a programming style, e.g. 'reactive programming'.

  • 'Push' dataflow interfaces can be used for either synchronous or asynchronous dataflows. So in ALA we default to using push style dataflows unless 'pull' has a particular advantage in a particular case. This allows instances of abstractions to be wired either synchronoulsy or asynchronously. In other words the choice of syncronous or asynchronous is deferred until the application user stories are written. Asynchronous can be chosen for example when two instance of abstractions will communicate over a network, or on on differnt threads, and synchronous can be chosen when the two will communicate on the same thread.

    'Push' style dataflows (reactive extensions) appear to be less popular. I don’t understand why. Perhaps it’s because the IObservable interface isn’t a true push style since the destination usually starts the flow of data by Subscribing (cold observables)? This mix of pull and push behaviour in the IObservable/IObserver pair is confusing and not easily amenable to network or miltithreaded systems that would otherwise suit push programming paradigms. Hot observables do not need the pull to initiate the data flow, but they have to avoid using both OnCompleted and OnError, otherwise the whole chain must be resubscribed. So they don’t use the full benefits of the IObserver interface.

  • ALA programming paradigms, which are usually interfaces, are analogous to monad interfaces. ALA programming paradigm interfaces can do anything that monad interfaces can do, such as IMaybe, IEnumerable, or Task or futures.

  • A monad’s Bind function is partially analogous to ALA’s WireTo function, because it implements the wiring. However the Bind function is different for every different monad type because it includes the deferred, run-time, common, execution code of the monad. ALA’s WireTo function only does the wiring. It does not normally include any common run-time code, although it can sometimes be overridden to do special wiring. Instead, in ALA, that common code goes into programming paradigm interface, which may use intermediary objects. WireTo is generally the same WireTo for all programming paradigms and therefore all wiring up of an entire application.

  • Monads usually use deferred execution and ALA always uses deferred execution, so in this respect they are similar. Both build an object structure which you then run after the wiring up is completed. They both have two phases, the wiring up phase and the run-time execution phase. However, in ALA, we always separate out all the wiring code for the entire application and then set the whole application running. Deferred monads are often wired up and then executed in the same code statement.

    By building the entire application first, ALA completely separates code into a top layer at the abstraction level of specific user stories, and a second layer that consists of domain abstractions that contain all the code that executes at run-time. In this way the top layer has all the declarative code that expresses the application and the second layer has all the imperative code that knows how to do general computation work at runtime.

  • ALA’s application layer corresponds loosely with functional code that composes functions. ALA’s programming paradigms correspond loosely with Bind functions. And ALA domain abstractions correspond loosely with the set of methods that generally come with a monad library such as Select or Where. With respect to these three layers, monads are a pattern that is consistent with ALA’s fundamental constraints.

6.2.1. Composing with plain objects instead of functions.

By using plain objects the barrier to understanding seems lower than for monads, at least for developers already familiar with objects. Functional programming, and monads in particular, seem to have quite a high barrier to entry unless you are a mathematician. The world needs the programmers who are able to understand objects but do not necessarily understand mathematical notation. I’m not sure what would happen if all universities only taught functional programming so that everyone is introduced to pure functions first. Perhaps then it would be objects which have a barrier to entry.

ALA’s domain abstraction objects are easier to understand than monads because they are plain objects. The mental model of composition in ALA is wiring instances of domain abstractions by their ports, which is conceptually known as the component model. Monads compose functions so the mental model is primarily oriented to composing a chain of functions as a dataflow. To make an analogy with electronics, ALA is like composing ICs (integrated circuits with many pins) and monads is more like composing two-port components such as resistors, capacitors, inductors and transistors.

There seems to exist computing problems that are best described using state. Objects are the language feature that provides for this. Monads end up using objects with state anyway - they are just hidden beneath the covers.

The only slightly unusual thing about ALA’s domain abstraction objects and plain object oriented objects is the use of ports. Posrt are used for all run-time input and outputs. Any programmer with familiarity with dependency injection can understand that a port is just an implementiion of dependency injection. A port is implemented simply as a field of the type of an interface, or is an implemented interface. As with normal dependency injection, the field is assigned a reference to another object that implements the port interface.

Unlike conventional dependency injection, the field is not assigned by the constructor or any setters. Instead the field is always assigned through use of WireTo or WireIn.

The bigger difference between ALA and conventional dependency injection is that the interface used must be more abstract than either of the classes. It is not even an abstract base class. It is even more polymorphic than that. This type of interface is called a programming paradigm, and can be implemented by many disparate classes. Therefore, the dependency injection cannot be container based. Instead the application code must explicitly instantiate the objects and then wire them together.

Because ALA uses plain objects, and plain interfaces as their ports, ALA developers can add new domain abstractions and programming paradigms themselves to build a DSL for expressing their user stories. In the functional world, developers can certainly write new monad types, but it doesn’t seem that easy, and seems generally left to library developers. The abstraction level of these libraries is therefore generally not as close to the domain, and does not make a DSL.

6.2.2. ALA vs monad syntax

Although ALA supports multiple programming paradigms, the dataflow programming paradigm is quite a common one. So we will inevitably need domain abstractions like those that come with monad libraries like Select, Where and Sum. So it is worth comparing the syntax of dataflows.

Here we are comparing the code in the top layer, the code that describes a user story. Both monads and ALA use fluent style with dot operators. For monads, the composed abstractions are functions that return the same interface, so they can be chained directly:

source.Filter(x=>x>=0).Select(x=>sqrt(x))

In the Monad version above, the Filter and Select functions effectively do both the wiring and specify the operation to be wired.

In ALA, WireIn is just a generalized object composition operator, so we need to separately instantiate an object that does the operation we want:

source.WireIn(new Filter<int,bool>(x=>x>=0)).WireIn(new Select<int,int>(x=>sqrt(x))

Usually this code is generated from a diagram. However, there is nothing stopping us achieving exactly the same syntax as the monad version if we really want to. We just create some extension methods:

namespace DomainAbstractions
{
    static class ExtensionMethods
    {
        public static IBindable Select<T, U>(this IBindable source, Func<T,U> function)
        {
            var select = new Select<T, U>(function);
            source.WireIn(select);
            return select;
        }
    }
}
namespace DomainAbstractions
{
    {
        public static IBindable Filter<T>(this IBindable source, Func<T,T> function)
        {
            var filter = new Filter<T>(function);
            source.WireIn(filter);
            return filter;
        }
    }
}

The code for these extension methods go in the same abstractions as the Select and Filter classes respectively.

Note that IDataFlow is the interface that is actually being wired. IDataFlow goes in the forward direction (the same direction as the data flow). So the Select and Filter extension methods can’t be defined on IDataFlow or IObservePush. We need an interface on which to define Select and Filter. This interface must go in the reverse direction towards the source. So that’s what IBindable is for. IDataFlow and IBindable are somewhat analogous to the IObserver and IObservable interfaces respectively. Note, though, that IBindable only exists to give us an interface on which to define the extension methods. It doesn’t do anything else, so it contains no methods:

interface IBindable {}

Note: normally IBindable would have a type parameter: IBindable<T>. That would allow type inference to be used for the type parameter of methods that are defined on it such as Bind<T,U> and Select<T,U>, etc. However, the compiler can’t always successfully use type inference for that second type parameter. It can if the second parameter passed to Bind is a function that returns a certain type. But it can’t if the second parameter is an action, which is the case for the ObServerPushAction domain abstraction. Therefore I have removed the type parameters from IBindable so that its less confusing, and Bind etc will always need to have its types passed explicitly.

In ALA we use the explicit WireIn and new operators for the following reasons:

  • In ALA, Domain abstractions are generally at a slightly more specific level of abstraction than monad library functions (specific to the domain to support the construction of user stories). So, domain abstractions are written by the application developer much more frequently than new monads are written. They are extremely simple to write once the concept of ports is understood, because the ports make them zero coupled with one another and with the application layer above. The only difference from plain classes is that you have to know that input ports must use implemented interfaces from the programming paradigms layer, and output ports must be plain private fields of the types of these same interfaces. We don’t want the extra burden of adding a corresponding extension method.

  • In ALA we can choose between WireIn and WireTo depending on whether we want to chain instances of abstractions or do fanout wiring. Monad library functions alway return the next object in the chain, so only naturally wire up chains.

  • The mental model of components with ports that you explicitly wire up is more versatile than the mental model of composing functions as a dataflow chain. Functions can be thought of as have multiple ports, for example the merge function can have two input streams, but the fluent syntax of combining monadic functions does not suit it.

  • Composing monad functions is only a dataflow programming paradigm. In ALA many diverse programming paradigms can be used. The diverse programming paradigms represent different meanings of composition. For example, we can compose the UI. The code below puts a Button, TextBox and Grid inside a window.

    ALA wiring code
        window.WireTo(new Button().WireIn(...))
              .WireTo(new TextBox().WireIn(...))
              .WireTo(new Grid().WireIn(new DynamicDataSource().WireTo(...)));

    The button can be further wired using an event driven programming paradigm. The Textbox can be further wired to its data using dataflow. The Grid can be wired using a dynamic dataflow programming paradigm to a dynamic data source, which could itself be wired using a schema programming paradigm.

  • Deferred monads look like operations on data, but obscure the fact that they build a structure of objects for later execution. This is confusing until you get used to it. The WireTo and WireIn operators together with the new operator make it explicit that you are building a structure of objects as a program that you can then set running.

  • Because domain abstractions can have multiple ports, WireIn and WireTo allows us to specify which port we want to wire when it could be ambiguous.

  • Inherent in the requirements of a typical application is really a network of relationships. This network is often best represented by a diagram. Explicit WireIn and WireTo operators allow us to directly translate a diagram to code. Also, diagramming tool can automatically generate the wiring code containing using WireTo and new.

6.3. Using monads in an ALA application

Although composing with objects is generally more versatile than composing with functions, if you already have a monad library containing functions such as SelectMany, Select, Where, Sort, Aggregate, etc, we would certainly want to make use of it in ALA applications. There would be no sense in reinventing that functionality as 2-port classes. You can use the monad library for some dataflow parts of the program.

In this section we discuss two methods to use monads within an ALA application:

  1. The first method is to use IObservable as the interface for some of the ports of your domain abstractions. Then two instances of these domain abstractions can be wired with a reactive extension expression inbetween. Although n IEnumerabe version is possible, we will only show an example using the IObservable interface because that is more compatible with how ALA programming paradigms gnerally work (push by default). We will give examples for both static and dynamic type dataflows.

  2. The second method is to write a general purpose domain abstraction that can be configured with a monad expression. The domain abstraction has input and output ports using ALA’s DataFlow interface. We will do both an IEnumerable and an IObservable version of this domain abstraction.

6.3.1. Domain abstractions with IObservable ports

Statically typed

The first way to use monads with ALA is to use the monad interface, such as IObservable, for the ports on some domain abstractions. For example, we could have a domain abstraction for a CSV file reader that has an output port of type IObservable<DataType>. Then we can have a domain abstraction called ObservableToSerial that has an input port of type IObservable.

We can then wire there two end instances via some .Where or .Select functions inbetween using monad functions that already exist in the reactive extension library. Here is some example application code:

class Program
{

    static void Main()
    {
        var outputer = (6)
        ((IObservable<DataType>)new CSVFileReaderWriter<DataType>() { FilePath = "DataFile1.txt" }) (1)
        .Select(x => new { Firstname = x.Name.SubWord(0), Number = x.Number+1 } ) (3)
        .Where(x => x.Number>48) (4)
        .WireInR<T>(new ObservableToSerial<T>(Console.Writeline)); (5)

        var program = new StartEvent().WireTo(outputer); (7)
        program.Run(); (8)
    }

    private class DataType (2)
    {
        public string Name { get; set; }
        public int Number { get; set; }
    }
}
1 We start the chain by instantiating a CSVFileReaderWriter and providing it with a filepath.
2 We also give CSVFileReadWriter a type, DataType, which corresponds with the fields in the CSV file we are going to read. (This is not a dynamic CSV file, so we are going to do this program completely with compile-time type checking using type inference.)

The CSVFileReaderWriter domain abstraction can have multiple output ports of different types, but the one we are going to wire is an IObservable<DataType>. CSVFileReaderWriter implements this interface. To specify which port we are wiring we simply cast the CSVFileReaderWriter to IObservable<DataType>. It’s a shame we had to have DataType appear twice in the program.

1 We wire the CSVFileReaderWriter’s IObservable port to a Select function. Like a monad, Select returns another IObservable, with a different type. The compiler can use type inference to generate this type.
2 We wire the output of Select to a Where function. Where returns yet another IObservable with a type using type inference.
3 We wire the output of Where to a new domain abstraction called ObservableToSerial. (The type inference doesn’t work here, but we will fix that soon.)
4 We store the ObservableToSerial in a local variable called outputer because we need to wire to it in another place.
5 outputer has an IEvent input port which is used to start the transfer. With IObervables, the data transfer is started from the destination end. We wire a StartEvent domain abstraction to the outputer. StartEvent has an IEvent output port and can be used to set a program running. We store the StartEvent in a variable called program.

<9) To start the program running we call program.Run(), which is a method in the StartEvent.

The line labelled 5 in the listing doesn’t compile. It’s what we would like to have written to get the type inferencing working starting from the CSVFileReaderWriter right through to the outputer. The reason it doesn’t compile is that new ObservableToSerial<T> needs a type to be specified for T. The WireInR<T> knows the type from its this parameter. But you can’t get the compiler to transfer that type to the second parameter of WireInR, the new ObservableToSerial<T>.

The solution is to use an extension method to do the WireInR and the new ObservableToSerial<T>. Here is a suitable extension method:

public static ObservableToSerial<T> ToConsole<T>(this IObservable<T> observable) where T : class
{
    var o = new ObservableToSerial<T>(Console.WriteLine);
    observable.WireInR(o);
    return o;
}

Using this extension method, here is the application again:

    static void Application()
    {
        var outputer =
        ((IObservable<DataType>)new CSVFileReaderWriter<DataType>() { FilePath = "DataFile1.txt" })
        .Select(x => new { Firstname = x.Name.SubWord(0), Number = x.Number + 1 })
        .Where(x => x.Number > 48)
        .ToConsole();

        var program = new StartEvent().WireTo(outputer);
        program.Run();
    }
}

Type inference now works all the way through the dataflow chain. We only had to specify the type of the data in the CSV file.

Dynamically typed

This next example does the same functionaility as the previous example, that is demonstrating mixed use of domain abstractions with IObservable ports and reactive extension monads. However, this time it does not statically define the data type at compile-time. In other words, it makes no assumptions about the data schema in the CSV file. Instead, it determines everything at run-time. If any code tries to access specific data that doesn’t exist or has the wrong type, run-time exceptions are thrown rather than compiler errors.

Since the CSV file is now considered dynamic, it has two header lines, one to name the columns and one that defines the types of the columns. Knowledge about these header lines is contained in the CSVFileReaderWriter abstraction. The first half of the code writes some data to the CSV file to ensure header lines are created.

static void Application()
{
    var csvrw = new CSVFileReaderWriter() { FilePath = "DataFile2.txt" }; (1)

    // First write some data to the file

    IObserverPush<ExpandoObject> writer = csvrw; // writer port (2)
    writer.OnStart(); (3)

    dynamic eo = new ExpandoObject(); (4)
    eo.Number = 47; (5)
    eo.Name = "Jack Up";
    writer.OnNext(eo); (6)

    eo.Number = 48; (7)
    eo.Name = "Wynn Takeall";
    writer.OnNext(eo);

    eo.Number = 49;
    eo.Name = "Rich Busted";
    writer.OnNext(eo);

    writer.OnCompleted(); (8)

    // Now wire the output port of the CSVFileReaderWriter via a Select and a Where to an outputter.

    var outputer = new ObservableToSerial<ExpandoObject>(Console.WriteLine); (9)(10)

    ((IObservable<dynamic>)csvrw)  (11)
        .Select(x => new { Firstname = ((string)x.Name).SubWord(0), Number = x.Number + 1 })
        .Where(x => x.Number > 48)
        .WireInR(outputer);

    var program = new StartEvent().WireTo(outputer); (12)
    program.Run(); (13)
}
1 First we instantiate a CSVFileReaderWriter domain abstraction. Notice how this abstraction is not generic like we had before. Instead its ports use the ExpandoObject class.
2 Get a reference to the input port of the CSVFileReaderWriter. This input port has type IObserverPush<ExpandoObject>. The IObserverPush programming paradigm is like IDataFlow, but can handle batches of data:
interface IObserverPush<T> : IObserver<T>
{
    void OnStart();
}

As you can see it is a standalone version of IObserver. It doesn’t need a corrsposnding IObservable interface. It operates by itself. It is explained in detail later.

3 Calling OnStart on the input port causes CSVFileReaderWriter to create a new file.
4 Create a temporary ExpandoObject. This is a usefull class when using dynamic typing which can have properties added at run-time.
5 Add fields to the ExpandoObject.
6 Give the ExpandoObject to the input port of the CSVFileReaderWriter, which will write the data to the CSV file.
7 Write more records in the same way.
8 Complete writing the CSV file.
9 Instantiate an ObservableToSerial for the end of the chain. ObservableToSerial has an IEvent input port called trigger that is used each time we want the program to go (by subscribing to its data source).
10 Configure the ObservableToSerial to output to the console.
11 Wire the chain up starting from the CSVFileReaderWriter through to the ObservableToSerial via two LINQ operators. IObservable ports are used the whole way. The IObservable output port of the CSVFileReaderWriter is selected by the cast.

One unusual thing you may notice about this ALA program is the use of WireInR instead of WireIn that we would normally use to wire things in a chain. A.WireInR(new B()) actually wires in the reverse direction from normal, that is from B to A. You use it like you would use WireIn, in the same direction as the dataflow, but it actually wires in the opposite direction. This is because IObservable, the programming paradigm interface being used, must be wired in the opposite direction as the dataflow, like a pull interface. The A object implements IObservable and the B object has a field of type IObservable. So the wiring must go in the reverse direction of the data flow. WireInR is implemented simply as WireInR(this object A, object B) {WireIn(B, A);}

12 Wire an instance of a StartEvent to the ObservableToSerial Trigger input port.
13 Make the program run by telling the StartEvent to output an event.

6.3.2. Domain abstraction configured with monads

This is the second method of using monads in an ALA application.

It uses a domain abstraction that can be configured with a monad expression. This domain abstraction uses your normal ALA dataflow programming paradigm forits its input and output ports. We will do two versions, one configured with an IEnumerable chain and one configured with an IObservable chain.

Configuring with IEnumerable monads

Let’s call the domain abstraction EnumerableQuery. You configure an instance of EnumerableQuery with a LINQ expression.

Here is an example program using EnumerableQuery. EnumerableQuery uses IDataFlow as the programming paradigm for its ports. We chain up three of them and configure them all with a similar query:

static void Application()
{
    var proxySource1 = new EnumerableProxySource<int>(); (1)
    var query1 = proxySource1.SelectMany(x => new[] { x * 10 + 1, x * 10 + 2, x * 10 + 3 }).Select(x => x + 1); (2)
    var proxySource2 = new EnumerableProxySource<int>(); (1)
    var query2 = proxySource2.SelectMany(x => new[] { x * 10 + 1, x * 10 + 2, x * 10 + 3 }).Select(x => x + 2); (2)
    var proxySource3 = new EnumerableProxySource<int>(); (1)
    var query3 = proxySource3.SelectMany(x => new[] { x * 10 + 1, x * 10 + 2, x * 10 + 3 }).Select(x => x + 3); (2)

    var userStory = new StartEvent(); (3)
    userStory
    .WireIn(new ValueToDataFlow<int>(0)) (4)
    .WireIn(new EnumerableQuery<int, int>(proxySource1, query1) { instanceName = "Query1" }) (5)
    .WireIn(new EnumerableQuery<int, int>(proxySource2, query2) { instanceName = "Query2" }) (6)
    .WireIn(new EnumerableQuery<int, int>(proxySource3, query3) { instanceName = "Query3" })
    .WireIn(new DataFlowToSerial<int>(Console.Write)); (7)

    userStory.Run(); (8)
}
1 To build a LINQ expression, you need to start with a source that implements IEnumerable. Since we don’t have an actual source, we will use a proxy for the source. That’s what the EnumerableProxySource is.
2 An example LINQ expression consisting of a SelectMany and a Select.
3 Instantiate a StartEvent domain abstraction, which we will use to tell the user story to run.
4 Instantaite a domain abstraction that represents a simple scalar value and will output that value to its output IDataFlow port when told to by its IEvent input port.
5 Wire in the first of the three EnumerableQuery domain abstractions. EnumerbaleQuery has an input IDataFlow port and an output IDataFlow port. Configure it with the LINQ expression. To do that we give it both the proxySource object and the LINQ expression object. EnumerableQuery will receive data pushed to its input port, apply the LINQ expression to it, and push the result out its output port.
6 We chain up three of the EnumberableQuerys to demonstrate normal ALA wiring up of this abstraction.
7 We wire the final output to the console using an instance of a DataFlowToSerial domain abstraction configured to give its serial stream to the Console.
8 Tell the user story to run.

Here is the output of the program:

ConsoleOutputListMonad.png
Figure 52. Output of demo code using EnumerableQuery domain abstraction

The initial value of zero expands to 27 numbers because of the SelectMany in each of the LINQ expressions.

Internally, what EnumerableQuery does is every time an input data arrives, it executes a foreach on the query. When the query asks for for data from the proxySource, the proxySource returns the data that came into the input port.

Here is the EnumerableQuery domain abstraction:

class EnumerableQuery<T, U> : IDataFlow<T>  // input port (1)
{
    private readonly EnumerableProxySource<T> proxySource;
    private readonly IEnumerable<U> query;

    public EnumerableQuery(EnumerableProxySource<T> proxySource, IEnumerable<U> query) { this.proxySource = proxySource; this.query = query; proxySource.Enumerable = getIEnumerableForInputData(); } (3)

    private IDataFlow<U> output;  // output port (2)

    private T inputData;

    private IEnumerable<T> getIEnumerableForInputData() (4)
    {
        yield return inputData;
    }


    void IDataFlow<T>.Push(T data) (5)
    {
        this.inputData = data;
        foreach (var x in query) output?.Push(x);
    }
}
1 The input port is an IDataFlow<T>
2 The output port is an IDataFlow<T>
3 The constructor takes a LINQ expression (both its proxySource object and the query itself) and saves them to local variables. The constructor also sets up proxySource to get its data from an IEnumerable.
4 The IEnumerable is returned by getIEnumerableForInputData. This IEnumerable simply returns the data that has come in on the input port. The IEnumerable is implemented with a method that contains a yield return.
5 The implementation of the input port is what drives the domain abstraction. It first saves the incoming data so that the query can use it as its source, then enumerates the LINQ query. The results are given to the output port.

The code above can be found in a working example program on Github in the IEnumerableMonad repository here: https://github.com/johnspray74

Configuring with IObservable monads

In the previous example, we created a domain abstraction that can be configured with a LINQ query. In this next example, we create a domain abstraction that can be configured using reactive extensions.

When all else is equal, I prefer reactive extensions over LINQ because it can do asynchronous and synchronous.

We could have done this example using IDataFlow ports just as we did in the previous example. However, the IObserverPush programming paradigm that we briefly introduced earlier is more appropriate. IDataflow handles an open-ended stream of data, whereas IObserverPush can handle open-ended batches of data.

Here is a user story example of using this domain abstraction.

    static void Application()
    {
        var subject1 = new Subject<int>(); (1)
        var query1 = subject1.SelectMany(MutiplyBy10AndAdd1Then2Then3).Select(x => x + 1); (2)
        var subject2 = new Subject<int>();
        var query2 = subject2.SelectMany(MutiplyBy10AndAdd1Then2Then3).Select(x => x + 2);
        var subject3 = new Subject<int>();
        var query3 = subject3.SelectMany(MutiplyBy10AndAdd1Then2Then3).Select(x => x + 3);

        var userStory = new StartEvent(); (3)
        userStory
        .WireIn(new ValueToObserverPush<int>(0)) (4)
        .WireIn(new ObservableQuery<int, int>(subject1, query1)) (5)
        .WireIn(new ObservableQuery<int, int>(subject2, query2))
        .WireIn(new ObservableQuery<int, int>(subject3, query3))
        .WireIn(new ObserverPushToSerial<int>(Console.Write)); (6)

        userStory.Run(); (7)
    }


    static IObservable<int> MutiplyBy10AndAdd1Then2Then3(int x) (8)
    {
        return Observable.Create<int>(observer =>
        {
            observer.OnNext(x * 10 + 1);
            observer.OnNext(x * 10 + 2);
            observer.OnNext(x * 10 + 3);
            observer.OnCompleted();
            return Disposable.Empty;
        });
    }
1 First we need a proxy source object that implements IObservable on which to build our reactive extensions expression. The RX library provides a suitable class that we can use for this called Subject.
2 Create the RX expression consisting of a SelectMany and a Select.
3 The user story will consist of three ObServableQuerys wired up in a chain. Data transfers are intitaited at the source. So we instantiate a StartEvent domain abstraction to give us a way of starting (or restarting) the dataflow.
4 The StartEvent instance is wired to the start port of an instance of ValueToObserverPush using an IEvent programming paradigm. ValueToObserverPush is a simple domain abstraction that is configured with a single value. It has an IObserverPush output, and will output its value when it gets a signal on its start port.
5 The ValueToObserverPush is wired to an ObservableQuery using the IObserverPush programming paradigm. ObservableQuery is configured with an RX expression. Both the subject and the expression itself must be passed in.
6 After the three ObservableQuerys are wired, the output is wired to an instance of ObServerPushToSerial, where it is converted to text to be displayed on the Console.
7 Now that the user story is all wired up, we can run it. It can be run more than once.
8 The function passed to the SelectMany is implemented using the Observable.Create method, which is a convenient way to do it for our purposes here.

Here is the code for ObServableQuery:

class ObservableQuery<T, U> : IObserverPush<T>  // input port (1)
{
    private readonly Subject<T> queryFrontEnd;
    private readonly IObservable<U> query;
    public ObservableQuery(Subject<T> queryFrontEnd, IObservable<U> query) { this.queryFrontEnd = queryFrontEnd; this.query = query; } (3)

    private IObserverPush<U> output;  // output port (2)


    private IDisposable subscription = null;

    void IObserverPush<T>.OnStart() (4)
    {
        output.OnStart(); (5)
        subscription?.Dispose(); (7)
        subscription = query.Subscribe( (6)
            (data) => output.OnNext(data),
            (ex) => { output.OnError(ex); terminated = true; }
            () => output.OnCompleted());
    }

    void IObserver<T>.OnNext(T data) (8)
    {
        queryFrontEnd.OnNext(data);
    }

    void IObserver<T>.OnError(Exception ex) (9)
    {
        queryFronEnd output.OnError(ex);
    }

    void IObserver<T>.OnCompleted() (10)
    {
        queryFronEnd.OnCompleted();
    }

}
1 The input port is an IObserverPush
2 The output port is an IObserverPush
3 The constructor configures the domain abstraction with an IObservable expression. Both the front end Subject object and the RX expression object itself are passed in. These are saved as local variables.
4 An OnStart call prepares for a new batch of data.
5 The OnStart signal is propagated to the output port so it goes right through the chain to prepare the entire chain for the data sequence.
6 The query that we were configured with is subscribed to. This does not cause data to flow from the RX expression yet. Outputs from the RX expression are routed directly to the domain abstraction’s output port. The subscribe must be done here rather than in the constructor because if OnStart is called again for a subsequent batch of data, Subscribing needs to be done again to 'reset' the RX expression if it had completed.
7 If the query had previously been subscribed to, it probably had an OnCompleted or OnError which would prevent it working until it is subscribed to again. But we don’t want to subscribe to it twice, so we first unsubscribe.
8 When data arrives at the input port, it is given to the query via the Subject object.
9 Any exception coming in from the input is passed through to the output via the query. If the query has already generated an exception it will likely discard it.
10 The OnCompleted from the input is also passed through to the output via the query.

That completes our examples of how you can use monads within an ALA application.

This has been a long section contrasting monads and ALA. In summary ALA solves the same problem that monads solve, that of composing more abstract computational units to create more specific computational units. But where monads are about composing functions, ALA composes objects. While pure functions are mathematically simpler than objects, many computations are more naturally expressed using state. That’s why ALA is object oriented. In other words, sometimes objects just make better abstractions than functions.

But if you already have an existing monad library, it makes sense to use it within an ALA application.

6.3.3. IObserverPush interface

We used the programming paradigm interface IObserverPush a couple of times in the examples previously.

Here it is again:

interface IObserverPush<T> : IObserver<T>
{
    void OnStart();
}

Nothice that it is the IObserver interface with one added method: OnStart.

Now we can explain the reasoning behind this programming paradigm, and compare it with the IObserver/IObservable pair.

IObserverPush<T> is similar to IDataFlow<T>, but with the ability to handle batched data and to propagate errors down the data flow.

This interface is the 'pure push' version of the IObservable/IObserver pair. Remember that while IObserver is a push style interface, IObservable is not. The Subscribe method of IObservable is more of a pull style that usually gets the flow of data started. So its hard to use IObservable asynchronously or over a network.

Data transfers using IObserver are usually triggered from the destination end. It does so via IObservable. For example, ObservableToSerial in the previous section must subscribe to 'pull' the data. So Subscribe does two things: 1) it wires the IObserver interface in the opposite direction, and 2) it (usually) starts the data transfer. The source will then push the data back using the IObserver interface.

Sometimes Subscribe only wires the IObserver, and thereafter the source initiates the data transfer whenever it likes. This is called a hot observable. OnCompleted and OnError cannot really be used with hot observables because they usually stop everything. If the source does use OnCompleted or OnError, then the source must really send the data in response to the Subscribe. If OnCompleted or OnError are called, the detsination must unsubscribe and resubscribe to get the next batch of data. The Subscribe method, therefore, is not just used for wiring - it is usually used to start the transfer.

I find this behaviour of IObservable/IObserver doesn’t suit permanently wired user stories like we do in ALA. Besides, IObservable and IObserver seems to be a weird mix of 'push' and 'pull' styles. What I want is a purely push programming paradigm, that can be permanently wired, can batch the data, and can propagate errors down the dataflow chain.

The other problem with the IObservable interface is that the destination wires itself to the source. The destination of a communication should never wire itself to the source if its in the same layer. It’s fine if the source is in a lower (more abstract) layer. But general wiring up within an abstraction layer should always be done by a higher layer.

So to fix all these problems with the IObservable/IObserver pair, I use IObserverPush as an ALA programming paradigm.

The OnStart method effectively takes the place of the Subscribe method in that it will allow data to flow again after an OnCompleted or OnError. In other words, we can permanently wire IObseverPush, and it will work for ongoing batches of data even after OnCompleted or OnError occurs in each batch. The wiring aspect of the Subscribe method is not needed. In ALA the layer above will wire up the IObserverPush interface. The IObservable interface is therefore completely redundant.

In summary, IObserverPush

  • is used instead of IObserver/IObservable

  • is a pure push programming paradigm

  • like IObserver, goes in the direction of the dataflow

  • requires the layer aobve to wire it up

  • is designed to be permanently wired, but can still handle batches of data using OnCompleted, or OnErrors.

6.4. Encapsulation, polymorphism and inheritance

ALA replaces encapsulation with abstraction.

ALA removes associations and inheritance and instead uses composition (provided the composition uses a more abstract abstraction).

ALA replaces polymorphism with zero coupling.

The first two we know as fundamental principles in ALA, and have already been discussed in chapter three.

The third statement requires some elaboration.

In the meme pool of software engineering we have at least five memes for the one concept. These are polymorphism, information hiding, protected variations, dependency inversion principle and open closed principle.

I shall argue in their individual discussion later that none of them is a principle. All five are just a simple pattern. The motivation is that if you have code that couples knowledge of different 'things', you extract the knowledge into their own modules. Now when the 'thing' changes, you can change it or swap it out without affecting the client module. Switch statements were a smell in traditional code that different things were mixed.

You may already have separated out one implementation of a thing. So now your client code talks to a concrete thing. The conical example is a particular database. But now you need to use a different thing. Instead of putting in a switch statement everywhere to talk to different databases, you use the polymorphism / information hiding / protected variations / dependency inversion / open closed pattern.

The pattern itself consists of an interface. That’s it. All those memes all trying to tell you to use an interface. Oh, and another one - if you have heard the phrase "program to interfaces".

On top of that, single responsibility also pretty much forces the use of an interface. Referring to a peer concrete object is always a second resposibility.

ALA does not use this pattern.

To understand why, lets call the client module B and the modules that implement the interface, C1, C2 etc. B doesn’t know which of the C modules it is talking to at run-time. If we want it to be C2 for a particular application, we have higher level code that injects C2 into B.

It’s important that we realize that in this pattern the interface is owned by B. It describes what B requires. It is cohesive with B. It is part of abstraction B. This still the case even if the interface is split out into a module or even a different compilation unit of its own.

Therefore C1, C2 etc have a dependency on B. They implement B’s requirements. They collaborate with it. The dependency in the design is just inverted from what it might have been. C1 & C2 are coupled with B.

So this is illegal in ALA (assuming B and C1, C2 etc are all at a similar level of abstraction, which they likely are. That’s why for ALA I have stated that the equivalent is zero coupling. ALA replaces the dependency with nothing at all between A and C1, C2 etc.

We have talked about how ALA still works in Chapters three and four. It does still use an interface but it is not owned by B (or C1 or C2). It is at a much more abstract level, the level of a programming paradigm. For example if abstractions B, C1 and C2 know about the event-driven programming paradigm, then instances of them may be wired together.

ALA further requires that the higher level code that does the injecting is also an abstraction. It is just one that is specific to a user story. Let’s call it A. A needs to cohesively do all the wirings of all the instances of domain abstractions to implement a whole user story in a cohesive way.

These five memes don’t have anything to say about that. They are redundant with respect to ALA. By just using ALA the job is done in a better way.

The SRP, DIP amd OCP are discussed further in the sections below.

6.5. SOLID Principles

The SOLID principles collated by Robert Martin are confusing. Their one or two sentence descriptions don’t describe them very well, so you have to go a read a lot to understand them. Unfortunately they are collected up into the catchy acrostic "SOLID" with a meaning that is undeserved. This has made the collection more well known than it deserves, as we shall explain.

6.5.1. Single Responsibility Principle

The SRP strangly worded differently from it’s name. It states that a module (function, class or package) should have only one reason to change. I find this s strange formulation of the name.

By using abstractions, the SRP is complied with in terms of reasons to change. However, some abstractions arguably have more than responsibility. I often use the question "What do you know about?" to an abstraction. It is always one thing it knows about, but it may have multiple responsibilites for that thing.

Examples:

  • An ADC driver (analog to digital converter hardware) knows all about a particular ADC chip. It has the responsibilies of initializing it and getting the readings from it. It changes only if the HW chip changes.

  • A protocol abstraction knows about a protocol. It has the responsibility to send data using the protocol and to receive it. It changes only if the protocol changes.

  • A file format abstraction, such as CSVFileReaderWriter knows about a file format. It has the responsibility to both read it and write it. It changes if the file format changes.

My advice is that the SRP is made redundant by thinking in terms of abstractions, which accomplishes the intention of the SRP better.

6.5.2. Open Closed Principle

Talk about confusing. Firstly Betrand Meyer coins the phrase, which is impossible to understand without further reading. On further reading you find that Robert Martin has a completely different principle by exactly the same name. Then he has two verions of that, one for modules in the same compilation unit and one for when the client is in a different compilation unit and is already published. By the way, being already published was also the context of Meyers OCP.

None of them are principles - they would need to be used in the right conext at best. They have associated patterns anyway (or anti-patterns relative to ALA).

Martins version

The sources of knowledge about the meanings of these memes are:

Craig Larman Kevlin Henny

6.5.3. Liskov Substitution Principle

TBD

6.5.4. Interface segregation principle

TBD

6.5.5. Dependency Inversion Principle

The DIP is stated:

  1. High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces).

  2. Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.

This sounds the same as the ALA fundamental rule that all dependencies must be on abstractions that are more abstract.

The Dependency Inversion Principle, and its associated pattern goes some way toward ALA in one respect and far too far in another respect.

Firstly ALA uses the word abstraction for the unit of code. The DIP really only uses the word abstraction as a synonym for interface – e.g. abstract class. The essence of the difference is that when ALA allows a dependency on an abstraction, it means more abstract than what DIP does. In both cases an interface is introduced. But in DIP, that interface is owned by the first module, and expresses what that module requires, so it’s highly coupled with the module, not really more abstract than it. ALA’s interfaces don’t belong to domain abstractions but go all by themselves in a lower layer. They are so much more abstract that we call them programming paradigms.

To be more precise, the DIP (as its name suggests) reverses a dependency used for communication between two classes, but ALA completely removes it. But the ALA wiring pattern also adds other dependencies. It adds a dependency on each module from a higher layer for dependency injection and it adds dependencies from each module to a programming paradigm interface in a lower layer for ports.

Let’s start with conventional code where B talks to C. It uses a dependency:

B ---→ C

DIP does this:

B < --- C

ALA does this:

B ---- > I

C ---- > I

Those who know the DIP might immediately say “no the DIP has a version where the interface is put into its own separate package like that as well”. The DIP allows for the interface to be placed in a different compilation package than B. Lets call it IB. Theoretically this allows C (the implementer of IB) to be reused without B. However, this is a superficial change from the point of view of abstraction level. Simply moving IB doesn’t make it more abstract. That interface is still owned by B - it represents what B requires. So as it still just a part of the B abstraction.

With DIP, you get to choose a specific implementation, C, to satisfy what B requires. In ALA you get a port with a programming paradigm that will take any domain abstraction instance with a compatible port of the same programming paradigm.

Both DIP and ALA require dependency injection. So let’s draw the injection dependencies as well:

Conventional code version:

B ---→ C

DIP version:

A --→ B

A --→ C

C ←-- B

ALA version

A ---→ B

A ---→ C

B ---- > I

C ---- > I

DIP effectively moves the interface from C to B. B gains an interface that does a similar job to C. C then implements it and B uses it.

Because the new interface is owned by B, it may be different from the one in C because now it’s about what B requires rather than what C provides.

Because of this, it might often be an adapter that implements the interface, and then the adapter uses the original interface of C.

TBD

Think of B as being some business logic and C being the database. B no longer depends directly on a specific database. But the databases do now depend on B. To avoid changing the databases, you would use adapters. The pattern is designed to increase the reuse potential of B, the business logic, because different databases can be plugged into it. But it likely decreases the reuse potential of the things around the business logic unless adapters are used. The DIPs application is primarily around making business logic reusable, and leads to hexagonal architecture, which has the business logic in the middle, and all the peripherals are plugged into its interfaces.

Returning to the sentence in the DIP that states: “High-level modules should not import anything from low-level modules.”.

The 2nd ALA dependency rule is in a way less constraining than the DIP here. If a low-level module is much more abstract, ALA allows to keep the dependency. This is what allows the dependencies between the application user stories and the domain abstractions. It comes down to what is meant by high-level and low-level in Martin’s writings. I think by ‘low-level’ he refers to what would have been depended on in conventional code. Things like the database, middleware for communications, and frameworks.(e.g. for supporting asynchronous events.)

In ALA, yes you would wire the specific database adapter and the specific middleware adapter (and the specific UI), but you wouldn’t wire in the framework. It doesn’t matter that the abstraction depended on is low level. I want to commit to only one implementation of the framework. It would be silly to have to use ports on every single domain abstraction so I can wire in a framework of my choice, and have to wire it to every single domain abstraction, when I want to commit to using one. This becomes more obvious as you get to even lower levels such as math libraries. I don’t need to allow for swapping out the math library implementation. So ALA allows dependencies on more abstract abstraction even if they are low-level modules. In fairness, Martin probably doesn’t mean to include all low-level modules in the DIP, just certain ones that should be decoupled.

6.6. Dependency injection pattern

By now we know that ALA uses dependency injection. It uses it for wiring up all instances or all domain abstractions.

We have favoured using reflection to do the injection in our examples, but that is just a syntactic shortcut that allows domain abstractions to have many ports without also having many setters. It also allowed us to keep the ports private from direct access by the application layer. It allows ports to be implemented very simply, without the need for setters at all. It allows some other interesting things to be done. For example, after an instances port has been wired, there may events in the interface of the port that need internally wiring to event handler methods. The wireTo method can look for and call a method in the instance to do this immediately after wiring.

ALA always uses explicit wiring. This is one of the most important aspects of ALA. It’s usually in the form of a diagram, because the wiring is usually an arbitrary graph. ALA never does dependency with automatic wiring. Having a dependency injection container means that the wiring itself is implicit in the interface types. If one module requires an interface, and the container has a module that implements it, that means these two modules get wired together. This type of implicit wiring is indirect and obfuscated and illegal in ALA.

In ALA, abstraction pairs don’t have their own interfaces for their instances to communicate. So we don’t have the situation where class A has a dependency on class B, and so an object of class B (or one of its subclasses) is injected into class A. Similarly, we wouldn’t have the situation where class A requires an interface that is implemented by class B.

In ALA the interfaces must be programming paradigm interfaces, which are a whole abstraction layer more abstract. So we need to be thinking that if class A accepts or implements a certain programming paradigm interface, there could be any number of other abstraction instances that could be wired to it. Furthermore, we could build arbitrarily large compositions. Some abstractions will have some ports that don’t need to be wired to anything. So it doesn’t really make sense to call what we are injecting 'dependencies'. We just think of it as wiring things up. You wouldn’t describe what an electronics engineer does as dependency injecting components into each other.

In ALA, the explicit wiring should not be XML or JSON. I do not consider these readable programming languages. They are data languages.

Usually user stories contain a graph structure of relationships. So the wiring should be a diagram to best show that structure.

However, if the graph is mostly a tree structure (with relatively few cross connections), then it may still make sense to avoid the weight of a diagramming tool, and represent the wiring in text form. But in this case I still much prefer the readability of code written in a programming language than XML or JSON. An argument can be made for the declarative nature of say XAML and that UI designers could learn this declarative language more easily than a programming language. But I would maintain that a the subset of the programming language needed to the equivalent of XML is declarative style. That’s what most of the wiring examples in this website are: declarative composition.

Besides, its not just about UI. For a given user story there will likely be UI, business logic, data transformations, and data storage. These should all be expressed togther cohesively. They should all be composed inside one abstraction. To handle the sometimes non-trivial configuration of the abstraction instances, normal code is sometimes needed, for example for lambda expressions or delegates. If we have a UI designer on the team, great, just teach him the subset of domain abstractions that are used for the UI, how to configure them, and how to compose them. Languages like XAML are not particularly easy just because they are declarative.

6.7. Physical boundaries

I was listening to a talk by Eric Evans where he said that Microservices works because it provides boundaries that are harder to cross. We have been trying to build logical boundaries for 60 years, he said, and failed. So now we use tools like Docker that force us to use say REST style interfaces in oder to have physical boundaries. I have also heard it suggested that using multiple MCUs in an embedded system is a good thing because it provides physical boundaries for our software components. And I think, really? Is that the only way we can be create a logical boundary? I can tell you that multiple MCUs for this reason is not a good idea if only because all those MCUs will need updating, and the mechanisms and infrastructure needed to do that make it not worth it. Unless there is a good reason, such as to make different parts of your code independently deployable, the extra infrastructure required for physical boundaries that are just logical boundaries is not necessary. Furthermore, physical boundaries, like modules do not necessarily make good abstractions. The only boundary that works at design-time is a good abstraction. So ALA achieves it’s design-time boundaries by using abstractions.

6.8. Test Driven Development

It is said that TDD’s main advantage is not so much the testing, but the improvement in the design. In other words, making modules independently testable makes better abstractions. This is probably true, but in my experience, TDD doesn’t create good abstractions nearly as well as pursuing that goal directly. The architecture resulting from TDD is better but still not great.

6.9. Observer pattern

TBD

6.10. Layer patterns

6.10.1. MVC

TBD

6.10.2. Application, Services, Drivers, Hardware

TBD

6.11. Factory method pattern

The Factory Method pattern in both the GOF book and in online examples has multiple variations. The only thing they seem to have in common is that the client doesn’t use "new ConcreteProduct()". It just wants an object that implements an interface, IProduct. For any reason it doesn’t want to be the one who will decides at design-time what that concrete product will be.

Here are some of the variations.

  • Several ConcreteCreators exists to encapsulate knowledge of how to use the ConcreteProduct constructor which has many parameters, in a consistent way to make a valid ConcreteProduct. The common example is different named pizzas or sandwiches.

  • The Client finds out at run-time what ConcreteProduct is needed (usually a string name). We want to move the switch statement out of the client and into a Creator class.)

  • The client knows when the objects are needed, but needs to be more stable. Which product is needed changes more often (although still known at design-time). So it goes into a class that changes.

In all cases we end up with two objects wired together through the IProduct interface. These two objects we will refer to as the Client and the ConcreteProduct (from the pattern terminology). To get them wired using the Factory Method pattern requires the use of a FactoryMethod. The FactoryMethod typically goes in an abstract class called ICreator, which may do the creating itself, or maybe overridden by one or more ConcreteCreators.

In the context of abstraction layers, ALA gives more insight into the FactoryMethods pattern. Remeber we expect lower layers to more stable. The IProduct and ICreator interfaces are in the ProgrammingParadigms layer (lowest layer). The Client and all the different ConcreteProducts are in the DomainAbstractions layer (middle layer). The ConcreteCreator is in the Application layer and wires one of the ConcreteProducts to the client. So now when we want to change the ConcreteProduct, only the ConcreteCreator in the application layer has to change.

But in ALA we typically accomplish that in a far simpler way. We commonly let the application code instantiate the right concrete class (that implements the interface, IProduct), and wire it to the Client object using the WireTo() method. This is nothing more than static wiring, but can only work when the required ConcreteProduct is known at design-time.

6.11.1. case 1

Now to the case in ALA where we have a client that needs a concrete product creating later than design-time, that is at run-time. Such a client is the Multiple Abstraction. It’s job is to make many instances of a Domain Abstraction. But it is an abstraction so can be used to make instances of any object. They don’t even have to implement a specific interface such as IProduct, because Multiple doesn’t interact with these instances itself.

6.11.2. case 2

Let’s say you have a Table domain abstraction that stores a table of data. In your application, you want to instantiate many Tables. Now lets suppose that we want these Table instances to persist their data. A database must be attached via an IPersistance interface. We don’t want the Table class to know about concrete Databases. We want the application layer at the top to do that. But we don’t want the application layer to have to wire the database to every instance that requires an IPersistance. We want the Application to be able to just use a Table as if it is a self-contained abstraction. We want the Table instances to take care of themselves for Persistence. So we make a Peristence abstraction in the Programming paradigms layer. The concept of Persistence is at the right abstraction level to go in this layer. The Table class can use this persistence abstraction through a FactoryMethod. A variable in the Persistence abstraction stores the IPeristence object. The application instantiates which database it wants to use and passes it to the Peristence abstraction.

6.12. Decorator pattern

TBD

6.13. Bridge pattern

TBD

6.14. Architecture styles

I am not an expert at these so called 'Architectural styles'. Any feedback about the accuracy of the following comparisons would be appreciated.

6.14.1. Components and connectors

David Garlan and Mary Shaw in their paper titled "An Introduction to Software Architecture" 1994 use components and connectors as a framework for viewing architectural styles. Depending on the style, the connectors can be a procedure call, event broadcast, database query, or pipe (which we call dataflow).

Similarities

ALA follows this idea closely.

Differences

In ALA we call the styles programming paradigms, and it is emphasised that multiple programming paradigms can be used in the one user story. The reason not to call them 'styles' in ALA is that the word style tends to imply using a single style throughout the program.

In ALA 'components' becomes 'abstractions' and 'connectors' becomes 'ports and wirings'. This change in terminology is to emphasis that the wiring is distinct from the abstractions themselves. The term components and connectors can (albeit not necessarily)) refer to an effectively monolithic system that is just separated into pieces and the pieces connected back together in a fixed rigid arrangement. This is especially true if the design methodology is decomposition of the system into elements and their relations. Such a system is loosely coupled at best. In ALA you can’t do that. Systems must be composed of instances of abstractions wired together by a higher layer abstraction that directs the wiring. Abstractions are necessarily zero-coupled with one another. They use ports that have the types of a small number of programming paradigms so that instances of them can be composed in (generally) an infinite variety of ways. The style where components being filters and connectors being pipes works this way.

I suspect that most components and connector systems use interfaces that are specific to the components.

Examples using the UML component diagram, even though it uses the term ports, show interfaces that rigidly couple their components to one another, for example, interfaces with names such as CustomerLookup. This would mean that only components that are implementations of that specific interface could be substituted. Usually there appears to be only one, making the components effectively just modules. In UML, components appear to be just containers. They are the first level of decomposition of a system, and themselves just contain connected classes. This type of architecture is incompatible with ALA.

6.14.2. Component Based Software Engineering

ALA uses many of the same methods found in component based engineering or the Components and Connector architectural style.

Similarities
  • Components are Abstractions.

  • Reusable software artefacts.

  • Connection ports for I/O.

  • Composability

  • Both instantiate components, specialize them by configuration, and compose them together to make a specific system.

  • ALA’s 3rd layer has interfaces used to wire abstractions in the 2nd layer, so at a lower level (more abstract) level. They represent something more like programming paradigms. The equivalent pattern in components engineering is "Abstract Interactions".

  • The architecture itself is composed of a generic part and a specific part. The general part is the ALA reference architecture itself and the components or the connectors architectural style. The specific part is the wiring diagram of the full system.

Differences
  • Component based engineering technologies such as CORBA primarily solve for platform and language interoperability in distributed system whereas ALA brings some of the resulting concepts and properties to everyday small-scale, non distributed development as well, where the only separation is logical.

  • In ALA there is perhaps more particular emphasis on making components clearly more abstract than the systems they are used in, and making the interfaces clearly more abstract than the components. The components are pushed down a layer and the interfaces down to a layer below that. Then all dependencies must be strictly downwards in these layers. In component based engineering, this structure is not necessarily enforced. If the components are just a decomposition of the system, then the system, components and interfaces may all be at the same level of abstraction, making the system as a whole complex.

  • ALA depends on the 'abstractness property' of components to get logical separation, and so calls them 'Abstractions' and not components to help them retain that property. Even if there will only be one use and one instance, it is still called an abstraction. This keeps them zero coupled and not collaborating with other abstractions they will be wired to.

  • ALA layers are knowledge dependency layers. Components may still be arranged in layers according to run-time dependencies, such as communication stacks. In ALA run-time dependencies are always implemented as explicit wiring inside another higher layer component.

  • ALA’s top layer must be a straight representation of the requirements, whereas components may tend to be decomposed pieces of the system.

  • ALA’s 2nd layer of components are designed for expressiveness of user stories or requirements, and provide DSL-like properties. ALA puts emphasis on the 2nd layer of components having the scope of a domain as the means of explicitly controlling the expressiveness of the pallet of components.

  • ALA is not fractal. In ALA the components of components are abstractions that become more abstract and thus ubiquitous and reusable. ALA therefore uses abstraction layers rather than hierarchies.

  • ALA forces decisions about which abstraction layers the software artefacts go into, and then controls knowledge (semantic) dependencies accordingly.

  • ALA tries to make the abstraction layers discrete and separated by a good margin.

  • ALA puts greater emphasis on wiring being able to represent any programming paradigm that suits the expression of requirements, and the use of many different paradigms in the same wiring diagram.

  • ALA emphasises the cohesion of functional parts of a system such as UI, logic and Data, by bringing them all together in one small diagram using domain level components

  • Instead of 'required' interfaces, in ALA they are called 'accepts' interfaces. This is because the abstractions are more abstract and composable, so, as with Lego blocks, there isn’t necessarily a connection to another instance.

6.14.3. Presentation, Business, Services, Persistence, Database

TBD

6.14.4. Presentation, Application, Domain, Infrastructure

The middle two layers appear to be the same as ALA’s. The Presentation (UI) only has run-time dependencies on the Application, and the Domain layer only has run-time dependencies on the Infrastructure (Persistence etc), so these layers are not present in ALA.

Instead Presentation is done in the same way as the rest of the application, by composing and configuring abstractions in the domain. The meaning of composition for UI elements (typically layout and navigation-flow) is different from the meaning of composition in the use-cases (typically workflow or dataflow).

In ALA, the foundation layer is also done in the same way as the rest of the application, at least a little. Domain abstractions that represent say a persistent table are in the Domain layer. The composition and configuration of them again goes in the Application layer. This time the meaning of composition is, for example, columns for the tables and schema relations.

If the implementation of any domain abstraction is not small (as is the case with the persistent Table abstraction mentioned above, which will need to be connected to a real database), it will be using other abstract interfaces (in the Programming Paradigms layer) connected to its runtime support abstractions in a technical domain, the same as in Hexagonal Architecture.

6.14.5. Object Oriented Programming

From my reading, it seems that the most characteristic feature of OOP is that when data and operations are cohesive, they are brought together in an object. Others may see it as enabling reuse, inheritance, and still others may see it as polymorphism. New graduates seem to be introduced to polymorphism in inheritance and not be introduced to interfaces at all, which is a shame because the concept of interfaces is much more important.

I have never been an expert at Object Oriented Design as I found the choice of classes difficult and the resulting designs only mediocre. But I think the most fundamental and important characterising feature of OOP is under-rated. That is the separation of the concepts of classes and objects. This separation is not so clearly marked when we use the terms modules or components. The separation is fundamentally important because it’s what allows us to remove all dependencies except knowledge dependencies. In the way described earlier in this article, you can represent the knowledge of most dependencies as a relationship between instances completely inside another abstraction. What OOP should have done is represent relationships between objects completely inside another class. The problem is that OOP doesn’t take advantage of this opportunity. Instead, it puts these relationships between objects inside those objects' classes, as associations or inheritance, thereby turning them into design-time dependencies, and destroying the abstract qualities of the classes. Abstractions, unlike classes, retain their zero coupling with one another.

ALA addresses the problem by calling classes abstractions and objects instances. Abstractions differ from classes by giving us a way to have logical zero coupling, as if they were on different physical platforms. Instances differ from objects by having ports because their classes give them no fixed relationships with other objects.

Of course, when you are writing ALA code, abstractions are implemented using classes, but you are not allowed associations or inheritance. Instances are implemented as objects but with ports for their connections. A port is a pair of interfaces that allow methods in both directions. The interfaces are defined in a lower layer.

In ALA, the UML class diagram completely loses relevance. Because classes have no relationships with each other, bar knowledge dependencies, a UML diagram in ALA would just be a lot of boxes in free space, like a pallet of things you can use. You could show them in their layers and you could even draw the downward composition relationships that represent the knowledge dependencies, but there would be no point to this except in explaining the concepts of ALA. When you are designing an actual system, the real diagram is the one inside of an abstraction, especially the uppermost one, the application. It shows boxes for instances of the abstractions it uses, with the name of the abstraction in the box, the configuration information for those instances, and of course the lines showing how they are wired together. The names inside the boxes would not even need to be underlined as in UML, because the boxes in such diagrams would always be instances.

Such a diagram is close to a UML object diagram. However, a UML object diagram is meant to be a snapshot of a dynamic system at one point in time. In ALA, any dynamic behaviour is captured in a static way by inventing a new abstraction to describe that dynamic behaviour. Thus the design-time view is always static. So the object diagram is static. The application class specifies a number of objects that must be instantiated, configured, and wired together to execute at run-time. Since the structure is always static, ideally this would be done by the compiler for best efficiency, but there is no such language yet. So, in the meantime, it is done at initialization time. The object diagram can be fairly elegantly turned into code using the fluent coding style shown in the XR5000 example.

6.15. DSLs

We briefly discussed ALA as a DSL in the structure chapter here

ALA includes the main idea of DSLs in that the fundamental method "represent[s] requirements by composition of domain abstractions". It shares the DSL property that you can implement a lot more requirements or user stories in a lot less code.

But ALA only tries to be a light-weight way of telling ordinary developers how to organise code written in your underlying language. Although the domain abstractions do form a language and the paradigm interfaces give it a grammar, ALA doesn’t pursue the idea of a language to the point of textural syntactic elegance. Instead, you end up with explicit wiring methods to combine domain entities, or plain old functional composition, or some other form of composition in the wider sense of the word. Often, the text form is only a result of hand translation of an executable diagram. ALA certainly doesn’t overlap with DSLs to the extent of an external DSL, nor does it try to sandbox you from the underlying language. It therefore does not require any parsing and doesn’t need a language workbench, things that may scare away 'plain old C' developers.

Like DSLs, ALA can be highly declarative depending on the paradigm interfaces being used to connect domain abstractions. It is better to have the properties of composition and composability in the your domain language even if they may not be in a perfectly elegant syntactic form. ALA may end up composing abstractions with calls to wireTo methods instead of spaces or dots. But often a diagram using lines is even better than spaces and dots.

In DSLs, it is important that different languages can be combined for different aspects of a problem. For example, a DSL that defines State machines (the state diagram) and a DSL for data organisation (Entity Relationship Diagram) may be needed in the same application. You don’t want to be stuck in one paradigm. ALA recognises this importance by having paradigm interfaces that are more abstract than the domain abstractions.

DSLs probably work by generating a lot of code from templates whereas ALA works by reusing code as instances of abstractions. Both of these methods are fine from the point of view of keeping application specific knowledge in its place, and domain knowledge in its place. Howver, the distinction between ALAs domain layer and programming paradigms layer is probably not so as clearly made in the implementation of the templates.

It is an advantage of DSLs that they can sandbox when needed. An example from the wiring pattern earlier is that the ports of instances do not need to be wired. Therefore, all abstractions need to check if there is something wired to a port before making a call on it. Enforcing this is a problem I have not yet addressed.

A possible solution, albeit inferior to a real DSL that would tell you at design-time, might be that when there are tools that generate wiring code from diagrams, they automatically put stubs on all unwired ports. These stubs either throw an exception at run-time, or just behave inertly.

ALA is different from external DSLs. ALA is just about helping programmers organise their code in a better way. It doesn’t try to make a syntactically elegant language, as a DSL does. Certainly an external DSL will end up representing requirements in a more elegant syntax. But that is not the most important thing in ALA. The most important thing is the separation of code that has knowledge of the requirements, which will cause the invention of abstractions that have zero coupling (because the coupling was really in each requirement - that is why a requirement is cohesive). ALA also avoids taking the average imperative language programmer out of their comfort zone. It does not require a language workbench and does not sandbox you from the underlying language.

ALA probably does fit into the broadest definition of an internal DSL. However, again, it does not target syntactic convenience in the expression of requirements so much as just separating the code that knows about those requirements from the code that implements them. An internal DSL usually aims to have a mini-language that is a subset of the host language, or it tries to extend the host language through clever meta-programming to look as if it has new features. ALA is about abstraction layering. It is about this design-time view of knowledge dependencies: what abstractions in lower layers are needed to understand a given piece of code.

6.16. Multi-tier Architecture

TBD

6.17. Clean Architecture

Clean architecture is initially viewed as concentric circles which are in effect layers. Entities are innermost, with business logic next, and the external system consisting of things like database, UI and communications on the outer. These layers are allowed to have dependencies going inwards.

In conventional code, dependencies tend to follow communications, and communications, when implemented in the form of direct function or method calls, flow from the initiator of the communications.

This gives rise, for example, to dependencies from the UI to the business logic, and then from the business logic to the database. In clean architecture, these are referred to as primary and secondary I/O with respect to the business logic. The idea in clean architecture is to invert the secondary dependencies so that all communications dependencies are now toward the business logic.

In this way the business logic at the core is reusable, and perhaps more importantly understandable without knowing details of a concrete database, middleware, or UI. It also facilitates easier testing of the business logic.

The business logic uses interfaces to communicate with the outside world. The primary communications have interfaces that the business logic implements (unchanged from conventional code). The secondary communications have interfaces which the business logic requires. The concrete implementations of database, etc are passed in or injected in. This wiring is specific to a unique application, so in ALA terms, it goes in the top layer.

From the point of view of the business logic only, this is compliant with ALA, except for the dependencies on entities, which is discussed below. The elements of the business logic, which in clean architecture are called use cases, can be considered abstractions that know about the business use cases and nothing else.

6.17.1. Adapters

In the clean architecture, dependencies, such as those between business logic and database, are reversed (following the dependency inversion principle) from what it would have been in conventional code. These reversed dependencies do not comply with ALA. I think most implementations recognise these as bad dependencies, and solve it by removing the dependencies altogether using adapters. This is now a lot closer to ALA compliance.

Something must pass-in or inject the adapters into each of the business logic use cases. If this logic is thought of as being in a higher layer, then this is also ALA compliant.

In terms of ALA abstraction layers, the use cases, the database, the UI, and other IO are all about the same level of abstraction. They all know about different types of details. While the use cases know about the domain and it’s requirements, the database knows about how to efficiently store data. They are all abstractions that are zero coupled with one another. The adapters go in a layer above, and are specific to a use case / external IO pairing. The main() (or a function it delegates) goes in a layer above that and wires everything up using (usually) constructor dependency injection on the use cases.

6.17.2. Entities

Clean architecture allows dependencies of use cases on entities. This is incompatible with ALA.

Entities typically hold all sorts of domain details, for example various informations about customers. When the requirements change, these will change. We expect requirements to change - that’s why we have agile.

Entities are an easy place to just add all fields to do with an identity. They will tend to hold some fields that, although they associate with an identify, really belong to separate use cases. These fields should be cohesive with their use cases. If entities hold information that is not significantly abstract with respect to use cases, such as the customer’s address, which is primarily used by one or two use cases only, then it is not ALA compliant. The customer identity abstraction’s responsibility should not be to know all data that can be associated with a customer, but to know about the idea of identity. It should not be used as the carrier of information between two use cases, which would expose all entity data to all use cases. Instead, use cases should all know about the abstraction, customer identity. A particular use case should only know about it’s own data, and only store it against a customer identity.

In other words, a user story should be able to have private data that is associated with an identity and still ultimately stored with all other data for that identity in the database. The only idea that is abstract enough to go in a layer below the use cases is the customer identity, which is likely to be reused by most new use cases. Subclassing, so that every use case has its own subclass may solve the problem in one way, but I expect would cause other problems.

Even if some customer detail needs to be shared with another use case, communicating this via a shared entity is bad. For example, consider a use case in a system that knows about the address that customers enter into the system. It could have an output port called 'address' that can be used to wire it to other use cases. This port will probably have a DTO type that belongs to it. The DTO cannot be shared with other features in the same layer without violating ALA constraints. A feature such as frieghtcost may need an address to calculate freight. Remember it is written separately from the address feature so is not coupled with it. It cannot know about the address feature. It can’t know the DTO of the address. Nor does it need the entire address. So it may be written, for example, to have input ports for country and zip code. Yet another feature is shipping. It needs an address for a shipping label. It may have an input port that takes a string for of address, because it isn’t interested in the content of the address, only in faithfully printing it. So these three ports are incompatible. The wiring layer, which knows that it needs to wire these three together also knows how to adapt them, which can be done quite simply by passing in a lambda expression into the WireTo method (analogous to a Select clause in LINQ).

More generally in ALA, such applications are best viewed primarily in terms of dataflows rather than abstracted entities. Dataflows to/or from the database, for example. It flows to particular use cases, and only the data that is needed by the use case. At any point in the flow, the flow has a type. It is still nice to have a compiler generated, anonymous, fully type checked class at each point in the flow. But nowhere do we want to create an explicit class for sharing a whole entity, or even a part of an entity.

The identity of a customer itself is probably an abstract concept that can be used by all features. We therefore want a shared abstraction for the identity (just knowing about a unique internal or external number or key).

It should be possible to add a feature that needs a new private field (private to the feature). The data can still be associated with an identity and be stored in the database. Adding this field should cause a database migration, but not changes to other use cases.

So the way entities should be handled is quite different in ALA.

TBD do a simple 'task list' application on Github in both ALA and clean architecture to show how entities are handled in ALA. Then add a feature such as e-mail notification on due date to show how a new feature can have it’s own private data stored against the task identity (the e-mail sent status) and communicate via a port with an existing feature (the due date feature).

6.17.3. Primary separation

There is a second major difference between clean architecture and ALA. In clean architecture, the UI and other externals IO such as the database are considered to be separated first. That is how it is shown on an architecture diagram, almost as if they are separate packages. You hear of being able to switch between a GUI or CLI based UI.

This view of primarily separating UI from business logic will likely lead to coupling. It is unlikely that the UI is so generic that it knows nothing about the business logic. It will need to specific to the data the business logic needs or produces. Similarly, the design of the UI will usually influence the way the business logic works. For example, the UI may be designed so that you enter all data first (like a form) and then submit, or it may be designed so that you select generally what you want to do, and then wizards guide the user through. The choice is likely to affect the way the business logic works.

In ALA, the primary separation is by features first. The UI and the business logic for a particular feature is considered to be cohesive with respect to that feature abstraction. The use case will wire up both the elements of the business logic and the elements of the UI (and those for the necessary database queries, etc). The UI elements used can still be swapped out for different ones, but that is an operation on the feature.

In the case that the UI design is not changing, but its implementation is, that involves swapping out the implementations of the UI domain abstractions. The abstraction themselves do not change, so the use cases wont change. But the new UI abstractions can shift to a different technology, shift from desktop to cloud, or the like.

6.17.4. DTOs

DTOs have two different uses.

  • part of an interface to group together related data that is sent through the interface at one time.

  • to collect data together to be transported together to cut down on the overhead of messaging.

interface DTOs

In ALA, DTOs are not generally abstractions in themselves. Therefore, they may not be put in a lower layer and shared by two abstractions to communicate. That would couple the knowledge inside the two abstractions. If many abstractions want to know about the same DTO, this is likely to be the case as new abstractions are added, then maybe it is sufficiently abstract to be in a lower layer and shared.

Otherwise in ALA, you need to use adapters. This can be as simple as a lambda expression passed to the WireTo operator, in the same way that you would pass a lambda expression to a .Select clause in LINQ.

Although this is ALA compliant, in ALA we generally prefer not to use adapters. Instead we use interfaces that are a significantly more abstract that are not owned by the business logic core. These are of course at the abstraction level of programming paradigms. These types of interfaces are heavily reused, allow composability in the wiring, and help tremendously to keep all abstractions from being implicitly coupled.

If a DTO can be avoided by, for example, having two dataflow ports that use primitive types, this will increase the abstraction level, reusability and composability of your abstractions.

transport DTOs

In ALA you wouldn’t use DTO for transport purposes. Instead, invent an abstraction say called multiplexer_demultiplexer for packing/unpacking (or serializing/deserializing) multiple input or output ports. Then instances of any two abstractions A and B, that would normally be compatible for wiring together, and which use asynchronous communications, can be physically deployed to opposite sides of the transport system. The wireTo operator, knowing they are in different physical locations, defers to a version that wire each of them to the respective multiplexer_demultiplexer instances.

6.17.5. Stability of wiring/adapter/feature layers

A system built from a wiring layer at the top, then an adapters layer below that, and then a layer below that for independent features, use cases, databases, UIs etc is ALA compliant. This is because the abstraction layers are more abstract as you go down. The top layer abstraction is a specific application. The second layer adapters are specific to pairs of things in the third. The third is the layer of fully reusable things. A database, even though we call it concrete, is a lot more reusable than a particular application, or a particular adapter.

An ALA application using these three types of layers is a little different from the layers we normally talk about, which uses domain abstractions that are wired directly together using compatible ports instead of via adapters in the layer above. To enable the ports to be compatible, there must be a layer below that provides abstract interfaces, which is what we call the programming paradigms. This latter arrangement has compositionality. For example, two domain abstractions currently wired together can have another domain abstraction, which is a decorator such as a filter, wired between them.

The two styles of layering can be used together.

6.17.6. Swapping out technology

In clean architecture, part of the reason for avoiding dependencies from business logic to things like a particular database or framework is to allow swapping out the technology. The database in the third layer can be exchanged for a completely different type - the coninical example is changing it from a relational database to a simple file. The business logic does not change. Only new adapters are needed, one for each use case. The top layer wiring of course also needs to change to use the different adapters.

An ALA application that uses the preferred layering scheme of application layer, domain abstractions layer, programming paradigms layer can also have its technologies swapped out. Let’s again use the canonical example of swapping a relational database for a simple file. The domain abstraction that implements persistence using a database will have a port that implements a suitable programming paradigm. Usually this port has a type like ITableDataflow. You only need to substitute this domain abstraction with one that uses the same programming paradigm, but implements it as a simple file. Effectively these domain abstractions are wrappers, not adapters.

The wiring again needs to change in all the places that were instantiating the database implementation. This is probably the only practical way to do it, as the database implementation probably needs different application specific configuration than what a simple file implementation would.

Now let’s consider swapping out the UI. Let’s say we are changing the UI from a desktop windowed application to a browser, or from a PC window to a CLI (Command Line Interface).

In the original PC application, the wiring instantiates UI GUI domain abstractions. These domain abstractions are wrappers for, say, WPF UI elements. The wrappers have ports which the wiring uses to connect them to the corresponding parts of the business logic. These ports are, or course, abstract interfaces from the programming paradigms layer.

To swap out the UI involves changing the wiring to instantiate from a different set of these UI domain abstractions. They will have the same ports that are still wired to their relevant place in the business logic as before.

In the case of the browser, these new domain abstraction work by changing elements of the HTML that will be returned by an initiating HTTP request. Just as the windowed domain abstractions were wired to their containing window, browser domain abstractions will be wired to their containing page. The containing page will request their content when it is time to send the response to the HTTP request.

The case of the CLI is more interesting. Whenever there is a case of either a GUI or a CLI user interface in conventional architecture, the business logic is tied to the CLI commands, and the GUI then uses the CLI. But in ALA we have the option to do this without coupling the design of the business logic to the design of the CLI commands.

This is how it could work. Imagine we have previously built the application as a desktop windowed application, just as we did before. Now we change the wiring to use a set of CLI domain abstractions instead. Actually we need only two abstractions, one called command and one called response. Instances of the command abstraction are configured with the command that they handle. The command has an output event port which fires when the command is entered. If there are parameters, the abstraction can have other output ports for them, which are output before the event port fires. Alternatively you could chain up a series of parameter abstractions, each with a single output port. The response abstraction has an input port, and just prints any input data it receive. Optionally it could have a configuration name so it can identity itself when it prints.

Just as there are containing domain abstractions that describe layout for the GUI types of UI domain abstractions, CLI domain abstractions would also connect to a common domain abstraction that receives commands in a general form and passes them to the handler that is configured for that command. It would also collate the responses, add newlines to the output, etc.

There is one other possibility. In the above cases of swapping out the UI, we changed the names of UI domain abstraction instantiated by the wiring. That was potentially all we needed to change.

It is possible that the configuration of the domain abstractions did not need to change. For example, CLI command abstractions need to be configured with the actual command string they will respond to, whereas their GUI equivalents, which are buttons, need to be configured with a button name. These could potentially be the same. If other configuration information of UI domain abstractions, such as style, is implemented in a generic way such as having a style port wired using WireMany, then it is possible that the wiring only needs to specify the UI domain abstraction names.

In this case we could name all equivalent UI domain abstraction with the same name. Then by which set of classes we include in the project, it will be built for different technologies. I’m not really proposing it be done this way, just exploring the idea.

6.18. Onion Architecture

TBD

6.19. Hexagonal Architecture (Ports and Adapters)

ALA includes the basic idea of hexagonal architecture, but with modification using the Bridge Pattern to keep cohesive knowledge belonging to the application from being split.

In a previous section we intimated that the sideways chains of interfaces going out in horizontal directions were the same as hexagonal architecture. While ALA shares this aspect of hexagonal architecture, there is still an important difference.

ALA retains domain abstractions of the UI, Database, communication and so on. For instance, in our XR5000 example, we had a domain abstraction for a persistent Table. We had domain abstractions for UI elements such as Page, Softkey etc. We don’t just have a port to the persistence adapter, we have an abstraction of persistence. We don’t just have a port for the UI to bind to, we have abstractions of the UI elements. The implementation of these abstractions will then use ports to connect to these external system components. Why is it important that we have domain abstractions of these external components?

  1. The Database and the UI will have a lot of application specific knowledge given them as configuration. Remember the creativity cycle. After instantiation of an abstraction comes configuration. The database will need a schema, and the knowledge for that schema is in the application. The Softkey UI elements will need labels, and that knowledge is in the application. By making domain abstractions for persistence and UI, the application can configure them like any other domain abstraction as it instantiates and wires up the application. To the application, these particular domain abstractions look like wrappers of the actual database and UI implementations, but they are more like proxies in that they just pass on the work.

    The Persistence abstraction then passes this configuration information, via the port interface to the actual database. The Softkey abstraction then passes its label, via the port interface, to the softkeys. Otherwise the Application would have to know about actual databases and actual softkeys.

    If you need a design where the UI can change, you just make the UI domain abstractions more abstract. A softkey may be a command abstraction. It is still configured with a label. But it may be connected to a softkey, a menu item, a CLI command, a web page button, or a Web API command.

  2. From the point of view of a DSL, it makes sense to have concepts of UI and persistence and communications in the DSL language. The application is cohesive knowledge of requirements. The UI and the need for persistence are part of the requirements. In fact, for product owners communicating requirements, the UI tends to be their view of requirements. They talk about them in terms of the UI. Many of the product owners I have worked with actually design the UI as part of the requirements (with the backing of their managers, who are easily convinced that software engineers can’t design UIs. PO can’t either, but that is another story.). The point here is that the UI layout, navigation, and connection to business logic is all highly cohesive. We explicitly do not want to separate that knowledge.

    As a restatement of an earlier tenet of ALA, it is much better to compose the application with abstractions of Business logic, UI and persistence than to decompose the application into UI, persistence and business logic.

  3. We want the application to have the property of composability. We have previously discussed how that means using programming paradigm interfaces for wiring up domain abstractions. By using domain abstractions to represent external components, the abstractions can implement the paradigm interfaces and then be composable with other domain abstractions. For example, the Table domain abstraction which represents persistence may need to be connected directly to a grid, or to other domain abstractions that map or reduce it. Indeed, the Table abstraction itself can be instantiated multiple times for different tables and be composed to form a schema using a schema programming paradigm interface. I have even had a table instance’s configuration interface wired to a another Table instance. (So its columns can be configured by the user of the application.)

  4. The fourth reason why it is important for the application to not directly have ports for external components of the system is that we don’t want the logical view of the architecture to become just one part of the physical view. If there is a communications port that goes to a different physical machine where there is more application logic, the application’s logical view should not know about that. It may be presented as an annotation on the application (lines) connecting certain instances, but it shouldn’t split the application up. At the application level, the collaboration between parts instantiated on different machines is still cohesive knowledge and belongs inside one place - the application.

6.20. Domain Driven Design

Domain Driven Design’s "Bounded Contexts" and ALA’s Domain Abstractions layer have the same goal, that of encapsulation of the domain specific knowledge.

Domain driven design appears to concentrate on common languages to allow pairs of elements to communicate, which ALA explicitly avoids. ALA tries to abstract the languages so that they are more abstract and fundamental than the domain, and more like programming paradigms.

6.21. Microservices

TBD

6.22. Architecture evaluation methods

Methods such as ATAM tell us how to evaluate an architecture for quality attributes such as maintainability, for instance by giving it modification scenarios to test how difficult the modifications would be to implement. There are several scenarios based methods to do this such as ATAM. Using this we could, theoretically, iteratively search over the entire architecture design space to find a satisfactory solution. It’s a bit analogous to numerically solving for the maxima of a complex algebraic formula. In contrast, ALA is analogous to an 'algebraic solution'. If the desired quality attributes, and all the software engineering topics listed above are the equations, ALA is the algebraic solution. It simplifies them down into a parameterised template architecture, ready for you to go ahead and express your requirements.

6.23. Reactive Extensions

TBD

6.24. WPF & XAML

TBD

6.25. Functional programming

TBD

6.26. Functional programming with monads

TBD

6.27. Functional Reactive Programming

TBD

6.28. Example project - Game scoreboard

For the example project for this chapter, we return to the ten-pin bowling and tennis scoring engines that we used in Chapter two, and add a scoreboard feature (well a simple ASCII scoreboard in a console application rather than real hardware).

As the requirement, say we want a console application that displays ASCII scoreboards that look like these examples:

Ten-pin

 -----+-----+-----+-----+-----+-----+-----+-----+-----+--------
|   1 |   2 |   3 |   4 |   5 |   6 |   7 |   8 |   9 |    10  |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| 1| 4| 4| 5| 6| /| 5| /|  | X| -| 1| 7| /| 6| /|  | X| 2| /| 6|
+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+--+
|   5 |  14 |  29 |  49 |  60 |  61 |  77 |  97 | 117 |   133  |
 -----+-----+-----+-----+-----+-----+-----+-----+-----+--------
Tennis

 -----++----+----+----+----+----++--------
|   1 ||  4 |  6 |  5 |    |    ||    30  |
|   2 ||  6 |  4 |  7 |    |    ||  love  |
 -----++----+----+----+----+----++--------

As usual in ALA, our methodology begins with expressing those requirements directly, and inventing abstractions to do so. So, we invent a 'Scorecard' abstraction. It will take a configuration which is an ASCII template. Here are the ascii templates that would be used for ten-pin and tennis:

 -------+-------+-------+-------+-------+-------+-------+-------+-------+-----------
|   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |     10    |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
|F00|F01|F10|F11|F20|F21|F30|F31|F40|F41|F50|F51|F60|F61|F70|F71|F80|F81|F90|F91|F92|
+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+---+
|  T0-  |  T1-  |  T2-  |  T3-  |  T4-  |  T5-  |  T6-  |  T7-  |  T8-  |    T9-    |
 -------+-------+-------+-------+-------+-------+-------+-------+-------+-------------
 -----++----+----+----+----+----++--------
| M0  ||S00 |S10 |S20 |S30 |S40 || G0---  |
| M1  ||S01 |S11 |S21 |S31 |S41 || G1---  |
 -----++----+----+----+----+----++--------

The scorecard ASCII template has letter place-holders for the scores. (A single letter is used so it doesn’t take up much space on the template design.) Different letters are used for different types of scores. Digits are used to specify where multiple scores of the same type are arranged on the scoreboard. They are like indexes. Either 1-dimensional or 2-dimensional indexes can be used in the scoreboard template. For example, the frame scores in ten-pin bowling have scores for each ball for each frame, F00, F01 etc, as shown in the example above.

The scorecard abstraction needs functions it can use to get the actual scores. The functions are configured into little 'binding' objects that we then wire to the scoreboard. The binding objects are configured with the letter that they return the score for.

6.28.1. Ten-pin

Having invented the Scorecard and Binding abstractions, we can now do the ten-pin application diagram:

diagram bowling 3

An abstraction we didn’t mention yet is the ConsoleGameRunner. Its job is to prompt for a score from each play, display the ASCII scoreboard, and repeat until the game completes.

The 'game' instance of the Frame abstraction on the right of the diagrams is the scoring engine we developed in Chapter Two. Together with this engine, we now have a complete application.

The rounded boxes in the diagram are instances of domain abstractions as usual for ALA diagrams. The sharp corner boxes are instances of Application layer abstractions. They are the mentioned functions for the Bindings. That code is application specific so goes in the application layer. They just do a simple query on the scoring engine.

Now tranlate the diagram into code. Here is the entire application layer code for ten-pin:

consolerunner = new ConsoleGameRunner("Enter number pins:", (pins, engine) => engine.Ball(0, pins))
.WireTo(game)
.WireTo(new Scorecard(
"-------------------------------------------------------------------------------------\n" +
"|F00|F01|F10|F11|F20|F21|F30|F31|F40|F41|F50|F51|F60|F61|F70|F71|F80|F81|F90|F91|F92|\n" +
"|    ---+    ---+    ---+    ---+    ---+    ---+    ---+    ---+    ---+    ---+----\n" +
"|  T0-  |  T1-  |  T2-  |  T3-  |  T4-  |  T5-  |  T6-  |  T7-  |  T8-  |    T9-    |\n" +
"-------------------------------------------------------------------------------------\n")
.WireTo(new ScoreBinding<List<List<string>>>("F",
    () => TranslateFrameScores(
        game.GetSubFrames().Select(f => f.GetSubFrames().Select(b => b.GetScore()[0]).ToList()).ToList())))
.WireTo(new ScoreBinding<List<int>>("T",
    () => game.GetSubFrames().Select(sf => sf.GetScore()[0]).Accumulate().ToList()))
);

If you compare this code with the diagram, you will see a pretty direct correspondence. Remember 'game' is the reference to the scoring engine project in the previous chapter.

That’s pretty much all the code in the application. Oh there is the 'translate' function, but it is pretty straight forward once you know the way a ten-pin scorecard works. For completeness here it is.

/// <summary>
/// Translate a ten-pin frame score such as 0,10 to X, / and - e.g. "-","X".
/// </summary>
/// <example>
/// 7,2 -> "7","2"
/// 7,0 -> "7","-"
/// -,3 -> "-","7"
/// 7,3 -> "7","/"
/// 10,0 -> "",X
/// 0,10 -> "-","/"
/// additional ninth frame translations:
/// 10,0 -> "X","-"
/// 7,3,2 -> "7","/","2"
/// 10,7,3 -> "X","7","/"
/// 0,10,10 -> "-","/","X"
/// 10,10,10 -> "X","X","X"
/// </example>
/// <param name="frames">
/// The parameter, frames, is a list of frames, each with a list of integers between 0 and 10 for the numbers of pins.
/// </param>
/// <returns>
/// return value will be exactly the same structure as the parameter but with strings instead of ints
/// </returns>
/// <remarks>
/// This function is an abstraction  (does not refer to local variables or have side effects)
/// </remarks>
private List<List<string>> TranslateFrameScores(List<List<int>> frames)
{
    // This function looks a bit daunting but actually it just methodically makes the above example tranlations of the frame pin scores
    List<List<string>> rv = new List<List<string>>();
    int frameNumber = 0;
    foreach (List<int> frame in frames)
    {
        var frameScoring = new List<string>();
        if (frame.Count > 0)
        {
            // The first 9 frames position the X in the second box on a real scorecard - handle this case separately
            if (frameNumber<9 && frame[0] == 10)
            {
                frameScoring.Add("");
                frameScoring.Add("X");
            }
            else
            {
                int ballNumber = 0;
                foreach (int pins in frame)
                {
                    if (pins == 0)
                    {
                        frameScoring.Add("-");
                    }
                    else
                    if (ballNumber>0 && frame[ballNumber]+frame[ballNumber-1] == 10)
                    {
                        frameScoring.Add(@"/");
                    }
                    else
                    if (pins == 10)
                    {
                        frameScoring.Add("X");
                    }
                    else
                    {
                        frameScoring.Add(pins.ToString());
                    }
                    ballNumber++;
                }

            }
        }
        rv.Add(frameScoring);
        frameNumber++;
    }
    return rv;
}

6.28.2. Tennis

So now that we have these domain abstractions for doing console game scoring applications, let’s do tennis:

diagram tennis 3

I left the code out of the GetGameOrTieBreakScore box as it is a little big for the diagram here. It is similar to the other queries but it must first determine if a tie break is in progress and get that if so. Also it translates game scores from like 1,0 to "15","love".

And here is the code for the Tennis diagram:

consolerunner = new ConsoleGameRunner("Enter winner 0 or 1", (winner, engine) => engine.Ball(winner, 1))
.WireTo(match)
.WireTo(new Scorecard(
        "--------------------------------------------\n" +
        "| M0  |S00|S10|S20|S30|S40|S50|S60|  G0--- |\n" +
        "| M1  |S01|S11|S21|S31|S41|S51|S61|  G1--- |\n" +
        "--------------------------------------------\n")
    .WireTo(new ScoreBinding<int[]>("M", () => match.GetScore()))
    .WireTo(new ScoreBinding<List<int[]>>("S", () =>
        match.GetSubFrames()
            .Select(sf => sf.GetSubFrames().First())
            .Select(s => s.GetScore())
            .ToList())
    .WireTo(new ScoreBinding<string[]>("G", () => GetGameOrTiebreakScore(match)))
);

If you compare this code with the diagram, you can see a pretty direct correspondence. match comes from the scoring engine project in Chapter two.

6.28.3. Concluding notes

Although the diagrams must be turned into text code to actually execute, it is important in ALA to do these architecture design diagrams first. They not only give you the application, they give you the architectural design by giving you the domain abstractions and programming paradigms as well. If you try to design an ALA structure in your head while you write it directly in code, you will get terribly confused and make a mess. Using UML class diagrams will make it even worse. Code at different abstraction levels will end up everywhere, and run-time dependencies will abound. Our programming languages, and the UML Class diagram, are just not designed to support abstraction layered thinking - it is too easy to add bad dependencies (function calls or 'new' keywords) into code in the wrong places.

Note that at run-time, not all dataflows have to go directly between wired up instances of domain abstractions. The data can come up into the application layer code, and then back down. This was the case when we did the functional composition example in Chapter One. In this application we are doing that with the code in the square boxes that get the score from the engine. The important thing is that all the code in the application is specific to the application requirements.

That completes our discussion of the console applications for ten-pin and tennis. The full project code can be viewed or downloaded here:

7. Chapter seven - The philosophy behind ALA

In this chapter we go into some of the deeper whys behind ALA. We did this somewhat in chapter three, but here we get more philosophical.

This chapter needs a review pass - some of thinking is out of date.

Once again each section of this chapter is a different perspective.

7.1. The human brain

In this perspective of ALA, we look at the problem of complexity in software in the context of how the human brain works.

Software design involves our intelligence or brain power. Understandability, readability, complexity are all things very closely related to the brain. Yet in the field of software engineering we pay little attention to how the brain understands our complicated world in order to understand how we should do our software.

Our brains do it primarily through one mechanism which we have come to call 'abstraction'. We learn abstractions from the commonality of multiple examples, and we then use abstractions without those examples cluttering up the common notion that was learned.

Paintings_from_the_Chauvet_cave.jpg
Figure 53. Paintings in the Chauvet cave

Our ancestors could use a word like 'bring your spear' and it had a simple meaning to them only because all the detail and knowledge that went into building a spear was replaced with the abstraction 'spear'. Without the abstraction, the sentence would have had to be more like "bring the object that we made by joining the object we made by applying blows to the hard material we found at the place…​, with the long material we cut in the place with the tall…​, by tying it with the long grass material using the gooey stuff we found at the…​". Even this sentence was only made possible by other abstractions: joining, material, blows, hard, long (twice), cut, tall, tying, gooey, and found. If we expanded all of them until we were only using a few basic concepts like 'object' and 'place', we would have a sentence so long that we could never communicate at all. That’s what abstractions do, and how our brains make use of them. The word spear, in turn can be used to create a new abstraction, a hunting plan, while all of that other detail remains hidden from that new context.

The problem with software engineering is we are not making use of this way that the brain works. Simply put, we are not creating good abstractions. This lets the complexity inside one 'module' spill out into other modules. Abstractions, not modules, are the only mechanism that allows us to hide details at design-time.

neuron.svg
Figure 54.

As software engineers we do learn and use many abstractions. For example if we want to protect simultaneous access to the resource, our brain should conjure up 'mutex'.

If the brain already 'knows about' an abstraction, the abstraction is like any other single line of code, such as a mutex. We can make use of a mutex without having to deal with the details and complexities of how it works. We don’t have to think about the fact that we may have to wait for the resource. Nor that another thread may start running if we have to wait. Nor that if a higher priority thread preempts us while we have the resource, we may have it for a long time. Nor that if a still higher priority thread needs it during that long time, we will be given temporary priority to finish our use of it. We can just simply use the abstraction for protecting a resource.

Abstractions like mutex, regex, SQL are already invented by the 'culture' of software engineering, much like memes in the real world have been passed down to us. Where we fall down is when we get into a particular domain where the abstractions have not yet been invented, and we need to invent them. It is not easy to invent new abstractions, but invent them we must, at a rate far higher than is normal for cultural evolution.

Good domain abstractions, introduced and learned by new developers in the domain, then appear to them as normal program elements - things they can use with extraordinary convenience like any other line of code.

7.2. Abstraction

Of the overwhelming list of engineering topics that we listed in Chapter One, this topic that is the most fundamental to ALA, and the one most needed for explaining it. It’s also probably the vaguest and most misunderstood topic in software engineering, so we will spend some time understanding it.

Abstraction will be the king. The short reason why we start with abstraction is that our quality attributes, complexity and understandability are very much to do with how our brains work, and for 100,000 years at least, our brains have worked with abstractions to understand our world. Abstractions are the only mechanism our brains use for dealing with otherwise complex things.

As in a chess game, winning is only about protecting the king. But this Abstraction king is benevolent. If he is destroyed, you do not lose the game immediately. It will take time, but you will lose.

There are other contenders to be king in the engineering topics list. For example, it is said that the best thing about TDD is not the testing but the emergence of better abstractions. TDD is like a lord that serves the king. It usually serves the king, causing you to make better abstractions. But sometimes it just serves its own purpose and makes the abstraction worse. It just produces code that works where it passes and no more.

Another contender is microservices. It is popular because it improves your abstractions by making them harder to destroy with cross coupling. But it too is just a lord. Because it provides physical boundaries that in normal software would be crossed, it serves the king. But by serving the abstraction king directly we can have logical boundaries, and all their benefits, even in 'monolithic' code.

Another contender to be king is 'no side effects' used by the functional mathematical purity guys. There are those who talk as if disobeying this king is absolute treason. But again, this lord is only effective because he usually serves the abstraction king. But, again, there are times when he doesn’t, and 'no side effects' is not enough to make a good abstraction.

ALA always follows the one true king.

7.2.1. Classes, Modules, functions and encapsulation.

Classes, Modules, functions and encapsulation are artefacts of the language and do their thing at compile-time. They are not necessarily abstractions. They have been around for about 60 years, not enough time for our brains to see them in the same way as the compiler does. Although abstractions are implemented using these artefacts, ALA needs them to also be abstractions. In ALA "abstraction" is the term we use for the artefacts of our design instead of classes, modules, functions, or components, all of which are extremely fragile as abstractions.

In ALA, classes can still be used inside an abstraction, but they are just a language construct to allow operations given a reference to an object, not to so much to associated data and methods, or to encapsulate data representation. Inside an abstraction everything is considered cohesive, not structured in any way. Classes don’t make the abstractions in themselves, although in ALA many abstractions are implemented using a single class with ports.

7.2.2. Wikipedia on abstraction

"Thinking in abstractions is considered by anthropologists, archaeologists, and sociologists to be one of the key traits in modern human behaviour, which is believed to have developed between 50 000 and 100 000 years ago. Its development is likely to have been closely connected with the development of human language, which (whether spoken or written) appears to both involve and facilitate abstract thinking."

In the real world, new abstractions come along infrequently, and are conceived of by few. People quickly begin using them to understand new insights or compose new things. They become so natural to us that we forget that they are abstractions. In no other field do we need to create them as fast as in software engineering. It is the most important skill a developer needs to have.

7.2.3. Defining abstraction

The term abstraction is arguably one of software engineering’s vaguest or most overloaded terms. Because it is the most fundamental concept in ALA, we try to provide a definition. I find the easiest way to define it is to provide a set of 'statements about', 'properties of', or 'what it is nots':

  • Etymology: 'to draw out commonality'

  • The concept or notion drawn out of what is common in multiple instances

  • Because it is a 'commonality', it is inherently reusable. Kruger says that abstraction and reuse are two sides of the same coin.

  • Has inherent stability - as stable as the concept itself

  • The only mechanism that separates and hides design-time knowledge

  • Its concept or notion is easier to remember than its implementation. For a good abstraction, it is much, much simpler.

  • Abstractness increases with scope of reuse

  • Knows nothing about peer abstractions

  • use ports (instances or interfaces) for IO instead of directly calling other abstractions.

  • Abstractness decreases with more ports

  • Abstractness decreases as you get closer to your specific application

  • Abstractness is not how far you are above physical hardware

  • An ability our brains evolved understand the world

  • The only way we have of dealing with complexity

7.2.4. Leaky abstractions

The code inside abstraction A could potentially tell how long B takes to execute. If it is sensitive to that, this is not a problem with ALA, but with the leaky abstraction. The design needs to be changed to eliminate the dependency on that leakage, or, if that is not possible, the coupling managed. But for the vast majority of code, the coupling between the insides of any two abstractions really is zero.

7.2.5. The three stages of creativity

creativity.jpg
Figure 55. The creativity cycle

A good abstraction separates the knowledge of different worlds. A clock is a good abstraction. On one side is the world of cog wheels. On the other side is someone trying to be on time in their busy daily schedule. Neither knows anything about the details of the other. SQL is another good abstraction. On one side is the world of fast indexing algorithms. On the other is finding all the orders for a particular customer. Let us consider a domain abstraction - the calculation of loan repayments. On one side is the world of mathematics with the derivation and implementation of a formula. On the other, the code is about a person wanting to know if they can afford to buy a house. If your abstractions don’t separate knowledge of different worlds like this, then you are probably just factoring common code. Find the abstraction in that common code. Make it hide something complicated that’s really easy to use and really useful, like a clock.

The creativity cycle starts with abstractions, such as cogs and hands, instantiates them, configures them for a particular use, then composes them into a new abstraction. In ALA we usually go around the creativity cycle three times, creating three layers on top of our base programming language.

7.2.6. Abstractions need ports

In traditional programs, inputs (or at least incoming function calls) are typically part of the module or class’s interface but outputs (or at least outgoing function calls) are typically just buried in the code.

This is fine if calling functions or methods in a lower abstraction layer. However, it is absolutely not fine if calling functions or methods of a peer in the same abstraction layer.

In ALA all inputs and outputs to or from peers in the same layer must be 'ports'. There should be one port for each peer that can be wired. This is the Interface Segregation Principle. A port is a logical wirable connection point. A port either implements or accepts an interface. Outgoing function calls buried in the code that at run-time will go to a peer must only go to the port, which has an indirection mechanism of some kind.

Programming languages encourage all outgoing function or method calls to refer directly to the destination, or the destination’s interface, so you have to make an effort to avoid doing this.

A port is not an artefact of programming languages (yet) so they must be implemented logically somehow as normal code. To code a logical port, you need to do two things.

  1. The interface type of the port must not be owned by another peer abstraction. The interface type must be from a lower abstraction layer.

  2. The name of the port is the name of the field that accepts the interface.

A port can have multiple interfaces. In this case I make the names of the multiple fields contain the port name.

If you are using an asynchronous event driven design, the equivalent of a conventional outgoing function call is typically written something like this:

Send(Event, Receiver, Priority);

where the Event is something the receiver defines.

Again, we are sending the event directly to a peer abstraction using the peer abstraction’s interface (its event).

In ALA, sending an event should be self-oriented, so written something like this:

Send(Sender, SenderPort)

The sender just sends the event out, not knowing where it goes, and the port identifies the event (or you could have both port and event). This just tells the event framework who the sender and sender’s port was. The event framework gets information from the application in the top layer to know what to do with the event. The application has the specific knowledge to know what an event from a given sender on a given port means, and therefore where it should go, and what the priority should be.

In general, classes, modules, components, functions should all have ports for both input and output. They should not own the interface types for these ports, whether they are incoming or outgoing.

An output port from an abstraction may say 'This has happened' or 'Here is my result', not 'do this next', or 'here is your input'.

There are multiple ways to implement the indirection inherent in ports for outgoing calls. They can be callbacks, signals & slots, dependency injection, or calls to a framework send function.

Note that inputs and outputs are not necessarily on different ports. We may want to wire both inputs and outputs between two instances or two abstractions with a single wiring operation. The general case is that a single wiring operation wires multiple interfaces that are logically one port. One contains methods going in one direction and the other contains methods going in the other.

7.3. Complexity

7.3.1. Philosophy of complexity

7.3.2. Dijkstra on complexity

"It has been suggested that there is some kind of law of nature telling us that the amount of intellectual effort needed grows with the square of program length. But, thank goodness, no one has been able to prove this law. And this is because it need not be true. We all know that the only mental tool by means of which a very finite piece of reasoning can cover a myriad cases is called “abstraction”; as a result the effective exploitation of his powers of abstraction must be regarded as one of the most vital activities of a competent programmer. In this connection it might be worth-while to point out that the purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise. Of course I have tried to find a fundamental cause that would prevent our abstraction mechanisms from being sufficiently effective. But no matter how hard I tried, I did not find such a cause. As a result I tend to the assumption —up till now not disproved by experience— that by suitable application of our powers of abstraction, the intellectual effort needed to conceive or to understand a program need not grow more than proportional to program length."

The "conceive" part I agree with, if by that we mean the development. However, the "intellectual effort to understand" part needs further insight. We shouldn’t have to read an entire program to understand a part of it. We ought to be able to understand any one part of it in isolation. The effort to read any one part should be approximately constant. In Chapter One of this article there was a quality graph of complexity here.

complexity curve

These graphs are qualitative in nature, based on experience. But now that we have a better understanding of ALA structure, we can explain how it manages to keep complexity from increasing.

In ALA, the design-time view of the system is nothing more than a static view of instances of abstractions composed together. In a typical application, there will be of the order of fifty different domain abstractions - not a difficult number to familiarize yourself with in a new domain.

Abstractions have no relationship with one another. Each is a standalone entity like a standalone program. If every abstraction contains say 500 lines of code, and the system itself contains 500 lines (instances of abstractions wired together) then the most complex the software gets is that of 500 lines of code.

Even if one abstraction is overly complex internally, say it conceals a piece of legacy code using a facade pattern, that doesn’t affect the complexity of any other part of the system.

ALA is based on the realization that abstraction is fundamentally the only mechanism available to us to achieve this constant complexity.

When doing this for the first time in a domain, it’s not easy to invent the abstractions. but the alternative is always runaway complexity.

The goal of software architecture should be to keep complexity constant.

7.4. Zero coupling, not loose Coupling

Here we meet the first meme from our list of software engineering topics that we must throw out. To many, this will seem a surprising one. Yes, I am saying 'loose coupling' is undesirable.

7.4.1. A common argument

An argument is sometimes stated along these lines: "There must be at least some coupling, otherwise the system wouldn’t do anything." Hence we have the common meme about "loose coupling and high cohesion". In this section we show how this argument is false and resolve the apparent dilemma. We will eliminate all forms of design-time coupling except one. That one remaining one is anything but loose and very desirable.

7.4.2. Classifying coupling

Think of some instances of dependencies you know of in a system and try to classify them into these three types by asking when the system would fail without it.

For example, let’s say that dataflows from an ADC (analog to digital converter) to a display as part of a digital thermometer. At run-time, both must exist. At compile-time both must have the same method signature:

diagram 01

Or the display may tell the ADC when to do the conversion. At run-time there is temporal coupling.

diagram 02

In this one there is an association from a Customer class to an Account class to facilitate communication between them. At run-time there is coupling. At compile-time there is coupling too - the type of the Account class must be exactly the same as expected by the Customer class:

diagram 04

In all the above diagrams, relationships shown in red indicate they are disallowed by the ALA constraints. Green is for desirable relationships, of which there is only one. When we disallow all these types of coupling, the modules, components, functions and classes can now be abstractions.

7.4.3. Run-time, Compile-time and Design-time

A few times already in the article, I have sneaked in a magic qualifier, 'design-time'. You know how we sometimes talk about run-time and compile-time with reference to binding. In ALA we recognise that understandability, complexity, etc, are all happening at design-time. By design-time I mean any time you are reading code, writing code, or changing code.

At run-time, the CPU processes data. At compile-time, the compiler processes code. At design-time the brain is processing abstractions.

In conventional code, it is common for all forms of coupling, run-time, compile-time, and design-time, to appear as coupling between modules or classes.

You can work out what type of dependency you have by when it first breaks. A run-time dependency doesn’t break until the program runs. The program can still be compiled and it can still be understood.

A compile-time dependency first breaks at compile-time. At design-time the code can still be understandable.

A design-time dependency prevents code from even being understood. The code loses its meaning.

7.4.4. Layers

In everyday design, knowledge dependencies are not normally shown as lines. You simply use the abstraction by its name. But in this article, just so we can explain the meta-architecture, we will sometimes draw knowledge dependencies like this (always downward).

diagram 05

This represents that the implementation of abstraction C knows about abstraction A. A is more abstract than C. C and A cannot therefore be peers, as was the case with the components above. Peer abstractions cannot have any coupling with one another.

7.4.5. Whole-Part pattern

If you are familiar with the Whole-Part pattern, ALA uses it extensively. But there is a constraint. The Whole-Part pattern is only used with knowledge dependencies (since that is the only relationship you are allowed). It may of course be used in other forms inside an abstraction, provided it is completely contained in a single abstraction.

A real world example of the Whole-Part Pattern with knowledge dependencies is Molecules and Atoms. A water molecule, for example, is the whole.

diagram 06

Oxygen and hydrogen are the parts. Note that oxygen and hydrogen are abstractions, and they are more abstract than water because they are more ubiquitous, more reusable and more stable (as a concept) than any specific molecule. We could make a different molecule but still use exactly the same oxygen and hydrogen as parts to compose the new molecule.

When we use the word 'ubiquitous', it refers to the number of times the abstraction is used in a Whole-Part pattern to make other abstractions. It doesn’t refer to the number of abstractions that are instantiated. So just because there is a lot of water, that doesn’t make the abstraction ubiquitous. In comparing the abstraction levels of Oxygen and Hydrogen with water, Oxygen and Hydrogen are more ubiquitous because they are used to make more abstractions than water is.

The molecules and atoms analogy with ALA is very close, and we will return to it when we come to explain in more detail how run-time and compiler-time dependencies are moved inside a single abstraction.

For now we just need to remember that we are using the whole-part pattern with knowledge dependencies only. At design-time, the whole is explained and reasoned about in terms of the parts, just as the water molecule is in terms of the oxygen and hydrogen.

7.4.6. Run-time/design-time congruence

A software program can be temporally confusing. Everything that happens at design-time is in preparation for what will happen at run-time. Our low-level imperative languages tend to keep the two congruent. The statements in the program at design-time follow in the same order as they will execute at run-time. The only difference between the two is a time shift and the speeding up of the clock.

When we want the knowledge of run-time dependencies to be moved inside another abstraction, this congruence between design-time and run-time must be broken. Unfortunately, developers start out by learning a low-level imperative language, so it becomes unnatural to them to architect their programs without this congruence. Indeed, breaking this congruence needs a pattern to be learned, and then carefully protected from the temptations of our imperative languages. I call it the Ẃiring pattern'.

Before going into the pattern, we need to round out the most important aspects of ALA.

7.5. Wiring pattern - Part one

We now introduce the pattern that both solves the congruence problem just discussed in the previous section, and provides the alternative to all those disallowed coupling types discussed earlier. This pattern is usually an important part of ALA.

Note: The wiring pattern is not necessarily a part of an ALA architecture. For example, if your whole problem is just an algorithm, and therefore suits a functional programming style, then you can still compose abstractions with function abstractions, provided all function calls are knowledge dependencies, and not say, just passing data or events.

If you are using monads, especially I/O monads, or RX (reactive extensions), especially with hot observables, you are already using the wiring pattern. The pipes and filter pattern is also an example of the wiring pattern. Labview or Node-Red can use the wiring pattern. There are many other examples of the wiring pattern. Most support a dataflow programming paradigm. Here we generalise the pattern to support any programming paradigm.

The wiring pattern may be the same as the "Component Pattern" in some literature if used with what is referred to as 'configurable modularity' or 'abstracted interactions'.

The wiring pattern allows lines on your application diagram to mean any programming paradigm you want that express your requirements. It also allows you to implement multiple programming paradigms together in the same diagram.

If you are using dependency injection with explicit code for the wiring (not auto-wiring), then you are half way there.

The wiring pattern separates design-time/run-time congruence. It works by having a 'wiring-time' that is separated from run-time. 'Wiring-time' can happen any time before run-time. It can happen immediately before it, as for instance in LINQ statements or RX with a cold observable. It becomes powerful when we make wiring-time congruent with design-time. Usually the wiring code will actually run at initialization time, when the program first starts running. That initialization code becomes the architectural design.

Let’s suppose you have designed your system with two modules, A and B. There will be one of each in your system.

diagram 07

At run-time we know that A will talk to B. So we design A to have an association with B. The association may first appear on a UML model, or it may go straight into the code something like this:

static component A
{
   B_method();
}
static component B
{
   public B_method() { }
}

A and B may be implemented as non-static, with only one instance of each. The association is still there.

component A
{
   private var b = new B();
   b.method();
}
component B
{
   public method() { }
}

A may create B itself, which is a composition relationship, as above. Or A may have a local variable of type B passed in by some kind of dependency injection, which is still an association relationship.

component A
{
   B b;
   public setter(B _b) {b = _b}
   b.method();
}

Note that although dependency injection was used, it only eliminated part of the dependency, that of which particular subtype of B it is going to talk to, but A still knows the general type B, which is not allowed in ALA. (Part of the problem here is that A and B were probably arrived at by decomposition, and so they have subtle knowledge of each other, for example of how they collaborate.)

If A and B are collaborating, they are not abstractions. Their knowledge of each other at design-time (to enable their relationship at run-time) binds them to each other so that neither can be reused in any other way. And if they can’t be reused, they can’t be abstract.

Let’s revisit the water molecule analogy we discussed earlier for the Whole-Part pattern, and develop it further to be clearer how these dependencies affect abstractions. Let’s say we have decomposed water into two components, Oxygen and Hydrogen. Oxygen will talk to Hydrogen to get an electron, so we write:

component Oxygen
{
   var h1 = new Hydrogen();
   var h2 = new Hydrogen();
   h1.getElectron();
   h2.getElectron();
}

The diagram for that looks like this:

diagram 08

In the real world, oxygen is a very useful abstraction for making other molecules. In writing code this way to make water, we have tied it to hydrogen. Oxygen can’t be used anywhere else, at least not without bringing with it two hydrogens, rendering it useless. By implementing the Oxygen-Hydrogen relationship needed to make water in oxygen, we have destroyed the oxygen abstraction. We never even made the water abstraction. To understand water, we would have to read the code inside oxygen, where the parts about water have become entangled with the inner workings of oxygen, protons and neutrons and all that stuff. Oxygen is also used to make caffeine. We could never make coffee!

Caffeine molecule.png
Figure 56. caffeine - oxygen atoms are red

Abstractions are fragile and get destroyed easily, so we have to take care to protect them. What we needed to do was to put the knowledge about the relationship between oxygen and hydrogen to make water in a new abstraction called Water.

diagram 09

In general, to break coupling between peer modules A and B, we move the knowledge of the coupling to a higher level abstraction (less abstract level) where it belongs. Let’s call it C. C is a more specific abstraction. The knowledge is encapsulated there - it never appears as a dependency of any kind. And it is cohesive with other knowledge that may be contained inside abstraction C.

diagram 10

becomes

diagram 11

The diagram above is only to show the ALA knowledge dependency relationships between the three abstractions. It doesn’t yet show explicitly that an instance of Abstraction A will be wired to an instance of Abstraction B. In practice we never actually draw knowledge dependencies. We are just doing so here to show how ALA works. We would draw it in this way instead:

diagram 12
diagram 13

Now we have the explicit wiring. It looks a lot like the original diagram where we had no C. But where the knowledge is coded is very different. Because it is C and not A that has the knowledge of the relationship between A and B, Abstractions A and B do not change. They continue to know nothing of the connection. They remain abstractions. They remain re-usable.

It may seem at first that adding the extra entity C is a cost, but in fact C is an asset. It shows the structure of the system. It shows it explicitly. It shows it in one small understandable place. And it is executable - it is not a model.

The original abstractions were left below C to show that they still exist as free abstractions to be used elsewhere. They are not contained by C in any way as modules from a decomposition process would be. The A and B inside C are only instances. We wouldn’t normally bother to draw the abstractions below. So we just draw this:

diagram 14

C must achieve the connection between A and B either at compile-time or run-time. With current languages, the easiest time to do this is at initialization time, when the program first starts running. This is similar to dependency injection, except that we are not going to inject the instance of B into A.

This is what the code inside C might look like:

Abstraction C
{
   var a = new A();
   var b = new B();
   a.wireTo(b);
}

Typically we will write the code using the fluent pattern, with the wireTo method always returning the object on which it is called, or the wireIn method always returning the object wired to. The constructor already returns the created object by default.

Abstraction C
{
   new A().wireTo(new B());
}

If A and B are static modules, this produces something like:

Abstraction C
{
   A_setcallback(B_method);
}

7.6. Wiring pattern - part two

We are half-way through explaining the wiring pattern. Now we turn our attention to how A and B can communicate without knowing anything about each other.

This part of the pattern is also called "Abstract Interactions"

Of course, one way is that C acts as the intermediary. This way is less preferred because it adds to C’s responsibilities. But it is sometimes necessary if there are some abstractions brought in from outside. Such abstractions will 'own' their own interfaces or may come with a contract which C will have to know about. C will usually have to wire in an adapter, or handle the communications between the two abstractions itself.

A better way, because it leads to an architectural property of composability, is that A and B know about a 4th abstraction that is more abstract than either of them. This is legal because it is a design-time knowledge dependency. Let’s call it I.

diagram 15

I is an interface of some kind. It may or may not be an actual artefact. What it must be is knowledge that is more abstract than A and B and therefore knows nothing of A and B. It is more ubiquitous and more reusable than A and B are. In other words we can’t just design I to meet the particular communication needs of A and B. That would cause A and B to have some form of coupling or collaboration with each other, and again destroy them as abstractions.

I is so abstract, ubiquitous and reusable, that it corresponds to the concept of a programming paradigm. We will cover programming paradigm abstractions in following sections because they are a critically important part of ALA. We will see that ALA is polyglot with respect to programming paradigms.

circuit diagram.gif
Figure 57. In an electronic schematic, the components are abstractions that are composed using two paradigm interfaces - live analog signals and live digital signals

Returning to a software example, let’s choose a single simple programming paradigm: activity flow. This programming paradigm is the same as the UML Activity diagram. When we wire A to B and they use this paradigm, it means that B starts after A finishes. If A and B accept and provide this interface respectively, then wiring them together by drawing an arrow will have that meaning, and cause that to happen at run-time.

diagram 16

It is easy to create an interface for the activity-flow programming paradigm. It has a single method, let’s call it 'start'. Many abstractions at the level of A and B can either provide or accept this paradigm interface. Then instances of them can be wired up in any order and they will follow in sequence just like an Activity diagram.

Note that the Activity Diagram is not necessarily imperative in that any Activity can take an amount of time to complete that is not congruent with the actual CPU execution of code. In other words activities can be asynchronous with the underlying code execution, and for example, delay themselves during their execution, or wait for something else to finish, etc.

The code in Abstraction A could look something like this. Don’t take too much notice of the exact method used to accomplish the wiring. There are many ways to do this using only knowledge dependencies. The important thing is that A continues to know nothing about its peers, continues to be an abstraction, and yet can be wired with its peers to take part in any specific activity flow sequence:

 Abstraction A : IActivity
 {
    private IActivity next = null;

    public IActivity wireTo(IActivity _next)
    {
        next = _next;
        return _next;
    }

    IActivty.start()
    {
        // start work
    }

    // code that runs when work is finished.
    // may be called from the end of start, or any time later
    private finishedWork()
    {
        if (next!=null) next.start();
    }
 }

Abstraction A both provides and accepts the interface. This allows it to be wired before or after any of its peer abstractions. In ALA we use the word 'accepts' rather than 'requires' because there is often an end to a chain of abstraction instances wired together. If no next interface is wired in, the activity flow ends.

Abstraction B would be written in the same way, as it also knows about the Activity flow interface:

 Abstraction B : IActivity
 {
    private IActivity next = null;

    public IActivity wireTo(IActivity _next)
    {
        next = _next;
        return _next;
    }

    IActivty.start()
    {
        // start work
    }

    // code that runs when work is finished.
    // may be called from the end of start, or asychronously later
    private finishedWork()
    {
        if (next!=null) next.start();
    }
 }
As an aside, in C# projects, we wrote wireTo as an extension method for all objects. It used reflection to look at the private interface variables in the source class and the interfaces provided by the destination class. It would then match up the interface types and do the wiring automatically. It could even use port names to explicitly wire ports of the same types.

Now let’s revisit the molecule analogy. By now we would know to put the knowledge that Oxygen is bonded to two Hydrogens inside the water abstraction where it belongs.

diagram 17

In terms of knowledge dependencies it means this:

diagram 18

The programming paradigm here is a polar bond. It is more abstract (more ubiquitous and reusable) than any particular atom. We could have a second programming paradigm, a covalent bond, as well. Again, the important thing here is not what the code does - that is arbitrary (and not actually correct chemistry) but how the atoms can be made to interact while retaining their abstract properties with only design-time knowledge dependencies:

Abstraction PolarBond
{
   GiveElectron();
}
 Abstraction Oxygen
 {
    private PolarBond hole1 = null;
    private PolarBond hole2 = null;

    public Oxygen wireIn(PolarBond _pb)
    {
        if (hole1==null) hole1 = _pb; else
        if (hole2==null) hole2 = _pb;
        return this;
    }

    public Initialize()
    {
        if (hole1!=null) { hole1.getElectron(); BecomeNegativelyCharged(); }
        if (hole2!=null) { hole2.getElectron(); BecomeNegativelyCharged(); }
    }
 }
 Abstraction Hydrogen : PolarBond
 {
    PolarBond.getElectron()
    {
        BecomePositivelyCharged();
    }
 }
 Abstraction Water
 {
    new Oxygen()
        .wireTo(new Hydrogen())
        .wireTo(new Hydrogen())
        .Initialize();
 }

Let’s do one more example, this time with a dataflow programming paradigm. I have found that dataflow is the most useful programming paradigm in practice. It is useful in a a large range of problems.

Let’s construct a thermometer. Assume we already have in our domain several useful abstractions: an ADC (Analog Digital Converter) that knows how to read data from the outside world, a Thermistor abstraction that knows how to linearise a thermistor, a Scale abstraction that knows how to offset and scale data, a filter abstraction that knows how to smooth data, and a display abstraction that knows how to display data.

All these domain abstractions will use the dataflow programming paradigm. Note that none of them know anything about a Thermometer, nor the meaning of the data they process.

So we can go ahead and create a Thermometer application just by doing this:

diagram 19

Note that we configure all the abstraction instances for use in the Thermometer by adding configuration information into rows on the instances.

When we manually compile the diagram (assuming we don’t have automated code generation), it might look something like this (again using fluent coding style):

Abstraction Thermometer
{
   new ADC(Port2, Pin3)
       .setFrequency(1000)
       .wireTo(new Thermister().setType('K').setInputRange(20,1023)
           .wireTo(new Scale(32,0.013)
               .wireTo(newDisplay().setDigits(4).setDecimals(1))
           )
       );
}
The configuration setters and the WireTo extension method return the object on which the call is made to support the fluent coding style.

The diagram is the requirements, the solution and the architecture of the application, and is executable. The diagram has all the cohesive knowledge that is a thermometer, and no other knowledge.

The diagram can be read stand-alone, because all the dependencies in it are knowledge dependencies on abstractions we would already know in the domain.

Let’s say when the Thermometer runs, there is a performance issue in that the ADC is producing data at 1kHz, and we don’t need the display to be showing Temperatures at that rate. Also the temperature readings are noisy (jumping around). Let’s make a modification to the Thermometer by adding a filter to reduce the rate and the noise:

diagram 20

If the domain abstractions are not already implemented, we have got the architecture to the point where we can ask any developer to implement them, provided we first give them knowledge of ALA and of the programming paradigm(s) being used.

But let’s look how the dataflow paradigm might work.

If you are familiar with RX (Reactive extensions) with a hot observable source (which is an example of the wiring pattern), this is similar in concept although RX tries to have duality with for-loops iterating through the data. The dataflow paradigm we set up here will just be a stream of data. The IDataFlow interface corresponds to IObserver, and the wireTo method corresponds to the Subscribe method.
The ideal would be a language where we don’t have to decide if the dataflow will be push or pull, synchronous or asynchronous, buffered or unbuffered or other characteristics of communications. The abstractions would not need to know these things - they would just have logical I/O ports, and the type of communications could be binded in at compile-time as part of the performance configuration of the system.
Later we will introduce an asynchronous (event driven) execution model. It is preferable to do the dataflow paradigm interface using that because it allows better performance of other parts of the system without resorting to threads.

For simplicity, we will just implement a synchronous push system. Again, don’t worry about the filter itself. The code is just there to see how the LowPassFilter fits in with the dataflow programming paradigm, and how simple doing that can be.

Interface IDataFlow<T>
{
   push(T data);
}
 /// LowPassFilter is a DataFlow paradigm decorator to be used in an ALA archtecture.
 /// 1. Decimates the incoming data rate down by the setCutoff configuration
 /// 2. Smooths the data with a single pole filter with cutoff frequency equall to the input frequency divided by the cutoff. T must be a numeric type.
 /// Normal checks and exceptions removed to simplify
 Class LowPassFilter<T> : IDataFlow<T>
 {
    private Dataflow next;

    // This is normally done by a general extension method
    public IDataflow wireTo(IDataflow _next)
    {
        next = _next;
        return _next;
    }

    integer cutoff;

    setCutoff(integer _cutoff)
    {
        cutoff = _cutoff;
    }

    int count = 0;
    T filterState = NAN;

    IDataFlow.push(T newData)
    {
        if (filterState==NAN) filterState = newData * cutoff;
        filterState = filterState - filterState/cutoff + newData;
        count++;
        if (count==cutoff)
        {
            count = 0;
            if (next!=null) next.push(filterState/cutoff);
        }
    }
 }

You will notice that both the Domain abstraction, Filter, and the Programming Paradigm abstract interface, IDataFlow, use a parameterised type. This makes sense because only the application, the Thermometer, knows the actual types it needs to use.

7.7. Expression of requirements

One of the fundamental aspects of ALA is that the abstraction level of the application is fixed and defined by:

The succinct description of requirements

This is a similar concept to a DSL (but not quite the same). If the abstraction level were more specific, we wouldn’t have the versatility to describe changing requirements or new applications in the domain (too expressive). If it were were more general, we would have to write more code to describe the requirements (not expressive enough).

I noticed during 40 years of code bases written at our company, two did not deteriorate under maintenance. They always remained as easy to maintain as they were in the beginning, if not easier. All others deteriorated badly. Some deteriorated so badly that they could no longer be maintained at all. At the time we din’t know why and could not predict which way it would go. It seemed as if you just got lucky or unlucky.

Perhaps it was the type of changes that came along? But the two code bases that were easy to maintain seemed to be easy for any kinds of change. And the ones that were hard were hard for any change. This continued to hold for years on end. Of course, most changes were changes to requirements, but often enough, changes would be for performance or other reasons. These also seemed easy in these two code bases, but hard everywhere else.

I began to look at the structure and style of the easy and hard code. The easy code was not complicated while the hard code had degenerated well into the complex. The two easy code bases were doing very different things in very different ways, so there was apparently not a common structure or style. But they did have one thing in common. The code that represented the knowledge of the requirements was separated out. That code only described requirements, and it was expressed in terms of other things that were relatively independent, reusable and easy to understand (what we call abstractions).

This is what first gave rise to one of the core tenets in ALA. The first separation is not along the lines of functional or physical parts of the system, such as UI, Business logic, and Data model. The first separation is code that just describes requirements.

Of course this has a strong parallel with how DSLs work. Is ALA just DSLs? There are several differences. Firstly in ALA we don’t try to create a sandbox language for a domain expert to maintain applications. We don’t go as far as an external DSL. It’s for the developer and we don’t want to cut him off from the power he already has when it is needed. We just give him a way to organise the code and a process to get him there - describe the requirements knowledge in terms of abstractions and then trust that those abstractions, when written, will make it work.

7.8. No two modules know the meaning of data or a message.

The two modules will have collaborative knowledge. We reason that the sender must know the meaning to formulate the message, and the receiver must know the meaning to interpret the message. So how can it be avoided? The answer is to make the sender and receiver in same abstraction. They both know the same knowledge, so they are cohesive, so they should be together. In the logical view of the system, they are two instances of the one abstraction. We let the physical view fact that the sender and receiver will be deployed in different places drive them to be different modules.

7.9. Expressiveness

Requirements are usually understated initially in terms of abnormal conditions. However, they are usually communicated quite quickly relative to the time to write the code. In ALA, they are separately represented. The precise expression of the requirements using the right programming paradigms should take about the same amount of information as the English explanation of them.

In general, ALA probably requires about the same amount of total code. But once the requirements are represented, the domain abstractions are known and they are independent small programs with dependencies only on the programming paradigm interfaces used. This independence should make them much easier to write. As the system matures, the effort to modify gets less as more domain abstractions come on line as tested, mature and useful building blocks. The final cost of maintenance should be much less than an equivalent ball of mud architecture.

7.10. No models

Leave out details only inside abstractions

It is generally accepted that a software architecture must, by necessity, leave out some details. Somehow we need to find a satisfactory architecture without considering all the details. Often models are used to represent the architecture. Like its metaphor in the real world, a model leaves out details. The problem is they can leave out arbitrary details. We can’t be sure that some omitted detail won’t turn out to be important to the architectural design.

ALA therefore does not use the model metaphor. Instead, it uses diagrams (if not plain old text). Of course, this distinction comes down to semantics. I define a diagram as different from a model in that it does not leave out details arbitrarily. The only way to leave out details in an ALA diagram is inside the boxes, in other words inside abstractions. Because abstractions already have the required meaning when used in the diagram, the details omitted can’t be important to the diagram, and can’t affect the architectural design.

7.10.1. Executable architecture

Your architecture should be executable

The distinction between diagrams and models explained in the previous section gives rise to an interesting property of the ALA architecture. Diagrams are executable. Therefore the architecture itself will be executable. When the implementation of the abstractions is complete, there will be no work left to do to make the architecture execute (apart from practical considerations of bugs, misinterpretations of the requirements, performance issues, improvements to the initially conceived set of domain abstractions, and the like).

There should be two aspects of an architecture, the meta-architecture and the specific architecture. If using ALA, ALA itself is the meta-architecture and the top level application diagram is the specific architecture.

If your specific architecture is executable, it is also code. There is no separate documentation or model trying to act as a second source of truth.

7.10.2. Granularity

The final architecture of your software will consist only of abstractions. These abstractions will need to be independently readable and understandable. To meet this need, all of the abstractions will be small, even the 'system level' ones.

Conversely, none should be too small. We want them small enough to allow the human brain to understand them, but there is no need for them to be smaller, or we will just end up with an inordinate number of them. This inordinate number will tax the brain in a different way, by causing it to have to learn more abstractions than necessary in a given domain.

The ideal abstraction size is probably in the range of 50 to 500 lines of code.

7.10.3. Modules, Components, Layers

The common terms, modules, components, or layers often result from a decomposition process and therefore are parts of a specific system. The system may have only one of each type. The parts have a lower abstraction level than the system because they are just specific parts of it. In ALA we want to reverse this so that parts are more abstract than the system.

But say you do end up with some single use abstractions and implement it in a static way, it is important to still see these entities as two aspects in one: an abstraction and an instance.

7.11. Abstraction Layers

7.11.1. Layers pattern

With only design-time knowledge dependencies to deal with, layers are used for organising these dependencies so that there are no circular dependencies, and that they all go toward more abstract, more stable abstractions. As the name "Abstraction Layered Architecture" suggests, layers are crucially important to ALA.

In the section on the wiring pattern we ended with three layers:

diagram 21

There is a Layers pattern that also controls dependencies, but since most systems have numerous run-time dependencies between elements represented as design-time dependencies, these layers are used for the run-time dependencies. It is usually explained that each layer is built on services provided by the layer below it.

One example is the UI/Business Logic/Data model. Another example is the OSI communications model, where the layers are Application, Presentation, Session, Transport, Network, Data link, and Physical. In ALA, each of these ends up being turned 90 degrees. Metaphorically they become chains. In ALA each component wouldn’t know about the components next to it. That applies symmetrically, to the left and to the right. Data goes in both directions. At run-time, everything must exist for the system to work. It doesn’t really make sense to use a asymmetrical layers metaphor.

The design pattern for layers does have one or two examples of layering used by knowledge dependencies. The term ‘layer’ is therefore an overloaded term in software engineering. When used for knowledge dependencies, the English term 'layer' is a better metaphor. If a lower layer of a wall were to be removed, the layers above would literally collapse, and that’s exactly what would happen in knowledge dependency layering. The layers above literally need the knowledge of abstractions in lower layers to make any sense.

ALA’s ripple effects are already under control because the only dependencies are on abstractions, which are inherently stable, and furthermore, those abstraction must be more abstract. However, to make these dependencies even less likely to cause an issue during maintenance, we try to make the abstraction layers discrete, and separated by approximately an order of magnitude. In other words each layer is approximately an order of magnitude more abstract than the one above it. More abstract means more ubiquitous, so the layers contain abstractions which have greater scope, and greater potential reuse as you go down the layers.

We won’t need many layers. If you think about abstraction layers in the real world, we can get from atoms to the human brain in four layers. Remember the creativity cycle early in this article. We only need to go around the cycle four times to make a brain: Atoms, Molecules such as proteins, Cells such as neurons, neural nets, and finally the brain itself.

7.11.2. The four layers

We start with four layers. They have increasing scope as you go down. This type of layering was described by Meiler Page-Jones. Meiler Page-Jones’ names for the four layers are: "Application domain", "Business domain", "Architecture domain", and "Foundation domain".

Layers.png
Figure 58. Four ALA layers

ALA uses slightly different names: Application layer, Domain Abstractions layer, Programming Paradigms layer, and Language layer.

Application layer

The top layer has knowledge specific to the application, and nothing but knowledge specific to the application, i.e. representing your requirements.

A simple Application might wire a grid directly to a table. When Business logic is needed, any number of decorators (that do validation, constraints, calculations, filtering, sorting, etc.) can be inserted in between the grid and the table by changing the wiring of the application.

Domain abstractions layer

Knowledge specific to the domain goes in this layer. A domain might correspond to a company or a department. As such, teams can collaborate on the set of abstractions to be provided there.

Applications have knowledge dependencies reaching into this layer.

Programming Paradigms layer

All knowledge specific to the types of computing problems you are solving, such as execution models, programming paradigm interfaces and any frameworks to support these, is in this layer.

The Programming Paradigms layer will abstract away how the processor is managed to execute different pieces of code at the right time. Execution models are covered in detail in chapter four.

This layer is also where we arrange for our domain abstractions to have common simple connections instead of having a specific language for each pair of modules that communicate. The Programming Paradigms layer abstracts away ubiquitous communications languages (which we have been referring to as programming paradigms in this article.)

Let’s use the clock as a real world example. (This is the same clock example we used in section 2.9 when introducing the role abstractions play in the creative process.) One of the the domain abstractions for clocks is a cog wheel. Cog wheels communicate with one another. But they don’t do it with communications languages specific to each pair, even though each pair must have the correct diameters and tooth sizes to mesh correctly. The cog abstraction just knows about the common paradigm of meshing teeth, a more abstract language in this lower layer. This language is analogous to a programming paradigm. With it, the clock abstraction (which is in the highest layer) can then instantiate two cogs and configure them to mesh. The concept of cog thus remains an abstraction and instances of it are composable. The clock, which already knows that two instances of cogs are needed, also knows where they will be fitted and what their diameters must be. The knowledge in the clock abstraction is cohesive.

Language layer

The language layer is included to show what is below the other three layers. It is not hardware as you would find in many other layering schemes, nor is it a database, because it is not run-time dependencies we are layering. The lowest layer has the remaining knowledge you need to understand your code, that of the languages, libraries and any very generic APIs you may use.

The hardware and database do have a place, but we will cover it later. Being a run-time dependency, it will be well off to one side and slightly higher up.

Domain Abstractions API

The boundary between the application layer and the domain abstractions layer is an API that supports the solution space of your requirements (within the constraints of your domain).

The scope of the Domain Abstractions layer defines the expressiveness available to the application. The greater the scope (or bigger the domain), the more applications are able to do. The cost is expressiveness. The applications will have to be longer to specify what is to be done. Conversely, a smaller domain allows less versatility in the applications, but there is greater expressiveness, which means you write less code.

Possible extra layers

The domain is an approximation of all the potential applications and all the modifications you are likely to make. If the domain is large because it is enterprise wide, you could have an additional layer for small domains. The enterprise domain would include enterprise wide abstractions such as a person identity, and the smaller domains would add additional, more specific abstractions, such as a customer (by composition).

If the applications are large and themselves need to be composed of features, an additional layer that supports plug-in style abstractions may work well. Plug-in abstractions may actually be instances of domain abstractions, such as a settings Menu, or a customer Table. A feature can then add settings to the menu, or columns to the table that remain unknown to any other features.

Programming Paradigms API

The boundary between all higher layers and the Programming Paradigms layer is another API. It separates the domain knowledge from the programming paradigm implementation knowledge. It almost always takes care of the ‘execution flow’, the way the computer CPU itself will be controlled to execute all the various parts of the code and when, often using a framework. On the other hand, the Programming Paradigms layer doesn’t necessarily have any code at all. Remember that the layers are ‘knowledge dependencies’, not run-time dependencies, so the paradigm could be a ‘computational model’ that just provides the knowledge of patterns of how to constracut the code in higher layers. The decisions about use of the patterns and about the way the code is executed have already been made and exist in the Programming Paradigms layer.

Rate of change of knowledge

The knowledge in each of the four layers has different change rates.

  • The Language layer contains knowledge that will likely change only a few times in your career.

  • The Programming Paradigms layer knowledge changes when you move to different computing problems types, or discover different approaches to solving a broad range of problems. For example, if you have not yet used an event driven execution model or state machines in your career, and you move into the embedded systems space, you will very likely need to have those skills.

  • The Domain Abstractions layer has knowledge that changes when you change the company you work for. It will change at the rate that the company’s domain is changing, or is becoming better understood. If your company uses lean principles, one of the things you want to do is capture knowledge for reuse. This is the whole point of the Domain Abstractions layer, it is a set of artefacts that capture the company’s reusable knowledge.

  • The Application layer has the fastest changing knowledge, the knowledge that changes at the rate that an application gets maintained.

7.12. Composition versus decomposition

Here we revisit the important idea introduced in chapter 3 to do with the pitfalls of thinking in terms of hierarchical decomposition.

In decomposition methods, we are taught to decomposes the system into smaller elements or components with relations between them. Then decompose those into still smaller ones. The process continues until the pieces are simple enough to understand and implement. Each decomposition artefact is completely contained inside its parent artefact, so it forms a hierarchical encapsulation structure.

Decomposition of the system into elements and their interactions.

The decomposition approach is often the de facto or informal method used by developers because it is encouraged by many architecture styles and patterns, for example components or MVC. It is the method used in ADD (Attribute Driven Design). Indeed some definitions of software architecture sound like this meme:

  • From Wikipedia quoting from Clements, Paul; Felix Bachmann; Len Bass; David Garlan; James Ivers; Reed Little; Paulo Merson; Robert Nord; Judith Stafford (2010:

    "Each structure comprises software elements, relations among them, and properties of both elements and relations."
  • IBM.com

    "Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution. [IEEE 1471]
  • synopsys.com

    "Architecture also focuses on how the elements and components within a system interact with one another."
  • From an article on coupling by Martin Fowler https://www.martinfowler.com/ieeeSoftware/coupling.pdf

    "You can break a program into modules, but these modules will need to communicate in some way—otherwise, you’d just have multiple programs."
  • Loose coupling and high cohesion

The "loose coupling and high cohesion" meme suggests that loose coupling is the best we can do. We are told that modules or components must collaborate in some way. It seems reasonable and even self-evident. So why is it completely wrong? It’s because we are thinking in terms of decomposition. There is another way - composition.

To be fair, some of the examples above are vague enough to be interpreted in either way. But all are misleading in that they are suggestive of the idea of decomposition.

To fix the problem, we should re-word the meme:

Expression of the requirements by composition of abstractions.

All four big words are changed and some are exact opposites. Indeed, the architecture that comes out of this method is "inside out" when compared to the decomposition method.

Let’s contrast two pseudo-structures: one that results from the decomposition approach and one that results from the composition approach.