The Craft of Complex Software Development
I thought that I’d make an effort for a nice change. Granted, things have been crazy for myself recently. However, I thought that I’d make sure that I dedicate some time to my blog. It feels like I’ve not posted anything juicy in some time now. I thought that I’d share some of my thoughts & some of my experience with regards to developing & maintaining complex software applications. But not from a technological standpoint. Rather, the domain standpoint. Ironically, this seems to be a reoccurring issue amongst some set of developers, where they just struggle to get it right.
In all honesty, due to life changes, career demands, etc. One big life change has been bringing a new born baby boy into the world, so as you can imagine, I’ve been exceptionally busy as of late. 😅
Unfortunately due to all the chaos, it is taking me a very long time to work on personal projects, it is taking me a long time to write up some blog posts that aren’t just a 5 minute, half arsed write up, etc. SO without me blabbering on, let’s talk about building applications that are simply complex by the requirements. Without a doubt, tons of developers have talked about this subject area, but I thought why not throw in my 2 pence piece as well?
In my previous roles, I’ve not had to deal with all that complex business rules. Traditionally, in my past roles, the that has complexity has always been strictly technological. Ironically, what is considered best practice which can sometimes contradict best practice with exceptionally complex business rules & use cases.
An example could be something as simple as a view model or a request object, or whatever you want to call your object or class. You may be in some situation where you want to simply reuse that same object. But given the two different use cases, you’re not using the same set of properties. Sometimes you don’t need to know the ID. An example being where you create an entity where the ID is typically generated via the database. Other times you need to know the ID. An example being where you need to update some record(s) within your database. You need to know the ID(s) while running some update query, because without the ID(s) you’re likely to corrupt the data within your database. If you have a multi tenant application, the other tenants aren’t likely to be pleased with such an outcome. But in order to achieve maximum performance, we can generally agree that you should only ever work with small amounts of data at a time. It’s generally faster to run a lot of really small queries against the database as opposed to large & really complex ones, tis arguably common sense.
In such a scenario, granted this might not be the best scenario in the world, but theoretically, you could create two separate objects. This way, as you grow your product, the complexity of the code base continues to increase, etc. These two objects can continue to grow in different ways, isolated from one another. This can prove to make the general maintainability of the system easier in future. It can also make unit testing a lot easier. It can help prevent over engineering something simple. If you couple this approach with other concepts such as a screaming architecture, in my eyes, you’re off to the races!
Application Layer Architecture
Speaking briefly of architecture, ironically sometimes there’s a conflict of interest here as sometimes introducing more & more layers of code into the mix can sometimes cause an unnecessary amount of complexity. Let’s use an example where your business logic needs to talk with some database.
Some may by default decide to use a simple service & repository type of pattern. But if there’s an obscene amount of complexity behind the feature & then there’s also an obscene amount of complexity buried within these two additional classes. Is it really wise to attempt to abstract this code away?
Granted, for the most part, it’s likely that it’s fine for people to abstract away some code into these type of patterns. But when the complexity is so high that you end up hating yourself for trying to jump between 3+ files, you may begin to see a flaw in your initial architecture. Granted, you should always try to simplify your code, thus using more layers doesn’t really introduce any unnecessary complexity. But there are some cases where the complexity is unavoidable at times. An example may be the feature that looks at transferring money from one bank account to another. There may be features that run fraud detection, there may be features that do all sorts of amazing & fascinating things. The list of features that could be involved in such a process could be infinitely long.
But this isn’t necessarily due to technological complexity, this could in fact have a very simple technological implementation behind the scenes, using a nice ol’ pub/sub architecture. Simple, cheap, scalable, what’s not to love? But there’s just the simple fact that there’s a lot of processes that need to happen, they all need to succeed, behave correctly, run as expected, etc. But that’s all tied to the business domain, that’s not really complexity derived from the technological solution or anything like that.
A lot of well known software engineers have mentioned a lot of concepts surrounding complex application development. But a lot of people may have complex technical problems, but generally simplistic domain problems, an example being how you might be able to implement some solution where you can create a relatively basic CRUD application. There may be some need for some complex technological solutions behind the scenes. An example might be dealing scalability.
Scalability is ironically a problem that’s a double edge sword, it’s a problem that you want to have. But there are a lot of scenarios and situations where the requirements of the application are exceptionally complex. The technical implementation might be ironically quite simplistic. As mentioned above, you may have the need for an asynchronous architecture, which can be achieved using the likes of Azure Service Bus. Followed by an Azure function to subscribe to the bus & an Azure web application to publish to the bus. That’s a pretty simple architecture right there, it’s somewhat scalable & it could be quite affordable too.
In my eyes, the situations where the business domain introduces complexity are substantially more interesting. An example might be how your application may integrate with an array of different services. However, based on user configuration, you might want to consume different endpoints, you may want to treat the data differently based on the configuration. Granted this might not be the best example. But it’s an example either way! 🤷♂️
A real world example of a complex business domain is QuoteOnSite, trust me, I know the quirks & the devil that lives within the detail. There’s a lot of complexity due to the fact that there are just so many edge cases, one small, yet pretty simple example being the way QuoteOnSite calculates tax for an array of line items. Some systems, such as Xero will calculate tax somewhat differently, the output may end up being the same, but fundamentally, the calculations are completely different. There’s so much configuration that goes into QuoteOnSite, that just from a user perspective it can be considered quite complex. Regardless of how neat, clean or easy the UI is.
It’s very likely that you’ve heard of concepts such as domain driven development, behaviour driven development, etc. And there’s no wonder. If you’ve ever had to develop some really complex application, it’s likely that you’ve encountered challenges quite early on.
This might be just conversing with colleagues, be that other technical individuals, or non technical individuals. These are always the fun challenges as they simply require a thought process that goes beyond the technical mindset. As a result of poor of communication & not utilising the benefits that arise from DDD & BDD you will encounter the saying:
As software developers we fail in two ways:
We built the thing wrong, or we built the wrong thing.Steve Smith
This saying is fantastic, because it can be applied to 90% of software solutions out there, perhaps even more than 90%. Granted, I might even say that 90% of software solutions do not require these standards, such as DDD, such concepts only really apply to complex domains. Complex applications that have complex business logic, the technological requirements are arguably somewhat irrelevant with regards to DDD. An example being how you might need to allow for real time communication, low latency, very high concurrency, etc. All of which are technological problems, easy, not really, but still technological.
This is where the likes of using architecture approaches such as CQRS start to make a load of sense, plus it also means that if you have a clever database solution behind the scenes, you could have a read only database to read data from. Followed by another database that you write data to, just to ensure that while writing data to the database, it never causes read queries to slow down due to some insert or some update running.
There have been scenarios where I’ve encountered people trying to stick to standards for the sake of sticking to standards. As a general rule of thumb, standards are awesome & they’re typically there for very good reason. But to stick to a single approach for all kinds of problems is in itself problematic. But this is why the likes of diversification is a pretty damn nice choice! Much like investing.
An example being how in my eyes, sometimes REST just doesn’t make sense. Purely due to the fact that I’ve found that you end up needing to make some endpoints too flexible. In such scenarios, I find that protocols such as RPC make a lot more sense. Because you can be a damn sight more specific about what a given resource is going to do. This kinda solution with REST is considered somewhat bad practice, and that’s fair enough.
Typically I’ve found over the years, the more complex the application the more it makes sense to adopt the likes of a screaming architecture. Once you reach this level of complexity, sticking to a given architecture, such as a traditional MVC architecture, it slowly starts to make very little sense. Don’t get me wrong, I’m not saying there’s anything wrong with MVC, but typically it’s not flexible enough for exceptionally complex applications.
An example may be the onboarding process of your product. Some applications keep it as simple as humanly possible, such as registering & maybe confirming the email address they provided. This works for a large number of entities, however, given the context of your business, this may not be enough.
In the context of QuoteOnSite, it’s not enough. We need the user to provide the likes of their business name, their logo, their company branding or colours, etc. There’s a lot to it, but all of this is necessary in order for the application to make sense when the user logs in & begins to use the application. So when they do send documents to customers or third party entities, there’s some branding specific to that company on the document. Ironically, if one were to try to stick to something like MVC for this solution, it would be unnecessarily complex.
In this situation, it did not make sense to try & stick to a really basic architecture, just because of all the intricate business rules alone. An example being how if you don’t provide a specific property, it might make an assumption for the user & provide some predefined properties. Properties which the user can later configure within the application, but for the on boarding process, it’s an entirely different logical process.
Granted, it might be pretty lazy to state that it’s a matter of common sense & intuition. But there’s not much more that can be said without simply spouting off my own opinions on how a solution should be built & architect. All that I can say is that you should do your homework before you build. See mistakes that other developers have made over the years. See what the likes of uncle bob have to say on architecture, etc.
I will also say that you shouldn’t stick to a given approach or recommendation like it’s a religion, but use it to your own advantage. An example being Martin Folwer’s comments around domain specific languages & so on, the high level content is simply golden.