The Complexity Bleed

Introduction

In the world of software development, the odds are that you’ll at some point have to deal with building and/or maintaining a complex system. For those of you with a nice bit of experience can relate, I’m sure.

I have recently developed a little philosophy that granted, it’s not a one size fits all solution, but for me & the scenarios that I’ve encountered, I really like it.

The Philosophy

I’ve encountered scenarios where the entire ecosystem is quite literally mentally draining, because there’s just so much complexity at every level throughout the stack. For instance, there have been applications where there’s a lot of business logic on the backend. But there have also been situations where the data coming from the backend is in such a strange format & sometimes even in an incomplete format, the front end needs to jump through a lot of hoops. I’ve seen systems where it feels like you might need a PhD to deal with the database itself. And we’ve not even covered the infrastructure side of things yet.

I sat there for sometime the other day, pondering the question:

How does one effectively deal with all of that complexity?

My brain the other day

This is one of the questions I ask myself often, in addition to other subject matters such as performance, security & so on. One area of my work that I’ve previously taken pride in is developing software, regardless of the complexity, in such a way where it provides a pretty good developer experience.

So, how have I previously handled complexity? – The short answer, it depends, the annoying thing about our world is that context is pretty much the dictator. But one relatively simple tactic that I’ve seen myself lean more & more towards is encapsulating complexity, for example, if the database is poorly designed & massively over engineered, etc. Rather than pollute your backend code with unnecessary layers of complexity to accommodate to that, why not just make your queries really complex? You might consider using stored procedures?

Now granted, this might not always be elegant for the layer where you have all of your complexity. In my opinion, it allows you to essentially hide some of the complexity, you lock it up in one area to prevent other areas of your product having to deal with a lot of complexity.

But why though?

Why might be a perfectly valid question to ask, the simple answer being that it prevents the headache of having to deal with the complexity essentially cascading throughout your product. Take for example the theme that I’ve gone with where you’re dealing with a really complex database, I’m sure most of us have been there. But let’s run with that idea, by having one super complex query, it might prevent your application from having to make multiple requests to the database. Now that’s a good thing in the sense that it makes the architecture a little less chatty, but it might also mean that you don’t have to deal with as much query related logic at the application layer.

Let’s use a little example shall we? Let’s say I wanted to get a customer from a banking application, but I then wanted to get all of the customer credit products related to that customer. Let’s say for instance, the user interface has a nice tab system so all of my savings accounts, current accounts, etc, they’re in a different tab to my credit products such as a card finance product, a personal loan, a mortgage, etc.

Now one might say this is some totally reasonable logic to have at the application level:

Now granted, this isn’t the most crazy & complex code that I’ve ever seen, but let’s for arguments sake say that because the database was designed so badly, that there’s a need to do a lot of complex queries. I mean in this silly little example, it’s just doing the two database requests. But what if I told you that you could turn it into one?

So with this example, you can see that the condition of there being no customer, this no longer has any impact on the number of database requests that the application makes. It doesn’t care in that respect. Yes, the query itself is more complex & one might argue that this is less efficient than just making another database call or another couple of database calls. But in a sense, there’s an element of complexity that has been removed from the application layer, the application layer no longer cares if there’s no customer and if there’s no customer, then we don’t run another query. No, the query in a sense takes care of things for us, yes, at the application level, we still care about knowing if we have a customer or not, since that would impact the API response, but there’s just 1 database call as opposed to two.

Granted, this illustration doesn’t work so well on a simple use case, but let’s imagine that you’re dealing with a really complex application, where you might need to run a query against 30 different tables, or potentially more? Let’s imagine that you’re running a SaaS product, where there’s some level of authorisation involved & you have admin users that can dictate who can see & do what within the application. In such an application, clearly there’s a need for a users table & possibly a user permissions table. Granted, you could use something like active directory & store roles against users, but come on, where’s the fun in that? Let’s just run with the idea of being in a bootstrapped startup where you just have to hack your way to success.

Now let’s look at the scenario above, where it’s the same database, but on a different portal, let’s say this is the admin portal. The admin might need the basic user permission to manage customers, but then there might be an additional table that might act like some look up table or something, that states if a user can manage a specific customer or not. It might be that the company has a policy in place where users can’t manage their own accounts or accounts of relative, etc. This might be an ethical concern as opposed to a technical concern. But you can see with this example, the complexity is quickly starting to grow. Now if you were to include the likes of reporting, you might need to include more & more conditional logic again, etc.

Conclusion

Even having written this article, I think it proves a point where this kinda philosophy only really applies at scale. If your application isn’t that complex, then the odds are that you don’t need to think about such things. But when you’re at a level where you’re dealing with just a mind boggling amount of functionality, layers upon layers of complexity, then it starts to get complex.

I guess you can place this thought process in the same pool as topics like DDD, you certainly don’t need to use DDD on really simple applications. But as you grow, scale & develop your product further, using a framework such as DDD quickly begins to pay off.

Even if you’ve thought I’m talking complete & utter nonsense, which fair enough, everyone has an opinion, I hope that you’ve at least enjoyed the article. At best, I hope I might’ve given you some food for thought.