Tech Talk: "Breaking the Monolith" mit Maurizio Rinder von willhaben

Tech Talk: "Breaking the Monolith" mit Maurizio Rinder von willhaben

Hey guys! I would like to start my very first TechTalk now with a other title-slide at the beginning.

So I would like to jump straight into a little problem that my team and I have currently – so we have kind of a situation. I think, a situation well known for everyone. Let me guide you through the situation first, before we start the actual presentation.

So, currently I work at willhaben in the jobs-team, which is kind of a separate project, or a separate back end project so to say in willhaben. And here are some hard facts now about this project: currently it’s kind of a mid-sized project, about 150.000 lines of code for back end and front end. It has already – let’s say – around 120 database tables, tendency a bit growing – not that much, but still, linearly growing. I guess those facts that you can read here are well known to everyone of us, dealing with a monolith, so we are facing a classical layered-architecture, which is only horizontally sliced. So you have the classical packages like service, API and in the packages there are just a bunch of classes and implementation and interfaces, so… every developer’s dream – so to say – when you start somewhere. Consequently for that – it’s a mid sized project, so not that big – it already comes to a phase where we see that we don’t get features out that quickly. They take still weeks to be finished and so on. This is due to the not-so-good architecture, but also due to test environments that we have a lack of, because our project is currently deployed in virtual machines – you have a kind of limited set of available virtual machines. We are currently around – let’s say – 10 to 12 developers working on different stuff, so it really happens very often that those test environments are occupied.

You could do two things: either have one long long coffee, or you try to get one of these test environments, your choice. Maybe you also know, many software development teams already try to go more into this agile development, like Kanban, Scrum and so on – same for us, but agile development only works if you also have a good basis in the technical part. This architecture is very bad for agile development, because you have to touch so much stuff and you have so many merge conflicts or potential merge conflicts with other tickets and so on. It’s quite tricky.

Also – what you don’t have to forget – your project is kind of complicated, it also takes a lot of time to onboard new colleagues and new developers to this. They also need to learn for weeks how this stuff works and maybe they also need a couple of days to set everything up. Kind of unsatisfying – still, it’s a small project, but already unsatisfying.

The positive part is, we have currently an opportunity where we can change this situation, so we have to big projects in our jobs area, one of it is that we are currently exchanging our search-technology – so the technology for how we search for jobs and how we categorize them and so on – kind of a big element or artifact in our software or in our back end. The other thing is, that we are also upgrading our front end at the moment. We have a lot of places where we change or have to refactor many things. Exchanging the search engine is a lot of back end stuff like DAO stuff, service layers and so on. And for the front end it’s mostly API stuff, so creating new API controllers for them and so on.

With that I want to introduce you to my topic today! There is a chance for a journey of improvements in our team. Our decision as a team was that we use these opportunities to break the monolith and make from one monolith two smaller services. So, welcome to my presentation! I’m going to talk about breaking the monolith – it’s a smaller story, a willhaben-jobs-team story so to say. My name is Maurizio Rinder, I’m working for willhaben very close to two years right now, it’s a very cool company. Currently, for this project, I’m the so called TPL – Technical Project Lead. So I’m responsible for the whole organization and planning of getting this search-technology out to you, so that you have a more satisfactory result list for your search requests and so on.

What I want to give you today are not many deep technical things, but more a way of how we are currently organizing this project. So we introduced many procedures or tools that help us to kind of have a common understanding and also a good overview and a good navigation through the project without – let’s say – forgetting things, getting stressed, getting more grey hair that you already have and so on.

Those are the four points that I want to take a closer look at: Preparation and documentation – as you know, big project, preparation is key! Then I would like to introduce you to how we organize our tasks, so our workload, how we split it up. Then, one thing that I’d like to mention here today is how important the team and the stakeholders are for such a project. At the end, I also – yeah, we’re in a TechTalk – give a kind of overview on the techstack we have and what we chose for this back end to work with.

But first, let’s go for the documentation – a not so nice topic maybe for many people here. Because… documentation, I mean, come on. But the thing is: actually you need it! You need it on different layers, so that all of the stakeholders you deal with understand what you want to do. What is the target? What does the endproduct look like? How does the architecture look like? How do the back ends communicate to each other? You kind of have to put some stuff already in this documentation. The important question is always: How much should I put there?

Maybe you know, the classical waterfall model looks like „okay, I’m typing everything down, very in-detail, then I have everything 100% planned, then we start!“

Well, that was – let’s say – in the 1990s, we are now in 2021, so we can do something different here. On the right side you see the first sketch of our new little infrastructure. We do it in this program called Miro. So, we sketched it down a bit and put in all the ingredients that we need for this infrastructure, but the thing is, such a sketch is not the best thing. I can also write it down on a tissue paper, it’s the same. You’d never find it again and never get really comfortable with it. Rudimentary sketches are a good thing, but you need better documentation. For us, one of the ways to go here was that we chose arc42 for the first documentation. There were already some talks – I think in the last year or something – at the Domain Driven Design meetup and so on. Quite an interesting topic! We heard about this once in such a talk and I kind of – especially for this project – got hooked.

So I took a look at the webpage itself and I found, okay there are 12 chapters. Every chapter has kind of its own points on what you should document about your architecture, about your plan and so on. I said to myself, whoa, that’s insane! For starting, that’s a lot. I’m going back to the waterfall model – no no, I don’t want to!

But I got inspiration from one of the founders here of this arc42 specification Dr. Gernot Starke. I think I watched one INNOQ Technology Lunch, it’s a 30-40 minute presentation. And this Gernot Starke presented this arc42 and also gave tips on how to start with this. Especially what he suggested is that you don’t need to document all 12 points, you just need a subset of them – the most important things. Which are those listed here. Those are the current confluence links that we have. You put down your requirements and goals, constraints that you have, then the system scope – so that everyone has an overview – you have of course a concept, which means where you put your architecture and stuff. Feel free to watch this video! Unfortunately it’s in german, but it has kind of good inspiration on where to go with first documentation and also one other thing that he mentions – also comes in the presentation here – is keep decision records! It’s a very good presentation, I put the link into the slides, so you can go there whenever you like.

Yeah, the next thing – so after the documentation or let’s say the first plan was set up – is bunch of work. I saw in the time schedule that we need around 6 months for this whole project, especially when you also include the front end with this. They work together now. One of the tricky parts in this project is also that we need only change stuff in the back end, but in this project there are around five different platforms involved. It’s the back end, the web front end, Android, iOS and the older back end – you have five platforms where you have to do changes and stuff like this. Since we have a quite skilled agile coach with us since about 1.5 years, I got also inspired by her about this user-story concept. She was very keen on bringing in this user-story concept and clean up the backlog or our sprints, so that you have at first artifacts that everyone understands, so even a project manager who has never touched technological stuff can understand user-stories, because they’re one/two sentences and criterias on what you need to fulfill to get this out. It’s kind of a wishlist, so to say. On the right side in this organigram you actually see how our project is now organized here. Actually we’re organizing this thing in JIRA, so this „EPIC“ stuff – on the top – is JIRA. Currently this is the „Exchange Search Technology“ project, so there’s nothing big inside, just the topic that we’re going to work on. But then, this project is split into so-called milestones – I call them also work packages. They’re very cool, because you can say for my project this is the chapter „Result List“, „Filter Parameters“, then some other stuff and so on – so you have work packages where you can put the user-stories inside that they belong to. A cool thing! Currently we also use these working packages to track our progress, so we can also talk to the business like „okay, we’re working on those things and they are currently X% ready.“ We can also say „yeah, this working package is now 100% ready.“

So yeah, kind of a good overview here. There the user-story, as I said is more the business view of it. For the developers, actually we kind of have this classical thing like sprint planning 1, 2 and also the refinement meeting – refinement meeting is more this thing where you can discuss the user-story, estimate the user-story by storypoints, T-Shirt size, whatever you like! And in the sprint planning 1, there go the user stories inside, sprint planning 2 is then very important for this concept, because there you sit together with all the developers and discuss how we solve this user story, what needs to go in this user story. Obviously you do the obvious things, creating an API endpoint, the service routine, database routine, whatever. But you can also decide, okay we have some refactoring tasks to do here – maybe we can also attach this to the user story. The outcome of this is – since we’re doing this with JIRA and Confluence, so all this documentation stuff – you have a very good tracking here on what has changed with each user-story. You have good backtracking.

Whenever I open up a user-story I can see an Android ticket, a front end ticket, a back end ticket and I know, okay, on these platforms I changed something. Quite a nice thing! With this setup you’re kind of very well set up. If you keep this process all the time; sometimes it’s stupid at the beginning because it looks damn simple and many people are maybe not so keen on doing simple things, but I tell you – also being half-italian – simplicity at its best is a good thing! It’s also difficult to handle, because you need to stick to it!

So yeah, the next thing: teamwork. Your developers, the stakeholders and so on – very important! Especially as a project lead, you need some communication skills, you need to know when to involve this person, that person and so on. You also need to kind of – how would we say – know when you want to talk with the project manager and the business and what you want to give them and how you present it also – I guess you know, if you go too deep into technical stuff, you 100% lose those people. You need this skill to be on a surface where they still understand you and know that there is progress and optimism. But besides the non-dev stakeholders, within the development team you also need to try to involve everyone who is taking part in this project. It’s not a good thing – like the classical way – to go into the corner, develop everything and show everyone „hey, it’s ready!“

No, especially for breaking the monolith, you have technical decisions to make. What architecture should we take, which database technology, how should we do the back end to back end communication for stuff like this. In our case, what we introduced here is that we have a bi-weekly jour fixe. So, we are just four back end developers for concretely this project. The day after the refinement meeting, we have this opportunity to meet together, talk about the current user stories, maybe technical decisions. Or maybe also some social stuff, how we work, is it fine for everyone how we’re doing this. Maybe also talking about how we should split the user-stories, so the dev tasks. Should it be one dev task, or maybe multiple dev tasks for one platform – so many topics.

What I also like is that everyone in your team – every developer in your team has some particular abilities. Strengths, topics that the person is keen on. Maybe there is someone who knows every library outside by memory, or knows which version has this and that inside. Or you have someone who is very good with database stuff. Another one is working more on the REST controller side – this has more dev information that you could have.

I would say, using these abilities of each developer is a very good thing. They feel happy, you get happy, you have good quality. At the end, I mean honestly – whoever is watching this video and also the people here in this room – who knows 100% of all IT? No one! So you should be open minded for let’s say the knowledge of the other people.

This is one example now that I’d like to share with you here. This was the very first appointment of one of our jour fixe and what we discussed here was which architecture should we take? Should we still stick to this classical layered architecture? Horizontally sliced? Vertically sliced? How big, how small, how many layers? And so on…

We had kind of a discussion, the four of us. At the end, the decision record is logged, you can see here the record of this, what we discussed and what we decided. We came to the conclusion that we go with the hexagonal architecture – I know that you already had one guy here, talking about this, I was so nice to put his link also in the slides, feel free to watch this thing about Hexagonal Architecture too. I personally have to say, this is the first time I used this style and I really like this! What you can do here is that you have a very good flexibility and also a good management for your cross-dependencies and so on. They have a kind of a defined direction. The dependencies go to the domain and to this service layer and other than that you only have this adaptive module or package – you name it – where you can goof around however you like. You can plug it in and plug it out – perfect. This is agile development at its best! With these user-stories combined with this architecture you can really say „okay, you go to the database, you go to the REST controller, we meet each other in the center and we integrate later on. Everyone can work on their own.“ Yeah, so quite a nice thing! Also, what’s cool here is what I’m going to try out – or I already tried out here – I also take other developers into it. Maybe for example you see here the current status – we have a web front end and a app front end and both of the are kind of heterogenous, so they take different response structures and also data. The web likes SEO stuff, the apps don’t – and other stuff. So I decided, yeah, okay we have front end developers for web and app, why not bring them to this concrete adapter and say „yeah, you can do stuff here too, it’s quite easy, you can’t destroy many things, because it’s very decoupled and you just use the domain service, there is need to go to these deep adapters and service and domain stuff as soon as a real back end developer did this already.“

So, quite a nice thing – it really kind of gives you the opportunity to get T-shaped developers and have more of the collaboration that you want in your team. As I said – agile development works here quite good. You can do stuff in a decoupled way. Try it out! Whenever possible! Or if you split something like this, you should give this architecture a try. I know the learning curve in the beginning is quite tricky because we had some problems… hmm, adapters, ports, how does this work, how should we name it, how should we package it and so on. But I think it comes with every feature ticket that you have, you kind of constantly refactor those things or improve these things – it comes with experience, I would say. It’s like a little journey, with every step in your journey you learn something new and kind of introduce it with your behaviour and with your skills and so on.

Another thing that we decided together is which techstack we’re going to use. There are a couple of things here, like we write our back end in Kotlin – quite a good choice I must say! I haven’t used Kotlin before, but I like it a lot, because you get to this point where you kind of prevent the classical NullPointerException and stuff like this, so it’s quite stable and quite nice to use and very easy going. If someone mainly wrote TypeScript stuff, you can also get them on board quite easily, because the syntax and stuff is quite similar.

For the database and back end classical Spring Boot and PostgreSQL. We have our sourcecode on GitLab and also the deployment and CI/CD stuff. One thing that we specifically – because for the architecture and some other stuff – we tried out is we used ArchUnit to kind of check on a constant basis if the rules are kind of still valid. So if you write something wrong here, or you tried to access a different adapter with your adapter, then you will have a problem with ArchUnit and you have to fix this. For the back end to back end communication we will use Kafka, but there is still a decision that needs to be made – I have this on the next slide. And one other thing that we are going to try out now – since we have to front ends – at this moment is to introduce pact with this. Kind of this consumer back end consumer provider contracting thing. We also have automation tests here whether you fulfill the contract with your front end guys.

For Kafka, there are two things, two technologies or two ways we want to discover. Obviously we use Kafka Connect as this component to load up our configurations, but there is still a hot discussion about wether to use Debezium or this JDBC Source Connector. Debezium is currently in use in our back end for data warehouse tracking, it works quite nice, but it has some issues with PostgreSQL clusters, so if one PostgreSQL instance goes down with the application and so on, then you need to reboot the other one and need to be sure that every application slots… it’s quite a thing. I like this technology, because it’s more pushing stuff, but others are more for the JDBC Source Connector, because it’s a polling mechanism, maybe has higher availability and so on – but we’ll see. This is an example now also put into a meeting that we will have at the end of july or something, where we talk about this, put in the pros and cons of everything, yeah.

That is kind of a journey – I personally like this! It gives experience and lets you grow. At the end you kind of have a very satisfying product. Maybe it’s also kind of an investment for your future self all the time.

Before I end this presentation, here are the links that I mentioned. The first one is kind of a blog where you can get more information about this user-story stuff, how you split user-stories, how you create them, what is their purpose – feel free to look at this. The hexagonal architecture – as I said – this is the video from the guys here from devjobs. Then there the is this INNOQ Arc42 video that you can look up – grab a lunch, plate of pasta or something, sit down and watch it. It’s worth the time! The last one is one of my personal blogs about Debezium, it was the starting experience that I have.

With that, I will end this and say also that we are hiring. There are still positions and so on open, so if you’ve already watched all the links before, then maybe you also want to come by and take a look at jobsbei.willhaben.at and see which jobs are open!

The last thing I would like to say here, which is kind of my personal experience that I have and I like. Either if it’s in starting a new job or starting a new project or going for something new – so to say – is:

Life begins, once you get out of your comfort zone. And that’s something that you should do on a constant basis.

Thank you!