Tech Talk: "Drones Swim" mit Thomas Lutz von Frequentis

Tech Talk: "Drones Swim" mit Thomas Lutz von Frequentis

Hello everybody! Thanks for the devjobs.at TechTalk invite. I am here today to present a bit about drones and how they SWIM. Sounds strange but that happens in real life. Speaking of „real“: It is about real time systems, but actually its not really about real time systems, its about real life systems. Things that happen in our environment, in our daily life, that you might not notice but still they’re around.

Before we dive in a bit of plan and agenda. The next 10 minutes will be some introductions – me and who we are. It will be a bit of context – what does „real life system“ mean – what do we in our daily work in our company deal with, what’s the challenges we have to solve? I picked one concrete task – adding something to this real life system – and from that task I picked one challenge – managing the information part of the systems. And finally I try to present an approach that at least for us works in dealing with such real life systems.

Let’s dive in – introductions first. And before I start with that I am really sorry, there is no radically new thing in there. To be precise: there won’t even be one line of code but still I think it’s worth listening.

Introductions: First of all me – I am Tom, Thomas Lutz, system architect at Frequentis. Currently I am running drones. „Running drones“ means in new business development this is the department I am currently assigned to. I do a bit of development, a bit of project management, a bit of research. Driving the topics – drone traffic management – through the different stages in our innovation process. From an idea via research project to something that is commercially exploitable and that actually sooner or later bring some profit. Get in touch, you’ll figure it out – email or LinkedIn – I guess you will find me. I said before: I am not a developer anymore. I still do code so I have to say sorry. And to be honest I used to be sharp but at least I have some experience to share.

A bit about the company – who we are: It’s in a nutshell everywhere – and everywhere with a special focus. And the special focus is on control centers. I have a few numbers here: 240.000 km of shoreline a managed by our systems. 90 % of all air traffic passengers have been – sooner or later – in touch with one of our software, hardware, whatsoever components in one flight from A to B. In times like these 90 % is a saver number but actually this means billions of passengers. Before corona hit it was 3 billions of passengers per year that have been handled by our systems. You don’t feel it, you won’t have a ticket that is scanned by one of our end user devices – it’s more less in the background, in the tower, in the public safety emergency control – but we will come to that a bit later.

A bit later means context. Context described by a few images. You can see control centers in our five business units, strategic business units, that we have. Left side: ATM, air traffic management civil – this means a tower. It is not always this tall building on an airport, sometimes it’s quite hidden the area control centre that manage traffic on routes over continents but in a nutshell they all look like this. Means to see where traffic is, means to see what will be in a day or in two days, to flow management, to keep planes separate from each other to ensure safety in a nutshell. Quite the same in defense: air traffic management systems ensuring separation, ensuring that the people can land, can depart, can manage the flights. Public safety – slightly different – no planes but in that that case police cars resources. Call tackling, if you call the 112 or 911 in quite a lot of countries actually, you might end up in one of our systems. They calltech operates a headset, they can take details what’s going on, who is where, what do we need to do, what do we need to support. In a nutshell, like in an ATM system, the position where a controller can take care of people’s needs, in that case emergency needs. Public transport – it is one our newer business units, same concept: if you for example drive in Vienna using the underground, those people in there in the control rooms, they need to communicate. The drivers, they have radios or phones, people running in the stations they have phones that runs via our communications systems as well. Trains, railroad networks all over Europe – knowing which train is where, knowing which wagon is where – that’s part of the control centers that we have there. And last but not least maritime – we had it before. 250.000 km of shore line. You might get the pattern behind, it doesn’t really matter if it’s a moving object, be it a plane, be it a police car, be it a ship – you need to manage it somehow and this is what we do, this is the context.

And now the task. What if all those things come together? You might wonder how would this happen? Why would a police officer have something to do with an air traffic controller? And the answer is easy, the answer is: drones. Drones are easy to get, quite cheap, they range from a few euros up to few 100k. But it very much depends what you want to do with them but for decent drones you just need to go around the corner to your favourite shop, buy one and fly. This is not only something that people at home do for their hobby use – private pilots – but it is well something that for example public safety units found out. It’s easy – you get a drone, you put it in your trunk. If you come to a traffic accident – like we have here – you open the trunk, you start the drone and within seconds you have an overview. Who is injured, who you need to send, how many cars, did you miss somebody – maybe there was a car flying off the street that is now in the forest. Same for maritime. You need drones in ports. Instead of sending out a boat who scan for damages on the ship, or to see if somebody has something mounted to some places on the ship to smuggle in and out stuff that maybe they don’t want to have declared. Again, quit a nice use for drones. Trains – some of them, especially those that do building stuff have drones on board. They fly ahead 5 km, scan what they need to do, what they need to repair. Again same concept – easy to get, lots of traffic. Numbers. Right now – or before last year – we had average 30k flights over Europe per day. The number actually did not go down that much – passengers went down a bit, but the number of flights indicates all the planes that have either filed a flight plan or called somebody to announce themselves, typically this is carriers – you flying the business trip, you flying on vacation. They basically fill out a form, say I want to go from Vienna to – I don’t know – Ibiza and they declare the type of plane, speeds, the average altitude they want to fly – and then somebody says OK. There's forecasts that say in 2035 we will have 17 million flights – 16.8 – average a day over Europe. It's a forecast, so add a zero, remove a zero. It's in 15 years. Who knows? But one thing is granted, it's significantly more. And not a surprise, none of the systems that we currently have is made for such a load. That's just too much traffic. If you imagine air traffic controllers adding a factor hundreds or thousands, how would they do it? They could not approve, they could not follow, they could not ensure separation. Long story short, we need to do something. This „we“ actually means Europe or the US or China or actually the world. We need to define a concept that can handle those things. There’s different versions and the European one is called U-space. Space is not meant to be something physical like airspace or a room. It's meant to be a holistic approach. A set – as it's written here – of new services and procedures that is designed to support safe, efficient and secure access to airspace – and the important part – for a large number of drones. What we had before the control centers, public safety, ATM, defense, maritime and public transport – they will have to deal with that. ATM mostly, because there is already planes, there’s already something existing, and that something existing needs to cope with the load. To indicate a bit how much it is you might see down here, this small little drone. From the size of that box, this is the current traffic and forecast says in 2035 this is traffic that they need to handle. Let's see.

In a nutshell, this is the task that we need to right now tackle: change a system that has been grown since like the 1910s. First of all it was paper writing, then it was a bit of IT, now it's mostly IT based and there's a lot of machines in the background dealing with carriers, dealing with general aviation, with gliders, with paragliders, with hobby pilots. And now you need to squeeze in drones. What do you do? Rip it into pieces, completely rebuild or based on experience or try to learn or just figure out what the best approach is. The good news is you don't need to do this alone. So the key part is there's more than that. And in Europe there's a thing called SESAR – Single European Sky is actually an EU initiative that unites a lot of different entities, academia, universities, companies, startups, organizations that already deal with traffic like air navigation service providers. And people from the companies – like me – then can join research projects and those research projects in a holistic approach – with people that invent, with people that build, with people that use and afterwards with people to manage that – together aim to figure out how you could deal with that load that we've seen before. Nowadays, this is called public private partnership. It's been around for quite a while and actually really successful. In Europe currently there's like 3.000 people working on stuff like that – from research, from software developers to project managers to financial guys trying to figure out business cases. You can see on the right side all the different entities to take part here. Huge initiative really bringing things forward.

One challenge from that task: you have a lot of organizations and all those organizations want to do things and in an IT world they need to talk to each other. One example of such a project is the GOF project, Gulf of Finland. Gulf of Finland was one of the U-space projects. All together 19 partners, air navigation service providers, organizations that run air traffic, startups, new companies that build systems just to manage drones, drone operators, airlines for drones, existing ATM technology providers like us, startups that were focused on consulting and a lot of – lot of, lot of – local organizations trying to figure out how such a system could work.

And the goal was really simple – really, really simple: Make everybody aware of what is currently going on. In more simple terms: provide them with an overview. We just would have simple dots indicating planes, drones and maybe other traffic. In such a project it's less paperwork than you would think. So what we actually did is we took it from use cases, so we were trying to figure out before we got the project what would happen in real world. And then we chose a few use cases. You can see them at the bottom – one like I described before: police intervention. Police has a drone in its car, starts it and just want to fly. Typically they are in a hurry. Other use cases: cooperation over the sea with maritime units in trying to rescue people, search and rescue people. Quite a challenging one actually because first you send a drone, then the drone already has hopefully a fix on the person in need and sooner or later helicopter comes and tries to rescue the drone. And then you need to coordinate the ship with drone, with helicopter. There’s a few other use cases as well. A famous one: parcel delivery, Amazon, Google, quite active in this area. I have my doubts, at least in Europe. I am not sure if this really will take off but we will see. In the states it definitely make sense. If you look at their structure, how much time they need to reach the last house in one of these huge areas where a lot of people live it does make sense. In Europe if you look at old cities like Vienna, it will be hard to actually deliver something. But still there is a lot of business in there and a lot of challenges in there because this is huge numbers. parcel deliveries is the large extent of the forecast. Last use case, a typically one, inspection flights, infrastructure, power lines, harbours, we had it before, something that is not so many people, where helicopters are expensive and if you have a decent drone – that can easily do 100 km flights – just helps you in scanning power lines. They have sensors to tell you from a 10 m distance if they need to change one mount, screw or a piece of that power line mast just by flying by. And then they always can decide later to send a helicopter. Lots of different entities, air traffic management, unmanned traffic management, drone operators, coast guards, police, cities they need to improve as well. And you need to make them talk to each other. You need to make their IT systems talk to each other. I'm in one of those departments to do mostly new things. But still if you now go to an existing unit, like a police, for sure they will already have something. And it's like with financial systems, never touch a running system – so most of the systems could be potentially quite old. Some would say „legacy“. I tend to say „proven“. That sounds a bit better and actually it's true – why change it if it works? Doesn't make sense. And on the other hand, you have systems from drones doing really cool stuff using modern technology and then you need to bring both worlds together. And this is what we did in this project.

And how we did it – this is now actually the core part. I will press a button then an animation will start. But first I need to explain a few things. This is all the services that we identified to be present in that project. You have things like the U-space service provider, pure software entity, providing services, flight planning, advanced weather forecasts, data recording, partially even things like sensor analysis. This is the „New Kids on the Block“, as they were called in the project. Actually guys from any IT domain that try to enter aviation. And it makes sense because they have different concepts. If you fly a drone there's a few subtle differences. First of all there's no people on board. And to challenge the existing environment, one could say if I just make sure that the drone doesn't do much damage if things go wrong, I don't need all those very special and very hard to reach requirements that I have on manned deviation. Those companies do that. They have a pure focused, pure software approach they don't have too much hardware, they live in the internet. Whereas other systems, older systems might have closed networks, hardware involved, completely different beasts. Still they don't live alone. So they need to talk to other systems. They need to talk to so called ground control stations. This is the remote control – the one thing that flies the drone, ranges from a notebook with software to something that you might know your RC-car. Bigger ground control stations – especially for professional drone operators – that might be a container. Two or three operators in there flying the drone, operating the sensors, capturing data and acting with the things that they capture. The ground control station is connected to the actual unmanned aerial vehicle – the drone – the aircraft flying. And on the drone, you have systems as well. What else do we have? On the right side we have the general public. Just to mention one challenge – if you are in front of a stadium or if you’re in front of your house and you see a drone approaching you might want to know who that drone is, if it is ok that the drone is there. Difficult topic, but still with the low „entry barrier“ that drones have its not so easy to make people aware that they shouldn’t fly in front of a soccer stadium that is fully booked or that they shouldn’t fly close to an airport. So we need some means for the general air public to interact. Best case those that fly, announced their flights. On top we have authorities – air navigation service providers, those that currently manage traffic and other authorities like police, we have seen them before. I start. That’s one flow that actually tells a drone operator that he should turn around because there is conflicting traffic. And you see it running through without going into details how many services are involved and how many interactions between those services are happening. And what we did in the project is we did not look too much on what is exactly going on in something like the service called „tactical deconfliction“ but we looked at what information we need to exchange so that the entity providing that service can connect to the entity providing this service. Even though this is hardware and this is software. And creating this drawing took us like 4 weeks in a gang of 35 people, lots of discussions, long nights – some working, some having fun – that is part of this whole research project. And the we figured out this is not so complex. What you need to share is things like geofences – walls in the sky where drones shall not pass through. You need to share things like telemetry – complex term for position, where the drone is and where the current speed is. You need to share flight plans, a declaration of intent: „I as a drone operator intend to fly from here to here.“ That’s one use case, we had like 50. And then we tried to identify from these 50 use cases, and from all those 50 arrows, connectors between the different systems, how we would design things. Looking at systems to system interfaces less than the actual software in one respective system. Challenging part, especially because some of the U-space service providers have a lot of investment in their services. So they're not too happy to talk about what's actually going on inside. And after four weeks there was a simple conclusion. Oh shit – what happens if one record is misunderstood or wrong? One position that has the wrong format. Typical mistake: you swap latitude and longitude. From an Austrian position you then end up in Saudi Arabia and that might actually cause a lot of havok. And then we looked at concepts that are around to solve an mitigate that. And we found something that's actually quite simple and around for a while. It's called SWIM: System Wide Information Management. It's an 80 page standard from ICAO, an aviation standardisation community. And in one sentence, it's a simple summary: Managing information means that you need to assure that commonly understood information is provided in high quality to the right people at the right time. Ten years ago, 15 years ago, just in time production, one big change in industry – same in software. Need to know principle, quite famous now with GDPR, keep things simple and make sure that everybody understands. And that everybody understands is actually quite hard to achieve. Remember before, a policeman and an air traffic controller, they are completely different domains. You don't want a policeman to be trained ATC guy and the other way around because they both have their job to focus on. And still they need to agree on my car is here, my drone is here, a plane is here and we make sure that the whole thing doesn't conflict with each other. Creating understanding is a key part in here. How do we do that in the system to systems approach? We talk and then we write. We create a thing called service specification. That's a document. Could be a wiki page, could be a word document, could be whatsoever – just something in writing. And that's something the writing has mostly three things: a data model, interfaces of services to interact with the data model and you need to describe the behaviour. Could be as simple as you need to log in, you need to provide your role and then you can do something. Or it could be a more complex interaction, especially for flight planning – request, aprove, maybe propose something else, then go back to the beginning. But you need to have a clear understanding of what happens if you talk to that interface. And you need to make that in a way so that both systems – let's say this is a ground control station and this is some police system that already exists – map their existing technology, their existing API, web services, SOAP services, file based interfaces, whatever it could be, they map that to the service specification. And if you do that, if you get those two people on one task and make them discuss, they sooner or later understand each other. And they do this because the service specification is technology agnostic. We've all done that. My code is better than your code. My technology is cooler than yours. I'm Java, you're .NET. I am Microsoft, you're Linux. It disappears right now. But still technology wars are something that you absolutely need to avoid if you have big systems like this. And as soon you have documented: „this is a position here“, „this is a position there“ an we both map to one document – a least common denominator if you want – you then can start to implement. Lessons learned from us, you talk for 3 weeks, 4 weeks depends on how many meetings you have and if you get people on board, once you have that documented, writing the actual adaptor, bridge whatever pattern you choose is just a matter of a few hours. And with all the documentation you produced before it's actually quite easy to test and you can make sure that if those domains come together and one record is broken no bad things happen.

A bit more formal. The framework itself is quite complex, so we've got like 80 pages of documentation on how to fill that out but in general means we decouple documentation. We start at the very top – hard to reach – with a service specification, you might be able to read it on the left side. It's high level, technology independent, use something that has no special meaning to any developer group. UML might be a nice candidate. Put requirements in to explain why you need that thing but keep it high level – everybody should be able to understand. That service specification has a logical data model. Same story here, SQL, NoSQL – whatever product you want to choose, it doesn't matter, what you need to make sure is that people understand how that record looks like. Second level: a technical design, now it's technology. That might be REST, GraphQL, SOAP – choose something. As long it maps to the service specification you're fine. And if you have two technical designs, one of them could be the one translating. And how would you do that? Link both technical designs to the same service specification based on the common understanding you've created before. Last one. That's a simple one and it's low access: service instance. You might want your service to be used, actually. So you're just deploy it somewhere and put in information how you can reach that. My famous example iceberg service – as a mariner you might want to know where an iceberg is. Captain of the Titanic. Titanic could have used that service. So in the service specification you define it's got a size, it might have a risk associated, it has a position and a drift. That's it. That's a simple iceberg but you don't need more. Technical design. Simplest possible case back then. You do have a voice communication system. You have somewhere a phone and if you're a captain you can talk to that phone and say “hey I'm the Titanic, I'm here, is there something that is dangerous to me?” And then the other one could say no. And then I say „okay, please let me know.“ And the other one says „OK give me your phone number or you're radio ID and I will call you back.“ Request response, publish subscribe in very simple terms using voice communication. Or you have modern state-of-the-art something Node.js-based, sockets to send updates on-boarding integration with ground control stations would be the same concept – you ask a question: „Am I OK?“ – „Yes/No“ – „Let me know, here's my contact.“ Last one, it could as well be an image, something like a webmap service to be read by a human. It's always the same concept and it's described in the technical design. What it needs is especially the data model. I’ve put up one thing. Yes, UML is alive. This is a position. And it's not just latitude, longitude, this is the least common denominator that came out of all those discussions with public safety units, ATM, U-space service providers, drone operators. Why is it so much? You have a few interesting things that you typically don't need in everyday life but if you manage traffic they could be interesting. Like here: What is it? Is it a bird, is it a plane, is it a drone? How do I approach it to handle it? Most of it is optional. So the mandatory stuff is just on the left upper side, one or more altitude, you need to know that, how high and one position to identify that aircraft or that moving object. Drawing something like this: again 4 weeks. But definitely worth it, we use it now in minor changes since five years. And it works pretty well because it can take it to anybody. You can take it to top level management – they are able to read stuff like that – you could take it to a developers – and it's easy to implement as soon as you have a UML, coming from a tool there's not so many mistakes that are made in the modeling part, implementing is actually a walk in the park. And everybody from test up to C-level has the same understanding of what actually is happening in the system. Still it's boring. So why do we do this? Remember the project before. Achieving that all people see the same thing. You can see that here this is a shot from the drone going from Estonia to Finland in that case with an international flight. That's a screen cap in Google showing how the drones spiraled up. So that was connected. On the left side you have a typically ATC display. This is something the tower people use, the air traffic management to separate traffic. You can even see some birds here. This is the way to Russia. Closed airspace, here we put up the drone corridor where the drone was flying and on top you got mostly web based systems showing the same situation overview with your target that you UTM operators. They have different focus, they need different things on their screen. Five-ish million project to achieve the same display on four different screens. Sounds simple, takes a while and only works if you manage information.

Which brings me to one of the last slides: Checkout.

  • Introductions; done. Tom Lutz, you'll find me.
  • Context; software for real life. Control centers, onboard systems in drones, onboard systems in trains, everything that happens in real life we focus on safety.
  • Task; add to those systems, improve them, add new domains, add new concepts, bring domains together.
  • Challenge; one record could stop everything. I did not put it on the slides but it's easy to say it: if something goes wrong this has consequences – we do ATM stuff so if there's a bug in the software that makes this display stop, people don't die right away but it's definitely a severe, very critical situation because the outcome, they can’t do their job anymore. Most of our systems especially those in the really safetycritical domains, they have a lot of nines behind the comma. They have a lot of and very intense test procedures because it actually does matter what we do and how we code. And this is why concepts like these are so important. If you need to act you need to be fast, you need to understand.
  • Approach; SWIM – looking at long term cycles. On the last slide, it's easy to remember – things like that, you can keep them. The position of a moving object, that's a real life object and that position would work for a wagon in the middle ages or it would work for a ship from the egyptians – the concept of moving objects moves rather slow – sometimes they move to the third dimension, who knows what will happen in 100 years, maybe we get faster, warp, things that we know from Enterprise – but the concept itself is quite stable. Technology changes every two or three years. When I started, we had to explain why we want to use XML and not a standard plaintext file. Then we moved to CSV, back to plaintext file, JSON, SOAP, REST interfaces – what is cooler, what is better? It's just technology, it doesn't matter. What actually matters is the concept.

And the concept mostly deals with information. This is my three-sentence summary: Managing information – especially the flow – is super important in systems like ours. As important as making everybody understand, technical people, management, people that run it – you want a police officer to know what that system actually does. That's quite easy if you have drawings like these. The last one – decouple documentation. You're can keep them at separate speed. You can have new services, based on the existing concept, just in a new technology to see if it works and if you can integrate it.

This is what we do. If you're interested – talk to us! How do you talk to us? You find us in the internet, obviously: „Frequentis".

There is a talent network there, there's quite a lot of job opportunities there. If you want to look around first, of course we are on the typical channels.

So much for today, if you're interested, get in touch! I say thank you for your time and I hope you enjoyed. Bye!