3D mapping LiDAR scanner for fast incident response

Facilitator:

Brad Glassington and Leonie Ball from SafeWork New South Wales. And they're going to be discussing LiDAR technology and the LiDAR scanner, its benefits and some of its limitations. They’ll talk about the use of LiDAR at incident first response operations and some case studies and show some example images. So please take it away, Brad.

Brad:

Thank you, Andrew. And welcome everybody. So, yes, my name is Brad Glassington, I'm an Assistant State Inspector with SafeWork New South Wales. And I'm attached to the Investigations and Emergency Response Directorate. And with me today is Leonie Ball who is also an Assistant State Inspector attached to the Investigations and Emergency Response Directorate. I'll be leading the presentation today and I'll have Leonie here for moral support in case I get any curly questions.

Neither of us are experts in this subject but Leonie has a fair bit more experience in using the technology than I do. So I'm sure between the two of us we can cover off on any question that you guys might have at the end of the presentation. So bear with me as I share my presentation with you guys.

OK. I trust that everyone can see that, my presentation there. While my primary role is as an investigations inspector, one of my ancillary roles is to manage a portfolio of work that’s focused on the implementation of technology solutions within our directorate. Our LiDAR solution was introduced to the Directorate prior to the creation of this technology portfolio. However, it was the work that was done in implementing the LiDAR solution that made us realise the importance of having a similar implementation process for all the technology solutions that we might introduce in the future.

The key purpose of the Investigations and Emergency Response Directorate’s technology portfolio is to identify technology solutions that increase the Directorate’s efficiency, improve the quality of our work or increase our capabilities or all of the above. Then once any given technology solution has been identified, it’s the responsibility of the technology portfolio to manage the process from conception through implementation to an operational readiness stage. We consider that our LiDAR solution is in the operational readiness stage.

So what are we going to cover today? We’re going to have a quick look at what actually LiDAR is. I'm sure there's a lot of you guys that have – some may not have heard the term and some have heard the term but don’t know what it is. But we’ll have a quick explanation of what LiDAR actually is. We’ll look at the LiDAR scanner itself and then we’ll go into how that scanner actually works. We’ll have a look at the scan analysis software that we use once we’ve taken the scan and then we need to take it through an analysis process.

And then we’ll look at some examples and case studies of where we’ve deployed and applied the LiDAR technology at some of our responses to serious and fatal incidents. Then we’ll have a look at the LiDAR operations at SafeWork New South Wales and how we’ve implemented the processes. And, finally, we’ll close out by looking at some of the specific deliverables that LiDAR provides us. We’ll also look at the challenges that we’ve identified between now and the time that we’ve sort of started to use the LiDAR solution.

And then we’ll close out with looking at some of the benefits of using the LiDAR technology. And then we’ll go into some questions to the extent that you guys have any questions for us.

LiDAR technology has been around since the 1960s when laser scanners were mounted to planes. But it wasn’t until the late 1980s, with the introduction of commercially viable GPS systems, that LiDAR data became a useful tool for providing accurate geospatial measurements. LiDAR is a remote sensing technology which uses the pulse from a laser to collect measurements which can then be used to create 3D models and maps of objects and environments. LiDAR is an acronym and it stands for light detection and ranging.

LiDAR works in a similar way to radar and sonar yet it uses light waves from a laser instead of the radio waves and sound waves that radar and sonar uses. A LiDAR system calculates how long it takes for the light to hit an object or a surface and reflect back to the scanner. And depending on the sensor used, LiDAR units can fire hundreds of thousands of pulses per second. Each of these measurements or returns can then be processed into a 3D visualisation known as a point cloud. And we’ll show you some examples of those point clouds in the presentation today.

LiDAR systems are commonly used for surveying tasks because of their ability to collect three-dimensional measurements. LiDAR scanning systems are popular for surveying the built environment such as buildings, road networks and railways. As well as creating digital terrain and elevation models for specific landscapes. These are all the applications that can benefit us as inspectors when we’re looking at firstly, preserving, recording and then recreating incident scenes.

So there are many different mobile LiDAR solutions available, however, the scanner device that we use is a handheld device. It can also be mounted on a pole for ease of use. The device is relatively lightweight, however, on long scans it can get heavy due to the requirement to hold the device above head height. And the requirement to hold the device above head height is so that your body doesn’t disrupt the 360 degree scan. And when it’s down below head height, as you can imagine, your body would create a shadow within that 360 degree scan.

The cylindrical section of the scanner shown in the image that you can see here is the actual scan. So the cylindrical section is the actual scanner itself. The square housing below it contains what is effectively a computer that runs a Linux Ubuntu distribution as its operating system. Input, output devices can be attached using the USB ports around the computer housing. And a monitor can be attached either physically or using a wireless dongle, depending on the monitor that you're using.

We have both options, we have monitors that we can connect through the USB ports or we have a monitor that we can look at wirelessly. I think we’re using an iPad for that option. The unit itself is powered by a 5 volt battery that’s connected to the scanner during use. So the scanner itself doesn’t have its own battery connected, it’s an external battery. But it’s a fairly lightweight and small battery that generally fits in a large pocket. We’ve got our safety vests that we use that have the front pockets and it’s a perfect fit for that.

When operating the scanner with a monitor, a monitor attached, you can see the scan in real time as you're conducting the scan. However – and this is just a personal choice – I've found that it’s simpler to conduct the scan without an attached monitor. If you use some simple techniques when conducting the scan you can ensure that you end up with a quality scan. And, for me, sometimes the monitor can be a distraction and get in the way when you're conducting the scan. But that’s a personal choice. I know that we have some of our users that do like to have the monitor connected at the same time.

Having said that, it’s easy to end up with a scan that’s suboptimal. But if you have a good understanding of the limitations of the scanner and basic understanding of how the scanner operates, scanning issues can mostly be avoided.

So how does the scanner work? The scanner’s a mobile scanner. So it answers two fundamental questions: where am I, what's my location and what is around me, what does my environment look like? The solution to these is known as simultaneous localisation and mapping or SLAM for short. SLAM is the basis of a lot of the work that is going into autonomous vehicles and robotics and LiDAR is one solution to the SLAM problem in those areas.

Our needs, however, are, thankfully, somewhat less than the needs of an autonomous vehicle operating in an unknown location in an unknown environment. Our system uses GPS to identify the location and the LiDAR scanner to map the environment. But it goes a step further in that once some aspect of the environment is scanned and mapped, by feeding back the increasingly map environment into the real time processing algorithm. The location of the scanner is refined even further than the fundamental accuracy of the GPS.

And this allows a scanner to have very accurate location data as well as avoiding issues like GPS drift. The accurate mapping range is approximately 100 metres from the scanner with an accuracy which is measured in millimetres out at the 100 metre range. Obviously accuracy increases for points closest to the scanner.

So how are these scans analysed? There's many software programmes available for the analysis of point clouds produced by the LiDAR scanners. The one that’s recommended by our LiDAR suppliers and the one that we use is an open source software package called CloudCompare. You can see an example screen shot of the software on the screen. It’s beyond the scope of this presentation to do a deep dive into the specific capabilities of the software. But CloudCompare is a 3D point cloud processing software. It’s a generic point cloud processing software package that can perform several advanced algorithms. We just barely scrape the surface of its capability.

However, one of the key advantages of doing the LiDAR scans is that when our analysis needs go beyond our limited capabilities we have the option of bringing in third-party experts to work on our scans.

So let’s have a look at some examples of a LiDAR scan. We’ll be showing you some scans that we’ve taken in actual incident responses. As I've said earlier, the LiDAR scan produces a point cloud which is simply that, a cloud of points in space. However, each individual point in the cloud has specific location data that can be referenced in the point cloud allowing us to build the 3D visualisation. And, more importantly, for us, to take accurate point to point measurements.

You will note also that the image on the screen is not a photo realistic image. And even to get it to this point in terms of readability or viewability, if that’s a word, we’ve had to run it through the software analysis process. This particular scan was taken by walking or tracking around the inside of the factory unit where the incident occurred. I won't go into the specifics of this particular incident as the matter is still within the courts. But one of the key benefits of obtaining a LiDAR scan is the ability to obtain multiple viewpoints from one scan. This scan was obtained, as I said, by tracking the scanner inside the factory. Some of the outside at the front is captured as you can see from the pallets at the front door.

But once we have the scan or the 3D point cloud you can look at the scene from multiple angles. From the same scan you can get a plan view or a layout view of the factory and from there you can obtain meaningful measurements. And you can see here the image has been annotated with measurement information. I'll go into a little bit more detail on the measurement data in the next few slides but I just wanted to show you, in this example, the ability to obtain the different viewpoints from one scan.

Looking at another example. This particular incident involved a worker coming into contact with overhead high voltage 11kV powerlines sustaining serious injuries that required the amputation of both hands. In this scan we were able to get a full scan of the incident site which included the scaffolding the worker was standing on when the incident occurred. And you can see clearly the proximity of the powerlines in this scan. As you can imagine, in these types of incidents, it’s inherently difficult to get accurate measurements of the distance between structures and powerlines.

Looking at the same scan from a different angle, in this case, top down view, we can start to get a more detailed sense of the distance between the powerlines and the scaffolding structure. And you can see that the powerlines in this image are clearly displayed running in front of the building at the bottom of the image and running besides the building to the right of the image.

Once again, looking at the same scan, once we’ve established the correct angle from which a meaningful measurement can be taken we can then use the CloudCompare software to take a point to point measurement. And, as you can see in this case, we’ve annotated the image with distance markers and distance labels. I'm not sure if you can see the actual numbers on the screen but each of the three labels annotated in this image is a distance label.

And each distance label shows one, the location of each point, specifically the X, Y and Z coordinates or point cloud coordinates for each point. And two, the calculated distance between the two points. In this case, it’s shown in metres to six decimal places. Looking at another example, this scan was taken at an incident involving a worker falling from a height through an unprotected void on a residential construction site. The worker sustained fatal injuries.

Note, in this scan, the surrounding environment that the scan has picked up. I noted earlier that the scan has a range of 100 metres. The inspector undertaking this scan has conducted an outside scan as well as an inside scan. This is an important point to make about the techniques and the methods we can use to put a scan together. The CloudCompare software that we are using has the ability to merge two overlapping scans.

So, in this case, it’s possible for the attending inspector to take an outside scan of the front and surroundings of the building site and a separate scan of the inside of the building site. But the key is that there needs to be some overlap to allow the software to accurately merge the two scans. Often when we’re at an incident site it can be scanned using one single scan moving from outside to inside. But it’s important to know that multiple scans can be merged if you're confronted with a situation that requires it.

This is a view from above the site looking down into the top of the site and it’s the same scan. And this is a closer more detailed look at the void looking down from the top of the site. Unfortunately I don’t have all the details relating to this particular incident so I can't accurately indicate the location of the void. But I wanted to use this example to discuss the different methods of taking a scan specifically relating to the ability to merge two or more scans in the software tool that we use.

And just a quick final example, this incident involved a tailgate failure on a tip truck carrying a full load of hot bitumen that unexpectedly released onto a worker who was positioned at the rear of the truck. In this case, the LiDAR scan allowed us to capture the location of the bitumen truck on the site and specifically get accurate measurements of the angle that the tipper body was raised to at the time of the incident.

So now I'd like to show you a quick look at how we’ve implemented LiDAR operations at SafeWork New South Wales. At this point in time, the Investigations and Emergency Response Directorate are the owners of the LiDAR technology within SafeWork New South Wales. However, our engineering team have recently purchased a unit and they're also looking at building a capability in this area. And there's some obvious applications for LiDAR in that space.

Within the Investigations and Emergency Response Directorate the LiDAR is operated as part of our first response function. Now, the reality is that we are investigations inspectors, we are not technology experts. So it was recognised early that trying to ensure that every inspector within the directorate was trained and able to maintain the skills and knowledge to effectively operate the LiDAR scanners was simply not a practical proposition.

So the Directorate trained a group of interested users, these LiDAR users have been trained in the techniques for conducting LiDAR scans and the analysis of the scans using the CloudCompare software for use in our Briefs of Evidence. And when a LiDAR scan is required we draw on these users to conduct the scans ancillary to the regular first response functions and activities conducted by our investigations inspectors.

So what are some of the LiDAR scanning deliverables? Mostly, to date, we are presenting the LiDAR scans as annotated images with accurate measurements. We can present 2D plans and sections but we could also do 3D fly throughs or walk through simulations, videos of 3D walk throughs using the CloudCompare software.

We can build 3D models and digital scene reconstruction and you’ve already seen that we can look at the scans from different views or arbitrary angles. And this is particularly useful when we’re looking at verifying eyewitness evidence. We actually haven't done it yet but there's also the capability of building 3D physical models using 3D printing technology. We can also, I guess, produce videos that combines both photography, point cloud data and the 3D models that come out of that. So these are future deliverables that we haven't had a need for to date.

But, by using the LiDAR technology, we’ve basically positioned ourselves to take advantage of these deliverables in the event that they become requirements in the future as our investigations progress. So what are some of the challenges that we’ve found in terms of the implementation of LiDAR technology at SafeWork New South Wales, particularly the Investigations and Emergency Response Directorate?

One of the first ones that we came up against is the management of the scanning units. The cost of the scanners mean that we have a limited number of scanners. We have two scanners within the Investigations and Emergency Response Directorate. Within SafeWork New South Wales we have investigations inspectors responding to incidents from five SafeWork New South Wales offices. And they're scattered throughout the state.

The challenge is ensuring that a scanner is available in the right location at the time that it’s needed. This means that often the scanners need to be moved between opposites. So we’ve set up a system of self-reporting of scanner movements and periodic location audits to ensure the location of the scanner units are adequately and transparently tracked. And this helps us when we get an incident that we need to use a scanner at. One of the other challenges is, I guess, the management and the maintenance of the knowledge and skills required to operate the scanners.

There is a technique involved in getting good scans and inspectors in the LiDAR user group are required to maintain the skills of using these techniques. Analysing the scans also requires a basic knowledge of the CloudCompare software. These skills and knowledge are perishable. So this poses a significant challenge to our LiDAR users as there is often a period of time between each usage for our LiDAR users.

This is alleviated by ongoing training. We provide both in house training and training via our third party local suppliers who provide training in both the scanning techniques and the use of the CloudCompare software packages. But we also encourage our users to get the LiDAR equipment out and play around with it and practice using it in situations without the sort of pressure of deploying the scanners during a first response incident or a response to an incident.

One of the other challenges is the idea of how do we represent the deliverables at court? Due to the nature of our investigations and potential prosecutions there's normally a long time between the LiDAR scan and a presentation at court. To date, in our jurisdiction, the use of this technology has not been tested at court. This is something that we are keeping an eye on as this has a potential to impact on our processes and our procedures and use of the LiDAR.

And I'm not sure if LiDAR has been used and tested in other jurisdictions. But, in the event that there is ever a question as to the accuracy and validity of the measurements taken from LiDAR scans, we can call on our third party experts to provide that evidence if required. However, to date, we haven't had to.

So what are some of the benefits we’ve found from using the LiDAR technology? The big one for us is accurate measurements. But, more importantly, accurate measurements that are otherwise impossible or difficult to obtain using other traditional methods. But there's some other generic benefits and these include the fact that a LiDAR scan records virtually a full dome 360 degree point of view. That the scans are free form bias capturing all aspects of the scene. And this is important because details may only become relevant as an investigation progresses. The integrity of the scene remains as it was when the scan was taken and LiDAR scans can be revisited at any time, at any occasion.

So this brings me to the conclusion of my presentation. Thank you for your interest and I think now we’ll be able to take some questions.

Facilitator:

Terrific, Brad. There's been lots of questions come through. Questions around weight which I think you have answered already that they're quite portable and depends on the particular unit. Expense, they are prohibitively expensive. Are they an expensive bit of kit?

Brad:

Yes, they are very expensive. I don’t have an exact cost but it’s up around the $80,000 mark, I'm pretty sure that’s where it’s at. In terms of weight, I'm not sure if Leonie knows the exact weight. It’s relatively lightweight and you can probably see – I'm not sure if the image is still up but you can see a blurred-out view of Leonie holding the scanner above head height. The thing about the weight of the scanner is really the position that you need to hold it at so –

Facilitator:

Yep. And if you're commenting, Leonie, you're on mute there if there's a –

Leonie:

Yeah. No, I wasn’t commenting.

Facilitator: OK.

Leonie:

But Brad’s right. It actually isn't especially heavy and I feel like it’s around a three or four kilo mark. But exactly what Brad said, it’s more about the fact that you need to hold it up over your head that sort of prevents you from holding it there for very long without it becoming quite tiring.

Brad:

And we’ve actually got some – I mentioned it in the presentation. One of the – we’ve looked at different solutions and there's harnesses that you can buy. But I think the solution that we found we fell down on was mounting it onto, I guess, a single pole tripod type thing which is not really – a single pole. I guess you can't really call it a tripod. And that sort of alleviates the manual handling issues that you can have when you're doing a really long scan and holding it above your head.

Facilitator:

I'm guessing somewhat like a surveyor’s pole or the like?

Brad:

Yeah. I think so. You see the surveyors out on the side of the road with their tripods. I'm not sure, but I've been told that that’s actually they're setting up a LiDAR system. So this particular unit that we've got is effectively something very similar but they're sort of mounted onto a computer and allowed us to take a mobile scan.

Facilitator:

There's a few comments coming through about the new iPhones which come with a LiDAR feature.

Brad: Yeah.

Facilitator:

And around whether you're thinking that that will shift the normalisation and whether you see that as a potential substitute at some point.

Brad:

Yeah. Look, it’s an interesting comment. Something that I've thought about myself. I think, at this stage, we’re in the very early stages of it being a replacement. A lot of the stuff – first of all, I think just by the very nature of the iPhones and the iPads, it’s very much a one-directional scan. I'm not sure of the resolution of those scans either. And then you’ve got the – a lot of the work that’s happening in these units, which I'm sure the iPhones and the iPads are capable of, but there's a lot of algorithmic real time processing happening as well.

So, look, it’s certainly an interesting space to watch in the consumer market to see whether or not it comes in. But I'm pretty confident they're not at that point.

Facilitator:

Yep. And there's also quite a few questions. You showed some of the images and there were questions around the colours and what they represent.

Brad:

So that’s all part of the actual analysis process and that’s very much a – you would have seen that there were some images that had different colours and there were other images that had other colours. It’s very much a user preference point of view and just trying to get the best readability out of it. Because what the software allows you to do is change the colour range. Now, sometimes the colours represent – you can change the colours to represent elevation. So if, at a certain colour, you know it’s a low elevation and then it changes colour as it goes up the height scale. But you can change the colour to represent other factors.

I'm not sure – Leonie, I don’t want to flick past, but what are some other ways that you might use the colours?

Leonie:

So I think sometimes – and you're right Brad, in that it represents, I suppose, heights and stuff like that and it will depend on what you allocate when you're doing the processing. But, say, for example, for that factory one, so if I wanted to do a measurement of height, say, or maybe more width, you make sure that the point that you are selecting is in the same colour scale. And that will help with getting a more accurate reading as opposed to, say, if I was to try and measure something from two different colours. I wouldn’t be on the same plane I think is what I'm trying to say. So it can help with that when you're trying to select points to conduct measurements.

Facilitator:

And there's quite a few questions around practical use, can they be used in all weather conditions?

Brad:

That’s actually a good question. And look, I've only ever used it in good conditions and that’s just a luck of the draw from my part. I'm not sure if any of our user group has a difference on that. I suspect that you wouldn’t be using it in torrential rain conditions. It’s effectively a computer and it doesn’t look overly waterproof and I would argue that it’s not. And I guess it then becomes a manager of managing time in terms of – managing, I guess, the incident scene itself in terms of preserving or throwing a non-disturbance on it and coming back at a time when you can safely do the scan.

Facilitator:

Yeah. There's comment here around could you strap it to a helmet while you walk around?

Brad:

Look, interesting idea.

Yeah. Look, one of the techniques in using the scan is being able to tilt it. And what happens when you tilt the scan, because it does have a – I think I'm on the screen now – but it sort of does have an arc if that makes sense. So you'll see that the top of the scanner itself, while it does 360 degrees, that arc does finish. So one of the things, traps I guess, when you're conducting a scan, is that you assume that it’s scanning outside of that arc when it isn't. And sometimes, in the early days, you'd come back from a scan and you'd go, I only got halfway up that building and I didn’t get the whole building sort of thing.

So the technique is around once you’ve got a datum scan you can then do that sort of thing. So, to answer the question, I like the thinking but you might have to do some head wobbles.

Facilitator:

Great. I just missed my question which I had lined up. And so there was one that was around bird’s eye view or drone. And I think you might have covered this a little bit about the ability to merge two scans together. So maybe you'd like to comment on drone use and handheld use, whether you’ve combined or –

Brad:

Yeah. Look, at this stage, we haven't but I know there are solutions out in the world where LiDAR units have been mounted onto drones. We’re in the very embryonic stages of looking at drone technology ourselves so that’s a watch this space for us. But I do know that LiDAR scanners have been mounted onto drones out in the world. And you can get some – and that just adds another layer to your ability to get really good three-dimensional images.

Facilitator:

Yep. OK. And so now coming down to its use in prosecutions and court, has it been used in prosecutions and testing in court much in the work health safety? And are you aware if it’s been used by police or other jurisdictions much?

Brad:

I guess the short answer to both of those questions is no but in terms of whether we’ve used it and whether I'm aware of it being used in other jurisdictions. However, from our experience and that was one of the things I checked with our user group and, to date, it hasn’t been tested in court as far as I'm aware in our jurisdiction. It’s being used in matters but those matters ultimately, for various reasons, haven't been defended I guess. So it hasn’t had to undergo the scrutiny of the judicial process.

Now, it may be that the scanner might have had a positive impact or, depending on which way you're looking at it, in terms of allowing the parties to come to a decision around whether or not they're going to defend the matter or not. But, to date, we haven't actually – it hasn’t been tested in our jurisdiction. And, as I said, I'm not aware of whether or not it has been beyond our jurisdiction or not. It’s probably something that we want to do some research on.

Facilitator:

Terrific. Thanks. And we’ve probably just got one final question to wrap up with. And that’s around where do you see the future of LiDAR going and its use as a regulatory tool or do you see you guys scaling out or scaling up?

Brad:

Look, at this stage – and I probably mentioned it in the last couple of slides in the context of we’re just scraping the surface of what LiDAR can produce. So, at the moment, our key deliverable is a printed out image on a screen that’s annotated with accurate measurements. But, as I said, even now, we’ve got the capability to produce three-dimensional fly throughs of an incident scene. We can produce walk throughs. So I guess, for us, it just becomes a matter of we need to educate our investigations inspectors as to what we can produce out of these so that, when the needs arise, we know that they’ve got a go to if that makes sense. So I guess, yeah, watch this space.

As I said, we’re very conscious that we’re just scraping the surface of this technology and it’s just a matter of looking at the – and look, the software and the scans. The scans are one thing but once the software gets – that will improve over time as well. So, as I said in the presentation, the images aren’t photorealistic but I know that there's technology coming down the pipeline that allows the scans to become a lot more photorealistic.