Blue Shed - a Point Cloud test in photogrammetry

In this test I set out to produce an inexpensive sample point cloud for use in the Unreal Engine (UE) game engine. Many new versions of UE contain a LiDAR plugin that allows the direct import of point cloud data into the UE project, and the 4.26.0 version I used was no exception. Note, Unity also has the ability to import point clouds.

Using software, point clouds and panoramic views can be created from a sequence fo images of an object or scene. Many pieces of software can do this ranging from paid commercial products to free opensource projects. A 2008 Microsoft project even had a web service and later a downloadable program to stitch together multiple images into a panorama, it was known as PhotoSynth.  And it was featured in an episode of CSI at the time.. Regard3D is another such program, as is AgiSoft Metashape. A 30 day trial version of the Agisoft product was used for this test.

The process to create a point cloud at a high level

The sample video was shot on an Apple iPhone 11 Pro Max at 4k resolution and 60 frames per second. The high resolution and fast frame rate are required to produce clean crisp images in the next step. The video from the iPhone was clear and crisp so no video post production touch ups were applied in my test, but you may want to review the video and adjust colour or sharpness.

In order to create the image sequence OpenShot video editor was used. Other video editors can produce image sequences. In OpenShot, a new video output profile was create to produce a 2 fps 4k video. This output profile was then utilized with the output image sequence option. The process created 186 high resolution 4k pictures from the sample video.

Sample from image sequence

Sample sequence image, TestTwo-00010-min.png

Agisoft Metashape was used to create the point cloud. It does this by aligning and depth sampling the 186 photos. Processing the images to point cloud and various models along with the learning curve was relatively quick and finished in under 2 hours. But I have a lot more to learn about the tool as I’m sure the output can be improved with a better understanding of the tool. During this stage, the Metashpae options were generally set to “medium” settings and .laz, .pts, .obj, and .fbx files output. The .pts and .laz are the point cloud variants of the model while the .obj and .fbx are 3D models. The 3D models are skinned with a coloured image.

Screenshot from Metashape

Screenshot from Agisoft Metashape with point cloud processed.

The Metashape "report" option is interesting in that it summarizes the current project and includes visualizations for height map data, and estimated camera locations.

The pdf report is avaialbe at this link.

The .obj and .fbx files were viewed in Blender and viewed fine. 3D objects created in this fashion are typically large and contain lots of surfaces or points, these were no exception.

Sample render from Blender

Sample render of the .fbx model in Blender

In the Unreal Engine system, a new project was started, the LiDar plugin enabled for the project, and the .laz point cloud was imported. The point cloud needed to be scaled to 0,254 of the imported size to compensate for mismatched scale units. The project was then configured for an Android/Oculus Quest build and packaged. The application package was loaded to a Quest 2. It was now possible to view and walk around the neighbourhood blue shed.

Work visualizing the point cloud in UE is on-going.

Screenshot from Unreal Engine

A screenshot from Unreal Engine showing the .laz point cloud.

The fbx version of the model was also uploaded to Skecthfab as a test for visualizing in a web browser, The sample link is Blue Shed - 3D model by scbenoit (@scbenoit) [aaef432] - Sketchfab

I'm looking forward to experimenting with LiDar point data from both the early 2020 iPad Pro, and the newest iPhone 12 Pro in comparison to this video based approach.

In conclusion, it was relatively straight forward to get a usable point cloud, and a 3D model, from a video of a subject. However, more time with the tools would produce a better result.

Software used:

OpenShot video editor v2.4.3, OpenShot Video Editor | Download
Agisoft MetaShape v1.6.5 (30 day trial) , Installer
Blender v2.83.0, Download —
Unreal Engine 4.26.0 , EpicGames Unreal Engine

Hardware used:

Computer: AMD Rysen 1950x Threadripper based, Windows 10 computer
16 cores (32 threads) wiith 32G of ram, NVME loacal disks and RTX2070 GPU

Video camera: Apple iPhone 11 Pro Max