Everything is being scanned. From avocados to Amsterdam, from your living room to the local park, it has become easier than ever to take 3D ‘snapshots’ of physical things. This has profound implications for the next major shift in computing: to one in which spatial awareness, machine learning and new devices combine to radically change, well, everything. And yet this shift is nearly invisible to those who don’t work in simulated reality ( Apple’s term for XR/AR/VR/AcronymX). It can even get a bit confusing inside the industry, with so many terms floating around. But Matt Miesnieks makes an important point: Doug Thompson #AR@Dusanwriter · Aug 29, 2020 If a scan uses LiDAR is it still #photogrammetry ? Matt Miesnieks@mattmiesnieks What if it uses 6dof camera pose and/or processed in real time? Or uses a neural network? It’s all the same thing. 3D reconstruction just diff quality, speed, cost 11:10 AM - Aug 29, 2020 See Matt Miesnieks's other Tweets There are dozens of ways to capture reality. And they’re arriving at different resolutions. LiDAR, Cameras and Scans for Everyone Some people like to see photos of food, fashion or cute animals in their social feeds. Myself? I like looking at the latest scans. Call me geeky, but I find this way more sexy than a photo of the latest loaf of bread someone baked: LokiEliot@Demonkid New Photogrammetry of my shoe, Dense Cloud shows bottom tred worked this time. 11:13 AM - Aug 23, 2020 See LokiEliot's other Tweets Now, this isn’t that much different, really, than taking a selfie in Snapchat and adding a face filter. Your camera is being used to detect the dimensions of what it sees. Snapchat is simply taking a ‘scan’ of your face, detecting its contours, and the ‘filter’ then attaches effects and elements to this unseen scan. But with photogrammetry, you can export this scan. This, for example, isn’t “real”: it’s a scan that uses RealityCapture software to export and use the asset that was created: miguelbandera #photogrammetry@miguelbandera #photogrammetry in @RealityCapture_ #japan 9:27 AM - Sep 5, 2020 See miguelbandera #photogrammetry's other Tweets The above is a walk-through of an entirely digital ‘twin’ based on interpreting a camera “walk-through’ of a physical space: Until recently. the camera on your phone was the most powerful way to scan 3D objects, faces and spaces. “Behind” the camera was a lot of computing power. Being able to interpret video and convert it into a 3D digital object using software was a big lift. Over time, the software became more accessible. But if you wanted, for example, to scan a whole city, you needed a Google car driving around with a lot of equipment on the roof. Apple Brings LiDAR to the Masses When Apple added LiDAR to the iPad Pro , it maybe didn’t give you the ability to easily scan a whole city (there’s a limited range to what the LiDAR can see), but it HAS allowed developers to do some pretty detailed scans of nearby rooms and objects. Combined with advances to ARKit, the software that developers use to create AR experiences for iOS, you can use the LiDAR to rapidly detect the contours of the world around you: Tim Field@nobbis iOS 14 continues to improve LiDAR quality. Beta 5 adds new "smoothedSceneDepth" API. 3:15 PM - Aug 18, 2020 This will allow for AR experiences that are richer and more ‘realistic’. But combined with machine learning, it will also power a new generation of apps in other categories. I’m expecting, for example, that with the launch of iOS 14, that there will be an explosion of fitness apps which give you a ‘virtual coach’: Large Scale Scans and Machine Learning So, the ability to scan nearby objects and environments is becoming more and more accessible because of advances in how well your phone’s camera can “see”, as well as the addition of LiDAR to devices like the iPad (and the upcoming iPhone). As we shift our view to a larger scale, the kinds of scans that were once reserved to Google Street View cars or owners of highly specialized (and expensive) equipment are now possible at a lower price. This has shifted large-scale scanning from being either a niche business or available only to the big players like Apple and Google. Once a leader in “professional” LiDAR scans, Velodyne now has competition, with companies in the field raising millions of dollars in funding to support hundreds of customers. We can stitch together multiple scans to create high-resolution digital twins of large spaces that contain a lot of objects: René Schulte@rschu New AI model for large-scale 3D reconstruction. Convolutional Occupancy Networks presented at #ECCV2020 by @songyoupeng works with implicit neural representations even on large structures and great detail. 10:52 AM - Aug 24, 2020 See René Schulte's other Tweets Pixel8 , meanwhile, is creating a “common ground truth” which aligns different inputs to create a digital twin of large-scale spaces. They took a group of 25 volunteers with phones and were able to combine the data
