AR Measurement Essentials on iOS

Published in

Hipo

5 min readAug 31, 2020

Bauer Fit app allows measuring and automated equipment suggestions

We recently worked with the amazing folks at Bauer for a really fun and challenging app called Bauer Fit. One of the requirements was to implement a screen on which users can measure their hands or elbows. In this article, I will be sharing how I implemented the measurement flow and solved some issues I encountered along the way, hoping that can help someone out there. Also, I will explain things on a sample application which I put together for this post.

Result Screen

Screen Layout

The layout of the screen is very straightforward as you can see here. There are three buttons, undo, clear, add, and an image view for center dot.

Sample App’s Functionality

The functionality of app is pretty straightforward too; Users will be able to measure things just by using three buttons.

Add button functionality:

When tapped, a dot node is added on screen, if necessary.

Clear button funtionality:

Clears everything on screen, which added in measurement session.

Undo button funtionality:

Undos last operation, if necessary.

Adding dot nodes

A dot should be put and kept on screen each time add button is tapped. We can achieve this by adding an SCNNode on any ARCNSceneView.

I used the method; func hitTest(_ point: CGPoint, types: ARHitTestResult.ResultType) -> [ARHitTestResult]to get real world coordinates based on center of screen.

The hitTest method on SceneKit view returns results for real-world objects or surfaces detected based on 2D coordinate(CGPoint) we deliver.

As result I ended up the method below as ARSCNView extension;

func realWorldPosition(for point: CGPoint) -> SCNVector3? {
    let result = self.hitTest(
        point, 
        types: [ .featurePoint, .existingPlaneUsingExtent ]
    )
    
    guard let hitResult = result.last else {
        return nil
    }
    
    let hitTransform = SCNMatrix4(hitResult.worldTransform)    let hitVector = SCNVector3Make(
        hitTransform.m41,
        hitTransform.m42,
        hitTransform.m43
    )    return hitVector
}

The method simply takes a CGPoint as a parameter and converts the hitTest’s result into a SCNVector which is needed to determine dot node(SCNNode) positions. Also, I was only interested in the first result because the results are sorted from nearest to farthest from the camera.

Note that the method is called with the ARSCNView’s center point, which corresponds to the dot image’s position according to the screen’s UI.realWorldPosition(for: view.center)

Drawing the line

In order to draw a line, we need to add another SCNNode, in a line shape, between two points, and also another node to display measurement results.

When the add button tapped for the first time, in addition to adding a dot node on the screen its position is also used as the start point of the line, inSCNSceneRendererDelegate's method renderer:updateAtTime.

func renderer(
    _ renderer: SCNSceneRenderer, 
    updateAtTime time: TimeInterval
) {
    let view = self.measurementView    DispatchQueue.main.async {
        let position = view.centerRealWorldPosition()
        let pointOfView = view.pointOfView
        
        if let line = self.draft.drawLineIfNeeded(
            to: position, 
            withPointOfView: pointOfView
        ) {
            self.measurementView.addChildNodes(line.nodes)
        }
        ...
    }
}

During each update, the center point of the screen determines the end position of the line, therefore the line is redrawn with the updated endpoint each time the user moves the camera to another point.

Note that if the add button is tapped for the second time, the line is kept at its recent position. I achieved this by keeping/checking a step logic on ARMeasurement struct.

struct ARMeasurement {
    let steps: [Step] = [.first, .second, .last]
    var currentStep: Step = .first
}
    
...func drawLineIfNeeded(
    to position: SCNVector3?,
    withPointOfView pointOfView: SCNNode?
) -> Line? {
    if measurement.isCompleted {
        return nil
    }
    
    return addLine(to: position, withPointOfView: pointOfView)
}private func addLine(
    to position: SCNVector3?,
    withPointOfView pointOfView: SCNNode?
) -> Line? {
    guard
        measurement.currentStep == .second,
        let fromPosition = startDotNode?.position,
        let toPosition = position,
        let pointOfView = pointOfView
        else {
            return nil
    }
    
    let line = Line(
        fromVector: fromPosition,
        toVector: toPosition,
        pointOfView: pointOfView
    )
    
    self.line?.nodes.forEach { $0.removeFromParentNode() }
    self.line?.removeFromParentNode()
    self.line = line
    
    return line
}

Displaying text on line

I used another SCNNode, setting anSCNText as its geometry, to display measurements. Also, I set its position to endpoint of the line to make sure it's always visible.

private final class TextNode: SCNNode {
    private var textGeometry = SCNText()
    private lazy var textWrapperNode = SCNNode(
        geometry: textGeometry
    )    ...
}

Plane indicator

The plane indicator is important to acknowledge users about the detected plane instantly so that the user can make sure whether the nodes will be added on the right plane, which is essential for making the right measurement.

The indicator is only visible if there is a detected plane, which is determined by the method, realWorldPosition(for point: CGPoint) -> SCNVector3?. If the method returns a position addIndicatorNode() is called and the indicator is displayed on the center of the screen. When the method doesn't detect anything and returns nil, then the indicator is removed from the screen via removeIndicatorNode() method.

Also to update indicators position, resetIndicatorPosition() method is called in renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor.

private func addIndicatorNode() {
    let geometry = SCNPlane(width: 0.05, height: 0.05)
    let material = SCNMaterial()
    
    material.diffuse.contents = UIImage(
        named: "icon-plane-indicator"
    )
    indicatorNode.geometry = geometry
    indicatorNode.geometry?.firstMaterial = material
    
    scene.rootNode.addChildNode(indicatorNode)
}func resetIndicatorPosition() {
    if self.indicatorNode.parent == nil {
        self.addIndicatorNode()
    }
    
    guard let position = self.centerRealWorldPosition() else {
        self.removeIndicatorNode()
        return
    }
    
    self.indicatorNode.eulerAngles.x = -.pi / 2
    self.indicatorNode.position = position
}func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    DispatchQueue.main.async {
        self.measurementView.resetIndicatorPosition()
        ...
    }
}

AR Session feedback

Other than plane indicator, I also used ARSessionDelegate's methods to give additional info about the ar session. The message Move your phone on a flat surface until it's detected is displayed when no plane is detected in the ar session.

private func updateSessionInfoLabel(
        for frame: ARFrame,
        trackingState: ARCamera.TrackingState
    ) {
    let message: String
    
    switch trackingState {
    case .normal where frame.anchors.isEmpty, // No planes detected
         .limited(.excessiveMotion),
         .limited(.initializing),
         .limited(.insufficientFeatures),
         .notAvailable:
        message = "Move your phone on a flat surface until it's detected"
        
    default:
        message = ""
    }
    
    measurementView.info = message
}

Conclusion

In conclusion, this was my first experience with ARKit and although things seemed a little unfamiliar in the beginning, I was able to put things together with the help of sources on topic on the net, hoping that this one also be one of them, thanks for reading.