The Challenge of Developing a Birthday Celebration Filter Using ARKit

Reference: Apple Developer Documentation

ARKit's ARSCNView is a SceneKit-based 3D view. ARSCNView processes camera input data in real time to display virtual objects. ARSCNView requires an active session to function, and ARSession settings are configured through ARConfiguration.

The types of ARConfiguration provided by ARKit are as follows. Each configuration enables users to have different AR experiences.

• ARWorldTrackingConfiguration

• AR Geo-Tracking Configuration

• AR Orientation Tracking Configuration

• ARBodyTrackingConfiguration

• ARFaceTrackingConfiguration

Among these, ARFaceTrackingConfiguration recognizes faces from the information received via the camera and provides facial information through FaceAnchor. Then, in the ARSCNViewDelegate functions,

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode?

This allows you to create new nodes based on new ARAnchors detected by ARKit.

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
    guard let device = MTLCreateSystemDefaultDevice() else { return nil }
    let faceGeometry = ARSCNFaceGeometry(device: device)

    let faceNode = FaceFilterNode(geometry: faceGeometry)
    let headNode = HeadNode(with: "filter_hat")
    let eyeNode = EyeNode(with: "filter_sunglasses")
    faceNode.addChildNode(headNode)
    faceNode.addChildNode(eyeNode)

    return faceNode
}

The EyeNode and HeadNode are defined by inheriting from SCNNode as shown below. They are responsible for applying filter images to the head and eye positions.

class EyeNode: SCNNode, FilterNodeElementProtocol {
    init(with imageName: String, width: CGFloat = 0.2) {
        super.init()
        setup(with: imageName, width: width)
    }
    func updatePosition(for vectors: [vector_float3]) {
        let eyeIndex = 12
        if let newPosition = vectors[safe: eyeIndex] {
            self.position = SCNVector3(newPosition)
        }
    }

    //...
}

The vectors array received as a parameter in the updatePosition function is FaceAnchor's ARFaceGeometry an array containing the vertex elements of each face provided by,

• The center of the eye

• Center of the mouth

• tip of the nose

It provides detailed information on approximately 1,220 elements.

Once you've created a new node like this, you need to update it every frame. Another ARSCNViewDelegate function,

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor)

This is handling it.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    guard let faceAnchor = anchor as? ARFaceAnchor,
    let filterNode = node as? FilterNode else { return }

    filterNode.geometry.update(from: faceAnchor.geometry)
    let vertices = anchor.geometry.vertices
    filterNode.subNodes.forEach { $0.updatePosition(for: vertices) }
}

This process updates the new geometry information in the FilterNode created to generate the filter image, and uses the newly acquired vector data to update the positions and orientations of the EyeNode and HeadNode.

Then, you'll get a cool filter feature like this. Pretty simple, right? 😁