The Journey of Developing a Birthday Celebration Filter Using ARKit

See: Apple Developer Documentation

ARKit's ARSCNView is a SceneKit-based 3D view. ARSCNView processes camera input data in real time to display virtual objects. ARSCNView requires an active session to function, and ARSession settings are configured using ARConfiguration.

ARKit offers the following types of ARConfiguration. Each configuration allows users to have a different AR experience.

• ARWorldTrackingConfiguration

• ARGeoTrackingConfiguration

• AROrientationTrackingConfiguration

• AR Body Tracking Configuration

• ARFaceTrackingConfiguration

Among these, ARFaceTrackingConfiguration recognizes faces from the data received via the camera and provides facial information through FaceAnchor. Then, within the ARSCNViewDelegate functions,

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode?

This allows you to create a new node based on a new ARAnchor detected by ARKit.

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
    guard let device = MTLCreateSystemDefaultDevice() else { return nil }
    let faceGeometry = ARSCNFaceGeometry(device: device)

    let faceNode = FaceFilterNode(geometry: faceGeometry)
    let headNode = HeadNode(with: "filter_hat")
    let eyeNode = EyeNode(with: "filter_sunglasses")
    faceNode.addChildNode(headNode)
    faceNode.addChildNode(eyeNode)

    return faceNode
}

The EyeNode and HeadNode above are defined as subclasses of SCNNode, as shown below. They are responsible for applying filter images to the head and eye positions.

class EyeNode: SCNNode, FilterNodeElementProtocol {
    init(with imageName: String, width: CGFloat = 0.2) {
        super.init()
        setup(with: imageName, width: width)
    }
    func updatePosition(for vectors: [vector_float3]) {
        let eyeIndex = 12
        if let newPosition = vectors[safe: eyeIndex] {
            self.position = SCNVector3(newPosition)
        }
    }

    //...
}

The `vectors` array passed as a parameter to the `updatePosition` function is from FaceAnchor's ARFaceGeometry an array containing the vertex elements of each face provided by,

• The center of the eye

• The center of the mouth

• tip of the nose

It provides detailed information on approximately 1,220 elements, among others.

Once you've created a new node like this, you need to update it every frame. Another ARSCNViewDelegate method is

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor)

is handling this.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    guard let faceAnchor = anchor as? ARFaceAnchor,
    let filterNode = node as? FilterNode else { return }

    filterNode.geometry.update(from: faceAnchor.geometry)
    let vertices = anchor.geometry.vertices
    filterNode.subNodes.forEach { $0.updatePosition(for: vertices) }
}

This updates the geometry information in the FilterNode created to generate the filter image, and uses the newly obtained vector data to update the positions and orientations of the EyeNodes and HeadNodes.

Then, you’ll end up with a cool filter feature like the one below. It’s super simple, right? 😁