Many people do not want someone else to be able to use their phone. For such users, developers are coming up with new ways to block mobile devices. At first, the phone information was protected by a PIN code, then a pattern appeared, then a fingerprint scanner, and finally face unlock. Xiaomi decided to keep up with the new standard and therefore, with a firmware update, added this function to many of its devices that did not initially support it. You can learn more about the technology in this review.
The main differences between Xiaomi technology and Apple
Despite the fact that the function first appeared on a Samsung phone, it gained popularity thanks to Apple . The American giant managed to improve the technology mechanism, making it almost ideal.
Is everything so good in Xiaomi's face scanner? Has the Chinese company presented the user with a similar option that costs several times less? Let's try to figure it out.
Apple - Face ID
In addition to the regular front camera, special sensors . They process about 30 thousand points on the face , as a result of which information about the owner is remembered as accurately as possible.
Special neurons penetrate to a certain depth, and due to this, the quality of recognition increases significantly. The user can grow a beard, put on a hat and glasses - and the scanner will definitely recognize him . If there is insufficient lighting, no problems will arise either . Being in a dark room or at dusk, the owner can not worry and can safely use his phone.
Xiaomi - Face Unlock
Here, unfortunately, face unlock has some disadvantages. Only the selfie camera is used as a recognizer . The situation is a little better with the flagship Mi 9 and Mi 8, where infrared sensors are present .
If enough light enters the frame and there are no changes in appearance, Face Unlock will work just as well as on the iPhone. But vegetation, hats, glasses and scarves are already a hindrance. In previous Xiaomi models, unlocking also took place using a photograph, but in MIUI 11 and MIUI 10 this flaw has been successfully corrected.
Small conclusions. If you don’t often use a face password, and you can’t buy an expensive iPhone X, take a closer look at Mi 8. You’ll get all the basic functionality of Face ID without any problems. Do you need constant and accurate unlocking without the slightest error? Then only Apple products will help.
Delaunay triangulation to construct a mask
Let's move on to the practical part. Let's try to build a simple mask on the face using the obtained landmarks. The expected result will be a mask like:
Figure 6. Mask visualizing the Delaunay triangulation algorithm
Delaunay triangulation is a triangulation for a set of points S on a plane in which, for any triangle, all points from S, with the exception of points that are its vertices, lie outside the circle circumscribed around the triangle. First described in 1934 by Soviet mathematician Boris Delaunay.
Figure 7. An example of Delaunay triangulation. From each point a circle is generated, passing through the two nearest ones in the Euclidean metric
Xiaomi models with face unlock support
Officially, the face unlock feature is only available on certain models . Most often these are either flagships or phones in the mid-price segment.
We are talking specifically about factory support, but no one excludes custom firmware. In addition, with the release of new versions of Miyuai, the function may be added to some phones .
In the meantime, let’s look at smartphones that already have frontal unlocking from the factory:
- Mi Mix 3;
- Mi Mix 2/2S;
- All Redmi 5 models;
- The entire Black Shark and Pocophone series;
- The entire Mi 8 line;
- The entire line of Mi 9, Mi 10, Redmi K30 and Redmi K20;
- Redmi 6 and 6A, Redmi Note 6;
- Android One series;
- Redmi Note 7, Note 7 Pro, Redmi 7, Redmi 7A;
- The entire CC9 line. Including CC9 Pro (Mi Note 10);
- Redmi Note 8, Redmi Note 8 Pro, Redmi 8, Redmi Note 8T, Redmi 8A and Redmi Note 9 Pro (Max).
All other models not listed above do not have facial recognition functionality. If this is not the case, indicate your model in the comments and we will correct the list.
By the way, the first Xiaomi phone with Face ID was supposed to be Mi Note 3. Alas, the flagship never received the necessary software from the manufacturers.
In which regions does Face ID work?
Unfortunately, this function is not yet available in Europe (partially available in Russia). Currently it operates in the following countries:
- India;
- Indonesia;
- Hong Kong;
- Taiwan;
- Malaysia;
- Singapore;
- Partially Europe.
How to set up face unlock on Xiaomi
And finally, we begin the practice itself. Activating face unlock is very simple . I will describe detailed instructions including switching to one of the above regions using Redmi 5+.
- Go to “Settings”, scroll down a little and click on “Advanced settings”.
- Next, “Region” and select any suitable one from the list, the first one will be India, and select it. Be prepared that after selecting a new region, the system will automatically change the configurations of some applications + add local ones (in my case, I downloaded 5+ applications: Amazon, Mi Store, Netflix and others).
- Now we go down to the item “System and device” . Go to the section “Blocking and protection” .
- On the new page we see tabs responsible for adding a fingerprint, locking the screen, etc. In the middle is the inscription “Adding face data” . We click on it.
If you don’t have alternative protection, the system will require you to add it. This can be a alphanumeric marking, a graphic key or a fingerprint.
Then a warning appears that face protection is not the safest option. We agree, and the scanning begins. Look straight into the front camera so that your entire face is captured in the lens. Make sure there is enough light. When the photo is taken, check it carefully and click on the big “Done” button. So we have enabled facial recognition on Xiaomi.
Practical implementation of the algorithm
Let's implement the Delaunay triangulation algorithm for our face in the camera.
Step 1: Inside you will see a wrapper that takes an array of points in 2D space and returns an array of triangles.
public final class Triangle { public var vertex1: Vertex public var vertex2: Vertex public var vertex3: Vertex public init(vertex1: Vertex, vertex2: Vertex, vertex3: Vertex) { self.vertex1 = vertex1 self.vertex2 = vertex2 self.vertex3 = vertex3 } }
And vertex is a wrapper for CGPoint, additionally containing the number of a specific landmark.
public final class Vertex { public let point: CGPoint // Point identifier. From 0 to 67. There are 68 values for dlib. Or 65 for vision public let identifier: Int public init(point: CGPoint, id: Int) { self.point = point self.identifier = id } }
Step 2. Let's move on to drawing polygons on the face. Turn on the camera and show the image from the camera on the screen:
final class ViewController: UIViewController { private var session: AVCaptureSession? private let faceDetection = VNDetectFaceRectanglesRequest() private let faceLandmarks = VNDetectFaceLandmarksRequest() private let faceLandmarksDetectionRequest = VNSequenceRequestHandler() private let faceDetectionRequest = VNSequenceRequestHandler() private lazy var previewLayer: AVCaptureVideoPreviewLayer? = { guard let session = self.session else { return nil } var previewLayer = AVCaptureVideoPreviewLayer(session: session) previewLayer.videoGravity = .resizeAspectFill return previewLayer }() private lazy var triangleView: TriangleView = { TriangleView(frame: view.bounds) }() private var frontCamera: AVCaptureDevice? = { AVCaptureDevice.default(AVCaptureDevice.DeviceType.builtInWideAngleCamera, for: AVMediaType.video, position: .front) }() override func viewDidLoad() { super.viewDidLoad() sessionPrepare() session?.startRunning() guard let previewLayer = previewLayer else { return } view.layer.addSublayer(previewLayer) view.insertSubview(triangleView, at: Int.max) } override func viewDidLayoutSubviews() { super.viewDidLayoutSubviews() previewLayer?.frame = view.frame } private func sessionPrepare( ) { session = AVCaptureSession() guard let session = session, let captureDevice = frontCamera else { return } do { let deviceInput = try AVCaptureDeviceInput(device: captureDevice) session.beginConfiguration() if session.canAddInput(deviceInput) { session.addInput( deviceInput) } let output = AVCaptureVideoDataOutput() output.videoSettings = [ String(kCVPixelBufferPixelFormatTypeKey): Int(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) ] output.alwaysDiscardsLateVideoFrames = true if session.canAddOutput(output) { session.addOutput (output) } session.commitConfiguration() let queue = DispatchQueue(label: "output.queue") output.setSampleBufferDelegate(self, queue: queue) print("setup delegate") } catch { print("can't setup session") } } }
Step 3. Next we get frames from the camera
Fig 8. Example of a received frame from a camera
extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } guard let attachments = CMCopyDictionaryOfAttachments( kCFAllocatorDefault, sampleBuffer, kCMAttachmentMode_ShouldPropagate) as? [String: Any] else { return } let ciImage = CIImage(cvImageBuffer: pixelBuffer, options: attachments) // leftMirrored for front camera let ciImageWithOrientation = ciImage.oriented(forExifOrientation: Int32(UIImageOrientation.leftMirrored.rawValue)) detectFace(on: ciImageWithOrientation) } }
Step 4. Looking for faces in the frame
fileprivate func detectFace(on image: CIImage) { try? faceDetectionRequest.perform([faceDetection], on: image) if let results = faceDetection.results as? [VNFaceObservation] { if !results.isEmpty { faceLandmarks.inputFaceObservations = results detectLandmarks(on: image) } } }
Step 5. Looking for landmarks on the face
Fig 9. Example of found landmarks on a face
private func detectLandmarks(on image: CIImage) { try? faceLandmarksDetectionRequest.perform([faceLandmarks], on: image) guard let landmarksResults = faceLandmarks.results as? [VNFaceObservation] else { return } for observation in landmarksResults { if let boundingBox = faceLandmarks.inputFaceObservations?.first?.boundingBox { let faceBoundingBox = boundingBox.scaled(to: UIScreen.main.bounds.size) var maparr = [Vertex]( ) for (index, element) in convertPointsForFace(observation.landmarks?.allPoints, faceBoundingBox).enumerated() { let point = CGPoint(x: (Double(UIScreen.main.bounds.size.width - element.point.x) ), y: (Double(UIScreen.main.bounds.size.height - element.point.y))) maparr.append(Vertex(point: point, id: index)) } triangleView.recalculate(vertexes: maparr) } } } private func convertPointsForFace(_ landmark: VNFaceLandmarkRegion2D?, _ boundingBox: CGRect) -> [Vertex] { guard let points = landmark?.normalizedPoints else { return [] } let faceLandmarkPoints = points.map { (point: CGPoint) - > Vertex in let pointX = point.x * boundingBox.width + boundingBox.origin.x let pointY = point.y * boundingBox.height + boundingBox.origin.y return Vertex(point: CGPoint(x: Double(pointX), y : Double(pointY)), id: 0) } return faceLandmarkPoints }
Step 6. Next, draw our mask on top. We take the resulting triangles from the Delaunay algorithm and draw them in the form of layers.
Fig 10. The final result is a simple mask over the face
A complete implementation of the Delaunay triangulation algorithm in Swift is here.
And a couple of optimization tips for the experienced. Drawing new layers every time is an expensive operation. Constantly calculating the coordinates of triangles using the Delaunay algorithm is also expensive. Therefore, we take a high-resolution and good-quality face that looks into the camera, and run the Delaunay triangulation algorithm once on this photo. We save the resulting triangles into a text file, and then we use these triangles and change their coordinates.
