Object recognition is a technology capable of identifying objects and people in an image or video. Although it has been around for many decades, it has only recently become widely used in social media and software. Object and face recognition technology can be utilized in many areas and industries, such as security, surveillance and robotics.

This  is a tutorial on how to build an AI application, that recognizes and tracks faces real-time in photos or videos. For this purpose, we will use the tracking.js library. The tracking.js library utilizes many vision techniques and algorithms into the browser environment by using JavaScript and HTML5. It is a lightweight library that allows you to implement face and feature detection, real time color tracking and more. We will start a project creating a simple HTML page that detects faces in an image and build moving forward to video feeds. By the end, you should have a solid foundation on how to create an application to identify and track faces in images and videos.

The following topics will be covered in this tutorial:

• Creating and hosting a simple web page in a local HTTP Server
• Implementing tracking.js in a web page
• Scanning for faces in a photo
• Using training classifiers
• Tracking faces in a video

Requirements

For this tutorial, you will need the following components:

• The tracking.js library
• The jQuery JavaScript Library, version 3.3.1
• The Bootstrap  HTML, CSS, and JS library, version 4.2.1
• Your favorite HTML/JS Code Editor. We will be using Visual Studio Code.
• A basic HTTP server to serve your pages. We will be using Node.js

Download the tracking.js library from this repository:
https://github.com/eduardolundgren/tracking.js/tree/master/build

We will reference the jQuery and Bootstrap libraries using CDN (Content Delivery Network) ,so we will also need an active Internet connection. If you want your application to run completely offline, you will have to download the files and reference them in your HTML page accordingly.

To install Visual Studio Code please go through the following steps:

1. Open the Visual Studio Code website
   https://code.visualstudio.com/
2. Find the correct download package for your platform, Windows, Linux or MacOS
   
3. Download the executable and Save it to your computer
4. Run (or Unzip) it to complete the installation.

To install Node.js to your machine please go through the following steps:

1. Open the Node.js website
   https://nodejs.org/en/download/
2. Find the correct download package for your platform, Windows, Linux or MacOS
   
3. Download the executable and Save it to your computer
4. Run (or Unzip) it to complete the installation.

The complete source code for this project is in this GitHub Repository:

https://github.com/georgiakalyva/face-tracking-js

Facial Recognition

An object or face recognition system is a technology capable of identifying objects and people in an image or video. To identify faces in images, most of the techniques available map the facial features to detect a face, and then compare it to a database to complete the identification. Face recognition has several limitation since it depends on head tilt and direction, lighting and image clarity. However it is preferred in biometric identification, since it does not require the individual’s participation like fingerprint or iris identification.

There are many algorithms and techniques used in face recognition, but generally face recognition has the following process. The first step is to detect the position of the face features by analyzing the relative position, size, and/or shape of the eyes, nose, and jaw line. To train an algorithm to detect those points, a machine learning model is used based on large face databases. There are many free available for educational and research purposes like CalTech University Datasets http://www.vision.caltech.edu/html-files/archive.html. Then, depending on the angle, the points are normalized and finally compared against a database to determine if  match is found and the person is identified. Face detection and tracking in videos uses a similar process. Since a video is a succession of images, all that is needed to implement face tracking  is to run image recognition in every frame of the video.

Face recognition has various application in many areas and industries. Social Media employ face recognition to tag faces and suggest photos that might be of the same person. There are social media engines, that search for the same person profile across all social media or someone can simply use Google Image Search to find similar images across the Internet.

As identity verification goes, most electronic devices such as mobile phones and laptops allow the user to unlock your device via the owners’ facial features, and make mobile payments using face together with other biometrics like fingerprint.

Law enforcement and border authorities use face identification to look for criminals and identify suspects. Security in banks, airports, stores and other businesses use face recognition software to look for suspicious behavior.

So lets move on to learn how to build a face detection and tracking application.

Creating and hosting the project

In this section, we will go through the steps of creating a new HTML page from scratch and serve it using a local web server. For those familiar with HTML5, the steps will be straightforward. For anyone new, everything will be explained in detail so make sure you have everything downloaded and installed and we are ready to begin.

Creating a new HTML Page

The first step is to create a new empty HTML page. Create a directory in your local drive to add all the project files. Open the project directory and create the following three folders and name them assets, css and tracking. In the assets directory we will put all assets like images etc. In the css directory we will put all css related files. In the tracking directory, we will put the tracking.js library files. Here’s how we would go about the process:

1. Open Visual Studio Code or your preferred code editor, create a new HTML page in that directory and name it Index.html. Open the page you just created, and adjust the content like this:

<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Artificial Intelligence - Face Tracking with JS </title>
</head>
<body>
<script>
</script>
</body>
</html>

2. In the <head> section after the <title> tag we will reference all the CSS and JavaScript Libraries. First, add some Bootstrap meta-tags and reference the Bootstrap CSS via CDN.

<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!--CSS-->
<link rel="stylesheet" ref="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css">

3. Create a new .css file and add it in the css folder. Name it demo.css and add the following styling rules in it.

.rect {
border: 2px solid white;
left: -1000px;
position: absolute;
top: -1000px;
}
video, canvas {
margin-left: 50px;
margin-top: 50px;
position: absolute;
}

4. Then go back to Index.html and add a reference to it, under the Bootstrap.css declaration.

<link rel="stylesheet" href="../css/demo.css">

1. Find the folder where you downloaded the tracking.js library and unzip the project somewhere on your local drive. Find the /build folder and copy the files into your project folder. Then add the file references after the CSS declarations.

<!--Tracking JS-->
<script src="../tracking/tracking-min.js"></script>
<script src="../tracking/data/face-min.js"></script>
<script src="../tracking/data/eye-min.js"></script>
<script src="../tracking/data/mouth-min.js"></script>

6. Finally, add the jQuery and Bootstrap CDNs.

<!--jQuery/Bootstrap-->
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.6/umd/popper.min.js"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js"></script>

7. Now that we have finished with the declarations, it is time to add some content. Add the following code inside the <body> tag above the <script> tag, for the navigation bar.

<nav class="navbar navbar-expand-sm bg-dark navbar-dark">
 <a class="navbar-brand" href="#"><img src="../assets/logo.png" height="20px"/></a>
  <!-- Links -->
  <ul class="navbar-nav">
    <li class="nav-item">
     <a class="nav-link" href="/Index.html">Face Recognition (Photo)</a>
    </li>
    <li class="nav-item">
     <a class="nav-link" href="/Camera.html">Face Tracking (Camera)</a>
    </li>
    <li class="nav-item">
     <a class="nav-link" href="/Video.html">Face Tracking (Video)</a>
    </li>
  </ul>
</nav>

Using Node.js to serve content

If you try to run the page right now it loads fine. When we add the tracking.js code later, if you try to run the HTML page locally you might run into this error:

or errors like:

net::ERR_FILE_NOT_FOUND

This happens because when you add the tracking.js and css code into your application that are not hosted on a server, somewhere there will be some form of URL loading request. You can’t load images or any other content via this method from a local file system. Your images need to be loaded via a web server, and accessed via a proper HTTP URL, so this is why we need to use some kind of server environment.

In this section, we will show you how to use Node.js to serve your pages. We have chosen Node.js,  because it is an open source server environment, that runs on almost every platform (Windows, Linux, Unix, Mac OS X, etc.). If you are familiar with another server technology, you can modify the code accordingly and go ahead and skip this next part.

Create a file named server.js in your project folder and add the following code to it.

var http = require('http');
var fs = require("fs");
http.createServer(function(request, response){
      fs.readFile("index.html", function(err, data){
        response.writeHead(200, {'Content-Type': 'text/html'});
        response.write(data);
        response.end();
    });
}).listen(3000);

Open a Command prompt window (or a Terminal) inside the folder and run the following command to install the http-server module. Http-server is a zero-configuration command-line HTTP server. It can be used for production, but it’s simple enough to be used for learning and development purposes as well.

npm install http-server -g

Press enter and the installations will begin.

After the installation is completed, all you need to do to run the server is launch the Command Prompt (or Terminal) inside the project folder and run the following http-server command.

http-server

In the Command Prompt window, you can see the IP address and port the application is running on. Also, you can monitor all requests in the Command Prompt window.

Open your browser and go to http://localhost:8080/ to view the page we created. The result should be similar to this:

Implementing Face Detection in photos

Now that everything is set up we can start with face detection. In this section, we will explain the key concepts of working with tracking.js in as much detail as possible.

First, choose an image that contains faces and upload it to the assets directory. Then open Index.html which we created in the previous section and add the following code under the navigation bar. In the src attribute add the link to your image.

<div class="container-fluid">
     <div class="demo-image">
         <img id="imageToFindFaces" src="../assets/1.jpeg" />
     </div>
</div>

We will create a function called FaceDetection for clarity. You can put the code directly in the $(document).ready() function if you wish.

Instantiating Trackers

The first step is to load the image you want to start tracking to a variable.

var img = document.getElementById('imageToFindFaces');

Then instantiate the Tracker you want to use. We are going to use the ObjectRecognition Tracker to detect faces. You must define the stepSize property, which specifies the block step size.

var tracker = new tracking.ObjectTracker('face');
tracker.setStepSize(1.7);

Once you have the track instance and the image you want to detect faces you can start tracking. The tracking.js library provides a utility that handles the tracking, provided you have instantiated the tracker correctly. All you have to do is pass as arguments the tracker and the object id (image, video, canvas).

tracking.track('#imageToFindFaces', tracker);

Now that you have everything ready you need to know when something happens by listening for track events. For this purpose, we will use the track function.  In this function, you will find all the data you need by using the event variable the tracker returns. In case of an image this will run once to detect faces. By iterating the event.data you will find the coordinated of the faces in the image. Here you can add some extra validation, by checking the event.data.length property and if there are no data returned, a message can be shown to the user.

tracker.on('track', function(event) {
    if (event.data.length === 0) {
    
    } else {
        event.data.forEach(function(rectangle) {
            window.plot(rectangle.x, rectangle.y, rectangle.width, rectangle.height);
        });
    }
 });

Using those coordinates you can draw on top of the image a rectangle based on the data the tracker returned. Use the window.plot to draw the rectangle. You can use the following code block to create a div and insert it inside the parent div of the image, at the coordinates location.

window.plot = function(x, y, w, h) {
    var rectangle = document.createElement('div');
    document.querySelector('.demo-image').appendChild(rectangle);
    rectangle.classList.add('rect');
    rectangle.style.width = w + 'px';
    rectangle.style.height = h + 'px';
    rectangle.style.left = (img.offsetLeft + x) + 'px';
    rectangle.style.top = (img.offsetTop + y) + 'px';
};

Run the application in the browser. The end result should look similar to this.

Training Classifiers

To detect a face or an object, we use what we call a training classifier. Training classifiers teach the tracker the object you want to track. If you noticed, in the beginning, we referenced three files in addition to the tracking.js library.

<script src="../tracking/data/face-min.js"></script>
<script src="../tracking/data/eye-min.js"></script>
<script src="../tracking/data/mouth-min.js"></script>

In the previous section, we used the face training classifier to detect a face. Now let’s add the eye and mouth classifiers to detect eyes and mouths in the face, in addition to the face detection. To achieve this, all we need to do is add more parameters at the instantiation of the tracker. Instead of a string, we will pass an array of strings, with the desired classifiers.

var tracker = new tracking.ObjectTracker(['face', 'eye', 'mouth']);

Now run the sample again. The end result should look similar to this.

Implementing Face Tracking in videos

In this section, we are going to implement face tracking in a video feed from the camera. Implementing a video tracker is a little different from face detection from an image. When we have an image, the track event runs only once, but in a video the track event is executed for every frame of the video.

Create a new page and name it Camera.html. The declarations and everything else will be the same, except the demo-image div, which will be replaced by the following code.

<div class="demo-camera">
 <video id="video" width="480" height="320" preload autoplay loop muted>
 </video>
 <canvas id="canvas" width="480" height="320"></canvas> 
</div>

The video element is responsible for taking the video stream from your webcam and displaying it on the screen. We will use the canvas element to display the results on top of the video feed. In the $(document).ready() function, load the video and the canvas to a variable. Then set the canvas context to return a 2d object.

var video = document.getElementById('video');
 var canvas = document.getElementById('canvas');
 var context = canvas.getContext('2d');

The instantiation of the tracker is almost the same. Note the two extra properties we need to define at this point, setInitialScale and setEdgesDensity. The setInitialScale property specifies the initial scale to start the feature block scaling. The setEdgesDensity property specifies the edges density of a block in order to decide whether to skip it or not.

var tracker = new tracking.ObjectTracker('face');
tracker.setInitialScale(4);
tracker.setStepSize(2);
tracker.setEdgesDensity(0.1);

Start the tracking by passing as arguments the initialized tracker and video feed, and set the camera option to true.

tracking.track('#video', tracker, { camera: true });

Because we have a video feed, the track event will run for every frame in the video. For this purpose, we have to modify the tracker listener to clear the rectangle each time it executes. Then we will redraw the rectangle for every frame.

tracker.on('track', function(event) {
context.clearRect(0, 0, canvas.width, canvas.height);
event.data.forEach(function(rect) {
context.strokeStyle = '#ffffff';
context.strokeRect(rect.x, rect.y, rect.width, rect.height);
context.font = '14px Helvetica';
context.fillStyle = "#fff";
context.fillText('x:'+rect.x+'px',rect.x+rect.width+5,rect.y+11);
context.fillText('y:'+rect.y+'px',rect.x+rect.width+5,rect.y+22);
});

Run the sample to see the results. If you don’t have a webcam on your computer the process of implementing tracking in a video from a local file is quite simple. Just add the <source src="../assets/girls.mp4" type="video/mp4"> inside the <video> tag like in the code snippet below.

<div class="demo-camera"> 
<video id="video" width="480" height="320" preload autoplay loop muted controls>
<source src="../assets/face/girls.mp4" type="video/mp4">
</video>
<canvas id="canvas" width="480" height="320"></canvas> 
</div>

The results should look similar to this.

Summary

In this tutorial we created a new web page from scratch, learned to set up a local HTTP server to host our web pages and implemented face tracking in pictures and video using the tracker.js library. We learned to draw a rectangle around the features we detected, so now you can take this knowledge and draw anything using the coordinates provided by the library. For example we can use this to overlap the photo with flowers and glasses instead of a rectangle, just like Instagram filters, or blur the photo to hide the faces. And we did it all in the environment of the browser just by using HTML and JavaScript.

As great as it is to have come this far, this is only the start of the capabilities the tracking.js has to offer, so at this point only your imagination is the limit. Since we have a solid base to build on, you can begin experimenting with the color tracking capabilities or create your own training classifier and bring the power of computer vision to your applications.