Applying Eyelashes and Lipstick using OpenCV

Hola! Are you sad about your favorite beauty app getting banned? Why don’t you build something of your own?

Wondering how? Then stop wondering and start reading 😛

In today’s blog, we’ll cover a tutorial on how to create your own tools for applying beautiful lipstick and gorgeous eyelashes on your face.

Overview

A pre-trained deep learning model has been trained to recognize the human face and its parts like: eyes, nose, mouth, etc. We will be using it to recognize the facial features and then we’ll apply various makeup tools.

Importing libraires

We are using the following libraries

• imutils -> for image processing
• numpy -> to do basic mathematics
• dlib-> it has the basic face recognition tools
• cv2-> for basic image processing
• sys-> for system commands

from imutils import face_utils
import numpy as np
import dlib
import cv2
import sys

Loading Pre-trained model

Now, we’ll load our facial feature detecting pre-trained deep learning model and store it in a variable named ‘shape_predictor‘.

shape_predictor= "shape_predictor_68_face_landmarks.dat"

Moving further, we’ll be initializing dlib face detector. We’ll be using it for facial landmark recognition.

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor(shape_predictor)

Loading the image and detecting Facial Features

We’ll start by converting the image into grayscale. For detecting facial features, we’ll use the grayscale image.

Firstly, we’ll initialize an empty list ‘a‘. Then, we’ll run a loop for extracting the shape of the facial features (in form of points/coordinates ) followed by converting it into a numpy array and finally, we’ll store it in a variable ‘shape’. Next, we’ll store the facial landmark shapes and their position in the list ‘a’. Variable ‘shape‘ only has list of coordinates in a 2-D array, where as ‘a’ has the names of facial features detected and their exact position in the variable ‘shape’.

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
rects = detector(gray, 1)
# loop over the face detections
a=[]
for (i, rect) in enumerate(rects):
    shape = predictor(gray, rect)
    shape = face_utils.shape_to_np(shape)
    a=list(face_utils.FACIAL_LANDMARKS_IDXS.items())
   

Function to apply Lipstick

If the features were detected, the length of ‘a’ must be greater than 0. If so, then we’ll store the name of the facial features detected in the variable ‘name’. Since, lips are detected first and stored at position ‘0’, we are extracting data only at index ‘0’. We are also using i,j variables for specifying the position in 2-D array where the points/coordinates of lips are stored.

Now, every point in shape variable is appended into a list variable ‘points’. We’ll now convert the variable ‘points’ into numpy array of suitable dimension.

At last, we’ll fill the polygon made by these points, i.e. lips, with purple color.

def apply_lipstick(img)  :    
    if(len(a)):
        name, (i, j)=a[0]
        points= []
        
        for (x, y) in shape[i:j]:
            points.append([x,y])
        points= np.reshape(points, (-1, 1, 2))    
        cv2.fillPoly(img, [points], (84,44, 150), 8)        
    
    return img

Function to applying Eyelashes and Eyeliner

The length of ‘a’ must be greater than 0 if the features were detected. If so, then we’ll store the name of the facial features detected in the variable ‘name’. Data at indices ‘4’ and ‘5’ will be extracted as left eye and right eye are detected and stored at position ‘4’ and ‘5’ respectively.. We have also used i,j variables for specifying the position in 2-D array where the points/coordinates of eyes are stored.

We’ll be extracting the first 4 variables of the left eye as they stand for upper eye.

To apply a liner, we’ll be taking 4 points and draw a line connecting them. We will select 2 points of the eye to apply eye lashes.

Then, we’ll be extracting the coordinates and width of eye in the variable ‘weye’. Moving further, we’ll read the image which is a lash for the left eye. followed by resizing the lash according to the shape of the eye.

The region of interest in image where lash is to be applied is stored in variable ‘roi’. We’ll visit every point in the lash and add it to the image.

We’ll follow the same approach for right eye and finally we’ll return the image.

def apply_lashes(img)  : 
    if(len(a)):
      name,(i,j)  = a[4]   #left eye    
    points= []
    cnt=0  
    
    if len(shape):
        for (x, y) in shape[i:j]:
            points.append([x,y]) 
            cnt+=1
            if cnt==4:
                break
            
    if len(points):  #liner
        left= points[0]
        left= (left[0], left[1])
        right= points[1]
        right= (right[0], right[1])
        pt= (0,0,0)
        cv2.line(img,left, right, pt,1)
        left= points[1]
        left= (left[0], left[1])
        right= points[2]
        right= (right[0], right[1])
        cv2.line(img,left, right, pt,1)
        left= points[2]
        left= (left[0], left[1])
        right= points[3]
        right= (right[0], right[1])
        cv2.line(img,left, right, pt,1)
        
        #extreme left el, er
        el= points[0]         
        er= points[2]
              
        #diff in height
        x_left, y_left, x_r, y_r= el[0] , el[1], er[0], er[1]
        weye= abs(x_r- x_left)*2
               
        lash= cv2.imread("one_lash.jpeg",-1)
        if lash is None:
            print(" not opened")
            return img
        lash= cv2.resize(lash,(weye, int(weye/4)))
 
       roi= img[ y_left-lash.shape[0]:y_left, x_r-lash.shape[1]:x_r] 
        
        for x in range(0, lash.shape[0]):
            for y in range(0,lash.shape[1]):
                a,b,c= lash[x,y]
                if not(a>=150 or b>=150 or c>=150):
                 roi[x,y]= (a,b,c)
                                       
        img[ y_left-lash.shape[0]:y_left, x_r-lash.shape[1]:x_r]  = roi
        
    ############right one
    a=[]
    shape=[]
    for (i, rect) in enumerate(rects):
    	shape = predictor(gray, rect)
    	shape = face_utils.shape_to_np(shape)
    	a=list(face_utils.FACIAL_LANDMARKS_IDXS.items())    
    if(len(a)):
        name,(i,j)  = a[5]   #right eye,    
    points=[]
    cnt=0    
    if len(shape):
        for (x, y) in shape[i:j]:
            points.append([x,y]) 
            cnt+=1
            if cnt==4:
                break
            
    if len(points):  #liner
        left= points[0]
        left= (left[0], left[1])
        right= points[1]
        right= (right[0], right[1])
        pt= (0,0,0)
        cv2.line(img,left, right, pt,1)
        left= points[1]
        left= (left[0], left[1])
        right= points[2]
        right= (right[0], right[1])
        cv2.line(img,left, right, pt,1)
        left= points[2]
        left= (left[0], left[1])
        right= points[3]
        right= (right[0], right[1])
        cv2.line(img,left, right, pt,1)
        
        el= points[1]         
        er= points[3]
        
            
        x_left, y_left, x_r, y_r= el[0] , el[1], er[0], er[1]
        weye= abs(x_r- x_left)*2
        
        lash= cv2.imread("one_lash_r.jpeg",-1)
        if lash is None:
            print(" not opened")
            return img
        lash= cv2.resize(lash,(weye, int(weye/4)))
        #print(lash.shape)
        
        roi= img[ y_left-lash.shape[0]:y_left, x_left:x_left+lash.shape[1] ]
        
        for x in range(0, lash.shape[0]):
            for y in range(0,lash.shape[1]):
                a,b,c= lash[x,y]
                if not(a>=150 and b>=150 and c>=150):
                 roi[x,y]= (a,b,c)
                                        
        img[ y_left-lash.shape[0]:y_left, x_left:x_left+lash.shape[1] ] = roi
        return img 

Function to enter image

We’ll create a function ask_image() which asks the user to input image name. If the image name is incorrect, it will re-ask the user. If count become greater then 5, the program will quit. The input image will be stored in a list ‘stack‘.

count=0
stack=[]

def ask_image(): 
    ##user interaction
    print("enter the image name: ")
    img_name= input()
    global count
    img = cv2.imread(img_name)
    if img is None:
        if count > 5:
            print("maximum limit reached re-run program")
            sys.exit()
            
        print(" no such image exits\n press 1 to reenter name \n press any other key to exit")    
        x = input()
        if x=="1":
            count+=1          
            ask_image()
        else:
            sys.exit()
    else:
        stack.append(img)
        return [img, img_name]

Function to input choice from user

The below function choice() will ask user to input the choices.

User will input:

1. applying lipstick.
2. applying lashes.
3. adjusting brows.
4. viewing the image.

def choice(img,img_name):
    print(""" enter 1 to apply lipstick
                    enter 2 to apply lashes and liner
                    enter 3 to adjust brows 
                    enter 4 view 
                    enter any other key to exit and save """)
    x= input()
    if x=="1":
        curr= stack[-1]
        curr= apply_lipstick(curr)
        stack.append(curr)
        view(curr)
        choice(img,img_name)
    elif x=="2":
        curr= stack[-1]
        curr= apply_lashes(curr)
        stack.append(curr)
        view(curr)
        choice(img,img_name)  
    elif x=="3":
        curr= stack[-1]
        curr=eye_brow(curr)
        stack.append(curr)
        view(curr)
        choice(img,img_name)
    elif x=="4":
        view(stack[-1])
        choice(img,img_name)   
    else:
        cv2.imwrite("edited"+img_name,stack[-1])   
        print(" file saved ")
        sys.exit()

Function to view the image

c=0
def view(img):
    if img is None or img.shape[0]==0 or img.shape[1]==0: return
    cv2.imshow('image press any key to close', img)
    global c
    cv2.imwrite("edited"+str(c)+img_name,stack[-1])
    c+=1
    cv2.waitKey(0) 
    cv2.destroyAllWindows()

Lastly, we’ll ask user to input the image followed by asking for choices.

choice(ask_image())

Implementation

Conclusion

Cheers!!

Finally, you are able to create your own beauty app tools. What are you waiting for? Go and try them on your own! Let us know what would you do differently in the comments section.

–Suniti Jain