Motion tracking II: 2D pose estimation via OpenPose

Overview

In this script, we take the videos prepared in the previous script and perform OpenPose 2D pose estimation (Cao et al. 2019) on them.

We are using the BODY_135 model with 135 keypoints.

Demo of this pipeline has been published on EnvisionBOX

See OpenPose documentation for more information.

import os
import subprocess
import glob
import tempfile
from IPython.display import Video
import random

curfolder = os.getcwd()

# Openpose demo.exe location
openposefol = curfolder+'/openpose/'
openpose_demo_loc = openposefol + '/bin/OpenPoseDemo.exe'

# This is the model to employ
model_to_employ = 'BODY_135'

# List folders in a main folder
folderstotrack = glob.glob(curfolder +'/projectdata/*')

# Get all folders per participant, per session
pcnfolders = []

for i in folderstotrack:
    pcn1folders = glob.glob(i + '/P0/*')
    pcn2folders = glob.glob(i + '/P1/*')
    pcnfolders_in_session = pcn1folders + pcn2folders

    # Append to the list
    pcnfolders = pcnfolders + pcnfolders_in_session

# There might be some other things we don't want now
pcnfolders = [x for x in pcnfolders if 'Config' not in x]
pcnfolders = [x for x in pcnfolders if 'opensim' not in x]
pcnfolders = [x for x in pcnfolders if 'xml' not in x]
pcnfolders = [x for x in pcnfolders if 'ResultsInverseDynamics' not in x]
pcnfolders = [x for x in pcnfolders if 'ResultsInverseKinematics' not in x]
pcnfolders = [x for x in pcnfolders if 'sto' not in x]
pcnfolders = [x for x in pcnfolders if 'txt' not in x]

print(pcnfolders[0:10])

Running OpenPose as a subprocess

Instead of running OpenPose in the notebook, we will send commands to the OpenPose executable as a subprocess. This is more efficient and allows us to run OpenPose in parallel for multiple videos.

def runcommand(command):
    # Run the command using subprocess for OPENPOSE TRACKING
    try:
        subprocess.run(command, shell=True, check=True)
    except subprocess.CalledProcessError as e:
        print(f"Command execution failed with error code {e.returncode}")
    except FileNotFoundError:
        print("The OpenPoseDemo.exe executable was not found.")

for i in pcnfolders:
    os.chdir(openposefol)
    print('working on ' + i)

    # Identify all avi files in folder
    direc = glob.glob(i + '/raw-2d/' +'*.avi')

    # 3 cameras
    video0 = direc[0]
    video1 = direc[1]
    video2 = direc[2]

    videolist = [video0, video1, video2]
    
    # Make a new directory if it doesn't exist
    if not os.path.exists(i+'/pose/'):
        os.makedirs(i+'/pose/')
    if not os.path.exists(i+'/pose/pose_cam1_json/'):
        os.makedirs(i+'/pose/pose_cam1_json/')
    if not os.path.exists(i+'/pose/pose_cam2_json/'):
        os.makedirs(i+'/pose/pose_cam2_json/')
    if not os.path.exists(i+'/pose/pose_cam3_json/'):
        os.makedirs(i+'/pose/pose_cam3_json/')
    

    # Also make directory for openpose videos (pose-2d-trackingvideos)
    if not os.path.exists(i+'/pose-2d-trackingvideos/'):
        os.makedirs(i+'/pose-2d-trackingvideos/')

    # Initialize the pose2 folder
    outputfol1 = i+'/pose/pose_cam1_json/'
    outputfol2 = i+'/pose/pose_cam2_json/'
    outputfol3 = i+'/pose/pose_cam3_json/'

    outputfollist = [outputfol1, outputfol2, outputfol3]

    for it, j in enumerate(outputfollist):
        # Prepare the command
        openposelocation = ' ' + openpose_demo_loc + ' '
        model = '--model_pose' + ' ' + model_to_employ + ' '
        video = '--video ' + videolist[it] + ' '
        todo = '--write_json '
        outputfol = j + ' '
        videoadd = '--write_video '
        videopath = i+'/pose-2d-trackingvideos/' + 'video'+str(it)+'.avi' + ' '
        # Send the command via subprocess
        command = r' '+openposelocation+model+video+todo+outputfol+videoadd+videopath
        print('were going to send this to command prompt: ' + command)
        runcommand(command)
    

The results are saved in /pose folder in json format, and are used for triangulation via Pose2sim (Pagnon, Domalain, and Reveret 2022) in the next script.

The videos with the estimated poses are saved in the /pose-2d-trackingvideos folder

Here is an example of the output:

References

Cao, Zhe, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.” 2019. https://doi.org/10.48550/arXiv.1812.08008.

Pagnon, David, Mathieu Domalain, and Lionel Reveret. 2022. “Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy.” Sensors 22 (7, 7): 2712. https://doi.org/10.3390/s22072712.