12 KiB

Raw Blame History

Exercise 2: Gesture recognition

Advised duration: 4 hours
Maximum advised duration: 6 hours

This exercise can be splitted in three parts but they are all bound and you should do them in the right order.

The main objective is to develop a simple gesture recognition system. It is not based on AI or Machine Learning. It is rather made of Q&D¹ solutions.

Just a glimpse on the work to be done

Our gesture recognition will be able to:

recognize postures ie static body poses (your body does not move, it is tested on a single frame),
recognize gestures ie dynamic body movements (your body does move, it is tested during several frames),
do mapping of body movement on whatever computer thing (a cursor, an ellipse, a dinosaur...).

You will have to develop the following assemblies (except Exercise3.WpfApp, and the Nuget Packages Microsoft.Kinect and CommunityToolkit.Mvvm):

flowchart LR
subgraph Exercise2
    MyGesturesBank
    KinectUtils
    Tests
end
MyGesturesBank -.-> KinectUtils 
KinectUtils -.-> Microsoft.Kinect
KinectUtils -.-> CommunityToolkit.Mvvm
Tests -.-> MyGesturesBank
Tests -.-> KinectUtils
style Microsoft.Kinect fill:#eee,color:#111,stroke:#999
style CommunityToolkit.Mvvm fill:#eee,color:#111,stroke:#999
subgraph Exercise3
    WpfApp
end
WpfApp -.-> MyGesturesBank
WpfApp -.-> KinectUtils

KinectUtils will contain all the base types to do gesture recognition and mapping. It will be updated in the three parts of this exercise.

MyGesturesBank will contain specific implementation of postures and gestures that could then be used in different applications.

Tests are only one or several Console app tests to verify that our system is working well.

Here is just a glimpse of the classes you will develop. More details will be given later or by your teacher on demand:

BaseGesture will represent any gesture, ie any Posture or Gesture (dynamic). To be tested, you will have to call TestGesture that will fire the event GestureRecognized if it is recognized.
Posture is a specialization of BaseGesture for postures.
Gesture is a specialization of BaseGesture for dynamic gestures.
GestureManager is a static mediator for all gestures. It will ease the use of the gesture recognition system as the user will have to subscribe only once to the GestureManager.GestureRecognized event and not to every gesture event.
BaseMapping<T> is a generic class that allow mapping a body on a T. It will use BaseGesture to start or stop the mapping.
To avoid plumbery of creating every gesture for every app or tests, we will finally add a IGestureFactory storing gestures to be used in an app.

classDiagram
class BaseGesture {
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized()
    +GestureName : string
}
class Gesture {
    +IsRecognitionRunning : bool
    #MinNbOfFrames : int
    #MaxNbOfFrames : int
    -currentFrameCount : int
    #TestInitialConditions(Body body)* bool
    #TestPosture(Body body)* bool
    #TestRunningGesture(Body body)* bool
    #TestEndConditions(Body body)* bool
    +TestGesture(Body body)*
}
class Posture {
    +TestGesture(Body body)*
    #TestPosture(Body body)* bool
}
class IGestureFactory {
    +CreateGestures() IEnumerable~BaseGesture~
}
<<interface>> IGestureFactory

class GestureManager {
    <<static>>
    +EventHandler~GestureRecognizedEventArgs~ GestureRecognized$ 
    +AddGestures(IGestureFactory: factory)$
    +AddGestures(params BaseGesture[])$
    +RemoveGesture(BaseGesture)$
    +StartAcquiringFrames(KinectManager)$
    +StopAcquiringFrame()$
}

BaseGesture <|-- Gesture
BaseGesture <|-- Posture
Posture <|-- PostureRightHandUp
Posture <|-- PostureTwoHandsDragon
Gesture <|-- SwipeRightHand
Gesture <|-- ClapHands
IGestureFactory "1" <-- GestureManager : Factory

class BaseMapping~T~ {
    +SubscribeToStartGesture(gesture: BaseGesture)
    +SubscribeToEndGesture(gesture: BaseGesture)
    +SubscribeToToggleGesture(gesture: BaseGesture)
    #Mapping(Body body)* T
    +TestMapping(Body body, out T ouput) bool
}

BaseMapping~T~ <|-- MapLeftHandToMouse : T <- Point

class MapLeftHandToMouse {
    #Mapping(Body body)* Point
}

BaseGesture <.. BaseMapping~T~ 
BaseGesture "*" <-- GestureManager : +KnownGestures

IGestureFactory <|.. AllGesturesFactory
GestureManager --> "1" KinectManager : KinectManager

Deeper informations

Here are more details about what we are going to do. Do not hesitate to come back to this section later.

BaseGesture: it represents the base class for every gesture, ie a posture (static position of a body) or a dynamic gesture (a movement). It owns a name and sends and event GestureRecognized when the gesture is ... recognized. To test the gesture, the type owns an abstract method TestGesture that will be overriden in the different subclasses.

classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}

Posture: this abstract subclass of BaseGesture is the base class for every posture (ie every static position of a body). It overrides the TestGesture method that will send the event GestureRecognized event if the TestPosture abstract method returns true. It is the base class for every other posture class of your coming application (ie PostureRightHandUp, PostureTwoHandsDragon).

classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class Posture {
    <<abstract>>
    +TestGesture(Body body)
    #TestPosture(Body body)* bool
}
BaseGesture <|-- Posture
Posture <|-- RightHandUp
Posture <|-- TwoHandsDragon

Gesture: it represents a dynamic gesture. It is also a subclass of BaseGesture and overrides the TestGesture method to fire the event GestureRecognized event, but its algorithm is a little bit more complicated, and based on four abstract methods that will be explained later. But here is a summary:
- TestInitialConditions: tests the body to see if the posture corresponds to a possible initial condition of this gesture
- TestPose: tests the body to see if its current pose is valid for this gesture (like a picture or a pause in a movie)
- TestDynamicPoses: tests the current pose but in comparison with the previous ones, to see if the dynamic of the movement corresponds to the gesture
- TestEndingConditions: tests the current pose to see if it corresponds to the end of the gesture.
- Moreover, it also checks that the gesture is done in more than a minimum number of frames (otherwise it is just noise) or in less than a maximum number of frames (otherwise, it is too slow to be intentional).
Note:
This algorithm is far from being efficient, but it is simple...

Subclasses of Gesture will only have to override these methods.

classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class Posture {
    <<abstract>>
    +TestGesture(Body body)
    #TestPosture(Body body)* bool
}
class Gesture {
    +IsRecognitionRunning : bool
    #MinNbOfFrames : int
    #MaxNbOfFrames : int
    -mCurrentFrameCount : int
    #TestInitialConditions(Body body)* bool
    #TestPosture(Body body)* bool
    #TestRunningGesture(Body body)* bool
    #TestEndConditions(Body body)* bool
    +TestGesture(Body body)*
}
BaseGesture <|-- Posture
BaseGesture <|-- Gesture
Gesture <|-- SwipeRightHand
Gesture <|-- ClapHands

To simplify the creation of different gestures and postures, we will use gesture factories. Every gesture factory implements the interface IGestureFactory whose contract asks for just one method CreateGestures returning the collection of BaseGesture to be tested in your app. This way, you won't have to test a hundred colliding gestures if you need only a dozen.

classDiagram
direction TB
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class IGestureFactory {
    <<interface>>
    +CreateGestures() IEnumerable~BaseGesture~
}
BaseGesture <.. IGestureFactory
IGestureFactory <|.. AllGesturesFactory

At last, to avoid the user to subscribe to multiple GestureRecognized events (one per gesture), the static class GestureManager plays the role of a mediator or a façade for our library, simplifying the subscription and use of the gestures. It also uses the previous factory.
One particular precious thing, is that it sends a static event GestureRecognized which you can subscribe to. This event is only one to which the user should subscribe. Its subscription will internally subscribes to all knownGestures. Easier and beautiful.

classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class IGestureFactory {
    <<interface>>
    +CreateGestures() IEnumerable~BaseGesture~
}

class GestureManager {
    <<static>>
    +EventHandler~GestureRecognizedEventArgs~ GestureRecognized$ 
    +StartAcquiringFrames()$
    +StopAcquiringFrames()$
}
BaseGesture "*" <-- GestureManager : -mKnownGestures
IGestureFactory "1" <-- GestureManager : Factory

Plan

Part1: Gesture Recognition (Postures)

The first part will focus on the basics of our gesture recognition system and postures.
In other words, you will have to code:

BaseGesture
Posture
GestureManager
and some concrete postures to be tested in a test

Part2: Gesture Recognition (Dynamic Gestures)

In the second part, you will code:

Gesture
some concrete gestures to be tested in a console app
MyGesturesBank
IGestureFactory
a concrete factory with the previous gestures and postures

Part3: Gesture Recognition (Mapping)

In this last part, you will have to code:

BaseMapping<T>
a concrete mapping
and a final test.

¹ Q&D stands for Quick & Dirty in tribute to Seattle Computer Products and QDOS

While preparing this practical works, I was listening to...

In The Court Of The Crimson King

King Crimson (1969)

12 KiB Raw Blame History