Kinect-Project/exo2_subject.md

[Home](./README.md)
| [Exercise 1 - Kinect Streams](./exo1_subject.md)
| **Exercise 2 - Introduction**
| [Exercise 2 part 1 - Postures](./exo2_1_subject.md) 
| [Exercise 2 part 2 - Gestures](./exo2_2_subject.md) 
| [Exercise 2 part 3 - Mapping](./exo2_3_subject.md) 
| [Exercise 3 - Free App](./exo3_subject.md)

# Exercise 2: Gesture recognition

*Advised duration*: 4 hours  
*Maximum advised duration*: 6 hours

This exercise can be splitted in three parts but they are all bound and you should do them in the right order.  
  
The main objective is to develop a simple gesture recognition system. It is not based on AI or *Machine Learning*. It is rather made of **Q&D**¹ solutions.  

# Just a glimpse on the work to be done

Our **gesture recognition** will be able to:
- recognize **postures** _ie_ static body poses (your body does not move, it is tested on a single frame),
- recognize **gestures** _ie_ dynamic body movements (your body does move, it is tested during several frames),
- do **mapping** of body movement on whatever computer thing (a cursor, an ellipse, a dinosaur...).

You will have to develop the following assemblies (except *Exercise3.WpfApp*, and the Nuget Packages *Microsoft.Kinect* and *CommunityToolkit.Mvvm*):
```mermaid
flowchart LR
subgraph Exercise2
    MyGesturesBank
    KinectUtils
    Tests
end
MyGesturesBank -.-> KinectUtils 
KinectUtils -.-> Microsoft.Kinect
KinectUtils -.-> CommunityToolkit.Mvvm
Tests -.-> MyGesturesBank
Tests -.-> KinectUtils
style Microsoft.Kinect fill:#eee,color:#111,stroke:#999
style CommunityToolkit.Mvvm fill:#eee,color:#111,stroke:#999
subgraph Exercise3
    WpfApp
end
WpfApp -.-> MyGesturesBank
WpfApp -.-> KinectUtils
```

**KinectUtils** will contain all the base types to do gesture recognition and mapping. It will be updated in the three parts of this exercise.  

**MyGesturesBank** will contain specific implementation of postures and gestures that could then be used in different applications.  
  
**Tests** are only one or several *Console app* tests to verify that our system is working well.  

Here is just a glimpse of the classes you will develop. More details will be given later or by your teacher on demand:
- ```BaseGesture``` will represent any gesture, _ie_ any ```Posture``` or ```Gesture``` (dynamic). To be tested, you will have to call ```TestGesture``` that will fire the event ```GestureRecognized``` if it is recognized.
- ```Posture``` is a specialization of ```BaseGesture``` for postures.
- ```Gesture``` is a specialization of ```BaseGesture``` for dynamic gestures. 
- ```GestureManager``` is a static mediator for all gestures. It will ease the use of the gesture recognition system as the user will have to subscribe only once to the ```GestureManager.GestureRecognized``` event and not to every gesture event.
- ```BaseMapping<T>``` is a generic class that allow mapping a body on a ```T```. It will use ```BaseGesture``` to start or stop the mapping.  
- To avoid plumbery of creating every gesture for every app or tests, we will finally add a ```IGestureFactory``` storing gestures to be used in an app.  
  
```mermaid  
classDiagram
class BaseGesture {
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized()
    +GestureName : string
}
class Gesture {
    +IsRecognitionRunning : bool
    #MinNbOfFrames : int
    #MaxNbOfFrames : int
    -currentFrameCount : int
    #TestInitialConditions(Body body)* bool
    #TestPosture(Body body)* bool
    #TestRunningGesture(Body body)* bool
    #TestEndConditions(Body body)* bool
    +TestGesture(Body body)*
}
class Posture {
    +TestGesture(Body body)*
    #TestPosture(Body body)* bool
}
class IGestureFactory {
    +CreateGestures() IEnumerable~BaseGesture~
}
<<interface>> IGestureFactory

class GestureManager {
    <<static>>
    +EventHandler~GestureRecognizedEventArgs~ GestureRecognized$ 
    +AddGestures(IGestureFactory: factory)$
    +AddGestures(params BaseGesture[])$
    +RemoveGesture(BaseGesture)$
    +StartAcquiringFrames(KinectManager)$
    +StopAcquiringFrame()$
}

BaseGesture <|-- Gesture
BaseGesture <|-- Posture
Posture <|-- PostureRightHandUp
Posture <|-- PostureTwoHandsDragon
Gesture <|-- SwipeRightHand
Gesture <|-- ClapHands
IGestureFactory "1" <-- GestureManager : Factory

class BaseMapping~T~ {
    +SubscribeToStartGesture(gesture: BaseGesture)
    +SubscribeToEndGesture(gesture: BaseGesture)
    +SubscribeToToggleGesture(gesture: BaseGesture)
    #Mapping(Body body)* T
    +TestMapping(Body body, out T ouput) bool
}

BaseMapping~T~ <|-- MapLeftHandToMouse : T <- Point

class MapLeftHandToMouse {
    #Mapping(Body body)* Point
}

BaseGesture <.. BaseMapping~T~ 
BaseGesture "*" <-- GestureManager : +KnownGestures

IGestureFactory <|.. AllGesturesFactory
GestureManager --> "1" KinectManager : KinectManager

```  

## Deeper informations
Here are more details about what we are going to do. **Do not hesitate to come back to this section later**.  

- ```BaseGesture```: it represents the base class for every gesture, _ie_ a **posture** (static position of a body) or a **dynamic gesture** (a movement). It owns a name and sends and event ```GestureRecognized``` when the gesture is ... recognized. To test the gesture, the type owns an abstract method ```TestGesture``` that will be overriden in the different subclasses.
```mermaid  
classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
```
- ```Posture```: this abstract subclass of ```BaseGesture``` is the base class for every **posture** (_ie_ every static position of a body). It overrides the ```TestGesture``` method that will send the event ```GestureRecognized``` event if the ```TestPosture``` abstract method returns ```true```. It is the base class for every other **posture** class of your coming application (_ie_ ```PostureRightHandUp```, ```PostureTwoHandsDragon```).
```mermaid  
classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class Posture {
    <<abstract>>
    +TestGesture(Body body)
    #TestPosture(Body body)* bool
}
BaseGesture <|-- Posture
Posture <|-- RightHandUp
Posture <|-- TwoHandsDragon
```
- ```Gesture```: it represents a **dynamic gesture**. It is also a subclass of ```BaseGesture``` and overrides the ```TestGesture``` method to fire the event ```GestureRecognized``` event, but its algorithm is a little bit more complicated, and based on four abstract methods that will be explained later. But here is a summary:  
  - ```TestInitialConditions```: tests the body to see if the posture corresponds to a possible initial condition of this gesture
  - ```TestPose```: tests the body to see if its current pose is valid for this gesture (like a picture or a pause in a movie)
  - ```TestDynamicPoses```: tests the current pose but in comparison with the previous ones, to see if the dynamic of the movement corresponds to the gesture
  - ```TestEndingConditions```: tests the current pose to see if it corresponds to the end of the gesture.  
  - Moreover, it also checks that the gesture is done in more than a minimum number of frames (otherwise it is just noise) or in less than a maximum number of frames (otherwise, it is too slow to be intentional).
  > Note:  
  > This algorithm is far from being efficient, but it is simple...   

  Subclasses of ```Gesture``` will only have to override these methods.  
```mermaid  
classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class Posture {
    <<abstract>>
    +TestGesture(Body body)
    #TestPosture(Body body)* bool
}
class Gesture {
    +IsRecognitionRunning : bool
    #MinNbOfFrames : int
    #MaxNbOfFrames : int
    -mCurrentFrameCount : int
    #TestInitialConditions(Body body)* bool
    #TestPosture(Body body)* bool
    #TestRunningGesture(Body body)* bool
    #TestEndConditions(Body body)* bool
    +TestGesture(Body body)*
}
BaseGesture <|-- Posture
BaseGesture <|-- Gesture
Gesture <|-- SwipeRightHand
Gesture <|-- ClapHands
```
- To simplify the creation of different gestures and postures, we will use **gesture factories**. Every gesture factory implements the interface ```IGestureFactory``` whose contract asks for just one method ```CreateGestures``` returning the collection of ```BaseGesture``` to be tested in your app. This way, you won't have to test a hundred colliding gestures if you need only a dozen.
```mermaid  
classDiagram
direction TB
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class IGestureFactory {
    <<interface>>
    +CreateGestures() IEnumerable~BaseGesture~
}
BaseGesture <.. IGestureFactory
IGestureFactory <|.. AllGesturesFactory
```
- At last, to avoid the user to subscribe to multiple ```GestureRecognized``` events (one per gesture), the static class ```GestureManager``` plays the role of a mediator or a façade for our library, simplifying the subscription and use of the gestures. It also uses the previous factory.  
One particular precious thing, is that it sends a static event ```GestureRecognized``` which you can subscribe to. This event is only one to which the user should subscribe. Its subscription will internally subscribes to all ```knownGestures```. Easier and beautiful.   
```mermaid  
classDiagram
class BaseGesture {
    <<abstract>>
    +TestGesture(Body body)*
    +GestureRecognized : EventHandler~GestureRecognizedEventArgs~
    #OnGestureRecognized(Body)
    +GestureName : string
}
class IGestureFactory {
    <<interface>>
    +CreateGestures() IEnumerable~BaseGesture~
}

class GestureManager {
    <<static>>
    +EventHandler~GestureRecognizedEventArgs~ GestureRecognized$ 
    +StartAcquiringFrames()$
    +StopAcquiringFrames()$
}
BaseGesture "*" <-- GestureManager : -mKnownGestures
IGestureFactory "1" <-- GestureManager : Factory
```  

#  Plan

## Part1: Gesture Recognition (Postures)
The first part will focus on the basics of our gesture recognition system and postures.  
In other words, you will have to code:
- ```BaseGesture```
- ```Posture```
- ```GestureManager```
- and some concrete postures to be tested in a test

## Part2: Gesture Recognition (Dynamic Gestures)
In the second part, you will code:
- ```Gesture```
- some concrete gestures to be tested in a console app
- *MyGesturesBank*
- ```IGestureFactory``` 
- a concrete factory with the previous gestures and postures

## Part3: Gesture Recognition (Mapping)
In this last part, you will have to code:
- ```BaseMapping<T>```
- a concrete mapping
- and a final test.

--- 

¹ *Q&D stands for Quick & Dirty in tribute to Seattle Computer Products and QDOS*

  
---

[Home](./README.md)
| [Exercise 1 - Kinect Streams](./exo1_subject.md)
| **Exercise 2 - Introduction**
| [Exercise 2 part 1 - Postures](./exo2_1_subject.md) 
| [Exercise 2 part 2 - Gestures](./exo2_2_subject.md) 
| [Exercise 2 part 3 - Mapping](./exo2_3_subject.md) 
| [Exercise 3 - Free App](./exo3_subject.md)

---

Copyright © 2023-2024 Marc Chevaldonné

--- 

While preparing this practical works, I was listening to...  

<table>
    <tr>
        <td>
            <img src="./images/in_the_court_of_the_crimson_king.jpg" width="120"/>
        </td>
        <td>
            <div>
                <p><b>In The Court Of The Crimson King</b></p>
                <p><i>King Crimson</i> (1969)</p>
            </div>
        </td>
    </tr>
</table>