[Home](./README.md) | [Exercise 1 - Kinect Streams](./exo1_subject.md) | **Exercise 2 - Introduction** | [Exercise 2 part 1 - Postures](./exo2_1_subject.md) | [Exercise 2 part 2 - Gestures](./exo2_2_subject.md) | [Exercise 2 part 3 - Mapping](./exo2_3_subject.md) | [Exercise 3 - Free App](./exo3_subject.md) # Exercise 2: Gesture recognition *Advised duration*: 4 hours *Maximum advised duration*: 6 hours This exercise can be splitted in three parts but they are all bound and you should do them in the right order. The main objective is to develop a simple gesture recognition system. It is not based on AI or *Machine Learning*. It is rather made of **Q&D**¹ solutions. # Just a glimpse on the work to be done Our **gesture recognition** will be able to: - recognize **postures** _ie_ static body poses (your body does not move, it is tested on a single frame), - recognize **gestures** _ie_ dynamic body movements (your body does move, it is tested during several frames), - do **mapping** of body movement on whatever computer thing (a cursor, an ellipse, a dinosaur...). You will have to develop the following assemblies (except *Exercise3.WpfApp*, and the Nuget Packages *Microsoft.Kinect* and *CommunityToolkit.Mvvm*): ```mermaid flowchart LR subgraph Exercise2 MyGesturesBank KinectUtils Tests end MyGesturesBank -.-> KinectUtils KinectUtils -.-> Microsoft.Kinect KinectUtils -.-> CommunityToolkit.Mvvm Tests -.-> MyGesturesBank Tests -.-> KinectUtils style Microsoft.Kinect fill:#eee,color:#111,stroke:#999 style CommunityToolkit.Mvvm fill:#eee,color:#111,stroke:#999 subgraph Exercise3 WpfApp end WpfApp -.-> MyGesturesBank WpfApp -.-> KinectUtils ``` **KinectUtils** will contain all the base types to do gesture recognition and mapping. It will be updated in the three parts of this exercise. **MyGesturesBank** will contain specific implementation of postures and gestures that could then be used in different applications. **Tests** are only one or several *Console app* tests to verify that our system is working well. Here is just a glimpse of the classes you will develop. More details will be given later or by your teacher on demand: - ```BaseGesture``` will represent any gesture, _ie_ any ```Posture``` or ```Gesture``` (dynamic). To be tested, you will have to call ```TestGesture``` that will fire the event ```GestureRecognized``` if it is recognized. - ```Posture``` is a specialization of ```BaseGesture``` for postures. - ```Gesture``` is a specialization of ```BaseGesture``` for dynamic gestures. - ```GestureManager``` is a static mediator for all gestures. It will ease the use of the gesture recognition system as the user will have to subscribe only once to the ```GestureManager.GestureRecognized``` event and not to every gesture event. - ```BaseMapping``` is a generic class that allow mapping a body on a ```T```. It will use ```BaseGesture``` to start or stop the mapping. - To avoid plumbery of creating every gesture for every app or tests, we will finally add a ```IGestureFactory``` storing gestures to be used in an app. ```mermaid classDiagram class BaseGesture { +TestGesture(Body body)* +GestureRecognized : EventHandler~GestureRecognizedEventArgs~ #OnGestureRecognized() +GestureName : string } class Gesture { +IsRecognitionRunning : bool #MinNbOfFrames : int #MaxNbOfFrames : int -currentFrameCount : int #TestInitialConditions(Body body)* bool #TestPosture(Body body)* bool #TestRunningGesture(Body body)* bool #TestEndConditions(Body body)* bool +TestGesture(Body body)* } class Posture { +TestGesture(Body body)* #TestPosture(Body body)* bool } class IGestureFactory { +CreateGestures() IEnumerable~BaseGesture~ } <> IGestureFactory class GestureManager { <> +EventHandler~GestureRecognizedEventArgs~ GestureRecognized$ +AddGestures(IGestureFactory: factory)$ +AddGestures(params BaseGesture[])$ +RemoveGesture(BaseGesture)$ +StartAcquiringFrames(KinectManager)$ +StopAcquiringFrame()$ } BaseGesture <|-- Gesture BaseGesture <|-- Posture Posture <|-- PostureRightHandUp Posture <|-- PostureTwoHandsDragon Gesture <|-- SwipeRightHand Gesture <|-- ClapHands IGestureFactory "1" <-- GestureManager : Factory class BaseMapping~T~ { +SubscribeToStartGesture(gesture: BaseGesture) +SubscribeToEndGesture(gesture: BaseGesture) +SubscribeToToggleGesture(gesture: BaseGesture) #Mapping(Body body)* T +TestMapping(Body body, out T ouput) bool } BaseMapping~T~ <|-- MapLeftHandToMouse : T <- Point class MapLeftHandToMouse { #Mapping(Body body)* Point } BaseGesture <.. BaseMapping~T~ BaseGesture "*" <-- GestureManager : +KnownGestures IGestureFactory <|.. AllGesturesFactory GestureManager --> "1" KinectManager : KinectManager ``` ## Deeper informations Here are more details about what we are going to do. **Do not hesitate to come back to this section later**. - ```BaseGesture```: it represents the base class for every gesture, _ie_ a **posture** (static position of a body) or a **dynamic gesture** (a movement). It owns a name and sends and event ```GestureRecognized``` when the gesture is ... recognized. To test the gesture, the type owns an abstract method ```TestGesture``` that will be overriden in the different subclasses. ```mermaid classDiagram class BaseGesture { <> +TestGesture(Body body)* +GestureRecognized : EventHandler~GestureRecognizedEventArgs~ #OnGestureRecognized(Body) +GestureName : string } ``` - ```Posture```: this abstract subclass of ```BaseGesture``` is the base class for every **posture** (_ie_ every static position of a body). It overrides the ```TestGesture``` method that will send the event ```GestureRecognized``` event if the ```TestPosture``` abstract method returns ```true```. It is the base class for every other **posture** class of your coming application (_ie_ ```PostureRightHandUp```, ```PostureTwoHandsDragon```). ```mermaid classDiagram class BaseGesture { <> +TestGesture(Body body)* +GestureRecognized : EventHandler~GestureRecognizedEventArgs~ #OnGestureRecognized(Body) +GestureName : string } class Posture { <> +TestGesture(Body body) #TestPosture(Body body)* bool } BaseGesture <|-- Posture Posture <|-- RightHandUp Posture <|-- TwoHandsDragon ``` - ```Gesture```: it represents a **dynamic gesture**. It is also a subclass of ```BaseGesture``` and overrides the ```TestGesture``` method to fire the event ```GestureRecognized``` event, but its algorithm is a little bit more complicated, and based on four abstract methods that will be explained later. But here is a summary: - ```TestInitialConditions```: tests the body to see if the posture corresponds to a possible initial condition of this gesture - ```TestPose```: tests the body to see if its current pose is valid for this gesture (like a picture or a pause in a movie) - ```TestDynamicPoses```: tests the current pose but in comparison with the previous ones, to see if the dynamic of the movement corresponds to the gesture - ```TestEndingConditions```: tests the current pose to see if it corresponds to the end of the gesture. - Moreover, it also checks that the gesture is done in more than a minimum number of frames (otherwise it is just noise) or in less than a maximum number of frames (otherwise, it is too slow to be intentional). > Note: > This algorithm is far from being efficient, but it is simple... Subclasses of ```Gesture``` will only have to override these methods. ```mermaid classDiagram class BaseGesture { <> +TestGesture(Body body)* +GestureRecognized : EventHandler~GestureRecognizedEventArgs~ #OnGestureRecognized(Body) +GestureName : string } class Posture { <> +TestGesture(Body body) #TestPosture(Body body)* bool } class Gesture { +IsRecognitionRunning : bool #MinNbOfFrames : int #MaxNbOfFrames : int -mCurrentFrameCount : int #TestInitialConditions(Body body)* bool #TestPosture(Body body)* bool #TestRunningGesture(Body body)* bool #TestEndConditions(Body body)* bool +TestGesture(Body body)* } BaseGesture <|-- Posture BaseGesture <|-- Gesture Gesture <|-- SwipeRightHand Gesture <|-- ClapHands ``` - To simplify the creation of different gestures and postures, we will use **gesture factories**. Every gesture factory implements the interface ```IGestureFactory``` whose contract asks for just one method ```CreateGestures``` returning the collection of ```BaseGesture``` to be tested in your app. This way, you won't have to test a hundred colliding gestures if you need only a dozen. ```mermaid classDiagram direction TB class BaseGesture { <> +TestGesture(Body body)* +GestureRecognized : EventHandler~GestureRecognizedEventArgs~ #OnGestureRecognized(Body) +GestureName : string } class IGestureFactory { <> +CreateGestures() IEnumerable~BaseGesture~ } BaseGesture <.. IGestureFactory IGestureFactory <|.. AllGesturesFactory ``` - At last, to avoid the user to subscribe to multiple ```GestureRecognized``` events (one per gesture), the static class ```GestureManager``` plays the role of a mediator or a façade for our library, simplifying the subscription and use of the gestures. It also uses the previous factory. One particular precious thing, is that it sends a static event ```GestureRecognized``` which you can subscribe to. This event is only one to which the user should subscribe. Its subscription will internally subscribes to all ```knownGestures```. Easier and beautiful. ```mermaid classDiagram class BaseGesture { <> +TestGesture(Body body)* +GestureRecognized : EventHandler~GestureRecognizedEventArgs~ #OnGestureRecognized(Body) +GestureName : string } class IGestureFactory { <> +CreateGestures() IEnumerable~BaseGesture~ } class GestureManager { <> +EventHandler~GestureRecognizedEventArgs~ GestureRecognized$ +StartAcquiringFrames()$ +StopAcquiringFrames()$ } BaseGesture "*" <-- GestureManager : -mKnownGestures IGestureFactory "1" <-- GestureManager : Factory ``` # Plan ## Part1: Gesture Recognition (Postures) The first part will focus on the basics of our gesture recognition system and postures. In other words, you will have to code: - ```BaseGesture``` - ```Posture``` - ```GestureManager``` - and some concrete postures to be tested in a test ## Part2: Gesture Recognition (Dynamic Gestures) In the second part, you will code: - ```Gesture``` - some concrete gestures to be tested in a console app - *MyGesturesBank* - ```IGestureFactory``` - a concrete factory with the previous gestures and postures ## Part3: Gesture Recognition (Mapping) In this last part, you will have to code: - ```BaseMapping``` - a concrete mapping - and a final test. --- ¹ *Q&D stands for Quick & Dirty in tribute to Seattle Computer Products and QDOS* --- [Home](./README.md) | [Exercise 1 - Kinect Streams](./exo1_subject.md) | **Exercise 2 - Introduction** | [Exercise 2 part 1 - Postures](./exo2_1_subject.md) | [Exercise 2 part 2 - Gestures](./exo2_2_subject.md) | [Exercise 2 part 3 - Mapping](./exo2_3_subject.md) | [Exercise 3 - Free App](./exo3_subject.md) --- Copyright © 2023-2024 Marc Chevaldonné --- While preparing this practical works, I was listening to...

In The Court Of The Crimson King

King Crimson (1969)