ml5js

Templates, examples and exercises

0) p5js online editor

Setup an account with the p5js online editor

1) Train a classification algorithm

Simple template: Webcam classifier. Alternative press and hold version

Exercise 1: Train a classifier to distinguish between two different classes. Make the sketch output a descriptive text for each class.

Webcam classifier + image

Webcam classifier + sound

Advanced example: Trainable Camera

Advanced example: Image Classifier flexible classes

Exercise 2: Try to change the Webcam classifier + image example so it outputs images that represent your two classes

2) MobileNet pretrained classification

Simple template: Mobilenet pretrained classification

Advanced example: Classification to speech

Advanced example: Poems about things desktop -> mobile -> About the project

ImageNet list of classes

Tensorflow.js YOLO multiple object detection (not in p5js editor)

3) Train a regression algorithm

Simple template: Webcam regressor

Advanced example: Playback rate camera

Advanced example: Control Mister Bean with regression

Exercise 3: Train the regressor and use the continuous output value slider.value() to control something. It is okay to leave the code as it is, and just focus on training the system

4) PoseNet pose detection

Simple template: Webcam PoseNet part selection single users

Advanced example: Classify poses

Advanced example: Draw with your nose

Exercise 4: Pick A or B
A) Edit the "Simple template" and use the x-y value of a bodypart to control something (e.g. the position of some images)
B) Use "Classify poses" as a starting point and make something happen for three different (yoga?) poses

5) Beyond ml5js: Sound, speech and face

Pretrained Facial Emotions (with clm tracker)

Facetracking + estimates of age, gender and mood with face-api source code (not in p5js editor)

Exercise 5A: Use one of the values from the four emotions (angryVal, sadVal, surprisedVal or happyVal) from the "Pretrained Facial Emotions" example to control something.

Speech recognition (with annyang.js)

Exercise 5B: Discuss ideas for a project you could build using speech recognition (a game, a service, a campaign site or something else) and try to build a small part of it to prove the interaction.

6) Start prototyping!

Exercise 6 (group):
Get into groups, do a quick brainstorm, pick a concept and build a small prototype that shows a key interaction of your idea. You can make something creative, useful and/or silly - up to you - just make sure it has a machine learning component somewhere! It is okay to build something on top of a previous exercise. How much coding you do is up to you, if you don't feel comfortable, you can choose to focus on training the interactions. Informal mini-presentations start at 16:20.