Sniff is a "Scratch-like" programming language that's designed to help Scratchers move gently from Scratch to more conventional languages. They can start writing programs, without having to learn a new language because Sniff is based on Scratch. They learn a little more about variables, compiling, syntax errors (!), and they can have fun controlling real hardware while they're doing it.

Wednesday, 8 July 2015

Playing Rock/Paper/Scissors with Sniff and the Leap Motion

This is the 100th post to the Sniff blog, and its a really fun one. The Leap Motion controller is like a Kinect for your hands (though the way it works is very different). It tracks your hands as you wave them around, and when it works its really amazing. In principle it can recognise the position of each joint of each finger and track it to milimeter accuracy. In practise it does that sometimes, then freaks out for a while, but its still super fun to play with.

The first thing I set out to do with it was teach it to recognise rock/paper/scissors - they're pretty distinct hand gestures, so we should be able to tell them apart. Assuming that you've already got the tracking software installed (you don't need the leap SDK), the next thing we do is set up a leap motion device in Sniff:

make leap leapmotion device
make handName string
make fingerNum number
make leapFound boolean
make leapPosX number
make leapPosY number
make leapPosZ number
make leapDirX number
make leapDirY number
make leapDirZ number
make leapVelX number
make leapVelY number
make leapVelZ number

That's a lot of variables - because the Leap produces a lot of data! However we're not going to be using the direction or the velocity in this example - just the positions.

when start
..set handName to ""
..set fingerNum to 0
..tell leap to "getPosition"
..if leapFound
...say join "Hand:" handName
...say join "X:" [leapPosX]
...say join "Y:" [leapPosY]
...say join "Z:" [leapPosZ]
..wait 0.5 secs

To get started here's the basic code to track a hand. We don't know which hand (yet), so we set the handName to "", and we're not interested in a specific finger, so we set fingerNum to 0. Then we ask the leap to do its thing!

If it finds a hand, then it tells is which it found, and fills in all the other variables, so we can print the hands position. EASY! You could take the XY positions and use that to move a sprite round the screen in a game!

We can use fingerNum to track specific fingers, so we're pretty much ready to go and build R/P/S. The easiest way to do this is figure the distance that each finger tip is away from the centre of the hand - big number means extended, small number means closed in.

make centerX number
make centerY number
make centerZ number

make indexDist number
make middleDist number
make ringDist number
make pinkyDist number

when start
..set handName to ""
..set fingerNum to 0
..tell leap to "getPosition"
..if leapFound
...set centerX to leapPosX
...set centerY to leapPosY
...set centerZ to leapPosZ
...set fingerNum to 2
...tell leap to "getPosition"
...set indexDist to sqrt of ((leapPosX-centerX)*(leapPosX-centerX)+(leapPosY-centerY)*(leapPosY-centerY)+(leapPosZ-centerZ)*(leapPosZ-centerZ))
...set fingerNum to 3
...tell leap to "getPosition"
...set middleDist to sqrt of ((leapPosX-centerX)*(leapPosX-centerX)+(leapPosY-centerY)*(leapPosY-centerY)+(leapPosZ-centerZ)*(leapPosZ-centerZ))
...set fingerNum to 4
...tell leap to "getPosition"
...set ringDist to sqrt of ((leapPosX-centerX)*(leapPosX-centerX)+(leapPosY-centerY)*(leapPosY-centerY)+(leapPosZ-centerZ)*(leapPosZ-centerZ))
...set fingerNum to 5
...tell leap to "getPosition"
...set pinkyDist to sqrt of ((leapPosX-centerX)*(leapPosX-centerX)+(leapPosY-centerY)*(leapPosY-centerY)+(leapPosZ-centerZ)*(leapPosZ-centerZ))

We start by getting a hand. We then go through and get the position for each finger, and use pythagoras to work out the distance from the hand to the finger tip. Note that asking for the hand position will set a handName, which means that when we ask for fingers (2-5 - we ignore the thumb) we're using the hand we've already found.

Finally we just need to parse those distances to check for each of the hand gestures:

...set threshold to 70
...if indexDist<threshold and middleDist<threshold and ringDist<threshold and pinkyDist<threshold
....say "Rock!"
...if indexDist>threshold and middleDist>threshold and ringDist>threshold and pinkyDist>threshold
....say "Paper!"
...if indexDist>threshold and middleDist>threshold and ringDist<threshold and pinkyDist<threshold
....say "Scissors!"
...wait 1 secs

Distances are in mm, and for my (large-ish!) hands I found 70m worked well as a cut of point between fingers being "in" or extended - adjust based on personal physique! If all the fingers are less than threshold then its "Rock!". If index and middle are the only two extended its "Scissors!", and if all are extended its "Paper!".

The results are surprisingly successful! It occasionally does loose track but its still pretty impressive (it can be helpful to just view the gestures in the Leap "visualizer" tool - if your gestures aren't clear in the visualiser then your own sw isn't going to stand a chance!). We've cut down the Leap API to produce something that's really easy to program, but still keeps most of the potential. The code for this is included in the current Sniff release. We've tested on a Mac, but it should work on other platforms - our older Windows and Linux build boxes are too slow to run the leap motion tracker, so let us know how you get on!

Playing Rock Paper Scissors with a computer is REALLY fun!

No comments:

Post a Comment