{"items": [{"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928523893552", "anchor": "fb-928523893552", "service": "fb", "text": "David Bronwyn Bryce", "timestamp": "1519313262"}, {"author": "Rohin", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982", "anchor": "fb-928536652982", "service": "fb", "text": "Some thoughts:
- The simplest thing that could work is to finetune pretrained ImageNet weights for your task. I haven't done this myself but I expect it's quite easy to set up. This isn't a great fit for your task, because the convolutions used in image classifiers are meant to ignore the positions of particular objects in the image, whereas you want to look at the relative positions of your fingers to identify the chord. That said, people have successfully used ImageNet features to do eg. optical flow, so it can still work. You can then run it on the input camera image as often as possible while still being realtime.
- It might be hard to get an accurate model that runs fast enough to work in real time. On a CPU classifying an image with a good model can take on the order of seconds. There are slimmed down versions like MobileNet but that could sacrifice accuracy.
- Good models on ImageNet still have non-negligible test error, enough that it would be unacceptable for use in live performance. However, ImageNet data is pretty messy (human performance on ImageNet is ~95%), and your task seems a lot easier. I'd guess you can get near 100% accuracy with the simple approach, and it seems especially likely that you could have the model say \"I don't know\" (if the model is < 70% confident in its answer, say) instead of making a mistake.
- I'm more worried about distribution shift. I'd be more confident in it working if you took videos of you playing the mandolin normally (ideally during practice performances in different locations) and labeled that. That's decidedly more effort of course, but it eliminates annoying problems like the classifier only being able to recognize chords when you play them slowly and carefully.", "timestamp": "1519315740"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928538304672", "anchor": "fb-928536652982_928538304672", "service": "fb", "text": "→ Taking substantial fractions of a second to run the classification would be a problem. This would need to run in a few ms at most.", "timestamp": "1519316673"}, {"author": "Rohin", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928545440372", "anchor": "fb-928536652982_928545440372", "service": "fb", "text": "→ MobileNet might get down to a few ms? I don't know much about model minification.
You might be able to parallelize, though this would not be an entry-level tool, because parallelization usually means having multiple cores each processing a different image, not having multiple cores processing the same image.", "timestamp": "1519319962"}, {"author": "Ben", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928664257262", "anchor": "fb-928536652982_928664257262", "service": "fb", "text": "→ Just use some hardware with a gpu.
But regardless, I don't think transfer learning will work here. As you mentioned, the whole point of convolutional neural nets is that they pick out the shape of an object regardless of it's location in the image.
This is not a good application for this type of technology.", "timestamp": "1519365563"}, {"author": "Rohin", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928724127282", "anchor": "fb-928536652982_928724127282", "service": "fb", "text": "→ GPUs give you speedup by processing multiple training points in parallel, drastically increasing throughput, but this doesn't help with latency. You probably *could* use a GPU to improve latency too but I'd bet it's a fair amount of work -- you can't just tell Tensorflow \"run this on the GPU\" and expect to get lower latency.", "timestamp": "1519407944"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928725025482", "anchor": "fb-928536652982_928725025482", "service": "fb", "text": "→ Ben: \"pick out the shape of an object regardless of it's location in the image\"
That's useful for two big components of this problem:
* Identify the neck regardless of where it is in the image
* Identify the hand shape regardless of where it is on the neck
Then you also need to identify how far up the neck the hand is, though.", "timestamp": "1519408460"}, {"author": "Ben", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928735863762", "anchor": "fb-928536652982_928735863762", "service": "fb", "text": "→ Well I suppose you could use a NN for image segmentation to put a bounding box/centerline on each part, then feed that into a different algorithm (or simple MLP).
You'd still need a lot of hand labeled data for training, though. A couple thousand images may suffice.
For hardware, check out Nvidia's Jetson. The onboard gpu was designed for feed forward inference. It's used with NN driven drones using onboard video.", "timestamp": "1519411492"}, {"author": "Ben", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536652982&reply_comment_id=928741257952", "anchor": "fb-928536652982_928741257952", "service": "fb", "text": "→ Eh, I take it back. Neural nets are universal approximators; no need to feed it into a different algorithm. A couple dense layers at the end may just do the trick.", "timestamp": "1519413981"}, {"author": "Andrew", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536902482", "anchor": "fb-928536902482", "service": "fb", "text": "midi?", "timestamp": "1519315879"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536902482&reply_comment_id=928538224832", "anchor": "fb-928536902482_928538224832", "service": "fb", "text": "→ Are you suggesting I put sensors on the mandolin neck?", "timestamp": "1519316621"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536902482&reply_comment_id=928538354572", "anchor": "fb-928536902482_928538354572", "service": "fb", "text": "→ Actually, tiny buttons on the back of the neck would be an interesting approach; what else is my thumb for?", "timestamp": "1519316716"}, {"author": "David", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536902482&reply_comment_id=928545744762", "anchor": "fb-928536902482_928545744762", "service": "fb", "text": "→ Are the strings conductive? Maybe you could run thin wires between the frets, and use the strings to close circuits.", "timestamp": "1519320049"}, {"author": "Andrew", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536902482&reply_comment_id=928554477262", "anchor": "fb-928536902482_928554477262", "service": "fb", "text": "→ I was thinking that midi input from a pickup might be sufficient. I'm not an expert, but I googled for mandolin midi and saw that some people were doing that. I figured it would be easier than visually recognizing chord shapes from your left hand", "timestamp": "1519322394"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928536902482&reply_comment_id=928575729672", "anchor": "fb-928536902482_928575729672", "service": "fb", "text": "→ Do you mean an audio pickup? Because that can't do midi except by interpreting the audio and trying to deduce the notes being played.", "timestamp": "1519327509"}, {"author": "Mark", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928538668942", "anchor": "fb-928538668942", "service": "fb", "text": "I wonder if it would be easier to use a foot-activated keyboard and just play the desired bass notes with your feet, like an organ.", "timestamp": "1519316882"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928538668942&reply_comment_id=928539053172", "anchor": "fb-928538668942_928539053172", "service": "fb", "text": "→ I've tried this, and I'm not good enough at it. I would need one thing to select the note, and another thing to trigger it.", "timestamp": "1519317043"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928538668942&reply_comment_id=928540959352", "anchor": "fb-928538668942_928540959352", "service": "fb", "text": "→ I might be able to make the two foot version of this pretty well on my own, actually...
One set of midi pedals for selecting the note (12 is nice, could do fewer) then my existing setup for triggering it.", "timestamp": "1519318102"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928538858562", "anchor": "fb-928538858562", "service": "fb", "text": "Hmm, maybe something like http://www.instructables.com/.../DIY-Guitar-Midi.../ would be better", "timestamp": "1519317016"}, {"author": "Greg", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928538858562&reply_comment_id=928540240792", "anchor": "fb-928538858562_928540240792", "service": "fb", "text": "→ This has too many problems to be useful, but it's a neat idea. The thing he links, the solange 6, that has an idea that might actually work but it is definitely both costly and complicated to implement. It's \"proprietary\" but from \"Industrial Radio's proprietry Fretsense technology reads pitch directly from the frets rather than using outdated pitch-to-voltage techniques.\" I am guessing that they have a piezoelectric pickup in every fret, which solves all of the polyphony problems and all of the extra noise and unsafe voltages running on the strings.", "timestamp": "1519317792"}, {"author": "Greg", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928538858562&reply_comment_id=928541084102", "anchor": "fb-928538858562_928541084102", "service": "fb", "text": "→ I've done installations of systems like this: http://www.graphtech.com/products/brands/ghost or this: https://www.sweetwater.com/.../GK3--roland-gk-3-divided...
Noticeably not perfect, but they do an alright job if you aren't playing complicated chords.
Installing a piezo pickup on a mandolin.... remove your frets, route a space for wiring, install the piezos + wires, and ... man that's a complicated setup.
BUT I still like it way more than getting the note via voltage on a charged wire.", "timestamp": "1519318161"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928539871532", "anchor": "fb-928539871532", "service": "fb", "text": "The first sanity check is to take some video and confirm that you can extract the chords by sight from still frames. One possible problem that comes to mind is that it may be difficult to distinguish between a finger hovering over an open string, and that same finger pressed down on that string, especially if the camera is pointing straight down the -z axis.
Latency is going to be a challenge. It's not obvious to me that the correct latency is greater than zero; it might be that the correct timing to start a bass note is milliseconds prior to the mandolin chord. This should be pretty easy to determine on a multi-track mixer. Your hands may have enough lead time to make that work, though.
The other problem is jitter. You're going to want to extract the beat from audio, and time chords to that, rather than try to get the beat from the video.", "timestamp": "1519317449"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928539871532&reply_comment_id=928540590092", "anchor": "fb-928539871532_928540590092", "service": "fb", "text": "→ I can definitely do this by sight for the particular chords I play, which are very easy to distinguish from each other. Doing the fully general version is hard and I don't think is worth it.
The bass note definitely needs to start before the mandolin chord sounds, but I think it still starts after the fingers are in place to play the chord, or at least I could play that way.
I'm planning to get the beat from my foot; see the post.", "timestamp": "1519317987"}, {"author": "Bil", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928540470332", "anchor": "fb-928540470332", "service": "fb", "text": "Who cares if it's practical? It's a totally cool thing to do.", "timestamp": "1519317862"}, {"author": "Danner", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928544861532", "anchor": "fb-928544861532", "service": "fb", "text": "https://youtu.be/qv6UVOQ0F44
This feels like the type of thing that could be gleaned from an image based network.
Or was it this one that looked at an image... https://youtu.be/Ipi40cb_RsI", "timestamp": "1519319555"}, {"author": "Ben", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928549562112", "anchor": "fb-928549562112", "service": "fb", "text": "Oh wow! I\u2019d love to train a network to turn music I like into sheet music!", "timestamp": "1519321668"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928549562112&reply_comment_id=928575909312", "anchor": "fb-928549562112_928575909312", "service": "fb", "text": "→ This is already a thing people are working on, and have been for a while.", "timestamp": "1519327554"}, {"author": "Bob", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512", "anchor": "fb-928558843512", "service": "fb", "text": "i\u2019m a geek from way back \u2013 put a midi pickup on my Taylor 812c in 1986 \u2013 but, that said... listen to the bass on Walking in Memphis (just to pick One thing i\u2019ve heard so far today) and tell me why this is still a good idea.", "timestamp": "1519323809"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512&reply_comment_id=928576293542", "anchor": "fb-928558843512_928576293542", "service": "fb", "text": "→ The idea is that sometimes I use a kick drum along side my mandolin playing, and I think it would be musically interesting to be able to tune it to match my playing on the fly.", "timestamp": "1519327654"}, {"author": "Hollis", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512&reply_comment_id=928643154552", "anchor": "fb-928558843512_928643154552", "service": "fb", "text": "→ (just for clarity: you're saying that WiM uses this style of bassline, and that it's un-compelling?)", "timestamp": "1519356082"}, {"author": "Bob", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512&reply_comment_id=928643673512", "anchor": "fb-928558843512_928643673512", "service": "fb", "text": "→ no, quite the reverse. there are numerous elements of that bassline that no machine is going to play for 20 years.", "timestamp": "1519356427"}, {"author": "Bob", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512&reply_comment_id=928647760322", "anchor": "fb-928558843512_928647760322", "service": "fb", "text": "→ and just picking out, say, the root note of the current chord gets nowhere near it, nor most decent bass lines.", "timestamp": "1519358086"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512&reply_comment_id=928648119602", "anchor": "fb-928558843512_928648119602", "service": "fb", "text": "→ Right, I'm not trying to replace bass players. I'm trying to make my own playing more versatile.", "timestamp": "1519358375"}, {"author": "Bob", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928558843512&reply_comment_id=928650320192", "anchor": "fb-928558843512_928650320192", "service": "fb", "text": "→ got it, Jeff; that was more a response to Hollis.", "timestamp": "1519359314"}, {"author": "Daniel", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928596098852", "anchor": "fb-928596098852", "service": "fb", "text": "I see where you are going with this. In the future, instead of playing music by manipulating an \"instrument\", a musician will just stand in front of a video camera making weird hand gestures, which the computer will interpret and emit the desired sound. Brave new world", "timestamp": "1519336305"}, {"author": "Jim", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928596098852&reply_comment_id=928597535972", "anchor": "fb-928596098852_928597535972", "service": "fb", "text": "→ That's actually not a bad idea. A theremin is a fairly expensive instrument, but you might be able to emulate one with a camera. And you wouldn't have to emulate its properties exactly, you could make a related-but-better instrument.", "timestamp": "1519337174"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928596098852&reply_comment_id=928601223582", "anchor": "fb-928596098852_928601223582", "service": "fb", "text": "→ Theremins expensive? You can make one yourself pretty easily, or buy one online for under $100.
But they're also extremely difficult to play precisely or quickly; they're not really great instruments, at least not along the axes I care about.", "timestamp": "1519338985"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928596098852&reply_comment_id=928601398232", "anchor": "fb-928596098852_928601398232", "service": "fb", "text": "→ When we want to talk to computers efficiently we use our hands on physical controls; we don't gesture. So my guess is gesturing to a video camera would not make a good instrument.", "timestamp": "1519339064"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928596098852&reply_comment_id=928601522982", "anchor": "fb-928596098852_928601522982", "service": "fb", "text": "→ A major thing we need from instruments is rapid feedback, and tactile feedback (which can happen even before sounding the note) is the best we've found.", "timestamp": "1519339115"}, {"author": "Zera", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928596098852&reply_comment_id=928606004002", "anchor": "fb-928596098852_928606004002", "service": "fb", "text": "→ Something that mapped beautiful/interesting dance movements onto pleasant/interesting sounds might make for a nice camera-based instrument.", "timestamp": "1519341225"}, {"author": "Gawain", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152", "anchor": "fb-928624392152", "service": "fb", "text": "Just learn the accordion, then you can play bass, chords, and melody at the same time", "timestamp": "1519349311"}, {"author": "Gawain", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928624721492", "anchor": "fb-928624392152_928624721492", "service": "fb", "text": "→ (and in case you don't want an instrument that sounds like an accordion, they make midi accordions!)", "timestamp": "1519349422"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928624846242", "anchor": "fb-928624392152_928624846242", "service": "fb", "text": "→ Accordion isn't very percussive, which is a lot of my mandolin playing.", "timestamp": "1519349491"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928624866202", "anchor": "fb-928624392152_928624866202", "service": "fb", "text": "→ (also accordion hurts my wrists)", "timestamp": "1519349504"}, {"author": "Gawain", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928625829272", "anchor": "fb-928624392152_928625829272", "service": "fb", "text": "→ Jeff Kaufman I disagree, it can be played quite percussively. But hurting wrists is a problem", "timestamp": "1519349843"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928631627652", "anchor": "fb-928624392152_928631627652", "service": "fb", "text": "→ Gawain how can it be played percussively? Specifically, how would you make a very short burst of high frequency sound like a fiddle or mandolin scratch?", "timestamp": "1519351716"}, {"author": "Gawain", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928637306272", "anchor": "fb-928624392152_928637306272", "service": "fb", "text": "→ Open all the stops, pull hard and play a bunch of keys extra staccato. I use that all the time. Also, depending on how it's mic'd you can use the sound of the register buttons but that's more of an extended technique", "timestamp": "1519353660"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/928519482392?comment_id=928624392152&reply_comment_id=928647435972", "anchor": "fb-928624392152_928647435972", "service": "fb", "text": "→ Do you have a recording it something where I can hear what you're describing? As I'm imagining it the sound isn't what I'm looking for.", "timestamp": "1519357974"}]}