"so I want to tell you guys about speech recognition and deep learning I think deep learning has been playing an increasingly large role in speech recognition and one of the things I think is most exci..."

"participated in along with Stanford and UW is to show that for even something straight forward that we sort of take for granted as an application of speech which is just texting someone with voice or ..."

"just said there's also other tasks though like word spotting or triggering so for example if my phone is sitting over there and I want to say hey phone go do something for me actually has to be listen..."

"almost effortlessly and for machines this has historically been incredibly hard so you think of this is like one of those sort of consummate AI tasks so the goal of building a speech pipeline is if yo..."

"future so traditional systems break the problem of converting an audio wave of taking audio and break and turning it into a transcription into a bunch of different pieces so I'm going to start out wit..."

"involve some modeling efforts and machine learning algorithms is to find the sequence of words W that maximizes this probability the probability of the particular sequence W given your audio that's st..."

"word hello what I might try to do is break it down into these units of sound so the first one is like the that H sound in hello and then an a sound which is actually only one possible pronunciation of..."

"phonemes so once you have this phoneme representation unfortunately it adds even more complexity to this traditional pipeline because now my acoustic model doesn't associate this audio feature with wo..."

"drop it right in that's great but it's also really hard to get working well if you start from scratch with this system and you have no experience in speech recognition it's actually quite confusing an..."

"model and there are lots of different methods for training this thing so take your favorite machine learning algorithm you can probably find someone who is trained in acoustic model with that algorith..."

"previously so this is in some sense the first generation of deep learning for speech recognition which is I take one of these components and I swap it out for for my favorite deep learning algorithm s..."

"the question that many in the research community including folks if I do have been trying to answer is can we go to a next-generation version of this insight can we for instance build a speech engine ..."

"tutorial together there is going to be some starter code live for the basic pipeline the deep learning part of the pipeline that we're talking about so there are some open source implementations of th..."

"bunch of different formats for audio but typically this one-dimensional wave that that is actually me saying something like hello world is something like 8,000 samples per second or 16,000 samples per..."

"would have a length of either say 8,000 or 16,000 samples and each element would be say a floating-point number that I had extracted from this eight or 16-bit sample this is really simple now once I h..."

"second we lose a little bit of information when we do this but it turns out not to not to be a huge difference now I said a moment ago that I think probably this is going to go away in the long run an..."

"this audio clip the first thing I'm going to do is cut out a little window that's typically about 20 milliseconds long and when you get down to that scale it's usually very clear that these audio sign..."

"can just think of this as a vector so now instead of representing this little 20 millisecond slice as sort of a sequence of audio samples instead I'm going to represent it as a vector here where each ..."

"this representation is to build what I'm going to call an acoustic model but which is really to the extent we can make it happen is really going to be an entire speech engine that is represented by a ..."

"and hopefully I can engineer my network in such a way but I can just read the transcription off of these output neurons so that's kind of the the intuitive picture of what we want to accomplish so as ..."

"problem if you were building a speech engine several years ago is to just try to bootstrap the whole system so I had actually train a neural network to correctly predict the sounds at every frame usin..."

"possible symbol sequences below that and what you'll notice is that if I go through every possible combination of symbols here there are several combinations that all map to the same transcription so ..."

"searching through all the possible alignments right I could shift these characters around a little bit I can move them forward backward I could expand them by adding duplicates or squish them up depen..."

"then if we have a whole bunch of other possibilities in here like the word yellow-eyed compute them in the same way and so this equation just says to sum over all the character sequences see so that w..."

"the same way we did for an H right so whenever you see two characters together like this where they're adjacent duplicates you sort of squeeze all those duplicates out and you just keep one of them bu..."

"okay very good okay so once I've defined this right I just gave you a formula to compute the probability of a string given the audio so as as with every good starting to a machine learning algorithm w..."

"this with the equation that I had on the last page that says in order to compute the probability of a given transcription I have to sum over all of the possible symbol sequences that could have given ..."

"can also just give you back the gradient so I won't go into the equations here instead I'll tell you that there are a whole bunch of implementations on the web that you can now use as part of deep lea..."

"there are pieces of off-the-shelf software that will compute for you the CTC cost function they'll compute this log likelihood given a transcription and the output neurons from your recurrent Network ..."

"so the first one that we've been calling sort of grad in the vein of all of the grad algorithms out there is basically a trick to help with recurrent neural networks so it turns out that when you try ..."

"difficult so we call this sort of grad it's basically a curriculum learning method and so you can see some work from yoshua bengio and his team on a whole bunch of strategies for this but you can thin..."

"without it and in addition another strategy that's extremely helpful for recurrent networks and very deep neural networks is batch normalization so so this becoming very popular and it's also availabl..."

"so this turns out not to be the optimal thing this actually doesn't give you the most likely transcription because it's not accounting for the fact that every transcription might have multiple sequenc..."

"cook so I'll give you a more detailed example in a second of how that happens all right so these are all the concepts of our of our very simple pipeline and the demo code that we're going to put up on..."

"so because this is like my favorite speech recognition party game I wanted to show you a few more of these so here's the max decoded output the poor little things cried Cynthia think of them having be..."

"later did you see the true label the poor little things cried Cynthia and that the last word is actually all these years and there isn't a word hanging off at the end so here's another one that is tru..."

"of the language itself okay so let's go back to decoding we just talked about max decoding which is sort of an approximate way of going from these probability vectors to a transcription Y and if you w..."

"resort to some kind of generic search strategy and so one proposed in the original paper briefly is a sort of prefix decoding strategy so I don't want to spend a ton of time on this instead I want to ..."

"step backward from a perfect end-to-end system and make make these transcriptions better so as I said the real problem here is that you don't have enough audio available to learn all these things if w..."

"models is to get examples like this correct so there are a couple of solutions one would be to just step back to a more traditional pipeline right use phonemes because then we can bake new words in al..."

"you off to the fact that Tchaikovsky concerto is a reasonable thing for a person to ask and that this particular transcription which we have seen in the past trike offski concerto even though composed..."

"from new data to keep track of whatever the trending words are today that your speech engine might need to deal with and we also have the need to query this thing very rapidly inside our decoding loop..."

"over here is that as you're multiplying in all of these terms you tend to penalize long transcriptions a bit too much and so having a little bonus or penalty at the end to tweak to get the transcripti..."

"represents the end of a word and so in addition to adding the probability of the space into my current estimate this gives me the chance to go look up that word in my language model and fold that into..."

"okay so that is really the scale model that is the set of concepts that you need to get a working speech recognition engine based on deep learning and so the thing that's left to go to state-of-the-ar..."

"that you don't get from from just reading The Wall Street Journal and finally there's something called the Lombard effect that happens when people are in noisy environments so if you're in like a nois..."

"so for example if we want to simulate noise in a cafe here here's just me talking to my laptop in a quiet room hello how are you so if I'm just asking how are you and then here's the sound of a cafe s..."

"audio and so the idea here is that it's just much easier to engineer your data pipeline to be robust than it is to engineer the speech engine itself to be robust so whenever you encounter an environme..."

"okay let's talk about computation because the caveat on that last slide is yes more data will help if you have a big enough model and big models usually mean lots of computation so what I haven't talk..."

"respectable so there are a whole bunch of ways to use GPUs if I do we've been using synchronous SGD it turns out that you've got to optimize things like all reduce code once you leave one node you hav..."

"the libraries that you're using and you're actually suffering a huge setback that you don't need to be feeling right now so one of the things we found back in November is that in libraries like Kublai..."

"batch and and this kind of modifies your your algorithm a little bit but in the end is worthwhile all right this is kind of all I want to say about computation if you're if you've got a few GPUs keep ..."

"here any questions so far yeah go ahead yep sorry yeah so the question is if humans have such a hard time coming up with the correct transcription how do you know what the truth is and the real answer..."

"right empirically determined yeah so the question is for a spectrogram with we talked about these little spectrogram frames being computed from 20 milliseconds of audio and is that number special is t..."

"I think in a lot of the models the in the demo for example we don't use overlapping windows they're just adjacent yeah yeah so those results are from from in-house software it Baidu if you use somethi..."

"so I want to take a few more questions at the end and maybe we can run into the break a little bit I wanted to just dive right through a few comments about production here so of course the ultimate go..."

"so in practice a lot of the neural networks that we use in research papers because they're awesome for beating benchmark results turn out not to work that well on a production engine so one in particu..."

"so that it could get more context on its own but it kind of turns out to only do that a little bit in practice it's really tough to control it and so if you find that you're doing much worse sometimes..."

"batch that we're working on the hunks of audio that we're working on are are in a big enough batch so if we just process one stream of audio so that my GPU is multiplying matrices times vectors then m..."

"very much I think we're right at the end of time sounds good alright we had one in the back who's waiting patiently go ahead more than one voice simultaneously so the question is how does the engine h..."