"All right. Hello everybody. Welcome back. Glad you came back. Today, we will unveil the first tutorial. The first project is DeepTraffic, code named "DeepTraffic," where your task is to solve the tr..."

"images that are out there on the Internet and then you think about ImageNet, a data set where every image is labeled, the size of that ImageNet data set is a tiny subset of all the images available on..."

"the only thing we can count on is that truth that comes in the form of a reward. So the standard supervised learning pipeline is you have some raw data, the inputs. you have Ground Truth, the labels, ..."

"And the computational building block of a neural network is a neuron. A perceptron is a type of neuron. It's the original old school neuron where the output is binary, a zero or one. It's not real val..."

"in any order, the output is a 1. Otherwise, it's a 0. The cool thing about a NAND gate is that it's a universal gate that you can build up any computer you have where you have your phone in your pocke..."

"It's a sort of one basic computational truth you can hold on to as we talk about some of the magical things neural networks can do because if you compare a circuit of NAND gates and a circuit of neuro..."

"we use any kind of smooth function. Sigmoid, where the output can change gradually as you change the weights and the bias. And this is a basic but critical step and so learning is generally the proces..."

"and your task is given that image to say what number is in that image. Now, what is an image? An image is a collection of pixels; in this case 28 X 28 pixels. That's a total of 784 numbers; those numb..."

"And when they don't, we adjust the weights in such a way that they get closer to zero and closer to one depending on whether this is the correct neuron associated with a picture. We'll talk about the ..."

"resulting from the forward pass. So when you input that number of a 6 and outputs, whatever it outputs that's "a", a 10 dimensional vector. And it's summed over the inputs to produce the squared error..."

"It has to exist in this world that's maybe very different from the Ground Truth. So the take away from supervised learning is that a neural network's a great memorization but in the sort of philosophi..."

"and from the environment side, the environment receives the action amidst the observation. So your action changes the world, therefore, that world has to change and then tell you about it and give you..."

"Going from 5 to 4 to 3, down to 0. And so the goal is to select at any one moment the action that maximizes future reward. Without any knowledge of what a reward is in the greater sense of the word, a..."

"actually the biggest problem with everything you'll see today, is the model as how we represent the environment. And we'll see today some amazing things that neural networks can achieve on a relativel..."

"and probability of .1 going left and right. This is the optimal policy. Now if we punish every single step with a -2 as opposed to a -0.04. So every time you take a step,it hurts. You're going to try ..."

"and that reward is discounted because the world is stochasted, we can't expect the reward to come along to us in the way that we hope it does based on the policy, based on the way we choose to act. An..."

"The input to the Q-Function is a state at time t, "st". An action they choose to take and that state "at". and your goal is in that state to choose an action which maximizes the reward in the next ste..."

"When the learning rate is 0, you don't learn when alpha is 0. You never change your world view based on the new incoming evidence. When alpha is 1, every time change your world evaluation based on the..."

"in whatever you're doing with Reinforcement Learning or driving, you have no preconceived notion of what's good and bad, it's random. Or however you choose to initialize it. And the fact that it learn..."

"So the way you model, so let's start, is very simplistic world of Atari paddle. You think you model it as a paddle that can move left and right and there's some blocks and you model the physics of the..."

"Yeah. That's a great question. And so the question was: "what is the robustness of this model if the way you represent the world is at all, even slightly different, from the way you thought that worl..."

"Oh, the effect in that sense? No, no. It's just a coin flip. And if that Epsilon is 0.5, half the time you're going to take a random action. So there's no specific- It's not like you'll take the bes..."

"So you look at 4 images; so, current image and 3 images back and say, they're gray scale with 256 gray levels that size of the Q-Table that the Q value function has to learn is whatever that number is..."

"but still needs a formalized definition of that world which is much easier to do when you're able to take in sensors like an image So Deep Q-Learning, deep version. So instead of learning a Q-Table, a..."

"is that using a convolutional neural work, which I'll talk about tomorrow, but it's a vanilla network, like any other like I talk about earlier today, just a regular network That takes the raw pixels,..."

"And so we replace the basic of their rule, in the previous pseudo code, by taking a forward pass through the network given that S-state. We'll look at what the predicted Q-value is of that action. We ..."

"So as opposed to letting an agent, So you're learning this big network that tries to build a model of what's good to do in the world and what's not. And you're learning as you go. With experience repl..."

"on the left, so that's very little training, what you get is a paddle that learns hardly anything and it just keeps dying. It goes from 5 to 4 to 2 to 2 to 1, Those are the number of lives left. Then ..."

"A very small example of it but its general. It's general purpose, it knows nothing about games and knows nothing about paddles or physics. It's just take answer input of the game and they've did the s..."

"No, you don't have the- Well, yeah you play one step of the game. So you take action in a state and then you observe that. So you have that simulation. I mean, really that's one of the biggest proble..."

"is beat the game of Go is they learn from both expert games and by playing itself. So, you can do this in a distributed way and you could do the learning in a distributor way so you can scale. And in ..."

"build a network that achieves a speed of 65 miles an hour or higher. There is a leaderboard and you get to submit the model you come up with with a simple click of a button. So all of this runs in the..."

"There is a drop down for different display options. The default is non, in terms of stuff you show on the road. Then there is the learning input which is the, while that whole space is descritized, yo..."

"That block when it's empty is set to the 85 miles eighty miles an hour. And when it's occupied, it's set to the number that is the speed of the car. And then, the blocks that the red car is occupying ..."

"And those actions are given as input. That action was produced by the what's called here, the brain. The brain takes in the current state as input, the last reward, and produces and learns and uses th..."

"It provides a value of 0 in those blocks. If we set the patches behind to be 10, it looks 10 patches back behind starting at the 1 patch back is starting from the front of the car. The scoring for the..."

"that you've done up to this point or resets it and start the training again. So save often and there's a save button. So the training is done a separate thread in Web Workers which are exciting things..."

"and watching everybody pass. So what's being shown live is the lost function which is pretty poor. So in order to train, like I said, a thousand frames a second you just press the "Run Training" butto..."

"As as you play around with stuff if you want to save the code you could do so by pressing the "Save Code" button. That saves the various javascript configurations and that saves the network layout to ..."

"if it's a 0, 1 through 9, here the output is one of the five actions: left, right, stay in place, speed up or slow down. The ConvNet.JS settings is you can select a number of inputs if you want to mes..."

"is done in the browser using HTML5 canvas. So here is a simple specification of a blue box with canvas and this is very efficient and easy to work with. And the thing that a lot of us are excited abou..."

"So the very best of deep reinforcement learning is the most exciting accomplishment, I think, is when the game- When I first started as a freshman, took "Intro to Artificial Intelligence" it was said..."