Project Perseus AI ~ Post #2 ~ Digging Deeper Into how Machines Learn

By

 









Project Perseus AI ~ Post #2 ~ Digging Deeper Into how Machines Learn

This week I built off of what I learned from last week! I didn't take giant leaps in stretching my horizons of the subject, but rather, I really fleshed out some more basic principles (something I could probably do forever), but at some point I would have to move on. However, what exactly did I learn about this week?

What I Learned this Week

I primarily learned about what an activation function is, and what the different types of activations there are. I also learned about logistic regression, which is a logical "next step" to linear regression which I talked about in Post #1. In a way, last week I learned how machines learn, this week I learned about how they think which their learning is a result from. So I just went a little bit deeper into the mechanics of things.

Okay, let's start off with the meat of the material:

Logistic Regression

Many times, in machine learning contexts, you want the system to output a probability. One way to do this is to apply a function over every output of the system that converts the outputs to a number between 0 and 1 (a probability can never be greater than 1 or less than zero). 

A common function that can do this is the sigmoid function, shown below.

$$y' = \frac{1}{1 + e^{-z}}$$

Where z (the input of this function) is actually the sum of all of the weights and biases of every neuron in the model (remember what I talked about in the first post), or essentially the output of the system.

This function then takes the entire output of the machine learning model (with the weights and biases calculated using gradient descent) and plugs it into this equation to get a number between 0 and 1, or, the probability.

Now, why is this important, you may be asking?

It's very important. Whereas with linear regression, you get the model's answer, with logistic regression you get how confident the model is. And based on my newbie knowledge of machine learning, I think this is something that is very crucial. What the probability is, is actually the computer giving you its probability of being right about a given scenario. What's better, is that logistic regression models are very easy to train and they are not very computationally "heavy."

How does Logistic Regression Work?

Remember how a common loss function from linear regression was the mean squared error function? It basically just told the computer how wrong its prediction is (ouch). With logistic regression, the loss function is called Log Loss (also called binary crossentropy).

Why? This is because the rate of change of a logistic regression is not constant like with linear regression. Remember the sigmoid function I described above? That curve is like an S-shape. Not linear. Inputs to the sigmoid function that are large negatively or large positively give a small change, but values closer to zero give more intense changes. Therefore, it's helpful to use a logarithm function to calculate the loss.

The log loss function is given below:

$$\text{Log Loss} = \sum_{(x, y) \in D} \left[ -y \log(y') - (1 - y) \log(1 - y') \right]$$

This function returns the logarithm of the magnitude of the change. If mean squared error is used instead, you would need more and more memory to record the very small changes (since the number of decimal places keep increasing!) For example: 0.9997, then 0.99997, and so on.

Log loss takes in a data set with labels (y values) that are either 1 or 0. You can think of this as either being true or false. y' in the log loss equation is the computer's predicted response, which is the probability of it being right, or rather, it's "confidence." There's a huge error when the model is confident but wrong, and not a whole lot of loss when the model is unsure but closer to the right answer, and barely no error at all when the model is both confident and correct.

There's also another key aspect to logistic regression and that's regularization, but I will get into that in a future post.

Activation Functions

Activation functions are arguably the most important part of machine learning models. This is because these functions introduce non-linearity to the system.

Here's the brutal truth that I've learned regarding these functions. Without them, it actually doesn't matter how many layers (sets of neurons) you actually have. It could be 1 or 100. If none of them utilize an activation function, all their doing is linear regression...over...and over...and over again. This makes finding patterns in complex data pretty much impossible.

However, activation functions allow the model to change its input space and warp data a certain way so as to be able to find non-linear patterns.

The sigmoid function discussed above is actually an example of an activation function. other functions like it also take linear input (weights times the features plus a bias) and then in turn plugs that into the function which is the activation function.

Then, the output of the activation function gets passed along to the next layer as its input. This is how non-linearity is introduced in the model

A famous activation function is ReLU (Rectified Linear Unit) and it is shown below:


$$f(x) = max(0, x)$$

This means that all negative values are now treated as 0 and all positive values are treated as is. (x = x).

This does have a problem with what's called "dying neurons," which are neurons that always get treated as zero and there's another form of the function called the Leaky ReLU which multiplies negative values by just a very small number instead of just treating them straight up as zero.

In Conclusion

I think it's time to wrap up this post; it's taken me too long to write and think about! I didn't include my code portion for this because it wasn't very different from last time. Like I said I kind of just learned more about what machine learning models do under the hood. I did toy around with the matplotlib python library to make a chart of predicted versus actual performance for my model which was neat. I think I may show that in the next post.

As always, thank you for reading and for joining me on my personal educational journey through machine learning! See you on the next post!

Where thoughts orbit stars and dreams power suns...







Comments

Post a Comment

Popular posts from this blog

By
Image
  Project Perseus Introduction Dear reader, Thank you for visiting the Project Perseus blog. I hope that the topics I write about bring you as much joy as they do for me. The world is a wonderful place...if you know where to look. My goal is to help readers reach that place of finding what's wonderful, beautiful, and captivating in this world by giving you a unique perspective and allowing you to take an innocent look at this universe's beauty. But before I begin, I should address some important questions you undoubtedly have: Who are you? My name is Nathaniel Comstock Moll. I am a current student in the fields of biochemistry, molecular biology, and physics. Outside of the technical aspect of my studies I enjoy writing novels, composing original pieces for the piano, doing some amateur backyard astronomy (sometimes volunteering at my school's local observatory), and now apparently writing blog posts. Essentially, I love to learn and to stay curious; two aspects that have c...
Read more

Project Perseus AI ~ Post #1 ~ Machine Learning Origins

By
Image
  Project Perseus AI ~ Post #1 ~ Machine Learning Origins This is the first post in hopefully a series of many that will document my educational journey through the topics of AI and machine learning. Every post will discuss a topic that I've reviewed and/or models that I've programmed to help me better understand these machine learning concepts.  What is the goal with these posts? Again, the goal is to primarily give a record of my self-taught journey in machine learning and to one day create my own AI model which I will name Perseus. I'm quite far away from that point at present, but if I'm diligent enough, I believe I can get there.  I started this journey because I've recognized that the 21st century will most likely be defined as the advent of organic and artificial intelligence cooperation. I believe that a healthy and beneficial cooperation is possible, but that it will require deep understanding on the humans' part and (hopefully) from the AI as well. My ...
Read more

Learning and Experimenting with Dynamic Light Scattering

By
Image
  Learning and Experimenting with Dynamic Light Scattering This past week at my Ole Miss REU research program has been...interesting to say the least. Not in a bad way, just in a I'm-going-insane-because-all-my-data-is-wacky-no-matter-what-I-do kind of way. The purpose of these experiments that I've been trying to do involves gaining data about the size and distribution of DNA/mercury complexes in solution. I outlined in my first post of this series how Dr. Wadkins has demonstrated that mercury complexes with DNA and induces it to form secondary structures. It's also postulated that these structures can be thought of as fully-fledged nanoparticles.  Dynamic Light Scattering (DLS) is a common way to obtain data about nanoparticles. Therefore, getting usable data from the DLS instrument from my DNA/mercury samples was my first order of business in order to develop an idea of how increased mercury concentrations correlates with particle size in solution. I thought it would be ...
Read more

Project Perseus AI ~ Post #3 ~ Evaluating Models and Learning TensorFlow

By
Image
  What did I Learn this Week? This week in my educational journey in machine learning, I learned how to track model metrics, such as accuracy and precision, I learned how another kind of model works, called binary classification , and finally I learned some Tensorflow basics. Tensorflow is a really neat library created for the python programming language that lets you easily set up and train models using its Keras API . I started going through some videos that taught me the basics of how to import data, compile the model, fit the model, and evaluate the model's effectiveness.  So, I've boiled down everything that I've learned in the past couple weeks into a post that briefly describes the ML concepts I went through and the code I used to practice those concepts. Let's get started! Basics of a Binary Classification Model A binary classification model is one that seeks to categorize data points as one option out of two possible options. For example, it could be used to ...
Read more