Understanding AI: An Introduction

Much like when I started my cryptography 101 series, I want to preface this with a very clear disclaimer: I’m definitely not an expert in this subject.

It actually took me a while to convince myself to embark on this new journey. For some reason, writing about Artificial Intelligence (AI, or Super Intelligence as it was recently branded, or however it's called when you're reading this) and the tightly related concept of Machine Learning in general feels more daunting than writing about cryptography or blockchain.

Part of it might be due to the huge boom the area has seen in the past few years. It somehow feels like the amount of available knowledge is almost impossible to process, let alone cover in a few articles.

Still, AI is simply too fascinating of a technology to not want to sink your teeth into, at least to me. It's amazing how far the field has come, but what's even more amazing is that despite having reached such high highs, we're already wired to expect it to keep evolving.

I mean, we’re already talking close to sci-fi levels here: some feats are nothing short of Jarvis-level intelligence.

It is that fascination that drives me to write this series, and perhaps to help some of you that are in the same boat as me - that is, you kind of understand the underlying mechanisms, but can’t fully grasp how all the pieces fit together.

The way I would like this series to be interpreted is as a set of curated notes from my independent studies, which I do in my free time. Nothing you read here will be an expert opinion, although, as always, I'll try to be as thorough and clear as possible.

And so, I’m starting this series as an attempt at understanding this amazing and vast topic.

Maybe we'll become experts together? Who can say!

Alas, I don’t think it wise to start our adventure by jumping right into the math behind all this. There are some ideas we ought to discuss first.

Introduction

Back in high-school (ah, the good ol' days), I had this science teacher that once gave us a really interesting assignment. She took a closed box into class, and asked us a very simple question: can you determine what's inside the box?

An innocent and even kinda silly question, right? But it had a nice twist that stuck with me. We were allowed to interact with the box however we wanted, but the catch was that we were not allowed to open it.

My classmates and I spent the better part of the lesson flinging the box around, listening to what noises it made, trying to determine if there were any patterns, etc. We formulated all sorts of crazy theories about what it contained. And when she finally opened the box, it was amazing to see how simple the setup was: a few beads of different weights strung along a taut cable in the center of the box.

I bet you're wondering why I'm telling you all this. The reason is beautifully simple, but it has profound implications. So here's the big revelation:

That's exactly how we navigate the world around usThat's exactly how we navigate the world around us

Yup. Reality itself is a closed box.

Think about it. When we're trying to understand the how things work, we rarely draw conclusions out of thin air.

At least not by modern standards! People used to attribute things to Gods being pissed off or whatever, but we don't do that anymore. Except for flat-earthers, of course.

What we do is gather data about the world, and then formulate what we call models to describe it. We then proceed to test those models against the collected data, and validate how accurately our model can predict our observations.

In the classroom example, the thumps and rattles were our data, and our crazy theories about the beads were our models. Shaking the box again to check whether a theory predicted the next sound? That was validation!

For example, and despite what pop culture has to say about it, we don’t have a very precise model of gravity only because an apple fell on Isaac Newton’s head. We do because after the supposed event, he was able to come up with a way to explain his experimental observations through a mathematical function:

$$F = G \frac{m_1 m_2}{r^2}$$

A model!

And for such a simple equation, this does a really good job.

Crucially, he couldn’t have done this without gathering data. Perhaps he started with something as simple as measuring the time it took for an object to fall to the ground. And maybe afterwards, he turned his gaze to the skies for more answers, observing the motion of the planets as many others had done before him.

I cannot say for sure, but what’s guaranteed is that he collected relevant data, and only afterwards was he able to validate his model.

Understanding our reality through careful analysis of data is at the crux of our business today. After all, data is everywhere, and it may reveal a lot about not only our physical world, but also about things such as our behaviors, our preferences, and many more things.

This is fundamentally what the area of Machine Learning seeks: to be able to produce these models that allow us to predict how something will behave, based on the available data. To reveal the patterns hidden in our observations.

Machine Learning vs Artificial Intelligence

You don't hear Machine Learning too often these days though.

But before AI had its mainstream boom back in 2022 with the release of tools such as ChatGPT (even though the race started earlier, roughly around 2016-2017), Machine Learning was very much the buzzword of the moment.

Indeed, this term has now mostly fizzled out, probably because nowadays you stamp “AI” into any product, and it seems to sell better. But does that mean that Artificial Intelligence is much different from Machine Learning? Should we focus on the former, and not worry too much about the latter?

While not the same thing, these two are closely related. Machine Learning is the universe of tools, models, and techniques that allow us to build good predictors out of data. Some of these models even exhibit behavior that resembles intelligence, and the systems built around them are what we usually refer to as Artificial Intelligence.

In other words, the idea of “Artificial Intelligence” refers to this capacity of some models and the tools around them to behave logically and intelligently.

In this time and age, the amount of tools and techniques that have been developed is incredibly huge. But you can probably imagine that this was not how the history of learning from data started. The perceptron, one of the pivotal components in the development of neural networks (which are foundational for modern Machine Learning), was only proposed in 1958 - and Newton, who had to understand gravity without the help of LLMs, passed away in 1727, long before having the powerful tools we have at hand today.

Just imagine if the dude had all the gadgets we now have at our disposal…

Thus, our starting point must not be AI itself. It’s much more important that we build our Machine Learning knowledge from the ground up, as it will be the basis for any serious AI development we can hope for later down the road!

Likewise, we will not skip right to the latest algorithms and techniques, so as to not get lost in the way. Instead, we’ll take it slow, and first explore the simplest models out there - and we’ll build up from there.

Before we try that though, we must not forget that there are two things at play here: models, and data. And we must pay special attention to the latter, as it is as important (or even more important) than the models themselves.

An Examination of Data

I know, I haven’t been very concrete yet. Allow me then to compensate by switching to a more practical approach; we'll continue by means of an example.

Imagine we want to predict whether a viewer will enjoy a particular movie, so that we can build a recommendation engine. Simple enough, right? All we need is to collect some data about people and their movie preferences, and then build some form of predictor around that information.

But as we’ll see, things get complicated very quickly.

Choosing What to Measure

For starters, what kind of information do we even care about?

Our first job is to decide what attributes actually matter. Should we track a person’s age? Their favorite genre? The time of day they usually watch movies at? Whether they prefer salt or sweet popcorn? The list could go on and on, almost forever. When do we stop?

That’s the first challenge: feature selection.

It might be very tempting to simply say “you know what? the more attributes, the better!”, and gather as much information as possible. But more attributes don't necessarily translate to better results. For example, someone’s shoe size won’t matter too much for our movie recommendation system. At best, we’ll waste our time and effort collecting that data, and at worst, it may even confuse our model.

Just imagine Newton trying to understand gravity, but only measuring the color of falling apples. No matter how much data he gathered, he wouldn’t have made much progress!

On the other hand, we might miss crucial information. Perhaps we assume that tracking whether someone watches movies alone, with their family, or with friends is unimportant… but it may be a deciding factor in whether they will want to watch a movie or not.

In some cases even, we may know which attributes to look at, but we may not be able to collect values for them. For example, when a doctor needs to diagnose a patient, they cannot ask for several exams at once, because each exam costs money. And so, there are situations in which we’ll have to make do with what we have.

What we just described is what’s usually called irrelevant attributes and missing attributes, and it’s important to keep tabs on them, because they can hurt our models.

Or rather, they will hurt our ability to create a good predictor. We’ll see this in action as we get to the models themselves!

So yeah, it gets tricky. However, if we happen to have a good intuition on the subject we’re working on (and sometimes, that’s quite a tall task), we may be able to select good attributes that can in principle produce a good model.

Unfortunately though, our problems don't end there.

Information Quality

Even when we choose the right attributes, the information we gather might be problematic in itself.

Let’s say we ask people to rate movies on a scale of 1 to 10. Again, it sounds super straightforward, but the problem is that people are inconsistent. Someone might rate a movie they enjoyed as an $8$ on a regular day, but if you asked them again after a bad week at work, they might say it was a $6$.

This introduces noise in the data: deviations from the “real” value.

In truth, it’s hard to say what “real” even means, because noise is always present in real-world data. Sensors malfunction, people misremember, typos happen during data entry.

Sometimes, noise is small and manageable, and it doesn’t greatly affect our ability to obtain good predictors. Other times though, it can completely throw off our models. Thus, we’ll be required to thoroughly examine data before and even during the process of finding good models (usually called training), and evaluate what to keep and what to discard.

I reckon all of this probably sounds daunting already, and we haven’t even started!

Every step of the way requires careful analysis, and things are not as well-defined as in other more purely-theoretical areas, due to the nature of what we’re doing: learning from noisy, imperfect data.

There are even more potential problems we need to consider, such as biases in data collection, scale, and time dependence, just to name a few.

So to set your mind at ease just a little, what I'll say is that we'll take it slowly and methodically. These concerns will arise naturally as we go, so we’ll cover those as we get deeper into the series.

And as this book puts it:

Machine Learning is more art than science.Machine Learning is more art than science.

A bold claim, perhaps, but having just seen how many judgment calls hide in even the first steps (what to measure, what to keep, what to discard), it's hard to disagree. And since AI is built on top of Machine Learning, the same goes for it.

Before we go, one last thing.

Models

I want to briefly talk about the models we’ll be working with.

Models come in all sorts of colors and flavors. They even vary in the degree of intelligence they may exhibit, ranging from simple reflex-based models designed to deterministically predict the value of an output based on input attributes, to models that seem to be capable of understanding logic, and may even change their predictions based on context.

Clearly, the space of possible predictors (the mathematical functions that act as models) is extremely big. It is therefore very important that we know what type of model to choose depending on the problem we want to solve.

For example, our recommendations engine looks very “dumb” in terms of intelligence. It just needs to do some calculations, and output a simple yes or no to the question “will this user like this movie?” (we can turn those 1 to 10 ratings into a yes/no label by picking a cutoff, say, 7 or above means “liked”). Such a model is called a classifier (a binary classifier, to be more precise), and even though it can be quite complex, it doesn’t need to think to get the job done, as we shall soon see.

Contrast this to a Large Language Model (LLM), which seems to adapt based on what you ask and what information you provide. However, this kind of model doesn’t seem to be a good fit for our recommendations engine - or at least, it seems to be a bit of an overkill.

It's like killing a mosquito with a bazooka.

And it's not only a matter of wasted effort. The more flexible a model is (that is, the wider the range of patterns it can bend itself to fit), the easier it becomes for it to latch onto the noise in our data, mistaking it for a real pattern.

So a good model isn't the most powerful one. It's the one that's just flexible enough for the problem at hand - and getting that choice right may determine whether we end up with a good predictor, or a fabulously tremendous mess. We'll see exactly what that means once we get our hands on some actual models.

Summary

Although this short introduction doesn’t address how exactly Artificial Intelligence works, we took our first steps into understanding what learning from data is all about, and already went through some important considerations.

Like with the closed box, we can never open reality to check what’s inside. All we have are our observations, and the models we build to explain them. How good those models turn out depends on the quality of the data we feed them.

In all honesty, it may sound like we didn’t cover much ground. But I assure you: it’s easy to forget about these foundational ideas about data and models as one starts getting their hands dirty - and they may entirely undermine our efforts moving forward if we're not careful.

Now that we’ve established that we need a solid Machine Learning basis to continue, our next step in the series will be dedicated to laying some groundwork. In particular, there are three areas of mathematics that are indispensable for what’s coming: probability and statistics, linear algebra, and optimization.

And so next time, we’ll start learning some of the fundamental concepts in these areas, which will prove indispensable as we move forward in our journey. After that, we'll be ready to start exploring the simplest models, and build up from there.

Stay tuned!