Join us for a fun dive into caching & the memory hierarchy! Learn how CPU registers, RAM, & hard drives balance speed & storage to make your computer lightning fast. From cache hits to internet browsing, we break it down in a clear, beginner-friendly way. Subscribe for more tech explainers & hit that like button to support the channel! Scan the QR code for more tutorials & resources. Let’s geek out together!
Introduction to Caching 00:00:00
What is Caching? 00:00:10
Memory Units: Size vs. Speed 00:00:14
CPU Registers vs. System Memory 00:00:33
Benefits of Caching 00:01:05
Access Patterns and Algorithms 00:01:24
Cache Hit and Miss 00:02:48
Data Movement in Cache 00:04:51
Cache Purging 00:05:57
Multiple Cache Levels 00:07:15
Internet Caching Example 00:09:28
Memory Hierarchy Pyramid 00:12:34
Volatile Memory and Registers 00:13:16
Processor Cache and RAM 00:13:48
Slower Storage Types 00:14:38
Conclusion and Subscribe Request 00:15:12
Thanks for watching!
Find us on other social media here:
- https://www.NeuralLantern.com/social
Please help support us!
- Subscribing + Sharing on Social Media
- Leaving a comment or suggestion
- Subscribing to our Blog
- Watching the main “pinned” video of this channel for offers and extras
Hello there. I’d like to talk to you about caching and the hierarchy of memory.
So for starters, what is caching?
Simply put, sometimes we have memory units or storage units that are very, very big because they’re cheap,
but that also makes them very, very slow.
And sometimes we have storage units that are very, very fast and thus expensive and thus very, very, very small.
and thus very, very, very small.
For example, if we had like a, let’s say a CPU
and we were just talking about the memory
that the CPU itself could hold in its CPU registers,
we have a very small number of registers inside of the CPU
compared to system RAM or disks or whatever.
So we can’t actually hold that much memory,
but at the same time, the CPU registers are lightning fast,
lightning fast, even compared to system memory.
So that kind of sucks, right?
right because the system memory is way slower than the CPU registers why can’t
we just have more registers well it’s too expensive so caching is something
that you can kind of use to get the best of both worlds you can have something
that’s very very fast be your primary place that you try to store things and
look for things to get them and set them and then when it runs out of space you
can just kind of push things to something that’s a little bit slower but
there’s a pattern of access that actually makes sense.
Hopefully you’re not accessing your data and instructions in a totally random way.
So that when you access in a pattern,
then algorithms inside the machine and the operating system can kind of figure out
which items are best to take out of the small storage device.
And that way we can sort of keep the most frequently or the most often
or the most recently used things inside of the fastest device.
This is kind of a way to get the best of both worlds, kind of a way to leverage the best of both worlds.
Okay, so what am I kind of talking about?
Let’s just suppose for the sake of argument, let me draw this a little bit.
Suppose for the sake of argument, you have something that’s really, really slow, but it’s huge.
A huge storage thing.
Maybe this is your disk.
Maybe this is your system RAM.
I don’t know.
There’s many levels to this.
a little bit faster and thus a little bit more expensive but it’s slower and then maybe you have
something else that is like really really fast but it’s super expensive so we can only make a
really really small version of it right so imagine that uh you know you are the i don’t know we’ll
say the user the user of these things is actually usually just like the cpu or something but let’s
say that you’re the user and you want to look and see if some piece of data you want is stored in
is stored in there. Maybe you have memory location of 45 or something or 44 that you
want to check for. You’re like what what is the data at memory location 44? I’m
totally making this up that we wouldn’t look at memory location 44 but suppose
you are. So if memory location 44 or this item whatever that you’re looking for is
in the super small and fast thing then awesome you don’t need to look any
further. But if it’s not we’ll call that a cache miss. We’ll say well it’s not in
we’ll say well it’s not in there whoops it’s not in there so we call that a miss for a cache and
that means we just have to check the next thing we’ll say all right well if it’s not in there is
it in the next fastest thing the thing that’s like a little bit slower and a little bit bigger if it’s
in there we call that a hit we would call it a hit if we found it on the first try also but if it’s
in there we call it a hit and if it’s not in there we call it another miss so we’ll say miss and then
And then finally we’ll check maybe the biggest thing that we have available and we’ll go all right is it is it in the giant thing?
And usually it is going to be in the giant thing because if we’re talking about program instructions or program data or something like that
You know the data is at least going to be on your disk drive in almost every case right so or your hard drive
so
You could imagine maybe this green thing is the
CPU registers or the CPU cache or something very very very fast and you can imagine this blue thing here is
is perhaps system RAM.
It’s like a little bit bigger, a little bit faster.
And then the red thing is your hard disk,
which does definitely have all the data,
but it’s really, really, really slow.
And we’re hoping not to try to touch the red thing
because it’s going to cost us a lot more time.
And then same thing for the blue.
So the first time we look for this piece of data,
you know, we missed twice
because it was only in the red thing,
you know, the disk, that we’ll call it.
But then suppose we come along later
later and we look for that piece of data again maybe we’re inside of a loop maybe our function
gets called multiple times and for whatever reason we check whenever when we when we originally found
the data in the red box by the way it was sent up through the cache and sort of copied into all the
other devices so you know when we couldn’t find our data anywhere in the green or blue the it was
copied from the red to the blue and then to the green also so it’s actually everywhere and then
And then the next time we try to look for that same piece of data, we’ll have a hit because it’s just sitting in the CPU or sorry, not the CPU, but just the fastest thing, which you could imagine as being the CPU or the CPU cache or something.
Right. So notice how once we have a hit, there’s not really any need to go look onto the slower devices.
based on how often you need that data or how recently you’ve used that data or whatever.
So great. We have a lot of hits. We’re now getting lightning speed because we’re just like reading
only from that green area. Suppose some time passes and you start filling up the green thing,
the fastest thing with lots of other data and it overflows because it is a small place for data.
So eventually we’ll have to start purging some of the data that we’re not using anymore. And maybe
is awful i’m still getting used to this pen let me just try one more time oh when i lift up the
pen it kind of drags a bit um so we’ve kind of filled up the green area and there’s not enough
room for all the data that we want the system realizes that ox44 hasn’t been used in a while
it doesn’t seem to be a part of the current working window data set and so we’ll just
purge it so the system just says all right let’s just erase that from the green
it from the green the fastest storage device it might not erase it from the
blue because the blue is bigger it can hold more data before we start having to
erase things right so that means there’s a pretty good chance it’s sitting in the
blue so if we try to access it again I’ll just put a red X here to indicate
that we have a miss so we miss the fastest device that sucks then we check
the next fastest device and it’s in there so we’re gonna call that a hit so
we didn’t get to use the fastest device but at the same time we also didn’t have to use
the slowest device so this is still kind of a win right suppose oh and then by the way once we find
that hit once we actually hit it from the blue device then the data gets carried back to the
green device the fastest device because the system now considers ox44 that address or that value
whatever to be part of the working set of data that we’re currently using so the moment we hit
it gets brought up to the fastest device in the hopes that maybe we’ll try to request it again
in the near future and save a bunch of processing time.
So suppose for the sake of argument, then a bunch more time passes
and it has to be purged again from the fastest device.
So, you know, we’re accessing lots of data.
We’re filling up that green thing again.
Eventually there’s not enough room.
We have to purge some old stuff.
So we purge the 44 there.
And then even more time passes.
And we have to also purge it from the medium speed device, the blue square.
Because that also can fill up at some point.
So that sucks.
We’ll never purge from the giant storage device,
which is like your disk drive usually or your hard drive.
So it’ll still be there.
Which means eventually when we try to request 44 again,
we’ll go through the original process.
We’ll just say, you know, going through…
Well, I’ll just write the word miss.
I’ll just write the word miss.
We’ll just go miss here.
And then we look here and it’s another miss.
And then we look here and it’s a hit.
I’ll put hit.
So in that case, when a lot of time passed
and all the other caches kind of filled up,
then we didn’t really get any benefit,
any performance boost from using caches.
In fact, we took a little bit of a penalty
on those two misses because we got to check one thing,
whoops, and then check another thing, whoops.
And then finally we hit the red thing,
red thing which is giant and has the data we wanted we could have just gone
there in the first place and saved a couple of you know some CPU cycles
because we wouldn’t have had to do the two misses but checking and missing is
part of the process of using a cache and so it’s worth it because you know most
of the time we’ll hopefully be hitting a lot of cache in the green area and even
more cache in the blue area so that’s the basic idea of caching kind of within
Let’s see what else that I want to show you here.
We got slow, medium, and large.
And maybe just another example that you might relate to more.
Suppose you go to an internet website and on the website there’s lots of images and
things like that.
Well, those images are usually cached on your local machine inside of your browser’s
cache area.
What am I even talking about?
First time you go to a website ever, the images get loaded from the internet.
very very very big because it’s like at this point in well I think since 2020 it
was over 64 zettabytes and a zettabyte is a trillion gigabytes so 64 trillion
gigabytes of data on the internet so huge right but the internet is kind of
slow even if you have a lightning fast connection it’s probably still slower
than accessing your local hard drive definitely way slower than accessing
your local memory although most people when I recorded this video have internet
slower than their disk drive or their SSD. Anyway, so the internet is huge, but it’s slow.
And I guess it’s cheap compared to the amount of storage and the speed you get.
So you download the image from the website from the internet. And that’s like takes a lot of time.
If you do that every single time, and there’s lots of images, you might feel like your internet
experience is slow, you might see images kind of like loading. You know, even in the modern era,
if they take just like a moment to load, then you kind of think like, whoa, there’s like a lot of
delay here, right? So, but then the first time we would call that, I guess, a miss on your local
cache, and then we would hit on the internet. So the first time you load it, it actually,
your browser will actually save a copy of that data to your local disk area. And well, the next
time you go to that website, instead of looking directly to the internet to get the images,
get the images your browser will look to see if you have a copy saved already locally on your on
your disk drive or your hard drive and if that’s true then your computer will just load the images
from the local machine and not from the internet and then your pages will seem like they load a lot
lot faster so that’s kind of the concept of caching in a slightly different way you know the fast
expensive thing is your is your hard drive the slow cheap thing is the internet you can imagine
many levels i mean this diagram that i drew there’s three levels in your cpu you have cpu
registers the fastest thing and then you have l1 cache which are almost as fast as the registers
and you have like some amount of kilobytes that you can store and then your cpu your cpu also has
l2 cache usually and then depending on what kind of cpu you have you might have l3 cache so that’s
four levels right there and then beyond that you know there’s system ram that can cache some data
cache some data for your CPU. It’s a lot slower than the CPU caches, but you know, it’s larger.
And, uh, uh, you know, eventually we can just get basically down to your hard drive,
uh, in the slowest case. So let’s see, what else can I tell you real fast? Oh,
I wanted to show you a diagram. Let’s see. So let me show you this diagram right here.
I found a diagram of this on Wikipedia. It’s called computer memory hierarchy.
And so this is what I talked about in the beginning of the video. We want to talk about
the beginning of the video we want to talk about like the hierarchy of memory
the hierarchy is kind of like a pyramid the reason it’s a pyramid is so that we
can kind of remember that stuff on the top is very small notice how the thing
on the top it has less area to work with and the things at the bottom have a lot
more area to work with so the thing that’s written in the middle here
actually let me go to let me go to the actual image of it because I think I
like it the way it looks a little bit better oh no I wanted to do this because
I wanted to do this because I already had it.
Okay.
So the things at the top have a very small size and a very small capacity.
And you can imagine some examples are processor registers, like our CPU registers.
But they’re very, very fast and they’re very expensive.
And there’s this little term written down here or this little description saying power on immediate term.
That just basically means this is volatile memory.
And so, you know, the moment you disable, the moment you disconnect power,
shut down your computer, you just lose that.
It’s just gone forever.
Then kind of, you know, getting a little bit bigger and also a little bit cheaper,
but also a little bit slower.
We have the small size, small capacity category.
You could imagine this is your processor cache, which is just, you know,
the cache on your processor is just a little data storage area.
You know, like I said before, that sort of backs up the CPU registers.
it’s where the registers first look for their cache. Still very fast and very expensive,
only a little bit cheaper and thus slower than the CPU registers. Then we have like medium size
and small size and large size and whatever. Different sizes and categories. So kind of
going from the fastest to the slowest here, once we hit cache, then we can consider this to be
the disk, but random access memory or system RAM is just a lot slower than the CPU, but
it’s faster than the disk.
And then you can imagine other things that are a little bit slower, but a little bit
larger could be flash memory, USB memory and so forth.
And then and then pretty slow here, the slowest that I ever deal with personally or hard drives,
even your SSD being way faster than your old disk drives is still considered very slow
compared to system RAM, but they’re huge, right?
But they’re huge, right?
And then kind of like at the very bottom of this pyramid is tape backup,
incredibly slow, but incredibly large for the money.
And that’s why this is sometimes used still.
And that, my friends, is the hierarchy of memory.
I hope you learned a little bit of stuff and you had a little bit of fun.
I’ll see you in the next video.
Hey, everybody.
Thanks for watching this video again from the bottom of my heart.
I really appreciate it.
I really appreciate it. I do hope you did learn something and have some fun
If you could do me a please a small little favor
Could you please subscribe and follow this channel or these videos or whatever it is?
You do on the current social media website that you’re looking at right now
It would really mean the world to me and it’ll help make more videos and grow this community
So we’ll be able to do more videos longer videos better videos or just I’ll be able to keep making videos in general
so please do do me a kindness and
Do me a kindness and subscribe.
You know, sometimes I’m sleeping in the middle of the night
and I just wake up because I know somebody subscribed or followed.
It just wakes me up and I get filled with joy.
That’s exactly what happens every single time.
So you could do it as a nice favor to me
or you could troll me if you want to just wake me up in the middle of the night.
Just subscribe and then I’ll just wake up.
I promise that’s what will happen.
Also, if you look at the middle of the screen right now,
you should see a QR code which you can scan in order to go to the website,
which I think is also named somewhere at the bottom of this video and it’ll take
you to my main website where you can just kind of like see all the videos I
published and the services and tutorials and things that I offer and all that
good stuff and if you have a suggestion for clarifications or errata or just
future videos that you want to see please leave a comment or if you just
want to say hey what’s up what’s going on you know just send me a comment
those in the middle of the night i get i wake up in a cold sweat and i’m like it would really it
really mean the world to me i would really appreciate it so again thank you so much for
watching this video and um enjoy the cool music as as i fade into the darkness which is coming for us
Thank you.
