Search Results

15 matches found for 'disk'

Virtual Memory

What is Virtual Memory? Virtual Memory is best described as a swap file on your hard disk that holds memory information for your running applications. Memory is structured and managed in two different ways; paging and segmentation.

September 27, 2020

B-Trees vs. LSM Trees

... usage, most notably SQL databases. With a B-Tree indexing structure, data is written onto the disk in fixed size page segments. These page segments are often about 4 KB in size, and have key value pairs sorted by the key.

March 14, 2018

Big Data Processing: Batching vs. Streaming

... relies on bounded data. Traditionally, the input for batch processing are files stored on disk. This is the case for MapReduce implementations such as Hadoop. These files may come from daily cronjobs or are exported from copies of OLTP (Online Transaction Processing) databases, such as a SQL database for inventory or customer purchases.

January 3, 2022

Apache Kafka and Event Streaming

... message broker. Log-based message brokers will, as the name implies, append entries to a log on disk, meaning they are durable. This is what allows Kafka (and other log-based message brokers) to replay events that might have already have been consumed by another client.

March 31, 2021

Traditional Message Queues vs. Log-based Message Brokers

... message broker. Log-based message brokers will, as the name implies, append entries to a log on disk, meaning they are durable. This is what allows Kafka (and other log-based message brokers) to replay events that might have already have been consumed by another client, or recover to a good state from machine failures.

January 20, 2022

A primer on MapReduce

... not fit into RAM, and a single machine will also not be able to hold the entire file in its hard disk either. Therefore, the entire file has to be split into pieces, scattered throughout multiple machines.

July 11, 2020

OS 101

... kernel, Linux kernel Hypervisor A hypervisor manages hardware resources such as CPU, memory, disk space as an abstraction across multiple operating systems or (virtual) instances.

December 21, 2022

Working with Production at Amazon Retail Website

... can include any kind of metrics for the hosts that are vital to its uptime. For example, low disk space, high CPU usage or high memory usage can indicate a server just waiting to crash and go down, causing your end users to suffer.

December 28, 2021

Web Development 101

... and core clock frequency has capped at around 3ghz for quite some time. You can expand your hard disk space and RAM, but even they have size limits. You can spend $1,000 on a single machine to get the state-of-the-art equipment, but spending $10,000 more on that machine for the best-of-the-best isn't going to immensely boost the machine's performance relative to the first boost.

November 9, 2017

Seattle Conference on Scalability: YouTube Scalability

... by 20%. Linux sees 5 volumes instead of 1 logical volume, allowing to more aggresively schedule disk I/O. Same hardware. Eventually did Database Partitions to spread writes AND read, partition by user.

December 25, 2020

Quick Numbers in Software Engineering Cheatsheet

... \(1,000,000 ns\) \(1,000 μs\) \(1 ms\) Somewhat fast HDD seek (i.e. 7200 RPM disk drives) \(10,000,000 ns\) \(10,000 μs\) \(10 ms\) Read 1 MB seque

January 23, 2021

Comparison Charts of File Storage Formats

... convenient way for developers and end-users, with less importance on the data size (in memory or disk). For this reason, these formats are typically human readable. For example, CSV and TSV are very popular output formats for data analysts who may use programs like Microsoft Excel.

January 24, 2022

Data stores in Software Architectures

... for objects that are rarely fetched. An alternative approach is to store data into actual hard disks (i.e. Amazon EBS), however there are major scaling drawbacks to this, as well as resiliency issues.

July 19, 2021

RDBMS Indexing

... for those new entries as well. This means that writes take longer, and takes up a lot more disk space. Therefore, if you are write-heavy, then you might want to avoid over-indexing, or be wary of which fields you index.

January 24, 2021

2PC - Two Phase Commit and Why it Sucks

... Prepare, Commit), it will record this state change onto the transaction log, which is also on disk. This is to provide a recovery mechanism in the case of failures such as system restarts on the Coordinator host.

January 18, 2022