Search Results

15 matches found for 'file system'

A primer on MapReduce

To first understand this very popular backend technology called MapReduce, let's take a look at Map and Reduce. Terminology The terms Map and Reduce are actually very popular higher-order functions used in functional programming.

July 11, 2020

OS 101

... conflicts Memory management (e.g. virtual memory) IPC File system (e.g. virtual file system abstractions) Examples: Windows kernel, Linux kernel Hypervisor A hypervisor manages hardware resources such as CPU, memory, disk space as an abstraction across multiple operating systems or (virtual) instances.

December 21, 2022

Comparison Charts of File Storage Formats

Big Data Encodings These encodings are often used with HDFS or some other distributed file system. Since the data can be as large as terabytes or petabytes, it is crucial to encode files in a space optimal way and also allow themselves to be read or written in an optimal way.

January 24, 2022

NoSQL - the Radical Databases

NoSQL NoSQL is a category of databases that aren't relational. For example, MySQL would be a relational database, where as MongoDB would be a NoSQL database. Back then, relational databases were the tried-and-true, prevalent and reliable data stores.

December 31, 2020

Big Data Processing: Batching vs. Streaming

... not limited to) Kafka, Flume, and Amazon Kinesis. This processed data can then be pushed out to file systems, databases, and live dashboards. One thing to note however is that Spark Streaming is more accurately a micro-batch data processing framework.

January 3, 2022

Virtual Memory

What is Virtual Memory? Virtual Memory is best described as a swap file on your hard disk that holds memory information for your running applications. Memory is structured and managed in two different ways; paging and segmentation.

September 27, 2020

Escape from the Strings

What is escaping? Escaping is used for characters that are not intended to be shown. Consider the following text: <strong>Hi there</strong> This text is saved onto a HTML file, foo.

April 23, 2023

Seattle Conference on Scalability: YouTube Scalability

Notes Apache isn't that great at serving static content for a large number of requests vs. NetScaler load balancing Python is fast enough There are many other bottlenecks such as waiting for calls from DB, cache, etc.

December 25, 2020

Design Concepts

In this article, I want to go over some fundamental design concepts that are useful for coming up with system design. Requirements Functional Requirements Describes specific behaviors i.e. If a URL is generated, it is composed of a Base64 encoded alias Non-functional Requirements Describes architectural requirements i.

November 20, 2019

Web Development 101

HTTP vs. HTTPS HTTP stands for Hypertext Transfer Protocol. It typically runs on TCP port 80. It is a protocol for sending data through browsers in the form of webpages and such. One major flaw with HTTP is that it is vulnerable to man in the middle attacks.

November 9, 2017

Google Protocol Buffers

From Wikipedia: Protocol Buffers are a method of serializing structured data. It is useful in developing programs to communicate with each other over a wire or for storing data. Here are some key characteristics of protobuf: Serialization into bytecode means that data is condensed -- small in size.

June 3, 2018

AWS Lambda and other Maven projects

Background AWS Lambda is a FaaS (Function as a service) that is event-driven and serverless. It is termed event-driven due to how AWS Lambda functions are invoked - the event that triggers a AWS Lambda function can be of many different types in the AWS realm.

April 22, 2019

Webpack: Usage Examples

Webpack has been around since 2012 and it is a very popular tool nowadays. You'll see it mentioned in a lot of front-end stacks. I've personally used it to power this blog and a handful of my own React projects such as https://classic-ah.

June 13, 2020

Asynchrony vs. Multithreading

Asynchrony Asynchronous programming, also known as event-driven programming, is built on foundations of Futures/promises. The basic idea is that instead of having a thread wait for a blocked call to finish (i.

June 20, 2020

Data stores in Software Architectures

Use Cases There are many ways to store your data. In this article we'll walk through some examples of data storage in common system designs. Reminder: There is no single best storage choice and they may vary heavily depending on things such as access patterns and scale.

July 19, 2021