Search Results

8 matches found for 'batch'

Big Data Processing: Batching vs. Streaming

... work with large amounts of data. The way in which this data is gathered comes in a few variants: batching, where we aggregate a collection of data (e.g., by hourly time), streaming for data that needs to be processed in real-time, and a unified variant which simply does not distinguish the technical difference between batching and streaming, allowing you to programmatically use the same API for both.

January 3, 2022

Design Concepts

... can constantly poll for the result of their request. It is a great solution for offline batch processing and making the request/response model more interactive if desired. Ticketing Systems Two-phase locking (for locking reservations/sessions) Collaborative Filtering Used by recommender systems to make automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating).

November 20, 2019

Scaling Instragram Infrastructure

... happen on replicas (Master-Slave Replication) To deal with increased latency with writes, by batching requests wherever possible Replication lag from Master to slave replicas was not a big issue (for them) Cassandra NoSQL (wide column store) to store user feeds, activities, etc.

December 26, 2020

Atomic operations with Elasticsearch

... live (in production), and another table that is a copy of the live table. Suppose we have a big batch of updates to the search table, daily. Instead of updating the live table, what if we just update the copy table instead, and then switch over to the copy table after the update process is complete? The underlying idea above is the exact use case for the Elasticsearch Aliases API.

May 6, 2020

NoSQL - the Radical Databases

... since HDFS doesn't support random read/write. In a nutshell, Hadoop is suited for offline data batch-processing while HBase is suited for real-time data needs. Graph What its like: A graph Optimized to represent complex relationships with many foreign keys or many-to-many relationships Suitable if you need to analyze relationships in data.

December 31, 2020

Misconceptions of Software Engineer interviews at FAANG

... interview, but you might not get an offer because someone had performed a bit better out of the batch of interviewees, or someone may have vetoed you for the hiring decision, etc. Again, it is best to just resume studying and keep moving forward.

August 15, 2020

Web Development 101

... similar to SQL style queries. Pros: A transaction involving one or more operations can be batched into one GraphQL call. Reduces the amount of network calls needed for complex query operations in traditional REST.

November 9, 2017

Data stores in Software Architectures

... we don't need real-time processing We can input this data into a Hadoop cluster and have a batch ETL (extract, transform and load) job to store the data into a Data Warehouse, such as Amazon Redshift.

July 19, 2021