Turning on query logging for PostgreSQL in Amazon RDS2018-03-09-Source: https://www.postgresql.org/docs/9.4/runtime-config-logging.html This is a very specific topic, but I needed to do it to track down performance of a slow web service call. First, go to your Amazon RDS instances tab and select the database that needs logging turned on, then go to “See details” under instance actions. Find the Parameter Group attached to … Continue reading "Turning on query logging for PostgreSQL in Amazon RDS"
On syncing data2017-07-17-I’m working on an ETL pipeline, which pulls from a REST API, does some transformations, and upserts it into a database. The REST API occasionally has invalid data (e.g. alphabet characters in the zip code, or misspelled US state names), and also occasionally throws HTTP 500 error pages. I’m supposed to stall the entire pipeline … Continue reading "On syncing data"
Testing data processing applications2017-05-08-We’re testing a data processing applications. There are multiple pipeline stages, the boundaries (i.e. inputs and outputs) of which are well defined. The bulk of the code deals with reshaping data from one form to another; there is very little functional logic. Therefore, testing should be focused mainly on how the code reacts to: Null … Continue reading "Testing data processing applications"
Lumens vs Candelas vs Lux2017-03-13-Whether you’re growing plants or have an obsession with flashlights, you will encounter the terms candela, lux and lumens. These all refer to measuring light, but what’s the difference? Radiant flux Let’s start from the beginning. Light is composed of photons, which have a specific energy depending on the wavelength. The radiant flux is simply … Continue reading "Lumens vs Candelas vs Lux"