Web Engineering Whys (v0.1)

Last updated: August 26th 2024

Table of Contents #

  1. Table of Contents
  2. Latest updates
  3. Target audience
  4. Preface
  5. What topics are included?
  6. Frontend programming
    1. JavaScript
    2. Browser compatibility and optimization
    3. Browser image optimization
    4. Browser storage mechanisms
    5. CSS
    6. React
    7. Redux
  7. Frontend tooling
    1. Webpack
  8. Project tooling
    1. Moonrepo
  9. Backend programming
    1. Node
    2. NestJS
  10. Fullstack engineering
    1. Server components
  11. Web performance
    1. Profiling
  12. Infrastructure
    1. Cloud
    2. Docker
    3. Compose
  13. System design
    1. Distributed systems
    2. Kafka
    3. Kubernetes
    4. Domain-Driven Design
    5. Tolerance and prevention
    6. Scaling
  14. Typing
    1. Static types
    2. TypeScript
  15. Testing
  16. Software design
    1. Functional abstractions
    2. Categorical design
    3. Algebraic design
    4. Type-theoretical design
    5. FP-TS
    6. Effect
  17. Data encoding
    1. JSON
    2. XML
    3. Protocol Buffers
  18. Database engineering
    1. Database indexing
    2. Query planning
    3. SQL
  19. Networking
    1. TCP
    2. HTTP
    3. Caching
  20. Web security
  21. Authentication and Authorization
    1. JWT
    2. OAuth 2.0
    3. OIDC
  22. Web architecture
    1. Server-side rendering
    2. Micro-frontends
  23. Software architecture
  24. API networking
    1. General
    2. REST
    3. gRPC
  25. Algorithms
    1. Big O notation
    2. Game tree search
  26. Appendix
    1. Additional references

Latest updates #

Target audience #

This is for the software engineer who is doing job interviews (either as interviewer or interviewee). Or the software engineer who feels shaky about the whys of what he does daily.

Preface #

"What" and "How" are inferior ways to learn about technology.

It's your understanding and ability to explain the "Why" of a thing – a technology, an algorithm, a programming library, etc. – that reveals your level of understanding.

Any monkey brain can spit out a definition (What) or memorize some recipe (How), but knowing the Why requires actual contextual and historical understanding of the thing.

So I'm writing this document, which consists purely of the "Why"s of a lot of things we do in software engineering, particularly web systems.

What topics are included? #

Everything and anything "web dev"-y that I deem relevant to whatever I'm doing at the moment in the software industry and/or my own products.

(I'll add a Table of Contents later.)

Frontend programming #

JavaScript #

(TODO: Improve this stub section, without turning it into a book.)

  1. Why are == and === different? docs
    1. Because == does type coercion to make more "truthy" / "falsy" values match each other, whereas === doesn't.
    2. (Which makes == "convenient", but === less error-prone and faster.)
  2. Why closures?
  3. Why callbacks?
    1. For separation of concerns in a "functional" way.
      • ("Functional" in the function construct sense, not the FP sense.)
    2. Eg. so my library doesn't have to know anything about your domain logic, types, etc. (In OO, think "Visitor" pattern.)
  4. Why asynchronous callbacks for concurrency?
    1. Because if you're already using callbacks for software design, using them to also deal with concurrency is natural.
    2. (ie. setTimeout(f, 1000) to schedule a call to f 1000 secs from now, without stopping the main thread to wait.)
  5. Why Promises?
    1. The usual answer: Because "it solves Callback Hell."
      • The more I think about my project history, the more Callback Hell seems like a myth. It was never the main problem for me.
    2. The realistic answer: Because promises increase separation of concerns:
      • In library function, f as a callback gives some separation of concerns – we don't have to know what f does.
      • But f is still a thing with an input type, so the library must know how to send it inputs and receive its output.
      • Whereas with promises, the library just returns the promise, and lets the user decide what to do.
  6. Why not Promises?
    1. Because sometimes separation of concerns isn't a concern!
  7. Why Immer.js? docs
    1. Because immutability "by hand" becomes error-prone and/or verbose once objects become nested.
      • By needing possibly many .slice()s and spread operators at different levels of a nested object, just to change one property somewhere inside.
      • (And "functional libraries" that try to provide Haskell-style "lenses" usually go for the "property path as a string" which isn't type-safe.)

References↑ #

  1. (Text/HTML) Equality comparisons and sameness @ Developer.Mozilla.Org
  2. (Text/HTML) Type coercion @ Developer.Mozilla.Org
  3. (Text/HTML) ImmerJS @ ImmerJS.GitHub.IO

Browser compatibility and optimization #

  1. Why polyfills?
    1. Because not every browser installed on every device in the world supports every DOM API the same way, or at all, so we add these javascript scripts that add the APIs so that our code can assume they exist.
    2. Because we don't want to be doing if (isExplorer) { doThis() } else if (isFirefox) { doThat() } like we used to in ancient times.
  2. Why compress?
    1. Obviously because smaller files load faster.
  3. Why "uglify" / "minify"?
    1. Because it makes the JS source code smaller:
      • By renaming variables, etc. to one-letter names.
      • By removing all unnecessary white-space and other syntactically optional stuff.
    2. (Note: "Uglified" code is a bit harder to reverse-engineer / plagiarize. But it's not something you should rely on. Any developer with enough time in their hands (and nowadays, AI tools) will be able to reverse engineer your "uglified" source.)
  4. Why source maps?
    1. Because we still want to debug code running on production, which is a pain with uglified/minified sources.
  5. Why code splitting?
    1. Because it makes the initial load faster, but not downloading everything upfront, and instead downloading each additional bit of code as needed.
  6. Why tree shaking?
    1. Because not every dependency is needed at all times, so tools like webpack will try to get rid of those dependencies which don't seem to be needed/imported by a given "bundle."
  7. Why WebP?
    1. Because JPEGs suck for images that are meant to be sharp, PNGs are too heavy when images are complex, and SVG gets expensive really fast for complex graphics (or impossible, for things like photos.)
    2. (Note: 50% of the bytes that travel the wire from any given page are image data.)
  8. Why lazy loading?
    1. See React.lazy().
  9. Why sourceSet? docs
    1. Because sometimes we want to requests different formats and sizes of an image based on the width and pixel density of the screen.
      <img srcset"header640.png 640w, header960.png 960w">
  10. Why CDNs?
    1. Because we want our users to load assets (images, script, audios, etc.) from dedicated (and ideally, geographically nearby) asset servers, instead of having our app's server handle that load too.

References↑ #

  1. (Text/HTML) HTMLImageElement - srcset @ Mozilla.Org

Resources↑ #

  1. tinypng
    • Free online image compressor for faster websites.
  2. caniuse.com
    • Free browser support tables for HTML5, CSS3, etc.
  3. imagekit.io
    • Paid (with free trial) url-based image and video optimizations, transformations.

Browser image optimization #

  1. Why not lossy?
    • Because it increases size with sharp edges and small details in otherwise undetailed areas.
    • (Eg. JPEG)
  2. Why not lossless?
    • Because it increases size with frequent, unpredictable changes in color, and large number of colors.
    • (Eg. PNG)
  3. Why not vectors?

References↑ #

  1. (Video/HTML) Image compression deep-dive @ YouTube

Browser storage mechanisms #

  1. Why cookies?
    1. Because, being lightweight, they are sent with each HTTP request, which makes them useful for things such as passing around authentication/authorization-related data.
  2. Why local storage?
    1. Because, unlike cookies, it doesn't have an expiration time.
    2. Because the size limit is way larger (whereas cookies are limited to 4KB.)
    3. Because you may have data that should be kept on the client, but you don't want to send it with each request (like with cookies).
  3. Why session?
    1. Because of all the same reasons for local storage, except we use this one when we want the data cleared when the user closes the tab/window.
  4. Why indexed DB?

CSS #

  1. Why CSS?
    1. Because the original idea was that the web page is a "document" and matters of visual styling (font colors, etc.) should be kept separate from the HTML document.
    2. Because the HTML document is meant to contain "semantic" code.
    3. (Ie. "This is a section, this a paragraph, this is a header, etc.")
  2. Why CSS frameworks?
    1. Because (ideally) they save you time with coherent default styles for layout, forms, etc.
  3. Why not CSS frameworks?
    1. Because they may be too large.
    2. Because customizing them might not be worth the hassle compared to just copying some of base "reset" stuff to your code and going from there.
  4. Why CSS preprocessors?
    1. Because they provide certain tools for CSS reusability that CSS alone lacks.
  5. Why not CSS preprocessors?
    1. Because their costs may outweigh their benefits.
    2. Because if you're styling through JS, reusability is a non-issue.
  6. Why non-semantic CSS? (eg. tailwindcss)
    1. Because we've given up on trying to make CSS both semantic and reusable.
    2. <p class="text-center">Hello</p>.
      • Good because it's easy.
      • Bad because the document is "concerned with" styling.
    3. <p class="greeting">Hello</p>
      • Good because it's semantic.
      • Bad because the ".greeting" style rules might apply to something else, eg. a button, which isn't a "greeting" at all, so using the class "greeting" on it would be wrong.
      • (You may try making something more abstract than "greeting," but that only kicks the can down the road.)
    4. .author-bio__image (ie. "BEM"-style CSS)
      • Good because it's "semantic" again.
      • Bad because it doesn't solve the reusability problem.
    5. <p class="w-96 bg-white shadow rounded">Hello</p>
      • Bad because "semanticity" has been thrown out the window.
      • Good because it's as reusable as possible.
    6. (Note: CSS coders dealing with a perennial software design question: At what level of abstraction do we write code?)
      • .rust-in-peace-cover?
      • .megadeth-album-cover?
      • .album-cover?
      • .squared .dark-bordered .shadowed?
    7. (Note: It's been argued that not all semantics need to be content-derived.)
  7. Why Object Oriented CSS?
    1. To reuse code in terms of visual patterns, as opposed to content semantics.
    2. To separate structure and skin, and separate container and content.
  8. Why "CSS in JS"?
    1. Because for large messy projects in can provide better scoping and modularity, based on component trees, instead of CSS-selector-based targeting of UI elements.
  9. Why not "CSS in JS"?
    1. Because it makes CSS caching harder or impossible.
    2. Because it can reintroduce the old "white flash" problem because the CSS is no longer a separate file that the browser will parse before the JS.
    3. (There are workarounds, such as extracting the non-dynamic CSS from the JS source with webpack, but this has the costs of increased complexity, and an increased dependence on webpack's quirky features.)
    4. Because in React in can lead to oversized component trees.

References↑ #

  1. (Text/HTML) When to use @extend; when to use a mixin
  2. (Text/HTML) CSS Utility Classes and "Separation of Concerns"
  3. (Text/HTML) tailwindcss
  4. (Text/HTML) About HTML semantics and front-end architecture
  5. (Text/HTML) Object Oriented CSS

React #

  1. Why "components"?

    1. Because the concept promotes the idea of the UI being composed of modular, self-contained pieces that work well together.

  2. Why is the key property important?

    1. Because without it React can't keep track of component instances properly.
    2. Because if your component returns the same type of element, React will keep the existing instance, even if all other props changed.
    3. Because it allows you to force React to unmount the current instance and mount a new one, even if props are the same.
      • Ie. it allows you to "reset" component instances.

  3. Why component "lifecycle" methods?

    1. To perform certain actions at the most (or only) appropriate time.
      • Eg. Fetching data, adding event listeners, cleaning up event listeners, etc.
      • ("Lifecycle" as in its low level state in the DOM and React's management.)

  4. Why componentDidMount?

    1. To setup stuff as soon as the DOM object is added to the DOM ("mounted")
      • Eg. Add event listeners, setup network connections, etc.

  5. Why componentDidUpdate?

    1. To react to props or state changes right after they cause a re-render.
      • Eg. Your class component gets an artistId and based on it it loads some data and renders stuff. When this artistId changes, you want to do the necessary clean up and regeneration of the UI.

  6. Why componentWillUnmount?

    1. To clean up.
      • Eg. Remove event listeners.

  7. Why getDerivedStateFromError, ie. "error boundaries"? docs

    1. To localize component explosions, so that a component failing won't crash the whole UI.
    2. (Note from docs: "There is no direct equivalent for static getDerivedStateFromError in function components yet. If you’d like to avoid creating class components, write a single ErrorBoundary component like above and use it throughout your app. Alternatively, use the react-error-boundary package which does that.")

  8. Why hooks?

    1. Because reusing stateful (and effectful, etc.) code is cumbersome with traditional class components (wrapper components, render props, etc.)
    2. Because stateful logic ends up spread across various component lifecycle methods (eg. listener registration in a lifecycle method, clean up in another, etc.) which makes it hard to do extraction refactors later.
    3. Because the switch from stateful/effectful class component to function component should result in simpler code, by moving:
      • Setup from componentDidMount to useEffect.
      • Clean-up from componentWillUnmount to useEffect's return function.
      • Reaction to prop/state change from componentDidUpdate to useEffect dependencies array.
      • this.state use to setState use.

  9. Why useState? docs

    1. Because we want to let React update the UI based on state changes in a function component.
    2. (Note: The expression EXPR in useState(EXPR) is the initial state, ie. what we would set as this.state = EXPR in classical components.)

  10. Why useState with a callback?

    1. Because the expression EXPR in useState(EXPR) might be too expensive.
    2. Ie. the computation for the initial state might be too expensive to be evaluated on every re-render, so we do useState(() => EXPR) so React knows to evaluate it just once.
    3. (Note: Sort of a useEffect(() => {}, []) equivalent for useState)

  11. Why useEffect? docs

    1. To have a controlled way to do (side) effects
    2. Why does useEffect take "dependencies"?
      • To control when the effect should run.
    3. Why does useEffect run after browser paint?
      • To avoid delaying the browser's screen updates.
    4. Why useEffect(f, []), ie. empty dependencies array?
    5. Why the optional return function in useEffect?
      • Because you may need to clean up, etc.
      • (What we used to do in componentWillUnmount.)
      • Eg. Removing event handlers.
      • Eg. Clearing intervals.
      • Eg. Aborting data fetching on component unmount.

  12. Why not useEffect?

    1. Because you might not need it.
      • Eg. It's unnecessary for computing fullName from useState-managed firstName and lastName values.

  13. Why is it important to understand referential equality?

    • To avoid confusing behavior related to useEffect's second argument (its dependencies). Eg.
      function C() {
  const [ name, setName ] = useState("");
  const [ age, setAge ] = useState(0);
  const person = { name, age };

  const [ unrelated, setUnrelated ] = useState();

  useEffect(() => {
     // This will run when `setUnrelated` is called.
     // Because `person` is always a new object on every re-render.
  }, [ person ]);

}
    • (^ Use [ name, age ] instead of [ person ] as dependencies.)
    • (^ Define person with useMemo and [ name, age ] as dependencies.)

  14. Why useContext? docs

    1. To pass state down a component tree without manually passing it as props.
      • Eg. user profile data, UI theme state, locale, authetication state, feature flags, any sort of global preference, etc.
      • ([Dependency Injection type of thing.)
      • (Scala implicit arguments type of thing.)
      • Eg. ThemeContext, use <ThemeContext.Provider value="dark"> ... </ThemeContext> at the top of the app tree, then any component (no matter how deep in the tree) can access the theme with useContext.
    2. In React jargon, to avoid "prop drilling."

  15. Why useRef?

    1. To mutate values without triggering re-renders.
    2. To reference DOM objects directly.
      • const inputEl = useRef(null);
        const onClick = () => { inputEl.current.focus() };
      1. (Note: one can pass a callback to the ref prop):
        • <input ref={inputEl => inputEl.focus()} />
    3. To avoid unnecessary uses of useState. Eg.
      function SomeForm() {
  const bandRef = useRef();
  const albumRef = useRef();
  const onSave = () => {
    const band = bandRef.current.value;
    const album = albumRef.current.value;
    // ...
  };
  return (
    <div>
      <input placeholder="Band" ref={bandRef} />
      <input placeholder="Album" ref={albumRef} />
      <button onClick={onSave}>Save</button>
    </div>
  );
}

  16. Why useCallback? docs

    1. Because it's usually unnecessary, and bad for performance, to recreate a function, eg. an onClick handler, on every render.
      function SomeForm() {
  // instead of this
  // const onSave = () => {
  // };
  // do this
  const onSave = useCallback(() => {
    f(foo)
  }, [ foo ]);
  return (
    <div>
      <button onClick={onSave}>Save</button>
    </div>
  );
}
      • (Note: Do make sure to specify as dependencies whatever variables from the component are used in the callback. Eg. foo in example above.)

  17. Why useReducer? (Why "reducers"?)

    1. To keep all "state update logic" in the same place.
    2. Because you're trying to have a module with the "brains" of the operation (as far as state updates goes.)
    3. Because a reducer can be easier to maintain than setStates everywhere.
    4. Eg. To migrate from direct state management to reducers, you migrate:
      • From direct state setting with useState, to "action" dispatching.
      • From state-setting logic in event handlers to data updates in reducers.
      • (Reducer logic is usually just a big switch.)
      • ("Action" is a plain object like { type: "delete", id: artistId })
      • ("Action" describes to the reducer what the user just did.)
    5. Eg.
      • const [foos, dispatch] = useReducer(foosReducer, []);
      • Where foosReducer is the function with the big switch, ie. the reducer.
      • Where [] is an array with any actions you want to run by default.
      • Where dispatch is a function used to dispatch actions.
    6. Note: One can combine contexts with reducers to further simplify a codebase.
      • Centralize state update logic in a reducer.
      • Pass state and dispatch functions down implicitly with context.

  18. Why useLayoutEffect? docs

    1. Because sometimes the effectful logic has layout-related actions (moving things, etc.) that we want to run before the next browser paint, but with the most up-to-date DOM layout values.
      • Ie. This hook is for when our effectful logic involves changing the DOM in a way visible to the user.
      • Eg. Placing a pop-up / modal. Some boolean decides whether or not to show it. The browser painting of the pop-up will occur first. User will see it wherever its DOM element lands naturally. Then your useEffect logic will run, and place it where it really belongs. Eg. using somebody's .getBoundingClientRect(). So the user will see it "flash" from the initial placement to the right one. With useLayoutEffect, you modify the DOM element's position with the latest layout values, but before the next browser paint, so no flashing will occur.
    2. (Note: useEffect happens asynchronously, ie. when a dependency changes, the work is scheduled to happen concurrently at some point, without blocking painting, thus will run after painting. useLayoutEffect happens synchronously, ie. the work runs synchronously in between calculating the latest layout values (thus the hook's name) and the next browser paint. )

  19. Why React.createPortal()? docs

    1. Because sometimes you want to render some children into some arbitrary part of the DOM.
    2. Eg. rendering the UI for a modal dialog on document.body.
        <>
    <button onClick={() => setShowModal(true)}>Show modal</button>
    {showModal && createPortal(
      <ModalContent onClose={() => setShowModal(false)} />,
      document.body
    )}
  </>

  20. Why React.lazy()?

    1. To defer loading component code until it is rendered for the first time. spec
      import { lazy } from 'react';
const MarkdownPreview = lazy(() => import('./MarkdownPreview.js'));
    2. Because not all users need all of the app's code all the time (eg. non-admin users don't need the code for AdminDashboard).

  21. Why <Suspense>?

    1. To display fallback UIs until children finish loading.
      • No more foo ? <C foo={foo} /> : <Spinner /> everywhere.
    2. To deal with the "flashing content" issue caused by conditional rendering logic usually related to waiting for data fetches.
      <Suspense fallback={<p>Loading...</p>}>
  <TwitterStats />
  <YouTubeStats />
  <McDonaldsStats />
</Suspense>
    3. (^ The child components above can then also get rid of the usual ternary conditional boilerplate when waiting for data.)
    4. (Note: Only Suspense-enabled data sources will activate the Suspense component)

  22. Why useTransition? docs

    1. Because that way we can have slow state updates that don't block the UI. Ie. So that we can mark certain state updates as "low priority" for the renderer. (Eg. loading a large array of components when switching tabs)
      const [isPending, startTransition] = useTransition()
    2. (Note: startTransition for classical components.)

  23. Why useDeferredValue? docs

    1. Because sometimes we want React to finish a re-render with an old state value X, while the X setter is finishing its work. Ie. keep displaying the previous known resolved state value (as opposed to using Suspense with some fallback "loading" UI) while re-rendering, to keep the UI feeling snappy.
    2. Eg.
      const [query, setQuery] = useState('');
const deferredQuery = useDeferredValue(query);
const isStale = query !== deferredQuery;

  24. Why Concurrent React (useTransition, useDeferredValue, fiber architecture, etc.)?

    1. Because React by default is synchronous, ie. it finishes all of the work involved in a component update in a single uninterrupted block in the main thread, potentially blocking user events.
    2. Because we want React to jump to different tasks, eg. rendering a list vs. processing user input vs. animations, concurrently, based on their priority. "Cooperative multitasking" style.
      • Eg. Partially rendering a tree, without committing it to the DOM.

  25. Why Flux?

    1. Because Facebook needed a way to keep state updates under control, and found that it could do so by:
      • Having "singleton" stores. PostsStore, CommentsStore, etc.
      • Register each store with a Dispatcher.
      • Having dispatcher calls be the only way to trigger a store update.
    2. Because having any random part of the app mutate state makes maintenance hard.
    3. (Note: Facebook described this "Flux Architecture" in 2014. This inspired the creation of React and Redux.)

  26. Why a "pull" approach for UI computation?

    1. Because with a push-based approach you have to schedule the work, whereas with a pull-based approach, the framework does it for you.

  27. Why "reconciliation" / "virtual DOM"?

    1. Because an intermediate tree structure allows React to optimize its interpretation of elements into DOM (or iOS, Android, etc.) updates.
    2. Because separating rendering from reconciliation allows all the various renderers (DOM, React Native, pedagogical examples, etc.) to use the same clever algorithm for UI representation.

  28. Why JSX?

    1. To write HTML-like markup inside JS files.
    2. Because React.createElement() by hand gets ugly fast.
      • JS: React.createElement(Foo, {a: 42, b: "B"}, "Text").
      • JSX: <Foo a={42} b="B">Text</Foo>.

  29. Why the fiber architecture?

    1. To prioritize different types of updates. Eg. Animation updates over data store updates.
    2. To be able to pause/abort/reuse chunks of rendering work.
    3. (The original stack-based reconciliation algorithm couldn't do this.)
    4. (The original algorithm would render subtrees immediately on update.)

  30. Why this "fiber" abstraction?

    1. To represent a "unit of work," so work can be "split."
    2. Eg. Schedule high priority work with requestAnimationFrame().
    3. Eg. Schedule lower priority work with requestIdleCallback().
    4. (Note: A fiber can be thought of as a "virtual stack frame.")
    5. (Note: Fiber system = More flexible "call stack.")

  31. Why automatic batching?

    1. Because grouping multiple state updates (eg. calls to useState setters) together into one re-render improves performance.

  32. Why create-react-app (CRA)? docs

    1. Because it's a quick way to create a new react app with "no configuration."
      • (Note: when the project grows out of triviality, we can "eject" it.)
      • (Ie. npm run eject will remove the create-react-app dependency and copy all config files (.babelrc, .jestconfig, etc.) to our directory so we can start managing them manually from then on.)
      • (Note: Obviously, npm run eject will work only once.)

References↑ #

  1. (Text/HTML) React @ GitHub
  2. (Text/HTML) Design Principles @ Legacy.ReactJS
  3. (Text/HTML) Understanding React's key prop @ KentCDodds.com
  4. (Text/HTML) Built-in React Hooks @ React.dev
  5. (Text/HTML) How to fetch data with React Hooks @ RobinWieruch.de
  6. (Text/HTML) React hooks - useState @ React.dev
  7. (Text/HTML) React hooks - useEffect @ React.dev
  8. (Text/HTML) React Hooks - useContext @ React.dev
  9. (Text/HTML) React Hooks - useRef @ React.dev
  10. (Text/HTML) React Hooks - useCallback @ React.dev
  11. (Text/HTML) Extracting State Logic into a Reducer @ React.dev
  12. (Text/HTML) Component - getDerivedStateFromError @ React.dev
  13. (Text/HTML) React Hooks - useLayoutEffect @ React.dev
  14. (Text/HTML) React - createPortal @ React.dev
  15. (Text/HTML) Scaling Up with Reducer and Context @ React.dev
  16. (Text/HTML) React - lazy @ React.dev
  17. (Text/HTML) React - Suspense @ React.dev
  18. (Video/HTML) A Quick Intro to Suspense in React 18 @ YouTube
  19. (Text/HTML) React hooks - useTransition @ React.dev
  20. (Text/HTML) React hooks - useDeferredValue @ React.dev
  21. (Text/HTML) Writing Markup with JSX @ React.dev
  22. (Text/HTML) Hello World Custom React Renderer @ Medium
  23. (Text/HTML) A (Brief) History of Redux @ Redux.JS.org
  24. (Text/HTML) React Fiber Architecture @ GitHub
  25. (Text/HTML) Reconciliation versus rendering @ GitHub
  26. (Text/HTML) What is a fiber? @ GitHub
  27. (Text/HTML) Window: requestIdleCallback() method @ MDN
  28. (Text/HTML) Window: requestAnimationFrame() method @ MDN
  29. (Text/HTML) Automatic batching for fewer renders in React 18 @ GitHub
  30. (Text/HTML) What is Concurrent React? @ React.dev
  31. (Text/HTML) create-react-app @ GitHub

Redux #

  1. Why Redux? docs
    1. Because we want global state in our React application.
    2. (Note: Doesn't have to be a React application, but that's almost always the context of the Redux, "state managers" conversation.)
  2. Why Redux Toolkit (RTK)? docs
    1. Because Redux by hand involves a lot of boilerplate.

References #

  1. (Text/HTML) Redux @ Redux.JS.org
  2. (Text/HTML) Why Redux Toolkit is How To Use Redux Today @ Redux.JS.org

Frontend tooling #

Webpack #

  1. Why can't webpack "tree-shake" when using CommonJS / require() for modules?
    1. Because require()s are "dynamic," ie. are resolved at runtime, whereas webpack's tree-shaking relies on static analysis of ES6-style imports and exports.
  2. Why is the "dependency graph" important?
    1. Because that's the structure webpack uses to represent what code to bundle.

Project tooling #

Moonrepo #

  1. Why moonrepo? docs
    1. Because managing things with npm scripts is a mess.
    2. Eg.
      projects:
  client: "apps/client"
  server: "apps/server"
  2. Why ./moon.yml at the root of each project?
    1. Because we need a place to define file groups, tasks, dependencies, etc. that are unique to that project.
  3. Why ./moon/tasks.yml?
    1. Because some file groups and tasks are inherited by all projects in the workspace, and we want to standardize them.
      • Eg. Linting, typechecking, code formatting.
      • Eg. Might want all projects to have these:
        tasks:
  format:
    command: 'prettier --check .'
  lint:
    command: 'eslint --no-error-on-unmatched-pattern .'
  test:
    command: 'jest --passWithNoTests .'
  typecheck:
    command: 'ts --build'
  4. Why the project configuration field?
    1. Because we need a way to specify project ownership in a large monorepo.
      • Eg.
        type: 'tool'
language: 'typescript'

project:
  name: 'moon'
  description: 'A repo management tool.'
  channel: '#moon'
  owner: 'infra.platform'
  maintainers: ['miles.johnson']
  5. Why toolchains?
    1. Because managing, downloading, and installing Node.js and other languages by hand or with adhoc scripts is a mess.
    2. Because platform-specific (ie. language and environment-specific) task running by hand or with adhoc scripts is a mess.

References↑ #

  1. (Text/HTML) Moonrepo - Introduction @ Moonrepo

Backend programming #

Node #

  1. Why cluster? docs
    1. Because it's a simple, out-of-the-box way to run multiple instances of node, to distribute the workload.
    2. (Note: If process isolation is not needed, worker_threads is enough to parallelize a single Node instance over multiple threads.)
    3. (Note: With cluster, our server.js will be spawned into multiple processes, all listening to the same port. We decide what to do when an instance crashes.)
  2. Why pnpm? docs
    1. Because npm's "hoisting" modules is bad, because it causes our code to access dependencies that haven't been explicitly specified in the package.json.
    2. Because it's faster than npm and yarn, because it uses a better module resolution algorithm.

References #

  1. (Text/HTML) Cluster @ NodeJS.Org
  2. (Text/HTML) pnpm docs @ PNPM.io
  3. (Text/HTML) What makes pnpm performant | Zoltan Kochan | ViteConf 2022 @ YouTube

NestJS #

  1. Why NestJS? docs
    1. Because for some developers need the OO-heavy MVC stuff to make their codebases maintainable.
    2. Because sometimes you just want a framework that tells you exactly where and how to write your code.
      • Ie. "Code to handle the response goes here," "code to talk to the service goes there," "errors are handled this way," "this is how to test," etc.
      • Ie. An "idiomatic" framework.
  2. Why decorators?
    1. So we can
      • Mark class methods as GET/POST/PATCH/DELETE controllers. Eg.
        import { Controller, Get } from '@nestjs/common';
@Controller('cats')
export class CatsController {
  @Get()
  findAll(): string {
    return 'This action returns all cats';
  }
}
      • Access query params. Eg.
        import { Controller, Get, Query } from '@nestjs/common';
@Controller('cats')
export class CatsController {
  @Get()
  findAll(@Query("order") order?: "asc" | "desc"): string {
    return 'This action returns all cats';
  }
}
  3. Why the NestJS command line (CLI)? docs
    1. So you can generate source files that follow the architectural patterns of the framework. Eg.
      • npx nest g module users.
      • npx nest g service users.
      • npx nest g controller users.
  4. Why services?
    1. To separate concerns, eg. HTTP-response-related "logic" goes in the controller, domain logic goes in the "service," etc. They often encapsulate calls to the database, other services, etc.
  5. Why providers?
    1. To do the "dependency injection" thing by decorating classes (which are usually services) as @Injectable().
    2. Ie. to have things that can be "injected" (into our controllers) as dependencies. And "instances" that can be managed by the framework, to avoid unnecessary instantiation with manual new's in the controllers.
  6. Why pipes? docs
    1. To have middleware-style request data validation and transformation.
      • Eg. ValidationPipe, ParseIntPipe.

References #

  1. (Text/HTML) NestJS - Documentation @ NestJS.Com
  2. (Text/HTML) NestJS - Documentation - CLI @ NestJS.Com
  3. (Text/HTML) NestJS - Documentation - Pipes @ NestJS.Com

Fullstack engineering #

Server components #

  1. Why Server components?

    1. Because it can improve performance to move data requests to the server, by reducing the time it takes to fetch the data and the number of requests the client side to make, as well as caching them to reuse them across users.
    2. Because it improves security to keep sensitive tokens and API keys on the server.
    3. Because it can improve the performance of weak devices and slow internet connections by rendering non-interactive UI components on the server.
      • It's JS code that doesn't get transmitted and executed on the client.
    4. Because initial page load can be faster and the server-rendered HTML makes the website SEO-friendly again.

  2. (Text/HTML) Server Components @ NextJS.Org

Web performance #

Profiling #

  1. Why should I care about reflows and repaints?
    1. To ensure that rendering a frame to screen tk 16.6ms or less (to maintain the 60fps),
    2. Ie. To compute the JS, styles, layout, paint, and compositing in <16.6ms.
    3. (Ideally 10ms or less, do to additional overhead operations on the browser.)
  2. Why do reflows occur?
    1. Because some style changed causing the need to recompute the placement/positioning of DOM elements.
  3. Why take heap snapshots?
    1. To investigate memory issues (leaks, etc.)
    2. To see how your JS objects and DOM nodes are being distributed in memory, before/after a certain UI events.

References↑ #

  1. (Video/HTML) Profiling JavaScript Like a Pro @ YouTube
  2. (Text/HTML) Record heap snapshots @ Developer.Chrome

Infrastructure #

Cloud #

  1. Why AWS, Azure, or Google Cloud?
    1. Resilience.
    2. Networking.
    3. Computation.
    4. Storage types.
    5. Economics (discounts, purchase models, etc.)

Docker #

  1. Why Docker?
    1. Because we want to containerize, because we want lightweight virtualization, because we want consistent development and deployment experience.
    2. Because old school virtualization, with VMs, was resource-heavy as hell, because it ran a full OS (managed by a "hypervisor," such as VMWare or VirtualBox.)
  2. Why dockerhub?
    1. Because we want a repository-type place for prebuilt images that we can just docker run. Eg. docker run hello-world.
  3. Why images?
    1. Because we want a sort of file system which we build from a Dockerfile, for the container to run.
  4. Why a Dockerfile?
    1. Because we want a standard way to create an image. Ie. specify what we want our image to have.
  5. Why a "container"?
    1. To run/stop an instance of an image.
  6. Why a registry?
    1. To share images.

Docker - Compose #

  1. Why Docker compose?
    1. For defining and running multi-container Docker applications, described/configured by a YAML file.
  2. Why services?
    1. To define the "computing components of an application." spec
    2. To define the "computing resource within an application which can be scaled or replaced independently from other components." spec
  3. Why networks?
    1. For services to communicate with each other. Eg.
      services:
  frontend:
    image: example/webapp
    networks:
      - front-tier
      - back-tier

networks:
  front-tier:
  back-tier:
  4. Why volumes?
    1. To "store and share persistent data." spec
    2. To have named data stores that can be reused across multiple services. Eg.
      services:
  backend:
    image: example/database
    volumes:
      - db-data:/etc/data

  backup:
    image: backup-service
    volumes:
      - db-data:/var/lib/backup/data

volumes:
  db-data:
    3. (Note: It's not straightforward to find the actual file implementation of the "volume" on macOS, because it lives inside an abstraction created by docker.)

References↑ #

  1. (Text/HTML) The Compose Specification @ GitHub
  2. (Text/HTML) Services top-level element @ GitHub
  3. (Text/HTML) Networks top-level element @ GitHub
  4. (Text/HTML) Volumes top-level element @ GitHub

System design #

Distributed systems #

  1. Why distributed systems?
    1. Because there aren't enough resources for one gigantic machine to do everything.
    2. To do storage and computing on multiple computers because a single computer can't handle it.
    3. (^ "multiple computers" = mid-range, commodity hardware.)
    4. According to Kleppmann:
      • Because some domains are inherently distributed (eg. mobile telecommunications)
      • Reliability: If a node fails, system as a whole keeps running.
      • Performance: Get data from nearby node rather than halfway round the world.
      • Solve bigger problems: For some problems there's no single supercomputer powerful enough.
  2. Why decentralized systems?
    1. Because sometimes we need our resources and process spread across multiple computers.
    2. (Eg. Blockchain.)
  3. Why do distributed systems typically require complex configuration?
    1. Because they need to be cluster aware, deal with timeouts, etc.?
  4. Why are distributed objects usually a bad idea?
    1. Because you can't encapsulate the remote/in-process distinction
      • An in-process method is fast and successful, so it makes sense to make many fine-grained calls.
      • A remote method is slow and error-prone, so it makes sense to make few coarse-grained calls.
  5. Why are some distributed systems needlessly complex and full of patches?
    1. Because designers make several fallacies when designing.
    2. (The fallacies, according to Peter Deutsch and James Gosling:
      • The network is reliable.
      • The network is secure.
      • The network is homogeneous.
      • The topology does not change.
      • Latency is zero.
      • Bandwidth is infinite.
      • Transport cost is zero.
      • There is one administrator.)
  6. Why layered architectures?
    1. To achieve a conceptual separation of concerns, which ideally leads to ease of maintenance and team organization and specialization.
  7. Why service-oriented architectures?
    1. For loose coupling and further separation of concerns.
  8. Why publish-subscribe architectures?
    1. For decoupling data producers from data consumers in terms of codebase and technology as well as time and space (asynchronicity).
  9. Why Amazon's DynamoDB?
    1. Because you don't run it, Amazon does.
  10. Why Google's BigTable and MapReduce?
  11. Why Apache's Hadoop?
  12. Why Apache's Cassandra?
  13. Why distributed systems "As a Service"?
    1. So you can make the people who wrote the distributed software deal with the distributed-related ops issues, while you focus on your business domain matters.
  14. Why scalability, availability, performance, latency and fault tolerance?
  15. Why distinguish between fault prevention, tolerance, removal, and forecasting?
  16. Why is geographical scalability a tough problem?
    1. Because network latency is bound from below, so we need to copy data to locations closer to the client, which leads to the problems of maintaining consistency.
  17. Why "fault tolerance"?
    1. Because things will go wrong. Whether it's the user doing things wrong, or a hardware corrupting data, or a wifi dying, things will go wrong, and one should at least think about how to recover (if possible) from the main faults.
    2. (Eg. Apache Cassandra.)
  18. Why a strong consistency model?
    1. To replace a single server with a cluster of distributed nodes and not run into any problems.
    2. (Ie. It makes the distributed system's behavior indistinguishable from a single server's.)
    3. (Weak consistency models start to introduce anomalies that make them a "different beast.")
  19. Why is there tension between strong consistency and availability during partition?
    1. Because one can't prevent divergence between two replicas that cannot communicate with each other while both continue to accept writes.
  20. Why is there tension between strong consistency and performance?
    1. Because strong / single-copy consistence requires that nodes communicate and agree on every operation, which results in high latency.
  21. Why is there tension between caching and replication and consistency?
    1. Because we now have multiple copies of a resource, and modifying one copy makes that copy different from all the others.
  22. Why is global synchronization extremely hard or impossible?
    1. Because network latencies have a natural lower bound?
  23. Why does the FLP impossibility result matter?
    1. Because it highlights that algorithms that solve the consensus problem must either give up safety or liveness when the guarantees about message delivery do not hold.
    2. Because it imposes a hard constraint on the problems that we know are solvable in the asynchronous system model.
  24. Why does the CAP theorem matter?
    1. Because it is more relevant in practice because of its slightly different assumptions (network failures instead of node failures) leading to clearer practical implications.
  25. Why is time modeling important?
  26. Why is the replication problem important?
  27. Why prevent divergence?
  28. Why accept divergence?
  29. Why does adding a machine not increase performance and capacity linearly?
    1. Because of the overheads of having separate computers.
      • Copying.
      • Coordination.
      • Etc.
    2. (This is why various distributed algorithms exist.)
  30. Why is latency (short response time) a complex issue?
    1. Because it's harder to address financially than other aspects of performance.
    2. Because it's strongly connected to physical limitations.
  31. Why is availability mostly about being fault tolerant in practice?
    1. Because more components = higher probability of failure, so the system should compensate so as to not become less reliable as components are added.
    2. (Availability = uptime / (uptime + downtime))
  32. Why have timing and ordering assumptions?
    1. Because information can only travel at the speed of light, so nodes at different distances will receive messages at different times and potentially different order than other nodes.
  33. Why synchronous system model (assumption)?
    1. Because it allows the system designer to make assumptions about time and order.
    2. Because they're analitically easier (but unrealistic.)
  34. Why asynchronous system model (non-assumption)?
    1. Because sometimes the system designer just can't make assumptions about time and order.
  35. Why a client-centric consistency model?
    1. To avoid anomalies where a client sees older versions of values resurfacing.
  36. Why leaderless replication?
  37. Why single-leader replication?
  38. Why partitioning?
  39. Why Change Data Capture (CDC)?
    1. Because sometimes you want to push a message to, say, Kafka, whenever data mutation (eg. INSERT, UPDATE, DELETE on your MySQL) happens, so that other systems subscribed/listening to the Kafka topic will do something (eg. analytics, stream processing, etc.)
    2. For use cases such as:
      • Replicate data (send to a data warehouse, data lake, etc.)
      • Send a message to the user whenever his data changes.
      • Invalidate or update caches.
    3. To leverage on the Write-Ahead Log DB systems have to notify other services of changes to a database / source.
    4. To achieve "near-real-time data" (analytics, etc.) and/or historical data preservation.
  40. Why a Lambda function?
    1. Because sometimes you just wanna run a little piece of code (on, say, Amazon's server pool) whenever something happens, and don't want to maintain / pay for a server that's always running, or worry about where it will run in the distributed system (you leave all that to eg. Amazon).
  41. Why Streaming?
  42. Why Events?
  43. Why cross-service transactions?
  44. Why "eventual consistency"?
  45. Why a "2-Phase Commit"?
    1. To ensure that data is consistently committed across several different systems. ("All or nothing.")
  46. Why Conflict-Free Replicated Data Types (CRDTs)?
    1. Because it's a way of managing data appropriate for offline features, allowing different replicas to make progress independently from each other, even if there's no communication possible at points. Particularly useful for collaborative state mutation apps.
    2. (Note: "Conflict-free" is a bit of a misnomer.)
  47. Questions for design session:
    1. What are we optimizing for?
      • Reads?
      • Writes?
    2. Capacity estimates.
      • Eg. Characters per post (Twitter), code blob size (GitHub), Posts-per-day (Twitter, FB)
    3. Sketch the main operations.
      • Eg. Fetching followers/following (Twitter)
    4. Are we going to have gigantic transactions? Can they be avoided by design?

References↑ #

  1. (Text/HTML) Getting Real About Distributed System Reliability @ Blog.Empathybox.Com
  2. (PDF/HTML) Basic concepts and taxonomy of dependable and secure computing @ IEEE.org
  3. (PDF/HTML) Fallacies of Distributed Computing Explained @ UNSW.edu.au
  4. (Text/HTML) Microservices and the First Law of Distributed Objects @ MartinFowler.Com
  5. (Video/HTML) Change Data Capture (CDC) Explained (with examples) @ YouTube
  6. (Video/HTML) What Is Change Data Capture - Understanding Data Engineering 101 @ YouTube
  7. (Video/HTML) Using AWS Lambda As A Data Engineering @ YouTube
  8. (Video/HTML) Thinking in Events: From Databases to Distributed Collaboration Software (ACM DEBS 2021) @ YouTube

Mock Interviews↑ #

  1. (Video/HTML) Google Systems Design Interview With An Ex-Googler @ YouTube
  2. (Video/HTML) 12: Design Google Docs/Real Time Text Editor | Systems Design Interview Questions With Ex-Google SWE @ YouTube

Kafka #

  1. Why Apache Kafka? docs
    1. Because it's fast in terms of throughput, ie. it is designed to move a large number of records in a short amount of time, which it achieves by doing sequential I/O in the form of an append-only log.
      • (Note: As of 2024, sequential writes are measured in the 100MB/s, whereas random writes are measured in 100KB/s. Sequential is orders of magnitude faster.)
    2. Because it provides horizontal scalability. It can scale to 100s of brokers, and millions of messages per second, with latency usually less than 10 ms, ie. real-time.
      • Which is why Kafka was invented at LinkedIn, and is now used by Netflix, Uber, Airbnb, Walmart, to do messaging systems, activity tracking, gathering metrics from multiple sources, application logs gathering, stream processing (see: Kafka Streams API, Spark, etc.), database load reduction through decoupling, etc.
      • Eg. Netflix uses Kafka for its eventing, messaging, and stream processing needs, eg. to apply recommendations in real-time as the user watches TV shows.
      • Eg. Uber uses Kafka to gather user, car, and trip data in real-time to compute and forecast demand, and compute surge pricing in real-time.
      • Eg. LinkedIn uses Kafka to prevent spam, collect user interactions, and make better recommendations in real time.
      • (Note: See how in all these use cases, Kafka is a data transport mechanism. It's not where your application / business logic lives.)
    3. Because a pubsub messaging system, where a publisher can publish messages to a topic without requiring knowledge of the subscriber, and without requiring the subscriber to be running at any specific time in order to receive it (unlike client-server networking), allows us to scale communications, up to millions of messages per second.
      • Eg. Messaging systems, activity tracking, app logs, stream processing, decoupling, etc.
      • And with certain guarantees about message publishing. Eg. At least once (produce same message until consumer acknowledges; consumer may have to deal with deduplication), At most once (produce a message only once; ie. no retries), Exactly once (even if a producer sends the same message more than once, the consumer will receive it only once).
    4. Because sometimes you want 3 services to consistently reflect, in order, the series of actions (eg. data changes) the user is making on a web app.
    5. To solve concurrency-caused Isolation issues by using serial event logs.
      • Eg. The "choose username if it doesn't exist" issue. SQL transaction can still yield the problem of two users with the same username, because of concurrent execution of transactions. A solution with Kafka is to push a message about the "I want to use username 'foo'" event into the specilized totally-ordered serial log, and then the relevant user/account/etc. (micro)services will read that log in order.
    6. (Note: Some Kafka proponents call Kafka a distributed event-driven platform.)
  2. Why events (or "record" or "message")?
    1. Because we want to record what has happened in the world, and we want to do it fast.
  3. Why zero-copy?
    1. Because the common path from file to socket is too slow:
      • OS reads from disk to kernel-space pagecache.
      • Application reads from pagecache to user-space buffer.
      • Application writes back to kernel-space into socket buffer.
      • OS copies data from socket buffer to NIC buffer where its sent to the network.
  4. Why topics?
    1. Because we need a way to (logically) partition data.
      • Eg. "users" topic, which has consumers and producers.
    2. Because we want to represent a particular stream of data.
    3. (Note: "Topic" in Kafka. "Queue" in RabbitMQ. "Channels" in Redis. "Tables" in databases.)
    4. (Note: When you create a topic, Kafka will automatically distribute it, ie. its partitions, across all brokers.)
    5. (Note: If you have more brokers than partitions for some topic, then some brokers simply won't have any data for said topic.)
    6. (Best practice: If you produce to a topic that doesn't exist, Kafka will create it automatically. However, the default configurations (e.g., replication factor, number of partitions) will almost certainly not be suitable for production environments, especially regarding fault tolerance and performance. So you should always explicitly create a topic with the appropriate configurations before writing to it.)
    7. Why num.partitions?
      • Because we want to store the data in different areas of the cluster.
    8. Why replication.factor?
      • Because we want to specify how many copies of each of these partitions we should have.
    9. Why min.insync.replicas?
      • Because we want to specify how many of these copies should be up to date for us to consider the cluster healthy.
    10. Why cleanup.policy?
      • Because we want to specify how to get rid of old / stale data.
  5. Why brokers?
    1. Because we need something to hold our topics and partitions.
    2. So each broker holds certain topic partitions.
    3. (Note: A broker is essentially a server.)
    4. (Note: A broker is identified by an id, which must be an integer.)
    5. (Note: No broker has "all the data," because the point is for Kafka to be distributed.)
  6. Why clusters?
    1. Because we want to have multiple brokers for fault tolerance.
    2. (Note: A typical number of brokers per cluster is 3, but some clusters out there have over 100 brokers.)
  7. Why is a broker also called a bootstrap server?
    1. Because each broker knows about all brokers, topics, and partitions.
    2. (Note: So when you connect to any broker, you connect to its cluster.)
  8. Why an append-only, never-delete-anything, type of storage?
    1. Because computers are super fast at appending (as opposed to inserting stuff in the middle of other stuff.)
  9. Why a pull-based model for consumers (ie. consumers polling for data, as opposed to Rabbit, where the server pushes data to the user)?
  10. Why partition the topics?
    1. Because they get pretty large pretty fast.
      • So we have, eg. partition 1 of the users topic, with users A to D, then partition to with users E to G, etc.
      • (Note: Thus the "address" (so to speak) of a message is topic -> partition number -> offset number.)
      • (Note: Messages in partitions are kept only for a limited time. Default is 1 week.) 1 (Note: Messages in partitions are kept only for a limited time. Default is 1 week.)
    2. Because the whole approach to scaling with kafka is that the user will specify a distribution strategy, ie. how to distribute/partition events across the different partitions.
    3. (Note: In a partition, the position at which a message can be found is called an offset, conceptually as in "byte offset," since the kafka partition is a large sequential log file, for efficient access.)
    4. (Note: An offset is not literally a byte array offset, though. Messages in a partition have variable size, and Kafka keeps a map of offset to actual byte location in the sequential log.)
    5. (Note: Generally, a message with a lower offset is processed earlier by consumers, while a message with a higher offset is processed later.)
    6. (Note: Offsets are immutable and ever-increasing.)
    7. Why partitioner.class?
      • So we can specify the mechanism by which Kafka decides what partition to use for a message.
      • (Note: If no partitionar class specified, Kafka uses hash of key. If no key present, it'll use sticky partition.)
      • (Note: There's a RoundRobinPartitioner, which doesn't always in evenly distributed data. That's why we have the sticky partitioner.)
    8. Why partitioner.ignore.keys?
      • Because sometimes you want your data uniformly distributed but you want to have keys, so you set partitioner.class to None and partitioner.ignore.keys to true.
    9. Why partitioner.adaptive.partitioning.enable?
      • Because sometimes we want to look at the current set of brokers we want to send data to and adapt to send that data to faster brokers. Ie. ignore slow brokers, send more data to the faster ones.
    10. Why partitioner.availability.timeout.ms?
      • Because (in conjunction with the above) sometimes we want to ignore a partition for a while if it's been taking longer that some about of ms in responding. (ie. "It's slow, we don't wanna bog it down any more.")
  11. Why partition keys?
    1. Because sometimes we want some messages to go to the same partition, so we use the same key (which can be any arbitrary string of our choice, eg. "truckLatLng") so that Kafka puts them into the same partition.
      • (Note: Caveat: This works as long as the number of partitions remains constant.)
      • (Note: The way Kafka will put all messages for a given key in the same partition depends on number of partitions, among other things. See key hashing.)
  12. Why is the choice of partition key important?
    1. Because a poor choice of partition key could mess up scalability.
      • Eg. all of the data ending up on a single partition.
      • Good keys evenly distribute across the partition space.
      • (Ie. good keys maximize parallelism.)
    2. (Note: Nowadays these concerns are usually handled by dedicated "managed" kafka cloud providers.)
    3. (Note: If no key is present, the producer uses round robin to select which partition to send the message to.)
  13. Why replication?
    1. Because brokers will go down eventually, and when that happens, we want another broker to serve the data.
      • The replication factor defines how many copies per partition.
      • Eg. replication factor of 2 means each partition exists twice.
    2. Replicas are distributed across existing brokers. Eg. for some "Topic-A":
      • Broker 101: [ Part-0 of Topic-A ]
      • Broker 102: [ Part-1 of Topic-A, Replicated(Part-0 of Topic-A) ]
      • Broker 103: [ Replicated(Part-0 of Topic-A) ]
    3. (Note: When you create a topic, its replication factor is usually between 2 and 3. With 3 being the gold standard, and 2 considered a bit risky.)
    4. (Note: Replication factor of N means producers and consumers can tolerate up to N-1 brokers being down.)
  14. Why is a replication factor of 3 a rule of thumb?
    1. Because 1 broker can be taken down for maintenance, and another taken down unexpectedly.
    2. Because 3 is the minimum number that allows the system to tolerate a single failure while still maintaining a majority (quorum) for decision-making.
    3. Because as you go beyond 3, the diminishing returns in fault tolerance are not worth the increasing storage and network cost.
  15. Why a partition leader?
    1. So that only one broker can receive and send data for a partition, letting the others be passive replicas, called ISRs (for "in-sync replica"), ensuring consistency and coordination.
    2. (Note: Who's the leader and who are the ISRs is managed by Zookeeper, or KRaft in newer Zookeeper-less setups.)
  16. Why producers?
    1. Because we need a way to get data into Kafka!
    2. Because we need to send data to a broker who is the partition leader of a topic.
      • (Note: The producer does this without any intervening routing layer.)
      • (Note: The producer will attempt to do automatic batching, by accumulating data in memory to send multiple messages in a single requests.)
      • (Note: The producer will also automatically recover on broker failure.)
      • (Note: The producer will optionally receive acknowledgement of data writes. acks=0 for no acknowledgement, acks=1 for leader acknowledgement, acks=2 for leading + replicas acknowledgement.)
      • (Note: See min.insync.replicas config. Set to eg. 2 means that acks=all is satisfied when at least 2 ISR, including the leader, acknowledge.)
  17. Why idempotent producers?
    1. Because a producer can introduce duplicate messages in Kafka due to network errors,
      • Namely, when Kafka successfully appends its message, but the ack from Kafka doesn't reach the producer, making the producer retry the same data, unaware that it's already been written.
      • What the idempotent producer does is send extra metadata in its retries that let Kafka detect and discard duplicate messages.
    2. (Note: As of Kafka 3.2.0, idempotence is enabled by default.)
  18. Why producer batching?
    1. Because we can increase throughput and improve latency by introducing some lag (eg. linger.ms=5) to make the Producer not instantly send each message each time we ask it to, and instead group them together for a given time and then send them in a batch, thus reducing the number of requests that the client sends to Kafka.
    2. (Note: Ie. Make a single request to a broker with a lot of data, instead of many requests with less data.)
    3. (Note: For any given batch, its data is only destined for a single topic partition, on a single broker.)
    4. Why batch.size?
      • Clue is in the name!
      • (Note: This config is an upper bound, not the exact size every request to the broker will have.)
      • (Note: Large batch sizes may increase throughput, at the cost of increasing the risk of eg. running out of space on the client.)
    5. Why linger.ms?
      • Because we want to optimize how long to wait before sending batch data.
    6. Why buffer.memory?
      • Because we want to control how much to allocate in memory (chunked into segments of batch.size) in the producer during batching.
      • (Note: Needs to be larger than batch.size.)
  19. Why batching metrics?
    1. Because batching is an interplay of various configurations (as shown above), so we need to keep track of how well out batches are performing with metrics, to know if we're doing it right.
    2. Why batch-size-avg?
      • Sanity check to see if we're grouping data properly.
      • Eg. If avg size is lower than the configuration, then maybe linger.ms is too low (ie. we're not waiting long enough to fill up the batch.)
      • Eg. If avg size is lower than the configuration, but linger.ms is already pretty high, then we're probably adding unnecessary latency.
    3. Why records-per-request-avg?
      • Another sanity check. Make sure we're batching.
  20. Why consumers?
    1. Because we need a way to read data from Kafka!
      • (Note: Consumers know which broker to read from.)
      • (Note: Consumers read data in order within each partition.)
    2. (Note: If you want to have a high number of consumers, you need a high number of partitions.)
    3. (Note: If there are more consumers than partitions, some consumers will be inactive.)
  21. Why consumer groups?
    1. Because we want to represent an "application." Eg. Node app, where multiple consumers can work together to consume data from a topic.
    2. (Note: Each consumer within a group reads data from exclusive partitions.)
    3. Because Kafka set out to do both "queue" (publish once, consume once) and "pub sub" (publish once, consume many times) models.
    4. Because we don't want consumers to know about partitions.
    5. Because a single consumer would be like someone "drinking from a strong water hose," ie. unable to handle all the data, so we use multiple consumers, but then two consumers might read the same message (eg. "Argentina scores 1 goal") and the system might incorrectly take it as two events (ie. Argentina now has 2 points instead of 1), so we group consumers in groups and say that only one consumer per group can read a message. So we get the scalability of multiple users, without event consumption duplication.
  22. Why __consumer_offsets?
    1. Because Kafka needs to store the offsets at which a consumer group has been reading, so that if a consumer dies, it will be able to continue from where it left off when it comes back on.
    2. (Note: The consumer chooses when to commit the offsets.)
  23. Why the three delivery semantics?
    1. Because different applications handle different degrees of data loss.
      • At Most Once: Offsets are committed as soon as the message is received. If the process fails, the message won't be read again.
      • At Least Once: Offsets are committed after the message is received and processed. If the process fails, the message will be read again until it works.
      • Exactly Once: Can only be achieved with the Kafka Streams API, and only from Kafka to Kafka or Kafka to external with an idempotent consumer.
  24. Why are consumers and producers fully decoupled?
    1. Because if producers have to know about consumers, wait for them, communicate directly in any way, etc. we can't scale well.
    2. Because we want everyone to read/write at their own pace.
      • Eg. (Admin) Create a topic:
        kafka-topics.sh --create --topic my-events --bootstrap-server localhost:9092
      • Eg. (Producer) Send some messages to Kafka:
        kafka-console-producer.sh --topic my-events --bootstrap-server localhost:9092
>event 1 happened
>event 2 happened
      • Eg. (Consumer) Read messages from Kafka:
        kafka-console-consumer.sh --topic my-events --from-beginning --bootstrap-server localhost:9092
event 1 happened
event 2 happened
      • (Note: When using Zookeeper, all the above commands must include a --zookeeper 127.0.0.1:2181 (or whatever your Zookeeper's address is) parameter.)
      • Eg. (Consumer - In Java) Read messages from Kafka:
        public class ConsumerDemo {
  public static void main(String[] args) {
    String bootstrapServers = "localhost:9092";
    String groupId = "my-app";
    String topic = "some-topic";

    Properties props = new Properties();
    props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId);
    props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

    KafkaConsumer<String, String> consumer =
      new KafkaConsumer<String, String>(properties);

    consumer.subscribe(Collections.singleton(topic));

    while(true){
      ConsumerRecords<String, String> records =
      consumer.poll(Duration.ofMillis(100));

      for (ConsumerRecord<String, String> record : records){
        log.info("Key: " + record.key() + ", Value: " + record.value());
        log.info("Partition: " + record.partition() + ", Offset:" + record.offset());
      }
    }
  }
}
  25. Why Zookeeper?
    1. To maintain harmony across clusters.
    2. (Ie. synchronize leaders and followers in the distributed system, with a gossip protocol.)
    3. Because Kafka original development focused on it being a robust message log, without reinventing the wheel for distributed consensus and coordination, so these tasks were delegated to Zookeeper.
    4. (Note: A collection of Zookeeper machines is called an ensemble.)
    5. Eg. Example setup for a messaging system:
      • Machine 1: Kafka and Zookeeper. (Broker; Leader.)
      • Machine 2: Kafka and Zookeeper. (Broker.)
      • Machine 3: Kafka and Zookeeper. (Broker.)
      • (Number of machines must be odd, because we're going to need a quorum and majority, for distributed systems reasons.)
    6. (Note: Zookeeper is its own distributed system. Writes go to the Zookeeper leader nodes. Reads are from Zookeeper follower nodes.)
    7. (Note: Kafka and zookeeper are separate installs.)
    8. (Note: Zookeeper used to store consumer offsets prior to Kafka v0.10.)
    9. Eg. Running Zookeeper:
      • zookeeper-server-start <KAFKA_DOWNLOAD_HOME_DIR>/config/zookeeper.properties
  26. Why KRaft mode?
    1. Because you may want to have Kafka itself handle cluster metadata management, leader election, partition reassignment, failure handling, and other tasks, instead of Zookeeper (which is its own beast with its own operational quirks.)
    2. (Note: See Raft consensus algorithm.)
  27. Why controllers?
    1. Because we need someone to manage the cluster.
    2. So there's always an Active Controller that manages the cluster. Another node will take its place when it goes down.
      • (Note: Controller nodes have a voting mechanism to choose who's the active controller.)
      • (Note: As the active controller manages the cluster, it will publish the metadata changes it makes to the __cluster_metadata topic.)
  28. Why pull-based message queues (as in pub-sub "brokers")?
    1. Because in the request-response model, the request requires resources in the backend in order to be served, and sometimes neither the single-threaded (eg. nodejs) nor multithreaded (eg. Apache) servers will cut it, ie. requests take too long to be computed, no matter what (regardless of concurrency model, load balancing, etc.), so we want a specific type of server that always does one constant-time operation (namely: append something to a queue), and respond to the user with some sort of identifier, and responds immediately, so the client has something, and can start polling using the queue id, until it gets what it needs. The problem becomes more unmanageable as a system has to chain many request-response services.
  29. Why Schema Registry?
    1. Because it's useful to have a schema-based contract between clients who produce to and consume from Kafka.
      • (Note: Which is something that is inherently lacking in event-driven architectures and is a cost of "loosely coupled" design in general.)
      • (Note: An analogy is made between lawyers in real life handling contract breaches and Schema Registry as an arbiter during schema changes.)
      • Ie. We want clients to have well-defined contracts for the data that they share.
      • (Note: Schema retrieval occurs once per schema and from that point on it is cached.)
      • (Note: Schema Registry supports Avro, JSON Schema, and Protobuf schemas.)
      • (Note: Schema management consists of: registering, updating, viewing, downloading, testing.)
      • (Note: Schemas can be registered using the confluence CLI, the REST API, or the gradle plugin.)
      • (Note: The schema subject name is its unique name that doesn't change as it evolves. The three naming strategies being TopicNameStrategy, RecordNameStrategy, and TopicRecordNameStrategy.)
      • (Note: Compatibility mode can be set at the subject level, so it's possible to have schemas with different compatibility levels.)
    2. Because it helps keep clients in sync as application requirements change and schemas evolve to respond to these changes.
    3. Because we want to deal with the common assumption Kafka clients make:
      • Producer assumes consumer will understand the message.
      • Consumer assumes producer will continue to send message in the same format.
    4. Because when a producer sends a message, first the key and value must be serialized into bytes (using separate serializers respectively). Ie. Kafka only transfers data in byte format.
      • Ie. There is no data verification done at the Kafka cluster level.
      • Ie. Kafka doesn't know if it's receiving strings, or numbers, etc.
      • Ie. Kafka doesn't know the meaning of the data.
      • But, obviously, the downstream consumer needs to know the meaning.
      • So we could use a way to agree upon a common data type.
      • So we have another separate cluster running, which we call Schema Registry, which is a centralized storage for all our schema metadata.
    5. Because we want to handle schema evolution without downtime (if producers and consumers are properly considered and compatibility modes are properly set and errors properly handled.)
  30. Why log segments?
    1. Because for efficiency Kafka does not actually keep the whole log as a big log file, so Kafka stores data in segments (indexed by offset) in a partition.
      • (Note: We can say a Kafka cluster contains brokers which contain topics which contain partitions which contain segments.)
      • (Note: ...and each segment contains a group of files: .log for data, .index for (offset, record), .timeindex, and .snapshot for producer sequence numbers for idempotency.)
      • (Note: The maximum size of a log segment file is configured by log.segment.bytes in the server.properties.)
  31. Why internal topics?
    1. Because Kafka must track its state changes somehow, and using itself, so to speak, makes sense.
    2. Eg.
      • __cluster_metadata.
      • __consumer_offsets.
      • __transaction_state.

References↑ #

  1. (Text/HTML) Kafka 3.8 Documentation @ Apache.Org
  2. (Text/HTML) Kafka 3.8 Documentation - The Producer @ Apache.Org
  3. (Text/HTML) Kafka 3.8 Documentation - Notable changes in 3.2.0 @ Apache.Org
  4. (Text/HTML) Featuring Apache Kafka in the Netflix Studio @ Confluent.io
  5. (Text/HTML) Keystone Real-time Stream Processing Platform @ NetflixTechBlog.Com
  6. (Video/HTML) Martin Kleppmann | Kafka Summit SF 2018 Keynote (Is Kafka a Database?) @ YouTube
  7. (Video/HTML) Danica Fine – Brick-by-Brick: Exploring the Elements of Apache Kafka® @ YouTube
  8. (Text/HTML) System Design: Why is Kafka fast? @ YouTube
  9. (Video/HTML) Apache Kafka Crash Course @ YouTube
  10. (Video/HTML) What is a Message Queue and When should you use Messaging Queue Systems Like RabbitMQ and Kafka @ YouTube
  11. (Video/HTML) When to use a Publish-Subscribe System Like Kafka? @ YouTube
  12. (Video/HTML) Publish-Subscribe Pattern vs Message Queues vs Request Response (Detailed Discussions with Examples) @ YouTube
  13. (Text/HTML) Efficient data transfer through zero copy @ Developer.IBM.Com
  14. (Video/HTML) USENIX ATC '14 - In Search of an Understandable Consensus Algorithm @ YouTube
  15. (Video/HTML) Key Concepts of Schema Registry | Schema Registry 101 @ YouTube
  16. (Video/HTML) Introduction to Schema Registry in Kafka | Part 1 @ YouTube

Kubernetes #

  1. Why Kubernetes?
    1. Because while containers and containerization is a great thing (because we don't have to provision and manage fleets of servers to achieve scalability), it has also led to new needs:
      • To orchestrate / babysit the multitude of containers that our modern systems need to have running and replicated across machines, etc.
      • To treat clusters of machines as a single resource.
      • Automated health checks, service discovery, autoscaling, etc.
    2. (Note: Kubernetes originated as a google utility called "Borg" which they then open-sourced.)
    3. Cluster, node, control plane, data plane.
  2. Why a control loop?
  3. Why a container runtime interface?
  4. Why a container network interface?
  5. Why a container storage interface?
  6. Why these?
    1. helm. package manager / template engine.
    2. kubectl. kubernetes client.
    3. kubectx. kubectl add-on for easy cluster switching.
    4. kluctl. Improving Kubernetes configuration management.
    5. KinD. Deploy kubernetes within Docker. Simple local development.
      • KinD main configs (YAML): kind (eg. Cluster), nodes (role (eg. control-plane, worker)).
    6. ko. Containerizing go apps.
    7. k9s. Observing kubernetes clusters.
    8. oras. OCI registry client.
    9. yq. Parsing and manipulating YAML.
  7. Example story of setting up a cluster for local development:
    1. Configure and deploy nodes using KinD.
    2. Configure and run load balancers within the kind cluster.
    3. (Optional, recommended) Use something like Civo to deploy a cluster remotely, to better simulate day-to-day development in a real project.
      1. Log in on some CLI, configure keys. (And also likely check things on a web dashboard.)
      2. Create network and ports for our cluster development.
      3. Create cluster.
      4. (Deal with "First time I tried to create it it failed, so I delete it and created it again and it worked." issues.)
    4. Configure and run cluster remotely (eg. a GKE Cluster on Google Cloud. By default, they manage the control plane, you manage the worker nodes. They've got good cluster monitoring.)
    5. Make sure to set clean up scripts and processes, to make sure once the cluster work, there's nothing left running that some cloud service might be billing you for in the future.
    6. Start working with actual Kubernetes resources.
      1. Namespaces. (There are full by default: default, kube-system, kube-node-lease, kube-public. It is suggested to create custom ones.)
      2. Pods. The smallest deployable unit in Kubernetes. Can contain multiple containers (primary, init, and sidecar containers.) Many configurations: Listening ports, health probes, CPU and mem, resource requests/limits, security, env vars, volumes, DNS policies.
        • (Note: Please don't create pods straight from the command line. All pods should be specified by versioned configuration.).
        • (Note: In the context of the note above: Deleting a namespace recursively deletes the resources under it, including all the pods. Which is convenient.).
        • It's important to specify memory limits for a pod, because if some program in it has a memory leak, it will eventually start to compete for resources with other pods on the same node, and kubernetes will evict other nodes, causing stability.
      3. Set up a ReplicaSet (almost never done manually/directly in real projects), which wraps a pod into 1 or more "replicas"
  8. Why replica sets?
    1. Because they maintain a static definition of a pod and keep the number of instances of said pod alive.
  9. Why deployments?
    1. Because if we want to modify a pod (update the container image, modify the resource parameters, etc.) we don't do it directly on the pod, and the replica set can't handle it, so we go yet a layer further up.
    2. Because they allow for "rollouts" and "rollbacks."
    3. Because they're great for long-running stateless applications.
    4. (Note: Mental model: Deployment wraps a Replica wraps a Pod.)
    5. (Note: We create a Deployment, Kubernetes in turn creates a ReplicaSet, and ReplicaSet creates and manages the underlying pods.)
  10. Why jobs?
    1. Because sometimes our "application" is a program that we want to run to completion. Ie. Still built on a pod, but with the notion of a "completion."
  11. Why CronJob?
    1. Because sometimes we want a job and additionally we want to run it periodically on a schedule.
  12. kubectl cheatsheet:
    • (where alias k=kubectl)
    • k <VERB> <NOUN> -n <NAMESPACE> -o <FORMAT>
    • k get pods -n 04-pod
    • k get pods -A
    • k get pods -l key=value
    • k explain <NOUN>.path.to.field
      • eg. k explain job.spec.backoffLimit
    • k explain pod.spec.containers.image
    • k logs <POD_NAME>
    • k logs deployment<DEPLOYMENT_NAME>
    • k exec -it <POD_NAME> -c <CONTAINER_NAME> -- bash
    • k debug -t <POD_NAME> --image=<DEBUG_IMAGE> -- bash
    • k port-forward <POD_NAME> <LOCAL_PORT>:<POD_PORT>
    • k port-forward svc/<DEPLOYMENT_NAME> <LOCAL_PORT>:<POD_PORT>
    • k apply -f Namespace.yaml (or some other config)
    • (Note: "apply" because kubernetes then changes the state of the cluster to reflect what we declare in the given config, while ensuring there's now downtime, ie. it doesn't update all pods at once. Ie. we stay "declarative" and the system is always running.)
    • k rollout restart deployment <DEPLOYMENT_NAME>
    • k rollout undo deployment <DEPLOYMENT_NAME>
    • k get svc
    • kubens
    • symmetry placeholder
    • curl <IP>.<NAMESPACE>.svc.cluster.local
  13. Why services?
    1. Because we want to send (load-balanced) network traffic to our application.
    2. Types of services are ClusterIP (internal network communication between nodes), NodePort, LoadBalancer.
  14. Why DaemonSet?
    1. Because sometimes we need to have an instance on each of our different nodes.
      • Eg. logs, metrics aggregation, to scrape what's been happening across all nodes and report it to some other system.

References↑ #

  1. (Text/HTML) Cloud tooling landscape @ CNCF.Io

Domain-Driven Design #

  1. Why Domain-Driven Design?
    1. Because focusing on learning about the problem (domain) leads to better solutions.
    2. To ensure that the development process serves the business needs.
    3. To find political constraints and impediments early.
      • To avoid wasting money in projects that are doomed to fail.
    4. To keep the software language close to the business language.
    5. To keep every piece of code clear about which purpose it's serving.
  2. Why Bounded Contexts?
    1. To keep the amount of context required for understanding at a minimum.
    2. So a team can have isolation and ability to move without others.
  3. Why an Anticorruption layer?
    1. To ensure the legacy part doesn't corrupt new part and viceversa.
    2. (Analogy: The "No outside shoes indoors" rule some places have.)
  4. Why Theory of Constraints with DDD?
    1. To look for bottlenecks to choose the best model.
  5. Why do "microservices" and DDD go well together?
    1. Because bounded contexts help find the right granularity for services.

References↑ #

  1. (Text/HTML) Domain-Driven Design in 2020 @ Blog.Avanscoperta
  2. (Video/HTML) Bounded Contexts - Eric Evans - DDD Europe 2020 @ YouTube
  3. (Video/HTML) The Art of Discovering Bounded Contexts by Nick Tune @ YouTube

Tolerance and prevention #

  1. Why "fault tolerance"?
    1. Because things will go wrong. Whether it's the user doing things wrong, or a hardware corrupting data, or a wifi dying, things will go wrong, and one should at least think about how to recover (if possible) from the main faults.
  2. Why "fault prevention"?
    1. Because even though fault tolerance is what really makes a system reliable, some faults are too stupid not to prevent them, eg. having database backups, to prevent full data loss in case of DB server catastrophe.
      • (And then having a process to use said backups, to tolerate the catastrophe. Thus Fault Tolerance and Prevention go hand in hand.)
      • (Note: Catastrophe is not necessary for things going down. Eg. You might have your program hosted on some paid Linux server, and the administrator might decide that there's an update, say an urgent security patch, that has to be applied, and therefore all servers will be restarted. Your system should tolerate such "scheduled downtime.")

Scaling #

  1. Why "elastic" systems (eg. AWS and similar IaaS)?
    1. Because sometimes you don't need, or can't afford, a human specialist to manually monitor load parameters and add computing resources as needed, as the system grows.
  2. Why not "elastic" systems?
    1. Because sometimes manually scaled systems are more predictable operationally.
  3. Why "scale up"?
    1. Because sometimes one big expensive powerful machine is the right tool.
  4. Why "scale out"?
    1. Because sometimes lots of small cheaper less powerful machines is the right tool.
  5. Why "scale up" and "scale out"?
    1. Because sometimes the right combo of big expensive machine plus small cheaper machines is the best approach.

References #

  1. (Text/HTML) How to Quantify Scalability

Typing #

Static types #

  1. Why "strong static typing"?
    1. Because, much like with FP, when done right, it eliminates an entire class of problems. Namely, the most annoyingly unnecessary runtime exceptions. If shit's gonna explode in production, at least make the explosion interesting! As opposed to a stupid mistake that would have been caught by a decent typechecker while writing the code.
  2. Why "structural" typing?
    1. I don't know.
    2. I guess because sometimes, specially in web dev where everything is some kind of "JSON," it's convenient for a function to say "just give me anything that has the structure { name: string, age: number }.
  3. Why not "structural" typing?
    1. Because you'll eventually have type Robot = { name: string, age: number } and type Cow = { name: string, age: number } and then a function function milk(cow: Cow) { ... } that will happily compile when accidentally called with a Robot, thanks to structural typing. And now you're just back to the same type unsafety of dynamic typing.
  4. Why "nominal" typing?
    1. Because sometimes you don't wanna have the "RuntimeError: Trying to milk a Robot" issue described above.
    2. Because sometimes you just want a Java type thing, where class Cow and class Robot are automatically different just be virtue of being different type declarations, regardless of their inner structure.

TypeScript #

  1. Why TypeScript?
    1. Because a lot of runtime errors can be turned into compile-time errors.
  2. Why any?
    1. Because maybe you're a horrible person, so you use any.
  3. Why unknown?
    1. Because it's a type-safe counterpart of any that allows only equality checks until we narrow it to a usable (ie. known) type.
    2. (Note: A value of type any is assignable to all values, whereas a value of type unknown is assignable only to unknown and any.)
    3. (Note: An unknown can be narrowed using typeof or instanceof or custom type guard functions.
  4. Why generics?
    1. Because many functions should work with literally every type. But you don't want any, because any loses type information.
      • So instead of any, we pass types as parameters (ie. parametric polymorphism, not ad-hoc polymorphism) so the compiler doesn't lose type information.
      • Eg. The identity function. Clearly, it should work with all types, since it does nothing but return the argument. Ie.: id(x) == x for all types. Using any would lose type information, so it's better to us generics function id<T>(x: T): T { return x } which allows the compiler to preserve the type information.
    2. Because parametric polymorphism is the best form of documentation. Eg.
      • type Foo = <A,B>(a: A, g: (a: A) => B) => B tells you everything.
  5. Why unions?
    1. To specify exactly which values can inhabit a type. Eg.
      type Theme = "dark" | "light" | "sunny"
type ChessRow = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
  6. Why tuples?
    1. Because we sometimes want to keep various values together in a structure, but don't need to label them.
  7. Why type?
    1. To name/alias a type, so we can use (and operate on) it. Eg.
      • type Actions = { run: string; jump: string }
  8. Why not interface?
    1. It's just a more limited and verbose way to declare a type.
  9. Why conditional types?
    1. To decide what a type should be based on some type-level condition. Eg.
      type NonZero<T extends number> = T extends 0 ? never : T

function divide<T extends number>(a: number, b: NonZero<T>): number {
    return a / b;
}
const x = 0
// divide(8, x);
// Compile-time error.
    2. (Super basic poorman's "dependent types.")
  10. Why mapped types?
    1. To derive a type from another type.
    2. (Usually to grab keys from an object type to create another object type.)
      type ActionsKey = keyof Actions
type EventHandlers = { [key in ActionsKey]: () => void }
const playerHandlers : EventHandlers = {
  run: () => {},
  jump: () => {},
};
  11. Why type templates?
    1. To further manipulate/customize derived types. Eg.
      type EventHandlersV2 = {
  [key in ActionsKey as `on${Capitalize<key>}`]: () => void
}
const playerHandlersV2 : EventHandlersV2 = {
    onRun: () => {},
    onJump: () => {},
}
    2. To generate combinatorial unions. Eg.
      type ChessRow = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
type ChessCol = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h"
type ChessCell = `${ChessCol}${ChessRow}`

// const test0 : ChessCell = "z"
// ^ Error: "z"' is not assignable to type '"a1" | "a2" 
// | "a3" | "a4" | "a5" | "a6" | "a7" | "a8" | "b1" | "b2" ...

// const test1 : ChessCell = "a9"
// Compile-time error.

const test2 : ChessCell = "h6"
// OK.
    3. For stronger compile-time string checks. Eg.
      type Protocol = "http" | "https"
type Domain = string
type Digit = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
type Port = `${Digit}${Digit}${Digit}${Digit}`
type APIURL = `${Protocol}://${Domain}:${Port}` 

// const apiUrl0 : APIURL = "htt://localhost:8080"
// Compile-time error.

const apiUrl01 : APIURL = "http://localhost:8080"
// OK.
    4. Why public, private, and protected?
    5. Why any, unknown, and never?
    6. Why declare?
      1. Because sometimes you need to call 3rd-party code (eg. a JS library) but there are no typings for the function(s) you want to call, so you declare the typing yourself with declare.
    7. Why ambient declarations?

References↑ #

  1. (Text/HTML) The TypeScript Handbook
  2. (Video/HTML) Deep Dive into Advanced TypeScript - Christian Woerz - Oslo 2023

Testing #

  1. Why test?
    1. Because we wanna know if it works before I deploy it, obviously.
  2. Why unit test?
    1. Because we want some verification that at least some of the calls to a function/component produce the expected result.
  3. Why generative testing?
    1. Because we want a (way) more complete verification that the code does the right thing in extreme and corner-case-y circumstances.
  4. Why e2e testing?
    1. Because it is close to an actual user testing it.
  5. Why human Q/A testing?
    1. Because it is the closest thing to an actual user testing it.

Software design #

  1. Why pure functions?
    1. Because you can easily test them and easily type them.
    2. (Before you tell me "Easy is not Simple," yes I've seen that talk. I've seen all those software designer-philosopher-guru talks. The functional guy, the uncle bob guy, the stack overflow guy, the other stack overflow guy, the design-pattern guy, the angry guy, the JS conf pothead guy, the Scala showman-presenter guy, etc. Spare me the slogans.)
    3. Concretely, because a function like const greeting = x => "Hello, " + x, can be easily tested / played with in a command-line setting without any ceremony.
  2. Why side-effect-y functions?
    1. Because as fun as FP is, at the end of the day programs have to do stuff!
    2. Because sometimes you're making a video game, and mutating a number with bitwise operators is preferable to constantly copying arrays for the sake of functional purity and "equational reasoning."
    3. Because sometimes you can just do what the OOP people do: Write mocks for everything in order to write tests.
  3. Why Event-driven?
    1. Because what's more natural than "when User clicks button, execute this code"? Let's not overthink this.
    2. Note: The DOM people made the right choice. Functional Programming for coding (real, non-trivial) UIs is just... clunky. Sorry, Haskell, Elm, Purescript, etc. people. Event-driven is just the most natural.
  4. Why not Event-driven?
    1. Because in data processing, on the other hand, connecting functions is the natural thing. It's just a digital form of electronic circuit design. You plug the right input-output boxes (ie. functions) in the right place, and it works. Guitar to pre to amp to interface to computer. It makes sense.
  5. Why async syntax in JS?
    1. For the same reason there is do-notation in Haskell: We programmers love imperative code, so we make syntax sugar for it. (See my definition of imperative code below)

Functional abstractions #

  1. Why write Your Own Monad?
    1. (Note: As opposed to just having a module with a bunch of adhoc functions.)
    2. Because sometimes you happen to have a well-defined structure along with a set of operations that can be seen as instructions.
      • (Note: My recommendation: Begin with a loose "modules." Try to make functions that operate on some common structure, ie. form some algebra. Keep an eye for any properties that may arise.)
  2. Why is Category Theory even a thing?
    1. Because whenever humans are trying to describe things, they end up on a whiteboard, drawing a bunch of some kind of objects (dots, letters, words, symbols) and a bunch of some kind of arrows to describe relationships. And after a zillion years of doing this, some mathematicians and other nerds started to notice patterns in these messy drawings (regardless of the contents), and called them categories and started to, well, categorize them.
  3. Why learn Category Theory?
    1. The delusional answer: Because knowing the mathematical theory from where the programming abstractions came from, will somehow help you.
    2. The realistic answer: Because you want to belong to some nerd club to fill some emptiness in your soul.
    3. (You could just use the FP functions that come with any FP library, and read the docs if you don't know what something does. You know, like a programmer.)
  4. Why Functors?
    1. To have a common interface for a certain type of transformation of a large number of data structures.
  5. Why Covariant Functors?
    1. Because it's useful that, abstractly speaking, these things have the same behavior:
      • Array.map.
      • OtherCollection.map.
      • Promise.map.
      • And even something like Function<A,B>.map.
    2. Mental exercise:
      • Speaking of Function<A,B>#map: You can only (covariantly) map over the output (ie. the B). Do you see why?
      • (Hint: A covariant functor maps over the contents of a metaphorical "box." If I forced you to think of a Function<A,B> as a box, which one makes more sense to view as its "contents," the input A, or the output B?)
  6. Why Contravariant Functors?
    1. (Note: this complements the last comments above)
    2. Because sometimes you want to "adapt the input" of some thing that takes an input. Like when you go to another country and the power plug on the wall is "weird," so you grab your adapter and plug it on the wall, and now the input is what you can use. You've "contravariantly" mapped over the hotel wall's power plug!
  7. Why Applicative Functors?
    1. Because sometimes you have, say, 3 things wrappedX, wrappedY, and wrappedZ (perhaps they're three results from three calls to different I/O operations, and it's not certain whether they all have their contents), and you want to write code that, if you squint your eyes, kind looks like you're just doing f(x, y, z).
    2. (Except it'll look more like liftA3 f wrappedX wrappedY wrappedZ)
    3. (Note: The code above is Haskell.)
  8. Why Monads?
    1. Because, much like "mapping over the contents of a box" with (covariant) Functor is a ubiquitous pattern, the "I called a function f that gave me a Maybe<User>, now I want to call a function g that takes a User, but obviously only call it if there is a Maybe<User> in the maybe box" is also ubiquitous.
    2. Because, if you've ever writen "GET USER; IF USER GOTTEN, GET USER DATA; IF USER DATA ..." pyramids of if-else, where you do an uncertain operation followed by a check for its data followed by another uncertain operation followed by a check of its data followed by another uncertain operation, and so on and so forth, then you've been doing manually the pattern that a Monad interface is supposed to handle (granted you're in the right language and/or using the right libraries). And if there's a sensible abstraction (or even syntax sugar like in Haskell and Scala) to do things less manually with "flatter" code, why not use it? Go monads.
  9. Why Monoids?
  10. Why Free Monad?
    1. Because sometimes you have some functor for your structure F<A> and it makes sense to have "layers" of this structure.
      • Eg. for a simple structure with two holes data Bin a a, a binary try data type is literally just type Tree = Free Bin. (A tree node is its "Pure" a. The recursive rest is
  11. Why OOP (class-based)?
  12. Why OOP (message passing)?
  13. Why propagators?
  14. (Haskell) Why pipes/conduits/machines/streams?

Categorical design #

  1. Why the identity arrow?
    1. Because we want to express proofs, isomorphisms, etc. which involve equations in which some arrow or composition of arrows "equals the identity."
      • Eg. f and g are "isomorphic" if g . f = id.
  2. Why Hom-set?
    1. To tersely refer to the set of all arrows from an object to another.
      • Eg. Hom(A, B) is the set of all arrows from A to B.
      • Eg. Hom(–, B) is the set of all arrows to B.
      • Eg. Hom(number, boolean) = "all the functions from number to boolean."
      • Eg. it's what the type a -> b in a generic signature is saying: All functions from a to b. Ie. a -> b is Hom(a, b).
  3. Why Hom-functor?
    1. To tersely refer to "the ways" to map from an object to another.
      • Eg. Covariant Hom-functor: "Fix the second argument in Hom, let the first vary." – Eg. Contravariant Hom-functor: "Fix the first argument in Hom, let the second vary." – Eg. Representable Hom-functor: A functor that's "essentially the same" as some Hom-functor.
      • Eg. the Reader Monad is a Hom-functor. Reader r is equivalent to Hom(r,–). Ie. all functions from r.
      • Eg. given a type a, Reader r will give you "all the functions from r to a, ie. the Hom-set Hom(r,a).
      • (Note: Reader r a is Hom(r,a), the Hom-set, "already applied," whereas Reader r, ie. the type constructor partially applied, is Hom(r,–), the Hom-functor which, given some type x, gives you a Hom-set, namely the Hom(r,x). Ie. Reader r a is a set/type, Reader r is a functor/type constructor.).
  4. Why functor?
    1. Because we want a way to map between categories while preserving the structure.
      • (Where "preserving the structure" means that for every relationship between arrows and objects in the source category, there'll be an equivalent in the target category. Basically a structured projection.)
  5. Why functoriality?
    1. ...
      • Eg. Functor-functor interaction law. FX, FY -> X,Y is saying that given a computation FX and an environment FY, we can get a computed value X and a state of the environment Y.
  6. Why bifunctor?
    1. Because sometimes you can and want to "map over" two values at once.
      • Eg. bimap g f (x,y)
      • Eg. bimap g f someEitherValue
  7. Why contravariant functor?
    1. Because sometimes you're mapping over inputs, ie. type parameters in negative position.
      • Eg. Given a type Comparison a = a -> a -> Bool , and a function b -> a, you can morph a Compare a into a Compare b.
      • (Note: The a is in negative position, ie. Comparison is not "a box that contains as" but rather a box that consumes as.)
  8. Why profunctor?
    1. Because sometimes you have a big box with a contravariant functor on its input side, and a good ol' covariant functor on its output side.
      • Eg. dimap :: (a -> b) -> (c -> d) -> p b c -> p a d.
  9. Why monad?
    1. Because real world computations are layered and/or nested and/or dependent on things, and the "monadic" structures treat this with rigor.
      • Eg. .flatMap to combine effectful actions, eg. loadUser(id): SomeMonad<User> followed by loadUserImagesFromS3(user): SomeMonad<Image[]>.
  10. Why comonad?
    1. Because sometimes you have some "environment," or "store," or "configuration," from which you can extract what you want, and you can also "grow" / "duplicate" the whole thing.
      • Eg. data Store s a = Store (s -> a) s deriving Functor.
  11. Why monad transformers (and stacks)?
    1. To have some coherent custom combination of different monads representing different effectful things (asynchrony, failure, etc.)
  12. Why is bottom problematic conceptually?
    1. Because by having for every type we must include it, which means we no longer (really) have coproducts.
      • (Note: It's still fine to talk about Haskell having products/records and "coproducts"/sum types in Haskell.)
  13. Why free monads?
    1. Because sometimes you want to write a description of a program instead of a program.
      • Easier to transform, compose, and understand. Typesafe reflection and aspect-oriented programming (without annotation mess.)
      • In Free[F, A], A is the value produced by the program, and F is the set of operations, ie. algebra, of the program, ie. the D in DSL.
  14. Why not free monads?
    1. Because by definition, you can't statically inspect them. Ie. You can declare a structure that "only describes" a M[A] -> A -> M[B] -> B pipeline, but by definition you can't inspect the A (let alone the B) at compile-time. You can do lots of things as you evaluate it step by step at runtime, but you can't statically inspect the whole thing.
  15. Why Cartesian Closed Categories (CCCs)?
    1. Because theorems about CCCs are theorems about typed lambda-calculus.
    2. Because one can think of types and functions, eg. in Haskell, as objects and arrows of a category, despite some serious theoretical problems caused by the programming language.
      • (Note: It's still useful enough to say that Haskell types and functions form a category, called "Hask," that is worth of study.)
      • (Note: Also, as long as you write total functions, and have a way convert your program / proof to point-free style, you can usefully "do category theory" in Haskell.)
  16. Why adjunctions?
    1. Because equality is too rigid, as it requires objects to be the same, and even isomorphism can be too restrictive, as it requires a reversible structure-preserving map. Whereas adjunctions give us a "weaker" equivalence, by relating two structures through functors in both directions along with a natural transformation that ties said functors together.

References↑ #

  1. (Text/HTML) Part I: From Theory to Pretty Pictures @ SchoolOfHaskell.Com
  2. (Video/HTML) Tarmo Uustalu: Monad-comonad interaction laws, monad algebras, comonad coalgebras @ YouTube
  3. (Text/HTML) Re: [Haskell-cafe] Basic question concerning the category Hask (was: concerning data constructors) @ Mail-Archive.Com
  4. (Text/HTML) "What Category do Haskell Types and Functions Live In?" @ Sigfpe.Com
  5. (Text/HTML) Some thoughts on reasoning and monads @ Sigfpe.Com

Algebraic design #

  1. Why isomorphism?
    1. Because sometimes we have two things A and B which, in a given situation, "for all intents and purposes" are "the same."
    2. Because sometimes we want to say "A and B are not exactly the same, but there's a two-way transformation between them."
      • Eg. In Set, {x, {y,z}} and {{x,y}, z} are not literally the same thing, but clearly we can rigorously transform one to the other.
  2. Why monoid?
    1. Because combining things is important and monoid is a term used when doing it rigorously.
      • Eg. You've defined some config language. Monoidal rigor means configs can be combined coherently.
      • (Note: By doing composition "rigorously" we mean precisely the following: (a <> b) <> c = a <> (b <> c), and mempty <> a = a = a <> mempty are laws satisfied by the compositional system.)
      • (Note: Additional laws may be added, eg. semilattice laws for when a <> b = b <> a and a <> a = a, eg. merging objects/records, but without satisfying the two monoid laws above, you can't claim to be building a serious compositional system.)
      • Eg. Ints with addition form a monoid. In List(1, 2, 3, 4).fold(0)(_ + _), the numbers are arrows in the monoid category, with 0 being the identity/mempty.
  3. Why algebra?
    1. To tersely refer a set of operations that combine data of some type, the whole thing being conceptualized as a "structure."
      • Eg. If you have a Tree, you can "fold" it. (Catamorphism.)
  4. Why coalgebra?
    1. To tersely refer to the "unfolding" of a structure into simpler parts.
      • Eg. Design a coalgebraic process where you start with a high-level configuration which gets recursively "unfolded" into specific smaller configurations for interpretation, etc.
      • Eg. If you have a seed, you can "unfold" it into a Tree. (Anamorphism.)
      • (Note: Easy in Haskell because it's lazy.)
      • (Note: Finite but arbitrary size data.)
      • Eg. If A is the seed (the type called the "carrier"), obviously you need some f : A -> F A (where F A is a tree or something) to start "growing." You then need a way to "access the new seeds" in order to keep growing, ie. a functor.
      • Eg. (Continued) F A = 1 + 2 * A, where 1 is the empty node (terminal object), 2 is a doubleton (ie. a data type with two possibilities, eg. boolean), paired with the new seed (ie. A).
      • Eg. (Continued) the whole thing, ie. (A, A -> F A), is a coalgebra.
      • (Note: You can have many different coalgebras for the same functor.)
      • (Note: Finding a coalgebra morphism for your adhoc types might be impossible, but coalgebras do form a category. And a terminal object in the category of coalgebras, ie. all other coalgebras have a unique arrow to it. And this terminal object is a fix point.)
      • (Note: Anamorphism also generate "infinite" data structures.)
      • (Note: In traditional FP, ie. functional design that isn't particularly algebraic, many algorithms are done recursively or co-recursively. In real, complex scenarios, they are difficult to grasp. Using algebraic-coalgebraic design, we can let the data structure drive the logic, as opposed to clever use of "logic" / "control." Ie. structuring computation using data structures.)

Type-theoretical design #

  1. Why polarity?
    1. Because sometimes it matters whether a type parameter is in positive or negative position, in order understand certain abstractions.
      • Eg. the difference between covariant and contravariant functor.
      • (Note: A plain type is in positive position. Return types are in positive position. Function arguments are in negative position.).

Semantics #

  1. Why induction?
    1. Because it is the main weapon a compiler designer has to prove properties about his programming language.

FP-TS #

  1. Why fp-ts? docs
    1. Because it brings to TypeScript many FP abstractions functional programmers know and love from Haskell, Scalaz, etc.
    2. (Note: And as explained elsewhere, we use FP abstractions because we want to encode as many things as possible as values, so we can inspect, log, reuse, type and statically check them. Importantly: Tracking errors on a type level.)
  2. Why pipe?

References↑ #

  1. (Text/HTML) fp-ts @ GitHub.io

Effect #

  1. Why effect? docs
    1. Because it's the evolution of FP-TS, ie. extending the functional programming notion of making everything a value, allowing us to track errors and context on a type-level, with low-latency concurrency, in TypeScript.
  2. Why "imprecise" non-mathematical names for abstractions?
    1. Because imprecise names for something that people will use is better than precise name that scare people away.
  3. Why "service"?
    1. Because we want "reusable components" that provide specific functionality.
  4. Why "tag"?
    1. Because Effect needs a unique identifier to locate the service.
    2. (Note: Tracked by the type system at compile-time.)
  5. Why Context?
    1. Because we need a collection for storing the services.
      • Essentially, type Context = Map<Tag, Service>
  6. Why Layer?
    1. Because we want to keep service interfaces clean and focused.
    2. To separate implementation details from the service itself.
    3. Because service construction can get complex, due to services depending on other services. So layers manage dependencies during construction rather than at the service level.
      • Eg. Layer<Srv,Error,Deps> is a blueprint for constructing Srv.
      • Eg. Layer<Config>: Config does not depend on other services.
      • Eg. Layer<Logger, never, Config>.
      • Eg. Layer<DB,never,Config|Logger>.
      • Eg.
        const LoggerLive = Layer.effect(
  Logger,
  Effect.gen(function* () {
    const config = yield* Config
    return {
      log: (message) =>
        Effect.gen(function* () {
          const { logLevel } = yield* config.getConfig
          console.log(`[${logLevel}] ${message}`)
        })
    }
  })
)
  7. Why scopes?
    1. Because resource leaks are unacceptable in web applications.
      • Eg. Unclosed socket connections, database connections, file descriptors, etc.
    2. Because we want to ensure that every time we open a resource, we have a mechanism to close it.
      • (For both success and failure.)
    3. Because we want to represent the lifetime of our resources. Ie.
      • Add finalizers, which represent resource release.
      • Close the scope, which releases the resource.
      • (...and runs the finalizers in the right order.)
    4. (Note: Scopes are defined with eg. Effect.acquireRelease)
  8. Why do scope finalizers run in reverse order?
    1. Because the following order of operations is the proper one:
      • acquire local directory.
      • acquire network connection.
      • acquire remote file.
      • release remote file.
      • release network connection.
      • release local directory.
  9. Why the type parameters <R, E, A>?
    1. Because pretty much every task you'll ever do is representable by having a type for injectable resources (R), a type for errors (E), and a type for the output value (A).
  10. Why tags on types?
    1. Because they help us organize our dependency injection.
      • Eg.
        class UserRouter extends HttpRouter.Tag("UserRouter")<UserRouter>() {}

References↑ #

  1. (Text/HTML) effect @ Effect.Website

Data encoding #

JSON #

  1. Why JSON?
    1. Because everyone uses it because this guy Douglas Crockford one day decided that this was a good idea.

XML #

  1. Why XML?
    1. Because unlike JSON, you're allowed to specify a rich structure.

Protocol Buffers #

  1. Why protocol buffers? spec
    1. Because they support strongly typed schema definitions, are efficiently encoded in binary, and have wide tooling support.

References #

  1. (Text/HTML) Protocol Buffers Version 3 Language Specification @ Protobuf.dev

Database engineering #

  1. Why databases?
    1. To store data that can be found again.
  2. Why not just use files instead of databases?
    1. Because database engines optimize for minimal storage overhead as well as fast access.
  3. Why caches?
    1. To store the result of expensive (in time and/or space) operations.
  4. Why streams?
    1. In the node.js sense: To process large amounts of data without loading it all into memory first.
    2. In the general sense: Because sometimes you can't even try to load all data into memory because it doesn't even make sense. Eg. ongoing live stream from audio device without a "last" element.
  5. Why batch?
    1. Because sometimes periodically crunching accumulated data is what's needed.
  6. Why full-text search servers?
    1. Because sometimes the text-searching capabilities of the "main" application database (whether SQL or "NoSQL") isn't enough. So you use something like Solr which has some "understanding" of how words work, so you can have text searches that smarter than strict text-matching. Eg. Know to grab documents containing the world "apples" even though the user searched for "apple," as well as automatic generation of sub-searches and suggestions (eg. Solr's "faceted search")
  7. Why domain-specific databases?
    1. Because of the general case of the above point. Some databases are designed for a specific type of use-case.
  8. Why SQL?
    1. Because most data you will ever work with is relational.
    2. Because most projects you will work on do benefit from keeping good ol' relational schemas.
  9. Why NoSQL?
    1. Because sometimes you can get away with not thinking about schemas or relations upfront, and just throwing data in "collections" in Mongo or something, and then later figuring out how to related data either by reinventing relational logic at the application level, or using some relational features that NoSQL database have built into their engines.
    2. Because sometimes being able to just throw a complex, possibly nested, JSON-like object into a "collection," without specifying schemas, is enough.
  10. Why GraphQL?
    1. Because you're tired of reinventing "REST" API server/frontend for things such as specifying which fields from the user object should be included in the request, etc.
  11. Why not GraphQL?
    1. Because ultimately there's no free lunch and there is no one-size-fits-all solution for the more complex relational data issues (performance, etc.) that your project may have.
    2. (But if you're project is basic relation-wise, graphql could save you a ton of API server and client development time.)

Database indexing #

  1. Why database indexes?
    1. Because they help us find data faster, as well as aid with sorting.
    2. Because without them, you'd have to search the table from beginning to end, aka. Full Table Scans (FTS).
    3. Because Full Table Scans (FTS) are slow.
      • Eg. Scanning a million metal band rows before getting to the "Megadeth" row is too slow.
    4. Because we want to search one million stuff faster.
    5. (Analogy: It's like an old school phone book, where some pages have the alphabet's letters sticking out of some pages, so you have a shortcut to the data, ie. allows you to say "Ah, the phones for names that begin with M start at this page, so I can skip all the previous pages".)
    6. (Note: Ways to make searching a million things faster: Parallelize, ie. split the table and distribute. Partition, ie. values from id 1 to 1000 at one place, values from 1000 to 2000 at another, etc., ie. essentially break a table into multiple tables. Index, ie. create a separate data structure to work essentially as a phone book's index, ie. row 1000000 is where band names starting with M begin, so we can start searching for "Megadeth" from there.)
  2. Why not binary trees for indexing?
    1. Because the problem with vanilla binary trees is that in practice they're rarely "balanced."
  3. Why B-tree indexes? visualization
    1. Because they are basically binary trees that auto-balance on write (which has a performance cost, which is usually worth it, because it makes reads way faster.)
    2. (Note: In PostgreSQL, B-tree indexes allow a query's ORDER BY to be honored without a separate sorting step, because by default index entries are stored in ascending order with nulls last, so a forward scan of an index on column x produces output satisfying ORDER BY x ASC NULLS LAST.).
    3. (Note: B-tree nodes typically contain many data per node.).
    4. (Note" B-tree indexes allow us to get closer to the ideal "binary search" situation that binary trees don't yield in practice. Ie. Quickly discarding large chunks of the dataset, ie. big reductions of the search space, along with minimizing I/O operations.)
    5. (Note: B-trees were invented in the 70's by Rudolf Bayer and Edward McCreight, working at Boeing, to improve efficiency by decreasing seek operations when reading lots of data from disk.).
    6. (Note: It is actually B+-trees that are used by your favorite B-tree index database engine, but we still refer to them as B-trees.)
    7. (Note: And there are tons of B-Tree variations: B-tree, B+-Tree, Blink-Tree, DPTree, wB+-Tree, NV-Tree, FPTree, FASTFAIR, HiKV, Masstree, Skip List, ART, WORT, CDDS-Tree, Bw-Tree, HOT, KISS_Tree, VAST-Tree, FAST, HV-Tree, UB-Tree, LHAM, PBT, Hybrid B+-Tree.)
  4. Why are database indexes larger in MySQL compared to PostgreSQL?
    1. (Note: In PosgreSQL, an index is essentially a key-value store (indexedColumns) => [page, itemID], or rather a sorted list of [page, itemID] tuples.)
  5. Why is the index page size important?
    1. Because the more elements in a page, the more we can search in a single I/O operation.
    2. (Note: In PosgreSQL, the default page size is 8192 bytes, and indexes can have up to 32 columns).
  6. Why should one try to fit the index in RAM?
    1. Because we're searching on the index all the time.
    2. Because if the index fits in the RAM, there's no memory limitation, only CPU limitation, ie. we're "CPU-bound," ie. bound only by how many threads we can spin up to search the index.
    3. Because if the index doesn't fit in the RAM, then we're "I/O bound" (hard disk paging, etc.), which is worse.
  7. Why would an index result in worse performance?
    1. Because a poorly constructed query (and/or index) can result in more I/O operations than even a plain old FTS.
    2. (Note: It is said of indexes that one should have "As many as necessary, as few as possible.")
  8. Why would an engine fail to use an index?
    1. Because the planner/optimizer might be unable to do so, or it might decide that a plain scan is actually better for some particular case.
      • (MySQL) Using functions in WHERE clauses (generally: transforming the data which is used by the index). Eg. YEAR(created_at) in a WHERE clause, on a table indexed by that column. MySQL won't use the index because created_at != YEAR(created_at).
      • PostgreSQL won't use the index if the table is small, because the I/O of using the index is likely slower than a plain scan.
      • (Note: "There is no plan that can beat sequentially fetching 1 disk page." From the PostgreSQL docs on index usage.)
  9. Why are indexes Not a Free Lunch (NAFL)?
    1. Because they have write performance costs, as well as memory costs.
    2. (Note: B-tree writes normally involve random I/O, and updating multiple pages on disk.)
    3. (Note: See RUM conjecture. In short: You can only optimize two out of these three at once: Read, Write, Memory.)
  10. Why are indexes a developer concern, and not just a DBA concern?
    1. Because indexes are built for specific queries, with knowledge of how the application's data is going to be accessed.
  11. Why are execution plans useful?
    1. Because SQL queries are "declarative," ie. we write a description of the data we expect, not how to fetch it at the low-level. The "execution plan" (in most SQL engines, the output of an EXPLAIN query) tells us more about what's happening at the low level, so that we can debug and/or optimize.
      • In MySQL the output of EXPLAIN has, among other things, the "access type" of a query, ie. which tells us what type of indexing it's doing, if any. (ALL means it's not using any index, which is usually a sign that either the query and/or the index need fixing.)
      • In PostgreSQL, EXPLAIN ANALYZE <QUERY>.
  12. Why index-only queries whenever possible?
    1. Because if the data you need is in the index itself, no I/O operations are necessary, and the query is faster.
  13. Why partial indexes?
    1. Because one of the costs of indexes (in addition to slower writes) is the RAM needed to use them, which can get too large when many users are making reads, so in eg. PostgreSQL we can pass a predicate so that only rows that match it will be indexed.
      • Eg.
        CREATE INDEX ON <table> (<column>, ...)
WHERE <some condition>
    2. (Note: Smaller indexes are faster, too).
  14. Why bitmap indexes?
    1. Because they're really fast for large tables with low cardinality columns, ie. colums whose data types are small, and for applications which are read-heavy, eg. data warehouses.
    2. (Note: The bitmap index is essentially a table where each row has one of the possible values the indexed column can have (eg. for a credit card type: visa, mastercard, etc.), and a bitmap indicating which rows contain this value.")

References #

  1. (Video/HTML) PostgreSQL Indexing : How, why, and when @ YouTube
  2. (Text/HTML) Indexes and ORDER BY @ PostgreSQL.Org
  3. (Video/HTML) B-tree vs B+ tree in Database Systems @ YouTube
  4. (Text/HTML) Let's meet B-trees @ Apache.Org
  5. (Text/HTML) B-trees @ CS.USFCA.edu
  6. (Text/HTML) CREATE INDEX @ PostgreSQL.Org
  7. (Text/HTML) Examining Index Usage @ PostgreSQL.Org
  8. (Video/HTML) Things every developer absolutely, positively needs to know about database indexing @ YouTube
  9. (Video/HTML) Things every developer absolutely, positively needs to know about database indexing (2023) @ YouTube
  10. (Video/HTML) "Modern B-Tree techniques" by Dmitrii Dolgov (Strange Loop 2022) @ YouTube
  11. (PDF/HTML) Designing Access Methods: The RUM Conjecture @ Harvard.Edu

Query planning #

  1. Why a query planner?
    1. Because the best specific way to implement our "declarative" SQL (or "NoSQL") queries isn't part of the queries themselves, so most database engines typically kepp some up-to-date statistics on certain properties of our tables, etc. which they then use to decide on the best way to fetch the data.
    2. (Note: This is what EXPLAIN statements show us in eg. MySQL and PostgreSQL: The query plan, without running the query, ie. what the expected costs are.)
      • Eg. PostgreSQL's EXPLAIN docs:
        EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 7000;

                         QUERY PLAN
------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..483.00 rows=7001 width=244)
   Filter: (unique1 < 7000)
    3. (Note: This is what EXPLAIN statements show us in eg. MySQL and PostgreSQL: The query plan, without running the query, ie. what the expected costs are.)
    4. (Note: In PostgreSQL, EXPLAIN tells us: Execution time, planning time, sort method, scan method, join method, join order, number of rows, indexes used, trigger time, etc.)
  2. Why is EXPLAIN Not A Free Lunch (NAFL)?
    1. Because it won't tell you why the planner made its choices.
    2. Because it won't tell you about how other factors (multiple DB sessions hitting the same table, OS and FS-level stuff, network, etc.) are affecting the query's performance.

References #

  1. (Text/HTML) Using EXPLAIN @ PostgreSQL.Org
  2. (Video/HTML) A beginners guide to EXPLAIN ANALYZE – Michael Christofides @ YouTube
  3. (Video/HTML) Explaining the Postgres Query Optimizer | Citus Con: An Event for Postgres 2022 @ YouTube
  4. (Video/HTML) Episode 417: Alex Petrov on Database Storage Engine @ YouTube

SQL #

  1. Why join?
    1. Because you rarely will get all you need by querying a single table (particularly the more "normalized" the database is), so you bring data from another table, joined by some predicate.
    2. (Or, less common in my experience, but also useful) you may want to check multiple rows of the same table at once.
  2. Why inner join?
    1. Because a simple set intersection often is all you need.
  3. Why left or right join?
    1. Because sometimes you're like "I want all rows from A no matter what, but if they happen to have data in B, I'd like to have that too."
    2. Because sometimes a basic set intersection is too strict, and leaves rows out that you want to keep, so you need some form of set union instead.
  4. Why "normalize"?
    1. To reduce redundancy.
  5. Why reduce redundancy?
    1. To save space and keep data integrity.
    2. (Even in "NoSQL" database like mongo you end up reinventing relational logic by using ObjectIds as manual "foreign key"s across collections.)
  6. Why Universally Unique Lexicographically Sortable Identifier (ULID)?
    1. Because the randomness of UUIDs makes them subpar/impractical for matters of data storage and structure. Eg. sorting indexed rows, clustering things together, etc.

References↑ #

  1. (Text/HTML) PosgreSQL 16 documentation – 2.6. Joins Between Tables
  2. (Text/HTML) Universally Unique Lexicographically Sortable Identifier @ GitHub

Networking #

TCP #

  1. Why Transmission Control Protocol (TCP)? spec
    1. Because the internet (IP) is a strong but unreliable packet routing system, where packets often arrive out of order at the other end, and it would be a pain for server applications to have to handle reordering (among other things), so TCP was invented to handle this, and have a reliable ordered stream.
  2. Why the ARP Table for Address Translation?
    1. Because IP address and Ethernet address are selected independently, so they cannot be calculated algorithmically.

References↑ #

  1. (Text/HTML) TRANSMISSION CONTROL PROTOCOL @ IETF.Org
  2. (Text/HTML) A TCP/IP Tutorial @ IETF.Org
  3. (Text/HTML) 4.1 ARP Table for Address Translation @ IETF.Org

HTTP #

  1. Why HTTP/1?spec
    1. Because we want a standardized and platform-independent protocol for communication between clients and servers over the internet, based on the idea of "request" and "response," and "documents" that link to each other (by means of "hypertext").
  2. Why HTTP compression?
    1. Because most text has a lot of redundancy, and HTTP involves a lot of text.
      • (Eg. HTML, CSS, JS, JSON, XML, and SVG "images," are all just text.)
      • (Images and audios are typically already compressed binary formats.)
      • The most common compressor is gzip, which uses the Deflate algorithm.
      • Brotli, by Google, is a more recent contender.
      • Browsers say which to use in a header. Accept-Encoding: br, gzip.
  3. Why not compress non-text files?
    1. Because you might actually make a file larger.
      • Eg. Compressing an MP3, which is already maximally compressed, will likely just add extra headers and dictionaries.
  4. Why minification (or preprocessing in general) before compression?
    1. To give the compression algorithm compression-friendly input.
  5. Why gzip?
    1. Because although it's not the best, the tradeoffs are acceptable for most HTTP communication.
    2. Because it is fast at both compressing and decompressing.
      • Which is important for (de)compressing on the fly.
    3. Because the memory it uses is independent of the size the data, as it operates on fixed-size chunks of data at a time.
    4. Because there are free implementations that avoid patent trolls.
  6. Why HTTP/2?
    1. Because HTTP/1 is limited to one TCP connection per request.
  7. Why HTTP/3?
    1. Because HTTP/2 went wrong.
      • Head-of-line blocking hurts user experience, because TCP's hard ordering guarantees.
  8. Why is HTTP/3 "peculiar" or a "workaround"?
    1. Because it's based on top of UDP, which is designed for unreliable communication, so HTTP3's QUIC is a workaround the impossibility of optimizing TCP handshaking, because middlemen network nodes (eg. NATs) have ossified the transport layer.

References↑ #

  1. (Text/HTML) Hypertext Transfer Protocol -- HTTP/1.1 @ IETF.Org
  2. (Text/HTML) gzip
  3. (Text/HTML) DEFLATE Compressed Data Format Specification version 1.3
  4. (Text/HTML) Brotli
  5. (Video/HTML) Everything You Need to Know About QUIC and HTTP3 @ YouTube.Com
  6. (Video/HTML) Horrible, Helpful, http3 Hack - Computerphile @ YouTube.Com

Caching #

  1. Why cache-control? docs
    • So we can control whether the cached response associated with a certain requests belogs to the private (private) or shared (public) cache or no cache at all (no-store).

References&caching; #

  1. (Text/HTML) HTTP Caching @ Mozilla.Org

Web security #

  1. Why sanitization?
    1. Because user data might contain malicious executable code.
      • SQL injection, when the user is able to input an SQL into a form that the system assumes is clean data.
      • XSS attacks, when the user is able to input JS that will then be loaded by other users to programmatically steal or break data.

Authentication and Authorization #

JWT #

  1. Why JSON Web Token (JWT)? rfc
    1. Because it's stateless and scalable.
    2. Because XML-based schemes like SAML are too cumbersome and verbose for space constrained environments such as Authorization headers and URI query parameters.
    3. Because it's URL-safe.
      • (By using a URL-safe variation of Base64.)
    4. Because it is "easy to implement using widely available tools."
  2. Why is JWT "stateless" and "scalable"?
    1. "Stateless" because everything is in the JWT, signed and/or encrypted.
      • Unlike, say, "Session + Cookie" schemes, where the user sends a session ID in a cookie, and backend has to load the authorization data, ie. state.
    2. "Scalable" because more users does not mean more DB queries for authentication/authorization purposes.
    3. "Scalable" because the user may interact seamlessly with many different backends, since all his authorization data is in the token.
  3. Why always verify the JWT's signature?
    1. Because it's super easy to crack a JWT by guessing weak secrets.
    2. (^So you should never just jwt.decode() the payload.)
  4. Why JSON Web Encryption (JWE)? rfc
    1. Because JWTs are not encrypted by default. They're just signed (either symmetrically or asymetrically) to maintain integrity (and in the asymmetric case, non-repudiation), but anyone can inspect the payload.

References↑ #

  1. (Text/HTML) JSON Web Token (JWT) @ IETF.Org
  2. (Text/HTML) Introduction to JSON Web Tokens @ JWT.IO
  3. (Text/HTML) JSON Web Token Best Current Practices @ RFC-Editor.Org
  4. (Video/HTML) Cracking JSON Web Tokens @ YouTube.Com
  5. (Text/HTML) JSON Web Encryption (JWE) @ IETF.Org
  6. (Text/HTML) Understanding JSON Web Encryption (JWE) @ ScottBrady91.Com

OAuth 2.0 #

  1. Why OAuth 2.0? rfc
    1. Because of the delegated authorization problem.
      • "Give Yelp access to my Gmail contacts."
      • "Give Last.fm access to my Spotify history."
    2. Because we don't want to give our password to third-party apps.
      • Eg. What Yelp used to do: They'd asked for your gmail password so they could access your contacts. Which sounds insane nowadays!
    3. Because sometimes users want to allow third-party applications to obtain limited access to an HTTP service.
    4. Because we want to always input our password only to the app it belongs, and generate some "token" to serve as proof that we've consented to limited, granular access to our data.
    5. Because we want to clearly delineate between the actors in a delegated authorization scheme.
      • "Resource Owner": That's "you" (eg. your GMAIL account.)
      • "Resource Server": That's eg. the GMAIL server.
      • "Client": That's the third-party app that wants to access some of your GMAIL data.
    6. ("OAuth" is short for "Open Authorization.")
  2. Why scopes?
    1. Because we want to be granular about access levels.
  3. Why state?
    1. Because we want to prevent cross-site request forgery (CSRF) attacks.
  4. Why an Authorization Code separate from an Access Token?
    1. Because we want to reduce the risk of token theft.
    2. Because we don't want the user's browser (Front Channel) to see the access code, which could be captured by a plethora of malicious actors (browser extensions, malware, etc.), so the Authorization Server gives an authorization code to the browser, which pass it to the client app, who will use it through the Back Channel to get the Access Token from the Authorization Server.
    const express = require(‘express’);
const axios = require(‘axios’);
const app = express();

app.get(‘/auth/redirect’, async (req, res) => {
  const authCode = req.query.code;
  const RESOURCE_AUTH_URL = "https://oauth.gmail.com/token";
  const tokenResponse = await axios.post(RESOURCE_AUTH_URL, {
    code: authCode,
    client_id: ‘YOUR_CLIENT_ID’,
    client_secret: ‘YOUR_CLIENT_SECRET’,
    redirect_uri: ‘<http://localhost:3000/auth/redirect>’,
    grant_type: ‘authorization_code’
  });
  const accessToken = tokenResponse.data.access_token;
  // Use accesstoken to access protected resources
});
  5. Why is OAuth 2.0 prone to security vulnerabilities?
    1. Because many configuration settings necessary for keeping data secure are optional according to the specification.

References↑ #

  1. (Text/HTML) The OAuth 2.0 Authorization Framework @ IETF.Org
  2. (Video/HTML) OAuth 2.0 and OpenID Connect (in plain English) @ YouTube.Com
  3. (Text/HTML) Cross-Site Request Forgery @ IETF.Org
  4. (Text/HTML) OAuth 2.0 authentication vulnerabilities @ PortSwigger.Net

OIDC #

  1. Why OpenID Connect (OIDC)? spec
    1. Because OAuth 2.0 has been misused for authentication (instead of what it's intended for: authorization) often enough that finally a simple layer (OIDC) was added on top of OAuth 2.0 to do authentication right when an authorization server also provides end-user authentication.
  2. Why is JWT relevant to the discussion of OIDC?
    1. Because the primary extension that OIDC adds to OAuth 2.0 is the ID Token data structure, which is represented as a JWT.
  3. Why is non-repudiation an important property?
    1. Because you don't want an ID Token provider to be able to deny that they generated the token. This prevents disputes about token authenticity.

References↑ #

  1. (Text/HTML) @ OpenID.Net

Web architecture #

Server-side rendering #

  1. Why (modern) Server-side rendering (SSR)?
    1. Because we've outgrown the "SPA" approach, because the amount of JS code we send to the browser nowadays has become a UX issue.
      • (Which is a sort of "return to the old days" except with better tools for eg. streaming responses.)

References↑ #

  1. (Video/HTML) Understand the Next Phase of Web Development - Steve Sanderson - NDC London 2024 @ YouTube

Micro-frontends #

  1. Why micro-frontends?

Software architecture #

  1. Why have a software architect?
    1. Because it's useful to have someone dedicated to the slightly bigger picture, ie. someone who can answer at least these three questions:
      • How smoothly is the system running (ie. Operability)?
      • How understandable is the system (ie. Simplicity)?
      • How easy is it to make changes to it (ie. Extensibility)?
  2. Why microservices?
    1. Because sometimes extensibility grinds to a halt when a monolith grows beyond a certain size.
    2. Because sometimes you want components A and B to be worked on by different teams, as decoupled as possible and, in fact, on different repositories, using different programming languages.
    3. (Note: It's 2024 and it's still a pain to achieve truly "micro" microservices for frontend projects. Why?)
      • ("Mashups" are not it.)
  3. Why not microservices?
    1. Because if you do it wrong, you end up with just another monolith except more complex with more parts which are only decoupled in your imagination.

API networking #

General #

  1. Why SOAP?
    1. Because XML and schemas are cool (except when they're not).
  2. Why REST?
    1. Because ditching XML, going schemaless, and being able to do whatever the hell we want is better (except when it isn't).
  3. Why GraphQL?
    1. Because actually having a schema is cool again and having the server handle property-selection in API queries without us having to reinvent it for every project is practical (except when it isn't).

REST #

  1. Why is there tension between correctness/type-safety and REST/RESTful?
    1. Because normal programming interfaces can be statically checked, whereas REST calls are about strings and runtime-only checking.
    2. Eg. Compare:
      • import { bucket } from "aws"; bucket.create("a");
      • http.put("https://a.s3.amazonsws.com/)

References↑ #

  1. (PDF/HTML) Distributed Systems 4th edition @ Distributed-Systems.Net

gRPC #

  1. Why gRPC? docs
    1. Because it promotes the microservices design philosophy of coarse-grained message exchange between systems while avoiding the pitfalls of distributed objects and the fallacies of ignoring the network.
    2. Because it takes advantage of HTTP/2 features such as multiplexing, stream prioritization, and server pushing.
      • (It does work over HTTP/1.1 but it's not as performant there.)

References↑ #

  1. (Text/HTML) gRPC Documentation @ GRPC.io
  2. (Text/HTML) gRPC Motivation and Design Principles @ GRPC.io
  3. (Video/HTML) The RPC Revolution: Getting the Most Out of gRPC - Richard Belleville & Kevin Nilson, Google @ YouTube
  4. (Video/HTML) gRPC Crash Course - Modes, Examples, Pros & Cons and more @ YouTube

Algorithms #

Just some of the very few algorithms that actually interest me.

Big O notation #

  1. Why Big O notation?
    1. To tersely describe how something (a process, etc.) grows in required resources (CPU, memory) in a standardized way.
    2. Note by Noriega:
      • Just ask yourself "What shape/graph is this drawing?"
      • Then find the simplest O-notation to express the shape.
      • (O(n) for linear, O(n^2) for quadratic, etc.
  1. Why Minimax?
    1. To minimize the maximum loss (thus "minimax") in a game's turn, by looking ahead at all the possible outcomes, for as many turns ahead as practical.
    2. (There is an inverse algorithm for the inverse need, ie. maximizing the score, called "maximin.")

References↑ #

  1. (Text/HTML) Minimaxer Part 1 - Building a minimax library in Typescript

Appendix #

Additional references #

  1. (Text/HTML) How Web Works @ GitHub.Com/vasanthk
  2. (Text/HTML) What forces layout / reflow @ Gist.GitHub.Com
  3. (Text/HTML) On Layout & Web Performance @ Kellegous.Com
  4. (Text/HTML) You Won’t Believe This One Weird CPU Instruction! @ VaiBhavsagar.Com
  5. (Text/HTML) Compiling to categories @ Conal.Net

Related