A pleasant walk through computing

Comment for me? Send an email. I might even update the post!

Memstate: The Practical Argument for Big, In-Memory Data

Today I listened to a fascinating episode of Jamie Taylor's The .NET Core Podcast, featuring an interview with developer Robert Friberg. Do listen to it.

Friberg and his team have been working on Memstate, an in-memory database. When I first started listening, I thought "this sounds like one of those interesting, technical edge-cases, but I'm not sure I see the point." By the end, I not only understood, I wanted to start using it! But since I don't know how soon that will happen, here's the compelling meat.

Relational databases solve a problem that doesn't exist anymore. That problem is limited memory relative to dataset size.

Your OLTP data can probably all fit in RAM.

Here, simplistically, is how Memstate looks compared to a relational database.

In a relational database, the data is: Read into Memory > Changed in Memory > Saved to Transaction Log > Saved to Storage > (Potentially) Reread into Memory

In a memory image, the data is: Already in Memory > Changed in Memory > Change is saved to Transaction log

The idea of keeping all needed data in RAM has been around for awhile. Friberg recommends reading Martin Fowler's description of the pattern, and so do I. This tidbit helped me grasp how common the concept is:

The key element to a memory image is using event sourcing.... A familiar example of a system that uses event sourcing is a version control system. Every change is captured as a commit, and you can rebuild the current state of the code base by replaying the commits into an empty directory.

I think this is fantastically straightforward. Why persist state on every write? Why not just persist the change, and keep the state in memory? In the past, the answer was "We can't fit all the data into memory, so you have to read it each time." But as Friberg points out, you can configure an Azure virtual machine instance with 1 terabyte of RAM!

Friberg is clear that this method, like any, shouldn't be applied to all use cases. He nicely points out that even if you kept, for (his) example, all of IKEA's inventory levels for all its warehouses in memory, you wouldn't keep all the product descriptions and images. Consider what we already do today with blob assets:

  • We don't (typically) store images in version control. We reference it externally.
  • We keep images in cache because it's static.

(And, of course, it also doesn't mean you couldn't keep that data in memory.)

Imagine if your order entry system data was running fully in memory. Now, ask--not rhetorically--could it? The answer will very likely be yes.

What happens if you need to perform maintenance? Surely it would take forever to replay all those millions of commands (transactions).

Friberg gives an example where a very large dataset could be loaded back into memory in about ten minutes.

Finally, while most of the podcast discussed big data, I wonder about small data. One of the persistence stores for Memstate is--you guessed it--a file. I think this would be a great solution for any app that needs a small database. The state would load fast, the app would run fast, and there wouldn't be any of the overhead of using a database engine.1

Want to go a step further? If the store is a file, and if the amount of data isn't huge, and if absolutle top performance isn't necessary, then I'll bet the transaction information could be stored as plain text. And this means your data and history are future-proof.

And that's how I like my data.

References


  1. Not that, for instance, SQLite is big. I think it's just a .dll.

Grammar for Developers - Apostrophes: 3 Simple Rules, 3 Common Mistakes

It's a truism that (many) developers are not good writers. And it's not just style or composition, it's grammar. This has always puzzled me, since if a programmer makes a grammar mistake (using a backtick instead of an apostrophe, for instance), the program won't run. Maybe the problem is that they don't pay attention to the red squigglies in Word.

Arguably the worst error I see is wrongly using an apostrophe when "its" is possessive. Here's a sentence to illustrate. Every apostrophe is correct.

Harvey's car's in the shop to have its radiator fixed before it's blown to bits.

Expand the contractions and the differences between possessives and contractions are clear. Harvey's car is in the shop to have its radiator fixed before it is blown to bits.

Here are three rules that, if followed, would take care of about 70% of the most grievous errors I see in blog posts, emails, business proposals, and technical documentation.

I even made a fancy PDF.

Grammar for Developers - Apostrophes

3 Simple Rules

  1. Never make a pronoun possessive using an apostrophe
  2. Always make a noun possessive using an apostrophe
  3. Never use an apostrophe with a verb

3 Common Mistakes

Right: Unit testing has its benefits.
Wrong: Unit testing has it's benefits.

Right: Carol's code is clean. Her code's style is precise.
Wrong: Carols code is clean. Her codes style is precise.

Right: He runs that backup every day.
Wrong: He run's that backup every day.

Terms

A noun is a specific person, place, thing or idea: Carol, France, house, radish, memory, Chipotle.

A pronoun is a generic, non-specific person or thing. These are the common pronouns: it, he, she, her, them, they.

A verb is an action taken by the subject. He codes. She tests. They demonstrate.

Weekly Sugar: 'Accelerate', the Best Book About DevOps

I run a Meetup called Coffee. Quiet. Code. I want to encourage developers to work together in the same casual space without having to discuss anything or even interact much. It could be seen as a far less rigid version of Mycroft Holmes' "Dionysus Club", a way to enjoy shared community.

My inspiration came from people, often strangers, reading together in libraries, and also the sweet memories of my whole family reading their separate books they received as holiday gifts. In that spirit, here's a book recommendation.

Accelerate: The Science of Lean Software and DevOps is exactly what it claims to be: high-quality, peer-reviewed research answering the practical question "What's really working in software development across all industries and team sizes?" There are plenty of opinions out there on DevOps. This book deals in facts.

I wrote an extensive book summary for a client, which you'll find on my web site.

'Accelerate' Book Notes And Quotes | Software Meadows

   Older