Michael Herold | Articles

Extending ActiveRecord with custom types

2021-03-22T09:00:00-05:00

In the last article, we saw how to use the ActiveRecord Attributes API to make Ruby on Rails models more communicative. The two explicit benefits of doing so from that article were a more discoverable data model and a uniform way of defining different types of attributes. The article also hinted at using richer data types to model attributes. That is the topic of this article.

The Attributes API is extensible and allows you to define clean interfaces for using Ruby objects as attribute types. Instead of relying on primitives, you can create your domain logic as Ruby objects that ActiveRecord then coerces form data into. This is a powerful technique that allows you to limit the Primitive Obsession code smell in your projects.

This article details how to create your own ActiveRecord type classes. Once we have a type class, we’ll register it to easily use it within a model, though we’ll also cover how to use it without registering it. Then, we’ll go over a concrete example. Lastly, we’ll talk through some rules of thumb for creating custom type classes.

Type classes

To create a type class, you only need to define a plain old Ruby class that meets a four method interface:

#cast(value) performs a type cast on input, like when using the setter methods from accepting a form submission.
#serialize(value) casts the value from the Ruby type to the type needed by the database.
#deserialize(value) casts the value from the database into the Ruby type.
#type returns a Symbol corresponding to the Symbol used to register it with the type registry.

In some cases, you can use private helper methods to run similar transformations for these interface methods. For example, if you end up serializing the value into a String-like field, your #cast and #deserialize methods might be able to share code.

If you need to be able to configure your type, you will also want to define an #initialize method to set the instance variables you will need as configuration.

While you can use any class that conforms to this interface, sometimes it makes sense to inherit from a built-in base class from ActiveRecord.

Built-in base classes

ActiveRecord comes with base classes that can help you create type classes for certain patterns. For example, if you have an immutable value type, inheriting from ActiveRecord::Type::Value might make it easier to implement your value type.

Likewise, when you want to add special behavior on top of an existing one, it can help to look at the built-ins. ActiveRecord types are a superset of ActiveModel types. This means that you will need to look at both the ActiveRecord::Type and ActiveModel::Type modules to get the list of built-in types.

Registering a type class

Once you have created your type, the easiest way to make it usable from your model is to register it in the ActiveRecord::Type registry. Doing so looks like this:

ActiveRecord::Type.register :my_identifier, MyClass

With the above registration, you can use your new type class with the following:

class MyModel < ApplicationRecord
  attribute :my_attribute, :my_identifier
end

Configurable type declarations

For configurable types, you can pass arguments on the attribute declaration to forward them to the type class instance. For example, given a class called MyClass with an initializer like #initialize(limit:, precision:):

class MyModel < ApplicationRecord
  attribute(
    :my_attribute,
    :my_identifier,
    limit: 2,
    precision: 5
  )
end

Positional arguments work as well, like with a normal method call.

Using a type class without registering it

While registering the type makes it easier to use, you don’t have to register it to use it. To use an unregistered type, pass an instance of the class to the attribute class method:

class MyModel < ApplicationRecord
  attribute :field, MyClass.new
end

For configurable classes, pass the arguments to the constructor:

class MyModel < ApplicationRecord
  attribute :field, MyClass.new(limit: 2, precision: 5)
end

Now that you know how to create, register, and use a class, let’s look at an example.

Example: GlobalID

GlobalID is a default Rails gem that enables you to store a reference to an object using a URI specific to your application. In cases where you have multiple Rails applications — or Rails applications that talk to non-Rails applications — they can be handy for loading information from other services. The behavior of a GlobalID is what allows ActiveJob to transparently load records from the database.

Sometimes it makes sense to store a reference to an object in the database as a GlobalID. You could store the GlobalID as a simple string, but that makes for a clumsy way to look up the reference. Instead, let’s create an ActiveRecord type that accepts a string value but transforms it into a GlobalID object.

module ActiveRecord
  module Type
    class GlobalID < String
      def serialize(value)
        value.to_s.presence
      end

      def type
        :global_id
      end

      private

      def cast_value(value)
        value = value.to_gid if value.respond_to?(:to_gid)
        value = super(value)
        ::GlobalID.parse(value)
      end
    end

    register :global_id, GlobalID
  end
end

On line 3, you’ll see that the type inherits from the String type. This makes it so you can ignore a lot of the details of the other methods since this is a specialization of a String column in your database.

On lines 4-6, you’ll see the definition of the #serialize method. At this point, we know that the value we’re dealing with is a nilable GlobalID. For this method on this specific type, we need to ensure that we transform the value into a nilable String. Converting the GlobalID to a String happens with #to_s. However, this means it will place an empty string when the value is nil. To prevent that, #presence outputs the value when it’s present and otherwise outputs nil.

On lines 10-14 is a specialization of ActiveRecord::Type::Value’s #cast_value helper. It takes a value that can be an instance of an ActiveRecord model, a String, or a GlobalID. For an ActiveRecord model, it converts the model to a GlobalID. Then, it delegates to ActiveRecord::Type::String to cast the GlobalID or String to a frozen string. And lastly, it reconverts the String into a GlobalID. This is an inefficient, but effective, workflow that we can later optimize, if necessary.

Lastly, on line 17, you’ll see the registration of the type as :global_id to make it easily usable from your models.

By leaning on the built-in class, this is all the change we need to have a fully functional GlobalID type! To use it, you can do the following:

class MyModel < ApplicationRecord
  attribute :related_to, :global_id
end

MyModel.new(related_to: User.first).related_to
#=> 'gid://myapp/User/1'

MyModel
  .new(related_to: 'gid://otherapp/Foo/123')
  .related_to
  .locate
#=> loads the record from otherapp using GlobalID

MyModel.where(related_to: 'gid://otherapp/Foo/123')
MyModel.where(related_to: GlobalID.parse('gid://otherapp/Foo/123'))

The database stores the GlobalID value object as a simple string, but when ActiveRecord loads the record the string becomes a GlobalID with all of its rich behavior. An, instead of being a one-off for this class, you can now reuse this behavior in other models as well!

After seeing an example, hopefully your creativity is flowing, so let’s talk about some rules of thumb for making custom type classes.

Rules of thumb

I have two main rules of thumb for creating ActiveRecord type classes. First, make your types robust, not strict. Second, try to use value objects for your types as much as possible.

Robustness

When creating types in other cases, generally you want to be strict so that you don’t end up introducing invalid data into your system. However, with the Attributes API, it’s best to follow the robustness principle:

Be conservative in what you do, be liberal in what you accept.

It’s tempting to have your type classes raise an error whenever the value that you receive is invalid for the attribute. However, in cases where you are either adding typing to an existing application or where multiple writers are writing to the database from which you read, it’s better to take a gentler approach. Consider converting the value to nil and then adding a validation for the attribute. Or even better, encode the invalid data as an exceptional value with a validation.

By taking the gentler route, you prevent the case where bad data in the database makes it so you can’t even inspect the object without resorting to ActiveRecord::Connection#execute or a direct database connection. Since the casting of the value raises an error, any time bad record with an invalid datum loads it will raise the exception. This is indiscriminate and can happen in a customer-facing controller, an API endpoint, or a diagnostic terminal.

Value objects

The best type classes use value objects for their Ruby values. Value objects are immutable and have identity based on the values that they wrap, rather than identity of their own. Most of the prior art that you can reference uses value objects, so it’s easier to reference examples for them. Mutability also adds an additional set of methods that you have to either include or define.

When you are unable to use a value object, consider inheriting from the JSON (or, if you’re using PostgreSQL, JSONB type). It is already a mutable type class and will handle the changes you need for mutability out of the box for you.

Conclusion

In this article, we covered creating your own custom type classes for ActiveRecord. We then found how to register them to make them easier to use, but also talked about using them without registering them. Then we went through an example with a type that models GlobalIDs and discussed two rules of thumb for creating your own type classes.

In the next article, we’ll see how to use type modifiers to make your type classes even more extensible. That article will show two examples of custom type classes, so if you’re interested in learning more about this technique, make sure check it out next week!

Have you created your own custom ActiveRecord types? What types did you model? Did you run into any issues?

Use the Attributes API to make your Rails models more communicative

2021-03-15T09:00:00-05:00

ActiveRecord is the powerful object-relational mapper at the heart of Ruby on Rails. By default, it gives you tools to quickly and easily create new database tables and map them to domain models. It follows the ethos of “convention over configuration” that David Heinemeier Hansson coined with the release of Rails. As such, with little application code, you get a powerful, database-backed model that Just Works™.

However, few lines of code do not, necessarily, mean that the lines that are there are easy to understand. Does this model have a name or a description? How is that expiry field encoded, as a Date or a Time? What fields exist on this table, again? These are examples of common questions when working in a Ruby on Rails application.

To alleviate some of these issues, you can use the Attributes API when defining your models to make them more explicit. You can also use richer data types for your fields to ease working with subsets of your models. This article will share the history of the Attributes API, then show some concrete examples of using it.

What is the Attributes API?

Starting in Rails 4.2, then-maintainer of ActiveRecord, Sage Griffin, added an API to separate the concerns of the columns of the underlying database from the attributes that ActiveRecord sets on its record models. This helped to make the internals of ActiveRecord cleaner and easier to modify. As with many decoupling scenarios, this had the added benefit of allowing independent changes to the two systems and, thus, opening a public API for use by Rails application developers.

Before this change, the ActiveRecord directly set and modified the attributes of its models based on columns mapped from the database. The original design coupled these two things in the interest of making it easy to work with Rails; it was a good solution for the time and served its purpose well. However, it was also a limiting design in that, if you needed to color outside the lines of the framework, it was difficult to do.

As an illustration, consider this problem. You have a ticket model that contains a state machine controlling the transitions between the reserved, issued, redeemed, and voided states. You reserve a ticket while the customer purchases it. After the purchase, you issue the ticket. Once the customer arrives at the event, you redeem it. And if for any reason, you cancel the ticket, you consider it voided.

Figure 1: The state machine for the ticket example.

The important parts of this example are the following:

There are event names for transitioning between different states (e.g. purchase).
Some states cannot transition to other states (e.g. reserved cannot transition to redeemed).

This state machine is an example of logic that you should encapsulate in a “plain old Ruby object” and use as a collaborator in your model. Without the Attributes API, you might do the following:

class Ticket < ApplicationRecord
  # field: state
  def state
    TicketState.new(super)
  end
end

This isn’t harmful in itself, but it’s a symptom of missing functionality in the framework. A trickier bit to work around, and the source of why API exists, involves domain objects and SQL queries. For example:

state = TicketState.new("issued")
Ticket.where(state: state)

This SQL query requires you to handle the serialization of the TicketState class to the database. Your TicketState class shouldn’t know anything about databases; it’s only a state machine! Alternatively, you might attempt to always remember to write this query as Ticket.where(state: state.state). If you’re anything like me, you will forget this at the worst possible time: in the middle of an emergency.

The Attributes API gives you a place to encode all of this information and make it easy to use rich domain models as fields in your ActiveRecord models.

Using the Attributes API

There are two pieces to the Attributes API that you need to know about to use it:

ActiveRecord::Type classes for defining the types of data in your models.
attribute declarations in your model files.

There’s a lot of functionality within those two concepts, so let’s dig in.

ActiveRecord types

Within the Rails code base, type classes live in three different modules. ActiveRecord::Type classes inherit from ActiveModel::Type that implement similar behavior for use in ActiveModel and, sometimes, are only aliases to constant in ActiveModel. PostgreSQL-specific type classes exist in the ActiveRecord::ConnectionAdapters::PostgreSQL::OID module.

The contract for these classes consists of a slim, four methods¹:

#cast(value) performs a type cast on input, like when using the setter methods from accepting a form submission.
#serialize(value) casts the value from the Ruby type to the type needed by the database.
#deserialize(value) casts the value from the database into the Ruby type.
#type is a Symbol corresponding to the Symbol used to register it with the type registry.

Unless you’re defining your own types — and if you are, stay tuned for my next article — you won’t ever use these methods directly. However, it’s helpful to know the interface in case you see some unexpected behavior and need to diagnose the issue.

Attribute declarations

Within your models, you can use the attribute class method to declare an attribute with a type and any additional information about it. For example, these are all valid examples:

class MyModel < ApplicationRecord
  # A simple, documenting example
  attribute :hosted_on, :date

  # Have a default? Set it here! (Overrides database defaults.)
  attribute :likes, default: 0

  # Is your default something that needs recalculated?
  attribute :expiry, default: -> { 10.minutes.from_now }

  # For the JSONB type in PostgreSQL, like a document model
  attribute :metadata, :jsonb

  # Have a PostgreSQL array field? Mark it like so:
  attribute :tags, :string, array: true

  # How about a PostgreSQL date range?
  attribute :valid_in, :date, range: true
end

Attributes don’t need a database column associated with them. If you have some datum that you need to accept but not persist, you can create an attribute for it, set validations for it, and even use dirty tracking if you want!

Sadly, there is no API for listing the registered types. In Rails 6.1, you can run the following, but it’s not guaranteed to work in the future:

ActiveRecord::Type
  .registry
  .instance_variable_get(:@registrations)
  .map { |r| r.__send__(:name) }

The built-in types for all databases mirror the types you can use in your migrations, though with some different names (e.g. bigint in migrations vs. big_integer in attributes). Gems and database adapters may also add extra options so if you’re doing something exotic, it’s worth a look under the hood to see what’s available.

Benefits of declaring your attributes

There are three benefits to declaring your attributes in your models. First, you make the data model more discoverable when you onboard a new person into your project. Second, you use the uniform access principle to define different types of attributes for your model. And lastly, you enable richer behavior for your fields when it’s appropriate.

Convention over configuration means there’s little ceremony to working in Rails, which allows you to move quickly. But it comes at the cost of discoverability for future you and your teammates. When you end up with a large model, it’s easy to forget the name of a value or what type it is. Adding an attribute declaration, particularly when you have a standard ordering to the paragraphs of code in your models, makes it easy to reference in the file when you’re working with it.

Have you ever had to define a virtual attribute on a model? For example, you might have a confirmation system where you have your customers check a box, but don’t persist that to the database. If so, you probably defined that attribute using standard methods. That’s reasonable, but the Attributes API allows you to define that attribute just like a database-backed one so that all of the model’s attributes exist in the same paragraph of code. That makes it easier to see the shape of the class and what it does. This is an application of the uniform access principle, in a way, because it groups like things under the same interface.

Like we saw with the ticket example above, we sometimes need to have complex collaborators as part of our models. In the ticket’s case, we wanted a state machine to manage the state of the class but had to resort to using a wrapping method. Instead, what if we could define our own type to handle that behavior:

class Ticket < ApplicationRecord
  attribute :state, :ticket_state
end

This is a teaser of my next article, so check in next week to see how to do this!

Everything isn’t sunshine and rainbows though; there are detriments to using the Attributes API as well.

Detriments of the API

Like with anything in life, there are trade-offs to using the Attributes API. The two big ones are the introduction of a new tool and boilerplate.

There’s a common saying about projects where you only have so much “innovation budget” that you can spend on a project and hope that it succeeds. This guiding principle helps you to control the impulse to always use the hot, new technology or process on every project. Making decisions based on blog posts is one of the things you have to consider in this budget! There is overhead to using this technique. It’s relatively new to Rails, so you will likely have to teach it to your colleagues. Because it’s new, there will be friction for people using it for the first time. And because it’s something that you had to teach, you should also document the decision as well so that future you and your colleagues know why you made that decision.

The second detriment comes down to requiring boilerplate, which is not the Rails Way. You already have the information in the schema, so why duplicate it in the model when Rails can discover it for you? You also can make standard methods for the class, so why do it this way? Do you always use the Attributes API or only sometimes? The decision to use something that can seem like boilerplate is fraught. I get that. But I believe that the trade-off is worth it.

Conclusion

The Attributes API allows you to customize the way that Rails casts data into fields, both database-backed and virtual. The API is a public part of Rails so has certain compatibility guarantees around it. It’s powerful and extensible. And it solves many problems using a small amount of code, both at the application level and the framework level.

There are trade-offs to using the Attributes API. I believe the costs are worth it, particularly for complex problem sets. But if you have a limited amount of teaching ability in your team, it might be something to pause and think about before you adopt it.

This article was just a primer on how to use it. Make sure to read my next article, where I will show you how to customize it!

Have you used the Attributes API in a project? What was your experience?

I think the #type method is necessary. It’s marked with a “no doc” comment so that indicates that it’s not, but I have never seen a type without it. If I’m wrong, let me know! ↩

Solving the right problem

2021-03-08T09:00:00-06:00

Recently, I received a bug report on one of my open source projects. The bug report came from my co-maintainer on Hashie, dB, who I hold as an exemplary asker of questions. I quickly read through the issue and the background information on his issue and quickly sent him a reply that his suggestion sounded reasonable and that I would accept a contribution adding that behavior. I was attempting to be more like him and act as a facilitator, which is an attribute that I admire in him. However, I did a poor job of this task.

Notice what I said earlier: “I quickly read through the question and background information.” Any time you find yourself quickly doing something a non-rote task, alarm bells should go off in your head. When it comes to complex tasks or understanding, quick is the enemy of right.

Quick begets shortcuts. Shortcuts in understanding. Shortcuts in listening. Shortcuts in thinking.

Quick begets guesses. Guesses about meaning. Guesses about intent.

And quick begets mistakes.

This is a story about my mistake.

The report

The original report shares a confusing situation from the Grape framework. To debug a memory leak, a Grape programmer wrote a benchmark using my gem. The benchmark is not easy to understand. It tests switching from an Array to a Set as the backend to store some inheritable properties for an API endpoint.

Here is the benchmark they wrote, along with the output from running it, as written by James Lamont:

$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
require 'grape'
require 'benchmark/memory'

class VotingApi < Grape::API
  logger Logger.new(STDOUT)

  helpers do
    def logger
      VotingApi.logger
    end
  end

  namespace 'votes' do
    get do
      logger
    end
  end
end

class PostApi < Grape::API
  mount VotingApi
end

class CommentAPI < Grape::API
  mount VotingApi
end

env = Rack::MockRequest.env_for('/votes', method: 'GET')

PostApi.call(env)
CommentAPI.call(env)

setup_ar = VotingApi.instance_variable_get(:@setup).to_a
setup_set = VotingApi.instance_variable_get(:@setup).to_set

Benchmark.memory do |api|
  calls = 1000

  api.report('using Array') do
    VotingApi.instance_variable_set(:@setup, setup_ar)
    calls.times { PostApi.call(env) }
  end

  api.report('using Set') do
    VotingApi.instance_variable_set(:@setup, setup_set)
    calls.times { PostApi.call(env) }
  end

  api.compare!
end

Calculating -------------------------------------
         using Array    21.261M memsize (   274.472k retained)
                       178.001k objects (     2.012k retained)
                        18.000  strings (     0.000  retained)
           using Set    21.261M memsize (     2.472k retained)
                       178.001k objects (    12.000  retained)
                        18.000  strings (     0.000  retained)

Comparison:
         using Array:   21261040 allocated
           using Set:   21261040 allocated - same

How I heard it and my mistake

When quickly reading through this bug report and reading the benchmark, this certainly sounds like a bug.

The output here shows that both approaches use the same ~21MiB of memory. This intuitively makes no sense since the set should prevent duplicates.

The question that came from the bug report was, in essence, “should we run GC.start after the task runs?” After a quick read, of course, we should run the garbage collector! Otherwise, how would we know what items survive through a collection?

And there — right there in that last sentence — is my mistake. I answered with a quick, “Yeah, that makes sense. Go ahead and send a change!”

Surprise! My quick response was wrong; it doesn’t make sense. The memory profiler gem already does that. In fact, it runs multiple collections to encourage the garbage collector to fully clean unlinked memory. Without doing so, it wouldn’t be able to report the statistics that it does.

So we now know my mistake, but what was the correct thing to do?

Reading it again

To find the correct path, let’s summarize the situation.

Grape stores some configuration that appears to cause a memory leak.
James wrote a benchmark that showed the memory leak using my gem.
The output was confusing so dB filed an issue with my gem.
I read the benchmark and the explanation for the confusion.
The confusion centered around someone using the benchmark with the comparison to show the presence of a memory leak.
Memory leaks have to do with retaining objects, not necessarily with allocating them, and the comparison is about allocations.

After rereading and summarizing, I saw my mistake and decided upon two directions to approach the problem.

A better solution?

While there isn’t a bug in the behavior of the gem, there is a bug in the experience of using the gem. If I created the gem with the intent of using it to diagnose memory leaks, perhaps I would have nailed the experience of doing that with its DSL. However, I designed the gem to measure the space complexity of algorithms. That is a similar, yet distinct, job. Thus, I didn’t design the interface for that purpose and, when used for it, the gem confused the benchmarker.

Making a quick decision here, I could take the path of redesigning the DSL to make it so you can easily do the job of tracking memory leaks. This needn’t be a complicated change. And it can be backward-compatible as well. This is a task that I know how to do and I’m confident in doing it.

But is it what I should do first?

The right choice

Even the summarized version of events is complex. There is more than one conversation that I was not a part of. There are multiple stakeholders in the problem. And it’s unclear if I missed or misunderstood anything in any of the conversations.

Thus, the right choice here is to ask whether I have it correct. The only way to know whether I understand the problem correctly is to state how I understand it and check that for correctness. Only once I have done that and know that I understand the job to be done can I be sure that I am solving the right problem.

So, I did that and am currently waiting on an answer. The great thing now is that, even if my understanding isn’t complete, I now know that people use my gem for a purpose other than the one I designed it for: proving memory leaks. Since I know that is something that people need, I can spend some time thinking about how to make that process better. I can even take my time instead of being quick about it.

What to do next time

Maintaining my open source libraries is something that I do in my free time. I do it for enjoyment and to improve my skills. Free time, however, is a precious commodity at this point in my life. This makes me want to handle things quickly when I’m going through the issues on my repositories.

Next time, I will think about this experience and the mistake that I made. Most bug reports are technical and not easy to quickly understand. I need a process that allows me to feel like I am quickly handling issues even when I’m taking my time. These articles are one way of doing that. Another would be to write a development diary about the task so that I can reference it later.

I think the process that I outlined in this article is a reasonable approach for next time. When I feel myself doing something quickly on an open source project, I should write down my understanding of the problem and then check it with the person who filed the issue. Only once I have done that should I move forward with a change.

Save time by spending it wisely. It’s a finite resource, after all.

What kinds of mistakes have you made because you were acting quickly? Have you created a process to catch yourself when you’re doing this?

Using ox-hugo without duplicating content in the repository

2021-03-01T09:00:00-06:00

My new blogging setup mashes up Org mode with Hugo using the ox-hugo library. This is part of my attempt to bring more of my work product into the single format of Org mode. Hugo has rudimentary support for Org mode through go-org, but it admittedly only tries to cover the 80/20 use case of Org mode. Since I intend to use Org mode for everything, the chances are high that I will end up with something incompatible, thus requiring me to work around the problem. I didn’t want to do that, so I stuck with the Hugo export library.

Every example that I saw of a site that uses ox-hugo ends up duplicating the Org mode documents are Markdown in the repository. This felt noisy and redundant to me so I wanted to find a way to have Netlify render the Org document to its constituent Hugo pages as part of the deploy process. This article documents the process that I now use and shares the rendering script that I wrote to process the file as part of a deployment.

Project

The layout of my project looks like the following:

michaeljherold.com/
├── blog.org
├── config.toml
├── Makefile
├── netlify.toml
├── script/
│   └── publish
├── static/
└── themes/
    └── v1/

The blog.org file holds my Org mode blog pages in “one post per Org subtree” mode. This allows me to have the entire site live in a single file, which I think is pretty cool. As time goes on and I write more, perhaps I will choose to split it into multiple files but for now, I like how this works.

For Hugo, we have config.toml, static/ and themes/. Nothing is exciting in any of those files since I render the site with Hugo-flavored Markdown.

The netlify.toml file contains my Netlify-specific configuration. This includes some redirects and commands for making production deployments, deploy previews, and branch-based deployments. Again, there’s not a lot that is interesting here.

The Makefile is a simple affair that we will discuss in two sections.

The meat of this article lives in the script/publish file, which we will talk about next.

Hopefully, other than the complete lack of a content/ folder, none of this is surprising for you if you’re familiar with Hugo. I’m striving to have as little magic as possible as part of my publication routine so that there are fewer things for me to tweak instead of sitting down and writing.

Publish script

The solution to the problem of duplicating content in your Git repository lives in the script/publish file. This is an Emacs Lisp-in-shell script that I run during my Netlify deployment to generate all of the Markdown files for the pages on the site.

I will break the script down section-by-section here and explain what I’m doing in each section.

Run me a shell script, please

#!/usr/bin/env sh
:; set -e # -*- mode: emacs-lisp; lexical-binding: t -*-
:; emacs --no-site-file --script "$0" -- "$@" || __EXITCODE=$?
:; exit ${__EXITCODE:-0}

I borrowed the setup for this section from Henrik Lissner’s fantastic Doom Emacs scripts because I thought it was so ingenious. First, we set the shebang that makes the script run via a shell command. Then comes the “wow!” moment. To explain, first a little background.

Historically, POSIX shells didn’t have the concepts of true and false¹. They used : to mean something equivalent to true. So we could rewrite the :; on each of these lines to true;, which is a no-op. You might be asking, “why would you do that?” The reason requires a little bit of Emacs Lisp knowledge: the ; character denotes a comment and the : denotes a keyword. In this case, it’s the keyword :, which in Emacs is effectively a no-op since it’s only a reference to the keyword.

Combine all of this together and you get the following effect: each of these lines runs in a POSIX shell like Bash but does not run anything in Emacs. How cool is that?

So what do these lines do?

The first line sets the POSIX “consequence of shell errors” to be that any command that fails in the script exits the script with the same exit code as the failing command. It also uses Emacs file variables to tell Emacs to read the file as if it were an Emacs Lisp file and use lexical bindings for all its variables.

The second line runs this file (via --script "$0") in Emacs without using a blank profile (via --no-site-file), passing along any arguments you give to the script into Emacs (via -- "$@"). If Emacs exits with a non-zero exit code, it saves that exit code in the __EXITCODE variable².

The third line then exits the script with the exit code from the last line or, if we didn’t set it, a default of zero, or success. That means the shell will never execute the rest of the script, so we’re free to do whatever we want there! Again, Henrik’s pattern here is impressive.

Okay, with the entirety of the shell script covered, let’s move on to some Emacs Lisp!

Just kidding, actually run some Emacs code

The first section of Emacs Lisp is boring because it just sets up some boilerplate. But it’s necessary, so here we go:

(defvar bootstrap-version)
(defvar straight-base-dir)
(defvar straight-fix-org)
(defvar straight-vc-git-default-clone-depth 1)
(defvar publish--straight-repos-dir)

(setq gc-cons-threshold 83886080 ; 80MiB
      straight-base-dir (expand-file-name "../.." (or load-file-name buffer-file-name))
      straight-fix-org t
      straight-vc-git-default-clone-depth 1
      publish--straight-repos-dir (expand-file-name "straight/repos/" straight-base-dir))

The first paragraph defines some variables that we later use to hush some warnings in the byte compiler. The first four are for setting up the Straight package manager, which I’ll talk about shortly, and the last one is a variable for this script.

The second paragraph sets values for several variables, as so:

gc-cons-threshold controls the threshold that Emacs must hit before it allows the garbage collector to run. By default, Emacs uses a very low threshold of 800KB. This causes many garbage collection cycles, which slows down the process of running the script. You can see in the comment that I set it to 80MiB, which is a good deal larger than the default but still conservative enough that I don’t worry that Netlify will run out of memory in the build container.
straight-base-dir sets the location to which Straight will install itself, relative to the current file. This is the project’s root directory, confusingly. Straight then adds a straight/ folder to that directory for its files.
straight-fix-org sets a Boolean variable for Straight to run a “fix” on the Org mode package during download. Org mode doesn’t store its org-version variable in the source repository and instead waits till they release a package to set it. Since Straight pulls from a Git repository, we must set this variable or Org will not build correctly³.
straight-vc-git-default-clone-depth sets the depth to which Straight will check out the Git repository for each package when it downloads them. Since we’re only using this to build our Markdown files and not use it daily, we can safely limit this to a single commit to speed up the build process on Netlify.
publish--straight-repos-dir contains the directory in which we want to store the repositories for the packages we download. We set this to be straight/repos/ in the root of the project.

Now that we have all of that setup, we can run the commands to bootstrap and install Straight. I decided to go with Straight here because it’s declarative and flexible. Right now, I only need Org mode and ox-hugo, but using Straight allows me to quickly and easily add a new package or a monkey-patch to either of the packages that I currently use. It also uses a declarative syntax which helps to make sure we do the correct thing every time.

(let ((bootstrap-file (expand-file-name "straight/repos/straight.el/bootstrap.el" straight-base-dir))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

This bootstrap block comes almost verbatim from the Straight documentation. We install the bootstrap file in the Straight directory that we configured above and load it without displaying any messages. Please note that this, at the end of the day, downloads and executes a script from the Internet. Many people use Straight so I’m confident in my use case to do this, but consider it if you’re doing something where you are hosting sensitive data. I am not, so I am comfortable doing this.

Once we have downloaded and bootstrapped Straight, we can install the packages we need:

(straight-use-package
 '(org-mode :type git
            :host github
            :repo "emacs-straight/org-mode"
            :files ("*.el" "lisp/*.el" "contrib/lisp/*.el")))

(straight-use-package
 '(ox-hugo :type git
           :host github
           :repo "kaushalmodi/ox-hugo"
           :nonrecursive t))

As previously stated, we are only going to use two packages, Org mode and ox-hugo. The :files declaration on Org mode tells Straight to only create symbolic links for those particular files. The :nonrecursive declaration on ox-hugo tells Straight not to pull the submodules in the repository, which helps the download complete faster.

Now that we have our dependencies, we can run the actual script to publish the Org subtrees as Markdown files:

(with-current-buffer (find-file-noselect (expand-file-name "blog.org" "."))
  (org-next-visible-heading 1)

  (when (equal (org-entry-get (point) "CUSTOM_ID") "toc")
    (let ((inhibit-message t))
      (org-cut-subtree)))

  (message "Publishing...")
  (org-hugo-export-wim-to-md t))

This block opens the blog.org file at the root of the project. It removes my table of contents that I keep in the file because it prevents ox-hugo from rendering the file. To do this, it skips to the first heading in the file, checks to make sure it has the CUSTOM_ID property that I set on it called toc for “table of contents,” and, if so, cuts — or removes — the subtree from the file. Then, we echo “Publishing …” to standard out and use the org-hugo-export-wim-to-md function to export the whole file to its constituent Markdown files.

Et voilà! We now have all of our Markdown files in the content/ directoy, just as Hugo expects.

Makefile

To coordinate the use of our publish script, I find it useful to create Makefile so that we can write a simple command in our netlify.toml instead of a complex list of commands. Here is the content of the Makefile we need:

.PHONY: md production preview clean
md:
	./script/publish

production: md
	hugo

preview: md
	hugo -FD

clean:
	rm -rf ./content ./public ./resources

We have four phony targets that handle different pieces of the site. The md target runs our publish script to create our Markdown files. We list it as a dependency for the production and preview targets, which generate the published and published, future, and draft pages of our Markdown, respectively. And to help clean it all up, we have the clean target that removes everything we built.

With these targets, we can then fill out the relevant portions of our Netlify configuration.

`netlify.toml`

For this article, we only need a simple, three-section Netlify configuration. You can see it as follows:

[context.production]
command = "make production"

[context.deploy-preview]
command = "make preview"

[context.branch-deploy]
command = "make preview"

For production deployments, we only want to build published pages, so we use make production to generate the site. For previews, it’s handy to see future and draft pages as well, so we use make preview for those environments.

And that’s it! That’s all we need.

Conclusion

Because I am still a novice-to-journeyman Emacs Lisp programmer, it took me a while to figure out how to do the programmatic render of my Org mode file into Markdown. But, once I figured it out, the solution ended up being very simple and, I believe, future-proof.

This allows me to keep the size of my site repository down, which pays dividends going forward. I still have to worry about the size of assets in the repository, so that might be something I will look into in the future. But at least I don’t have two copies of the page contents that could only lead to confusion. Even better is the fact that it only uses software that is readily available in Netlify’s build images.

Did I make any mistakes? What do you think of Org mode? Have you ever written a website in it?

At least according to this StackOverflow answer. I haven’t been able to track down a primary source for this quirk. ↩
As I write this, I think I found an unnecessary feature here. Given that I just explained that set -e will cause the command to exit with an error code, I think this is not needed and that the next line can just be exit 0, but I will have to try it out to double-check. ↩
There’s nothing like writing an article for you to catch errors in your scripts. I looked this variable up and the Straight maintainer marked it as obsolete on 2020-10-23. I don’t think it’s necessary any more, but again, I have to double-check. ↩

Provisioning a Valheim server with Terraform

2021-02-22T09:00:00-06:00

My friends and I decided we were going to give Valheim, an early-access survival game, a try. Since many of us are parents now and we’re in a socially distancing world, colocating for an evening of fun was out of the question. I decided to use the event as an excuse to practice my DevOps skills.

I use Terraform at work for a variety of things, but I don’t write enough of it to be completely comfortable. As such, I decided to use Terraform to provision a dedicated server for us. This article documents how I went about it using DigitalOcean and Terraform.

Create an SSH key

When provisioning cloud servers, SSH is your best friend. When you need to perform any maintenance on the server, you will need SSH to connect to it. Since this is a single-purpose machine that we’ll be provisioning, let’s create a single-purpose SSH key to go with it.

ssh-keygen -t ed25519 -f ~/.ssh/id_valheim -P ""

This command generates an elliptic curve SSH key that we will use to connect to the server. It uses an empty passphrase since its only purpose is to connect to this throwaway server. We will use the DigitalOcean API to upload this key to them for use with our server.

Configure the Terraform provider

First, you will need to create an API token on DigitalOcean. You’ll want to keep this in a safe place because once you generate it you will not be able to see the token again. Once you have it, you’ll want to set it as an environment variable. I use direnv for this purpose and, as such, put the environment variable in a .envrc file in my project. That looks like the following:

export DIGITALOCEAN_TOKEN="mytokenthatigotfromdigitalocean"

Once you have that environment variable set, you can use the DigitalOcean Terraform provider. To configure Terraform to use that provider, you will need to specify it as a required provider¹ like so:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.5"
    }
  }
}

And then declare it as a provider:

provider "digitalocean" {
  # export DIGITALOCEAN_TOKEN
}

I like to add that comment in the provider block to state where its credentials come from; it’s not necessary. For those of you following at home, you can place all of the configuration in a single .tf file or design your own scheme for organizing your blocks.

To install the provider, initialize Terraform with:

terraform init

Set up the SSH key

The first resource we need to create is the SSH key. This is because DigitalOcean creates the server from an image that uses cloud-init to set default SSH keys for the root user. Without it, the other commands won’t work.

To create the SSH key resource, we specify it like so:

resource "digitalocean_ssh_key" "valheim" {
  name = "valheim"
  public_key = file("~/.ssh/id_valheim.pub")
}

This takes the file on our local machine located at ~/.ssh/id_valheim.pub and uploads it as an SSH key to DigitalOcean. If you chose to name your key something different in the previous step, use that filename here. The name shows up in DigitalOcean’s security settings to help you identify it later. Pick something memorable.

Create the server

Once we have the provider set up, we can write a recipe to provision the server. First, we have to set up the SSH key that we generated in the last step

For our purposes, a bit of anecdotal research showed that others successfully run a four-person server on a 2 CPU / 4 GiB server. This is a standard server size from DigitalOcean called s-2vcpu-4b. With that knowledge, we can write the following resource:

resource "digitalocean_droplet" "valheim" {
  image = "ubuntu-20-04-x64"
  name = "valheim"
  region = "nyc1"
  monitoring = true
  size = "s-2vcpu-4gb"
  tags = ["game"]
  ssh_keys = [digitalocean_ssh_key.valheim.fingerprint]
}

The image here is the latest Ubuntu LTS at the time of this writing. The name helps to identify your droplet in DigitalOcean’s cloud console. In my case, three of the four of us were playing from Ohio so I picked the New York City 1 data center (or region). Setting the monitoring flag to true enables DigitalOcean’s new agent-based resource monitoring that is more fine-grained than the default. I prefer it to their older metrics. The tags are a way to keep your droplets organized in the cloud console.

The most important field here is the ssh_keys field. This takes the SSH key that we uploaded in the previous resource and links it to the droplet so that we can log into the droplet via SSH. Without it, DigitalOcean will email you a root password, but that is incompatible with the rest of the Terraform configuration.

Configure the server

I very much label this section as caveat lector, reader beware. Because I am treating this server as a “create it and delete it later” server, I am using a Terraform provisioner to provision the box. As stated in the documentation, Hashicorp only intends you to use these provisioners as a last resort. If this were for a mission-critical system, I would probably use Packer to build an image and deploy it instead. But that’s a little too heavy for my current task, so here we are.

There are three steps that we will perform. First, we’ll create some necessary directories and attempt to temporarily disable the automated updates for Ubuntu Server. Next, we’ll upload some configuration files to the server. Lastly, we’ll install Docker and start the Valheim service.

Create necessary directories

Running simple scripts in the Terraform provisioner is simple. The remote-exec provisioner generates a script file and runs it with the shell. We’ll use this functionality for the first step:

resource "digitalocean_droplet" "valheim" {
  # ...
  provisioner "remote-exec" {
    inline = [
      "sed -i 's/1/0/g' /etc/apt/apt.conf.d/20auto-upgrades",
      "mkdir -p /opt/valheim",
      "mkdir -p /etc/sysconfig"
    ]

    connection {
      type = "ssh"
      user = "root"
      host = self.ipv4_address
      private_key = file("~/.ssh/id_valheim")
    }
  }
}

The first line is a hack that attempts to beat a race condition between the provisioning scripts and the Ubuntu automatic updates service. If the automatic update occurs just before any of our calls to Apt, we won’t be able to obtain the lock on the repositories and package caches and our scripts will fail. So we use sed to disable the automatic upgrades service, then create two directories.

The first directory, /opt/valheim, will contain our game data. The second contains an environment file for the Systemd unit we will use to manage the server process.

The connection block configures the SSH connection that we use to run the provisioner. This will be in each of the provisioner blocks that follow, as I don’t think there is a way to share connection information between provisioner blocks without something like Terragrunt. The block says “create an SSH connect using the root user at the IPv4 address that we just created and use the id_valheim key.”

Now that we have the basics, we can upload a Systemd service unit and its accompanying environment file. This takes two blocks, one for each file.

Upload Systemd unit

To upload the Systemd unit that manages the Valheim server, we can add this provisioner to the droplet resource:

resource "digitalocean_droplet" "valheim" {
  # ...
  provisioner "file" {
    source = "conf/valheim-server.service"
    destination = "/etc/systemd/system/valheim-server.service"

    connection {
      type = "ssh"
      user = "root"
      host = self.ipv4_address
      private_key = file("~/.ssh/id_valheim")
    }
  }
}

The file provisioner takes a local file (or content string) and uploads it as a file to the resource. In this case, it takes the Systemd unit below, which we store in conf/valheim-server.service at the root of our Terraform project.

[Unit]
Description=Valheim Server
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
EnvironmentFile=-/etc/sysconfig/valheim-server
ExecStartPre=-/usr/bin/docker stop %n
ExecStartPre=-/usr/bin/docker rm %n
ExecStartPre=/usr/bin/docker pull lloesche/valheim-server
ExecStart=/usr/bin/docker run \
          --name %n \
          --rm \
          -v /opt/valheim:/config:Z \
          -p 2456-2458:2456-2458/udp \
          -e SERVER_NAME \
          -e SERVER_PORT \
          -e WORLD_NAME \
          -e SERVER_PASS \
          -e SERVER_PUBLIC \
          --dns $DNS_1 \
          --dns $DNS_2 \
          lloesche/valheim-server
ExecStop=/usr/bin/docker stop %n
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

This is the default Systemd unit for the Docker container we use to run the server. It reads environment variables from an environment file that we’ll upload next and starts the Docker container with them. Running through line-by-line:

-v /opt/valheim:/config:Z mounts our game data directory as a volume as an unshared bind-mount within the container
-p 2456-2458:2456-2458/udp forwards the default game ports from the host to the container
-e SERVER_NAME will configure the name of the server within Valheim
-e SERVER_PORT sets the first port that Valheim listens on
-e WORLD_NAME sets the name of the world file for your identification
-e SERVER_PASS sets the password to enter the server, which must be 5+ characters according to the manual
-e SERVER_PUBLIC is a boolean flag for whether the server should show up in the public server list
The -e --dns lines set the DNS servers that the container should use
lloesche/valheim-server is the name of the container image to run

The rest of the unit file attempts to ensure the container is the only copy running and tells the unit to load the environment file that we upload next.

Upload environment file

The second file provisioner uploads the environment file that configures the Systemd unit:

resource "digitalocean_droplet" "valheim" {
  # ...
  provisioner "file" {
    content = templatefile("conf/valheim-server.conf.tpl", {
      server_name = var.server_name,
      server_pass = var.server_pass,
      world_name = var.world_name
    })
    destination = "/etc/sysconfig/valheim-server"

    connection {
      type = "ssh"
      user = "root"
      host = self.ipv4_address
      private_key = file("~/.ssh/id_valheim")
    }
  }
}

This reads a configuration file template (shown below) from the conf/valheim-server.conf.tpl file at the root of your Terraform project and inserts some Terraform variables. To enable the variables, we need three variable blocks in our Terraform file:

variable "server_name" {
  type = string
}

variable "server_pass" {
  type = string
}

variable "world_name" {
  type = string
}

You can set these as environment variables with the pattern TF_VAR_<variable>, which I recommend doing in your .envrc file if you use direnv.

The template itself looks like:

DNS_1=8.8.8.8
DNS_2=8.8.4.4
SERVER_NAME="${server_name}"
SERVER_PASS="${server_pass}"
SERVER_PORT=2456
SERVER_PUBLIC=1
WORLD_NAME="${world_name}"

These are the environment variables that you see referenced above in the System unit.

Now that we have all of that on the server, it’s time to install Docker and start the service!

Install Docker and start Valheim

This section is the longest part of provisioning the server because it does several things. First, Docker is not available from the default Ubuntu repositories so we have to add its repository and configuration, which takes a few commands. Then, we need to install Docker. Finally, we enable the Valheim service and re-enable the automatic upgrade system.

We do this with the following provisioner block:

resource "digitalocean_droplet" "valheim" {
  # ...
  provisioner "remote-exec" {
    inline = [
      "apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yq apt-transport-https ca-certificates curl gnupg-agent",
      "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -",
      "echo 'deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable' > /etc/apt/sources.list.d/docker.list",
      "apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yq docker-ce docker-ce-cli containerd.io",
      "systemctl daemon-reload",
      "systemctl enable --now valheim-server.service",
      "sed -i 's/0/1/g' /etc/apt/apt.conf.d/20auto-upgrades"
    ]

    connection {
      type = "ssh"
      user = "root"
      host = self.ipv4_address
      private_key = file("~/.ssh/id_valheim")
    }
  }
}

Dissecting this line-by-line, we see the following:

apt-get update [...] gnupg-agent installs the prerequisites for adding the Docker repository in a headless way so we aren’t prompted for input
curl [...] | apt-key add - adds the Docker GPG key to the Apt keychain
echo [...] > /etc/apt/sources.list.d/docker.list adds the Docker repository to Apt
apt-get update [...] containerd.io installs Docker, again in a headless way
systemctl daemon-reload ensures that Systemd has a reference to our new Docker service
systemctl enable --now valheim-server.service enables the Valheim server, which makes it start up upon reboot, and starts the server now
sed [...] re-enables Ubuntu’s automatic updates

Once this provisioner finishes, Terraform finalizes its actions. We’ll have a Valheim server running in approximately four minutes! To connect, put in the IP address of your droplet with the port :2457, confusingly.

Conclusion

One of the ways that I improved my skills as both a developer and systems operator was through running private services for my friends and me, like video game servers. This article shared a pared-down version of the DevOps setup I used for setting up a dedicated Valheim server. Using a single local tool, Terraform, we were able to go from nothing to a fully functioning server running on DigitalOcean.

Granted, this simple setup skips a larger step in provisioning and uses a hack to try to beat the race against Ubuntu’s auto-updates. Converting those steps to a repeatable process was beyond the scope of what I wanted to do, but there are many ways to do so with a variety of different tools. As an exercise, see what you can come up with!

If you’re interested in seeing my full setup using CloudFlare for DNS and DigitalOcean firewalls, you can find it on GitHub. Also, you can sign up for DigitalOcean and receive a $100 credit for 60 days so you can test this for free².

Do you ever use your leisure time for things like this? Have you ever used a cloud provider to host a game server?

As of this writing, I am using Terraform v0.14. It and its prior version, v0.13, require different provider setup than prior versions. To check which version you’re using, run terraform -version. ↩
Full disclosure: that link is a referral link. If you sign up and spend $25, I will receive a $25 credit. If you’d prefer not to participate in that referral, here is a link without referral. ↩

Duplicate cookies in Ruby on Rails

2021-02-15T09:00:00-06:00

Ruby on Rails has an easy-to-use cookie store for managing state between requests. It has affordances for storing clear-text values, tamper-proof signed values, and encrypted values. I previously showed how you can make good use of encrypted values for handling ActionCable authentication. However, there is a case where you might end up with duplicate cookies in Ruby on Rails applications.

This article discusses cookies, shared cookies, and how the use of the Ruby on Rails cookie store may lead to apparently duplicate cookies. From that, we’ll end with three rules of thumb for using cookies and avoiding the issue.

The Rails cookie store exposes a #cookies method that returns a Hash-like object on your controllers. For standard, visitor-editable cookies, the bracket-set operator is your friend:

cookies[:plain] = "value"

This results in a cookie that can be both read and edited on a client device:

A signed cookie works much the same from your application code. To create a signed cookie, send the #signed message with a bracket-set operator, like so:

cookies.signed[:signed] = "value"

This sets the value of the cookie to a signed blob of the form <payload>--<digest>. The payload is a base 64-encoded JSON document that has a base 64-encoded message containing the value set via your assignment. The digest is, as of Rails 6.1, an HMAC-SHA1 digest using your application’s secret_key_base as the key.

For the curious, the cookie value looks like this in the header:

eyJfcmFpbHMiOnsibWVzc2FnZSI6IkluWmhiSFZsSWc9PSIsImV4cCI6bnVsbCwicHVyIjoiY29va2llLnNpZ25lZCJ9fQ%3D%3D--20b6c117746e8b08ad4349ce4594e89bd47ef3c5

Lastly, an encrypted cookie follows the same pattern. It’s easy to access, only needing a change to #encrypted instead of #signed with the bracket-set operator:

cookies.signed[:encrypted] = "value"

Rails translates the cookie using a pattern similar to that of the signed cookie. It’s a blob of the form <payload>--<iv>--<auth_tag>. Breaking that down …

The payload is a base 64-encoded, encrypted form of the JSON document from the signed implementation.
The initialization vector (IV) the base 64-encoded value of the seed to the encryption algorithm, which we need to be able to later decrypt the value.
The authentication tag is a base 64-encoded value that you can use to verify both the encrypted and plain text values of the payload. It, along with its delimiting dashes, is only present if you are using an “authenticated encryption with associated data” (AEAD) algorithm, which Rails 6.1 does by default.

Whew, that’s a lot to take in! For the (morbidly?) curious, here’s the cookie in header form:

x8s62U2CFfX0TgjuAcyX2eLi4PTLzSdWkjvHeYJ5zh8QO22fFj%2F%2BvM%2Fsk%2FV1u1lTKowWKrEe3YpQcYJ2yqRGUiiNMMWUqT9qcA%3D%3D--kwVMgQ4wOkYWXdQS--%2FcJ8VB6zVAsJ5aShfNGHaA%3D%3D

Now that we’ve seen the three different strategies, we can talk about trade-offs.

Trade-offs between methods

To talk about trade-offs, it will be handy to see each of the values side-by-side:

value

eyJfcmFpbHMiOnsibWVzc2FnZSI6IkluWmhiSFZsSWc9PSIsImV4cCI6bnVsbCwicHVyIjoiY29va2llLnNpZ25lZCJ9fQ%3D%3D--20b6c117746e8b08ad4349ce4594e89bd47ef3c5

x8s62U2CFfX0TgjuAcyX2eLi4PTLzSdWkjvHeYJ5zh8QO22fFj%2F%2BvM%2Fsk%2FV1u1lTKowWKrEe3YpQcYJ2yqRGUiiNMMWUqT9qcA%3D%3D--kwVMgQ4wOkYWXdQS--%2FcJ8VB6zVAsJ5aShfNGHaA%3D%3D

All three of these cookies contain the value "value". The plain cookie is of size 10, the signed cookie is size 148, and the encrypted one is size 171¹. That shows that there is, at a minimum, a space trade-off when using each of these strategies. Since cookies cannot be larger than ~4,096 bytes² ^,³, the trade-off for signing and encrypting can add enough overhead to make problems for you.

However, you would not want to put your authentication cookie in plaintext for your visitors to monkey with! Imagine the trouble you could get in if they were able to masquerade as your account by changing a cookie value. Signed cookies prevent that from being an issue, but can still leak internal information.

For extremely important data, like said authentication cookies, use encrypted values and keep the value small.

Sometimes, you need to be able to access a cookie on more than one domain. Perhaps you want to be able to show your customer’s avatar on your marketing page, but it runs on a WordPress website instead of your Rails application. Or maybe you run a web application firewall in front of your application and it doesn’t support the WebSockets that you need for a feature (I’m looking at you Sucuri Firewall.)

Regardless of your use case, when you need to share a cookie between domains, you must change the syntax you use for setting cookies. It’s not a large change, but there is an extra level of memory needed for using this method.

First, a look at what the new form looks like:

# old
cookies[:key] = "value"

# new
cookies[:key] = { value: "value", domain: :all }

That :all is a wildcard specifier meaning *.<my_domain>, or any subdomain of the primary top-level domain. So if you’re running an app at app.cool.io and your marketing site runs on www.cool.io, the :all option will work for you.

Otherwise, you can specify the domain(s) to which the cookie will apply.

Duplicate cookies?

Consider the following scenario. In one place of your application, you set a shared cookie as follows:

cookies[:key] = { value: "value", domain: :all }

In another spot, you nil out the value to remove it from the cookie jar, but you do it like so:

cookies[:key] = nil

When you make several subsequent requests to the application, calling these lines in any number of orders and counts, what would you expect the behavior to be?

In some cases⁴, you might end up with a cookie set for both mysite.com and .mysite.com! That can lead to some unexpected behavior if you’re not aware it’s a possibility because the cookie that “wins” the fight is the last one to appear in your request headers.

When this cookie relates to authentication, like handling ActionCable authentication, it can lead to exasperating issues where every other request might come through as unauthenticated.

Rules of thumb

With all of these learnings, we can describe three rules of thumb for using cookies in your Ruby on Rails application:

Use the level of cookie that makes sense for the domain. When you handle authentication and authorization, prefer at a minimum, signed cookies. When it doesn’t truly matter if someone changes the value, prefer plain cookies.
Store smaller, ideally primitive, values in a cookie to keep the size down. If you need a rich structure, consider storing an identifier to a record in your database.
When you need to share a cookie between domains, ensure that you always use the same domain: parameter when setting the cookie. In fact, there’s a line in the ActionDispatch::Cookies documentation about that, but it’s easy to miss.

Applying the third rule can take many forms, but once you do, you will see the problem of duplicate cookies disappear.

Conclusion

The cookie store in Ruby on Rails is very easy to work with but has a few sharp edges. Because it’s easy to use, it’s easy to make mistakes. Knowing when and how to use the different types of cookies will help you avoid one class of error.

But even knowing those basics still leaves a spot for error. When you need to share cookies between domains, there’s a possibility that you end up with duplicate cookies in your app.

Apply your discipline to the use of cookies and you’ll see your errors vanish before you.

Have you ever experienced a duplicate cookie in your Ruby on Rails application? How about those tricky authentication cookies - have you ever made a mistake there or discovered one in another app?

It’s slightly important to note here that the values for the signed and encrypted cookies are inflated by 1 and 4, respectively. Cookie sizes are the size of the cookie name plus the size of the value. Since I used the cookie names “plain,” “signed,” and “encrypted,” the names are of varying size. Keep that in mind when you’re picking names for your cookies! ↩
Interestingly, the spec, also known as RFC6265, does not specify a maximum size. It’s entirely up to browser vendors to implement, though they all agree that 4KiB is the maximum length they’ll support. Some browsers have overhead though, so it’s wise not to approach this value. ↩
Please, consider the effect of large cookies. For every request and every response, the browser transmits the cookie. As such, if you have a 4,000-byte cookie you’re tacking on an extra 8KiB of data per page for someone using a mobile device. ↩
I haven’t been able to put my finger on when this happens, but it feels like either a race condition or an order-dependent condition. From anecdotal reloading of a page that non-deterministically sets the value of a cookie to the current domain or to :all, I end up seeing both cookies set after a few reloads. If I track down the exact behavior, I’ll update this post. Do you know the cause? If so, please let me know! ↩

3 Ways to Make Arel a Joy

2021-02-08T09:00:00-06:00

Arel, the SQL syntax tree constructor library backing ActiveRecord, allows you to express your SQL queries without the use of fragments. ActiveRecord uses it internally to represent a variety of queries. It does this transparently using the traditional where(key: value) syntax that we’ve seen since the early days of Rails.

In cases where you only need to write simple query predicates, the traditional syntax is what you should reach for, full stop. However, when you need to write a more complex query like grouped pagination or null-based ordering, I recommend reaching for Arel.

However, because Arel considers itself a “framework framework,” it’s not a truly enjoyable to use out-of-the-box. This article shares three ways to make Arel a joy to use.

Table accessor

Arel’s status as a private API makes it so there are not easy-to-use affordances for it by default. So, it stands to reason, the first step we should take is to make it easier to use Arel queries. That first step is a simple one-line method:

class ApplicationRecord < ActiveRecord::Base
  def self.[](field)
    arel_table[field]
  end
end

With this change, you can use Model[:field] to access the Arel attribute for any field or alias in your models. That allows you to make writing even simple queries less error-prone.

To show what I mean, take the following example from RubyGems.org.

def self.find_by_owner_handle!(handle)
  joins(:user).find_by(users: { handle: handle }) ||
    joins(:user).find_by!(users: { id: handle })
end

One thing that, no matter how long I write Rails applications, I make a mistake on the first try is switching between the singular and plural forms of the model versus the table. To paraphrase one of my daughter’s favorite books …

I understand that tables are plural. I know that models are singular. I know all that. I am aware.

But still I cannot remember in a query.

– Apologies to Mr. Collins

Inevitably, I will read the joins clause (joins(:user)) and intuitively write the predicate as user: { handle: handle } even though I know that it’s SQL that we’re writing in the Hash-based query.

It’s not a big deal.

It’s really isn’t.

But it’s just enough of a paper cut that it can cause me to lose my focus when my test unexpectedly blinks red.

As such, I’ve taken to writing these types of query like so:

def self.find_by_owner_handle!(handle)
  joins(:user).find_by(User[:handle].eq(handle)) ||
   joins(:user).find_by!(User[:id].eq(handle))
end

This is unambiguous. I’m writing more Ruby and less Hash-as-SQL. I don’t have to think about it and the way the Arel attribute works, it feels like I’m accessing a Hash so I have the muscle memory for it. No more paper cuts!

As an aside, because this is Ruby, there’s more than one way to do the same thing and I might instead use the merge method. But that’s because this is a simple example; in general, the principle still holds.

Operator aliasing

Another paper cut that I experience that makes using Arel not quite joyful is remembering the names of the operators. Is it gteq or gteql? Likewise, is it eq or eql … or was it equal? What about neq vs not_eq?

Abbreviations (or lack thereof) for these operators change across languages and, in some cases, include all of them for different uses! If you find yourself hopping between languages with any sort of regularity, it can be difficult to write this correctly on the first try:

Post[:created_at].gteq(1.day.ago)

To remove some ambiguity, you can use the superpowers of Ruby to define operators that you won’t mistake:

module Arel::Predications
  alias_method :>, :gt
  alias_method :>=, :gteq
  alias_method :<, :lt
  alias_method :<=, :lteq
end

That change allows you to write this with a maths-like syntax:

Post[:created_at] > 1.day.ago

Or you might choose to do something heretical and alias gt as after so you can write the following, thus saving some mental overhead when performing date maths:

Post[:created_at].after 1.day.ago

Just remember, with great power and all that.

Extending Arel and ActiveRecord

Sometimes the SQL standard moves more slowly than the industry does, leading to databases innovating ahead of the standard. That innovation is great because it helps raise the tide for everyone, but it can lead to opaque, hard-to-remember syntax that only works for a single database type.

For example, I can never remember any PostgreSQL array functions. Which set of symbols means “contains”? Gotta look it up.

To help with this, you can define custom Arel nodes like the following:

class Arel::Nodes::Contains < Arel::Nodes::InfixOperation
  def initialize(left, right)
    super(:'@>', left, right)
  end
end

This encodes the knowledge of symbol translate to “contains” in the Contains class, which is much easier to remember. Since this is a custom Arel node, you must let your database-specific visitor know how to handle it, which you can do this like this for PostgreSQL:

class Arel::Visitors::PostgreSQL < Arel::Visitors::ToSql
  alias_method(
    :visit_Arel_Nodes_Contains,
    :visit_Arel_Nodes_InfixOperation
  )
end

This change tells the visitor that the Contains node is just an Infix Operation (which is any operation of the form arg1 op arg2, like 1 + 2).

Once you have this change, you can use your custom Arel node in a query like the following:

Post.where(
  Arel::Nodes::Contains.new(
    Post[:tags],
    Arel::Nodes.build_quoted('ruby')
  )
)

This query translates to SELECT * FROM "posts" WHERE "posts"."tags" @> 'ruby'; to look for all posts that have a “ruby” tag. The bit here that you need to know is the build_quoted method, which is responsible for wrapping — or “quoting” — arguments for use as parameters in SQL queries.

You can, of course, abstract this one level higher to ActiveRecord, which I recommend doing. But that’s another post for another day.

Conclusion

Arel is a powerful way to compose your SQL queries. However, the Rails core team considers it a private API so it’s not a smooth experience to use it out-of-the-box.

But this is Ruby! With a little bit of love, Arel can be a joy to use. It can save you from silly mistakes, make your queries more expressive, and be easily extended once you know how it works. I encourage you to give it a try to see if it helps you in your day-to-day work.

Do you use Arel in your project? If so, have you run into any issues with it? What’s your favorite thing to do with it?

Null-based ordering in ActiveRecord

2021-02-01T09:00:00-06:00

When designing your domain model for an application, the natural solution might end up using a nil to model the absence of a value. In these cases, you might need to reach for null-based ordering within your database. When you’re writing a Ruby on Rails application, you’re most likely going to use ActiveRecord as the interface to your database. While ActiveRecord supports most queries out-of-the-box, sometimes you have to go off the beaten path.

Null-based ordering is one of those situations. Since ActiveRecord’s ordering works based on simple key: :direction pairs, it is hard to perform null-based ordering outside of using the built-in behavior of your database. This article outlines two ways you can structure this type of null-based ordering, along with the caveats of using them.

A simple model

For this article, consider the following model. You want to deliver notifications to your customers. Each notification has a header and a body. You deliver the notification to an account via a “delivery” and the owner of the account receives the delivery and reads it at two separate times. When showing notifications to the owner of an account, you want to show unread notifications first in reverse order of delivery, then show the read notifications in the order that the owner read them.

Let’s run through a Class-Responsibility-Collaborator card exercise to define your model. Looking at this short description, we see three nouns that are candidates for models: Account, Notification, and Delivery. We also see four properties: a header, a body, a delivered at time, and a read at time. We also see two responsibilities, “show unread notifications first” and “show read notifications in the order the owner read them.”

We can translate these findings to the following model:

class Account < ApplicationRecord
  has_many :deliveries, class_name: "Notifications::Delivery"
  has_many :notifications, through: :deliveries
end

class Notification < ApplicationRecord
  attribute :body, :string
  attribute :header, :string

  has_many :deliveries, class_name: "Notifications::Delivery"
  has_many :accounts, through: :deliveries
end

class Notifications::Delivery < ApplicationRecord
  alias_attribute :delivered_at, :created_at
  attribute :read_at, :time

  belongs_to :account
  belongs_to :notification
end

This simple model uses a join model to represent the act of delivering a notification. By modeling an act, the join model makes the model more flexible by allowing us to easily deliver notifications to multiple accounts. It also separates the two concerns — the content of the notification and the act of delivering it — so that they can be independently modified.

Now that we have a model, let’s think about how to satisfy the goal of the brief.

Default null ordering

By default, SQL considers NULL to be a value less than any present value. When you order by a nullable field, an ORDER BY read_at ASC places all NULL values at the end of the results, whereas an ORDER BY read_at DESC clause places them at the beginning.

With that knowledge, it’s straightforward on how to place the unread notifications at the start of the list:

Notification
  .joins(:deliveries)
  .order(
    '"notifications_deliveries"."read_at" DESC'
  )

I find SQL fragments to be frustrating to write because typos are less obvious and you can’t use Rails’ aliased attributes so we could refactor this to use ActiveRecord::Relation#merge to write each clause in the natural sense:

Notification
  .joins(:deliveries)
  .merge(Notifications::Deliveries.order(read_at: :desc))

Now we have the unread messages at the beginning of the list … but the most recently read last. That doesn’t quite match our brief. What we want is for the unread notifications to be listed first, but the read ones to be listed in ascending order. What can we do?

Null-based ordering in SQL:2003

The 2003 version of the ISO SQL added a null ordering clause that allows you to specify NULLS FIRST or NULLS LAST for a nullable field. This changes the default interpretation of null for the purposes of sorting. Using this logic, we could change the query to:

Notification
  .joins(:deliveries)
  .order(
    '"notifications_deliveries"."read_at" ASC NULLS FIRST'
  )

This gets the notifications in the order we want, except for when there are multiple unread messages. Let’s fix that:

Notification
  .joins(:deliveries)
  .order(
    '"notifications_deliveries"."read_at" ASC NULLS FIRST',
    '"notifications_deliveries"."created_at" DESC'
  )

The first ORDER BY clause handles “unread notifications first […] read notifications in the order in which the owner read them.” And the second handles “unread notifications in order of delivery” because the database will only apply it to break ties where read_at is the same value (usually only for unread notifications).

Refactoring to Arel

Since ActiveRecord does not, by default, support NULLS FIRST we resorted to two SQL fragments. But as I said earlier, I find SQL fragments to be difficult to write on the first go-around and hard to debug when they result in error. To fix this, we can use the Arel internal querying grammar¹. A refactor might looks something like this:

class Notifications::Delivery < ApplicationRecord
  def self.in_display_order
    order(
      arel_table[:read_at].asc.nulls_first,
      arel_table[:delivered_at].desc
    )
  end
end

With this, we have encapsulated the logic in a chainable method on the relation using fully-qualified field names, so we can go back to using #merge to run the full query:

Notification
  .join(:deliveries)
  .merge(Notifications::Delivery.in_display_order)

There are two extra things that I leave as exercises for the reader:

We are not scoping this query to a single account. How would you make sure you’re only showing deliveries to a particular account?
This is a lot to type and a lot to change if we want to show notifications in this order in multiple places. What could you add or refactor to make it easier to change this logic in the future? Hint: think about how can we hide the implementation detail of deliveries from a module that only wants to know about Notification.

Database support

Since ActiveRecord is a generic object-relational mapper over several different databases, you have to think about whether your database supports this syntax before you try to use this in your own application. Let’s take a look at database support for null-based ordering.

PostgreSQL

PostgreSQL has supported null-based ordering since v8.3. Since this version went end-of-life in 2013, you are most likely on a version newer than that so go ahead!

MySQL

As of v8.0.23, released on 2021-01-18, MySQL does not support this query². You can work around the problem by using some quirks of MySQL’s treatments of null.

To order null values first in a query, we can rely on the fact that IS NULL returns a 1 for a null value and a 0 otherwise. Using that knowledge, we can split the predicate into two:

class Notifications::Delivery < ApplicationRecord
  def self.in_display_order
    order(
      arel_table[:read_at].not_eq(nil),
      arel_table[:read_at].asc,
      arel_table[:delivered_at].desc
    )
  end
end

I’m not really sure how this will interact with indexes since I haven’t used MySQL in production since before the MariaDB split. Since the predicate checks for a null value, I think it will use indexes accordingly, but check before you roll it out to production.

If you use this and have different findings, please let me know!

SQLite

SQLite has supported null-based ordering since v3.30.0, released on 2019-10-30. As of this writing, Debian Stable (Buster) is only on v3.27.2 but Debian Testing and Unstable (Buster) are both on v3.34.1. Ubuntu 20.04, the latest LTS release, is on v3.31.1. And for the Docker users, Alpine is on v3.34.1.

However, as of Rails 6.1.1, the SQLite adapter does not support the Arel version of the query and the SQL fragment version requires a wrap of the NULLS FIRST ordering clause in Arel.sql to prevent an ActiveRecord::UnknownAttributeReference.

With those caveats, a modified version looks like:

class Notifications::Delivery < ApplicationRecord
  def self.in_display_order
    order(
        Arel.sql('"notifications_deliveries"."read_at" ASC NULLS FIRST'),
        '"notifications_deliveries"."created_at" DESC'
    )
  end
end

Conclusion

Null-based ordering allows you to have fine-grained control over the ordering of nullable fields within your database queries. For cases where you need to handle null values differently, using the SQL:2003 null ordering predicates can help you make your sorting logic easier-to-read and understand while also helping you keep querying logic in the database rather than Ruby.

PostgreSQL has excellent support for null-based ordering, both at the database level and the ActiveRecord level. MySQL does not support it, but there are reasonable workarounds. And SQLite requires a small workaround due to not supporting the query at the ActiveRecord level.

Using Arel makes it so you can even use your domain logic when you use alias_attribute, which is an added cherry on top. Even though it’s a private API, it’s stable enough that I feel fine using it with tests around it for regression coverage during Rails upgrades.

Have you ever used null-based ordering in your Rails application? What were you querying for? If not, has there ever been a case where you needed to work around the problem?

It’s important to note that the Rails core team considers Arel to be a private API. I use it extensively and it has been stable for my use cases, but I recommend writing tests around any query in which you use it to make sure that Rails upgrades don’t break your queries. ↩
I know MySQL is still very popular, but I don’t know why. Do you work somewhere that uses MySQL? I’d love to hear why! ↩

Memoization in Ruby using three different patterns

2021-01-25T09:00:00-06:00

Memoization is the process of performing an expensive calculation — a lookup from a database, a long-running function, etc. — and then caching its value to prevent the expensive action from running again. In computer science jargon, it trades space complexity for time complexity; that is, instead of releasing the calculated value for the garbage collector, it stores the value, thus increasing the space requirements of the object, to save on the time it takes to calculate the value more than once.

There are myriad ways to perform memoization in Ruby. The most common means is to use the ||= operator to conditionally set an instance variable upon first running the method. In most cases, this works well. However, there are certain cases where it does not do what you expect it to. This article discusses the semantics and gotchas of memoization in Ruby and shares different patterns you can use to ensure the correct behavior happens every time you choose to memoize a method.

Hello, `||=`, my old friend

One of the first things you see when source-diving or working in Ruby on a team is the use of the ||= operator to save method results as instance variables. Using a simple Rails controller as an example, this pattern looks like the following:

class ApplicationController < ActionController::Base
  def current_user
    @_current_user ||= User.find(session[:user_id])
  end
end

Once you have seen this pattern and understood it at the surface level, it’s easy to apply. The algorithm looks something like this:

Create a method that performs some expensive calculation.
When you see that you use the method more than once, apply @<method_name> ||= on the left-hand side of the method’s body.
Move on to your next task.

But what is really happening here? Let’s break down the semantics of the operator. Were we to expand that line, it would look like the following¹:

class ApplicationController < ActionController::Base
  def current_user
    @_current_user || @_current_user = User.find_by(id: session[:user_id])
  end
end

After seeing the expanded form written out, we can step through its semantics. Due to operator precedence and the everything-is-an-expression nature of Ruby (e.g. there is an implicit return on the last statement of a block or method), there are two steps to this algorithm:

If @_current_user is truthy, return the value from the method.
Otherwise, set @_current_user to the value returned by User.find_by(id: session[:user_id) and return the value from the method.

Simple enough, right? Use the value if it’s there, otherwise calculate it. That accomplishes our goal for much of the problem space, but writing out the algorithm in English helps to illustrate that it doesn’t fully memoize the answer for all types of work that we might do.

When is `||=` not enough?

Think about the English algorithm again. Do you see the problem? To illustrate the point, it will help to veer into Boolean algebra for a moment.

In traditional Boolean algebra, there are two values: true and false. Computers fully run on Boolean algebra as all binary coding and logic run on this branch of algebra. That’s all well and good for the electrons working their way through the silicon of your computer processor, but we are programmers, not electrical engineers.

Computer programming is the practice of telling a computer what to do. However, it’s not only that. It is also the practice of codifying your understanding of a problem such that other humans can tell what the computer will do. Because the designers of the languages that we program mean for your code to be understandable by both machine and human², we often take shortcuts to reduce the amount of mental clutter in our programs. One such shortcut is the introduction of the concepts of truthiness and falsiness, with their itinerant adjectives truthy and falsy.

When we say something is truthy, we mean that when we pass the value to an if statement, the program will take the success branch rather than the failure one. The opposite is true of falsy values. In Ruby, everything is truthy except for an instance of NilClass (i.e. the value nil) or an instance of FalseClass (i.e. the value false). This is one of the reasons why Rails introduced the #present? and #blank? methods and why you see them littered throughout the code base of Rails applications.

Now that we know what we mean when we say truthy, the problem becomes clear: in our method, when we set @_current_user to something falsy, the method will run the calculation again … and again and again, if you call it repeatedly. With that knowledge, how can we ameliorate this problem?

A simple amelioration

The semantics of variables in programming languages is a topic that is hotly debated in language design circles. When should a variable be considered in scope? When is it defined? Do you hoist the definition to the top of the scope or not consider it until the line that declares the variable? Exciting stuff, for those who like that type of thing.

In Ruby, we define a variable — whether it’s a class, instance, or local variable — by putting its name on the left-hand side of an assignment operator³ for the first time. Once we define a variable, we gain access to the keyword-cum-method #defined? for checking whether we defined it previously … even if it is falsy.

Now, with knowledge of #defined?, we can design a new memoization strategy that helps protect our nilable User.find_by method:

class ApplicationController < ActionController::Base
  def current_user
    return @_current_user if defined? @_current_user

    @_current_user = User.find_by(id: session[:user_id])
  end
end

The first time you call the method, the instance variable @_current_user will not be defined so the method body runs, setting @_current_user to the User or nil. The next time you call it, @_current_user is defined so it returns via the guard clause. Peachy!

A gotcha in Rails controllers

Since Rails controllers share their instance variables and methods with the view context, you must exercise your discipline with helper methods and instance variables. Don’t access your instance variables from your view; declare them as helper methods in your controllers. This is the reason why I prefix my instance variables with an underscore in my Rails examples. It helps to communicate that we should not access the instance variable and instead use a defined helper method.

The danger is that when you access the instance variable from the view, there is a chance that you might inadvertently define it. If by accident, you define the instance variable backing your #current_user method in the view, you might break the method for every other caller.

As such, I recommend instituting a policy in your application of not accessing instance variables in views, whether they are partials or not.

Memoizing methods with parameters

Sometimes, we write methods that take parameters and perform expensive actions based on the parameters sent to the method. In those cases, we often still want to make the same trade-off in space for time that we do for singular variables. How can we cache the results of a parameterized method?

Writing out our meaning in English can, again, help:

If we have called the method #expensive with the argument "foo" before, return the value that we cached last time.
Otherwise, calculate the value and cache the result.

Single-parameter methods are the easiest here because the argument is easy to visualize as a cache key. When you want to cache something, you want a fast lookup (ideally O(1)) from a list of values. Expressed in that manner, is there a data structure that comes to mind?

Consider the following:

def expensive(arg)
  return @_expensive_memo[arg] if defined? @_expensive_memo

  @_expensive_memo ||= Hash.new do |memo, key|
    memo[key] = do_expensive(arg)
  end

  @_expensive_memo[arg]
end

Hashes have O(1) lookup cost, so they make a natural fit for the cache storage. Once we make that decision, we can leap to this implementation. Let’s break it down:

The guard clause runs fetches from the cache and uses the fall-through when the value isn’t there, as long as the cache was previously defined.
The storage takes the form of a Hash with a #default_proc. The block takes two arguments: the first is the hash itself and the second is the key that you’re trying to access. By setting the value of the key within the storage hash inside the #default_proc, we ensure that a call calculates the value and caches it.
The call to @_expensive_memo[arg] ensures that we get the calculated value from the first time we run the #expensive method instead of the cache store.

This pattern works as long as arg has a stable #hash method. This is true for built-ins like String and the Numeric classes, ActiveRecord objects, and Structs, but is not true by default for plain old Ruby objects that you implement yourself, so keep that in mind if you need to pass rich objects to your memoized method.

Extending for multiple arguments

What if you have multiple arguments that you need to pass to the method? You can pass an Array of objects as a Hash key, so it’s a small tweak to get that to work:

def expensive(arg1, arg2)
  @_expensive_memo ||= Hash.new do |memo, (inner_arg1, inner_arg2)|
    memo[[inner_arg1, inner_arg2]] = do_expensive(inner_arg1, inner_arg2)
  end

  @_expensive_memo[[arg1, arg2]]
end

In this example, we are destructuring the cache key into its constituent parts and then recombining them back into an array when setting the key. You could also use the * or ** operators if you choose to break out the calculation part into its own method; it depends on how communicative you want the signature of the memoized method to be. That could look like the following:

def expensive(*args)
  @_expensive_memo ||= Hash.new do |memo, inner_args|
    memo[inner_args] = do_expensive(*inner_args)
  end

  @_expensive_memo[args]
end

Alternatives to memoization

The advantage of memoization is that it allows your objects to be lazy and not initialize their state until you need it. But sometimes you have state that you will always need to initialize, even though it’s expensive. In those cases, consider setting the state as part of the initializer and using a simple attr_reader to read the state. This is Ruby, after all; if there’s one way to do it, there are a dozen other ways as well.

Conclusion

When you reach for memoization in Ruby, don’t blindly fall into the habit of @my_var ||= "something expensive". First, think about whether your calculation can return a falsy result. Particularly when you’re fetching single models from ActiveRecord or calculating a Boolean predicate, you have to consider the values returned by the method. If there’s the possibility of a falsy result, then adjust your method to use the check for definition. And lastly, when you need to memoize multiple values from one method, reach for a Hash to store those results.

Memoization can be a powerful tool in your toolbelt, but like all tools, pay attention to overuse. Also, consider whether the lazy initialization buys you anything in the context of what you’re currently working on. Perhaps it would be better to instantiate the state up front instead.

Has erroneous memoization ever bit you? What’s the worst example of a mistake you’ve made using memoization? Do you have any particularly great wins from applying the pattern?

Using RubyVM::InstructionSequence.compile().disasm shows that this isn’t quite correct … but it is close enough to illustrate the point. ↩
We mean languages to be understandable with some exceptions. Malbolge comes to mind. ↩
Or a variety of other ways, including instance_variable_set, but that is beyond the scope of this article. ↩

Implementing account impersonation in ActionCable

2021-01-18T09:00:00-06:00

Account impersonation is when you allow accounts (usually privileged accounts, e.g. support staff or developers) to operate your Rails application as if they are the owner of another account entirely. It’s a helpful feature to have when diagnosing issues that your customers write in about. There are plenty of gems that implement impersonation for your Rails controllers, but I wasn’t able to turn up any suggestions for implementing this for ActionCable.

This post outlines a simple method for adding impersonation to your Rails application in both your controllers and your ActionCable channels. I discuss some ActionCable features that are not well-documented and that I found a little confusing, then show examples of how to use the same pattern for identifying both your active account and your impersonating account — called a “shadower” in this article — in your channels.

Active accounts in controllers

In Rails tutorials, it’s common to see the authors reach for a batteries-included authentication gem like Devise, Clearance, or AuthLogic. For the purposes of this post, we ignore all of these gems and hand-wave over the authentication bits in favor of a simple example. In the interest of shared understanding, we use the ApplicationController#current_user pattern common to all three of the big libraries¹.

class SessionsController < ActionController::Base
  def create
    if authenticated_user
      reset_session
      session[:user_id] = authenticated_user.id

      redirect_to root_path
    else
      # Do some error handling
    end
  end

  private

  def authenticated_user
    # Do something here to authenticate the session or return `nil`
  end
end

class ApplicationController < ActionController::Base
  protected

  def current_user
    @_current_user ||= User.find_by(id: session[:user_id])
  end
  helper_method :current_user
end

In this example, we see a simple way to load the operating User from the database. In our sessions controller, we authenticate the session and then store an account identifier in the session. In our ApplicationController, we then use that identifier to look the account up and set up our current_user value. This is the simplest implementation and works well enough for our purposes.

Rails gives us some straightforward, but powerful, tools for making the stateless web feel like we are writing a stateful application. Session management is one of those tools². By storing a small value in the session, we suddenly have access to the current account in any controller descending from ApplicationController. Neat!

Extending this to ActionCable channels

As nice as sessions are, they are a construct only available inside the controllers of your applications. Since ActionCable channels are not controllers, we must find a different way to share information between our controllers and channels. Even the controversial ActiveSupport::CurrentAttributes tool can’t help us here because your ActionCable channel works on a separate thread from the controller serving the page that starts the channel.

The behavior of decoding the session cookie belongs to the controller and is not accessible in ActionCable channels³. However, that does not mean that cookies are inaccessible in channels, only that the session isn’t available. In fact, the Rails Guides give an example of using a cookie to identify the account for the channel. Wire this in as an encrypted cookie⁴ at the same place that you set the session value:

class SessionsController < ActionController::Base
  def create
    if authenticated_user
      reset_session
      session[:user_id] = authenticated_user.id
      cookies.encrypted[:user_id] = { values: authenticated_user.id, expires: 1.hour }

      redirect_to root_path
    else
      # Do some error handling
    end
  end
end

An important decision here is the expires value. Depending on what you’re going to do with ActionCable, you might need this to be longer or shorter. If you’re using ActionCable only for short-lived actions and most of the time your customers navigate through your application via controllers, prefer a short expiry. But if you’re building something like a persistent chat service where your customers spend a long time on one page, a longer expiry is likely what you want.

The trade-off between too short and too long is a trade-off between convenience and security. If you have an overly long expiry and an attacker manages to get hold of your customer’s cookie, then the attacker has a long time to perform session hijacking attacks against your system and your customer. But if you have an expiry that is too short and the customer gets disconnected — whether that’s through a dropped connection, a redeploy, or any other of a number of reasons — they either must refresh the page for their web socket to reconnect or use some JavaScript to refresh their cookie from an API endpoint.

Once you’ve decided upon an expiry length and set your cookie, access its value in your channel like the Rails guide shows.

class ApplicationCable::Connection < ActionCable::Connection::Base
  identified_by :current_user

  def connect
    unless (self.current_user = User.find_by(id: cookies.encrypted[:user_id]))
      reject_unauthorized_connection
    end
  end
end

The identified_by call metaprograms three things for us:

It creates an attr_accessor on the connection for current_user, which we then use in the connect callback.
It also adds a delegated method in your ApplicationCable::Channel class that exposes the Connection’s current_user as an instance method on the Channel.
It sets up connection identification for the connection when the browser initiates a subscription. This helps ActionCable to route broadcasts meant for a particular account.

Our simple implementation of #connect:

Attempts to find the current_user via the cookie that we set in the SessionsController.
If the cookie is not set or the User does not exist, the channel rejects the connection and disconnects.

Once we have this setup, we can stream broadcasts in a particular channel for the current_user.

class CommandsChannel < ApplicationCable::Channel
  def subscribed
    stream_for current_user
  end
end

This simple channel now streams any messages you broadcast to the current_user. To broadcast a message, use the ActionCable::Channel.broadcast_to method, like so:

CommandsChannel.broadcast_to(User.first, data)

Simple impersonation in controllers

Now that you have your current_user set up for both controllers and channels, it’s time to add the behavior for shadowing a user.

class ShadowsController < ActionController::Base
  # Secure this to make sure it's only accessible
  # to those who should have access
  def create
    if shadowed_user
      session[:shadower_id] = session.delete(:user_id)
      session[:user_id] = shadowed_user.id

      redirect_to root_path
    else
      # Do some error handling
    end
  end

  private

  def shadowed_user
    @_shadowed_user ||= User.find_by(id: params[:user_id])
  end
end

class ApplicationController < ActionController::Base
  protected

  def current_user
    @_current_user ||= User.find_by(id: session[:user_id])
  end

  # Secure this to make sure it's only accessible
  # to those who should have access
  def shadower
    @_shadower ||= User.find_by(id: session[:shadower_id])
  end
end

This simple addition retains the meaning of session[:user_id] to mean “the account whose permissions we should use for accessing resources” and layers on a shadower, an account who is “shadowing” or “looking over the shoulder” of the other account.

As an exercise, think about how you would adjust your access policies for specific resources based on who is shadowing whom.

Impersonation in channels

Extending our shadowing system to ActionCable channels is simple, but leaves some open questions as to how to handle the feature. First, we set the cookies in the controller:

class ShadowsController < ActionController::Base
  def create
    if shadowed_user
      session[:shadower_id] = session.delete(:user_id)
      session[:user_id] = shadowed_user.id
      cookies.encrypted[:user_id] = { value: session[:user_id], expires: 1.hour }
      cookies.encrypted[:shadower_id] = { value: session[:shadower_id], expires: 1.hour }

      redirect_to root_path
    else
      # ...
    end
  end

  private

  def shadowed_user
    @_shadowed_user ||= User.find_by(id: params[:user_id])
  end
end

Next, we add information to ApplicationCable::Connection about a subscription’s identification:

class ApplicationCable::Connection < ActionCable::Connection::Base
  identified_by :current_user, :shadower

  def connect
    unless (self.current_user = User.find_by(id: cookies.encrypted[:user_id]))
      reject_unauthorized_connection and return
    end
    self.shadower = User.find_by(id: cookies.encrypted[:shadower_id])
  end
end

Notice how we do not reject the connection when a shadower doesn’t exist; that’s because it’s the normal mode of operation for the app. Now there are two states for “who is viewing this session”: an account with a shadower and one without. Depending on the type of application and the type of channel that you’re writing, you now have to think about broadcasting strategies for messages within your channel.

Broadcasting strategies

When there is only a single account viewing a session at a time, it’s easy to decide the receiver for a given broadcast: the account. However, when you have two means of identifying a session, you have to decide between two different broadcasting strategies for messages⁵:

The broadcast makes sense to show on sessions for only the account without a shadower.
The broadcast makes sense to show on sessions for an account with a shadower.

Broadcasting to accounts, regardless of shadower

For messages that should show up for any session involving the shadowed account (whether operated by the account owner or a shadower), you must stream messages for that account and broadcast to that account:

class CommandsChannel < ApplicationCable::Channel
  def subscribed
    stream_for current_user
  end
end

As shown in the table below, messages on this stream get delivered anywhere the “current user” matches the “broadcast to,” whether or not the session involves a shadower.

Current User	Shadower	Broadcast to	Received
Alice	nil	Alice	x
Alice	nil	Bob
Alice	nil	[Alice, Bob]
Alice	Bob	Alice	x
Alice	Bob	Bob
Alice	Bob	[Alice, Bob]

This broadcasting strategy is suitable for functionality like notifications, messages, or other streams of information.

Broadcasting only to shadowed sessions

For messages that should show up on any current-shadowed sessions, stream messages to an identifier that uses both accounts by passing them as an array:

class CommandsChannel < ApplicationCable::Channel
  def subscribed
    stream_for [current_user, shadower]
  end
end

This channel streams messages only where shadower currently shadows current_user, as shown below. Note that if shadower is nil, it’s not equivalent to stream_for current_user because the identifier looks different.

Current User	Shadower	Broadcast to	Received
Alice	nil	Alice
Alice	nil	Bob
Alice	nil	[Alice, Bob]
Alice	Bob	Alice
Alice	Bob	Bob
Alice	Bob	[Alice, Bob]	x

This broadcasting strategy is suitable for functionality like transactional actions where the message governs the action to take after a command.

Mixing and matching streams

If you have a general channel like, in our example, a CommandsChannel that’s accessible for both the shadowed account and the shadower, you can stream messages from multiple sources:

class CommandsChannel < ApplicationCable::Channel
  def subscribed
    stream_for current_user
    stream_for [current_user, shadower]
  end
end

This channel streams messages that match either of the preceding patterns, as shown in the following table.

Current User	Shadower	Broadcast to	Received
Alice	nil	Alice	x
Alice	nil	Bob
Alice	nil	[Alice, Bob]
Alice	Bob	Alice	x
Alice	Bob	Bob
Alice	Bob	[Alice, Bob]	x

If you use the pattern shown here, you might prefer this construction, where it subscribes to a shadowed session if there is a shadower, otherwise subscribing as the current user:

class CommandChannel < ApplicationCable::Channel
  def subscribed
    stream_for [current_user, shadower].compact
  end
end

The semantics of issuing a broadcast get a little tricky here. As an exercise, think of how you would write an easy-to-use interface for broadcasting all of these types of strategies without needing to litter your codebase with that logic.

Conclusion

Because ActionCable does not, necessarily, have access to the session from the browser, you have to replicate the behavior for tracking the current user and shadower in the encrypted (or signed) cookie store. There are security implications with your decision, so be aware when you’re selecting an expiry. Tracking a session identifier that you control outside the cookie store is also a good idea; keep in mind the sessions section of the Rails security guide for tips on securely handling cookies.

Once you have the shadowing system set up, you must think about the broadcasting strategy for each channel that you implement. Depending on the type of information you broadcast to the channel, you might want to make sure to only broadcast to sessions by the same “identity pair” so that messages don’t intermingle between an account and a shadowed account; these types of messages are for transactional changes. Or, perhaps, you have something like a notifications channel that you want to send to anyone viewing an account. Your choice of strategy can make either of those things happen.

Do you allow shadowing in your application? If so, who do you allow to shadow? Do customers shadow each other, or is it only for support?

I truly detest the term “user” to mean a person who is currently operating our software. Only in drug addiction, abusive relationships, and software development do we find it acceptable to call someone a “user”. But since this is the established pattern, we bend to societal pressure for clarity. ↩
For more information about sessions, Justin Weiss has a nice article about Rails sessions and how they interoperate with cookies by default. ↩
Incidentally, if you use the built-in cookie store from Rails, you can access session[:user_id] using cookies.encrypted[Rails.application.config.session_options.fetch(:key, '_session')]. I don’t recommend doing this because there’s no guarantee that it will continue to work in future versions of Rails outside of being briefly mentioned in the ActionCable overview. ↩
You can also use cookies.signed here instead of cookies.encrypted. I recommend an encrypted cookie instead of a signed cookie because there’s no reason you should expose an internal identifier for your models. ↩
In actuality, there are four different configurations: current_user, [current_user, shadower], shadower, and [shadower, current_user], but the last two don’t have cases where I think it makes sense to use them if you’re using the former two. “Shadower-only” streams don’t do what you’d expect them to; they get delivered to other sessions of the shadower that might still be open without shadowing anyone. [shadower, current_user] reads more confusingly to me than [current_user, shadower] and has the same semantics, as long as you’re consistent. ↩

Michael Herold | Articles

Extending ActiveRecord with custom types

Type classes

Built-in base classes

Registering a type class

Configurable type declarations

Using a type class without registering it

Example: GlobalID

Rules of thumb

Robustness

Value objects

Conclusion

Use the Attributes API to make your Rails models more communicative

What is the Attributes API?

Using the Attributes API

ActiveRecord types

Attribute declarations

Benefits of declaring your attributes

Detriments of the API

Conclusion

Solving the right problem

The report

How I heard it and my mistake

Reading it again

A better solution?

The right choice

What to do next time

Using ox-hugo without duplicating content in the repository

Project

Publish script

Run me a shell script, please

Just kidding, actually run some Emacs code

Makefile

netlify.toml

Conclusion

Provisioning a Valheim server with Terraform

Create an SSH key

Configure the Terraform provider

Set up the SSH key

Create the server

Configure the server

Create necessary directories

Upload Systemd unit

Upload environment file

Install Docker and start Valheim

Conclusion

Duplicate cookies in Ruby on Rails

Basic cookie use

Trade-offs between methods

Sharing them across multiple domains

Duplicate cookies?

Rules of thumb

Conclusion

3 Ways to Make Arel a Joy

Table accessor

Operator aliasing

Extending Arel and ActiveRecord

Conclusion

Null-based ordering in ActiveRecord

A simple model

Default null ordering

Null-based ordering in SQL:2003

Refactoring to Arel

Database support

PostgreSQL

MySQL

SQLite

Conclusion

Memoization in Ruby using three different patterns

Hello, ||=, my old friend

When is ||= not enough?

A simple amelioration

A gotcha in Rails controllers

Memoizing methods with parameters

Extending for multiple arguments

Alternatives to memoization

Conclusion

Implementing account impersonation in ActionCable

Active accounts in controllers

Extending this to ActionCable channels

`netlify.toml`

Hello, `||=`, my old friend

When is `||=` not enough?