Database patterns in Scala

This is the first in a three part series about our database access layer and patterns we’ve adopted (part 2: Clean and re-usable Slick modules). We use Slick to type-check and build our queries, but many of these ideas may help with other libraries.

There’s a decent amount of material out about Slick basics (how to query), but not much about building a larger, multi-module app using it. This post is a soft introduction into some of the lessons we’ve learned and patterns that keep us sane.

• Type what’s important. In Scala, typing is easy (and with value classes, cheap). So, type your values. Id[User] is much better than Long. The compiler will make sure you don’t mix your IDs, and it’s always clear what you’re working with.

  case class Id[T](id: Long) { override def toString = id.toString }   // And if you're in the Play! world, these help too: object Id { // Slick MappedColumnType for any Id[M] implicit def idMapper[M <: Model[M]] = MappedColumnType.base[Id[M], Long](_.id, Id[M])   // JSON formatter def format[T]: Format[Id[T]] = Format(__.read[Long].map(Id(_)), new Writes[Id[T]]{ def writes(o: Id[T]) = JsNumber(o.id) })   // Binders for Play query string and paths implicit def queryStringBinder[T](implicit longBinder: QueryStringBindable[Long]) = new QueryStringBindable[Id[T]] { override def bind(key: String, params: Map[String, Seq[String]]): Option[Either[String, Id[T]]] = { longBinder.bind(key, params) map { case Right(id) => Right(Id(id)) case _ => Left("Unable to bind an Id") } } override def unbind(key: String, id: Id[T]): String = { longBinder.unbind(key, id.id) } }   implicit def pathBinder[T](implicit longBinder: PathBindable[Long]) = new PathBindable[Id[T]] { override def bind(key: String, value: String): Either[String, Id[T]] = longBinder.bind(key, value) match { case Right(id) => Right(Id(id)) case _ => Left("Unable to bind an Id") } override def unbind(key: String, id: Id[T]): String = id.id.toString } }

• Determine common patterns between your tables, and only write it once. All of our tables have id, createdAt, and updatedAt columns, so we implement them on the RepoTable (which extends Slick’s Table) level. Additionally, some of our tables have an externalId column, so it’s an optional mix-in.
• Log your query statements, watch out for inefficient generated SQL. The easiest way to do this is to wrap java.sql.Connection with logging around prepareStatement, prepareCall, nativeSQL, and createStatement.
• Sometimes, hand writing SQL is the best option. Slick makes it easier than writing your own prepared statements.

  def getOverdueCount(due: DateTime = currentDateTime)(implicit session: RSession): Int = { import StaticQuery.interpolation sql"select count(*) from scrape_info where next_scrape < $due".as[Int].first // not an injection! :) }

• Precompile your hot queries. Slick lets you compile queries ahead of time, and provide parameters as needed. It’s more verbose, and isn’t nearly as flexible, but in our benchmarks, you can sometimes save ~1ms per query.
• Use a caching layer for complex queries. Of course, cache invalidation can get complex, so be careful. Yasuhiro Matsuda wrote a wonderful transactional caching layer that sits on top of in-memory and memcached caches. A little birdie told me he’ll write about it soon, so stay tuned.

  override def get(id: Id[User])(implicit session: RSession): User = { idCache.getOrElse(UserIdKey(id)) { (for(f <- rows if f.id is id) yield f).firstOption.getOrElse(throw NotFoundException(id)) } }