Zefira's Digital GardenOut beyond ideas of wrongdoing and rightdoing there is a garden. Let us meet there and talk about code.Zola2015-08-01T00:00:00+00:00https://zefira.dev/atom.xmlUsing Datomic Pro with Boot2015-08-01T00:00:00+00:002015-08-01T00:00:00+00:00https://zefira.dev/posts/using-datomic-pro-with-boot/<p>I first heard about <a href="http://www.datomic.com/">Datomic</a> shortly after it was initially
released. I think I almost immediately went and read all of the
documents that <del><a href="http://thinkrelevance.com/">Relevance</a></del> <a href="http://cognitect.com/">Cognitect</a> released about
it's architecture. I was duly impressed, and really interested in
using it. In particular, the potential for
<a href="http://www.datomic.com/benefits.html">time-travelling</a> was really interesting and
exciting to me.</p>
<p>At the time I had been tasked with writing some code to determine
usage metrics for our system. One of my boss's hopes was that we could
generate usage data from some time ago as well to compare with our
current levels. Because we were using traditional relational databases
and hadn't planned for this use case, that turned out to be
impossible. However, if we had been using Datomic, that back-dated
query would have been absolutely trivial.</p>
<!--more-->
<p>When I attempted to actually learn to use Datomic however, I was
overwhelmed by the intensity and relative scarcity of the
documentation. My general inexperience meant that I wasn't prepared to
be an early adopter of this technology.</p>
<blockquote>
<p>My problem with the vast majority [of Datomic tutorials] is that
they seem to be written for people who don't need a tutorial, and by
and large all have the stench of "read the code and you'll
understand," quickly coupled with "and if you don't, you're not smart
enough to use this technology anyway."</p>
</blockquote>
<p>I read this in a <a href="http://ben.vandgrift.com/2014/04/24/a-clojure-datomic-web-app-tutorial.html">tutorial blog about Datomic</a> that I
recently ran across. It neatly sums up how I felt about Datomic at
that time. So nothing much came of my first attempt to learn Datomic.</p>
<p>Recently though, my interest was piqued again by a talk given by one
of my current co-workers. At the same time, I've been inspired
recently to pick up again a project I started around the time I was
first interested in Datomic, <a href="https://github.com/RadicalZephyr/rotateam">Rotateam</a>. Given my recent
love affair/obsession with Clojure, and in particular the
<a href="https://github.com/boot-clj/boot">Boot</a> project for creating Clojure <a href="http://boot-clj.com/">build tooling</a>, I
obviously wanted to rebuild the project using a
Boot/Clojure/Clojurescript stack. And what better database technology
to use than Datomic!</p>
<p>I chose to tackle the part I was least familiar with first: Datomic. I
followed the links and instructions from the Datomic home page until I
found out about the new (to me at least) free license for
<a href="http://www.datomic.com/get-datomic.html">Datomic Pro Starter Edition</a>. From there I just followed
the getting started instructions and the tutorial.</p>
<p>Since Datomic is very proprietary software, the jar's for it aren't
available from public repositories. Instead, when you register with My
Datomic you get a generated password that gives you access to their
private, password-protected Maven repository. Conveniently, when you
sign up for the Starter Edition you are presented with a page that
tells you how to setup Maven and Leiningen projects to pull from these
repositories. They show you an excellent setup too, that doesn't
require the security-sin of committing secrets to a Git repository.</p>
<p>Unfortunately, since Boot is still a relative newcomer to the Clojure
ecosystem, there are no official Cognitect-supported instructions for
securely using the My Datomic Maven repository with Boot.</p>
<p>Sounds like the perfect opportunity for a blog post :D</p>
<p><a id="top-down"></a></p>
<h2 id="top-down"><a href="#top-down" aria-label="Anchor link for: top-down"><i class="fa fa-link" aria-hidden="true" ></i></a>
Top Down</h2>
<p>Let's start from what we want to be able to do. In an ideal world, I
would be able to simply include Datomic Pro as a dependency in my
<code>:dependencies</code> list. In a typical <code>build.boot</code> file that would look
like this:</p>
<h5 id="build-boot"><a href="#build-boot" aria-label="Anchor link for: build-boot"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]])
</span></code></pre>
<p>If we try this out it pretty clearly fails. On my machine Boot spat
out a huge amount of junk that boiled down to "Sorry chap, I couldn't
find version <code>0.9.5206</code> of <code>com.datomic/datomic-pro</code> for you." Of
course, this is because we haven't told boot how to look in the Maven
repository where it exists: <code>https://my.datomic.com/repo</code>.</p>
<p>Let's try the simplest thing that could possibly work. If we look at
the Boot
<a href="https://github.com/boot-clj/boot/wiki/Boot-Environment#env-keys">documentation on the keys in the Boot environment</a>, we can
see that there is a handy <code>:repositories</code> key that we should probably
be setting with the details of our My Datomic credentials. If we check
out the <a href="https://github.com/cemerick/pomegranate/blob/pomegranate-0.3.0/src/main/clojure/cemerick/pomegranate/aether.clj#L639-L650">documentation in Pomegranate</a> for what the values of
the <code>:repositories</code> vector should look like, we can see that to
include authentication credentials, we need to specify it as a
map. Concretely our <code>build.boot</code> should look more like this:</p>
<p><a id="first-working"></a></p>
<h5 id="build-boot-1"><a href="#build-boot-1" aria-label="Anchor link for: build-boot-1"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]]
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">" </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:username </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">notmyemail@example.com</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:password </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">obviously-not-my-password</span><span style="color:#d6d6d680;">"</span><span>}]))
</span></code></pre>
<p>The reason for the funny syntax of specifying a lambda function as the
repositories value is because we want to update the value
<code>:repositories</code> by adding (<code>conj</code>ing) the <code>my-datomic</code> repository. We
can't just blindly override the repositories, because by default Boot
adds entries for pulling from <a href="http://search.maven.org/">Maven Central</a> and
<a href="https://clojars.org/">Clojars</a> which we probably don't want to blow away.</p>
<p><a id="side-note"></a></p>
<h3 id="side-note"><a href="#side-note" aria-label="Anchor link for: side-note"><i class="fa fa-link" aria-hidden="true" ></i></a>
Side Note</h3>
<p>Rather than bothering to go and read the Pomegranate documentation, we
also could have inspected Boot's default environment. Boot ships with a
handy task called <code>show</code> which is useful for this sort of inspection.
For Leiningen users, it's sort of equivalent to <code>lein pprint</code>. In this
case, since we're interested in what's in the environment we want to
run <code>boot show -e</code> or <code>boot show --env</code>. And of course, as with all
Boot tasks we could find out this information by running <code>boot show -h</code>. Okay, <code></side note></code>.</p>
<p><a id="back-to-it"></a></p>
<h2 id="back-to-it"><a href="#back-to-it" aria-label="Anchor link for: back-to-it"><i class="fa fa-link" aria-hidden="true" ></i></a>
Back to It</h2>
<p>Obviously you'd need to put your real My Datomic credentials in
there. Just as obviously, this can't be the final form our solution
takes. The <code>build.boot</code> file pretty much <em>needs</em> to be under source
control, and those secrets need to <strong><em>not</em></strong> be present in source
control.</p>
<p>So <a href="https://zefira.dev/posts/using-datomic-pro-with-boot/#first-working">that</a> works. But to develop a more elegant and
secure solution, we're probably going to need a better understanding
of just how Boot goes about loading dependencies. Now, we could go and
read the source. Or we could treat Boot as a black box and play with
it until we have a better understanding of how it works. I vote the
latter.</p>
<p>Let's start with the simplest thing that probably <strong>won't</strong> work. What if
we move the <code>:repositories</code> update into a separate <code>set-env!</code> like this:</p>
<h5 id="build-boot-2"><a href="#build-boot-2" aria-label="Anchor link for: build-boot-2"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]])
</span><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">" </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:username </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">notmyemail@example.com</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:password </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">obviously-not-my-password</span><span style="color:#d6d6d680;">"</span><span>}]))
</span></code></pre>
<p>This again fails spectacularly for the same reason of not being able
to resolve the Datomic dependency. But this test has told us something
important. It tells us that the dependency resolution happened <em>during
the first call to <code>set-env!</code></em>. This is important, because it implies
that if we get the <code>my-datomic</code> repository configuration into the boot
environment before that call, then everything should work just
fine. Let's try it out:</p>
<h5 id="build-boot-3"><a href="#build-boot-3" aria-label="Anchor link for: build-boot-3"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">" </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:username </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">notmyemail@example.com</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:password </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">obviously-not-my-password</span><span style="color:#d6d6d680;">"</span><span>}]))
</span><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]])
</span></code></pre>
<p>Hey, presto! It works. Let's think about why this works for a moment
and what the implications are. Obviously, at some point during the
<code>set-env!</code> function, some code gets called that notices that a new
dependency was added, and attempts to resolve it. As long as the
repository required to resolve that dependency is present in the list
of repositories at that moment, then everything works fine. This is an
excellent example of what the Boot authors are talking about when they
say that Boot builds are programs.</p>
<p>If you're like me then long familiarity with declarative build systems
has lulled you into thinking of build description files as
fundamentally not code. Even though a Leiningen project map is
entirely made of Clojure data structures, my experiences have taught
me that it isn't really code. But a Boot build file is. It's really just
executing Clojure code.</p>
<p>When I was first discovering how this worked for myself, I was working
on an actual project, and the <code>build.boot</code> was significantly more
complex. As such, I broke out the Datomic specific portions into the
snippets that I've included in this blog post. But because of my
build's-as-specification indoctrination, I had fallen into a rhythm of
always having my <code>build.boot</code> files have a certain structure to them.</p>
<h5 id="my-prototypical-build-boot"><a href="#my-prototypical-build-boot" aria-label="Anchor link for: my-prototypical-build-boot"><i class="fa fa-link" aria-hidden="true" ></i></a>
My prototypical build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span style="color:#a0cfa1;">;;;</span><span style="color:#87ae86;"> Start with source paths and dependencies
</span><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:source-paths </span><span>#{</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">src</span><span style="color:#d6d6d680;">" "</span><span style="color:#d68686;">test</span><span style="color:#d6d6d680;">"</span><span>}
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[org.clojure/clojure </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">1.6.0</span><span style="color:#d6d6d680;">"</span><span>]
</span><span> [midje </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">1.7.0</span><span style="color:#d6d6d680;">" </span><span style="font-weight:bold;color:#cc9495;">:scope </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">test</span><span style="color:#d6d6d680;">"</span><span>]
</span><span> [zilti/boot-midje </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.2.1-SNAPSHOT</span><span style="color:#d6d6d680;">" </span><span style="font-weight:bold;color:#cc9495;">:scope </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">test</span><span style="color:#d6d6d680;">"</span><span>]
</span><span> [radicalzephyr/bootlaces </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.1.15-SNAPSHOT</span><span style="color:#d6d6d680;">"</span><span>]])
</span><span>
</span><span style="color:#a0cfa1;">;;;</span><span style="color:#87ae86;"> Require bootlaces and other boot tasks
</span><span>(require </span><span style="color:#ececec;">'</span><span>[radicalzephyr.bootlaces </span><span style="font-weight:bold;color:#cc9495;">:refer :all</span><span>]
</span><span> </span><span style="color:#ececec;">'</span><span>[zilti.boot-midje </span><span style="font-weight:bold;color:#cc9495;">:refer </span><span>[midje]])
</span><span>
</span><span style="color:#a0cfa1;">;;;</span><span style="color:#87ae86;"> Define the project version
</span><span>(def +version+ </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.1.1-SNAPSHOT</span><span style="color:#d6d6d680;">"</span><span>)
</span><span>
</span><span style="color:#a0cfa1;">;;;</span><span style="color:#87ae86;"> And use the bootlaces configuration
</span><span>(bootlaces! +version+)
</span><span>
</span><span style="color:#a0cfa1;">;;;</span><span style="color:#87ae86;"> Finally, configure any default task-options
</span><span>(task-options!
</span><span> pom {</span><span style="font-weight:bold;color:#cc9495;">:project </span><span style="color:#ececec;">'</span><span>radicalzephyr/rotateam
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:version </span><span>+version+
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:description </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">A web-app for scheduling team role rotations.</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://github.com/radicalzephyr/clj-rotateam</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:scm </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://github.com/radicalzephyr/clj-rotateam</span><span style="color:#d6d6d680;">"</span><span>}
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:license </span><span>{</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">Eclipse Public License</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">http://www.eclipse.org/legal/epl-v10.html</span><span style="color:#d6d6d680;">"</span><span>}})
</span></code></pre>
<p>This structure is <em>very</em> reminiscent of a <code>project.clj</code> file. It's
format is slightly different, but there's really nothing that takes
advantage of the fact that this is actually a regular Clojure
program. This is true for a reason though, and again it's mentioned in
Boot's rationale. Simple projects don't need the flexibility of having
their build be a real program. But here's the thing, simple projects
tend to become complex projects over time.</p>
<p><a id="back-to-datomic"></a></p>
<h2 id="back-to-datomic"><a href="#back-to-datomic" aria-label="Anchor link for: back-to-datomic"><i class="fa fa-link" aria-hidden="true" ></i></a>
Back to Datomic</h2>
<p>Okay, enough philosophizing. What does this build as program mean for
storing and accessing our My Datomic credentials securely? Well for
starters, it means we can do something really simple like following
the Heroku paradigm of putting secrets into environment
variables. Pulling them out is easy with a little bit of Java-interop.</p>
<h5 id="build-boot-4"><a href="#build-boot-4" aria-label="Anchor link for: build-boot-4"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]]
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">" </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:username </span><span>(System/getenv </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_USERNAME</span><span style="color:#d6d6d680;">"</span><span>)
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:password </span><span>(System/getenv </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_PASSWORD</span><span style="color:#d6d6d680;">"</span><span>)}]))
</span></code></pre>
<p>This again works perfectly. But why stop there? This solution only
works when you have your Datomic username and password set as
environment variables. Instead, we could fallback to prompting the
user for the credentials. Borrowing and adapting some code from
<a href="https://github.com/adzerk-oss/bootlaces">Adzerk's bootlaces</a>, we can provide a reasonable
fallback experience.</p>
<h5 id="build-boot-5"><a href="#build-boot-5" aria-label="Anchor link for: build-boot-5"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(let [[user pass] (mapv #(System/getenv %)
</span><span> [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_USERNAME</span><span style="color:#d6d6d680;">" "</span><span style="color:#d68686;">DATOMIC_PASS</span><span style="color:#d6d6d680;">"</span><span>])
</span><span> datomic-creds (atom {})]
</span><span> (if (and user pass)
</span><span> (swap! datomic-creds assoc </span><span style="font-weight:bold;color:#cc9495;">:username </span><span>user </span><span style="font-weight:bold;color:#cc9495;">:password </span><span>pass)
</span><span> (do (println (str </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_USERNAME and DATOMIC_PASS were not set;</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;"> please enter your Datomic credentials.</span><span style="color:#d6d6d680;">"</span><span>))
</span><span> (print </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">Username: </span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (#(swap! datomic-creds assoc </span><span style="font-weight:bold;color:#cc9495;">:username </span><span>%) (read-line))
</span><span> (print </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">Password: </span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (#(swap! datomic-creds assoc </span><span style="font-weight:bold;color:#cc9495;">:password </span><span>%)
</span><span> (apply str (.readPassword (System/console))))))
</span><span> (set-env! </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">"
</span><span> (merge </span><span style="color:#ececec;">@</span><span>datomic-creds
</span><span> {</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"</span><span>})])))
</span></code></pre>
<p>This code may look a bit intimidating, but it's mostly managing the
details of user friendly input and output. But again, why stop here?
This is <em>just Clojure code</em> here, so all of Clojure's ability to
define and use abstractions is right there.</p>
<h5 id="build-boot-6"><a href="#build-boot-6" aria-label="Anchor link for: build-boot-6"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(defn get-cleartext [prompt]
</span><span> (print prompt)
</span><span> (read-line))
</span><span>
</span><span>(defn get-password [prompt]
</span><span> (print prompt)
</span><span> (apply str (.readPassword (System/console))))
</span><span>
</span><span>(let [user
</span><span> (or (System/getenv </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_USERNAME</span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (get-cleartext </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_USERNAME was not defined.</span><span style="font-weight:bold;color:#ff8080;">\n</span><span style="color:#d68686;">Username: </span><span style="color:#d6d6d680;">"</span><span>))
</span><span> pass
</span><span> (or (System/getenv </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_PASSORD</span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (get-password </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_PASSWORD was not defined.</span><span style="font-weight:bold;color:#ff8080;">\n</span><span style="color:#d68686;">Password: </span><span style="color:#d6d6d680;">"</span><span>))]
</span><span> (set-env! </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">" </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:username </span><span>user
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:password </span><span>pass}])))
</span><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]])
</span></code></pre>
<p>There's still some obvious duplication in there. Let's see if we can
get rid of that too.</p>
<h5 id="build-boot-7"><a href="#build-boot-7" aria-label="Anchor link for: build-boot-7"><i class="fa fa-link" aria-hidden="true" ></i></a>
build.boot</h5>
<pre data-lang="clj" style="background-color:#393939;color:#dedede;" class="language-clj "><code class="language-clj" data-lang="clj"><span>(defn get-cleartext [prompt]
</span><span> (print prompt)
</span><span> (read-line))
</span><span>
</span><span>(defn get-password [prompt]
</span><span> (print prompt)
</span><span> (apply str (.readPassword (System/console))))
</span><span>
</span><span>(require </span><span style="color:#ececec;">'</span><span>[clojure.string </span><span style="font-weight:bold;color:#cc9495;">:as </span><span>str])
</span><span>
</span><span>(defn get-env-or-prompt [prefix prompt-fmt word get-fn]
</span><span> (let [env-name (str prefix word)]
</span><span> (or (System/getenv env-name)
</span><span> (get-fn (format prompt-fmt env-name (str/capitalize word))))))
</span><span>
</span><span>(let [[user pass] (mapv #(get-env-or-prompt </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">DATOMIC_</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">%s was not defined.</span><span style="font-weight:bold;color:#ff8080;">\n</span><span style="color:#d68686;">%s</span><span style="color:#d6d6d680;">"
</span><span> %1 %2)
</span><span> [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">USERNAME</span><span style="color:#d6d6d680;">" "</span><span style="color:#d68686;">PASSWORD</span><span style="color:#d6d6d680;">"</span><span>]
</span><span> [get-cleartext get-password])]
</span><span> (set-env! </span><span style="font-weight:bold;color:#cc9495;">:repositories
</span><span> #(conj % [</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">my-datomic</span><span style="color:#d6d6d680;">" </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:url </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">https://my.datomic.com/repo</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:username </span><span>user
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:password </span><span>pass}])))
</span><span>
</span><span>(set-env!
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:dependencies </span><span style="color:#ececec;">'</span><span>[[com.datomic/datomic-pro </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">0.9.5206</span><span style="color:#d6d6d680;">"</span><span>]])
</span></code></pre>
<p>The code is longer now, but it's been decomposed and de-duplicated
significantly. It also gained the ability to prompt for values only if
the corresponding environment variable isn't set. We could keep going
with this, and define that <code>let</code> block as a function. We could move
all this code into a Clojure source file in the <code>src</code> folder of the
current project, and then <code>require</code> it in. Or we could put it into a
separate library like <a href="https://github.com/radicalzephyr/rotateam">my bootlaces</a> and add that as a
dependency. Once we extract this functionality into a library we could
add tests for it, and then continue to expand it's functionality. We
could add another method for retrieving the credentials. Perhaps
storing them in an encrypted edn file, which we read in if it exists.</p>
<p>All of these various permutations are possible, and more. And we
always have the full power of Clojure at our disposal. Notice what we
didn't have to do at any point along this process. We didn't have to
write a plugin for our build tool, or try to get a patch merged into
the source code and wait for it's release. It's all just been regular
Clojure code, following a very natural and easy code growth
path. Start out with an inline-definition and usage, then slowly
abstract and tease apart into a separate package.</p>
<p>This is the philosophy of Lisp writ large in the paradigm of building
programs. There is no difference between what is built into Boot, and
what we define personally. There is nothing done in the Boot built-in
tasks that could not have been done by a Boot user. Based on a few
carefully chosen "primitives" an elegant and powerful structure can be
built. This is what happens when your code is just data, or your build
is just a program.</p>
The Importance of Focusing Small with TDD2015-07-23T00:00:00+00:002015-07-23T00:00:00+00:00https://zefira.dev/posts/the-importance-of-focusing-small-with-tdd/<p>Since I've been an apprentice at 8th Light, I've been focusing on
really trying to <em>get</em> TDD as a practice and a discipline. As part of
that I've done a lot of reading and reflecting on past reading's about
TDD. One thing that jumped out at me about the way that Martin Fowler
and Kent Beck all talk about TDD: they all tend to say mildly
self-deprecating things like: "I'm just not smart enough to hold all
that complexity in my head. TDD helps me get away with not seeing the
big picture." But I think this statement is fundamentally misleading.</p>
<!--more-->
<p>I was always a little confused when I read statements like that,
because to me the clarity of their writing and thoughts indicate to me
that they are in fact pretty smart. Another way they say it is that
"you don't have to be super smart to do TDD." I think this is a
genuine statement, but it's about marketing TDD than about explaining
how to do it.</p>
<p>When I was writing Tic-Tac-Toe in Ruby during the first month of my
apprenticeship though, I was trying to rigorously test-drive all of
the development in a very intentional way for the first time in my
coding career. As I got into the swing of things, I started noticing
that I was focusing on smaller portions of the code.</p>
<p>Normally when I'm programming, I have something close to
<a href="http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-a-programmer/">this experience</a>. I try and build up an understanding
and mental structure of as much of the program as I can hold in my
head at once. I do this largely because if I have the structure of the
whole program in my head then I know that when I make changes, they're
going to be good ones, and remain consistent with the rest of the
program.</p>
<p>One of the benefits of TDD is that you end up with a
comprehensive test suite for your program as you build it. So instead
of having to build a huge mental model of the software as you are
programming, you can simply make a change and then run the test
suite. This will tell you, with far greater accuracy, whether the
change is consistent with the state of the actual program. There's no
layer of indirection through a potentially inconsistent mental model.</p>
<p>So, having tests allows you not to have a large mental model of the
program you're building. Another (desirable!) consequence of rigorous
test-driving is a well-factored design. If the components in your
system have small, well-defined responsibilities and interact with
other components in reasonable and well-defined ways the system is
simpler. When the system is simpler you can get away with having only
a local view of the code when making changes, so you don't have to be
"as smart."</p>
<p>But here's the thing, these are reasons why you can still program
without a comprehensive understanding of the codebase when practicing
TDD. But the revelation that I had, what caused me to say that Martin
Fowler and company are being misleading, is that I think TDD is actually
<em>easier</em> to do when you let go of trying to build a comprehensive mental
model of the code. This is more true the more of that context you can
hold in your head.</p>
<p>I had a friend in college who was arguably a genius. Assignments that
the rest of our class would take a week to do, he would churn out in a
12-hour Mountain Dew-fueled frenzy of coding. He would intentionally
seek to make assignments more challenging - for example doing a simple
assembly language project, he developed an Object Oriented style of
assembly programming because it would have been boring to him
otherwise.</p>
<p>I ended up working with him on a number of projects though, and I
discovered something. I thought that I had a tendency to overengineer
solutions, but my proclivities in that regard were nothing compared to
my friend's.</p>
<p>One summer we decided to create an AI to play the video game StarCraft
and enter it in a contest at the end of the summer. When we were
planning the architecture of the AI , he immediately started
describing this incredibly elaborate structure, with pluggable modules
that would allow swapping out of different components of the AI, and a
tiered architecture of decision making that would separate the low
level movements of units from the tactical and strategic
decision-making required to win the game.</p>
<p>This structure was impressive and compelling, and we spent weeks
talking about it and sketching diagrams and class hierarchies on the
whiteboard. It would have been helpful to have such a modular
architecture if we had a large team of people working on it, and the
initial meeting seemed to indicate there was a lot of interest. But
after a few weeks, interest in the project waned and most people
drifted away and that grand architecture was never needed and never
built.</p>
<p>From what I've seen, most of the best programmers that I know have
this tendency to over-engineer solutions. From what I've read, this is
not an isolated phenomenon in my social circle.</p>
<p>This type of pure thought design work is actually very similar to
building a mental model of how a piece of software works. It's really
just a change in tense: over-engineering is just building a mental
model of how a piece of software <em>will</em> work, in the future. I'm sure
there is some fascinating psychology behind why this is such a common
tendency. I think what focusing on TDD in this intensive way taught me
is that the smaller your mental model of a piece of software is, the
more restricted your focus is, and the easier it is to refrain from
attempting to over-engineer it.</p>
TDD and the Art of Picking Tests2015-07-19T00:00:00+00:002015-07-19T00:00:00+00:00https://zefira.dev/posts/tdd-and-the-art-of-picking-tests/<blockquote>
<p>The first [problem] is stuckness, a mental stuckness that
accompanies the physical stuckness of whatever it is you're working
on. A screw sticks, for example, on a side cover assembly. You check
the manual to see if there might be any special cause for this screw
to come off so hard, but all it says is "Remove side cover plate" in
that wonderful terse technical style that never tells you what you
want to know. There's no earlier procedure left undone that might
cause the cover screws to stick.</p>
<p>If you're experienced you'd probably apply a penetrating liquid and
an impact driver at this point. But suppose you're inexperienced and
you attach a self-locking plier wrench to the shank of your
screwdriver and really twist it hard, a procedure you've had success
with in the past, but which this time succeeds only in tearing the
slot of the screw.</p>
<p>Your mind was already thinking ahead to what you would do when the
cover plate was off, and so it takes a little time to realize that
this irritating minor annoyance of a torn screw slot isn't just
irritating and minor. You're stuck. Stopped. Terminated. It's
absolutely stopped you from fixing the motorcycle.</p>
<p>-Robert Pirsig, <em>Zen and the Art of Motorcycle Maintenance</em></p>
</blockquote>
<p>There is so much interesting material here it could take a while to
dig into all of it.</p>
<!--more-->
<h2 id="mental-stuckness"><a href="#mental-stuckness" aria-label="Anchor link for: mental-stuckness"><i class="fa fa-link" aria-hidden="true" ></i></a>
Mental Stuckness</h2>
<p>Despite it's total lack of physical tangibility software can get stuck
too, and as software developers when our software gets stuck, so do
we. The canonical example of stuck software is when we need to add a
new feature to our application, but it can't be done without breaking
everything else. Usually we arrive in this situation because we don't
have tests, and so we can't refactor our code. We can only make
changes and then hope that everything still works.</p>
<p>There is another kind of stuckness though that is less specific to
software. The stuckness of a blank page. This is what Pirsig is
talking about in the above quote. Even when you're rigorously
practicing TDD, this kind of stuckness can still affect you. It's most
obvious when starting a new project from scratch. Then it really is
"blank page." But I've noticed a similar feeling when I'm adding a new
<em>type</em> of functionality to an application.</p>
<h3 id="cuttings"><a href="#cuttings" aria-label="Anchor link for: cuttings"><i class="fa fa-link" aria-hidden="true" ></i></a>
Cuttings</h3>
<p>Pirsig spends the remainder of the chapter examining in detail what
stuckness means, and how to deal with it. Many of the</p>
<p>He describes it as the point at which what you think you know about
something has boxed you into a place where you can't conceive of a way
to move forward.</p>
Secure Password Storage2015-07-13T00:00:00+00:002015-07-13T00:00:00+00:00https://zefira.dev/posts/secure-password-storage/<p>At a high-level, when storing secrets like passwords, the standard
best practice is to never store the plain-text version of the password
in the system. Instead, you store some information that is uniquely
and repeatably derived from the password in an irreversible fashion.
This is often called a "digest."</p>
<p>This preserves the ability to check whether someone has entered the
correct password for an account in order to authenticate them (you
simply recalculate the derived value and check those for equality),
but it prevents an attacker who somehow obtains the password digest
from immediately having access to the plain-text version of a user's
password. That's the process at a very high-level.</p>
<!--more-->
<p>Now there are a number of different types of attacks that can be
executed against a web application and a legitimate user to attempt to
intercept the plain text version of a single user's password. The
defenses for these are well known, and all of the countermeasures that
are part of the source code of the app are being used. There are some
further countermeasures (such as usage of https) that are a concern
of how the app is deployed, but we will certainly be implementing
those when we get to the point of production deployment.</p>
<p>This leaves the main attack vector for obtaining user passwords as
somehow obtaining access to the password digests that we store in our
database. In our meeting on Thursday, Paul outlined some of the
deployment choices we can make to help prevent this from ever
occurring, but for illustrative purposes let's assume that an attacker
has compromised our database in some manner and now has access to our
user password digests offline.</p>
<p>It is at this point that a detail I have glossed over becomes very
important: the method that we use to generate the password
digests. The industry standard is to use a key derivation function
(KDF). There are three current industry standard KDF's: PBKDF2,
bcrypt, and scrypt.</p>
<p>All three of these KDF's produce values that cannot be used to derive
the original key (i.e. the plain-text of the user password). This
leaves two avenues of attack: a brute-force search for the original
password plain-text (a "preimage attack"), or a brute-force search for
another password plain-text that produces the same digest (a "collision
attack").</p>
<p>It's important to realize that defending against a brute-force
password search is not a question of making it "impossible." It's
really a question of how expensive it would be, and how long it would
take. The goal in this case is to make it so that the attacker can't
try enough passwords in a "reasonable amount of time" that they will
get a sufficient return on the hardware they need to throw at the
problem, without making password authentication take so long that it
makes users upset.</p>
<p>At a high-level, all three of the industry standard KDF's are tunable
as to how much time they take, so they effectively have equivalent
resistance to a brute-force attack. However, PBKDF2 is a relatively
simple algorithm that can be very efficiently computed in specialized
hardware, or on commercial graphics cards. This makes it
substantially weaker than the other two because graphics cards and
specialized hardware can increase the number of passwords that an
attacker can test per unit of time.</p>
<p>To quote an piece of writing I found about bcrypt:</p>
<blockquote>
<p>Bcrypt has the best kind of repute that can be achieved for a
cryptographic algorithm: it has been around for quite some time,
used quite widely, "attracted attention", and yet remains unbroken
to date.</p>
</blockquote>
<p>Recently, advances in hardware have made bcrypt somewhat vulnerable in
a similar manner to PBKDF2. Scrypt was designed to be tunable in both
speed and memory usage. This tunable memory usage makes scrypt far
less susceptible to these specialized hardware attacks.</p>
<p>In general though these types of hardware attacks come at significant
cost, so for many applications tuned usage of PBKDF2 or bcrypt is more
than adequate. Using scrypt also carries another form of risk. Since
it is a much newer algorithm it hasn't had time to be subjected to the
same analysis and attack attempts as PBKDF2 and bcrypt. This means
that it is possible for a security vulnerability to be discovered in
scrypt at some point in the future that makes it significantly less
secure than the other two.</p>
Domain Driven Writing2015-07-03T00:00:00+00:002015-07-03T00:00:00+00:00https://zefira.dev/posts/domain-driven-writing/<p>I've been reading <em>Domain Driven Design</em> by Eric Evans for the past
month or so, and thinking about the ideas and concepts in it has been
consuming a lot of my mental space and time. One thing in particular
that I've found to be really interesting is his concept of developing
a <strong>Ubiquitous Language</strong> for the project.</p>
<p>At a very high level, the idea of having a Ubiquitous Language for a
project just means that there is specific vocabulary and jargon that
is used to describe certain key concepts about the domain of the
project and the particular implementation that the team is
developing. Or to put it even more simply, that when someone on the
team uses a word or a phrase, everyone else knows what they are
talking about.</p>
<!--more-->
<p>This is actually something that people do constantly. You build up a
common language with the people you are closest to: your family,
friends, teammates at work. I know that my group of close friends has
a very particular way of speaking with one another, that probably
sounds somewhat odd to "outsiders."</p>
<p>Sometimes it happens organically over a long period of time, and
sometimes we do it intentionally. My favorite example of this second
category are the talks that Rich Hickey gives. Some of them at least
have followed a particular format. He starts with the definition of a
word, possibly two. He defines them precisely, and he defines the
context that he is talking within. He then builds on this definition
to talk about some fundamental idea in programming, or some insight
that this rigorous definition gave him.</p>
<p>The key part though, is that a good portion of his talk is given to
building up a sort of basic common language so that he is confident
that when he uses a word, we understand as well as we possibly can
what he intended by saying it.</p>
<p>A good friend of mine often has a very hard time with this. I have had
many conversations with him about interesting and deep topics that
have turned into arguments, largely about what a particular word
means. The problem is often that he has a deeper context and specific
meaning that he intends when he says something. But because I haven't
studied the same things or shared the same experiences, I don't
realize he means something different than usual.</p>
<p>I wish I had a concrete example that I could talk about, because this
sounds very wish-washy.</p>
<h3 id="a-slight-digression"><a href="#a-slight-digression" aria-label="Anchor link for: a-slight-digression"><i class="fa fa-link" aria-hidden="true" ></i></a>
A Slight Digression</h3>
<p>I'm strangely reminded of the final program from Mark Pilgrim's
excellent - though sorely out-dated - <em>Dive Into Greasemonkey</em>. He
spends the first several tens of lines of code writing some facilities
that will make programming Javascript more reminiscent of programming
in Python because he's so familiar with Python.</p>
<p>One thing that Eric Evans returns to throughout <em>DDD</em> is leaning on
the <em>Ubiquitous Language</em> of the project to suss out design issues
early. He emphasizes that we should try and engage all of our ways of
communicating and understanding the ideas of the project: with
language, with diagrams, and with code.</p>
<p>Uncle Bob has written at length about how (good) tests are really
specifications for how a program works. They should clearly
communicate the intent of the system.</p>
<p>Gerald Sussman and Hal Abelson write in the preface to their classic
<em>Structure and Interpretation of Computer Programs</em> that they believe
that programming is not just an activity concerned with causing
computers to enact computations. It is much more importantly "a novel
formal medium for expressing ideas about process."</p>
<p>These two ideas, tests as specifications, and computer languages as
formal language describing processes have started to seem intimately
connected to me.</p>
<h3 id="automation-and-communication"><a href="#automation-and-communication" aria-label="Anchor link for: automation-and-communication"><i class="fa fa-link" aria-hidden="true" ></i></a>
Automation and Communication</h3>
<p>Several years ago I was working on a project to build a video game AI
with several other students at my university. Another student and I
were pioneering the project, and in particular I took on the task of
figuring out how to get the development environment working on the
Computer Science lab machines.</p>
<p>This setup was non-trivial. It involved setting environment variables,
setting compatibility flags on the game executable, and installing
plugins into the game's binary directory. On top of that, but because
of the way the lab machines were setup, there was additional setup
that needed to be done at the start of <em>every development session</em>.</p>
<p>Once I had gotten everything working through a long process of trial
and error, I then set about attempting to duplicate my success from a
clean profile. As I figured out the minimal set of setup steps, I
wrote them down. I then proceeded to test the quality of these
instructions by handing them off to another student to replicate with
me watching over his shoulder.</p>
<p>You will of course be shocked to learn that he had almost no success
in following the steps I had written down. This is because I had
written them in enough detail for me to understand and remember
what needed to be done, but there was still a huge amount of context
for those instructions that resided only in my own head. Context that
I head gained by the laborious trial, error and troubleshooting
process of setting the environment up initially.</p>
<p>I iterated on the instructions several more times, each time finding
some new guinea pig to test them on. Until eventually, I came to the
conclusion that I would actually need pictures of the GUI dialogs and
full nearly click-by-click instructions in order to get a set of
instructions that other students could follow without making mistakes.</p>
<p>At that moment, I realized that I would be far less work, both
up-front and continuing to simply figure out how to write code to
automate the development environment setup.</p>
<p>This seems like a no-brainer. Write two scripts one time, spend a few
hours figuring out the nuances of scripting Windows and then tell
people "Download those two scripts. Run this one once, then run that
one every time you want to work on the project." Done.</p>
<p>The problem for me is the same problem that a math tutor has with
simply telling the student the formula they need to use to calculate
the answer. There is a <em>lot</em> of important context that is missing.</p>
<p>My setup script was incredibly effective. Not only did it flawlessly
setup up the dev environment, but it was at least an order of
magnitude faster than for a human to accomplish the same tasks.</p>
<p>That small script I wrote, expressed the idea of how to setup that
development environment incredibly precisely. But there was now
something missing. There was no context for that process to fit in
to. The why of it was lost, and all that was left was for people to
run it.</p>
<p>Even reading the contents of the script, it was possible to determine
what it did, but the <em>why</em> was completely missing.</p>
<p>I think testing, at it's very best, can start to supplement this
deficiency in code. It can help to answer the question "Yes, it
works. But why does it do it <em>that</em> way?"</p>
<h4 id="p-s"><a href="#p-s" aria-label="Anchor link for: p-s"><i class="fa fa-link" aria-hidden="true" ></i></a>
p.s.</h4>
<p>Some interesting papers that are somewhat related to this post:</p>
<ul>
<li><a href="https://www.recurse.com/blog/58-paper-of-the-week-notation-as-a-tool-of-thought">Notation as a Tool of Thought</a></li>
<li><a href="https://www.recurse.com/blog/41-introducing-paper-of-the-week">Growing A Language</a></li>
</ul>
The Many Faces of the Repository Pattern2015-06-29T00:00:00+00:002015-06-29T00:00:00+00:00https://zefira.dev/posts/the-many-faces-of-the-repository-pattern/<p>Last week I spent quite a lot of time working closely with my a fellow
apprentice on refactoring a Rails application away from direct usage
of ActiveRecord for data persistence, and towards some incarnation of
the Repository Pattern. Now that we've finished, I thought some
reflection on the different possible implementations that we
considered, and what we ended up implementing might be in order.</p>
<!--more-->
<h2 id="what-is-a-repository"><a href="#what-is-a-repository" aria-label="Anchor link for: what-is-a-repository"><i class="fa fa-link" aria-hidden="true" ></i></a>
What is a "Repository"?</h2>
<p>First of all, let's talk briefly about the Repository Pattern
itself. My first exposure to the idea was through the discussion of it
in "Domain Driven Design" by Eric Evans. Martin Fowler has
<a href="http://martinfowler.com/eaaCatalog/repository.html">a page on Repositories</a> where he summarizes the pattern
with this description:</p>
<blockquote>
<p>Mediates between the domain and data mapping layers using a
collection-like interface for accessing domain objects.</p>
</blockquote>
<p>This is a good high-level summary of what Eric Evans describes as
well. Broadly speaking, the idea is that you use the Repository
Pattern to thoroughly isolate your domain layer from whatever
strategies are used to persist the domain data. Evans offers this
description of the problem that Repositories help to solve:</p>
<blockquote>
<p>A subset of persistent objects must be globally accessible through a
search based on object attributes. ... Providing access to other
objects muddies important distinctions. Free database queries can
actually breach the encapsulation of domain objects and
AGGREGATES. Exposure of technical infrastructure and database access
mechanisms complicates the client and obscures the MODEL-DRIVEN
DESIGN.</p>
</blockquote>
<p>He further explains that:</p>
<blockquote>
<p>A repository lifts a huge burden from the client, which can now talk
to a simple, intention-revealing interface, and ask for what it
needs in terms of the model. To support all this requires a lot of
complex technical infrastructure, but the interface is simple and
conceptually connected to the domain model.</p>
</blockquote>
<p>Finally, he lists the advantages that using Repositories confers:</p>
<ul>
<li>They present clients with a simple model for obtaining persistent
objects and managing their life cycle.</li>
<li>They decouple application and domain design from persistence
technology, multiple database strategies, or even multiple data
sources.</li>
<li>They communicate design decisions about object access.</li>
<li>They allow easy substitution of a dummy implementation, for use in
testing (typically using an in-memory collection).</li>
</ul>
<p>For someone seeking to implement the Repository Pattern in their own
project, this list serves as an excellent barometer of the quality of
their implementation.</p>
<h2 id="my-repository"><a href="#my-repository" aria-label="Anchor link for: my-repository"><i class="fa fa-link" aria-hidden="true" ></i></a>
My Repository</h2>
<p>A few weeks ago I started working on a new client project. It was a
greenfield application, and we started building it in Ruby on Rails,
using the basic Rails application structure, and all of the built-in
Rails Magic â„¢. This meant a standard MVC architecture,
ActiveRecord models, the whole nine yards.</p>
<p>We embraced the "Rails Way" largely to give the project a
jump-start. We were pretty confident that eventually we'd start
feeling the pain of using Rails and want to head towards some sort of
hybrid Rails architecture.</p>
<p>A few weeks ago, we got to that point and started looking at
implementing a repository for the new domain object we were about to
introduce. Unfortunately, I didn't return to the discussion from the
DDD book referenced earlier, and we basically started to feel our way
into implementing the Repository pattern based on my teammates prior
experience, what we generally knew about the pattern and some
interesting examples we found on blog posts regarding using the
Repository pattern with Rails' ActiveRecord technology.</p>
<p>We ended up with something that looked basically like this:</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#fed6af;">module </span><span style="font-weight:bold;color:#ff8000;">Notes
</span><span> </span><span style="color:#fed6af;">class </span><span style="color:#f4a020;">Repository
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">initialize</span><span>(dao </span><span style="color:#ececec;">= </span><span style="color:#d6d6ae;">Note</span><span>)
</span><span> </span><span style="color:#fed6af;">@dao </span><span style="color:#ececec;">=</span><span> dao
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">new</span><span>(attributes)
</span><span> </span><span style="color:#f4a020;">Notes</span><span>::</span><span style="color:#f4a020;">Note</span><span>.</span><span style="color:#fed6af;">new</span><span>(attributes)
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">find_by_id</span><span>(id)
</span><span> record </span><span style="color:#ececec;">= </span><span style="color:#fed6af;">@dao</span><span>.find_by_id(id)
</span><span> </span><span style="color:#f4a020;">Notes</span><span>::</span><span style="color:#f4a020;">Note</span><span>.</span><span style="color:#fed6af;">new</span><span>(record.attributes)
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">save</span><span>(note)
</span><span> </span><span style="color:#fed6af;">if</span><span> note.valid?
</span><span> record </span><span style="color:#ececec;">= </span><span style="color:#fed6af;">@dao</span><span>.</span><span style="color:#fed6af;">new</span><span>(note.attributes).save
</span><span> note.id </span><span style="color:#ececec;">=</span><span> record.id
</span><span> </span><span style="font-weight:bold;color:#d6d6ae;">true
</span><span> </span><span style="color:#fed6af;">else
</span><span> </span><span style="font-weight:bold;color:#d6d6ae;">false
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span style="color:#fed6af;">end
</span></code></pre>
<p>This looks pretty similar to the interface that ActiveRecord gives
us. If we go back to that list of benefits that repositories are
supposed to confer, it's not immediately obvious that this
implementation is really going to give us any of those, except
possibly the ability to swap database implementations.</p>
<p>But what are we to do instead? Well, as it turns out waiting a few
days provided an answer. Over the weekend one of the Craftsmen at 8th
Light took the initiative to toss our codebase headfirst into a
slightly different repository implementation. This was a tricky
prospect for a codebase that was fairly coupled to the use of
ActiveRecord, but once we got everything figured out we ended up with
something that looked sort of like this:</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#fed6af;">module </span><span style="font-weight:bold;color:#ff8000;">Repository
</span><span> </span><span style="color:#fed6af;">class </span><span style="color:#f4a020;">RepositoryBase
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">initialize</span><span>(db)
</span><span> </span><span style="color:#fed6af;">@db </span><span style="color:#ececec;">=</span><span> db
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">create</span><span>(attributes)
</span><span> convert(repository.create(attributes))
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">find_by_id</span><span>(id)
</span><span> convert(repository.find_by_id(id.</span><span style="color:#fffd87;">to_i</span><span>))
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">all
</span><span> repository.all.map { |e| convert(e) }
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">count
</span><span> repository.all.count
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">convert</span><span>(data)
</span><span> </span><span style="color:#fed6af;">@db</span><span>.adapter.convert(model_class, data)
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">model_class
</span><span> </span><span style="color:#fed6af;">raise </span><span style="color:#d6d6ae;">NotImplementedError
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">repository
</span><span> </span><span style="color:#fed6af;">raise </span><span style="color:#d6d6ae;">NotImplementedError
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span style="color:#fed6af;">end
</span><span>
</span><span style="color:#fed6af;">module </span><span style="font-weight:bold;color:#ff8000;">Notes
</span><span> </span><span style="color:#fed6af;">class </span><span style="color:#f4a020;">Repository </span><span>< </span><span style="color:#d78d1b;">Repository::RepositoryBase
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">find_all_by_user_id</span><span>(user_id)
</span><span> repository.where(</span><span style="font-weight:bold;color:#cc9495;">user_id:</span><span> user_id).map </span><span style="color:#fed6af;">do </span><span>|note|
</span><span> convert(note)
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">model_class
</span><span> </span><span style="color:#f4a020;">Notes</span><span>::Note
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">repository
</span><span> </span><span style="color:#fed6af;">@db</span><span>.notes
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span style="color:#fed6af;">end
</span></code></pre>
<p>This actually looks substantially similar, and in some ways bears even
more resemblance to an ActiveRecord API.</p>
<p>Since it's come up a few times, let's consider the differences between
ActiveRecord and the Repository pattern. From the Rails Guide on
ActiveRecord:</p>
<blockquote>
<p>In Active Record, objects carry both persistent data and behavior
which operates on that data. Active Record takes the opinion that
ensuring data access logic as part of the object will educate users
of that object on how to write to and read from the database.</p>
</blockquote>
<p>So the essential feature of ActiveRecord is that the Model class
represents both the domain model, and the data model. Contrast this
with one of the primary features of the Repository Pattern as
described by Eric Evans:</p>
<blockquote>
<p>[The repository] Mediates between the domain and data mapping layers
using a collection-like interface for accessing domain objects.</p>
</blockquote>
<p>Instead of combining domain and data access into the same object, the
Repository gives us an explicit boundary to partition our system
on. On one side, we can hide the often significant technical
complexity of storing and retrieving data. On the other side we can be
concerned solely with domain objects and their behaviour.</p>
<p>So it's not really the difference in API that we're looking for, it's
the architecture surrounding that API. In particular, simply by having
a separate Repository object we've started to achieve that separation
between data and domain. So despite the fact that our Repositories
don't try to fully emulate a collection of domain objects we've
actually managed to achieve one of the primary goals of the Repository
Pattern.</p>
<h2 id="taking-it-further"><a href="#taking-it-further" aria-label="Anchor link for: taking-it-further"><i class="fa fa-link" aria-hidden="true" ></i></a>
Taking it Further</h2>
<p>I think the most important question to ask when talking about
technical infrastructure is "What does it buy me?" In this case, what
might we be able to do now that we use repositories everywhere that
would have been difficult or impossible before? It just so happens
that one of the next features on our list was implementing users in
the system having a balance of some kind.</p>
<p>We decided that in the domain, this could naturally be represented by
each user having an Account. This <a href="https://en.wikipedia.org/wiki/Indirection">indirection</a> conveniently
gives us the ability to potentially model transfers to non-User
entities. But how would we actually store the data associated with an
Account? The most obvious (and somewhat naive) way would be to store a
simple integer balance for each Account. As an ActiveRecord migration
it would probably look something like this:</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span>create_table </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">accounts</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">force: :cascade </span><span style="color:#fed6af;">do </span><span>|t|
</span><span> t.integer </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">balance</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">null: </span><span style="font-weight:bold;color:#d6d6ae;">false
</span><span> t.integer </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">user_id</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">null: </span><span style="font-weight:bold;color:#d6d6ae;">false
</span><span style="color:#fed6af;">end
</span></code></pre>
<p>But a common model for recording balances in the domain of accounting
is typified by double entry book-keeping. In this model the sum of all
debits and credits must always be equal, and the balance for a
particular account can be calculated by the sum of debits and credits
associated with that account. Again, in ActiveRecord that might look
something like this:</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span>create_table </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">accounts</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">force: :cascade </span><span style="color:#fed6af;">do </span><span>|t|
</span><span> t.integer </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">user_id</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">null: </span><span style="font-weight:bold;color:#d6d6ae;">false
</span><span style="color:#fed6af;">end
</span><span>
</span><span>create_table </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">transactions</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">force: :cascade </span><span style="color:#fed6af;">do </span><span>|t|
</span><span> t.integer </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">account_id</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">null: </span><span style="font-weight:bold;color:#d6d6ae;">false
</span><span> t.integer </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">amount</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="font-weight:bold;color:#cc9495;">null: </span><span style="font-weight:bold;color:#d6d6ae;">false
</span><span style="color:#fed6af;">end
</span></code></pre>
<p>If we've implemented a successful Repository, it should be possible
for us to write the Repository using the naive strategy, develop
client code that utilizes the Repository, and then transition the
Repository to use the more sophisticated transaction scheme without
making any changes to the clients.</p>
<p>The first key element for implementing the accounts repository is
going to be an Account domain object. Ignoring how it gets created for
the moment, here's a basic implementation.</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#fed6af;">module </span><span style="font-weight:bold;color:#ff8000;">Accounts
</span><span> </span><span style="color:#fed6af;">class </span><span style="color:#f4a020;">Account
</span><span> </span><span style="color:#fed6af;">attr_reader </span><span style="font-weight:bold;color:#cc9495;">:id</span><span>, </span><span style="font-weight:bold;color:#cc9495;">:balance</span><span>, </span><span style="font-weight:bold;color:#cc9495;">:user_id
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">deposit</span><span>(amount)
</span><span> </span><span style="color:#fed6af;">@balance </span><span style="color:#ececec;">+=</span><span> amount
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">transfer_to</span><span>(account, amount)
</span><span> </span><span style="color:#fed6af;">if </span><span style="color:#ececec;">!</span><span>has_funds?(amount)
</span><span> </span><span style="color:#fed6af;">raise </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">Balance too low</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="color:#fed6af;">else
</span><span> </span><span style="color:#fed6af;">@balance </span><span style="color:#ececec;">-=</span><span> amount
</span><span> account.deposit(amount)
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">has_funds?</span><span>(amount)
</span><span> </span><span style="color:#fed6af;">@balance </span><span style="color:#ececec;">>=</span><span> amount
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">class </span><span style="color:#f4a020;">Repository </span><span>< </span><span style="color:#d78d1b;">Repository::RepoBase
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">create_for</span><span>(user)
</span><span> account </span><span style="color:#ececec;">=</span><span> create({</span><span style="font-weight:bold;color:#cc9495;">user_id:</span><span> user.id})
</span><span> account
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">find_by_user</span><span>(user)
</span><span> convert(</span><span style="color:#fed6af;">@db</span><span>.accounts.find_by_user_id(user.id))
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">model_class
</span><span> </span><span style="color:#f4a020;">Accounts</span><span>::Account
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span style="color:#fed6af;">end
</span></code></pre>
<p>Notice that this version of the account uses the same naive model of
storing a simple balance. We could rewrite this domain level code to
store a list of transactions as an array of hashes, and then collapse
the list of transactions to the new effective balance when saving the
object. So already, we can see that separating the concerns of data
storage and domain behaviour has achieved some level of isolation.</p>
<p>Now let's imagine we change the database implementation so there is no
longer any <code>db.accounts</code> object. Instead, we have <code>db.transactions</code>
which has methods like <code>find_all_transactions_for_account_id</code>.</p>
<p>Again ignoring details of how the transactions are created or saved:</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#fed6af;">module </span><span style="font-weight:bold;color:#ff8000;">Accounts
</span><span> </span><span style="color:#fed6af;">class </span><span style="color:#f4a020;">Repository </span><span>< </span><span style="color:#d78d1b;">Repository::RepoBase
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">create_for</span><span>(user)
</span><span> account </span><span style="color:#ececec;">=</span><span> create({</span><span style="font-weight:bold;color:#cc9495;">user_id:</span><span> user.id})
</span><span> account
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">find_by_user</span><span>(user)
</span><span> transactions </span><span style="color:#ececec;">= </span><span style="color:#fed6af;">@db</span><span>.transactions.find_by_user_id(user.id)
</span><span> balance </span><span style="color:#ececec;">=</span><span> transactions.reduce(</span><span style="font-weight:bold;color:#87d6d5;">0</span><span>) { |r,t| r </span><span style="color:#ececec;">+</span><span> t.amount }
</span><span> convert({</span><span style="font-weight:bold;color:#cc9495;">user_id:</span><span> user.id, </span><span style="font-weight:bold;color:#cc9495;">balance:</span><span> balance})
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">model_class
</span><span> </span><span style="color:#f4a020;">Accounts</span><span>::Account
</span><span> </span><span style="color:#fed6af;">end
</span><span> </span><span style="color:#fed6af;">end
</span><span style="color:#fed6af;">end
</span></code></pre>
<p>This was a very minor change, given that under the covers data is
being stored in an entirely different manner. Even better, the change
is totally isolated to the repository. Because of the original naive
implementation of the Account domain object, some changes will need to
be made to the way it handles transfers. This may look like the
details of the data storage are leaking into the domain, but in fact
this is a data requirement bubbling up. We can't store transactions if
we don't record them in the first place. And realistically, we're
probably going to need the concept of a Transaction to be added to the
domain model.</p>
<p>You may then find yourself wondering, "What value do we really get
from this separation if both sides of the boundary are going to need
to change anyhow?" The answer I think is two things: separation of
concerns and incremental changes. The details of how we handle
transactions in the domain layer are going to be different from the
details needed to handle storing data in the database. This separation
of concerns allows us to focus on the technical complexity of data
storage when writing that code, and to focus on the business
complexity when dealing with the domain code instead of mixing the two
together.</p>
<p>Second, this separation layer provides us an abstraction layer that we
can use to isolate changes to our code. Earlier I demonstrated how the
domain layer could stay the same even though storage had moved to a
transaction model. It's not too hard to imagine changing the domain
model to track transactions using pure Ruby data structures and having
the repository still store accounts with an explicit balance.</p>
<p>This separation layer then gives us a nice stepping stone to move from
one implementation to the other. This stepping stone is important
because it's a place where we should be able to stop making code
changes and have all of our tests pass. To me, being able to make real
changes in this stepwise manner is one of the hallmarks of
well-factored and decoupled code.</p>
Estimating is Hard2015-05-26T00:00:00+00:002015-05-26T00:00:00+00:00https://zefira.dev/posts/estimating-is-hard/<p>I'm coming up on the end of my first iteration at 8th Light where I
was asked to estimate how long each of my stories was going to
take. For the three stories that make up the majority of converting my
rack Tic-Tac-Toe from a multi-page vanilla HTML/CSS app, to a SPA
(Single Page App) that uses AJAX to communicate with the server I came
up with a total of about 11 points. Given that at 8th Light we want to
be nominally doing about 9 points per week that meant that I estimated
it would take me just a touch over one iteration to complete that
transformation. But looking at it right now, I don't think I'm on
track to meet that goal.</p>
<!--more-->
<p>Estimating is acknowledged to be hard. The advice from <em>The Pragmatic
Programmer</em> acknowledges that and says that basically the only way to
get better at it is to try and estimate things and then track how
(in)accurate your estimates are. Beyond that general acknowledgement
though, I think I have some personal habits/quirks that make my
estimates less than good.</p>
<p>First, I have a tendency to want to seem like I know what I'm doing,
even when I don't. This is obviously problematic for estimation since
in it's simplest form it might make me underestimate how long I think
something will take so I look more proficient. There's a more
insidious way that this attitude causes me problems though. It means I
have a tendency not to ask questions, and I probably tend to bang my
head against things that aren't working for too long before I ask for
help.</p>
<p>The second problem I have is reading things and ensuring I actually am
comprehending them. This is sort of related to the first problem since
I think it comes from a similar place. But at this point it's more of
a mental habit than an explicit choice. For instance, even though I
didn't have a short time limit to figure out my estimates, I still did
them fairly quickly and without really having a clear picture of what
work I would have to do for each story. As soon as I started
trying to implement the stories, I realized this.</p>
<h3 id="the-takeaway"><a href="#the-takeaway" aria-label="Anchor link for: the-takeaway"><i class="fa fa-link" aria-hidden="true" ></i></a>
The Takeaway</h3>
<p>I need a more systematic approach to the way I come up with
estimates. First off, I'm going to go re-read the article on it from
<em>The Pragmatic Programmer</em>. As I recall it has some excellent advice
that I don't remember half well enough. Second, I think I'm going to
try to partially adopt a strategy that my co-mentor Zach suggested
when I was reviewing a code-base's "viability for extension." His
advice was to walk through the code while considering what it would
take to make some sort of reasonable change.</p>
<p>This is quite nebulous for a large code-base, since I first have to
decide what "a reasonable change" might be. But as a tool for reifying
the goals that I'm trying to estimate it should work very
nicely. Basically, all that this means is that I will actually
seriously try to think through what I would do to implement the story
I'm trying to estimate instead of just throwing a number out from my
gut feeling.</p>
Domain Driven Design and Clojure2015-05-21T00:00:00+00:002015-05-21T00:00:00+00:00https://zefira.dev/posts/domain-driven-design-and-clojure/<p>I've been reading an excellent book recently: <a href="https://en.wikipedia.org/wiki/Domain-driven_design"><em>Domain Driven Design</em></a>
by Eric Evans. I just finished the section on Entities and Value
Objects, and it's caused me to start thinking about Clojure and the
"sane state management model" that language advocates/enforces.</p>
<!--more-->
<p>The Clojure state model bears some strong similarities to what Evans
talks about when discussing the needs and uses for Entities and Value
Objects. Entities he says should be used to represent things where
their continuity of identity is more important than their current
value. Value Objects are the opposite, the important part of a Value
Object is the data it holds or represents, to the extent that Value
Objects with the same value are totally interchangeable.</p>
<p>He goes on to talk about the benefits that immutability and safe
arbitrary sharing that this interchangeability can give you.
Specifically, the relative looseness of the constraints on Value
Objects gives a system designer more freedom to implement Value
Objects in a way that has extra benefits, such as increased
performance or memory efficiency.</p>
<p>I'm just going to pull some quotes directly from
<a href="http://clojure.org/state">the page on clojure.org about state</a>.</p>
<p>First, some words about identity:</p>
<blockquote>
<p>While some programs are merely large functions, e.g. compilers or
theorem provers, many others are not - they are more like working
models, and as such need to support what I'll refer to in this
discussion as identity. By identity I mean a stable logical entity
associated with a series of different values over time. Models need
identity for the same reasons humans need identity - to represent
the world. How could it work if identities like 'today' or 'America'
had to represent a single constant value for all time?</p>
</blockquote>
<p>And now, values:</p>
<blockquote>
<p>So, for this discussion, an identity is an entity that has a state,
which is its value at a point in time. And a value is something that
doesn't change. 42 doesn't change. June 29th 2008 doesn't
change. Points don't move, dates don't change, no matter what some
bad class libraries may cause you to believe. Even aggregates are
values. The set of my favorite foods doesn't change, i.e. if I
prefer different foods in the future, that will be a different set.</p>
</blockquote>
<p>Clearly Rich Hickey was thinking along the same lines as Eric
Evans. In fact, I would say that Clojure is a programming language
that is built on a firm conceptual model of the domain of
programming. I think this is one of the things that makes Clojure such
a joy to program in. Especially once you're familiar with the
language, the standard idioms and the general layout of it's
libraries. Everything is highly conceptually consistent. There have
been very few points where I learned something new about Clojure and
thought "Well that doesn't really make sense..." More often when I
learn something new it fits smoothly into the mental model that I
already have about how the language works.</p>
<p>I guess this Domain Driven Design stuff might really work...</p>
Learning Rack2015-05-19T00:00:00+00:002015-05-19T00:00:00+00:00https://zefira.dev/posts/learning-rack/<p>For the past two weeks or so I've been grappling with trying to build
a simple HTTP front-end for my ruby Tic-Tac-Toe program. There have
been some non-technical hurdles with scheduling and other projects
demanding my time, but I've also spent a fair amount of time simply
struggling to put the pieces together with how the Rack libraries
work.</p>
<!--more-->
<p>Rack as a concept is exceptionally simple. Inspired by <a href="https://www.python.org/dev/peps/pep-0333/">PEP 333</a>
which specified the Python Web Server Gateway Interface or WSGI, Rack
has come to be the de-facto way that Ruby applications interface with
web servers. Beyond the <a href="http://rack.github.io/">basic concept</a>, Rack provides a set of
helper libraries. These include many useful middleware and convenience
classes for things like static file serving, sessions, and request
parsing and response generation. However, the documentation for the
classes included in Rack is pretty dismal.</p>
<p>For the most part it is all auto-generated documentation, but many of
the classes have only a vague description of what they do, with no
indication of how they're meant to be used. Conspicuously absent or
good examples of configuration of middleware, especially in the
special DSL that is made available in <a href="https://github.com/rack/rack/wiki/%28tutorial%29-rackup-howto">rackup</a> files.</p>
<p>Given the number of different web frameworks that are built on top of
Rack, it's sort of shocking to me that the documentation is in this
sort of state. Though I guess it's consistent with my general
experience of Ruby libraries as a whole.</p>
<p>I've really worked in Ruby since I started my apprenticeship at 8th
Light, but from what I've seen, many Ruby libraries have awkward gaps
in their documentation. For instance, Guard is an amazing and vital
piece of a smooth Ruby TDD toolchain. But it's documentation is
incredibly cryptic about how to properly setup a Guardfile. There are
more paragraphs about how to debug problems in your Guardfile than
what the format is!</p>
<p>Now documentation is hard. But over the last several years, it's been
increasingly clear to me that <a href="https://jacobian.org/writing/great-documentation/">good documentation</a> is one of
the most valuable assets that a software product can have, especially
an open-source one.</p>
<p>So then why is writing good documentation still so elusively
difficult? I know I struggle with it immensely. The majority of
<a href="https://github.com/RadicalZephyr?tab=repositories">my projects</a> have little to no documentation, with the notable
exception of <a href="https://github.com/RadicalZephyr/hermit">Hermit</a>.</p>
<p>I think there are a few reasons. First off, writing in general is
hard. It takes effort, and the will to keep re-writing and trying new
things to end up with a nicely polished piece of writing that reads
well, and communicates the author's intent clearly and succinctly.</p>
<p>Another reason is that it's much easier to polish a piece of writing
with good critical feedback from someone else. The complicating factor
for code documentation is that you need to find the right audience to
give you feedback. Often this means someone who is totally unfamiliar
with your exact software, but if it's code documentation it probably
also can't be your friend who only uses their computer for writing
Word documents and checking their social media.</p>
<p>But I think the final reason is that we don't do it enough. Again,
like all other writing, the best way to get better at it, is simply by
writing lots and lots of documentation. Also, critically reading the
documentation that you read, and reflecting on what is helpful, and
what isn't.</p>
Chestnut - Annotated2015-05-10T00:00:00+00:002015-05-10T00:00:00+00:00https://zefira.dev/posts/chestnut-annotated/<h4 id="a-note-about-version-numbers"><a href="#a-note-about-version-numbers" aria-label="Anchor link for: a-note-about-version-numbers"><i class="fa fa-link" aria-hidden="true" ></i></a>
A Note about version numbers</h4>
<p>At the time of this writing, there were several newer versions of many
of the dependencies used in Chestnut. In particular, the ClojureScript
core team had fairly recently released a new version with
<a href="https://github.com/clojure/clojurescript/wiki/Running-REPLs">vastly simplified REPL setup requirements</a> which
triggered changes to many of the related ClojureScript tooling
libraries (Piggieback, and Weasel especially). So PLEASE! Use the
latest version of Chestnut, or if you're setting up your own project
then look up the latest versions on <a href="http://clojars.org/">Clojars</a></p>
<h2 id="on-to-the-annotating"><a href="#on-to-the-annotating" aria-label="Anchor link for: on-to-the-annotating"><i class="fa fa-link" aria-hidden="true" ></i></a>
On to the Annotating!</h2>
<p>As I wrote [recently][yaks], I recently dove head first into doing web
development with Clojure and Clojurescript. Along the way I learned a
whole heck of a lot about how to actually set up a <a href="http://leiningen.org/">Leiningen</a>
project to support a nice workflow for such a project. However, most
of my new-found knowledge has already been put together into a very
nice package called <a href="https://github.com/plexus/chestnut">Chestnut</a>. However, at first glance
(and second and third glance really) Chestnut projects are complex and
intimidating.</p>
<p>This post is an annotated walkthrough of the configuration that a new
Chestnut comes with. The best way to follow along would be to start
off by running <code>lein new chestnut tour</code> in shell and then exploring
the files as I talk about them.</p>
<p>[yaks]: {% post_url 2015-04-04-shaving-the-clojurescript-yaks %}</p>
<!--more-->
<p>Let's start by looking at the base directory.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span># Directories
</span><span>- resources
</span><span>- src
</span><span>- env
</span><span>
</span><span># Files
</span><span>- LICENSE
</span><span>- README.md
</span><span>- code_of_conduct.md
</span><span>- project.clj
</span><span>- Procfile
</span><span>- system.properties
</span></code></pre>
<p>If you're familiar with Clojure development at all, this should look
very familiar to you. But there a few unusual things: the <code>Procfile</code>
and <code>system.properties</code> files, and the <code>env</code> folder. The <code>Procfile</code>
is a file for letting <a href="https://www.heroku.com/">Heroku</a> know how to run your app, and the
contents of it are explained well in the
<a href="https://devcenter.heroku.com/articles/getting-started-with-clojure#define-a-procfile">Heroku guide to getting started with Clojure</a>.</p>
<p>If we use the awesome [tree][tree] program to visualize some of these
directories we can see some interesting stuff. First, <code>src</code> looks
pretty standard for Clojure, with the addition of a second tree for
cljs files.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>src
</span><span>├── clj
</span><span>│ └── tour
</span><span>│ └── server.clj
</span><span>└── cljs
</span><span> └── tour
</span><span> └── core.cljs
</span></code></pre>
<p>The same is true of the <code>test</code> folder. But what's really interesting
is what is in that new <code>env</code> folder.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>env
</span><span>├── dev
</span><span>│ ├── clj
</span><span>│ │ └── tour
</span><span>│ │ └── dev.clj
</span><span>│ └── cljs
</span><span>│ └── tour
</span><span>│ └── main.cljs
</span><span>├── prod
</span><span>│ ├── clj
</span><span>│ │ └── tour
</span><span>│ │ └── dev.clj
</span><span>│ └── cljs
</span><span>│ └── tour
</span><span>│ └── main.cljs
</span><span>└── test
</span><span> ├── js
</span><span> │ ├── polyfill.js
</span><span> │ └── unit-test.js
</span><span> └── unit-test.html
</span></code></pre>
<p>What we see is that there are three top-level directories underneath
<code>env</code>, and below each of these we see what looks like normal Clojure
and Clojurescript source trees. Veeery, interesting.</p>
<p>Now let's jump and take a look at the <code>project.clj</code>. This is where the
heart of the action is. The first is boring normal project meta-data,
and we're going to skip it. The next section is interesting though.</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span> </span><span style="font-weight:bold;color:#cc9495;">:source-paths </span><span>[</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">src/clj</span><span style="color:#d6d6d680;">"</span><span>]
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:repl-options </span><span>{</span><span style="font-weight:bold;color:#cc9495;">:timeout </span><span style="font-weight:bold;color:#87d6d5;">200000</span><span>} </span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> Defaults to 30000 (30 seconds)
</span><span>
</span><span> </span><span style="font-weight:bold;color:#cc9495;">:test-paths </span><span>[</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">spec/clj</span><span style="color:#d6d6d680;">"</span><span>]
</span></code></pre>
<p>The <code>:repl-options</code> is straightforward. It just increases the timeout
when launching a REPL. Presumably this is because Chestnut REPL's are
so filled with awesome that they take longer to load ;) The <code>*-paths</code>
options are simple too. They just override the default place that
Leiningen looks for Clojure source and test files by default.</p>
<blockquote>
<p>N.B. The version of Chestnut I generated this with actually has a
bug, since it generated "spec/clj" as the test path, but no "spec"
folder.</p>
</blockquote>
<p>Next come the dependencies and plugins needed by this project.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :dependencies [[org.clojure/clojure "1.6.0"]
</span><span> [org.clojure/clojurescript "0.0-2511" :scope "provided"]
</span><span> [ring "1.3.2"]
</span><span> [ring/ring-defaults "0.1.3"]
</span><span> [compojure "1.3.1"]
</span><span> [enlive "1.1.5"]
</span><span> [om "0.8.0-rc1"]
</span><span> [environ "1.0.0"]]
</span><span>
</span><span> :plugins [[lein-cljsbuild "1.0.3"]
</span><span> [lein-environ "1.0.0"]]
</span></code></pre>
<p>Clojure and Clojurescript are obviously necessary, and when you're
using Clojurescript with Leiningen, you probably want the
[lein-cljsbuild][cljsbuild] plugin for compiling your
Clojurescript. <a href="https://github.com/ring-clojure/ring#ring">Ring</a> is the standard Clojure web application's
library. <a href="https://github.com/ring-clojure/ring-defaults#ring-defaults">Ring-defaults</a> is a library for providing standard
configurations of <a href="https://github.com/ring-clojure/ring/wiki/Concepts#middleware">Ring middleware</a>. <a href="https://github.com/weavejester/compojure#compojure">Compojure</a>
is a routing library built on top of Ring.</p>
<sub>
I'm not quite sure why the Clojurescript dependency is marked as
`provided`...
</sub>
<p><a href="https://github.com/cgrand/enlive#enlive-">Enlive</a> is a "a selector-based (Ã la CSS) templating library
for Clojure." Very cool stuff, and wildly useful for many other HTML
processing/producing/consuming tasks than just templating.
Unfortunately very under-documented. <a href="https://github.com/omcljs/om#om">Om</a> is the only
Clojurescript specific dependency in the list, but it's a pretty cool
one. It's a Clojurescript interface to Facebook's React.js library.</p>
<p>Finally we have the <a href="https://github.com/weavejester/environ#environ">Environ</a> library, and it's associated
<code>lein-environ</code> plugin. As their README says:</p>
<blockquote>
<p>Environ is a Clojure library for managing environment settings from
a number of different sources.</p>
</blockquote>
<p>However, it is also the basis for a lot of the really neat things that
Chestnut does.</p>
<p>Next up, we have some more basic configuration stuff.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :min-lein-version "2.5.0"
</span><span>
</span><span> :uberjar-name "tour.jar"
</span></code></pre>
<p>These two settings are specifically to help with deployment to Heroku
which looks for the <code>:min-lein-version</code> key to determine what version
of lein to build your app with. Chestnut also changes the uberjar name
so that the Procfile can specify a specific filename. Normally, the
uberjar name is derived from the project name and the current
version. However, this makes it a moving target. Every time you bump
your version number you would have to update the Procfile to stay in
sync. This way, the uberjar always has one name and the Procfile never
needs to change.</p>
<p>Finally, we have the basic cljs build configuration:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :cljsbuild {:builds {:app {:source-paths ["src/cljs"]
</span><span> :compiler {:output-to "resources/public/js/app.js"
</span><span> :output-dir "resources/public/js/out"
</span><span> :source-map "resources/public/js/out.js.map"
</span><span> :preamble ["react/react.min.js"]
</span><span> :optimizations :none
</span><span> :pretty-print true}}}}
</span></code></pre>
<p>This is remarkably similar to the configuration from the
<a href="https://github.com/emezeske/lein-cljsbuild/blob/1.0.5/example-projects/none/project.clj">lein-cljsbuild none example project</a>, and a standard
setup to enable source-maps.</p>
<h3 id="profiles"><a href="#profiles" aria-label="Anchor link for: profiles"><i class="fa fa-link" aria-hidden="true" ></i></a>
Profiles</h3>
<p>So far, most of what we've seen is fairly standard
clojure/clojurescript configuration stuff. But one of the things that
makes Chestnut awesome is that it makes really good use of the
Leiningen profiles feature. In particular, it uses profiles to
concisely specify different clojurescript compilation settings and add
dependencies that are only needed for development. If this is the
first time you've heard about Leiningen profiles, you should probably
go <a href="https://github.com/technomancy/leiningen/blob/master/doc/PROFILES.md#profiles">read about it</a>. The basic summary is this though:</p>
<blockquote>
<p>You can place any arbitrary key/value pairs supported by defproject
into a given profile and they will be merged into the project map
when that profile is activated.</p>
</blockquote>
<p>Let's start with the extra <code>:dev</code> setups. It starts off nice and easy
just adding more clojure source and test paths and some extra
development-only dependencies.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :profiles {:dev {:source-paths ["env/dev/clj"]
</span><span> :test-paths ["test/clj"]
</span><span>
</span><span> :dependencies [[figwheel "0.2.1-SNAPSHOT"]
</span><span> [figwheel-sidecar "0.2.1-SNAPSHOT"]
</span><span> [com.cemerick/piggieback "0.1.3"]
</span><span> [weasel "0.4.2"]]
</span></code></pre>
<p>Notice that there is a similar entry under the <code>:uberjar</code> profile for
<code>:source-paths</code> but that the <code>:test-paths</code> are omitted (since the
uberjar is typically for distributing production code).</p>
<p><a href="https://github.com/bhauman/lein-figwheel#lein-figwheel">Figwheel</a> is a very cool project that "pushes ClojureScript code
changes to the client." This enables a very smooth Clojurescript
workflow. It's so seamless in fact that in some ways it makes
Clojurescript programming <em>more enjoyable</em> than working with Clojure!</p>
<p><a href="https://github.com/cemerick/piggieback#piggieback-">Piggieback</a> provides support for running a ClojureScript REPL
on top of an nREPL session. Chas goes into the reasons of why this is
a desirable thing in the README, so go check it out if you're interested.</p>
<p>Finally, <a href="https://github.com/tomjakubowski/weasel#weasel">Weasel</a> allows your ClojureScript REPL to use
WebSockets to communicate with your chosen execution environment.
Again, their README has good information about why this might be a
thing you want.</p>
<p>The next two sections are pretty much just setup for Piggieback and
Figwheel:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :repl-options {:init-ns tour.server
</span><span> :nrepl-middleware [cemerick.piggieback/wrap-cljs-repl]}
</span><span>
</span><span> :plugins [[lein-figwheel "0.2.1-SNAPSHOT"]]
</span><span>
</span><span> :figwheel {:http-server-root "public"
</span><span> :server-port 3449
</span><span> :css-dirs ["resources/public/css"]}
</span></code></pre>
<p>Then both the dev and uberjar profiles contain an <code>:env</code> map
specifying either that <code>:is-dev</code> is <code>true</code> or <code>false</code>. These
configurations hook into the Environ library and allow you to specify
the value of environment variables directly in the project.clj
file. This particular usage should be pretty straightforward; it's
basically a simple switch that allows us to tell in our code whether
or not this is a dev build/run or not.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :env {:is-dev true}
</span></code></pre>
<p>The only thing remaining in the dev profile now is the cljsbuild test
configuration. Though once again, notice that there is the small
addition of the <code>env/dev/cljs</code> path as a source for our previosly
configured <code>app</code> ClojureScript build.</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :cljsbuild {:test-commands { "test" ["phantomjs" "env/test/js/unit-test.js" "env/test/unit-test.html"] }
</span><span> :builds {:app {:source-paths ["env/dev/cljs"]}
</span><span> :test {:source-paths ["src/cljs" "test/cljs"]
</span><span> :compiler {:output-to "resources/public/js/app_test.js"
</span><span> :output-dir "resources/public/js/test"
</span><span> :source-map "resources/public/js/test.js.map"
</span><span> :preamble ["react/react.min.js"]
</span><span> :optimizations :whitespace
</span><span> :pretty-print false}}}}}
</span></code></pre>
<p>Compared to the <code>:dev</code> profile, the <code>:uberjar</code> is quite simple:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span> :uberjar {:source-paths ["env/prod/clj"]
</span><span> :hooks [leiningen.cljsbuild]
</span><span> :env {:production true}
</span><span> :omit-source true
</span><span> :aot :all
</span><span> :cljsbuild {:builds {:app
</span><span> {:source-paths ["env/prod/cljs"]
</span><span> :compiler
</span><span> {:optimizations :advanced
</span><span> :pretty-print false}}}}}})
</span></code></pre>
<p>The main difference is the addition of three pieces of configuration
that make this a true production build. The <code>:hooks</code> option causes
leiningen to run the <code>leiningen.cljsbuild/activate</code> function to let it
<a href="https://github.com/technomancy/leiningen/blob/master/doc/PLUGINS.md#hooks">hook into the defualt Leiningen tasks</a>.</p>
<p>Then <code>:omit-source</code> simply directs leiningen not to include the
clojure source files in the resulting uberjar, and <code>:aot :all</code> causes
the Clojure compiler to <a href="http://clojure.org/compilation">Ahead-of-time compile</a> all your clojure
code. The idea behind both these configs is to make your final uberjar
as lean and performant as possible, at the expense of some (probably
not needed) flexibility.</p>
<p>So that's it for Chestnut's <code>project.clj</code>. It looks intimidating at
first, but in the end it is quite approachable when you break it down
into it's component pieces.</p>
Testing and the Structures of Learning Environments2015-05-06T00:00:00+00:002015-05-06T00:00:00+00:00https://zefira.dev/posts/testing-and-the-structures-of-learning-environments/<p>I've been fortunate this last year in getting to spend time in several
radically different learning environments in a relatively short span
of time. These experiences have gotten me thinking about the relative
pros and cons and about how I respond to the amount of structure that
environment offers.</p>
<!--more-->
<h3 id="the-university"><a href="#the-university" aria-label="Anchor link for: the-university"><i class="fa fa-link" aria-hidden="true" ></i></a>
The University</h3>
<p>Okay, let's start off with the most structured of all, the
university. "formal education" is certainly the most structured of all
the environments I've been in recently. Apart from the lower-levels of
the same system, I'm having a hard time of thinking of a more
structured type of learning. I've written before quite extensively
about specific problems I have with the established school system, so
I'll just leave it at that. However, university courses are highly
structured. They are actually required to by the nature of the
system. Even Fairhaven College, a place which essentially strives for
less structure only achieves that, less structure.</p>
<p>An excellent symbol (and indeed, a representation) of that structure
is the course syllabus. Contained in that one document are supposed to
be all the expectations and requirements for "doing well" in the
class. I noticed something towards the end of my college career. These
documents started getting more formulaic. In fact, they seem to be
approaching the type of jargon-ified meanings as "lawyer-ese." For
instance, my university had a policy that no assignments could be
given to students during "dead week" - the week before finals -
<em>unless</em> the teacher had mentioned that it was possible in the
syllabus.</p>
<p>Because of this, many professors simply had a clause in their "stock"
syllabus that mentioned that there might be assignments due during
dead week, just to reserve the option. First of all, I have a problem
with a document that requires so much boilerplate that there can be a
"stock" version of it. Second, the system of rules and requirements
that caused teachers to always include a certain statement in the
syllabus regardless of their plans for the class, made the whole
system into a game, pitting students against each other, and often
also into opposition with their professors.</p>
<p>I resisted playing this game for a long time (most of my life
actually). After taking two years off from school when my mother fell
ill, I came back with a conviction that this game was a necessary evil
that I needed to play, and play well. So I did. I got excellent grades
in my last several quarters at Western in all of my classes, despite
taking heavy credit loads of all difficult classes. But I didn't learn
as much.</p>
<p>My normal method of doing school appears somewhat lackadaisical
compared to the studious diligence of many "good" students. But part
of the reason for this is that I tend to follow my interests, and go
well beyond what is being asked of me by the teacher. A classic
example was a paper I needed to write for a history class in high
school. I needed to do research on some historical topic, but on the
way I stumbled upon some fascinating information about dolphins. I
spent most of my evening reading and learning much about dolphins, and
not researching for the paper. My high school pre-Calculus teacher
summarized my attitude towards school by bestowing an award on me at
the end of the year: "Highest Learning to Grade ratio."</p>
<h3 id="the-recurse-center"><a href="#the-recurse-center" aria-label="Anchor link for: the-recurse-center"><i class="fa fa-link" aria-hidden="true" ></i></a>
The Recurse Center</h3>
<p>At the opposite end of the structure spectrum we have the Recurse
Center. It is often difficult to explain to people exactly why I chose
to go to the Recurse Center last fall. I usually fall back on
paraphrasing the tag line from their website:</p>
<blockquote>
<p>The Recurse Center is a free, self-directed, educational retreat for
people who want to get better at programming, whether they've been
coding for three decades or three months.</p>
</blockquote>
<p>The key part of that statement for me is the "self-directed"
portion. The vast majority of what I learned during my CS degree came
from entirely self-directed exploration so it seemed like a natural
fit.</p>
<p>But they are serious when they say self-directed. The structure of
their environment is at a very bare minimum. About as bare as you can
get I think and still build a sense of community around a shared
experience. The structure and workings of the Recurse Center have been
written about by many of the students as well as the people who run
it. But for comparison's sake, I want to mention the few structural
elements that make up the Recurse Center.</p>
<p>First, they spend a half day at the beginning of the batch with a
welcome meet-and-greet breakfast. This is followed by talks by the
founders and facilitators to "set the tone" and people's expectations
for the batch. These mostly consist of talking about what they want
the Recurse Center to be, and the lightweight social rules that they
follow to help achieve that. After that, there are a few logistical
matters like keycards and access to the chat client, but then you are
thrown head-first into being at the Recurse Center: i.e. sitting in
room with about 60 other people, and learning about programming.</p>
<p>There are a few more things, like the Monday night talks by the
current resident or other interesting person, or the daily checkins
with other RC'ers. But other than that pretty much everything is up to
the participants of the batch. If people want to organize a daily
reading group you can do that. Or you can find someone interested in
pairing on a project with you. Or you can go to a short
presentation/talk someone has decided to give.</p>
<p>If this sounds vague and confusing it's because of the lack of
structure. It sounds like a platitude, but spending three months at
the Recurse Center really can be whatever you want it to be. A lot of
people take some time to adjust to this looseness, and there are
facilitators whose job is precisely that, to facilitate people in
their self-directed learning. Whether that means brain-storming about
projects to work on, or giving code review, or pairing on a
particularly difficult problem their whole job is to be available to
help RC'ers make the most of their time their. As a bonus they get to
spend their days doing their own self-directed learning.</p>
<p>The structure - or lack thereof - of the Recurse Center was inspired
at least in part by the unschooling movement as talked about by Grace
Llewellyn. I've [written before][unschoolove] about how I fell in love
with the idea of unschooling. But my time at the Recurse Center was
the first time I actually got to fully and intentionally practice
it. Three months was not long enough.</p>
<p>[unschoolove]: {% post_url 2014-05-06-s-and-e-part-three:-the-teenage-liberation-handbook %}</p>
<p>Despite the brevity of the experience, it did me a world of good. My
last year at college playing their game to the hilt sort of killed
something in me. I stopped enjoying programming, and I stopped seeking
out learning on my own as much. Killed is a rather strong word but
that's how it felt. Maybe it would be more appropriate to say that it
simply wore me out. I had no energy, no [gumption] left to put into
anything. At the Recurse Center though, I built massive amounts of
gumption. So much so that I haven't seriously found myself lacking it
since. If for only that reason, the Recurse Center was the perfect
antidote to my time at university. And I rediscovered my joy in
programming.</p>
<p>[gumption]: {% post_url 2014-05-01-s-and-e-part-two:-gumption %}</p>
<h3 id="8th-light"><a href="#8th-light" aria-label="Anchor link for: 8th-light"><i class="fa fa-link" aria-hidden="true" ></i></a>
8th Light</h3>
<p>Now I'm an apprentice at 8th Light. I sought out 8th Light primarily
because of their apprenticeship program, and the remarkable similarity
it bore to [my own idealized conception][codeacademy] of how to teach
someone how to program.</p>
<p>[codeacademy]: {% post_url 2014-06-07-s-and-e-part-seven:-onward %}</p>
<p>It is sort of a fascinating and fortuitous coincidence that the amount
of structure here is in between that found at the Recurse Center and
in universities. Whether it is actually "just right" remains to be
seen. So far the balance between guidance and self-directed
exploration has been fairly ideal. Having weekly check-in's with my
mentors puts a firm bound on how long I might drift without
purpose. Though I haven't found that to be a problem here, it was
definitely something I needed to be mindful of during my time at the
Recurse Center. I also find it helpful to have a direction set for me,
with concrete and achievable tasks to mark the route.</p>
<p>One interesting thought I've had recently is an analogy between the
structure of these various learning environments and software testing.
I think that the Recurse Center is much like the undisciplined hacking
on a pet project. There may be tests, but they are impromptu and
somewhat haphazard. More projected onto the structure of the code
after completion than designed into it from the beginning. You can get
a lot done in this fashion, but it's also easy to get lost in the
weeds (though this is largely the point as far as many RC'ers are
concerned).</p>
<p>A university degree program is then more like poorly executed TDD. The
tests are written first - mostly. But they are written with a mind to
<em>how</em> the task will be accomplished and are tightly, almost
inextricably, coupled to the implementation. They define a rigid but
currently stable and verified system. But woe betide the person who
tries to change anything, because much of the code and all of the
tests will break.</p>
<p>8th Light then is TDD done well. The tests are written first and they
specify the behavior of the system, not the implementation. The
desired goal is achieved through (rapid) iteration and feedback.</p>
Iteration Two: Refactoring and Minimax2015-04-29T00:00:00+00:002015-04-29T00:00:00+00:00https://zefira.dev/posts/iteration-two-refactoring-and-minimax/<p>I'm just about finished with my second iteration as an apprentice at
8th Light. Where my first iteration centered a lot around reading and
learning how TDD and Ruby work, this week I focused more on coding. I
spent most of my time just implementing the several stories of my
iteration: some minor (but important!) output tweaks, implementing a
play again loop, and adding two AI's to the game. I also did quite a
bit of refactoring and general code cleanup.</p>
<!--more-->
<p>I spent several hours figuring out the mechanics of doing an extract
class refactoring, and then realizing that I had extracted the class
the wrong way and so the "container" class was the one being held by
it's containee. It turned out to be very awkward to try to invert
this relationship so I backed out the change and extracted the other
class so that the container/containee relationship was correct. I
consulted Martin Fowler's <em>Refactoring</em> book heavily here. It's really
neat how the mechanical steps he describes are actually an incredibly
useful guide.</p>
<p>I noticed an interesting thing while I was refactoring. I was changing
the code so much that I started losing track of what was actually
going on. At some point, my brain just sort of gave up on trying to
hold the whole application in my head. This was an interesting
experience because usually I can hold more complexity in my head than
this Tic Tac Toe game currently has. I think this is part of the trend
I noticed that in general, when doing TDD I feel less need to hold a
lot of the complexity in my mind at once.</p>
<p>The slightly scary flip-side of that was that because I was doing
extract class refactorings and moving methods and such, I also ended
up moving tests around and modifying the tests a lot to keep them
working. Mucking around with them seriously decreased my confidence in
them, and thus in my confidence in the system as a whole. Clearly I
need to think more about how the test modification process goes along
with refactoring.</p>
<p>Another piece of functionality that took me surprisingly long to
implement was the play again functionality. The interesting part of
that process was that I wrote tests, and then wrote an implementation
that I thought was correct. The tests failed in a confusing manner and
so I assumed that the tests were incorrect. After poking around for
quite a while, firing up the debugger (<a href="http://pryrepl.org/">Pry</a> and <a href="https://github.com/deivid-rodriguez/pry-byebug">byebug</a> are
awesome!) I eventually discovered that the test was correct, and it
was even failing exactly correctly. I had simply forgotten to take
into account that once a game has been completed, one can't just start
playing it again from the beginning. So I had to write a reset method.</p>
<p>On Tuesday I spent most of the day test driving my way to a perfect
AI. This was an interesting experience since it was the first time
I've had a really clear idea of where I wanted to go with the code
while doing TDD. For the most part I've been trying to stick to the
letter of the acronym and let the tests lead me to the code
organization they want.</p>
<p>But the <a href="https://en.wikipedia.org/wiki/Minimax">minimax</a> algorithm is well known and an established way to do
the game state-space search that is required for a perfect Tic-Tac-Toe
AI. So how does one test drive that? Apparently I am most definitely
not alone in asking this question as both my mentors were expecting
it, and my fellow apprentice working on Tic-Tac-Toe (Ari) also spent
time last week puzzling over it.</p>
<p>I worried about it briefly at the start of the day, but then I quickly
adopted the outlook that I've found to be most successful with TDD;
don't think too much about the future. Instead, I focused on the
fundamentals. What's the simplest case an AI needs to deal with. I
turned to the Wikipedia page on Tic-Tac-Toe and it's bullet list of
how to be a perfect player. The most important rule is that an AI
should win immediately when it can. If that's not possible it should
block an opponent from winning.</p>
<p>These two situations are very easy to detect. It only takes a slight
modification to the code pattern used for detecting if someone has
already won, and no future board states need to be taken into
account. This approach got me moving forwards, which is often the
hardest part.</p>
<p>Once I had some momentum I kept going by then considering simple fork
creation, the next most important move type. After pondering how this
might work, I hit upon a simple solution. A fork is just a move that
results in a board where you have two possible ways to win! This was
attractive to me for several reasons. First, it was a (relatively
small) step towards searching the state-space of the game. By looking
one move ahead I felt I was moving in the direction of minimax. And it
was simple enough that it felt doable in a single TDD step.</p>
<p>This worked shockingly well. Not only did my very simple fork
look-ahead detect most forks that it could create, somehow it actually
was able to produce the correct fork-blocking behavior in the first
four cases that I came up with.</p>
<p>I was suspicious though, and so I crafted some devious fork tests that
were designed to expose the flaw in my simple algorithm. Eventually I
did that and it became clear that something more powerful was
needed. At this point, thinking about the simplest way to do things
produced no clear results. I was already doing a minimal look-ahead
and the analysis was insufficient. It seemed that the next step would
have to be going to a minimax algorithm. Here's basically what my
simple algorithm was at this point:</p>
<pre data-lang="ruby" style="background-color:#393939;color:#dedede;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">score</span><span>(node)
</span><span> get_wins(node.indexed_attack_sets).count
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">find_forks</span><span>(board)
</span><span> scores </span><span style="color:#ececec;">=</span><span> board.empty_spaces.map </span><span style="color:#fed6af;">do </span><span>|index|
</span><span> node </span><span style="color:#ececec;">=</span><span> board.speculative_move(</span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">X</span><span style="color:#d6d6d680;">"</span><span>, </span><span style="color:#fffd87;">index</span><span>)
</span><span> [</span><span style="color:#fffd87;">index</span><span>, score(node)]
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> scores.max_by { |i, s| s }.first
</span><span> </span><span style="color:#fed6af;">end
</span><span>
</span><span> </span><span style="color:#fed6af;">def </span><span style="color:#fffd87;">get_move</span><span>(board)
</span><span> attacks </span><span style="color:#ececec;">=</span><span> board.indexed_attack_sets
</span><span> win </span><span style="color:#ececec;">=</span><span> get_wins(attacks).first
</span><span> block </span><span style="color:#ececec;">=</span><span> get_blocks(attacks).first
</span><span>
</span><span> win </span><span style="color:#ececec;">or</span><span> block </span><span style="color:#ececec;">or</span><span> find_forks(board)
</span><span> </span><span style="color:#fed6af;">end
</span></code></pre>
<p>I tried to do get to minimax in several ways. First off, I actually
ended up using the <a href="https://en.wikipedia.org/wiki/Negamax">negamax</a> algorithm because it's simpler to code
because it takes advantage of the zero-sum property of
Tic-Tac-Toe. Initially, I tried to replace my whole algorithm with
negamax right off the bat. This did not work out well. Things got
complicated and many tests were failing. So I backed out and treated
negamax as a small refactoring. Specifically I treated it as a
refactoring of the <code>find_forks</code> function. This worked quite well, and
suddenly all of the forks tests were passing. I then incrementally
expanded it until negamax was the only thing being used for deciding
which moves to make.</p>
<h2 id="clojure-tdd"><a href="#clojure-tdd" aria-label="Anchor link for: clojure-tdd"><i class="fa fa-link" aria-hidden="true" ></i></a>
Clojure TDD</h2>
<p>I also spent some time pairing with both Emmanuel and Kristen on some
Clojure katas, the bowling game and coin changer respectively. This
was really fascinating for a couple of reasons. I consider myself
reasonably competent with Clojure. But I am almost totally unfamiliar
with trying to do TDD in Clojure.</p>
<p>The pairing that I did with both my fellow apprentices was in the
Ping-Pong style (mostly), where one of us wrote a test and the other
tried to make it pass. What was fascinating to me was that the
incremental development approach that this enforces pushed me towards
writing very non-idiomatic, highly stateful Clojure code.</p>
<p>After attempting the coin changer with Kristen, I was starting to
doubt my Clojure-chops and so I tried to TDD it out on my own. I ended
up arriving at a more satisfactory solution that used a stream model
of discrete state transformations. So a cleaner stateful solution.</p>
<p>I ended up talking about this with Emmanuel after our pairing session,
and he showed me a blog post and a screencast about doing the bowling
kata in Clojure. Both people ended up with similar solutions, which
basically involved the insight that a frame in bowling is only two
rolls when no strikes or spares are involved. In retrospect, this
seems sort of obvious, since a frame can't be scored until all of it's
rolls have occurred. But combining this with actually duplicating some
of the numbers from the stream of pin-counts is what leads to an
elegant Clojure solution.</p>
<p>Besides this key realization, I think I was also missing a critical
tool in my Clojure toolbox: custom lazy sequences. Both solutions
Emmanuel had found on the web ended up constructing a function that
produced a custom lazy sequence. This was necessary because of the
need to repeat certain elements from the stream.</p>
<p>But what I'm realizing now is that I think it's difficult to test
drive recursive solutions. But maybe I'm just doing it wrong. Often
when I write recursive functions I don't bother to write just the base
case first. I typically will just jump straight in and write both
parts at once. But now that I think about it, testing the base case of
a recursive algorithm would be really natural in TDD. You just have to
start at the simplest case.</p>
<p>Another thought that occurred to me about why TDD in Clojure is
harder. In Ruby or Java I have read about different design patterns
and there are established refactorings for manipulating class-based OO
code. So when I'm doing TDD in those languages I know the patterns of
it. Not patterns in the design patterns sense, but the flows of code
configurations you might say. Specifically, I'm familiar with the
intermediate stages and I can imagine them.</p>
<p>But the way I write functional code is somewhat different. Most of the
time, I experiment with a small piece of code in the REPL, until I've
produced the result I want, then I wrap it up into a function. This
process can proceed top-down where I conceive of the top-level
function first and then create the necessary pieces to make that
happen; or it can go bottom-up where I think about little pieces of
functionality in the domain I'm considering, and then start composing
those pieces to make more complex things.</p>
<p>In both of these styles of development, there isn't always working
code at all times though. In particular, the interesting sort of
high level behaviors often don't work until the whole thing comes
together at the end.</p>
<p>So I think I need at least two things before I can do TDD in Clojure
more effectively. I need to learn the patterns of Clojure code better,
and get more familiar with how it can be molded and changed from
simple and ill-factored to cleaner and well-factored. And second, I
need to adjust to the paradigm shift of thinking about code tests
first in a functional context.</p>
On Editors and their Benefits2015-04-16T00:00:00+00:002015-04-16T00:00:00+00:00https://zefira.dev/posts/on-editors-and-their-benefits/<p>I've been using Emacs for about as long as I've been programming. So
that makes it nearly eight years now. I like to think that I've gotten
reasonably competent with it, and somewhat more importantly, I have
customized my emacs to fit my mind like a glove.</p>
<p>I also touch type, but I use the Dvorak keyboard layout. These two
things combine to make it so that I can write programs at a speed
close to how fast I can conceive them. This is important, not because
it means I can code faster than other people, but because it means I
can get the thoughts out of my head fast enough that they don't slip
away.</p>
<p>This was brought into sharp relief for me a few days ago when I did
some pairing with my mentor Zach using his computer. You see, Zach
uses the standard QWERTY layout, and he also uses Vim.</p>
<!--more-->
<p>It seems that the accepted wisdom about pair programming is that it
works best when two people share the same computer, and swap control
of the keyboard back and forth. Some sources even recommend having two
sets of keyboard/mouse to lower the barrier to switching even further.
In the spirit of being open to trying new things I acquiesced to
Zach's implicit expectation that I would use his computer with QWERTY
and Vim.</p>
<p>It's been a long time since I've seriously attempted to use
QWERTY. When I first taught myself to touch-type in Dvorak I tried to
maintain some proficiency with QWERTY since it is ubiquitous. But I
soon discovered that it was hard to keep both locations for keys in
muscle memory at the same time. And that's really where efficient
touch typing requires that knowledge to reside. When you have to
consult your memory for the location of key, you've already lost a lot
of your speed.</p>
<p>Luckily for me, I could keep on typing QWERTY as I always had; that
is, by looking at the keyboard constantly and lifting my fingers high
enough off the keyboard that I could see the letters.</p>
<p>Muscle memory is important to my usage of Emacs too, and it builds
heavily on my Dvorak muscle memory. Emacs has a (deserved) reputation
for having a somewhat absurd number of key bindings. Trying to
remember them directly would be a monumental task. Instead, the most
commonly used key bindings migrate quickly from the realm of conscious
thought to muscle memory. I didn't even notice this happening until I
started trying to teach a friend of mine to use Emacs. Some of the
commands that I use, I know only by their location on the keyboard.
Given that I type in Dvorak and the keys are labeled in QWERTY this
means that the actual character I'm pressing is pretty opaque.</p>
<hr />
<p>When is proficiency or familiarity with a tool more important than
other considerations?</p>
<p>First let's look a bit at what those other considerations might
be. The first one that comes to mind is that using the same tools as
the team you're working on can be pretty important. Or maybe there is
another tool that has much more powerful or focused facilities for the
task/language/domain. Or to reverse that, possibly your tool of choice
simply lacks something that a lot of other tools support. Finally, I
think a really important consideration is whether you are holding on
to your tool(s) of choice out of habit and the comfort they provide.
Let's look at some examples of each of these.</p>
<p>A few years ago I was hired as an intern at Sage Bionetworks, a small
bio-tech start-up in Seattle. They happen to be an all Java shop,
and the standard editing environment there is Eclipse. At the time, I
was much less familiar with Emacs, I had never even opened Eclipse and
I didn't know any Java.</p>
<p>I spent my first few days getting through the administrative details
of starting a new job. But pretty quickly I got to the point where I
needed to setup my new computer for doing development on the Sage web
platform. They had a wiki, with several different pages on the
bootstrapping process for the various different aspects of the
process. A significant portion of it was focused on getting your
Eclipse installation setup correctly, with all the right plugins and
such.</p>
<p>Being totally in love with Emacs at that time, I determined that I was
going to figure out how to setup Emacs as a kick-ass Java editing
environment. My mentor was grudgingly amenable to this plan of
action. I struggled with that problem for a few days, until my mentor
came back and strongly suggested that I use Eclipse. His rationale was
essentially that this was the tool that the whole team had
standardized on. Since no one else really used the command line tools
to build or test the product I would be largely on my own in getting
things to work.</p>
<p>I gave in, and learned to use Eclipse. I made it bearable by
installing a plugin that simulated Emacs key bindings. I learned there
were some nice things about Eclipse - the automatic versioning, the
Java refactoring tools. But I also found it to generally be a vastly
inferior tool. While Eclipse has an extensive ecosystem of plugins and
add-ons, installing them is a nightmare of clicking through GUI menus
and needing to restart possibly several times. It is also highly
customizable, with good support for key maps and visual
modifications. However, neither your personally installed add-ins nor
your configurations can be saved in a reasonable way, and there is (as
far as I know) no way to automate the setup process. So every time you
move to a new development machine, you need to go through the same
process. Or, more likely, go through a very <em>similar</em> process, and
end up with a subtly different dev environment.</p>
<p>But I did get the benefit of being able to get the advice of the other
devs on the team when something wasn't working with my build. This
turned out to be critical since the process of getting a working (and
repeatable) build of their software was a highly non-trivial one.</p>
<p>When I was working with Zach last week and ended up using Vim with
QWERTY, he said something to me that reminded of my mentor's words at
Sage. He said that a lot of people at 8th Light use Vim, and that if I
want to do a lot of pairing with people it will probably be to my
benefit to learn how to use Vim at least a little bit.</p>
<p>This makes more sense in an environment where pairing is actively
practiced, but the general idea behind it is basically the prospect of
being able to sit down at someone else's computer and use it for
development at a reasonable level. I'm not sure that I agree with this
philosophy though.</p>
<p>One place where being able to use any given computer is key is as a
sysadmin. My college's CS department started participating in a
national competition during my second year, the Collegiate Cyber
Defense Competition or CCDC. The competition itself basically places
several teams of students in the role of a new system administration
team for some large corporation. The system is potentially in disarray
after a poorly documented transition from a previous team and the
students task is to secure the systems while maintaining a given level
of availability for one or more different services (such as email,
databases, servers, etc.).</p>
<p>I was very interested in participating in the club that was preparing
for the competition because they were learning both defensive and
offensive computer security skills. But I found it difficult to work
in the context of system's administration because I couldn't have my
environment setup just how I wanted it. Particularly during the
competition itself, there would be no time to try and install Emacs
(which is both large in footprint and memory usage and requires a ton
of dependencies), or switch the terminal at the computer I was using
to Dvorak. Thus I found that two choices I had made years before had
effectively precluded my participation in the CCDC.</p>
<p>Interestingly, my choice to learn the Dvorak layout is actually what
pushed me into learning Emacs. I had just gotten to the point of
actual touch-typing with Dvorak when I decided it was time to switch
from using PythonWin to a "real programmer's editor." Based on what I
had been reading on the internet, that choice seemed clear: it had to
be Vim.</p>
<p>But I was stalled almost immediately. Vim uses the letters 'h', 'j',
'k', and 'l' for text navigation, so you don't need to move your hand
to the arrow keys. This is convenient for usage under QWERTY, all
four keys are on the home row and are easily accessible - without
stretching - to your right hand. But on Dvorak they are all over the
keyboard, and there is no mnemonic for telling which key does
what.</p>
<p>I struggled with Vim for a few days, trying to learn how I could remap
the movement keys to the same location under Dvorak as they are in
QWERTY. But I was unable to find a solution and the prospect of the
cascading key remappings was enough to drive me to look at Emacs. By
contrast with Vim, most of the keys in Emacs have some kind of
mnemonic association. Moving forward a character is <code>ctrl f</code> and back
is <code>ctrl b</code>. Down one line is <code>ctrl n</code> and up is <code>ctrl p</code> for next and
previous. Not only that, but part of the very philosophy of Emacs is
easy customization, up to and including remapping every key on your
keyboard to do something different. More importantly, that philosophy
of customization is embedded in the Emacs community.</p>
<p>The seemingly simple choice of learning to type in Dvorak turned out
to be a key decision in my life as a programmer. It cut off certain
possibilities like learning to use Vim, and presents certain
challenges for pair programming, particularly in the quick
back-and-forth style where two people use the same computer. I have
also found that using Emacs appears to be the less common of the two,
and this is also limiting and isolating to some degree.</p>
<p>Here's the thing though, I <em>like</em> using Emacs; and typing in Dvorak
feels good to my fingers. So I'm a little bit stuck. I want to be
agreeable and able to collaborate and pair program with others
easily. But I also really like the tools that I use. They fit me. So
what do I do?</p>
<p>One possibility is to use more tools to overcome some of my
difficulties with pairing. Specifically, using a lightweight version
control system (like git) and a free code hosting site (like, say,
Github) you can arrange a style of pairing where two people work on
the same code, but each using their own device and tools and
setup. Pairing is the same during the actual coding process with
whatever Driver/Navigator or other dynamic you want to use. The
difference comes when you swap. Instead of simply sliding the keyboard
across to your partner, you commit, and push to your central
repository. Your partner then pulls down the latest code and starts
working on their machine. This is certainly a bit more work than just
handing off the keyboard, so it may not be suitable to very rapid
hand-offs. But I haven't found it to be overly onerous so far.</p>
<p>Another solution would be for me to maintain some skill at QWERTY and
Vim specifically for facilitating pairing. While more work for me
personally, it has the benefit of not requiring a new pairing
workflow. But this idea worries me a bit.</p>
<p>The human brain is <a href="https://en.wikipedia.org/wiki/Neuroplasticity">amazingly flexible</a>. My favorite
personal example of this is related to video games and how we control
them. Every first-person style game has at least one key setting in
the control options: whether the way you look upwards is by moving the
mouse or tilting the control stick upwards (usually known as "default"
or "normal"), or whether this mapping is "inverted" i.e. tilting up
looks down and tilting down looks up. My best friend and I have played
a lot of Halo together in our day. For a long time one of our favorite
pastimes was playing through Halo 2 levels on Legendary difficulty,
often with skulls active. Here's the thing though: he plays default,
and I play inverted.</p>
<p>Under most circumstances this doesn't matter. If we're playing
cooperatively it doesn't matter because we both have our own
controller and thus our own setting. It only becomes an issue when we
are handing a controller back and forth. But boy is it a problem
then. Invariably we both forget to switch back at the hand-off and
there ensues a brief period of confusion and panic when the game
character doesn't respond as our brain is wired to think it
should. What's really fascinating though is how quickly my brain
starts to retrain itself.</p>
<p>I've been playing games on inverted for years, possibly decades at
this point. As such you would think that my brain is very much
hardwired to expect that when I tilt the stick up, the view will move
down. And this is true. But sometimes, for one reason or another I'll
end up playing for a short time on the default setting. What happens
is remarkable and incredibly frustrating. If I focus on trying to
remember that up means up and down means down, I can usually get to a
level that is usable - at least when I pay full attention. But as soon
as something intense happens like a bad guy jumping out from behind a
bush, my reflexes revert to inverted settings. Even more interestingly
though, when I inevitably switch my control scheme back to inverted I
am unable to fully shed the tendency towards trying to play with
default look controls.</p>
<p>Back to keyboard layouts and editors. In short, I'm concerned that an
attempt to learn how to use QWERTY again and to memorize Vim keyboard
shortcuts will seriously undermine my ability to use Emacs effectively.</p>
<p>On the other hand, maybe this is just what I need. I'm also currently
trying to learn how to do Test Driven Development (TDD) and I'm
finding that a lot of the thought-habits I have around programming are
actually detrimental to doing TDD. Maybe breaking out of my
comfortable Emacs environment and using a new keyboard layout will
help me get into a totally new frame of mind and let me TDD more
effectively.</p>
<p>I think there is another reason to embrace the idea of learning a new
set of tools at this time though. A large part of the reason that I
wanted to do this apprenticeship with 8th Light was to learn. Learning
from others always requires some humility; at the very least you have
to be able to admit that there are things that you do not know. This
is difficult for me. But possibly, intentionally cultivating
<a href="https://en.wikipedia.org/wiki/Shoshin">Shoshin</a> or "beginner's mind" regarding my most basic level
of programming - how I edit my source code - will help me do it at all
levels.</p>
<blockquote>
<p>In the beginner's mind there are many possibilities, in the expert's
mind there are few. - Shunryu Suzuki</p>
</blockquote>
Learning TDD2015-04-14T00:00:00+00:002015-04-14T00:00:00+00:00https://zefira.dev/posts/learning-tdd/<p>This is my first week at 8th Light and I've been working on writing an
object-oriented Tic Tac Toe program. This would be no big deal for
typically, even though I'm doing it in a language I'm unfamiliar with.</p>
<p>Except! I'm trying to very rigorously drive the development with
tests.</p>
<!--more-->
<p>This is a fairly new experience for me. I mean, sure, when I first
read about TDD a while ago, I got really interested and excited about
it. I played with it a little bit, and tried to do some small stuff in
it. But I quickly got overwhelmed with the difficulty of "How do you
even get started?"</p>
<p>Monday, I made very minimal progress. I felt stuck not really sure
where to begin, and what to do. I ended up by basically writing tests
for a simple Board data class where I could make marks, and then see
that the marks were actually set. This felt highly unimpressive as the
output of my first day.</p>
<p>Tuesday and Wednesday I made some more progress. The thing that helped
for most was certainly reaching out for guidance from my mentors. I
had a good conversation with Brian. Despite it's somewhat brief and
vague nature it actually was very helpful. And I did some pairing with
Zach that really helped on the Ruby comfortability front. Just seeing
how someone who knows how to use Ruby approaches things made me feel
more confident and grounded in the language.</p>
<p>I also did quite a bit of Googling and reading about Ruby concepts and
paradigms. There wasn't anything really specific that I was looking
for, I was just trying to absorb some of the context of the community
by seeing what is talked about and the sorts of code snippets that are
out there. To an outside observer this probably seemed mostly like
procrastinating. To be fair, I sort of judged it that way myself.</p>
<p>The thing is though, in all my professional software development work,
I've noticed that whenever I end up procrastinating and sort of
"working around" a difficulty I'm having a peculiar thing happens. If
I try to force myself to face it head on and work at it, nothing
really good comes out of it. Often I just end up feeling stuck and
then I'm really unproductive, just staring at the keyboard.</p>
<p>But I find if I just relax into the procrastination, and try to stay
working on related things, that eventually everything sort of gels in
my mind, and then when I come back to the "hard" thing I'm usually
able to make some useful headway.</p>
<p>Today was that day for TDD in Ruby; I felt like everything sort of
came together. I came into work determined not to "procrastinate" like
I did for much of yesterday. I started off by starting to test drive a
Console or display class. But I quickly realized that this would just
be a thin wrapper on top of the IO class that already exists in Ruby!
So I went back to implementing my game class, and used mocks to
replace an instance of an IO class.</p>
<p>From there I got into a very nice flow. Even the tricky nature of
testing the main game loop didn't seem overly difficult in my mind. By
the end of the day I had gotten to the point where I couldn't really
think of any more functionality that would be needed to play an actual
game at the command line.</p>
<p>When I wired up the class to finally actually test it from end to end
and try to play a game, there were some unexpected problems. But the
basic logical structure of the game was pretty much entirely
correct. And I knew it would be, because of the tests that I wrote.</p>
<p>Very cool.</p>
Shaving the Clojurescript Yak(s)2015-04-04T00:00:00+00:002015-04-04T00:00:00+00:00https://zefira.dev/posts/shaving-the-clojurescript-yaks/<p>About a week ago I got fed up with a <a href="http://mangafox.me/">terrible website</a> that
had comics on it I wanted to read. So I decided to write a little web
app to make the reading experience more pleasant. Since I'm an avid
<a href="http://clojure.org/">Clojurian</a>, I've been interested in checking out
<a href="http://clojure.org/clojurescript">Clojurescript</a> for doing web development, and in particular
exploring the wonderful new world of React.js wrappers available in
Clojurescript. This is the story of how I learned to setup a
Clojurescript project.</p>
<!--more-->
<p>Since I've been interested in Clojurescript for a while, and even
toyed with it a few times, I came into this project with at least a
fair idea of what was out there. I knew about Om, and new basically
how the Clojurescript compilation process worked. I also am vaguely
comfortable with setting up a basic Clojure web app using <a href="https://github.com/ring-clojure/ring">Ring</a>
and <a href="https://github.com/weavejester/compojure">Compojure</a>.</p>
<p>I was also aware that there had been significant advances in the
Clojurescript workflow in the past few years. Most importantly I knew
about an apparently awesome Leiningen project template for
Clojure/Clojurescript web apps called <a href="https://github.com/plexus/chestnut">Chestnut</a>. However,
the times that I had tried to get up and running with Chestnut
previously I was totally overwhelmed by the amount of unfamiliar
code/configuration that it produced. Quite frankly, it seemed
excessive, and quickly led to me giving up on whatever
project-of-the-moment had inspired me.</p>
<p>So! This time, I determined to not start with Chestnut, and instead
build up slowly from the basic Lein app template that I was already
familiar with. And so commenced roughly five full days of mostly
yak-shaving. I'm not going to try and recount it all here; just the
highlights will be more than enough.</p>
<h2 id="how-i-shaved-my-yak"><a href="#how-i-shaved-my-yak" aria-label="Anchor link for: how-i-shaved-my-yak"><i class="fa fa-link" aria-hidden="true" ></i></a>
How I Shaved My Yak</h2>
<p>I started with off with a very vanilla <code>lein new app comic-reader</code>. From there, I copied the basic dependencies for a
jetty/ring/compojure web app from my <a href="https://github.com/RadicalZephyr/url-shortener">url-shortener</a>
project. I made some basic routes to make sure everything was working
correctly.</p>
<p>Next came adding Clojurescript into the project. This meant setuping
the <code>project.clj</code> to point to where the <code>*.cljs</code> files would live, and
then configuring the Clojurescript compiler. Basic Clojurescript
compilation with <a href="https://github.com/emezeske/lein-cljsbuild">lein-cljsbuild</a> is not totally trivial to
configure, especially since there are now many different options to
the Clojurescript compiler and many resources on the web have
older/outdated configuration examples, and typically no explanation
whatsoever of why they have it configured they way they do. But
overall it wasn't too tough. It helped significantly that I could
again copy setups I had previously found to work.</p>
<p>At this point, I was sick of shaving Yaks for a moment so I went and
learned how to use <a href="https://github.com/cgrand/enlive">Enlive</a> for doing
<a href="https://github.com/swannodette/enlive-tutorial#an-introduction-to-enlive">web scraping</a>! Then, feeling refreshed, I went back to the
Yak.</p>
<p>I knew that I wanted the awesome <a href="https://github.com/bhauman/lein-figwheel">Figwheel</a> plugin for an awesome
(mostly) reload-less Clojurescript experience. Again, Figwheel comes
with a Lein template that I didn't use directly. Instead, I made an
extra copy and then used it as a reference for when my configuration
based on reading the documentation didn't work.</p>
<p>Next step was adding in Om, and making a basic page setup
there. Again, nothing incredibly hard. I mostly just followed the
tutorial and everything came together fairly quickly. Only maybe an
hour of struggling and cursing at my computer. Then, I decided that I
wanted to build a single-page application (SPA). So I started looking
at libraries like <a href="https://github.com/gf3/secretary">Secretary</a> and
<a href="https://github.com/r0man/sablono">Sablono</a>. Eventually, after reading several blog posts and
pages of documentation, I decided that I actually wanted to use
<a href="https://github.com/reagent-project/reagent">Reagent</a> instead of Om. Luckily I hadn't written much actual
code before I came to that decision.</p>
<p>After playing with Reagent for a while, I started having difficulty
with thinking about how to use it as the basis for a SPA, especially
with in-browser routing happening, and changing the history token so
that different app states would be bookmark-able. (N.B. I've minimized
the explanation of this considerably. I spent a good chunk of time
wrestling with getting history integration working with Reagent before
realizing that Reagent's flow didn't make any damn sense to me.)</p>
<p>Back to the Google! After quite a lot of searching, sleeping, reading,
and searching again I found <a href="https://github.com/Day8/re-frame">re-frame</a> and it's epic
manifesto. After reading the whole damn thing (and all of the
"read-this-first" links), I decided that I would switch from vanilla
Reagent to re-frame. Again, thank goodness I hadn't really written any
significant code that was tightly coupled to reagent.</p>
<p>At this point, most everything worked pretty nicely, but there was
some significant ugliness about. I had my Figwheel configuration
inline in the same file as my main site code, and the clojurescript
configurations for production and dev were getting quite messy. After
incrementally gaining experience with most of the gaggle of
technologies that Chestnut uses, I felt prepared to tackle their
template again. So I started using it as a reference to enhance my own
configuration.</p>
<p>In particular, I started doing this when I wanted to deploy my app to
Heroku. It turns out that there are a whole raft of things that where
wrong with my configuration from Heroku's point of view. But after
about two hours of compare/edit/deploy cycles, I finally managed to
deploy my app to Heroku.</p>
<h2 id="wrapping-up"><a href="#wrapping-up" aria-label="Anchor link for: wrapping-up"><i class="fa fa-link" aria-hidden="true" ></i></a>
Wrapping Up</h2>
<p>To sum it all up, while this was in some ways a very frustrating
exercise (especially at times during the process!), overall it was
also a really excellent learning experience. Too often I try and
approach too many new things all at once. This often leads quickly to
getting overwhelmed by all the new-ness, and then often giving
up. It's not a great pattern.</p>
<p>The longer that I practice programming, the more firmly I come to
believe that incremental, evolutions of projects and knowledge are
fundamentally more approachable, sustainable and, quite simply, more
fun!</p>
<p>Now I'm feeling sort of pumped to do an "Annotated Chestnut"
walkthrough of what all the various configurations in Chestnut are
doing, and why they are useful and cool.</p>
Sane Static Site Setups2014-12-30T00:00:00+00:002014-12-30T00:00:00+00:00https://zefira.dev/posts/sane-static-site-setups/<p>During the last week of Hacker School I helped <a href="http://lmontopo.github.io/">Leta</a> sort out
some issues she had with her blog setup and restore everything to
sanity. It was a lot of fun and the setup is pretty straightforward so
I thought I'd do a short write-up on what we did and why.</p>
<p>To be clear, this blog post is about solving the particular problem of
how to organize a statically generated site/blog. The particulars I'm
going to discuss are for when you host the site on <a href="https://pages.github.com/">Github Pages</a>
but you need to generate the site locally because you're not using
vanilla <a href="http://jekyllrb.com/">Jekyll</a> or not using Jekyll at all.</p>
<!--more-->
<p>This isn't a tutorial about how to set up and use any particular
static site generator. There are <a href="http://jekyllrb.com/">quite</a> <a href="http://docs.getpelican.com/en/3.5.0/">a</a> <a href="http://wintersmith.io/">few</a> <a href="https://github.com/greghendershott/frog">out</a>
<a href="https://github.com/taylorchu/baker">there</a>, and <a href="https://github.com/hugoduncan/cl-blog-generator">they</a> <a href="https://staticsitegenerators.net/">all</a> seem to be quite good. So pick one and
get your site setup. You should be comfortable generating the content
of your site before worrying about what I'm describing in this post.</p>
<p>One of the best and worst things about using a static site generator (SSG)
is that the source for the site is fundamentally a separate thing from
the actual files that compose the site itself. The good news is that
the generated files are, well, generated. Given the source for a
site you can always regenerate the presentation files.</p>
<p>So clearly we want to keep the source for our site under version
control. If you're using Github Pages then git is a natural
choice. But Github Pages also requires that the generated content of
your site be in a git repository. This leads to an un-intuitive setup.
Because the source and published files don't actually share a common
history, it seems like they need to be stored in separate git
repositories. However, there is a fundamental relationship between
the files that dictates that organizationally they should always be
found together.</p>
<p>Luckily for us, git is flexible enough to allow us to achieve both
these seemingly conflicting goals. Since the usual workflow for a git
repository simply involves <code>git init</code> and then edit, <code>add</code>,
<code>commit</code> cycles, it's less well known that a git repository can
actually contain multiple independent "head" commits. Don't worry if
that doesn't totally make sense. The important thing is that we can
store two separate revision histories in the same git repository.</p>
<p>So, you have the source for a static site, and you've maybe written
some dummy (or real!) content for it and generated the site at least
once. Now, we want to make sure that we have a setup that will help us
preserve all of your hard work on making an awesome website.</p>
<p>The first thing we need to do is to make sure we have git repositories
in both the source and output directories by running <code>git init</code> in
both of them separately. Since most SSG's by default use a structure
where the output folder is a subdirectory of the source folder, make
sure that you have an entry in your gitignore file so that the output
isn't committed into the source repository. At this point we should
have two git repo's, one that <em>only</em> has the site source content
(including any files need by your SSG) and one that <em>only</em> has the
generated version of your site.</p>
<p>Now for the magic trick of combining the repositories. Let's say we
have this folder structure:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>- website/
</span><span> - .git/ # Git folder for source files
</span><span> - ... # Lots of awesome content files
</span><span> - output/
</span><span> - .git/ # Git folder for output files
</span><span> - ... # The actual generated content files
</span></code></pre>
<p>At a shell prompt in the directory <code>website</code>, you can run:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>git remote add output output/
</span><span>git push output master:source
</span></code></pre>
<p>Basically, what we're doing is setting one repository as a remote of
the other, and then pushing the content to new branch there. Now the
<code>output</code> repository contains both the source and output files in
separate git branches. Pushing all the branches of this to a hosting
site means you have a complete backup of your site.</p>
<p>Now, the process of updating your site is a little more complicated
than the general Github Pages workflow of edit, commit and push since
you need to generate the site yourself.</p>
<p>First, edit your site's content files. Commit as necessary for your
peace of mind. Once you're satisfied with how the content looks
(you've been previewing and generating the site right?), it's time to
commit the new content to the output branch, and then push it to your
hosting location.</p>
<p>That's basically it. It's a simple structure, but it's not totally
straightforward of how to get it set up, and it's also a bit more work
to maintain. Of course there are some interesting tools out there to
help with this process. <a href="http://octopress.org/">Octopress 2.0</a> tries to setup this
structure automatically for you, and provides a Rakefile for helping
to automate a lot of the normal tasks like site generation, previewing
and even deployment. <a href="http://docs.getpelican.com/en/3.5.0/">Pelican</a> offers to generate a Makefile and/or
a fabfile for doing the same sorts of things. Pelican also makes use
of the really neat <a href="https://github.com/davisp/ghp-import">GHP-Import</a> project to simplify the deploying
process.</p>
<p>Now go forth and blog!</p>
Getting started with LLVM and OCaml on OS X2014-12-14T00:00:00+00:002014-12-14T00:00:00+00:00https://zefira.dev/posts/getting-started-with-llvm-and-ocaml-on-os-x/<p>I [wrote recently][llvm-mac] about my (second) experience trying to
get started using LLVM on my Macbook. Shortly after that, I became
interested in combining my interest in LLVM with my interest OCaml. As
it turns out, this was a much easier task than I anticipated.</p>
<p>[llvm-mac]: {% post_url 2014-11-19-getting-started-with-llvm-on-os-x %}</p>
<!--more-->
<p>Since my whole experiment with Vagrant was so successful for using
the C++ LLVM libraries, I immediately started doing the same thing for
my OCaml experiments. I created a <code>Vagrantfile</code> and started trying to
write a bash script to correctly provision an Ubuntu instance with the
requirements for the OCaml LLVM bindings.</p>
<p>I was stalled fairly quickly by the fact that the version of
[OPAM][opam] in the Ubuntu apt repositories is somewhat old. Old
enough that it doesn't want to talk to the official OPAM servers for
package updates. So I futzed and fiddled a bit, and looked around for
alternate ways to install OCaml.</p>
<p>Eventually I got it figured out, and started trying to install the
OCaml bindings for LLVM. It didn't go well. The compilation of the
llvm opam package errored almost immediately. But while I was looking
through the error messages, I noticed something interesting. The
errors were related to nonexistent paths, and the paths that the
package was expecting all started with <code>/usr/local/Cellar</code>. For the
non brew-savvy, this is the default location that <a href="http://brew.sh/">Homebrew</a>
uses for all it's installations.</p>
<p>I knew that homebrew had an LLVM package because of my C++
meanderings. The error messages from OPAM on Ubuntu made me think that
the OCaml LLVM package was actually expecting to be run on a Mac, or
at least to work with a <code>brew install</code>ed LLVM.</p>
<p>So I gave it a shot. I cleaned out and re-installed my LLVM installation
with <code>brew rm llvm && brew install llvm</code>. And then I simply ran <code>opam install llvm</code>. As far as I remember (it was a couple weeks ago now ;),
everything went off without a hitch.</p>
<p><How was I able to validate the installation? Looked at Kaleidoscope
tutorial?></p>
<p>Finally, I wanted to be able to use the top-level to explore the LLVM
API, but that didn't appear to work right away. After some googling, I
found an answer on Stack Overflow (of course!) that said I had to
compile a custom version of utop with the LLVM libraries linked in.</p>
<p>I did that <describe it!! got to refigure out how I did that though>,
and then was able to get the wonderful utop completion stuff to work
with the totally unfamiliar LLVM bindings in OCaml.</p>
<p>Of course, then I needed to implement a <a href="https://github.com/RadicalZephyr/postfix-ocaml">language to actually compile
with LLVM</a>. And of course I still haven't actually done
anything with LLVM...</p>
Getting Started with LLVM on OS X2014-11-19T00:00:00+00:002014-11-19T00:00:00+00:00https://zefira.dev/posts/getting-started-with-llvm-on-os-x/<p>A few weeks ago, I decided that one of the things I wanted to tackle
during my time at Hacker School was getting familiar with the LLVM
project. To that end, myself and several other Hacker Schoolers formed
an informal group to work through the official
<a href="http://llvm.org/releases/3.5.0/docs/tutorial/index.html">LLVM Kaleidoscope</a> tutorial. We made reasonable progress at
first, but as soon as we actually had to start dealing with the LLVM
tools, I started encountering problems.</p>
<p>Long story short, I ended up getting frustrated with the state of the
documentation surrounding LLVM and moving on to working on other less
upsetting projects. This last weekend though I ended up getting back
into it. I tried two different approaches.</p>
<!--more-->
<p>First, I decided to use a Vagrant supported VM to do my LLVM
setups. This was for two reasons: the fact that I do my development on
a Mac running OS X seems to be problematic when trying to install LLVM
in a global manner. This is because <em>some</em> of the LLVM tools (like
Clang) make up the default build environment on OS X. But the toolset is
insufficient if you actually want to build languages with LLVM, and
the presence of these libraries makes it... complicated to try and
install a more complete version. As Homebrew says when you try to
install LLVM via <code>brew install llvm</code>:</p>
<blockquote>
<p>Mac OS X already provides this software and installing another
version in parallel can cause all kinds of trouble.</p>
</blockquote>
<p>This is essentially the problem that I ran into a couple weeks ago
that caused me to give up on working with LLVM. This time however, I
had the insight that I wasn't solely limited to the physical machine
that I had an the one operating system I have the space to install on
it. By using <a href="https://www.vagrantup.com/">Vagrant</a> I could pretty trivially have a working Ubuntu
environment to use as my development platform for working with LLVM.</p>
<p>So that's what <a href="https://github.com/RadicalZephyr/postfix-llvm">I did</a>. The most interesting thing about
that repository is how I ended up provisioning my Vagrant
VM. To quote briefly from the <a href="https://en.wikipedia.org/wiki/Provisioning#Server_provisioning">Wikipedia Article</a>:</p>
<blockquote>
<p>... provisioning is a set of actions to prepare a server with
appropriate systems, data and software, and make it ready...</p>
</blockquote>
<p>The provisioning is done via shell scripts (that's not the interesting
part!), but instead of using <code>apt-get</code> to install all the software
that I required I mostly had to build them from scratch. But let's go
through the story end to end.</p>
<p>First off, I did try to install all the software via <code>apt-get</code>. There
was some confusion for me about which version of LLVM to install. From
the reading I'd done on the [LLVM site] previously I thought that
since 3.5 has been officially released it would be considered the
stable version. However, when you install LLVM 3.5 via <code>apt</code> (with
<code>sudo apt-get install llvm-3.5</code>) the binaries don't seem to end up
getting installed on a path location. Or rather, they are on path, but
the names are all suffixed with <code>-3.5</code>.</p>
<p>This wasn't really what I wanted, so I also tried installing the 3.4
packages which it turns out are also the default set of packages
installed if you <code>apt-get install llvm</code>. This got the right names for
the tools onto my path, so step one check. So much easier to install
than with Homebrew! Well, to install and make sure that it was available
to me anyhow.</p>
<p>Then, since I was going to be doing <code>C++</code> development, I wanted to use
<a href="http://www.cmake.org/">CMake</a> as my build generator. If you've never used CMake but you do
<code>C/C++</code> development I'd highly recommend checking it out. It allows
you to specify your build process at a very high level. As a bonus,
it can then generate the files to process that build with <a href="http://www.gnu.org/software/make/">several</a>
<a href="https://eclipse.org/">different</a> <a href="http://msdn.microsoft.com/en-us/vstudio/aa718325.aspx">kinds</a> of actual <a href="http://www.cmake.org/cmake/help/v3.0/manual/cmake-generators.7.html#id4">build systems</a>. It's way more
convenient than writing your own <code>Makefiles</code> and much more modern than
the whole <a href="https://www.gnu.org/software/autoconf/">autoconf</a> system (which I can't say much about, I've never
learned it).</p>
<p>Anyhow, CMake has another handy benefit. It has a system for
<a href="http://www.cmake.org/Wiki/CMake:How_To_Find_Libraries">finding your dependencies</a>. Not like over the internet like
<a href="http://stackoverflow.com/questions/1541771/using-maven-for-c-c-projects">some build systems</a>, but still it's better than the basic
situation in C. Since CMake has actually been around for a while there
are many standard open source projects that provide the mechanics for
finding their libraries with CMake, including - handily enough - the
LLVM project.</p>
<p>One requirement of this system is that a "module file" be somewhere on
the CMake <code>modules-path</code>. For packages that support it, this should
happen when the package is installed and the locations of the critical
library and header files are actually known. Only problem is that the
CMake packages available in the Ubuntu repositories don't actually do
this. Turns out there was a <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735592">bug report</a> early in 2014 describing the
issue. Then a long back and forth with the package maintainer (I
think) and many cases of "Hey, it should be fixed now!", followed by a
response of "No, it actually still doesn't work..."</p>
<p>Then I found the LLVM <code>apt</code> <a href="http://llvm.org/apt/">nightly builds page</a> which
possibly answers why LLVM 3.4 is still the default package on Ubuntu,
since the LLVM project considers 3.4 to be "stable" and 3.5 is the
"qualification branch." This <code>apt</code> repository seemed like a good bet
for finding an LLVM package that would properly install the necessary
CMake files, but alas, I had no such luck.</p>
<p>So instead I built LLVM from scratch. This is both easier and harder
than it sounds. Building LLVM from source is a very automated process,
and the parts that aren't automatic by default (like downloading,
checking file signatures and unpacking archives) are highly
automatable. The painful part is that VM's are always slower than a
natively installed OS. And my poor Macbook Air takes about 45 minutes
to do a full LLVM build under the native OS X. So basically once I got
everything setup I let the LLVM build run and stopped thinking about
it except to check on it every few hours.</p>
<p>I had quite a few build failures mostly related to what seem
like out of memory errors in GCC. Sometimes just restarting the
compilation helped, but I also restarted Vagrant VM and gave it a full
2GB of memory which helped a lot.</p>
<p>Once the install finished though CMake was able to find it
immediately. As pointed out earlier, the code is up on Github. I was
surprised to discover that there is no Vagrant VM that exists for
doing LLVM development, so that's going to go on my list of
projects. Ideally, that <strong>won't</strong> involve building LLVM from source in
the provisioning stage, because of the computational issues with
VM's. It might be an interesting excuse to learn how to build .deb
packages and make a PPA with a CMake that properly installs the
<code>FindLLVM.cmake</code> files...</p>
Demonstrating demonstrate.sh2014-10-30T00:00:00+00:002014-10-30T00:00:00+00:00https://zefira.dev/posts/demonstrating-demonstrate-dot-sh/<p><em>Edit 2014/12/30: I ended up rewriting demonstrate in <a href="https://github.com/RadicalZephyr/demonstrate.sh">several</a>
<a href="https://github.com/RadicalZephyr/demonstrate">different</a> languages, but the implementation that I like the
best is in <a href="https://github.com/RadicalZephyr/demonstrate.py">python</a>.</em></p>
<p>So, in the course of trying to prepare a presentation for this
Thursday (today!), I ended up creating a program called
<code>demonstrate.sh</code>. Basically, it lets you write a script, meant for an
interpreter (like bash, python, irb, coffee etc.) and then execute it
on demand.</p>
<!--more-->
<p>What do I mean by "on demand"? Well, basically it invokes the
interpreter and then reads the script file one line at a time. Then it
prints out a fake prompt based on the interpreter you specified, and
then it prints out the line of input it's processing. When you hit
enter, the script sends that line of input to the interpreter and it
then prints its output to the screen. Oh, there's one little
caveat. When <code>demonstrate.sh</code> prints the line of output, it prints it
character by character with a small randomized delay. So it looks a
bit like you're actually typing out the command.</p>
<p>That's it! It's a simple script, but it allows you to create and then
run in a repeatable way a sequence of commands that you want to
<em>demonstrate</em>.</p>
<p>There are some improvements I'd like to add. The ability to
recursively run demonstrate scripts inside of a demonstrate script
would be awesome. Currently because I'm doing everything in the
simplest possible way, and the script is run in the background running
demonstrate inside of a demonstrate script doesn't work very well. I'm
actually not entirely sure why it doesn't work, but the results are
demonstrably not what I want.</p>
<p>It would also be nice to make it so that you can insert arbitrary
commands to the interpreter, instead of only being able to run the
commands in the script. This is a particularly key feature since if
you just start another interpreter to run ad-hoc commands, you don't
have the same environment. This isn't important if your script doesn't
contain any side-effectful things, but I think that a lot of scripts
will tend to.</p>
<p>Finally, it would be really nice if the <code>demonstrate.sh</code> didn't need
to fake the prompts for the interpreters. In an ideal world, those
prompts would be transparently printed directly to the screen. I'm
sure there's a way to do it, but it probably isn't possible/easy in
<code>bash</code>.</p>
<p>When I talked to <a href="https://github.com/doy">Jesse</a> about this, he asked the critical question.
Why not just record the output? My answer was essentially because I
wanted to eventually be able to run arbitrary commands in addition to
the script commands. This is sort of a weak answer, since it depends
on a feature I haven't yet implemented!</p>
<p>But there's another answer, that I discovered today when I was playing
with the classic Unix utility <code>script</code> that does exactly what Jesse
was talking about. The difference between recording an interaction
with <code>script</code> and writing a demonstrate script is this. Recording with
<code>script</code> is <em>still live coding</em>. This is especially true if you want
to use the timing feature to play it back at the same speed that it
happened at. In contrast, writing a <code>demonstrate</code> script is the same
as writing any regular script for the interpreter you're
targeting. You are totally offline, and you can write and edit the
script directly, and then run it with demonstrate to make sure it does
exactly what you want.</p>
<p>From what I've been told about the early days of programming, the
difference between recording an interaction with <code>script</code> and writing
a <code>demonstrate</code> script is approximately the same as the difference
between writing programs on punch cards and writing programs in a text
editor on a computer. If you mess up while punching the card, you have
to start over from the beginning (barring awesome hacks
<a href="https://twitter.com/francesc/status/521602168022118400">like this</a>).</p>
<p>Yup, that's it. If you're interested, check it out
<a href="https://github.com/RadicalZephyr/demonstrate.sh">on Github</a>. Pull requests welcome :)</p>
Playing with Org-mode2014-10-27T00:00:00+00:002014-10-27T00:00:00+00:00https://zefira.dev/posts/playing-with-org-mode/<p>So, as a way to procrastinate on actually starting to write/practice
my lightning talk for Thursday, I decided that I absolutely wanted to
make a slide show using the <a href="https://github.com/relevance/org-html-slideshow">org-html-slideshow</a>, a pretty neat piece
of Clojurescript written by those smart guys at
<del>Relevance</del> Cognitect.</p>
<p>Basically, it lets you take an outline/document written in <a href="http://orgmode.org/">Org-mode</a>
(another really cool piece of software!) and put some small
annotations in it to delimit your "slides" and then you have a
document that can be viewed as either a slideshow, or a web page. It's
harder to explain in words than it is to observe, so here's the
<a href="/demos/example.html">example page</a> from their github repo, and the
<a href="/demos/example.org">org-mode document</a> that produced that page.</p>
<!--more-->
<p>Now that is pretty neat. However, the workflow needed to create one of
these is sort of a pain. From the documentation on the project page,
you're expected to create a one-off web site that has the contents of
this production folder available and then manually generate the
html document from the org-mode source. Ick.</p>
<p>My blog is made with <a href="http://jekyllrb.com/">Jekyll</a>, which is a static site generator. None
of my posts are actually written in html, instead I write them in
<a href="http://daringfireball.net/projects/markdown/syntax">Markdown</a>, and then Jekyll handling converts the markdown into
html. When it does this I can also cause it to be inserted into a page
template (i.e. layout). The upshot is, I can create a static site but
not have to duplicate all of the code to have a common set of header
info, navigation toolbar, nifty sidebar etc. Because only the unique
things are specified in each document, and the general boilerplate is
automatically included for me by Jekyll.</p>
<p>So if I'm generating html from markdown automatically with Jekyll it
feels sort of wrong to be generating html manually with an interactive
emacs command and then committing this generated html into my git
repo. It would be so much nicer if I could just get Jekyll to invoke
emacs as the translator for turning <code>.org</code> files into <code>.html</code> files.</p>
<p>Well, pursuing that line of thought today I worked with
<a href="https://github.com/wallyqs">Waldemar Quevedo</a>. He showed me his incredible org-based workflow for
everything (he also gave an awesome lightning talk on it the first
week of Hacker School!). While it looks really cool, it's sort
heavy-weight for what I want to do. Frankly, I'm really comfortable
writing markdown, so I'll probably continue to write most of my posts
in it. But org-html-slideshow is exciting enough that I'd be willing
to write my presentations in Org-mode so I can utilize it.</p>
<p>I won't attempt to explain Wally's setup, but he's got some
<a href="https://github.com/wallyqs/org-ruby">pretty</a> <a href="https://github.com/eggcaker/jekyll-org">cool</a> <a href="https://github.com/punchagan/org-hyde">stuff</a> going on. In essence
though, he uses a ruby gem to parse the org syntax and transform it
into markdown. As of this writing though, it doesn't support the tags
feature that org-html-slideshow depends on for it's functionality.</p>
<p>While showing Wally the slideshow thing got him excited to try and
integrate it into his workflow, I started poking around and looking at
how to integrate it into my site. The solution I decided on was to
make <code>.org</code> files a generic member of the Jekyll source files. This
means that they'll still use a YAML frontmatter to mark that they need
to be transformed. This means that I need some way to transform a body
of org text into just the html content that it represents. Normally
when you generate an html page from org-mode, you get a complete html
page, <code>html</code>, <code>body</code> tags and all. What I needed was a way to invoke
the org->html transformation using the Elisp code in org-mode from
Jekyll.</p>
<p>With Wally's help, I was able to get this far towards that goal. This
is the elisp that will continuously read from the minibuffer until an
error occurs, and then transform that text using the org->html backend,
and finally print out the generated html.</p>
<pre data-lang="lisp" style="background-color:#393939;color:#dedede;" class="language-lisp "><code class="language-lisp" data-lang="lisp"><span>(</span><span style="color:#fffd87;">progn
</span><span> (</span><span style="color:#fffd87;">require</span><span> 'org)
</span><span> (</span><span style="color:#fffd87;">progn
</span><span> (condition</span><span style="color:#ececec;">-</span><span style="color:#fed6af;">case </span><span style="font-weight:bold;color:#d6d6ae;">nil
</span><span> (</span><span style="color:#fed6af;">while </span><span style="font-weight:bold;color:#d6d6ae;">t
</span><span> (insert (</span><span style="color:#fffd87;">read</span><span style="color:#ececec;">-</span><span style="color:#fffd87;">string </span><span style="color:#d6d6d680;">""</span><span>) </span><span style="color:#d6d6d680;">"</span><span style="font-weight:bold;color:#ff8080;">\n</span><span style="color:#d6d6d680;">"</span><span>))
</span><span> (</span><span style="color:#fffd87;">error </span><span style="font-weight:bold;color:#d6d6ae;">nil</span><span>))
</span><span> (</span><span style="color:#fffd87;">set</span><span style="color:#ececec;">-</span><span>buffer
</span><span> (org</span><span style="color:#ececec;">-</span><span style="color:#fffd87;">export</span><span style="color:#ececec;">-</span><span style="color:#fed6af;">to</span><span style="color:#ececec;">-</span><span>buffer 'html </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">*Org HTML Export*</span><span style="color:#d6d6d680;">"
</span><span> </span><span style="font-weight:bold;color:#d6d6ae;">nil nil nil t nil
</span><span> (</span><span style="color:#fffd87;">lambda </span><span>() </span><span style="font-weight:bold;color:#d6d6ae;">t</span><span>)))
</span><span> (message </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">%s</span><span style="color:#d6d6d680;">" </span><span>(buffer</span><span style="color:#ececec;">-</span><span style="color:#fffd87;">string</span><span>))))
</span></code></pre>
<p>And this is how you invoke the transformation of a file from org to
html. The <code>$emacs</code> is because I use <a href="http://emacsformacosx.com/">Emacs for Mac OSX</a> which means
that my emacs executable is buried inside of a mac <code>.app</code> file/folder
thing. Pipe a file into this script and if it's valid org syntax,
you'll get the corresponding html.</p>
<pre data-lang="bash" style="background-color:#393939;color:#dedede;" class="language-bash "><code class="language-bash" data-lang="bash"><span>tail -n+4 hello.org </span><span style="color:#ececec;">| </span><span style="color:#fed6af;">$</span><span>emacs --batch --eval </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">(progn (require 'org) (progn (condition-case nil (while t (insert (read-string </span><span style="font-weight:bold;color:#ff8080;">\"\"</span><span style="color:#d68686;">) </span><span style="font-weight:bold;color:#ff8080;">\"\\</span><span style="color:#d68686;">n</span><span style="font-weight:bold;color:#ff8080;">\"</span><span style="color:#d68686;">)) (error nil)) (set-buffer (org-export-to-buffer 'html </span><span style="font-weight:bold;color:#ff8080;">\"</span><span style="color:#d68686;">*Org HTML Export*</span><span style="font-weight:bold;color:#ff8080;">\"</span><span style="color:#d68686;"> nil nil nil t nil (lambda () t))) (message </span><span style="font-weight:bold;color:#ff8080;">\"</span><span style="color:#d68686;">%s</span><span style="font-weight:bold;color:#ff8080;">\"</span><span style="color:#d68686;"> (buffer-string))))</span><span style="color:#d6d6d680;">"
</span></code></pre>
<p>Now the only thing that's left is to write a Jekyll plugin to do this
invocation to produce the final output. <del>This is what I came up with
as a first attempt. Of course, it doesn't actually work for me,
because of the dumb location of my emacs executable... Oh well, that's
for tomorrow.</del></p>
<p><strong>Edit 10/28</strong>: So I spent the first half of today fixing this up.
Turned out to be more involved than I suspected. I had to evolve the
emacs lisp command enough that I actually made a separate git repo for
it. I fixed the problem of locating a proper emacs binary by deferring
to an appropriately named environment variable. I originally
implemented this shortcut by letting the user specify the location of
emacs in their <code>_config.yml</code>. But then I realized, this is not a
configuration that happens per-site, it's per machine so the site
config file is totally inappropriate. It's kind of annoying though
since if I forget to specify this environment variable then my site
generation won't work...</p>
<p>Anyhow, it works and as proof I made a version of the example
presentation that is <a href="/demos/example-plugin.html">generated by my plugin</a>, not by
hand. You don't have to take my word for it, but you will have to look
at the <a href="zephyrizing">git repository for this site</a>.</p>
<p>And here is the final code for my org-mode plugin generator:</p>
<p>{% include_code "Org Converter" lang:ruby ../plugins/org_converter.rb %}</p>
On the Importance of Distinguishing Between Stable and Development Documentation2014-10-26T00:00:00+00:002014-10-26T00:00:00+00:00https://zefira.dev/posts/on-the-importance-of-distinguishing-between-stable-and-development-documentation/<p>A few weeks ago I wrote about [Why Master Should Be Stable][stable]. I
spent most of that post explaining what had actually happened to me,
but not really talking about the fundamental underlying difficulty.
Well, now that I've encountered this issue again in a different
context, maybe it's time to dive in a bit and examine what I call the
Doc's Mismatch problem, and how it can be avoided.</p>
<p>[stable]: {% post_url 2014-10-07-dot-dot-dot-and-this-is-why-master-should-be-stable %}</p>
<!--more-->
<p>As the title hints, this issue is really more about being able to
match the documentation of a project to a particular version. The
problem with Flaskr was that when you went searching for the
documentation, you most easily found the documentation for the
currently in development branch. This isn't a problem when no major
changes have been made, but when there are major changes this becomes
an issue.</p>
<p>Part of the reason that I'm writing this blog post is because last
week, while I was going through the <a href="http://llvm.org">LLVM</a> project's <a href="http://llvm.org/docs/tutorial/index.html">tutorial</a> I ran
into the exact same issue! But I didn't spot that this version
mismatch was the issue until after I had found another workaround.</p>
<p>Not to belabor the point, but basically the LLVM tutorial linked above
is actually the version of the tutorial written for the upcoming 3.6
release. Since there are several breaking changes to the API occurring
in this release, when you have installed LLVM 3.5 the tutorial appears
to be rather broken.</p>
<p>This problem is sort of ironic as the majority of open source software
out there (most definitely including all of my projects :) have
little-to-no documentation, or what there is, is hopelessly
out-of-date. In contrast, the issue that I had with both Flaskr and
LLVM was actually a case of the most easily available documentation
being TOO up-to-date.</p>
<h3 id="how-can-we-do-better"><a href="#how-can-we-do-better" aria-label="Anchor link for: how-can-we-do-better"><i class="fa fa-link" aria-hidden="true" ></i></a>
How can we do better?</h3>
<p>Let's contrast these experiences with a project that gets it right:
<a href="https://www.djangoproject.com/">Django</a>. First off, I just want to praise the Django project for
their excellent documentation in general. There is a lot of it, and it
is fairly well-written and clear. There is also a good mixture of the
<a href="http://jacobian.org/writing/great-documentation/what-to-write/">three types of documentation</a> that every project
needs.<code></praise></code></p>
<p>Okay, now lets look at how Django avoids the Docs Mismatch
problem. First, look at the URL for the main
<a href="https://docs.djangoproject.com/en/1.7/">Django Documentation page</a>:
<code>https://docs.djangoproject.com/en/1.7/</code>. Notice that <code>1.7</code> at the
end? That indicates what version of Django these docs are for. At the
time of this writing, 1.7 is the latest <strong>stable</strong> release of
Django. So, right away Django is doing well. By default the link on
the Django homepage takes you to this page. And you can bet, that
when the next stable release of Django comes out, that homepage will
be updated to point at the 1.8 version.</p>
<p>However, there is still more that Django is getting fundamentally
right here. By embedding the version of the documentation in the URL,
they can host multiple different versions of the documentation. And
they do! This means that if you're forced to use an older version of
Django for whatever reason (legacy project, no install privileges
etc.) you can still go online to the Django website and find the
documentation for the version you're using. To make this even easier,
their documentation pages all have a permanent float in the bottom
right hand corner that shows the current version of the docs that
you're looking at, and has links to other recent versions of the
documentation. How cool is that!</p>
<p>If you look at the links in that little floated element, you'll notice
a non-numbered version is available <code>dev</code>. This is the documentation
for the [current development head][djangodevdoc]. More importantly,
notice that this is the equivalent of the documentation that was
easiest for me to find when dealing with both LLVM and Flaskr. But
with Django I had to specifically go looking for it. Why? Because most
people don't want to read the docs for the development head. And if
they do, they know it!</p>
<p>Django gets one more thing really right here. When you go to the dev
docs page, they automatically add a float at the top of the screen, in
an eye-catching color that effectively says: "These are the dev
docs. Are you sure you want to be reading them? They're probably
different."</p>
<p>This last feature is really the key to avoiding the Docs Mismatch
problem. The development docs are fairly rarely what people want to be
reading when they first pick up a new project. So they shouldn't be
the easiest (or only!) documentation to find for your project.</p>
<h3 id="how-do-we-fix-it"><a href="#how-do-we-fix-it" aria-label="Anchor link for: how-do-we-fix-it"><i class="fa fa-link" aria-hidden="true" ></i></a>
How do we fix it?</h3>
<p>The ideal situation is for your documentation to be structured like
Django's: multiple versions available, notifications when viewing the
dev version. The good news is that this is much easier to achieve
thany you might expect. <a href="https://readthedocs.org/">Read The Docs</a> makes it trivial to have this
kind of setup for your open source project. In fact, Django actually
uses Read The Docs for their own documentation. I highly recommend
checking out their page and seeing just how easy it is to set up for
your own projects. Of course, once you have this awesome system the
only issue is actually writing all the awesome documentation to
populate it...</p>
<h3 id="other-issues"><a href="#other-issues" aria-label="Anchor link for: other-issues"><i class="fa fa-link" aria-hidden="true" ></i></a>
Other issues</h3>
<p>A primary problem with the Docs Mismatch issue is that the people who
notice it are typically people just wanting to use the project, not
the people developing it. This is especially problematic because these
people are also the most easily discouraged from using a project.</p>
<p>Another problem is that often - especially in older projects - fixing
the Docs Mismatch issue will require either access to a server, or
changes in a release deployment process or something similar. None of
these things are possible for a beginner in a project to do. This
means that the people the most motivated to fix the issue, are the
least able to do anything about it.</p>
Adventures with Clojure macros2014-10-14T00:00:00+00:002014-10-14T00:00:00+00:00https://zefira.dev/posts/adventures-with-clojure-macros/<p>So, this is the second day of my second week of <a href="https://www.hackerschool.com/">Hacker School</a>. I'm
finally starting to feel like I'm getting into the
groove.</p>
<p>So I tackled <a href="http://mitpress.mit.edu/books/design-concepts-programming-languages">DCPL</a> in Clojure once again. I found two sort of scary
things. One, although my code was in a git repo, it wasn't pushed to
Github. Two, there was a big hairy macro that had apparently replaced
the core of my postfix program, but it wasn't checked in!!! Bad past
Zefira! Very bad, no good software practices...</p>
<!--more-->
<p>Anyhow, so I cleaned things up, and pushed up to Github (it's
<a href="https://github.com/RadicalZephyr/postfix-clj">here</a> FYI), and then I started playing with that macro
again. Turns out it pretty much worked. After a bit of mucking around
I managed to get it going. This quick success buoyed my spirits and I
thought, why don't I write another... replacing repetitive code is
awesome etc.</p>
<p>Then there I wrote some tests, and it turns out that it didn't
actually work. Repeat for the better part of two hours with testing
getting slowly better and my implementation getting slowly more
correct. But, there were lots of little successes throughout, so I
didn't get discouraged, and the thrill of surmounting the difficulties
I encountered gave me a warm glow when I overcame them (macro writing
is hard! I know, because <a href="http://www.paulgraham.com/avg.html">Paul Graham told me so</a> ;).</p>
<p>Anyhow, my success at writing this one macro made me want to write
another. And I immediately saw an opportunity! All of the binary
math operations for postfix are going to have the exact same form,
even with the last macro I wrote. etc. etc. about that macro, goes
pretty smooth.</p>
<p>Anyhow, so that went alright and now there are a bunch of tests I want
to write, just to validate that my macro is producing reasonable code
and continues to do so.</p>
<p>But... these tests are going to be SOOOO similar!!! Are you thinking
what I'm thinking? Awww, yeah! Time for another macro.</p>
<p>So I start writing a macro to generate some simple tests for me.</p>
<p>All the tests have this general form:</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>(</span><span style="color:#fffb9d;">deftest </span><span style="color:#fffd87;">subcommand-test
</span><span> (testing </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">Sub command</span><span style="color:#d6d6d680;">"
</span><span> (is (= (sub-cmd [</span><span style="font-weight:bold;color:#87d6d5;">1 2</span><span>])
</span><span> [</span><span style="font-weight:bold;color:#87d6d5;">-1</span><span>]))
</span><span> (is (= (sub-cmd [</span><span style="font-weight:bold;color:#87d6d5;">0 3 2</span><span>])
</span><span> [</span><span style="font-weight:bold;color:#87d6d5;">0 1</span><span>]))
</span><span> (is (thrown-with-msg? clojure.lang.ExceptionInfo
</span><span> </span><span style="color:#ececec;">#</span><span style="color:#d6d6d680;">"</span><span style="color:#c76f41;">sub: not enough values on the stack</span><span style="color:#d6d6d680;">"
</span><span> (sub-cmd [</span><span style="font-weight:bold;color:#87d6d5;">1</span><span>])))))
</span></code></pre>
<p>So I start my macro working on that basic outline. Over the course of
another hour or so, I tweak and work it up to be this:</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>(</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">defbinary-op-test </span><span>[cmd-name op]
</span><span> (let [cmd-name (name cmd-name)
</span><span> fn-name (symbol (str cmd-name </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">-cmd</span><span style="color:#d6d6d680;">"</span><span>))
</span><span> test-args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]
</span><span> op-result (vector (apply op (reverse test-args)))]
</span><span> (prn </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">test-args:</span><span style="color:#d6d6d680;">"</span><span> test-args </span><span style="color:#d6d6d680;">"</span><span style="font-weight:bold;color:#ff8080;">\n</span><span style="color:#d68686;">op result:</span><span style="color:#d6d6d680;">"</span><span> op-result)
</span><span> </span><span style="color:#ececec;">`</span><span>(testing </span><span style="color:#ececec;">~</span><span>(str cmd-name </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;"> command</span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (is (</span><span style="color:#ececec;">~'</span><span>= (</span><span style="color:#ececec;">~</span><span>fn-name </span><span style="color:#ececec;">~</span><span>test-args)
</span><span> </span><span style="color:#ececec;">~</span><span>op-result))
</span><span> (is (</span><span style="color:#ececec;">~'</span><span>= (</span><span style="color:#ececec;">~</span><span>fn-name </span><span style="color:#ececec;">~</span><span>(into [</span><span style="font-weight:bold;color:#87d6d5;">0</span><span>] test-args))
</span><span> </span><span style="color:#ececec;">~</span><span>(into [</span><span style="font-weight:bold;color:#87d6d5;">0</span><span>] op-result)))
</span><span> (is (</span><span style="color:#ececec;">~'</span><span>thrown-with-msg? clojure.lang.ExceptionInfo
</span><span> (re-pattern </span><span style="color:#ececec;">~</span><span>(str cmd-name
</span><span> </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">: not enough values on the stack</span><span style="color:#d6d6d680;">"</span><span>))
</span><span> (</span><span style="color:#ececec;">~</span><span>fn-name [</span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]))))))
</span></code></pre>
<p>Okay, it looks like a monster. But it's pretty straightforward.
Since my binary ops are actually implemented by the core functions
they represent I'm not concerned with the correctness or with testing
that they work per se. I'm more concerned with the vectors going in
and coming out. So I figure, I'll use the same numbers for all of
them, and have the macro generate the result of actually applying the
given binary operator to the data I've hardcoded.</p>
<p>That's whats going inside the <code>let</code> with <code>test-args</code> and <code>op-result</code>.</p>
<p>There are a couple of other tricky things I found out. For instance,
the <code>is</code> testing macro is looking for exactly the symbol <code>=</code> not for
<code>clojure.core/=</code> or for <code>postfix.core-test/thrown-with-msg?</code>. I know,
because those are the things <code>defbinary-op-test</code> produced before I put
in the unquoted-quote <code>~'</code> to stop the automatic namespace resolution,
as described in the <a href="http://www.manning.com/fogus2/">Joy of Clojure</a>.</p>
<p>But then, after I debugged all of these little problems, I discovered
that, though my tests were being generated syntactically correctly,
they were failing!</p>
<p>Now, given the way that I constructed the macro, this didn't seem
possible. I mean, I'm literally testing that my command, which uses
the given operator, produces the same thing as that operator.</p>
<p>So I dig in to make sure my macro is doing the right stuff, and pull
out <code>(clojure.pprint/pprint (macroexpand-1 '(defbinary-op-test add +)))</code></p>
<p>which produces this:</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>(clojure.test/testing
</span><span> </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">add command</span><span style="color:#d6d6d680;">"
</span><span> (clojure.test/is (= (add-cmd [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]) [</span><span style="font-weight:bold;color:#87d6d5;">3</span><span>]))
</span><span> (clojure.test/is (= (add-cmd [</span><span style="font-weight:bold;color:#87d6d5;">0 3 1</span><span>]) [</span><span style="font-weight:bold;color:#87d6d5;">0 3</span><span>]))
</span><span> (clojure.test/is
</span><span> (thrown-with-msg?
</span><span> clojure.lang.ExceptionInfo
</span><span> (clojure.core/re-pattern </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">add: not enough values on the stack</span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (add-cmd [</span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]))))
</span></code></pre>
<p>Okay, okay everything looks good... Wait a second... Did the macro
produce the result that 3 + 1 is 3? Okay, let's take a closer look at
that macro again. Specifically, the two relevant portions of the let:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>test-args [3 1]
</span><span>op-result (vector (apply op (reverse test-args)))]
</span></code></pre>
<p>and the part where <code>op-result</code> is used:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>(is (~'= (~fn-name ~test-args)
</span><span> ~op-result))
</span></code></pre>
<p>Okay, everthing looks pretty kosher, but to be sure, let's try it out
in the repl, with the appropriate values filled in.</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>(let [op +
</span><span> test-args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]]
</span><span> (vector (apply op (reverse test-args))))
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> => [4]
</span></code></pre>
<p>That's correct. But that macroexpand clearly shows a <code>[3]</code> where that
<code>[4]</code> should be.</p>
<p>Okay, maybe I didn't reproduce something correctly in the REPL. Let's
put some print statements in the macro before the expansion</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>(</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">defbinary-op-test </span><span>[cmd-name op]
</span><span> (let [cmd-name (name cmd-name)
</span><span> fn-name (symbol (str cmd-name </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">-cmd</span><span style="color:#d6d6d680;">"</span><span>))
</span><span> test-args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]
</span><span> op-result (vector (apply op (reverse test-args)))]
</span><span> (prn </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">test-args:</span><span style="color:#d6d6d680;">"</span><span> test-args </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">op result:</span><span style="color:#d6d6d680;">"</span><span> op-result)
</span><span> </span><span style="color:#ececec;">`</span><span>(testing </span><span style="color:#ececec;">~</span><span>(str cmd-name </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;"> command</span><span style="color:#d6d6d680;">"</span><span>)
</span><span> (is (</span><span style="color:#ececec;">~'</span><span>= (</span><span style="color:#ececec;">~</span><span>fn-name </span><span style="color:#ececec;">~</span><span>test-args)
</span><span> </span><span style="color:#ececec;">~</span><span>op-result))
</span><span> (is (</span><span style="color:#ececec;">~'</span><span>= (</span><span style="color:#ececec;">~</span><span>fn-name </span><span style="color:#ececec;">~</span><span>(into [</span><span style="font-weight:bold;color:#87d6d5;">0</span><span>] test-args))
</span><span> </span><span style="color:#ececec;">~</span><span>(into [</span><span style="font-weight:bold;color:#87d6d5;">0</span><span>] op-result)))
</span><span> (is (</span><span style="color:#ececec;">~'</span><span>thrown-with-msg? clojure.lang.ExceptionInfo
</span><span> (re-pattern </span><span style="color:#ececec;">~</span><span>(str cmd-name
</span><span> </span><span style="color:#d6d6d680;">"</span><span style="color:#d68686;">: not enough values on the stack</span><span style="color:#d6d6d680;">"</span><span>))
</span><span> (</span><span style="color:#ececec;">~</span><span>fn-name [</span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]))))))
</span></code></pre>
<p>When I reload the namespace I get this:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>:reloading (postfix.core postfix.core-test)
</span><span>"test-args:" [3 1] "op result:" [3]
</span><span>"test-args:" [3 1] "op result:" [3]
</span><span>"test-args:" [3 1] "op result:" [3]
</span><span>"test-args:" [3 1] "op result:" [3]
</span></code></pre>
<p>Okay, that's clearly what's happening but it's still really strange
that the <code>[3]</code> is showing up at all. So how about a
<a href="https://en.wikipedia.org/wiki/Minimal_Working_Example">minimal working example</a> (or rather, not-working in this case)?</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>(</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">deftestaddproblem </span><span>[op]
</span><span> (let [test-args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]
</span><span> op-result (vector (apply op (reverse test-args)))]
</span><span> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>test-args </span><span style="color:#ececec;">~</span><span>op-result]))
</span></code></pre>
<p>Then macroexpanding it:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>(clojure.pprint/pprint (macroexpand-1 '(deftestaddproblem +)))
</span><span>[[3 1] [3]]
</span></code></pre>
<p>Just for sanity, lets make sure it's the right version of <code>+</code>:</p>
<pre style="background-color:#393939;color:#dedede;"><code><span>(clojure.pprint/pprint (macroexpand-1 '(deftestaddproblem clojure.core/+)))
</span><span>[[3 1] [3]]
</span></code></pre>
<p>What is going on!?!?!?!?! Okay, back to the REPL. This time, a raw
REPL from <code>lein repl</code> in my home directory.</p>
<pre data-lang="clojure" style="background-color:#393939;color:#dedede;" class="language-clojure "><code class="language-clojure" data-lang="clojure"><span>user=> (</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">dotest </span><span>[op]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> (let [test-args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> op-result (vector (apply op (reverse test-args)))]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>test-args </span><span style="color:#ececec;">~</span><span>op-result]))
</span><span style="color:#ececec;">#'</span><span>user/dotest
</span><span>user=> (dotest +)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] [</span><span style="font-weight:bold;color:#87d6d5;">3</span><span>]]
</span><span>user=> (dotest -)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] [</span><span style="font-weight:bold;color:#87d6d5;">3</span><span>]]
</span><span>user=> (dotest *)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] [</span><span style="font-weight:bold;color:#87d6d5;">3</span><span>]]
</span><span>user=> (dotest /)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] [</span><span style="font-weight:bold;color:#87d6d5;">3</span><span>]]
</span><span>
</span><span style="color:#a0cfa1;">;;;</span><span style="color:#87ae86;"> And without using the macro preprocessing
</span><span>user=> (vector (apply + (reverse [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])))
</span><span>[</span><span style="font-weight:bold;color:#87d6d5;">4</span><span>]
</span><span>user=> (vector (apply - (reverse [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])))
</span><span>[</span><span style="font-weight:bold;color:#87d6d5;">-2</span><span>]
</span><span>user=> (vector (apply * (reverse [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])))
</span><span>[</span><span style="font-weight:bold;color:#87d6d5;">3</span><span>]
</span><span>user=> (vector (apply / (reverse [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])))
</span><span>[</span><span style="font-weight:bold;color:#87d6d5;">1/3</span><span>]
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> Removing more complexity...
</span><span>user=> (</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">dotest2 </span><span>[op]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> (let [test-args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> op-result (apply op test-args)]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>test-args </span><span style="color:#ececec;">~</span><span>op-result]))
</span><span style="color:#ececec;">#'</span><span>user/dotest2
</span><span>user=> (dotest2 +)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] </span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]
</span><span>user=> (dotest2 -)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] </span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]
</span><span>user=> (dotest2 *)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] </span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]
</span><span>user=> (dotest2 /)
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>] </span><span style="font-weight:bold;color:#87d6d5;">1</span><span>]
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> And again outside of the macro
</span><span>user=> (apply + [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span style="font-weight:bold;color:#87d6d5;">4
</span><span>user=> (apply - [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span style="font-weight:bold;color:#87d6d5;">2
</span><span>user=> (apply * [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span style="font-weight:bold;color:#87d6d5;">3
</span><span>user=> (apply / [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span style="font-weight:bold;color:#87d6d5;">3
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> More experiments
</span><span>user=> (</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">dotest3 </span><span>[]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> (let [+-res (apply + [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span> </span><span style="color:#ececec;">#_</span><span>=> --res (apply - [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span> </span><span style="color:#ececec;">#_</span><span>=> *-res (apply * [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])
</span><span> </span><span style="color:#ececec;">#_</span><span>=> div-res (apply / [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>])]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>+-res </span><span style="color:#ececec;">~</span><span>--res </span><span style="color:#ececec;">~</span><span>*-res </span><span style="color:#ececec;">~</span><span>div-res]))
</span><span style="font-weight:bold;color:#87d6d5;">3</span><span style="color:#ececec;">#'</span><span>user/dotest3
</span><span>user=> (dotest3)
</span><span>[</span><span style="font-weight:bold;color:#87d6d5;">4 2 3 3</span><span>]
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> And another
</span><span>user=> (</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">dotest4 </span><span>[]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> (let [args [</span><span style="font-weight:bold;color:#87d6d5;">3 1</span><span>]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> +-res (apply + args)
</span><span> </span><span style="color:#ececec;">#_</span><span>=> --res (apply - args)
</span><span> </span><span style="color:#ececec;">#_</span><span>=> *-res (apply * args)
</span><span> </span><span style="color:#ececec;">#_</span><span>=> div-res (apply / args)]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>+-res </span><span style="color:#ececec;">~</span><span>--res </span><span style="color:#ececec;">~</span><span>*-res </span><span style="color:#ececec;">~</span><span>div-res]))
</span><span style="color:#ececec;">#'</span><span>user/dotest4
</span><span>user=> (dotest4)
</span><span>[</span><span style="font-weight:bold;color:#87d6d5;">4 2 3 3</span><span>]
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> This might be important...
</span><span>user=> (</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">dotest5 </span><span>[op args]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> (let [res (apply op args)]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>op </span><span style="color:#ececec;">~</span><span>args </span><span style="color:#ececec;">~</span><span>res]))
</span><span style="color:#ececec;">#'</span><span>user/dotest5
</span><span>user=> (dotest5 + [</span><span style="font-weight:bold;color:#87d6d5;">3 1 2</span><span>])
</span><span>ArityException Wrong number of args (</span><span style="font-weight:bold;color:#87d6d5;">1</span><span>) passed to: Symbol clojure.lang.Compiler.macroexpand1 (Compiler.java:6557)
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> Or not... may just be a problem
</span><span>user=> (</span><span style="color:#fffb9d;">defmacro </span><span style="color:#fffd87;">dotest5 </span><span>[op args]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> (let [res (apply op args)]
</span><span> </span><span style="color:#ececec;">#_</span><span>=> </span><span style="color:#ececec;">`</span><span>[</span><span style="color:#ececec;">~</span><span>args </span><span style="color:#ececec;">~</span><span>res]))
</span><span style="color:#ececec;">#'</span><span>user/dotest5
</span><span>user=> (dotest5 + [</span><span style="font-weight:bold;color:#87d6d5;">1 2</span><span>])
</span><span>[[</span><span style="font-weight:bold;color:#87d6d5;">1 2</span><span>] </span><span style="font-weight:bold;color:#87d6d5;">2</span><span>]
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> For sanity checking purposes, since apply can only be used with
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> functions
</span><span>user=> (fn? +)
</span><span style="font-weight:bold;color:#d6d6ae;">true
</span><span>user=> (fn? -)
</span><span style="font-weight:bold;color:#d6d6ae;">true
</span><span>user=> (fn? *)
</span><span style="font-weight:bold;color:#d6d6ae;">true
</span><span>user=> (fn? /)
</span><span style="font-weight:bold;color:#d6d6ae;">true
</span><span>
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> And as Kevin Lynagh helpfully pointed out to me, macros don't
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> evaluate their arguments... Time to re-read the section on macros
</span><span style="color:#a0cfa1;">;;</span><span style="color:#87ae86;"> in Joy of Clojure
</span><span>user=> (apply </span><span style="color:#ececec;">'</span><span>+ [</span><span style="font-weight:bold;color:#87d6d5;">1 2</span><span>])
</span><span style="font-weight:bold;color:#87d6d5;">2
</span><span>user=> (apply (eval </span><span style="color:#ececec;">'</span><span>+) [</span><span style="font-weight:bold;color:#87d6d5;">1 2</span><span>])
</span><span style="font-weight:bold;color:#87d6d5;">3
</span><span>
</span></code></pre>
Hacker School: Week 1 Review2014-10-11T00:00:00+00:002014-10-11T00:00:00+00:00https://zefira.dev/posts/hacker-school-week-1-review/<p>So, I can't sleep, and it's the end of my first week at
<a href="https://www.hackerschool.com/">Hacker School</a>, so I thought I'd do some reflecting on what I did,
why I'm here and similar topics.</p>
<!--more-->
<h2 id="week-review"><a href="#week-review" aria-label="Anchor link for: week-review"><i class="fa fa-link" aria-hidden="true" ></i></a>
Week Review</h2>
<h3 id="monday"><a href="#monday" aria-label="Anchor link for: monday"><i class="fa fa-link" aria-hidden="true" ></i></a>
Monday</h3>
<p>This week has been a blur in some ways and incredibly long in others.
I started out on Monday with an interesting but ultimately abortive
attempt to pair with Alyssa Carter on her observational type system
implementation in Idris. Given that I know pretty much nothing about
either type theory or Idris, this outcome was probably somewhat
inevitable. In retrospect, while it was interesting to get some
exposure to the ideas and concepts that she talked about, I probably
should have admitted more freely when things she was saying were going
over my head. This is something that I've always struggled with, and I
think that Hacker School is a great place to try and tackle getting
over it.</p>
<p>The rest of Monday I spent flailing. And not really with any code;
mostly with just getting familiarized with Zulip and the HS (physical)
environment, and turning off most notifications in Zulip. I
discovered an interesting thing about myself that I hadn't been fully
aware of previously: when there are constant notifications happening
(like when you're highly subscribed to Zulip and desktop notifications
are turned on), I get really stressed out on a subconscious level.</p>
<p>It's sort of similar to how I start to feel inexplicably rushed when
my bladder is getting full, but before I actually notice the urge to
pee.</p>
<p>Anyhow, during my flailing I also poked a bit at my
<a href="https://github.com/RadicalZephyr/snake-puzzle-solver">snake-puzzle-solver</a> project, and discovered that I'd left it in a
sort of broken-ish state.</p>
<h3 id="tuesday"><a href="#tuesday" aria-label="Anchor link for: tuesday"><i class="fa fa-link" aria-hidden="true" ></i></a>
Tuesday</h3>
<p>On Tuesday I felt much calmer. I'm having a hard time remembering now
what exactly I worked on. I believe it was working through
<em>[Real World OCaml]</em> though. I didn't get very far, but I also spent
a bit of time [helping Sammy out][masterstable], and fixing up this
blog with Octopress.</p>
<p>[masterstable]: {% post_url 2014-10-07-dot-dot-dot-and-this-is-why-master-should-be-stable %}
[Real World OCaml]: https://realworldocaml.org/</p>
<h3 id="wednesday"><a href="#wednesday" aria-label="Anchor link for: wednesday"><i class="fa fa-link" aria-hidden="true" ></i></a>
Wednesday</h3>
<p>Through some poor decision-making Tuesday evening (i.e. eating two
slices of pizza, and also drinking a not inconsiderable amount) I was
either very gluten-sick or very, very hungover on Wednesday. No good,
very bad feels were had all day. At the time I was fairly convinced
that it was the pizza. In retrospect, the symptoms were very
hangover-like. Perhaps I just got much drunker than I thought I did?
Hard to say.</p>
<h3 id="thursday"><a href="#thursday" aria-label="Anchor link for: thursday"><i class="fa fa-link" aria-hidden="true" ></i></a>
Thursday</h3>
<p>On Thursday I powered through the first third or so of <em>Real World
OCaml</em>. That was awesome. OCaml is a fascinating language, and even
though it has perhaps more than it's share of syntactic quirks it's
been fun learning it. On a slightly meta-level I was sort of
dissatisfied throughout the day because of my (mostly self-imposed)
isolation. Not using Zulip much - because of how distracting it is -
makes me feel a bit cut-off from some of the social aspects of HS. And
I still haven't really found the courage to seek out somebody to pair
with. Another thing I want to work on. Also, the fact that I was doing
something so ordinary and mundane as simply reading a book made me
feel bad. Reading a book on a new language and working through the
code examples is something that I can, have, and - most likely - will
do again at home. So why am I "wasting" my time at Hacker School,
precious, precious time that it is, doing it here? Particularly since
I'm doing it alone!!</p>
<p>I don't really have answers to most of these dissatisfactions. Most of
them are probably just my social awkwardness manifesting, and the last
one sounds suspiciously like a strange form of Imposter Syndrome.</p>
<p>I stayed late in the evening though and started working on a neat
project that occurred to me when I suggested that another HS'er work
through Phil Nelson's minishell - implementing the minishell in
OCaml!! And even more importantly, I achieved a real-feeling amount of
success in the project! I was able, without any real documentation
(couldn't find accurate docs!) to utilize the OCaml Unix module well
enough to create the equivalent of the <code>minishell.c</code> that Nelson gives
out as the basis for the shell, and to complete the equivalent of
assignment 1.</p>
<h3 id="friday"><a href="#friday" aria-label="Anchor link for: friday"><i class="fa fa-link" aria-hidden="true" ></i></a>
Friday</h3>
<p>I somehow gained the impression that showing up at the space was less
expected on Friday. Now I'm really not sure where/how I got that
idea. But I'm also totally unsure if it's actually wrong... >:(</p>
<p>I did however use the day to catch up on some sleep, do some errands
(digital ones anyhow) and go hang out with a friend I haven't seen in
literally years. So, overall win even if I did totally miss out on a
HS day. But again, it makes me feel sort of disconnected to be not at
the space when others are.</p>
<h2 id="project-ideas"><a href="#project-ideas" aria-label="Anchor link for: project-ideas"><i class="fa fa-link" aria-hidden="true" ></i></a>
Project Ideas</h2>
<p>Given some of my meta-dissatisfaction that I felt on Thursday
regarding what I've been working on, I thought I might try to list
some other projects that seem more "Hacker School worthy".</p>
<p>Looking to the <a href="https://www.hackerschool.com/manual#sec-principles">three traits of an awesome Hacker Schooler</a>,
I started thinking about what are some challenging (strive for
greatness) problems that I could approach with rigor.</p>
<ul>
<li>Dive into the bug in auto-fill in emacs, try to fix it</li>
<li>Pursue learning more graphics stuff (since I've been sort of
intimidated by it for a while). Project-wise, it might be
interesting to try and discover what sort of work it would take to
update Emacs' rendering engine.</li>
<li>Try to dive into the Boomerang decompiler's source code and
understand it</li>
</ul>
<p>And some other random things I want to do (but might not be feasible):</p>
<ul>
<li>Implement a language that has lispy-prefix syntax but OCaml
semantics; in particular that has type-inferencing</li>
<li>To that end, implement the type inferencing engine that powers OCaml
type inferences</li>
<li>Port Boomerang to use the LLVM architecture</li>
</ul>
<h2 id="final-thoughts"><a href="#final-thoughts" aria-label="Anchor link for: final-thoughts"><i class="fa fa-link" aria-hidden="true" ></i></a>
Final Thoughts</h2>
<p>It's frustrating how many of the things that I want to do still have
what seems like huge mounds of prerequisites in front of them. I want
to do all of these interesting language things, but I feel like I'm
blocked from doing them because I want to do them in LLVM, but I don't
know anything about actually using the LLVM.</p>
<p>Also, when I was reflecting on what I need to do to become a better
programmer, one of the recurring thoughts I responded with is that I
need to write/work on a larger system/project. I feel like I have
quite a bit of experience taking an small projects. But not something
like YRAS... Which, I think, is one reason that I am absolutely
terrified of starting that project.</p>
...and this is why master should be stable2014-10-07T00:00:00+00:002014-10-07T00:00:00+00:00https://zefira.dev/posts/dot-dot-dot-and-this-is-why-master-should-be-stable/<p>Second day at Hacker School. I spent the morning doing two things:</p>
<ul>
<li>getting my blog setup to use <a href="http://octopress.org/">Octopress</a>.</li>
<li>helping another HS-er troubleshoot getting setup with <a href="http://flask.pocoo.org/">Flask</a> and <a href="http://flask.pocoo.org/docs/0.10/tutorial/introduction/">Flaskr</a>.</li>
</ul>
<!--more-->
<p>Yesterday, I was super frustrated by the difficulties I had in getting
Octopress going, and was reminded of the frustrations with setting up
Ruby environments that led me to wipe my computer and start from
scratch a few weeks ago. You know, that and a certain amount of boredom.</p>
<p>Anyhow, I was able to make more headway today, largely because I
sorted out all of the really silly stuff yesterday.</p>
<p>But helping Sammy with setting up <a href="http://flask.pocoo.org/">Flask</a> was a nightmare! TL;DR, we
figured it out, and it was incredibly simple.</p>
<p>First off, there were some minor SNAFU's with me showing her about
[virtualenv], and explaining briefly how it works and how why she
should use it. But then, we were confronted with these instructions
on the [Flaskr doc page][flaskr-github] on Github, a natural place to
look when the <a href="http://flask.pocoo.org/docs/0.10/tutorial/introduction/">"actual" docs</a> don't mention anything about how
to install or setup the damn thing.</p>
<p>We spent nearly two and a half hours searching for the mythical
<code>flask</code> binary, to no avail. Finally, after I inadvertently ran
aacross this [flask issue][issue], it dawned on me that we might be
having the same problem. Well, that's not quite true. In reality, I
got so frustrated that I decided to just try the simplest thing that I
could think of to get flaskr to run <code>python flaskr.py</code>. Low and
behold, it worked!! Then, in retrospect, having seen that issue
describing the disconnect between the docs for flaskr in the flask
<code>master</code> branch and what the latest stable version was, I went and
looked at the [last stable release docs][laststable] for Flaskr, and
there it was. Simple instructions to do the simple, straightforward
thing.</p>
<p>[[virtualenv]: http://virtualenv.readthedocs.org/en/latest/virtualenv.html
[flaskr-github]: https://github.com/mitsuhiko/flask/tree/master/examples/flaskr
[issue]: https://github.com/mitsuhiko/flask/issues/1180
[laststable]: https://github.com/mitsuhiko/flask/tree/0.10.1/examples/flaskr</p>
LeJOS and Clojure2014-04-17T00:00:00+00:002014-04-17T00:00:00+00:00https://zefira.dev/posts/lejos-and-clojure/<p>This is a short followup to my first post about
[experimenting with the Lego EV3's][lego-ev3] and <a href="http://lejos.org">leJOS</a>. Shortly
after I wrote that post, I went on the Clojure IRC channel and talked
to <a href="http://technomancy.us/">Phil Hagelberg</a> and some other helpful folks about my
issues. They steered me away from attempting to install <a href="http://leiningen.org/">Leiningen</a> on
the EV3 itself, which was a relief since I think that way lay
madness. Instead, they suggested that I have a small launcher program
that would just setup a REPL. Then I could run my robot-controlling
Clojure code from there.</p>
<p>[lego-ev3]: {% post_url 2013-10-31-leJOS-and-EV3 %}</p>
<!--more-->
<p>This advice led to the creation of <a href="https://github.com/RadicalZephyr/ev3-nrepl">ev3-nrepl</a>. It's basically a small
skeleton Clojure project to provide the basis for running Clojure code
on the EV3. It's a minimalist setup, so it should provide a good
starting point for other people to continue experimenting. I tried to
explain how to use it fairly clearly, as well as add all the links to
the necessary documentation to get leJOS running on the EV3.</p>
<p>Unfortunately, despite this seemingly great start, I never ended up
actually doing anything interesting with Clojure on the EV3. There are
two related reasons for this. First, the robotics class I was in
started having actual coding assignments (thanks to the fact that we
could now run leJOS). Second, it turns out that leJOS is designed from
an incredibly stateful and imperative point of view. This makes it
very hard to develop clean Clojure code that makes use of their
API. Specifically, leJOS makes the assumption that any code making use
of it is going to start-up, do robot stuff, and then die. A Clojure
REPL obviously breaks this assumption rather badly. As it turned out,
this was the most painful when I was trying to figure out how to
program and build my robot at the same time.</p>
<p>What I wanted to do was be able to open a Clojure REPL connect to it,
and then leave it running for a long time. This is my normal modus
operandi, but the EV3 makes it even more critical that this
work. Because of the limited hardware, it takes several minutes for
nrepl to actually boot up.</p>
<p>I think that using Clojure as a prototyping tool for building robots
with the EV3 could be incredibly powerful. All of the
<a href="http://www.infoq.com/presentations/Clojure-Java-Interop">arguments</a>
for why Clojure is a great language for exploring Java API's are
applicable to working with the EV3 API's. Possibly even more so, since
there is something really, really cool about typing some commands at
the REPL and having your robot start moving around and doing
stuff. Unfortunately, because of the current architecture of leJOS and
the limited processing power of the EV3's it's not quite there yet.</p>
Clojure, LeJOS and EV32013-10-31T00:00:00+00:002013-10-31T00:00:00+00:00https://zefira.dev/posts/lejos-and-ev3/<p>I've had the opportunity in the last couple weeks to start playing
with a Lego Mindstorms EV3. Since this hardware is such a new release
though, most of the normal contenders in the robot programming scene
(namely, RobotC) haven't had time to update their offerings to work
with it.</p>
<p>Enter <a href="http://lejos.org">LeJOS</a>. Apparently the maintainer of the LeJOS project
got early access to the developer documentation and other necessary
infos, so he's been able to develop a setup for allowing people to
program their EV3 using Java.</p>
<!--more-->
<p>It's a really elegant solution actually. Since the EV3 already runs a
full linux kernel, he just modified the base Legos linux image. You
then partition a microSD card and write this image to it. Now you can
boot the EV3 from the microSD card and ssh into it over either WiFi
(if you have an external USB wifi adapter that's compatible), or over
USB (a more recent feature).</p>
<p>My next goal is to get a clojure projet running on the ev3. There are
two approaches that I could take. First, I could try AOT compiling my
clojure source and then shipping just the uberjar to the EV3. This
seems the most straightforward. However, following this plan I almost
immediately ran into a stumbling block. The LeJOS java library cannot
be compiled using Leiningen when not on the EV3.</p>
<p>Apparently this is because something about the clojure compilation
process actually loads the EV3 classes into the JVM. This causes the
static initializers to be run. And the static initializers look for
specific files in the /dev directory to try and do their thing.</p>
<p>Obviously, when you're not on an EV3 these files don't exist.</p>
<p>So the second way of doing things would be to attempt to install
leiningen on the EV3 itself. I see a lot of potential problems with
this, first and foremost being how do I do it without a direct
connection to the internet? Secondly, the linux installation is very
minimal, and might not actually contain all the necessary tools to
install and use leiningen.</p>