A Blog about Business, Enterprise Architecture and Software Design by the Founders of ZenModeler

Practical Starter Tips for Clojure

In this article, I share my experience of developing with Clojure over the last months. I focus on the practical tips that will allow you to start seamlessly on Mac Os X (but should apply with Linux and Windows).

Clojure logo

Clojure Resources

When learning Clojure, first of all: have great resources to rely on and not get lost.

Books

If you begins with functional programming I suggest you to start with Clojure Programming from Christophe Grand, O'Reilly, once more experienced you can follow with the amazing Joy of clojure for hindsightful thoughts about programming applied with clojure (a 2nd edition is out). For those of you that prefer a more "example oriented" approach, have a look at the nice Clojure Cookbook. If you have a Java/OO background I suggest you to read Functional Programming for the Java Developer and Functional Programming for the Object Oriented Programmer. Of course, the bible, for more fundamental stuffs: Structure and Interpretation of Computer Programs (A good PDF and EPUB version are available).

Cheatsheet and documentation

I tend to have a Clojure Cheatsheet and a browser always opened on clojuredocs.org. I cloned the clojuredocs repo so I always access the clojuredocs on my Mac even when I'm offline. You can also access clojuredocs directly from the REPL with clojuredocs-client. You should also have a look at the Clojure Style Guide.

Update: there is now an up-to-date alternative to ClojureDocs named Clojure Grimoire, also Clojure Grimoire is available offline. The Clojure Cheatsheet is even updated to link to Grimoire instead of ClojureDocs.

Community news through newsletters and blogs

Clojure is already well-stocked with newsletters: Clojure Gazette, (def newsletter), Read Eval Print λove and agreggators: Planet Clojure and Clojurls.

Tero Parviainen, that organized ClojureCup, also has a great list of Clojure resources, another list of clojure resources.

I personaly gave a clojure presentation at the Lausanne Java User Group some times ago, here are my "Clojure Introduction slides".

Online resources

BraveClojure has great and detailed articles about Clojure and for exercising you've got 4Clojure and ClojureKoans. Clojure from the ground up is also a good introduction, and Clojure Distilled will introduce you to the mindset needed when using Clojure.

Conferences

Some conferences are entirely dedicated to Clojure: ClojureWest and clojure/conj in US, EuroClojure in Europe. Some have a good inclination towards Clojure like StrangeLoop or LambdaJam.

Common Environment for Clojure Programming

Java / Clojure / Leiningen

A JDK, Clojure, and Leiningen if you want your repl loaded with all the classpath dependencies of your project just type lein repl.

Make the non-US keyboard more hacker-friendly

The (caps-lock) key is almost unused and as a very convenient and accessible one I suggest you to remap it to ctrl. All the non-US keyboard layout are painful for daily use (I use a french one!), especially when you type a lot of (), {} and []. Therefore, I installed KeyRemap4MacBook and assign the [ to a single keystroke on the (shift) key. Of cource I assigned ( for the left one, and ) for the right one. The combination of of 'alt-shift' but by hitting the 'alt' or 'cmd' key first print me respectively a { or [ (and so on for the right side } and ]). Here is my modified KeyRemap4MacBook private.xml as a gist.

iTerm2

iTerm2 is a terminal emulator for Max OS X with impressive features, its configurability is the major one for me.

tmux

tmux is a terminal multiplexer allowing you to manage window and panes in one terminal and in a persistent way (that's to say you can close the terminal window, your session stay alive in the background with the tmux process). The best part of tmux is that it allows to script all my development environment in one command invocation: starting all the databases, the web server, the log tail, a window for the front-end, the backend, the infrastructure, etc.. The tmux book from pragprog is a must have to get started.

zsh

zsh is a shell designed for interactive use, hence the shortcuts and auto-completion it provides are simply awesome. Install oh-my-zsh to get a bundle of useful plugins in a seamless installation.

Here are some links to discover the power of zsh:

As a conclusion, here is a great blog post about the text triumvira that the combination of zsh, tmux and vim is.

Clojure Editor

The very first goal is to have an editor opened in one window and a clojure REPL beside, then to send code in the REPL for evaluation. That's all I need and it's called REPL-Oriented Development! beside the IDE choice, a good workflow with the REPL is also mandatory, here are some links about the subject: the famous Clojure Workflow Reloaded based on tools.namespace, Give your Clojure Workflow more flow.

There are basically 2 schools of thoughts on the editor subject: vim and emacs.

The vim School of Thought

Vim is a basic choice with limited features about the language comprehension as vim almost edit text only but also easier to start with. I don't cover the basics of working with vim as there is plenty of resources on the web, here are some:

Clojure and Vim

Install VimClojure and then C-c C-c will send the form under the cursor to the REPL hosted in a tmux pane. The first time you invoke the command it will ask you the tmux window.pane you want your selection to be sent. It's the most simpler and immediate development environment I used myself for some months. I didn't use the nailgun plugin for interacting with a repl as the level of clojure integration is very low with vim. At that time I prefered to just send basic text to the repl and switch between vim and the repl inside a tmux pane. A better integration and understanding of the edited language is also why I migrate to emacs.

I suggest you to read "The Text Triumvirate", a great blog post about the productive environment resulting from the use of the "vim/tmux/zsh".

The Emacs School of Thought

I switch from vim to emacs to get a better integration between the Clojure source and the repl. Emacs has a different philosophy compared to Vim, here is the emacs manual to get through it, and an Emacs CheatSheet for a reference.

Starting and Quitting Emacs

Emacs should be started in the background as its initialization is costly.

      $ emacs --daemon
   

then start instantaneously emacs with:

      $ emacsclient your_file.clj
   

Then to kill softly the daemon:

      $ emacsclient -r "(kill-emacs)"
   

Basic Functions I use with Emacs

  1. Start the cider repl: M-x cider-jack-in (should auto complete!)
  2. Evaluate the entire file: C-c C-k
Essential Emacs Functions key binding
Undo C-x u then q to quit the undo-tree
Load file C-x f
Save file C-x C-s
Split window vertically C-x 3
Go to the beginning of the file M-<
Go to the end of the file M->
Move to other window C-x o (should really work on this for a more straightforward keystroke as this function will be call several times per minutes)

Interacting with nrepl

Install emacs live

I find the "emacs live" bundle to be a good one to start with for clojure development, it bundles the cider project to boost emacs REPL capabilities.

nREPL Functions key binding
Switch the namespace of the repl buffer to the namespace of the current buffer C-cM-n
Evaluate the form preceding point and display the result in the echo area. If invoked with a prefix argument, insert the result into the current buffer C-xC-e
Evaluate the form preceding point and display the result in a popup buffer C-cC-p
Evaluate the top level form under point and display the result in the echo area. If invoked with a prefix argument, insert the result into the current buffer. C-M-x or C-cC-c
Evaluate the region and display the result in the echo area. C-cC-r
Interrupt any pending evaluations. C-cC-b
Invoke macroexpand-1 on the form at point and display the result in a macroexpansion buffer. If invoked with a prefix argument, macroexpand is used instead of macroexpand-1. C-cC-m
Invoke clojure.walk/macroexpand-all on the form at point and display the result in a macroexpansion buffer. C-cM-m
Eval the ns form C-cC-n
Select the REPL buffer. With a prefix argument - changes the namespace of the REPL buffer to the one of the currently visited source file. C-cC-z
Clear the entire REPL buffer, leaving only a prompt. Useful if you're running the REPL buffer in a side by side buffer. C-cM-o
Evaluate the current buffer. C-cC-k
Load a file. C-cC-l
Display doc string for the symbol at point. C-cC-d
Display the source for the symbol at point. C-cC-s
Display JavaDoc (in your default browser) for the symbol at point. C-cC-j
Jump to the definition of a symbol M-.
Return to your pre-jump location M-,
Complete the symbol at point. (For auto-complete integration, see ac-nrepl) M-TAB

I also wrote a short emacs-lisp function, called nrepl-send-to-repl, that send the form under the cursor as text to the repl.

paredit-mode

At first, I found the paredit-mode to be the most annoying one I ever met until I understood the "way of seeing it": think of the function as moving parens to the left or to the right and not moving the text (see the first two examples below). The 3 paredit functions I constantly use are the followings:

Paredit Function Binding Example
paredit-forward-slurp-sexp (foo (bar |baz) quux zot) ---> (foo (bar |baz quux) zot)
paredit-forward-barf-sexp (foo (bar |baz quux) zot) ---> (foo (bar |baz) quux zot)
paredit-splice-sexp M-s (foo (bar| baz) quux) ---> (foo bar| baz quux)
disable paredit

In short I constantly move the "end" paren to the right or to the left (I actually seldom move the "start" paren) end remove enclosing parens. The rest "nice-to-have" paredit functions are not as mandatory as that 3 ones and I often replace one paredit function with a sequence of that 3 basic ones (like splitting a (foo bar) into (foo) (bar) just ). Then I sometimes disable paredit to insert or remove a specific paren like any "dumb" editor. Finally, here is the best paredit explanation I stumbled upon.

The outsiders

The community is quite active in the IDE field. Here are some other options you should consider:

Being an occasional Sublime Text user and a Light table backer, I don't have an opinion about these environments. I'm quite happy with my emacs choice for the moment and should stick with it in the short term.

Conclusion

This post should give you a good overview of the resources to get started with Clojure and what it's like to install and use a Clojure development environment.

Introduction to Domain Driven Design - Entity and Value Object

The slides from My DDD Paris User Group talk hosted by Xebia

And here are the links towards the 3 videos (in french!):

About OO and FP Design

DDD distillation

Bounded Context

Books about DDD (free and not free)

And also some links towards the books out there about DDD:

Event Oriented Databases : a New Kind of Persistence Paradigm

Those days, I’ve been observing the emergence of radically innovative approaches to data persistence. This article starts with a quick summary of the – now a classic – NoSQL domain and then introduces the event based persistence approach that is emerging.

Time

NoSQL

It’s now been several years that the NoSQL databases are being talked about and are maturing with the experience feedback, sometimes with some difficulties. The concept of the “polyglot persistence” is spreading out. This term, first coined by Martin Fowler, means that several types of databases are used in the same software, each one fitting particular needs.

I also use that approach myself with ZenModeler (neo4j graph database, MongoDB document database, classical relational DB and I look with a strong interest towards event based databases). Each database type fulfills a precise need. The encapsulation of the data access behind a Repository object is essential according to me because it strongly helps to separate concerns between business logic and persistence, and then to encapsulate the persistence logic behind a clean business interface.

In short, the typology of NoSQL databases is :

  • Column-oriented Databases : store data by columns and not by rows, allows complex and efficient queries, useful for mainly read processing.
  • Key-Value databases : needs more efffort to query, but very easy to scale. Typical databases for parallel processing like “Map-Reduce”. Lots of people here, like Cassandra, Riak, Dynamo and the french guys at bureau14 with wrpme.
  • Document oriented databases : store document and access them with a key, the hype being to store document as JSON like with MongoDB (if you plan to use MongoDB, you have to check the »MongoDB gotchas »). The DDD Aggregate pattern is very useful with that kind of storage.
  • Graph databases : use concepts of nodes and edges, each holding properties (typical example : Neo4j).
  • Object databases or distributed caches : I’m feeling nostalgic about that one as I work with the ancestor Versant in 1999 for one of my internship :-). Nowadays, most of the object oriented databases switched to distributed caches in front of relational databases.

That‘s it for the technical aspects. There are big differences within the data structures used in each of them ranging from the narrower to the wider : maps, trees (as JSON or XML) or graphs. Keeping in mind that the more the better, a graph database can represent a tree and each node or relation can also store a map. For the substance I don’t see any differences because these tools don’t integrate any paradigm shift toward persistence approach…

Data Persistence Fundamentals

Let’s get back to the fundamental of what we store : data that describe reality !

A data is the relationship formed by a concept and its measure. A measure is characterized by a type : either quantitative (weight, amount, age) or qualitative (name, city, date). The composition of several data in a structure leads to the description of a fact.

For example, a user is browsing on an e-commerce website and is interacting with it : she adds some products to her cart and removes some, when she has finalized her purchases, she validates her order and then pays. Each of the user’s action is processed by the system. Overall, the later guarantees the ACID properties for her actions : atomicity, consistency, integrity and durability. The last point, durability, is essential in case a link of that extremely complex chain would fail. That durability is guaranteed by redundancy mechanisms and by writing on supports that “persist” in the event of a lack of electricity (sorry I need to get back to some basics before moving on :-)).

There is now two ways to analyze the user’s actions :

  • From the point of view in the cart: it changes along with the user’s actions (add a product, remove one, modify its quantity, recalculate the balance, etc.). The cart is always in a coherent state characterized by its identifier, the product list with its total quantity and the total balance. It is more or less the vision of “objects and states” of the OO paradigm.
  • From the user’s point of view: user is doing a sequence of actions (add, remove, change quantity, etc.). Of course, each action will be received by the cart which will process the necessary state transitions. The vision is more functionally oriented, where events trigger functions that make the state transitions. The whole thing being deterministic and, by definition, the event immutable…so it’s more or less the vision of « function » of the FP paradigm.

Event Oriented Persistence

An event describes a change that occurred in the past. Each event triggers an action that modifies the state of the object which receives it through a state transition. The receiving object becomes known in a specific state, following the transitions that occurred in the triggered functions associated with each event. Arriving in a specific state can also triggers another internal event.

For example : the addition of 3 products of $100 each results to a cart balance of $300.

If these events are replayed from the starting state in the same order, then the final state is identical in a deterministic manner. Moreover, if every event can be associated to its inverse that compensate the original one, the starting state can then be reversed forward from the final state.

Events are an implicit way of measuring the time that is passing by. The order of the events are a relative measure of time. The instant when the event occurred is the absolute one measured by our clocks, that’s to say an absolute measure of time.

The mutable objects, those performing state transitions, are the ones we are interested in their life cycles. The DDD approach names that kind of objects “Entities” rather than “Value Objects” (see that post about identity and state).

The following schema describes the relationship between the identity of the entity, its states and the events it received.

Event oriented interactions with cart object (identity + state)

The events are the « atoms » that can be stored in order to retrieve our object states at any point in time instead of the traditional way of persisting only the « head » or the last known state of our object.

In practice, the databases I think about while writing this article are Datomic and EventStore. EDIT : Rich Hickey just post an article on InfoQ describing the great design he did for the architecture of Datomic.

Conclusion

In conclusion, a fundamental paradigm shift is at work with the new kids on the block databases like Datomic as they force us to think about persistence in a completely new way: to wrap up, thinking about storing the state transition represented by the event and not only about storing the last known data. The approaches related to the « event » way of storing data are named Event sourcing, CQRS, etc.

My next article on that persistence topic will be about the blurred differences between fast memory and mass memory with the introduction of the memristor and the consequences in our software design.

10 Questions about Software Design

The ‘design’ word is one of the strangest beast that lives in the IT world. I hear that word very often as people talk a lot about it. Yet it has a very broad meaning and hence a misunderstood signifier among people, with inaccuracy and positive and negative myths about it… In this article I try to answer to the most common questions about Software Design. You can also replace the word "Design" with the word "Architecture", both are often poorly defined and understood within IT. I personaly prefer using "Design" as a verb and "Architecture" as a deliverable.

A picture I shot while writing this article in the Centre National de la Danse near where I live

What Design refers to?

‘Design’ defines both the activity with the verb ‘to design’ and the result of that activity with the noun ‘a design’.

Can a system have no design?

I had sometimes heard: “this system has no design”. That sentence refers to the lack of a proper design documentation about the system and also sometimes to the lack of attention from the developers to get some insight and structure the system wisely. But a system always has a particular design, sometimes poorly documented and not carefully crafted, but it has one because a source code exists and it runs.

What is a design made of?

A design is about all the decisions made to build the system, both the very large ones that impact the entire system and the small ones with an impact scoped to a small area (a function, a class, a sub-component of the system).

How to best make the design decisions?

  • First question: do I really need to make this decision now?
    • If not, how can I revert this decision later? that question is about flexibility and stuffs that are hard to change after they are done.
    • Or else, what can I do to defer this decision?

Also, always remember to question the validity of the user requirement with a simple « Why? ». One of the most famous experience feedback came from the F16 aircraft design, I advise you to look at that great experience feedback from the father of the F16, Harry Hillaker. In short, US Air Force (the client) ask for a plane capable of a speed over Mach2, but Harry Hillaker was the one who asked « Why? » and the response was « to be able to escape from combat” which was the real requirement. It allows Hillaker and his team to enlarge their perimeter of thinking for the solution and indeed provide a lot of innovations.

If the decision cannot be avoided, check the following three items:

  • Which user’s requirement this decision helps to fulfill?
  • What are the advantages/inconveniences of that decision?
  • Are the responsibilities still clearly separated among components after that decision?

How to properly document a software design?

You’ll have to talk about the “why” of the large decisions that impact the way the system will be structured or will behave. Because if we understand the “Why?”, the “How?” will be easier to grasp. The source code represents the ultimate design documentation of the softwares, but with a huge amount of details. Then we have to abstract that detailed representation, that is to say select only the meaningful information from that ocean of details. But we only get the ‘What?’ (the structure) and sometimes the ‘How?’ (the behavior), which is not very useful if not properly chosen.

What is a good design process?

A good design needs feedback from the user of the system, so the system has to be in front of the user (getting real) as soon as possible, with frequent releases of new versions once the users’ feedback has been received. Hence, a good design process is an iterative and incremental one. There is an acronym for the lack of an iterating process : BUFD for Big Up Front Design. You can also look at that great article from Jeff Atwood about the iterating process in a cat product.

How can a design emerge?

This is a concept coming from the agile world. In short, a design can emerge if we get feedback continuously. Hence TDD and BDD practices help the design to emerge, making all the decisions we need with good insights, because we have a direct and real usage of the system coming from the scenarios. I have had some project experiences where writing the scenarios right before coding – and only with the domain (no persistence, services, web layer, etc.) – saved me hours of refactoring after discovering mistakes I had made in my implementation. Neal Ford has good slides about the Emergent Design concept.

What is the goal of the design activity?

Deliver the best solution to the users given the 4 fundamental constraints of a building project: scope, resources (money is a good shortcut for resources…), time and the most important one, quality.

4 constraints of project

What are the 4 basics designer’s talents?

  • Understand user needs
  • Find an adequate solution
  • Abstract details
  • Separate responsabilities / concerns
    Each one of these « talents » needs a dedicated article because I have a lot to say about those talents.

In the end, what makes a good design?

I would say a design is good when there is a good balance between the different constraints at work, particulary those that are immediate to the end user. Fundamentally, a good user experience is what makes a design compelling. But the others internal quality attributes have their importance, like those defined by the Software Quality ISO norm.