My Go project structure

The first time I used Go was in late 2016. Since I was working with Java at that time, I remember everything I wanted to build in Go resembled Java code. It looked unnatural and I knew this is not the way it should look. After reading many blog posts about Go and developing many web services, I found a standard structure to use when building new services. Before I dive into the topic, I want to briefly say something about DDD (Domain Driven Design) and data structures & algorithms. Even though they are not language specific, I want to mention them because they have an influence on the way I build software and hopefully they will help clarify some decisions I made.

Domain driven design

Note: I'll try to explain only a tiny part of DDD mostly related to the layers.

In DDD, the core business logic of any software lives in a domain layer. This should be totally isolated from the rest of the application and shouldn't be communicating with any external sources (databases, filesystems, remote services etc.). It should contain only a pure code of the business logic. This layer contains enterprise wide business rules. Hereinafter: business logic.

There is also a layer called the application layer that sits on top of the domain layer. This layer has no business logic and its purpose is to delegate tasks to the domain layer to accomplish specific application tasks. It also deals with external resources like databases, other web services etc. and can also delegate tasks to these. This layer contains application specific business rules. Hereinafter: application logic.

The infrastructure layer is a layer where we put a real implementation of a database connections, web services, middlewares etc.

A simple example: Let's imagine we want to calculate your biological age and save it on a database based on your weight and gender. To get weight and gender we need to talk to a remote service over the network (it can be a totally separate web-service). In this case our core business logic that exists in the domain layer should calculate biological age based on these two parameters. It can be a simple function with two arguments as an input and one argument as an output. How we get these two parameters should be totally independent. Its only job is to calculate biological age based on given arguments and to return the result. In the infrastructure layer we will implement a real connection to the database in order to save the result and also implement a connection to the remote service to fetch weight and gender. The application layer is here to coordinate tasks and delegate jobs. That means this layer will first call the remote service in order to get desired values (it actually delegates tasks to the infrastructure layer), then it will delegate a calculating job to the domain layer passing these two values and at the end it will save the result in a database (again delegating it to the infrastructure layer). An important thing is that the application layer should not be aware of the infrastructure layer at all (can be done with dependency injection or higher order functions) but it is aware of the domain layer.

Note: the application layer is not the C in the MVC pattern.

There is much more to DDD than what I briefly explained here, and I use it as an idea on how to structure complex projects. If you want to know more about DDD I encourage you to read Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans and Implementing Domain-Driven Design by Vaughn Vernon.

Data structures & functions (algorithms)

Data structures and functions are two separate entities of software employed together to solve a specific task. Functions might be seen as black boxes that use different data structures as an input to produce new data structures as an output. By designing a good system the main focus should be on choosing the right data structures, not the functions.

Alan Perlis:

It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures.

Rob Pike:

Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.

With that in mind, when designing a system I pay attention specifically to organizing business wide data structures and separating them in specific areas so that they could be utilized by functions across packages. On the other side, task specific data structures stay as close as possible to the place of a usage. I am also trying to create as few different data structures as possible.

Go project structure

Let's get back to our Go code structure. Usually I have an app package in the project root and into it another one named core. The core package contains all business logic together with all application logic. In a fresh new project there are no sub-packages in it because the system is not too complex yet. Once it starts growing, I might create them in order to better organize core business logic. That means the application and the domain layers are just logically divided into different files but still live in the same place. This package is totally independent of the outside world.

At the same level as a core package there is a data package. A whole business wide data structure lives here. One can argue that this should go under the core package and to not put everything into the same place since it is a business domain data, which is correct, however for the sake of reducing complexity in the beginning phase I tend to put everything in one place. If projects start growing, these may be migrated inside, but for now, I keep them here.

The last package at the same level is infrastructure. Here, I put everything related to communication with the external world and all adapters (databases, remote web-services, handlers, middlewares, …). It might look something like this:

Golang structure business logic

Data

Here we find the business wide (domain) data structures. Since we have a very simple example, there will be only one data structure we need. So, only one file, for example named bio.go with this content:

package data

type BioData struct {
	Weight int
	Gender string
}

Core

Since our core package contains both an application and a domain layer, they will be only logically divided into the files in order to reduce complexity. No further sub-packaging needed at the beginning. As a domain layer, we might have one file named bio.go:

package core

import "my-project/app/data"

func calcBio(d data.BioData) int {
	...
	return result
}

An important thing here is to not let any external things creep in, only domain data are allowed to be used here. I usually keep the functions in the domain layer unexported. Functions should be as pure as possible.

As for the application layer, we will have another file in the same package named api.go. Here, all functions will be exported and this file will be used as an entry point into our business world. In the beginning we can put them all in one file, like api.go, but as soon as the project gets bigger, we may separate them into more files. Since the application layer’s responsibility is to delegate tasks, it should be totally aware of the domain layer, but not of the infrastructure layer. For that purpose, I use higher order functions. There is an example in our case (for a given userId we need to calculate a biological age and store it into the database):

package core

import "my-project/app/data"

func Calculate(get func(string) (data.BioData, error), store func(string, int) error) func(string) error {
	return func(userId string) error {
		bioData, err := get(userId)
		if err != nil {
			return err
		}

		age := calcBio(bioData)

		return store(userId, age)
	}
}

As we can see here, we insert the functions get for getting biological data from an external resource and store for storing the biological age. This is how we decouple it from a real implementation. Since the application layer is aware of a domain layer, we call calcBio directly.

Infrastructure

In this package we find all functions and settings that support our main core domain to run the app as a whole. Here goes: database connections, connection to remote services, handlers, servers, logging, middlewares etc. In our case it will be:

Golang, DDD infrastructure

Db

I usually name this package db but it can be changed to match the database we use, like postgres. In our example we will have only one file bio.go:

package db

import "database/sql"

func StoreBioAge(db *sql.DB) func(string, int) error {
	return func(userId string, bioAge int) error {
		// store bio age
		return nil
	}
}

As seen above, we use higher order functions in order to insert a db connection. That helps by testing and decoupling in the application layer (as we remember we have there a function signature store func(string, int) error )

Service

Here we can put the implementation of a fetching user data from a remote service. Let's name the file user.go:

package service

import "my-project/go-structure/app/data"

func GetBioData(userId string) (data.BioData, error) {
	// an implementation of a real connection to the remote user data service
	return data.BioData{}, nil
}

If any decoupling is needed here we can also use higher order functions for that purpose.

Server

The final step becomes the glue for all the pieces we have. We will add a server.go file and a handler sub-package. Also we can create a middleware sub-package as well or any other sub-package that helps build our software. Something like this:

Golang, Domain Driven Design

For example, all middlewares we want to create in the app should go into the middleware sub-package, e.g. auth middleware. Since we don't have any in our example, for the sake of simplicity we will leave it empty. In the handler package we store all handlers for our app. In our case we only have one, let's name it bio.go:

package handler

import "net/http"

func HandleBioAge(calculate func(string) error) func(w http.ResponseWriter, r *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
		userId := ... // get userId from http request

		err := calculate(userId)
		// handle error

		// send response
	}
}

And again unsurprisingly, we have higher order functions. Everything that matters for this handler is that it knows how to trigger a process of a bio age calculation and since we insert a simple function calculate, the only way forward is to execute that function. That's all that the handler needs to be aware of. That way the code is nicely decoupled and easily testable. As it can be seen, it returns a standard handler function func(w http.ResponseWriter, r *http.Request) and it can be simply used by our router.

And finally we have server.go file where we configure our server:

package server

import (
	"database/sql"
	"net/http"

	"github.com/gorilla/mux"
	"my-project/go-structure/app/core"
	"my-project/go-structure/app/infrastructure/db"
	"my-project/go-structure/app/infrastructure/server/handler"
	"my-project/go-structure/app/infrastructure/service"
)

func Run() error {
	db := // init db

	r := mux.NewRouter()

	initHandlers(r, db)

	srv := &http.Server{
		// ... server params
	}

	return srv.ListenAndServe()
}

func initHandlers(r *mux.Router, conn *sql.DB) {
	r.HandleFunc("/api/calculate/{id}", handler.HandleBioAge(
		core.Calculate(
			service.GetBioData,
			db.StoreBioAge(conn)))).Methods(http.MethodPost)
}

In the Run function we can put everything we need to create our server, but for the sake of simplicity I'll keep it short and tidy. More interesting here is the initHandlers function. That's the place all the gluing magic happens. Since our system consists of small building blocks, now we can easily combine them as a whole. Every part is easily exchangeable and testable.

In order to execute it, we could have a package structure in the root of our application like cmd/server. There we might have something like:

package main

import (
	"log"

	"my-project/go-structure/app/infrastructure/server"
)

func main() {
	log.Fatalf("Failed to start server: %v", server.Run())
}

And here is the whole structure:

Golang package structure

Testing

As you can see, I usually use higher order functions as my primary strategy for decoupling building blocks. That way every single part is nicely isolated and tested. Also it is possible to test it as a whole or simply to swap out some parts with other better suited implementations. I haven't implemented any tests in the example for the sake of simplicity.

Coupling

I've seen some developers try to decouple code always and at any price. Even though I like nicely decoupled code, as it can be seen above, I am not fond of doing it at any price and there is a reason for that. Let me give you an example.

Let's imagine we have an endpoint that accepts requests with body data that looks identically like our domain data BioData. We have two choices, to reuse it or to create a new one and convert it to the existing BioData. In this case I'd prefer to reuse it, since creating a new one adds unnecessary complexity. All together with new data structure we might need to create converters as well and maintain them during a time. The silliest thing here will be creating converters for converting one data structure to another one and have them both look exactly the same. And yes, that's called coupling but as long as we are aware of it and as long as we do it deliberately in order to reduce complexity, a little coupling is fine. But the important thing is that we are aware of it!

When these two data structures start differing, we can then create the new one and create the convertors. Usually I put a new data structure as close as possible to the place of usage (task specific data structures). Since it's handler related, in this case it might be something like this:

func handleSomething(w http.ResponseWriter, r *http.Request) {
	type bioBody struct {
		Height int `json:"height"`
	}
	b := bioBody{}

	err := json.NewDecoder(r.Body).Decode(&b)

	// do other stuff
}

Conclusion

That's the way I usually organize my go code project. If you asked whether it is super important to stick with exactly this folder structure, the answer would be no. For the smaller projects or pilot projects, the structure can be even more simpler. Sometimes I even don't create any folder structure if a project is super tiny, simple flat organization in one package is more than enough. The most important thing is knowing the responsibilities of any functions or packages we build, the way of interacting between them and their purpose. And then the code structure will emerge by itself.



see all posts
WHO AM I?

My name is Marko Krstic. I am a senior software engineer and support companies by automating their processes and creating online platforms from which they can reach a wider customer base. More about me

Your subscription could not be saved. Please try again.
Please confirm your email address to complete your subscription.

Newsletter

Subscribe to my newsletter and stay updated.