We like mage at Aserto. We use it (in lieu of make) to build our projects. Not having to use scripts (bash and make) to build scripts is amazing.

Just because you don’t ship the code that builds your stuff, doesn’t mean it shouldn’t conform to the same standards. You can use go modules to share common code across your projects.

You can reliably build, lint, and test your code.

We’re even starting to use mage in some repos that are not “Go” because we like it so much. Of course, you need Go, so if you’re not comfortable with Go itself, this might not be the correct solution for you.

mage-loot is a collection of mage goodies that includes dependency management using Depfile and helpers for automating the buf, dotnet and protoc CLIs. You’ll also find some opinionated build functions for Go and Docker. They cover typical use cases like building, linting, testing, and code generation.

Depfile

The dependency management helpers are quite powerful, so I’d like to detail this a bit.

One problem that we’ve had to repeatedly solve is making sure the human and robot teams all use the same tools, binaries, and libraries when building projects.

There are some solutions out there, but they didn’t fit our needs for various reasons (some are too hard to adopt, and some don’t have enough features).

With Depfile, we’re aiming for enough features so that you can reliably build complex projects, but simple enough that you understand how to use it in less than 10 min.

What we want to be able to do in a magefile is the following:

func Target() {
  var vault = deps.BinDep("vault")
  vault("write", "foo", "bar")
}

func Deps() {
  deps.GetAllDeps()
}

What we're saying here is that we have a binary dependency named "vault".

Even though we don't want to mention it in the code, we do want that binary to be procured securely, and we want to be able to set a specific version to use. Otherwise, different developers will have different versions of Vault. And some of those might even be compromised!

Furthermore, procurement should happen on the fly, as dependencies are needed. You'll notice the Deps target can download all of your dependencies on-demand, but calling this target is not required.

So how can you achieve all this? Depfile to the rescue! 🦸‍♂️

> Disclaimer: You have to buy into mage to use all this goodness.

Next to your magefile, you add a Depfile. It’s a YAML file with the following structure:

go:
  tool:
    importPath: "so.me/import/path"
    version: "v1.0.0"

bin:
  useful-binary:
    url: 'https://so.me/url/v{{.Version}}/protoc-{{.Version}}-{{if eq .OS "darwin"}}osx{{else}}{{.OS}}{{end}}-x86_64.zip'
    version: "1.1.0"
    sha:
      linux-amd64: "d4246a5136cf9cd1abc851c521a1ad6b8884df4feded8b9cbd5e2a2226d4b357"
      darwin-amd64: "68901eb7ef5b55d7f2df3241ab0b8d97ee5192d3902c59e7adf461adc058e9f1"
    zipPaths:
    - "bin/the.binary"
lib:
  useful-lib:
    url: "https://so.me/url/v{{.Version}}.tgz"
    version: "2.5.0"
    sha: "e8334c270a479f55ad9f264e798680ac536f473d7711593f6eadab3df2d1ddc3"
    libPrefix: "lib-{{.Version}}"
    tgzPaths:
    - "*/glob/patterns/*.proto"

You’ll notice we support 3 types of dependencies: go, binaries, and libraries.

For Go tools (where you need a go tool but it’s not a dependency of your app), we just use the Go toolchain to install the version you specify.

For binaries, we download the file or archive from a location we calculate based on a template, a version, OS, and architecture. If you’re downloading an archive, you can specify which file to extract from it. Binaries will live in a .ext/bin directory inside your project.

We also need you to give us the SHA of the artifact we’re downloading, so we can make sure there’s no trickery!

For libraries, we assume you’re downloading archives, either zip or tgz. You can again use a template for the download URL. Libraries live in .ext/lib. You can use globbing patterns to select which files to unpack from the archive.

Again, we need a SHA to verify integrity.

You can use the Depfile from mage-loot itself as an example.

Sample project

To demonstrate the use of mage-loot, we created a small sample project you can try out. The application is pretty simple: it fetches a random image of a shoe at build time and recreates the image as ascii-art. First let’s take a look at the Depfile:

go:
 sver:
   importPath: "github.com/aserto-dev/sver/cmd/sver"
   version: "v1.3.9"
 
bin:
 ascii-image-converter:
   url: https://github.com/TheZoraiz/ascii-image-converter/releases/download/v{{.Version}}/ascii-image-converter_{{ if eq .OS "linux" }}Linux{{ else }}{{if eq .OS "darwin" }}macOS{{ else }}Windows{{ end }}{{ end }}_{{ if eq .Arch "amd64" }}amd64_64bit{{ else }}arm64_64bit{{end}}.{{ if eq .OS "windows"}}zip{{else}}tar.gz{{end}}
   version: "1.11.0"
   entrypoint: '{{ if eq .OS "windows" }}ascii-image-converter.exe{{else}}ascii-image-converter{{end}}'
   tgzPaths:
     - ascii-image-converter_{{ if eq .OS "linux" }}Linux{{ else }}{{if eq .OS "darwin" }}macOS{{ else }}Windows{{ end }}{{ end }}_{{ if eq .Arch "amd64" }}amd64_64bit{{ else }}arm64_64bit{{end}}/ascii-image-converter
   zipPaths:
     - ascii-image-converter_Windows_{{ if eq .Arch "amd64" }}amd64_64bit{{ else }}arm64_64bit{{end}}/ascii-image-converter.exe
   sha:
     linux-amd64: "17f87445971fdb491c6ca9d4beed67044a330339b47775adffb8e3cb898ec15d"
     darwin-amd64: "c22c862818b4af5be9cdd11d49cf147cb2fea354a5d13ad3ef54702b5b1391a4"
     windows-amd64: "1c549e6aee6817dd7dd06377e6b198589cbaca17f848b09c523542c6b2a0cee4"

As you can see, we are using the Depfile to specify the particular versions of the ascii-image-converter package that are relevant for each OS and architecture. In our magefile, we use the Deps function to install all the dependencies:

// Deps installs dependency tools for the project
func Deps() {
 deps.GetAllDeps()
}

Next, the Build function retrieves the dependencies for the Shoe function on build time.

// Builds the binary
func Build() error {
  mg.SerialDeps(Shoe)
  return common.Build()
}


Finally, the Shoe function is where we use the dependency:

// Gets a random shoe image and makes ascii art
// It writes it to shoe.txt
func Shoe() error {
 toAscii := deps.BinDep("ascii-image-converter")
 
 err := downloadFile("shoe.jpg", "https://api.lorem.space/image/shoes?w=400&h=400")
 if err != nil {
   return err
 }
 
 return toAscii("./shoe.jpg", "--save-txt", "./cmd/shoe/", "--only-save")
}

At Aserto, we use mage-loot regularly, but we’d love to hear your thoughts about it. Let us know what you think in our community Slack or on twitter!