Coursier

2.1.17

2.0.0-RC2

July 12, 2019

Alexandre Archambault

coursier 2.0.0-RC2 was released some time ago. This release brings much less changes than 2.0.0-RC1. It mostly add scala 2.13.0 support, and a few fixes in the experimental install command.

2.0.0-RC1

July 12, 2019

Alexandre Archambault

This post describes the most notables changes of coursier 2.0.0-RC1.

Strict conflict manager

The CLI now accepts options to enable and configure the new strict conflict manager.

  • --strict enables the strict conflict manager,
  • --strict-exclude org:name excludes some modules from being inspected by the strict conflict manager,
  • --strict-include org:name includes some modules (even if they were excluded by --strict-exclude).

install command

The CLI of coursier now has an install command. This command was demo-ed during the Scala Days talk "All the fancy things you can do with flexible dependency management".

The environment variable COURSIER_EXPERIMENTAL needs to be true for this command to be enabled. It then allows things like

$ coursier install scalac sbt-launcher

It installs launchers for these applications in an OS-specific directory (~/Library/Application Support/Coursier/bin on OS X, ~/.local/share/coursier/bin on Linux), and will warn you if that directory isn't in your PATH.

It also allows to install GraalVM applications if GRAALVM_HOME is set (coursier install coursier-graalvm), and scala-native ones too if LLVM is installed (coursier install echo-native).

It should be described in more detail in a later post and / or dedicated page in the coursier website.

publish command

The CLI of coursier now has a publish command, allowing to publish arbitrary sbt projects and directories with a Maven repository-like structure. It's quite experimental, and should be documented in a later time.

Dynamically load scala-native tools

The --native option of the bootstrap command can now deal with several scala-native versions. It automatically detects the scala-native version of the dependencies it is passed, and downloads and uses the scala-native tools from that scala-native version

For example, both of these command work, and use either scala-native 0.3.9, or 0.4.0-M2:

$ coursier bootstrap io.get-coursier:echo_native0.3_2.11:1.0.1 \
    -o echo-native-0.3 \
    --native
$ coursier bootstrap io.get-coursier:echo_native0.4.0-M2_2.11:1.0.3 \
    -o echo-native-0.4.0-M2 \
    --native

Dependency string completion

coursier now has an API and a CLI command to complete dependency strings, like

val (fromIndex, completions) = coursier.Complete()
  .withInput("com.chuusai::shapeless:")
  .withScalaVersion("2.12.8")
  .complete()

$ coursier complete --scala-version 2.12.8 com.chuusai::shapeless:
2.3.2
2.3.3

Fixes

Fix bug in resolution preventing software.amazon.awssdk:utils:2.5.17 and software.amazon.awssdk:auth:2.5.17 to be resolved fine.

Other

  • --what-depends-on now accepts patterns like org:* or foo-*:name (thanks to @viqueen
  • upon checksum error, both the underlying file and checksum file are deleted from the cache, and downloaded again (no other attempt / deletion is the second one still fails)
  • global exclusions can be passed to coursier.Resolve and coursier.Fetch (rather than manually applied to each dependency)
  • early support for latest.release and latest.integration versions (improved upon in later versions)
  • versions like 0+ are now handled fine
  • launchers generated by the bootstrap command now read the environment variable JAVA_OPTS when they invoke java

1.1.0-M14

July 5, 2019

Alexandre Archambault

This post describes the most notables changes of coursier 1.1.0-M14.

Mirrors

coursier got support for mirrors, kind of like Maven has. Mirrors allow to substitute some repositories for others. This can be useful with internal Maven repositories mirroring Maven Central for example. In that case, you'd prefer requests to go to that mirror rather than to Maven Central.

Mirrors are automatically picked from the API (unless explicitly disabled) and the CLI.

They can be configured via configuration files, namely ~/.config/coursier/mirror.properties on Linux, and ~/Library/Preferences/Coursier/mirror.properties on OS X. For example, these settings allow to replace Maven Central with JCenter

jcenter.from=https://repo1.maven.org/maven2
jcenter.to=https://jcenter.bintray.com

Mirrors should be described in more detail in the coursier documentation in a later time.

New ways of passing credentials

The CLI and API of coursier now accepts credentials via the environment, via Java properties, or configuration files. These are described in more detail in the dedicated documentation page. Beware that some details may change a bit before the release of 2.0.0 final.

Refactoring

Redirections

HTTP redirections are now handled by some custom code, rather than by java.net.HttpURLConnection. (This introduced some regressions, now fixed in subsequent releases.) This allows a more refined handling of credentials when redirections happen (credentials can be stripped, or added if their settings allow it, in particular).

This should also allow users to more finely tune whether credentials should be sent upfront, without waiting for a 403 Unauthorized response (to be added to the dedicated documentation page).

Parsers

Most parsers were moved in dedicated objects under the coursier.parse namespace. That includes DependencyParser, ModuleParser, and RepositoryParser in particular.

High level API

The new high level API kept evolving / getting refined.

Fixes

XML errors

A regression in the code handling XML, since 1.1.0-12, giving SAXParseException, was fixed.

1.1.0-M13

March 5, 2019

Alexandre Archambault

coursier 1.1.0-M13 was just released.

The most notable changes of this milestone happened in the new high level API, that is nearing completion. A notion of "rules" or constraints is also being added to coursier, and should be described in more detail in a later post.

High level API

This API aims at being as simple to use as the CLI of coursier, while retaining the purity of the former low level API - without all its traps.

For example, one can fetch a few dependencies with

import coursier._

val files = Fetch()
  .addDependencies(
    dep"org.http4s:http4s-dsl_2.12:0.18.22",
    dep"org.tpolecat:doobie-core_2.12:0.6.0"
  )
  .run()

// files: Seq[java.io.File]

It relies on a "builder pattern", not unlike sttp. Numerous parameters can be adjusted by calling other methods before run, such as addRepositories, withResolutionParams, withCache, etc.

run() runs resolutions and downloads artifacts synchronously, but can be substituted by future() or io to run things in the background, via a Future or the IO monad of your choice.

More details can be found in the documentation.

The CLI and the sbt plugins of coursier already rely on this API. In the future, this should ensure that more advanced features, added in the CLI or the sbt plugin of coursier, are available to API users straightaway. It's also easier to evolve this API while retaining binary compatibility.

Rules

Motivated by SCP-20, support for enforcing some "rules" or constraints is being added to coursier. These are a bit similar to those of the Maven enforcer plugin, and allow to emulate the strict conflict manager of Ivy in particular.

While details are still being ironed out, the CLI already accepts rules.

For example, one can require all the jackson modules to have the same version with

$ coursier resolve org.apache.spark:spark-sql_2.12:2.4.0 \
    --rules 'SameVersion(com.fasterxml.jackson.*:jackson-*)'
…
com.fasterxml.jackson.core:jackson-annotations:2.7.9:default
com.fasterxml.jackson.core:jackson-core:2.7.9:default
com.fasterxml.jackson.core:jackson-databind:2.7.9:default
com.fasterxml.jackson.module:jackson-module-paranamer:2.7.9:default
com.fasterxml.jackson.module:jackson-module-scala_2.12:2.7.9:default
…

By default, if a rule doesn't pass, it can try to address the underlying issue. One can instead require resolution to fail, with

$ coursier resolve org.apache.spark:spark-sql_2.12:2.4.0 \
    --rules 'fail:SameVersion(com.fasterxml.jackson.*:jackson-*)'
Resolution error: Unsatisfiable rule …

An upcoming post should describe rules and future plans in more detail.

1.1.0-M12

March 4, 2019

Alexandre Archambault

Changes in 1.1.0-M12 are discussed along those of 1.1.0-M13.

1.1.0-M11

February 5, 2019

Alexandre Archambault

coursier 1.1.0-M11 was just released. This release contains numerous enhancements both in the core of coursier and its CLI, along with a few fixes for the CLI.

It occurs in the midst of the addition of the higher level API for coursier, that should be discussed in a later release. This higher level API aims at being as simple to use as the CLI of coursier.

The resolve, fetch, and launch commands were heavily refactored, to extract bits of the high level API from them. Although best efforts were made to prevent them, these refactoring may come with some regressions. Don't hesitate to report any if ever you run into one.

Tweaking / enhancements

Optimizations

Regular expressions were used during resolution to process Maven properties. Some handcrafted code now handles those, which provides nice performance gains.

Maven POM parsing used to be handled by parsing the POM file itself, then processing the resulting XML AST. It's now handled by a SAX parser, which brought some performance gains too.

Default repositories

In the CLI, and in the upcoming high level API, the default repositories are

These can now be tweaked via the environment, via

  • COURSIER_REPOSITORIES in the environment,
  • the coursier.repositories Java property.

This environment variable or Java property are expected to contain a list of repositories, separated by |, like central|sonatype:releases. The repositories themselves are parsed the same way as those passed via the -r or --repository options of the CLI.

Note that this does not affect the sbt plugin.

CLI

Download initial JAR from Central rather than Sonatype releases

The coursier launcher downloads a proguarded version of the CLI of coursier upon first launch. It is now ensured this proguarded JAR is downloaded from Maven Central, rather than Sonatype releases, the former being usually faster than the latter.

Ship completions with launcher

The completions of coursier now ship with the launcher itself. Run

$ coursier --completions zsh

to print the Zsh ones. (Zsh is the only supported shell for now.)

--sbt-plugin option

The resolve and fetch commands now accept a --sbt-plugin option to accept sbt plugin dependencies. Use like

$ coursier resolve --sbt-plugin com.typesafe.sbt:sbt-native-packager:1.3.3

Adjust the scala and sbt versions with the --scala-version and sbt-version options, like

$ coursier resolve \
    --sbt-plugin com.typesafe.sbt:sbt-native-packager:1.3.3 \
    --scala-version 2.10.7 \
    --sbt-version 0.13.8

Don't automatically resolve unrecognized dependencies against Scaladex

Formerly, things passed to the CLI not recognized as dependencies were assumed to be Scaladex queries, like

$ coursier launch ammonite

(Note that the Scaladex resolves this against the wrong module for now.)

Scaladex queries now have to be passed explicitly, like

$ coursier launch --scaladex ammonite

Conflicts handling from CLI (work in progress)

The resolve command can now print potential conflicts among dependencies, like

$ coursier resolve io.get-coursier:coursier-cli_2.12:1.1.0-M10 --conflicts
org.scala-lang:scala-library:2.12.8 was selected, but
  com.chuusai:shapeless_2.12:2.3.3                                wanted version 2.12.4
  com.github.alexarchambault:argonaut-shapeless_6.2_2.12:1.2.0-M8 wanted version 2.12.4
  com.github.alexarchambault:case-app-annotations_2.12:2.0.0-M5   wanted version 2.12.7
  com.github.alexarchambault:case-app-util_2.12:2.0.0-M5          wanted version 2.12.7
  com.github.alexarchambault:case-app_2.12:2.0.0-M5               wanted version 2.12.7
  org.scala-lang:scala-reflect:2.12.6                             wanted version 2.12.6
  org.scala-lang.modules:scala-xml_2.12:1.1.1                     wanted version 2.12.6
  org.typelevel:cats-core_2.12:1.5.0                              wanted version 2.12.7
  org.typelevel:cats-kernel_2.12:1.5.0                            wanted version 2.12.7
  org.typelevel:cats-macros_2.12:1.5.0                            wanted version 2.12.7
  org.typelevel:machinist_2.12:0.6.6                              wanted version 2.12.6
  org.typelevel:macro-compat_2.12:1.1.1                           wanted version 2.12.0

org.scala-lang:scala-reflect:2.12.6 was selected, but
  io.argonaut:argonaut_2.12:6.2.1 wanted version 2.12.4

Pass --fail-if-conflicts to have a non-zero exit code in case of conflicts.

Beware that these options are a work in progress, more options to filter out conflicts should be added in the future.

Reworked bootstrap launchers

The launchers generated by the bootstrap command underwent significant changes, most notably the "standalone" launchers.

Standalone launchers contain the JARs they're supposed to launch as resources. Previously, they were setting up a custom java.net.URLStreamHandlerFactory, to handle custom URL protocols for these JARs. This is no longer the case. Thanks to bits of code imported from the Spring boot loader, these JARs are handled via the jar protocol, resulting in URLs like jar:file:…!/. The logic imported from Spring also has the advantage of exposing those JARs to java.net.URLClassLoader in a way allowing for a performance on par with standard files.

Fixes

Fix stack overflow when printing trees with cyclic dependencies

Regression since 1.1.0-M2. For example, the following command works again

$ coursier resolve -t \
    -r http://cogcomp.cs.illinois.edu/m2repo \
    edu.illinois.cs.cogcomp:illinois-nlp-pipeline:0.1.21

Fail loudly on interrupted downloads from the launcher

The coursier launcher, and the bootstraps it generates, can download some JARs upon first launch. These downloads rely on java.net.HttpURLConnection for http and https, which sometimes succeeds even if the connection is abruptly interrupted, resulting in invalid JARs. These are now detected in the launcher, which then requests users to try running the launcher again.

1.1.0-M10

January 14, 2019

Alexandre Archambault

coursier 1.1.0-M10 was just released. This is the first release of coursier while it's new website is up and running, and sbt-ci-release handles publishing, from Travis CI.

Note that this is a milestone release, not necessarily aiming at retaining binary / source compatibility of the API with earlier versions. If you're using coursier via its API, it is recommended to stay on former versions unless you're interested in the very latest features and are ready to update the way you use the API in later versions.

New launcher URL

One notable change of this release is that the launchers of coursier are now being pushed to GitHub release pages rather than in the master branch of the coursier git repository. The URL to download it thus changes.

Download the latest launcher at either

Download a specific version via the release assets, e.g.

Also

--what-depends-on option

Add a --what-depends-on option to the resolve command, to quickly know which dependencies bring a particular dependency (#997), e.g.

$ coursier resolve \
    --what-depends-on org.htrace:htrace-core \
    org.apache.spark:spark-sql_2.12:2.4.0
  Result:
└─ org.htrace:htrace-core:3.0.4
   ├─ org.apache.hadoop:hadoop-common:2.6.5
   │  └─ org.apache.hadoop:hadoop-client:2.6.5
   │     └─ org.apache.spark:spark-core_2.12:2.4.0
   │        ├─ org.apache.spark:spark-catalyst_2.12:2.4.0
   │        │  └─ org.apache.spark:spark-sql_2.12:2.4.0
   │        └─ org.apache.spark:spark-sql_2.12:2.4.0
   └─ org.apache.hadoop:hadoop-hdfs:2.6.5
      └─ org.apache.hadoop:hadoop-client:2.6.5
         └─ org.apache.spark:spark-core_2.12:2.4.0
            ├─ org.apache.spark:spark-catalyst_2.12:2.4.0
            │  └─ org.apache.spark:spark-sql_2.12:2.4.0
            └─ org.apache.spark:spark-sql_2.12:2.4.0

(Adapted from the coursierWhatDependsOn command of sbt-coursier, itself inspired by the whatDependsOn command of sbt-dependency-graph.)

coursier-bootstrap library

Move the bootstrap generation logic of the bootstrap command to a separate standalone library (io.get-coursier::coursier-bootstrap:1.1.0-M10, #991).

Leaking of classes from the bootstraps

Prevent classes from the coursier launcher or of the logic of coursier bootstrap to leak in apps launched via the fetch command or bootstrap generated via the bootstrap command (#983 - coursier.CachePath or the classes of directories could leak in the launched application classpath.)

LDFLAGS for scala-native executable generation

Accept linking flags via LDFLAGS in the environment, when generating scala-native executables (#981).

Other

  • force fetching artifacts from the CLI with --force-fetch, even if resolution failed (#980)
  • fix JARs landing in the wrong classloader, when using the classloader isolation capabilities of the bootstrap command (#982)
  • refactoring of the cache module (#984)
  • switch to scala 2.12.8 (#986), add support for scala 2.13.0-M5
  • switch to fastparse 2 (#990, still shaded so that it doesn't appear as a dependency to downstream users of coursier)
  • allow to adjust the default number of parallel downloads via the coursier.parallel-download-count Java property (#994)