- Add Gradle build and release FAQ - Add SpringSource repository FAQ i.e. repo.springsource.org - Deprecate Building from source Explain that the document covers 3.1.x and earlier; that Gradle build instructions can now be found in the README Issue: SPR-8116master
cbeams
14 years ago
committed by
Chris Beams
11 changed files with 428 additions and 9 deletions
@ -0,0 +1,285 @@
@@ -0,0 +1,285 @@
|
||||
_This FAQ has been written with building and releasing the Spring Framework in mind, but is also general enough to serve as a guide for any Spring project that publishes to http://repo.springsource.org. The audience is primarily geared for project leads and committers, but should be valuable for anyone building from source or wanting to understand better how Spring is published, etc. The focus is on Gradle-based builds, but keep in mind that Maven-based Spring projects can take advantage of the same general steps, e.g. using the [Artifactory Maven 3 Task](http://wiki.jfrog.org/confluence/display/RTF/Bamboo+Artifactory+Plug-in) instead of the Artifactory Gradle Task, etc. Note that if you're unfamiliar with the benefits of Artifactory and repo.springsource.org in general, you may want to check out the [[SpringSource repository FAQ]]._ |
||||
|
||||
### Table of Contents |
||||
build-related |
||||
|
||||
* [How do I check out and build the Framework?](#wiki-check_out_and_build) |
||||
* [How long should a build take?](#wiki-build_duration) |
||||
* [Why are there so many Javadoc warnings?](#wiki-javadoc_warnings) |
||||
* [Why are compile-time warnings suppressed?](#wiki-compilation_warnings) |
||||
* [What are the most important tips for Gradle newbies?](#wiki-gradle_tips) |
||||
* [Why do I get 401/403 errors when downloading dependencies?](#wiki-authentication) |
||||
|
||||
release-related |
||||
|
||||
* [How and where are snapshots published?](#wiki-snapshot_publication) |
||||
* [How do I perform a Milestone, RC, or GA release?](#wiki-release_process) |
||||
* [What about publishing artifacts to Maven Central?](#wiki-maven_central) |
||||
* [How are docs and schemas and distribution zips published?](#wiki-docs_schema_dist_publication) |
||||
|
||||
*** |
||||
<a name="wiki-check_out_and_build"/> |
||||
# How do I check out and build the Framework? |
||||
See "Building from source" in the README in root of the 3.2.x (master) source tree. This also includes information on importing projects into Eclipse/STS/IDEA. |
||||
|
||||
*** |
||||
<a name="wiki-build_duration"/> |
||||
# How long should a build take? |
||||
When running `./gradlew build` for the first time, it is likely that the Gradle wrapper script will need to download Gradle for you. Gradle distributions run around 30MB, so this step will be connection-speed dependent. |
||||
|
||||
You will also need to download all of Spring's compile- and test-time dependencies. This currently runs around 120MB. Keep in mind here that almost all of these dependencies are optional at runtime for applications that use Spring. It's only when building from source that you actually need them all! |
||||
|
||||
Once you've bootstrapped a Gradle distribution and downloaded dependencies, you won't need to do it again; they are cached in your $HOME/.gradle directory. A complete `./gradle clean build` at this point will take between 5 and 15 minutes depending on your clock and disk speed. A solid state drive makes a huge difference here! |
||||
|
||||
As is also mentioned below in the 'tips' section, you'll want to break yourself of any habits executing the `clean` task during normal development iterations. Gradle has excellent _incremental build_ support, meaning that once you've built Javadoc, compiled, run test, etc, Gradle won't execute these tasks again unless the inputs for those tasks (e.g. .java files) have changed. As a general rule, just run `./gradle build` or `./gradle test` (without `clean`) to keep things snappy. |
||||
|
||||
Also, consider running with the `-a` flag to avoid evaluating other subprojects you depend on. For example, if you're iterating on changes in spring-webmvc, cd into the spring-webmvc directory and run `../gradlew -a build` to tell gradle to evaluate and build *only* that subproject. |
||||
|
||||
Finally, the Gradle daemon helps greatly in eliminating startup overhead. This feature will soon be 'on by default', but in the meantime you need to run gradle with the `--daemon` flag, or alternatively, export your `GRADLE_OPTS` environment variable to include `-Dorg.gradle.daemon=true` |
||||
|
||||
*** |
||||
<a name="wiki-javadoc_warnings"/> |
||||
# Why are there so many Javadoc warnings? |
||||
When running `./gradlew build` for the first time (or any time one runs `./gradlew clean build`), you will notice that most subprojects have emit many warnings during the `javadoc` task. This is simply because, over time, many warnings have accumulated. Committers are encouraged to fix these proactively, and pull requests are welcome from the community. It's a general goal to get these cleaned up as quickly as possible. |
||||
|
||||
*** |
||||
<a name="wiki-compilation_warnings"/> |
||||
# Why are compile-time warnings suppressed? |
||||
You'll notice that `build.gradle` includes the following line: |
||||
|
||||
[compileJava, compileTestJava]*.options*.compilerArgs = ['-Xlint:none'] |
||||
|
||||
This tells Gradle to suppress all warnings during compilation. The reason for this is that the framework currently has many warnings, most of which are related to generics usage -- particularly raw type warnings -- e.g. using `Class` instead of `Class<?>`. This is an artifact switching to Java 5 in Spring 3. As with the Javadoc warnings mentioned above, committers are encouraged to fix these warnings whenever possible. Once the bulk of them are eliminated, we can switch to -Xlint:all. In the meantime, it's just creates unnecessary noise in the build output. |
||||
|
||||
*** |
||||
<a name="wiki-gradle_tips"/> |
||||
# What are the most important tips for Gradle newbies? |
||||
|
||||
## Use the Gradle daemon |
||||
As mentioned above, the daemon greatly speeds up Gradle by eliminating startup overhead. Run with `--daemon` or set `GRADLE_OPTS=-Dorg.gradle.daemon=true`. Very occasionally, you may need or want to run `./gradlew --stop` to kill the daemon process. |
||||
|
||||
## Stop typing `clean` unless you really need it |
||||
As mentioned above, Gradle has excellent _incremental build_ capabilities, meaning that it will only compile, run tests, generate javadoc, etc. if content has actually changed. However, telling gradle to `clean` everything eliminates this benefit. It is a common practice -- particularly for longtime Maven users -- to type 'clean' with every build as a matter of course. Break yourself of this habit and you'll find builds running much faster. |
||||
|
||||
## Understand selective / task-specific cleaning |
||||
Following onto the item above, there are certainly cases in which you want to clean, but you may not want to clean _everything_. Instead of typing `./gradlew clean build`, consider using selective cleaning, in which you clean only the outputs for a given task. For example |
||||
|
||||
$ ./gradlew :spring-oxm:cleanTest build |
||||
|
||||
will eliminate the only the `spring-oxm/build/classes/test` directory and then run the entire build. This means that you still benefit from incremental build, but for sure will re-execute spring-oxm's test suite. The rule here is "clean<Task>", where <Task> is the capitalized name of the task you want to clean. Other examples would be: `cleanJar`, `cleanJavadoc`, `cleanDistZip`, etc. |
||||
|
||||
## Understand the `-a/--no-rebuild` switch |
||||
As mentioned earlier in this document, the `-a` switch tells Gradle to ignore evaluting and building any other subprojects. An example would be when iterating on changes to spring-webmvc. You may be running `./gradlew test` over and over again in the spring-webmvc directory, and each time you do this, Gradle will check all the other subprojects that spring-webmvc depends on to see if they've changed. This is generally useful functionality, but if you know better, you may want to save yourself the extra few seconds that these checks can take. Simply run `../gradlew -a test` within the spring-webmvc directory, and you'll see the difference. |
||||
|
||||
## Run only the tasks you need |
||||
Rather than running `./gradlew build` every time you want to simply compile and test, run `./gradlew test`. If you want only to generate the jar for a particular subproject, run `../gradlew jar` in that subproject's directory. |
||||
|
||||
Run `./gradlew tasks` to get an overview of available tasks and their descriptions. Run `./gradlew tasks --all` to get an even more exhaustive list. |
||||
|
||||
## Qualify tasks by :project for precision |
||||
When you run `./gradlew build` from the root directory, Gradle recurses through all projects, finding those that actually have a 'build' task, and executing it if it exists. This kind of heuristic behavior makes for convenient usage, but you can also be more specific. For example, from the root directory you can run `./gradlew :spring-core:test` and Gradle will run the 'test' task only within the spring-core subproject. |
||||
|
||||
A particularly relevant example of this is when running the integration tests for Spring Framework. As of the switch to Gradle, there is no longer a dedicated integration-tests subproject. Rather, these tests now live in the root project's src/test directory. If you run `./gradlew test` from the root directory, Gradle will run not only the tests in the root src/test directory, but also the tests for every subproject! To narrow it down and run _only_ the root project tests, type `./gradlew :test`. The ':' here says "the 'test' task in the root project". Contrast this again with `./gradlew :spring-core:test`. |
||||
|
||||
## Skip tasks with the `-x/--exclude-task` switch |
||||
Perhaps you'd like to run a complete `clean build`, but don't want to take the time to run the test suites? Do this with `./gradlew clean build -x test`. -x tells Gradle to exclude the task from the build lifecycle. Specify multiple tasks to exclude with multiple '-x' switches, e.g. `./gradlew clean build -x javadoc -x test` |
||||
|
||||
## Run a single test from the command line |
||||
Perhaps you're iterating on a particular unit test in spring-context and don't want to run the thousands of tests in its test suite just to see if your latest changes work. Run `./gradlew test -Dtest.single=MyTests`. There's no need to qualify the packaging here. Gradle will simply find any class named 'MyTests' and run just those tests. |
||||
|
||||
## Get more information with the `--info` switch |
||||
By default, Gradle's command line output is blissfully minimalistic. It will only tell you at a high level what it's doing right now, e.g.: |
||||
|
||||
:spring-asm:compileJava UP-TO-DATE |
||||
:spring-asm:processResources UP-TO-DATE |
||||
:spring-asm:classes UP-TO-DATE |
||||
:spring-asm:repackageAsm UP-TO-DATE |
||||
:spring-asm:jar UP-TO-DATE |
||||
:spring-core:compileJava |
||||
|
||||
If there are warnings, e.g. compiler or Javadoc warnings, those will show up, because Gradle wants you to see that there's something wrong. But if everything is going well, Gradle just keeps quiet. In certain cases, however, you might want to see exactly which tests are running when, or exactly what requests are being made to download dependencies, etc. Run `./gradlew -i` (or `--info`) to get a bit more verbosity. Run with `-d` / `--debug` to switch on the firehose. |
||||
|
||||
## Use Gradle's built-in IDE metadata generation |
||||
Prior to the switch to Gradle, we manually maintained Eclipse and IDEA project metadata. This a hassle and error-prone to say the least. Gradle provides robust support for generating this metadata, and for this reason we no longer check these files into source control. You'll notice that all these files now have entries in .gitignore. Please don't check them in! |
||||
|
||||
See the `import-into-eclipse.sh` and `import-into-idea.md` files in the root of the source tree for detailed instructions. |
||||
|
||||
Note that both IDEA and STS are hard at work on even more advanced Gradle tooling from the IDE side. At the time of this writing, neither one is quite capable of meeting all the needs of a build as complex as Spring's, but they'll get there soon. In the meantime, Gradle's built-in support works quite well. |
||||
|
||||
## Understand handling of 'optional' and 'provided' dependencies. |
||||
This commit provides all the detail you'll ever want on how we manage optional and provided dependencies with regard to our generated Maven poms: |
||||
|
||||
TODO - url to commit |
||||
|
||||
## Use the `init.gradle` init script to configure repo.springsource.org authentication |
||||
See the FAQ item on [authentication](#wiki-authentication) below. |
||||
|
||||
## Use `mavenLocal()` to resolve dependencies from your Maven cache |
||||
If you need to compile against a locally built version of a dependency, you'll need to add your local $HOME/.m2 repository to the set of repositories that Gradle searches during the build. |
||||
|
||||
This can be done directly in your build script by adding `mavenLocal()` as follows: |
||||
|
||||
repositories { |
||||
mavenLocal() |
||||
maven { url "http://repo.springsource.org/libs-release" } |
||||
} |
||||
|
||||
However, it's recommended that you do this via an init script, to avoid accidentally checking in the `mavenLocal()` entry. If you're already using the authentication init script described above, you can append the following to it, or if not, just create a new file in `$HOME/.gradle` named `init.gradle` with the following: |
||||
|
||||
allprojects { |
||||
repositories { |
||||
mavenLocal() |
||||
} |
||||
} |
||||
|
||||
## Use `gradle install` to publish artifacts to your local Maven cache |
||||
If you're building Spring artifacts from source in order to compile against them from another project (perhaps another Spring project, even, like Spring Batch or Spring Integration or the Spring Data-* family), you'll need to publish jars into your local $HOME/.m2 cache in order to pick them up from the dependent project. Do this with `./gradlew install`. |
||||
|
||||
## Consider using the 'find-gradle' convenience script |
||||
The Gradle wrapper is a big help, but it can be cumbersome to have to use relative pathing, e.g. `./gradlew`, `../gradlew` when working with the build. Follow the instructions here to install a simple shell script that allows you to type `gradle` anywhere and finds the nearest `gradlew` or falls back to your system-wide Gradle installation (if you have one): https://github.com/cbeams/shell-scripts/blob/master/find-gradle |
||||
|
||||
*** |
||||
<a name="wiki-authentication"/> |
||||
# Why do I get 401/403 errors when downloading dependencies? |
||||
Our Artifactory instance at http://repo.springsource.org is configured such that requesting and caching external dependencies, e.g. junit or cglib from Maven Central) can only be performed by authenticated users, i.e. SpringSource employees. Once such a dependency has been successfully cached in the repository, then even anonymous/unauthenticated users may download them. This configuration helps ensure that repo.springsource.org contains only Spring project artifacts and their transitive dependencies, and that we don't act as a kind of 'open proxy'. |
||||
|
||||
This means that even if you are not authenticated, you will not receive 401/403 unless _you're downloading a new dependency for the first time._ A typical example might be when upgrading to a newly-released version of JUnit or Hibernate. The solution to this problem is to provide your Artifactory authentication credentials to Gradle, such that it can successfully respond to 401 challenges when Artifactory issues them. The best way to do this is with a Gradle _init script_, which keeps your username and password separate from your build script. |
||||
|
||||
See https://github.com/SpringSource/gradle-init-scripts#readme for simple instructions on how to dowload and install an init script that does just this. This is recommended for all committers. |
||||
|
||||
*** |
||||
<a name="wiki-snapshot_publication"/> |
||||
# How and where are snapshots published? |
||||
The [3.2.x CI build plan](https://build.springsource.org/browse/SPR-B32X) is configured to use the [Artifactory Bamboo plugin](http://wiki.jfrog.org/confluence/display/RTF/Bamboo+Artifactory+Plug-in). Each time commits are pushed to [SpringSource/master](https://github.com/SpringSource/spring-framework/tree/master), this build plan |
||||
|
||||
1. executes `./gradlew build`, producing jars and distribution zip files |
||||
2. publishes these artifacts to http://repo.springsource.org/libs-snapshot-local ([browse in Artifactory tree view](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-snapshot-local%3Aorg%2Fspringframework)) |
||||
|
||||
Along with individual artifacts, [build metadata](http://wiki.jfrog.org/confluence/display/RTF/Build+Integration) is also published to Artifactory, including information about the environment and principal involved in the build, as well as allowing for bi-directional links between Bamboo and Artifactory. Click on the 'Artifactory' tab for any successful CI build to navigate to the associated Artifactory build. |
||||
|
||||
 |
||||
|
||||
Likewise, click on "Show in CI Server" from any individual Artifactory build to navigate back to the CI build that published it. |
||||
|
||||
 |
||||
|
||||
Users wishing to consume snapshot builds may add http://repo.springsource.org/snapshot to the list of repositories in their build script. See [[Downloading Spring artifacts]] for more details. |
||||
|
||||
*** |
||||
<a name="wiki-release_process"/> |
||||
# How do I perform a Milestone, RC, or GA release? |
||||
|
||||
The steps are simple, and almost everything is done via the Bamboo and Artifactory UIs. |
||||
|
||||
## Steps at a glance |
||||
|
||||
1. Stage the release into the libs-staging-local repository |
||||
2. Verify and test the staged artifacts |
||||
3. Promote the release to libs-milestone-local (or libs-release-local as appropriate). |
||||
4. Merge release branch |
||||
5. Announce the release |
||||
|
||||
## Steps in detail |
||||
|
||||
### 1. Stage the release |
||||
The Artifactory Bamboo plugin mentioned above also includes sophisticated [Release Management](http://wiki.jfrog.org/confluence/display/RTF/Bamboo+Artifactory+Plugin+-+Release+Management) capabilities. This feature allows for publishing releases _directly from CI_, including creating a release branch and/or tag; incrementing the project version; and publishing to the libs-staging-local, libs-milestone-local or libs-release-local repositories as appropriate. |
||||
|
||||
To access this feature, click on the "[Default Job](https://build.springsource.org/browse/SPR-B32X-JOB1)" for the Spring 3.2.x build plan, where you'll see a link to "[Artifactory Release Management](https://build.springsource.org/build/release/viewVersions.action?buildKey=SPR-B32X-JOB1)". Fill out the form fields there and click "Build and Release to Artifactory". Typical values -- in this case for a milestone release -- look something like the following: |
||||
|
||||
[[release-staging.png]] |
||||
|
||||
In the example above, the `version` property key refers to the property of the same name declared in `gradle.properties` in the root of the source tree. This value will actually be modified and updated in source control during the release process. |
||||
|
||||
Using a 'release branch' is optional, but recommended. This means that updates to the `gradle.properties` file will occur on a branch named 3.2.0.M1, helping to isolate the release from changes on the master branch, and also allowing for simplified rollback in case a last minute change needs to be made after staging the release. |
||||
|
||||
'Create VCS Tag' is also checked, indicating that a git tag named 'v3.2.0.M1' should be created, pointing to the commit where the `version` property is incremented. |
||||
|
||||
Notice that 'Next development version comment' is blank. Because this is a milestone release, the 'next integration value' of the `version` property is configured to return to it's previous value of '3.2.0.BUILD-SNAPSHOT'. The tooling is smart enough here to avoid creating an additional commit, so no comment is necessary. |
||||
|
||||
Importantly, notice that we're publishing to the 'libs-staging-local' repository - this is just what it sounds like: a staging area that allows us to test out the release internally before finally promoting it to the actual 'libs-milestone-local' repository and announcing it to the world. |
||||
|
||||
With these values supplied, click 'Build and Release to Artifactory'. |
||||
|
||||
### 2. Verify staged artifacts |
||||
When the staging build and release process is complete, you can navigate to the associated build record in Artifactory to verify that all modules were published as expected, e.g.: |
||||
|
||||
[[build-browser.png]] |
||||
|
||||
Note that in the Artifactory tree view, you can easily drill into jars and zips to inspect their contents, e.g. manifest files, javadoc, reference docs, etc, e.g.: |
||||
|
||||
[[tree-drill.png]] |
||||
|
||||
In the example above, clicking the 'Download' button will open the API javadocs in the browser -- a nice convenience. |
||||
|
||||
You may also wish to have internal team members 'smoke test' the release, e.g. change their sample projects and dependent framework builds to point to http://repo.springsource.org/libs-staging-local and compile/test/run against the staged artifacts. |
||||
|
||||
### 3. Promote the release |
||||
When verification is complete, return to the build in Bamboo from which you staged the release and click 'Artifactory'. You'll now see 'Promotion' options as follows: |
||||
|
||||
[[build-promotion.png]] |
||||
|
||||
The 'Target promotion repository' is set to 'libs-milestone-local'. Click 'Update' to move all artifacts from 'libs-staging-local' to 'libs-milestone-local'. |
||||
|
||||
### 4. Merge the release branch |
||||
At this point, the release is complete and successful, so the release branch should be merged back into master, e.g. |
||||
|
||||
$ cd spring-framework # your local spring-framework working copy |
||||
$ git checkout master |
||||
$ git fetch --all # to fetch the branch created during the release |
||||
$ git merge springsource/master # make sure you're up to date |
||||
$ git merge springsource/3.1.0.M1 # assuming your remote is named 'springsource' |
||||
$ git push springsource master:master |
||||
|
||||
### 5. Announce the release! |
||||
At this point, announcements may be made and users may consume the released artifacts by adding http://repo.springsource.org/libs-milestone-local to their build scripts. |
||||
|
||||
*** |
||||
<a name="wiki-maven_central"/> |
||||
# What about publishing artifacts to Maven Central? |
||||
|
||||
GA releases of Spring Framework are published not only to http://repo.springsource.org/libs-release-local, but also to Maven Central at http://repo1.maven.org. This allows for maximum convenience for the majority of Spring users, given that most users have Maven-based builds and Maven resolves artifacts by default from Maven Central. |
||||
|
||||
The preferred way of releasing artifacts to Maven Central is via Sonatype's Nexus server at oss.sonatype.org (OSO). This is explained in detail in [Sonatype's OSS usage guide](https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide). |
||||
|
||||
The SpringSource Artifactory repository has been customized with a "[nexus-push](https://github.com/JFrogDev/artifactory-user-plugins/tree/master/nexus-push)" plugin that allows for automatic propagation of builds from Artifactory to the Nexus server at OSO for publication into Maven Central. |
||||
|
||||
All Spring projects -- that is, all projects having groupid `org.springframework` -- can publish to OSO under the shared 'springsource' account. This has already been set up in the nexus-push plugin, so there's no additional setup necessary at OSO, even for new projects. |
||||
|
||||
The Artifactory Bamboo plugin supports use of the nexus-push plugin through it's UI. Step 3 of the the FAQ entry above on publishing releases described the process for promoting a build out of staging. If the build is a GA release, simply choose the 'Push to Nexus' option, and select 'libs-release-local' as the target repository: |
||||
|
||||
[[push-to-nexus.png]] |
||||
|
||||
Choosing this option means that the build will first be published into a staging repository at OSO and 'closed' in Nexus terminology. The 'closing' process will check your build artifacts to ensure they meet the [requirements for publication into Maven Central](https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide#SonatypeOSSMavenRepositoryUsageGuide-6.CentralSyncRequirement), e.g. that POMs are properly formed, that all artifacts have checksums, PGP signatures, etc. If there are any errors in the repository closing process, they will be displayed in the Bamboo UI and the promotion process will fail. At this point you'll need to correct the issues and walk through the release staging process described above once again. |
||||
|
||||
Note that with regard to requirements for OSO onboarding, Artifactory automatically generates sha1 and md5 checksums for all artifacts, so you don't need to worry about this. Furthermore, a [custom plugin](https://github.com/JFrogDev/artifactory-user-plugins/blob/master/pgp-sign/pgpSign.groovy) has been developed for repo.springsource.org that adds PGP signatures (.asc files) on the fly during upload using the buildmaster@springframework.org PGP key. You simply need to make sure that your build produces jars (including -sources and -javadoc jars) and well-formed poms. Any zip files such as distribution or doc zips are excluded from the promotion process to OSO. |
||||
|
||||
When the promotion process is complete, i.e. closing the staging repository at OSO succeeds, there is one additional step - you must log into http://oss.sonatype.org with the 'springsource' account. You will see your closed staging repository there, and you must manually click the 'promote' button. It looks something like this: |
||||
|
||||
 |
||||
|
||||
Pressing the promote button means that your artifacts will be published into Maven Central, but beware -- there's no going back after this point! |
||||
|
||||
*** |
||||
<a name="wiki-docs_schema_dist_publication"/> |
||||
# How are docs and schemas and distribution zips published? |
||||
First, you'll find three important tasks in the root build.gradle script: `docsZip`, `schemaZip`, and `distZip`. As you might guess, these create zip files containing docs and schemas in the case of _docsZip_ and _schemaZip_ respectively; _distZip_ aggregates the contents of the first two and adds in all classes jars, -sources jars and -javadoc jars. |
||||
|
||||
As described in the release process documentation above, all artifacts produced by the Spring Framework build are published into Artifactory, including these zip archives. This provides a consistent storage mechanism, but ultimately the docs and schema zips need to be published and unpacked at http://static.springframework.org/spring-framework/docs and http://www.springframework.org/schema, respectively. For example: |
||||
|
||||
* http://static.springsource.org/spring-framework/docs/3.1.0.RELEASE (changelog, api javadocs, reference docs) |
||||
* http://www.springframework.org/schema/context (xsd files for spring-context) |
||||
|
||||
Performing these uploads directly from the build script is problematic. It requires the build to use SSH libraries in order to SCP and unpack the zip files, which is already complex, but worse it requires that the operator of the build script has the correct SSH key authentication configured on the remote servers, and that they are within the VMware VPN. |
||||
|
||||
To avoid this complexity, a separate process called 'autorepo' runs periodically (every 10 minutes), querying Artifactory for these for docs and schema zips. When new ones are found, the autorepo process does the heavy SSH lifting to upload and unpack them at the sites mentioned above. This script is currently under development, but you can see the results of the prototype effort at http://static.springsource.org/autorepo/. |
||||
|
||||
In order for autorepo to work properly, these artifacts must be annotated in Artifactory with custom metadata. This metadata is attached to the artifacts on upload by the Artifactory Bamboo plugin. Here's the configuration in the Spring Framework 3.2.x build plan: |
||||
|
||||
[[artifact-properties.png]] |
||||
|
||||
Full documentation for this feature can be found in the [Artifactory Gradle plugin documentation](http://wiki.jfrog.org/confluence/display/RTF/Gradle+Artifactory+Plugin#GradleArtifactoryPlugin-ThePropertiesClosureDSL) |
||||
|
||||
The `type` property tells autorepo that the artifact is a 'docs-zip', 'schema-zip', or 'dist-zip'. The `deployed` property tells autorepo whether it has already uploaded and unpacked this artifact. When autorepo detects a new docs or schema zip (deployed == false), it performs the uploading and unpacking, and then sets the `deployed` property to `true`. |
||||
|
||||
The distribution zip, on the other hand, remains within Artifactory and the SpringSource [community download page](www.springsource.com/download/community) queries repo.springsource.org to provide the list of dist zip downloads for each project. It uses the same metadata mentioned above to perform the search. |
||||
|
||||
@ -1,5 +1,6 @@
@@ -1,5 +1,6 @@
|
||||
## Welcome! |
||||
|
||||
* [README](../spring-framework/#readme) |
||||
* [[Building from source]] |
||||
* [[Downloading Spring artifacts]] |
||||
* [[Gradle build and release FAQ]] |
||||
* [[SpringSource repository FAQ]] |
||||
|
||||
@ -0,0 +1,127 @@
@@ -0,0 +1,127 @@
|
||||
This document explains the purpose, nature, and best practices for use of the SpringSource repository at http://repo.springsource.org. For basic instructions on downloading Spring artifacts manually or using Maven and other build systems, see [[downloading Spring artifacts]]. |
||||
|
||||
### Table of Contents |
||||
* [For the impatient](#wiki-impatient) |
||||
* [What is the SpringSource repository?](#wiki-what_is_repo) |
||||
* [What repositories are available?](#wiki-available_repositories) |
||||
* [Can I resolve EBR artifacts via repo.springframework.org?](#wiki-ebr) |
||||
* [Will artifacts still be published to Maven Central?](#wiki-maven_central) |
||||
* [What about existing artifacts published to S3 / maven.springsource.org?](#wiki-s3) |
||||
* [What are the benefits of the SpringSource repository?](#wiki-benefits) |
||||
|
||||
*** |
||||
<a name="wiki-impatient"/> |
||||
# For the impatient |
||||
* End user application builds should use http://repo.springsource.org/release, /milestone and /snapshot to resolve Spring project artifacts. [details](#wiki-available_repositories) |
||||
* Spring project builds should use http://repo.springsource.org/libs-release, /libs-milestone, and /libs-snapshot to resolve Spring project artifacts and all transitive dependencies. [details](#wiki-available_repositories) |
||||
* Older-style http://maven.springsource.org URLs will continue to work, but switch to repo.springsource.org for new development. [details](#wiki-s3) |
||||
* GA Spring project artifacts will continue to be available via Maven Central. [details](#wiki-maven_central) |
||||
|
||||
*** |
||||
<a name="wiki-what_is_repo"/> |
||||
# What is the SpringSource repository? |
||||
The SpringSource repository is an [Artifactory](http://www.jfrog.com) instance hosted by JFrog on behalf of SpringSource at http://repo.springsource.org. Artifactory is a sophisticated _repository manager_ and it is used to host all SpringSource open source project artifacts as well as their transitive dependencies. It can be browsed and searched directly via the web at http://repo.springsource.org, and accessed by Maven, Gradle and Ivy as well as any other build system capable working against Maven/Ivy repository layouts. |
||||
|
||||
Spring project teams use repo.springsource.org as a single source for resolving all project dependencies and storing all project artifacts. Users of Spring projects can use repo.springsource.org to resolve Spring artifacts and their dependencies. |
||||
|
||||
*** |
||||
<a name="wiki-available_repositories"/> |
||||
# What repositories are available? |
||||
repo.springsource.org hosts three categories of repositories: _local_, _remote_ and _virtual_. The complete list of repositories can be browsed [here](http://repo.springsource.org/webapp/simplebrowserroot.html); below we describe the most important among these these. |
||||
|
||||
**Local repositories** typically host artifacts produced directly by Spring projects. The three main local repositories are: |
||||
|
||||
* [libs-snapshot-local](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-snapshot-local%3A): BUILD-SNAPSHOT (development) versions |
||||
* [libs-milestone-local](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-milestone-local%3A): M* (milestone) and RC (release candidate) versions |
||||
* [libs-release-local](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-release-local%3A): RELEASE (generally available) versions |
||||
|
||||
_Note: see [[spring artifact versioning|Downloading Spring artifacts#wiki-artifact_versioning]] for more information_ |
||||
|
||||
There also exist [plugins-snapshot-local](http://repo.springsource.org/webapp/browserepo.html?pathId=plugins-snapshot-local%3A) and [plugins-release-local](http://repo.springsource.org/webapp/browserepo.html?pathId=plugins-release-local%3A) repositories, used for hosting custom build system plugins, e.g. the [docbook-reference-plugin](https://github.com/cbeams/gradle-plugins). |
||||
|
||||
**Remote repositories** serve as caches for artifacts hosted elsewhere, for example [Maven Central](http://repo1.maven.org) or the [JBoss repository](http://repository.jboss.org/nexus). These repositories are not typically accessed directly, but rather are _aggregated_ by virtual repositories as discussed below. |
||||
|
||||
Three additional important repositories are [libs-snapshot-s3-cache](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-snapshot-s3-cache%3A), [libs-milestone-s3-cache](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-milestone-s3-cache%3A) and [libs-release-s3-cache](http://repo.springsource.org/webapp/browserepo.html?pathId=libs-release-s3-cache%3A). These cache Spring project artifacts previously deployed to s3://maven.springframework.org. More on these below. |
||||
|
||||
**Virtual repositories** aggregate any combination of _local_ and _remote_ repositories to provide convenient and controlled access to select sets of artifacts. For developers building applications against Spring projects, the most important virtual repositories are: |
||||
|
||||
* [snapshot](http://repo.springsource.org/snapshot) (aggregates _libs-snapshot-local_ and _libs-snapshot-s3-cache_) |
||||
* [milestone](http://repo.springsource.org/milestone) (aggregates _libs-milestone-local_ and _libs-milestone-s3-cache_) |
||||
* [release](http://repo.springsource.org/release) (aggregates _libs-milestone-local_ and _libs-milestone-s3-cache_) |
||||
|
||||
Therefore, adding http://repo.springsource.org/release as a repository within your build will ensure that you have seamless access to all SpringSource GA release artifacts, whether they were originally deployed against our older S3 repository or directly against repo.springsource.org. |
||||
|
||||
If you use http://maven.springframework.org URLs in your builds today, you'll be interested to know that DNS for maven.springframework.org now points to repo.springsource.org. These old repository urls (e.g. http://maven.springframework.org/snapshot) used to pull directly from S3, but they now pull from repo.springsource.org. There's nothing you need to change to adapt to this; you're free to keep these URLs in use (we'll preserve the DNS entry), but for clarity, we recommend switching to repo.springsource.org for any new development. |
||||
|
||||
Another class of virtual repository allows for resolution not only of SpringSource-produced artifacts, but also of their transitive dependencies: |
||||
|
||||
* [libs-release](http://repo.springsource.org/libs-release) (aggregates _release_, and all third-party GA _remote_ repositories) |
||||
* [libs-milestone](http://repo.springsource.org/libs-milestone) (aggregates _milestone_ and _libs-release_) |
||||
* [libs-snapshot](http://repo.springsource.org/libs-snapshot) (aggregates _snapshot_, _libs-milestone_, and all third-party snapshot _remote_ repositories) |
||||
|
||||
These virtual repositories are quite powerful. Declaring http://repo.springsource.org/libs-release in your build script ensures that you can resolve *all* Spring project artifacts and downstream dependencies, but *only* GA versions of them. |
||||
|
||||
By switching from _libs-release_ to _libs-milestone_, this constraint is relaxed to include not only all GA libs and dependencies, but also any marked as Milestone or RC. |
||||
|
||||
And finally, by declaring _libs-snapshot_, you open it all the way up, giving your build access to any Spring artifacts and downstream dependencies, from GA versions to development snapshots. |
||||
|
||||
The point here is that one _need not add additional repository entries_ in one's build script, but should rather _choose the repository with the necessary level of stability_. The goal is to leave Spring project build scripts with one URL for all their needs. Keep in mind that this approach helps ensure builds perform well, too -- the fewer repositories configured at the build script level, the fewer HTTP requests that need to be issued from your machine. Let the server do that work! |
||||
|
||||
Users of Spring projects (i.e. application developers) may also choose to use these repositories, but should note that for security reasons, only authenticated users (i.e. Spring project committers) may resolve and cache new dependencies against repo.springsource.org. Read [[this FAQ entry|Gradle-build-and-release-FAQ#wiki-authentication]] for complete details, but in short, non-authenticated users will probably not be able to resolve _every_ one of their application dependencies against repo.springsource.org, but will rather need to fall back to Maven Central or another third-party repository. |
||||
|
||||
_Note: naming may change in the near future for libs-release, libs-milestone and libs-snapshot in order to better express the hierarchical and stability-oriented nature of these repositories._ |
||||
|
||||
*** |
||||
<a name="wiki-ebr"/> |
||||
# Can I resolve EBR artifacts via the repo.springsource.org? |
||||
Certain third-party libraries, such as IBM's UOW jar or the atinject TCK jar exist *only* in the SpringSource EBR, and the Spring Framework has (optional) dependencies on several of these. For this reason, the [ebr-maven-external](http://repo.springsource.org/webapp/browserepo.html?pathId=ebr-maven-external-cache%3A) remote repository has been added to Artifactory, and if necessary it can add be added as an additional repository to your build configuration. Otherwise, Spring users resolving dependencies as bundles suitable for use in an OSGi container should continue to use the dedicated repository URLs as described in the [EBR FAQ](http://ebr.springsource.com/repository/app/faq#wiki-q8). |
||||
|
||||
*** |
||||
<a name="wiki-maven_central"/> |
||||
# Will artifacts still be published to Maven Central? |
||||
Yes. As always, GA (.RELEASE) versions of Spring Framework and all the other Spring projects will be [published to Maven Central](http://search.maven.org/#wiki-search%7Cga%7C1%7Cg%3A%22org.springframework%22) as well as repo.springsource.org. Spring project leads and committers should read [[this FAQ entry|Gradle build and release FAQ#wiki-maven_central]] for details. |
||||
|
||||
*** |
||||
<a name="wiki-s3"/> |
||||
# What about existing artifacts published to S3 / maven.springsource.org? |
||||
maven.springframework.org DNS now points to repo.springsource.org. /snapshot, /milestone and /release paths map seamlessly and pick up artifacts in S3 as well as in repo.springsource.org. Read the entry above on [available repositories](#wiki-available_repositories) for details. |
||||
|
||||
*** |
||||
<a name="wiki-benefits"/> |
||||
# What are the benefits of the SpringSource repository? |
||||
This is not an exhaustive list of Artifactory benefits, but rather a select list of the benefits that Spring project teams and developers using Spring projects can expect to enjoy from using repo.springsource.org. |
||||
|
||||
## simpler build scripts, simpler release process |
||||
As described [above](#wiki-available_repositories), **Spring project builds need only one repository URL** to resolve all dependencies. This means simpler configuration as well as faster builds, due to fewer HTTP requests being issued on the client. |
||||
|
||||
Thanks to Artifactory's deep CI integration, **build scripts do not need to manage deployment or publication of artifacts**. As described [[in this FAQ entry|Gradle build and release FAQ#wiki-release_process]], release management can be driven almost completely from the Bamboo UI at http://build.springsource.org. This results in a significant reduction of custom logic in individual project builds, and also **removes the need for VPN, S3 and SSH access**, because **all communication with repo.springsource.org happens over HTTP**. |
||||
|
||||
## one place for all Spring artifacts, one search box to find them |
||||
The SpringSource repository provides search over *all* Spring project artifacts -- jars (including source and javadoc), distribution zips, XSD files, and all dependencies. No need for finding an S3 browser or visiting the various Java artifact search portals. In addition to [quick](http://wiki.jfrog.org/confluence/display/RTF/Quick+Search) search, Artifactory also provides the ability to search the repository for artifacts containing a particular [class](http://wiki.jfrog.org/confluence/display/RTF/Class+Search), searching by groupId/artifactId/version/classifier ([GAVC](http://wiki.jfrog.org/confluence/display/RTF/GAVC+Search)), and even searching based on custom [properties](http://wiki.jfrog.org/confluence/display/RTF/Property+Search). All of this is possible not only via the web, but via the Artifactory [REST API](http://wiki.jfrog.org/confluence/display/RTF/Artifactory%27s+REST+API) as well. See [below](#wiki-customizability) for examples where custom properties and searching via the API help satisfy complex requirements. |
||||
|
||||
## tight control over dependency management |
||||
As described [above](#wiki-available_repositories), repo.springsource.org's virtual repositories allow Spring project teams and Spring users alike to restrict or allow access to snapshot, milestone, RC, and GA versions -- and all transitive dependencies -- with a single URL. For example, during different phases of project development, one may wish to disallow use of any snapshot dependencies, or disallow use of anything other than GA versions of dependencies. repo.springsource.org makes this simple. |
||||
|
||||
## improved build traceability and reproducibility |
||||
Again, thanks to Artifactory's CI integration with Bamboo, first-class metadata about builds and their artifacts are published to Artifactory, allowing for bi-directional linking between a particular build in Bamboo and the artifacts it published into Artifactory. Environment variables, system statistics, build tool versions, and principal information are also published as part of this build metadata, ensuring that artifacts can be reproduced exactly if necessary. |
||||
|
||||
## ability to host artifacts that do not exist or do not belong in Maven Central or other repositories |
||||
For example, a custom Gradle [docbook-reference-plugin](https://github.com/SpringSource/gradle-plugins) has been developed to generate HTML and PDF reference documentation for Spring projects. This plugin is specifically tailored to meet the needs of Spring projects and is not a general-purpose utility. Therefore, it need not live in Maven Central or any other repository. The [plugin-snapshot-local](http://repo.springsource.org/webapp/browserepo.html?pathId=plugins-snapshot-local%3A) and [plugin-release-local](http://repo.springsource.org/webapp/browserepo.html?pathId=plugins-release-local%3A) repositories at repo.springsource.org provide just the right kind of management for these artifacts. |
||||
|
||||
Another example is hosting non-jar artifacts such as distribution, docs, and schema archives. These could technically be stored in Maven Central, but repo.springsource.org is a more natural home. In addition, the Artifactory UI allows for browsing into these archives in-web, and even rendering HTML within them (as in the case of Javadocs or HTML reference docs). This is simple and convenient and intuitive. |
||||
|
||||
Yet another example is hosting build system configuration files, such as Maven settings.xml or Gradle init scripts. As described in the [gradle-init-scripts project readme](https://github.com/SpringSource/gradle-init-scripts#readme), this not only allows for a centralized location for project teams to store these files, but can also take advantage of Artifactory's support for [filtered resources](http://wiki.jfrog.org/confluence/display/RTF/Filtered+Resources), e.g. injecting usernames and passwords into these documents during download, etc. |
||||
|
||||
The list here can go on. In short, any artifact that we wish to share and make available for use by our build systems can be hosted at repo.springsource.org. This allows a degree of flexibility not otherwise available. |
||||
|
||||
## rich support for all major build tools |
||||
Artifactory provides dedicated tasks for building and publishing artifacts from Maven-, Gradle-, and Ant-based builds at both the [CI](http://wiki.jfrog.org/confluence/display/RTF/Bamboo+Artifactory+Plug-in#BambooArtifactoryPlug-in-ConfiguringMaven3%2CGradleandIvyBuilders) and [build](http://wiki.jfrog.org/confluence/display/RTF/Working+with+Maven) [script](http://wiki.jfrog.org/confluence/display/RTF/Working+with+Gradle) [levels](http://wiki.jfrog.org/confluence/display/RTF/Working+with+Ivy). Given that the various SpringSource projects use all three of these, this allows for a common approach to CI and release management while affording individual teams the freedom to choose the tools that work best for them. |
||||
|
||||
## watches and notifications |
||||
Project leads and team members can place [watches](http://wiki.jfrog.org/confluence/display/RTF/Watches) on any artifact, directory, or entire repository within repo.springsource.org in order to be notified by email when artifacts are added, changed or deleted. This can be useful for monitoring one's own project, but also for tracking published changes to projects that ones project depends on. |
||||
|
||||
<a name="wiki-customizability"/> |
||||
## customizability to complex requirements |
||||
For example, and as explained in [[this FAQ entry|Gradle build and release FAQ#wiki-docs_schema_dist_publication]], publishing Spring project documentation requires VPN and SSH access and detailed understanding of directory structures at static.springframework.org and other sites. By tagging docs and distribution zips with [custom properties](http://wiki.jfrog.org/confluence/display/RTF/Properties), we are able to search Artifactory programmatically and with precision via its [REST API](http://wiki.jfrog.org/confluence/display/RTF/Artifactory%27s+REST+API), and let an external cron script operating within the VPN handle this 'heavy lifting'. |
||||
|
||||
Another example is supporting the requirements for entry into Maven Central. As explained in [[this FAQ entry|Gradle build and release FAQ#wiki-maven_central]], a custom Artifactory plugin ensures that checksums and PGP signatures are created automatically for each artifact on upload, while another plugin facilitates automatic propagation of builds from repo.springsource.org into oss.sonatype.org staging repositories. All of this means less repetitive and error-prone work at the individual build-script level. |
||||
Loading…
Reference in new issue