Jacks (Jacks is an Automated Compiler Killing Suite)

Jacks Logo Betting on Java? You had better be holding a good hand.

What is Jacks?

Jacks is a free test suite designed to detect bugs in a Java compiler. We guarantee that it will find at least one bug in any supported Java compiler, in fact it may find more than one. Jacks is not designed to test a Java runtime (JVM) or Java class libraries. The Mauve project already aims to do that.

Jacks includes a collection of Java compiler tests contributed by people on the net. Jacks is free software licensed under the terms of the GPL. A test case must be licensed under the terms of the GPL to be suitable for inclusion into Jacks, but authors retain the original copyright. The license used by the Jacks test suite does not effect your ability to run the suite with any free or non-free Java compiler.

The layout of directories in the Jacks test suite mimics chapters from the JLS.
The Java Language Specification Version 2 can be found at:

  • http://java.sun.com/docs/books/jls/second_edition/html/jTOC.doc.html
  • http://java.sun.com/docs/books/jls/html (Old Version)

    Install Requirements

    Tcl version 8.3 is required as it provides the regression testing framework Jacks depends on. Older versions of Tcl will not work as they do not include the tcltest extension and they do not support Unicode character sets.

    Windows users can download the Tcl/Tk for Windows installer .

    Red Hat x86 Linux users can download the Tcl 8.3 RPM (Red Hat 6.x) (Red Hat 7.x) .

    Everyone else will need to build it from source: http://dev.scriptics.com/download/tcl/tcl8_3/tcl8.3.2.tar.gz .

    Once Tcl is installed, you will need the Jacks source code from the CVS. The CVS module for jacks can be fetched using the following CVS commands. 

    setenv CVSROOT :pserver:anoncvs@oss.software.ibm.com:/usr/cvs/jikes

cvs login
paswsd anoncvs

cvs checkout jacks

    Configure Jacks for your system

    Before running the tests, you will need to tell Jacks where you have installed each of the compilers you want to use. A setup file exists for each supported compiler. For example, if you want to run tests with the jikes compiler, you would need to edit the jikes_setup configuration file so that includes the path name where your jikes compiler is installed. You may also want to set the CLASSPATH that will be used for the tests and the optional path name of the Java runtime to use for runtime tests. You can add any number of configuration files. To remove a configuration you do not want, just delete the setup file for that configuration.

    Running Jacks regression tests

    To run the tests, invoke the jacks shell script and pass it the name of the compiler you want to test. You will want to put the toplevel directory on your PATH so that you can run the jacks shell script. Test the javac compiler like this:

    % jacks javac

    The above command will recurse into subdirectories and run all the tests it finds using the javac compiler. If any of the tests fail, an error will be displayed.

    You can also run tests with multiple compilers using a single command. This example runs the tests using javac and then jikes.

    % jacks "javac jikes"

    It is also possible to run tests using all of the supported compilers. The following command shows how this is done:

    % jacks all

    You can also run the tests in a specific subdirectory like this: 

    % cd tests/jls/packages/package-declarations/unnamed-packages
% jacks jikes

    In addition, you can pass patterns as the third argument. In this way you can run only those tests that match a given pattern. 

    % jacks jikes "*int*"
    The above command will run all the test with names that match the pattern *int*. You can also pass multiple patterns. 
    % jacks jikes "*int* test-1 *foo-1"

    The Jacks suite includes test that require a Java runtime in addition to a Java compiler. These tests are located in the tests/runtime directory.

    Tests that apply to only a specific compiler are included in a top level directory named after the compiler. For example, tests in the tests/gcj directory will only be run when testing with the gcj compiler.

    Adding a new test case

    Adding a new test case to Jacks is very easy. One of the main goals for Jacks was to create a set of tests that could be extended with minimal effort. A detailed example of Adding A Test Case is provided.

    Review existing tests

    A nicely cross referenced index of every test in the Jacks suite is provided. You can look up a test case by test number, find the JLS section that a given test directory refers to, review each of the test that apply to a given compiler, or simply review all the test that apply to a given JLS section.