A Dive in KSL

WOW, what an experience

The Dive

I just finally emerged from the deep ocean of the KSL (Kitchen Sink Language). That was one of my most challenging coding experience, ever.

The code is very complex (A compiler after all), very clean, using OO completely to the 3rd level, and having this strange results:

"100% code coverage, gives you 10% usage coverage"

My dive was guided by the Compiler group:

• Joseph D. Darcy's Sun Weblog So You Want To Change,
• Alexander Hristov Hacking the OpenJDK compiler
• and Matthias Ernst's Weblog Cascading First Patch which gave me the energy to try.

First, I wanted to use the OpenJDK project but the I did not manage to execute the Ant atsk for building and testing just the javac compiler. Furthermore, I'm an IntelliJ user (hard to change my habits) and reconfiguring the source paths to exclude all except the classes needed by javac was not on my agenda ;-)

Building the KSL javac
So I went for the Kitchen Sink Language project.
The README.html page after subversion checkout is quite helpful, and the build/test environment is good. You need to configure the "build.properties" like:

build.jdk.version = 1.7.0
build.release = ${build.jdk.version}-opensource
build.number = b00
build.user.release.suffix = ${user.name}_${build.fullversion.time}
build.full.version = ${build.release}-${build.user.release.suffix}-${build.number}

# Set jtreg.home to jtreg installation directory
jtreg.home=/work/java.net/jtreg

# Set test.jdk.home to baseline JDK used to run the tests
#test.jdk.home=jdk
test.jdk.home=/opt/java/1.7.0

compiler.source.level = 1.5

The /opt/java/1.7.0 is pointing to a previously build OpenJDK trunk (rev 237).

I did not manage to execute the jtreg test from ant. It seems that the jtreg Ant task lost the basedir?

So I modified it to run jtreg directly (for linux), with my personal filter for abstractEnum:

 <fail unless="jtreg.home" message="Property 'jtreg.home' needs to be set to the jtreg installation directory."/>
 <fail unless="test.jdk.home" message="Property 'test.jdk.home' needs to be set to the baseline JDK to be used to run the tests"/>
 <exec command="${jtreg.home}/linux/bin/jtreg">
    <arg value="-jdk:${test.jdk.home}"/>
    <arg value="-Xbootclasspath/p:${dist.javac}/lib/javac.jar"/>
    <arg value="-verbose"/>
    <arg value="-dir:test"/>
    <arg value="-r:${build.jtreg}/work"/>
    <arg value="-w:${build.jtreg}/report"/>
    <arg value="tools/javac/abstractEnum"/>
  </exec>

I also configured an IntelliJ project, containing the sources, the test sources and the Ant build.xml file.

Then I found out that I could not debug the compiler because it uses debuglevel="source,lines" when IntelliJ needs debuglevel="source,lines,vars", so I changed the build.xml.

Creating test cases

So, like every good Test Driven developer I started the work by writting the java code I wanted the compiler to accept. To integrate into the test suite you need jtreg. It is very easy to understand and use, and I really like the simple tags. I mainly developed by example looking at other tests definitions.

So my first test was:

public abstract enum AE1 {
    int i;
    AE1() {i=2;}
   AE1(int pi) {i=pi;}
    public int getI() {return i;}
}

To compile it, the parser needed to be changed and the clean code of the Parser class makes it easy. I used to work with javacc, and I really think direct java coding of a grammar is a lot more readable, "easier" to debug and to extend.

So, after the parser, I created another test:

public enum E1 extends AE1 {
    one(1),two;
    E1() {
   }
   E1(int pi) {
        super(pi);
   }
}

Here I got bad class file all the time when the resolving of AE1 took place. Wanting to move forward I created one java file with all the classes inside, to ease the loadClass():

public class AE2 {
    AbstractE2 ae2;

    public static void main(String[] args) {
        System.out.println("Before");
        AE2 t = new AE2();
        t.ae2 = E21.one;
        System.out.println("Youpi "+t.ae2.f()+"  "+t.ae2.full());
        t.ae2 = E22.twenty2;
        System.out.println("Youpi "+t.ae2.full());
        System.out.println("After");
        E21 e21 = E21.one;
        switch (e21) {
           case one:
               System.out.println("Got 1");
           break;
           case two:
               System.out.println("Got 2");
           break;
       }
   }
}

abstract enum AbstractE2<E> {
    public int f() {
        return ordinal();
   }
    public String full() {
        return name() + ":" + ordinal();
   }
}

enum E21 extends AbstractE2 {
   one, two;
}

enum E22 extends AbstractE2 {
   twenty2, twenty3;
}

This removed the loadClass issue and gave me all the errors related to actual compiler work. Here it took me some time to understand who is really doing what, and to control a step by step approach.

Customizing the environment

First, the compiler needs to be compiled with himself. So, the Ant does it in 2 steps, compile the bootstrap compiler then the javac compiler with the bootstrap. The problem is that if you destroyed the code, there is no way to move. To protect against this problem, you need to make sure you have a good set of if (...) that identify the special case you are working on and not anything else. For the "abstract enum" it took me too much time to identify this good tests (enum like before, enum with abstract declaration).

Second, There are a lot of steps in the compilation, and to control it found this in RecognizedOptions :

/* This is a back door to the compiler's option table.
* -XDx=y sets the option x to the value y.
* -XDx sets the option x to the value x.
*/
 new HiddenOption(XD) {
     String s;
     public boolean matches(String s) {
         this.s = s;
         return s.startsWith(name.optionName);
    }
     public boolean process(Options options, String option) {
         s = s.substring(name.optionName.length());
         int eq = s.indexOf('=');
         String key = (eq < 0) ? s : s.substring(0, eq);
         String value = (eq < 0) ? s : s.substring(eq+1);
         options.put(key, value);
         return false;
    }
},

And this in the JavaCompiler :

compilePolicy = CompilePolicy.decode(options.get("compilePolicy"));

static CompilePolicy decode(String option) {
if (option == null)
    return DEFAULT_COMPILE_POLICY;
else if (option.equals("attr"))
    return ATTR_ONLY;
else if (option.equals("check"))
    return CHECK_ONLY;
else if (option.equals("simple"))
    return SIMPLE;
else if (option.equals("byfile"))
    return BY_FILE;
else if (option.equals("bytodo"))
    return BY_TODO;
else
     return DEFAULT_COMPILE_POLICY;
}

There is also a lot of good assertions inside the compiler code that better be activated. So, I create a small java class playing with the options and activating Javac main directly.

And here is list of options I used:
For the JVM level options:

• -ea -esa This one is at the JVM level of the activation of the test code.
• -Xbootclasspath/p:dist/lib/javac.jar becareful you need jdk 1.7.0 to run it

For the javac options:

• -XDcompilePolicy=attr for attribute only pass
• -XDcompilePolicy=simple do the job in a very linear way, which makes the visitor on the code happens in a very different order than normal compile. I found it a very important check to verify that the code respect the visitor pattern and code of the compiler.
• -printsource is generating XXX.java files instead of XXX.class and do also a different visitor activation. Very important step to check the Parser/MemberEnter steps.
• -XDverboseCompilePolicy for showing what the compiler is doing. Very important.
• -verbose standard and sometimes annoying (I removed it after a while)
• -XDdev activate dev verbose (Don't know exactly what it means)
• -Xlint:unchecked verify when you messed up with generics ;-)
• -XDstdout very helpful to have compile error in standard out and compiler crashes and exception in error.

So here is my small test class that helped me a lot (Short cycle compile/test/debug).


The going down

This was really, really tough times for me. The issue is that java.lang.Enum does not have a default constructor but harcoded (String name, int ordinal) params. So Attr, TransTypes and even Lower, keep adding/removing this arguments types from the list of type parameters. The ''enum XXX'' and ''abstract enum YYY'' have the name and ordinal parameter added during the "synthetic sugar" phase of the Lower visitor. So, it does not match. I finally solved the issues the following ways:

1. Make some nice good test isDeclaredEnum() and isAbstractEnum() on the Symbol class
2. Allow enum to call super of an abstract enum
3. Found out that only the erasure_field method type has the 2 extra parameters, allowing a nice test (sugar already done or not)

When I finally manage to compile all the classes in one file "AE2.java", I went on the bad class file issue.

The final
I found out that a bad compiled class is a bad class. Logical no!

So, after solving all the issues above, my class was good. The last problem was the signature of abstract enum classes loaded from a XXX.class file. Here again the issue was with the erasure_field is null by default on class load, and in enum he is the only one to contain name and ordinal...

So, I just initialized it on ClassReader with the first well defined type.
Perfect, ''abstract enum'' at last...

Links
The patch and the tests are available here AbstractEnumOnKSL.tgz
Starting to work on the annotations now...