PATCH Abstract Enum with OpenJDK

The full solution in OpenJDK

Applying from KSL to OpenJDK

First, there are no issues (except a diff of 2 lines in the GPL license block ;-) to apply a patch from KSL to the OpenJDK project trunk checkout.

So I applied my KSL patch and I started modifying "sun/reflect/annotation/AnnotationParser.java" to retrieve the Enum.valueOf() from the parameter class type and not from the type definition. It means that in:

@AnnoAETest(ae2 = E21.one)
private String first;

where:

public @interface AnnoAETest {
  AbstractE2 ae2();
}

the AnnotationParser was doing Enum.valueOf(AbstractE2.class,"one"), but it had actually the class E21 defined, so I changed it to the good Enum.valueOf(E21.class,"one").
So the patch is:

 Index: j2se/src/share/classes/sun/reflect/annotation/AnnotationParser.java
===================================================================
--- j2se/src/share/classes/sun/reflect/annotation/AnnotationParser.java (revision 237)
+++ j2se/src/share/classes/sun/reflect/annotation/AnnotationParser.java (working copy)
@@ -422,9 +422,12 @@
             if (!enumType.getName().equals(typeName))
             return new AnnotationTypeMismatchExceptionProxy(
                 typeName + "." + constName);
-        } else if (enumType != parseSig(typeName, container)) {
-            return new AnnotationTypeMismatchExceptionProxy(
-                typeName + "." + constName);
+        } else {
+            Class paramEnumType = parseSig(typeName, container);
+            if (!enumType.isAssignableFrom(paramEnumType))
+                return new AnnotationTypeMismatchExceptionProxy(
+                    typeName + "." + constName);
+            enumType = paramEnumType;
         }

         try {

Now, another issue appears, is that the complicated test to know if a class is an enum with member or an abstract enum or anonymous class is complicated and the Class.isEnum() implementation was wrong.
After changing to the following implementation to remove specialized enum (anonymous):

 Index: j2se/src/share/classes/java/lang/Class.java
===================================================================
--- j2se/src/share/classes/java/lang/Class.java (revision 237)
+++ j2se/src/share/classes/java/lang/Class.java (working copy)
@@ -2877,8 +2877,7 @@
        // An enum must both directly extend java.lang.Enum and have
        // the ENUM bit set; classes for specialized enum constants
        // don't do the former.
-       return (this.getModifiers() & ENUM) != 0 &&
-       this.getSuperclass() == java.lang.Enum.class;
+       return ((this.getModifiers() & ENUM) != 0 && !isAnonymousClass());
     }

     // Fetches the factory for reflective objects

I'm getting for my following last failed test:

public class AnnoAEUsage {
  @AnnoAETest(ae2 = E21.one)
   private String first;

  public static void main(String[] args) throws Exception {
      System.out.println("Is enum "+E21.class.isEnum());
      E21 e21 = Enum.valueOf(E21.class, "one");
      switch (e21) {
          case one:
               System.out.println("Got 1");
              break;
          default:
              System.err.println("No got 1");
      }
      Field field = AnnoAEUsage.class.getDeclaredField("first");
      AnnoAETest anno = field.getAnnotation(AnnoAETest.class);
      AbstractE2 ae2 = anno.ae2();
      System.out.println("Youpi again "+ae2.full());
  }
}

enum E21 extends AbstractE2 {
  one, two;
}

The following output:

JDK Build:/work/java.net/ksl.trunk$/work/java.net/openjdk.trunk/control/build/linux-amd64/j2sdk-image/bin/java -cp build/test-classes abstractEnum.annotation.AnnoAEUsage
Is enum true
Got 1
Exception in thread "main" java.lang.IllegalAccessError: tried to access class abstractEnum.annotation.AbstractE2 from class $Proxy3
       at $Proxy3.ae2(Unknown Source)
       at abstractEnum.annotation.AnnoAEUsage.main(AnnoAEUsage.java:32)

Some more work...

Stupid isn't it
Well it turn out this one is logical...
It's related to an open bug: 6256803 which is not really a bug in my view. You just need to expose public your interfaces, since the dynamic proxy implementing annotation interface is not part of the package. So my AbstractE2 abstract enum, and the E22, E21 real enums were package protected, so: IllegalAccessError.
After making all the test classes public, it works...

The final OpenJDK Patch
For the b14 of openJDK I needed to apply for my platform: 6567018
The above patch file is for "hotspot/src/share/vm/gc_implementation/"

The root project for my patches (KSL and OpenJDK) and all tests files is here.

For the OpenJDK all patches were executed under:
URL: https://openjdk.dev.java.net/svn/openjdk/jdk/trunk
Repository Root: https://openjdk.dev.java.net/svn/openjdk
Revision: 239
And they can be downloaded here.

What's next

So, now what's left to do is to remove the "extends Enum" and make the generics declarations ">" generated by synthetic sugar.

I think that during the work on generics I will try to see the difficulty of having generics for Enum also?

But before all that I think I will start to apply this internal JDK to the fields-enum and other projects, to really judge the improvements...

The "reopened" RFE was accepted

Thank you for taking the time to suggest this enhancement to the Java Standard Edition.

We have determined that this report is an RFE and has been entered into our internal RFE tracking system under Bug Id: 6570766

1. Voting for the RFE
Click http://bugs.sun.com/bugdatabase/addVote.do?bug_id=6570766

2. Adding the report to your Bug Watch list.
You will receive an email notification when this RFE is updated. Click http://bugs.sun.com/bugdatabase/addBugWatch.do?bug_id=6570766

Abstract Enum on KSL

The reasons in few lines

Give the ability to create abstract classes between the final enum declaration containing the static enumerated values, and the abstract superclass for all enum java.lang.Enum.

So an AbstractColumn.java file can contain:

public abstract enum AbstractColumn {
String columnName;
int precision;
AbstractColumn() {
columnName = name();
precision = 0;
}
AbstractColumn(String c, int p) {
....
}
public String getColumnName() {return columnName();}
}

And inside an O/R Mapping framework an annotation can be:
public interface @Column {AbstractColumn definition();}

And then the application users have the level of indirection they need to provide logic to the framework using the static annotations. For example:

public enum CustomerColumn extends AbstractColumn {
firstName, lastName, age;
public String getColumnName() {
return StringUtils.capitalizeWithUnderscore(name());
}
}

@Entity
public class Customer {
@Column(definition = CustomerColumn.firstName)
private String firstName;
}

Then all the problems of hardcoded schema names in JPA entities (quite annoying), copy/paste, string based errors (no type safety), and so on disappear.
From my experience, the limitations of Annotations are really pushing developers away, and this feature can unlock the issue.

The KSL or OpenJDK patch
Please check A dive into the KSL A big dive (more like drowning) into the Kitchen Sink Language!, for how this patch was made and the references.

So, I finally manage to write a patch for abstract enum with the following limitations:

1. Abstract enum cannot extends other abstract enums (the extends close needs to be empty)
2. You need to declare the constructors you are using in the final enum extending the abstract enum and use super() calls if needed
3. You can declare the generics signature > in the abstract Enum, and use enum MyEnum for the final one. It does not mean generics are allowed for enum,
4. but for the moment I did not find synthetic sugar for generating generics that I could based my work on?
5. I had the following problem: '''The enum containing anonymous classes does not work anymore with this patch'''. The main problem is that after Check an implicit abstract flag is set on the enum and don't find a way to distinguish between the implicit and explicit abstract flag :-( I solved it using tree.mods and check for protected constructors in the second version of the patch.
6. So the only problem left is the usage of Abstract Enum in Annotations. It works at compile time but failed at runtime. So I need to work on the OpenJDK now. I'm getting:

java.lang.annotation.AnnotationTypeMismatchException: Incorrectly typed data found for annotation element public abstract abstractEnum.annotation.AbstractE2 abstractEnum.annotation.AnnoAETest.ae2() (Found data of type LabstractEnum/annotation/E21;.one)
at sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy.generateException(AnnotationTypeMismatchExceptionProxy.java:56)
at sun.reflect.annotation.AnnotationInvocationHandler.invoke(AnnotationInvocationHandler.java:74)
at $Proxy3.ae2(Unknown Source)
at abstractEnum.annotation.AnnoAEUsage.main(AnnoAEUsage.java:21)


The patch is here Abstract_Enum_in_KSL_second_version.patch which is based on "compiler-1.7.0-ea-b05". I applied the patch to "openjdk/jdk7/b13" in the folder "j2se" and it matches. I did not test the rebuild, yet.

The test code is based on a closed bug number about abstract enum (Bug ID: 6507006) is here here JtregTests.jar, and only the test of abstract enum in annotations fails.

The draft for JSL modifications
8.1.1.1 abstract Classes
Enum types (§8.9) can be declared abstract.

8.1.4 Superclasses and Subclasses
The optional extends clause in a normal class declaration specifies the direct
superclass of the current class.

Super:
extends ClassType

The following is repeated from §4.3 to make the presentation here clearer:

ClassType:
TypeDeclSpecifier TypeArgumentsopt

A class is said to be a direct subclass of its direct superclass. The direct super-class is the class from whose implementation the implementation of the current class is derived. The direct superclass of an enum type E is Enum, unless a SuperEnum is declared.

8.9 Enums
An enum declaration has the form:
EnumDeclaration:
ClassModifiers ''opt'' enum Identifier SuperEnum''opt'' Interfaces''opt'' EnumBody
SuperEnum:
extends EnumType
EnumType:
TypeDeclSpecifier
EnumBody:
{ EnumConstants''opt'' ,''opt'' EnumBodyDeclarations''opt'' }

The body of an enum type may contain enum constants, if the enum is not declared abstract. An enum constant defines an instance of the enum type. A non abstract enum type has no instances other than those defined by its enum constants.
The superclass of an enum can only be another enum declared abstract.
Generics declaration enum are not allowed, and will result in a compile-time error.


14.11 The switch Statement

The type of the Expression must be char, byte, short, int, Character,
Byte, Short, Integer, or a non abstract enum type (§8.9), or a compile-time error occurs.

A Dive in KSL

WOW, what an experience

The Dive

I just finally emerged from the deep ocean of the KSL (Kitchen Sink Language). That was one of my most challenging coding experience, ever.

The code is very complex (A compiler after all), very clean, using OO completely to the 3rd level, and having this strange results:

"100% code coverage, gives you 10% usage coverage"

My dive was guided by the Compiler group:

• Joseph D. Darcy's Sun Weblog So You Want To Change,
• Alexander Hristov Hacking the OpenJDK compiler
• and Matthias Ernst's Weblog Cascading First Patch which gave me the energy to try.

First, I wanted to use the OpenJDK project but the I did not manage to execute the Ant atsk for building and testing just the javac compiler. Furthermore, I'm an IntelliJ user (hard to change my habits) and reconfiguring the source paths to exclude all except the classes needed by javac was not on my agenda ;-)

Building the KSL javac
So I went for the Kitchen Sink Language project.
The README.html page after subversion checkout is quite helpful, and the build/test environment is good. You need to configure the "build.properties" like:

build.jdk.version = 1.7.0
build.release = ${build.jdk.version}-opensource
build.number = b00
build.user.release.suffix = ${user.name}_${build.fullversion.time}
build.full.version = ${build.release}-${build.user.release.suffix}-${build.number}

# Set jtreg.home to jtreg installation directory
jtreg.home=/work/java.net/jtreg

# Set test.jdk.home to baseline JDK used to run the tests
#test.jdk.home=jdk
test.jdk.home=/opt/java/1.7.0

compiler.source.level = 1.5

The /opt/java/1.7.0 is pointing to a previously build OpenJDK trunk (rev 237).

I did not manage to execute the jtreg test from ant. It seems that the jtreg Ant task lost the basedir?

So I modified it to run jtreg directly (for linux), with my personal filter for abstractEnum:

 <fail unless="jtreg.home" message="Property 'jtreg.home' needs to be set to the jtreg installation directory."/>
 <fail unless="test.jdk.home" message="Property 'test.jdk.home' needs to be set to the baseline JDK to be used to run the tests"/>
 <exec command="${jtreg.home}/linux/bin/jtreg">
    <arg value="-jdk:${test.jdk.home}"/>
    <arg value="-Xbootclasspath/p:${dist.javac}/lib/javac.jar"/>
    <arg value="-verbose"/>
    <arg value="-dir:test"/>
    <arg value="-r:${build.jtreg}/work"/>
    <arg value="-w:${build.jtreg}/report"/>
    <arg value="tools/javac/abstractEnum"/>
  </exec>

I also configured an IntelliJ project, containing the sources, the test sources and the Ant build.xml file.

Then I found out that I could not debug the compiler because it uses debuglevel="source,lines" when IntelliJ needs debuglevel="source,lines,vars", so I changed the build.xml.

Creating test cases

So, like every good Test Driven developer I started the work by writting the java code I wanted the compiler to accept. To integrate into the test suite you need jtreg. It is very easy to understand and use, and I really like the simple tags. I mainly developed by example looking at other tests definitions.

So my first test was:

public abstract enum AE1 {
    int i;
    AE1() {i=2;}
   AE1(int pi) {i=pi;}
    public int getI() {return i;}
}

To compile it, the parser needed to be changed and the clean code of the Parser class makes it easy. I used to work with javacc, and I really think direct java coding of a grammar is a lot more readable, "easier" to debug and to extend.

So, after the parser, I created another test:

public enum E1 extends AE1 {
    one(1),two;
    E1() {
   }
   E1(int pi) {
        super(pi);
   }
}

Here I got bad class file all the time when the resolving of AE1 took place. Wanting to move forward I created one java file with all the classes inside, to ease the loadClass():

public class AE2 {
    AbstractE2 ae2;

    public static void main(String[] args) {
        System.out.println("Before");
        AE2 t = new AE2();
        t.ae2 = E21.one;
        System.out.println("Youpi "+t.ae2.f()+"  "+t.ae2.full());
        t.ae2 = E22.twenty2;
        System.out.println("Youpi "+t.ae2.full());
        System.out.println("After");
        E21 e21 = E21.one;
        switch (e21) {
           case one:
               System.out.println("Got 1");
           break;
           case two:
               System.out.println("Got 2");
           break;
       }
   }
}

abstract enum AbstractE2<E> {
    public int f() {
        return ordinal();
   }
    public String full() {
        return name() + ":" + ordinal();
   }
}

enum E21 extends AbstractE2 {
   one, two;
}

enum E22 extends AbstractE2 {
   twenty2, twenty3;
}

This removed the loadClass issue and gave me all the errors related to actual compiler work. Here it took me some time to understand who is really doing what, and to control a step by step approach.

Customizing the environment

First, the compiler needs to be compiled with himself. So, the Ant does it in 2 steps, compile the bootstrap compiler then the javac compiler with the bootstrap. The problem is that if you destroyed the code, there is no way to move. To protect against this problem, you need to make sure you have a good set of if (...) that identify the special case you are working on and not anything else. For the "abstract enum" it took me too much time to identify this good tests (enum like before, enum with abstract declaration).

Second, There are a lot of steps in the compilation, and to control it found this in RecognizedOptions :

/* This is a back door to the compiler's option table.
* -XDx=y sets the option x to the value y.
* -XDx sets the option x to the value x.
*/
 new HiddenOption(XD) {
     String s;
     public boolean matches(String s) {
         this.s = s;
         return s.startsWith(name.optionName);
    }
     public boolean process(Options options, String option) {
         s = s.substring(name.optionName.length());
         int eq = s.indexOf('=');
         String key = (eq < 0) ? s : s.substring(0, eq);
         String value = (eq < 0) ? s : s.substring(eq+1);
         options.put(key, value);
         return false;
    }
},

And this in the JavaCompiler :

compilePolicy = CompilePolicy.decode(options.get("compilePolicy"));

static CompilePolicy decode(String option) {
if (option == null)
    return DEFAULT_COMPILE_POLICY;
else if (option.equals("attr"))
    return ATTR_ONLY;
else if (option.equals("check"))
    return CHECK_ONLY;
else if (option.equals("simple"))
    return SIMPLE;
else if (option.equals("byfile"))
    return BY_FILE;
else if (option.equals("bytodo"))
    return BY_TODO;
else
     return DEFAULT_COMPILE_POLICY;
}

There is also a lot of good assertions inside the compiler code that better be activated. So, I create a small java class playing with the options and activating Javac main directly.

And here is list of options I used:
For the JVM level options:

• -ea -esa This one is at the JVM level of the activation of the test code.
• -Xbootclasspath/p:dist/lib/javac.jar becareful you need jdk 1.7.0 to run it

For the javac options:

• -XDcompilePolicy=attr for attribute only pass
• -XDcompilePolicy=simple do the job in a very linear way, which makes the visitor on the code happens in a very different order than normal compile. I found it a very important check to verify that the code respect the visitor pattern and code of the compiler.
• -printsource is generating XXX.java files instead of XXX.class and do also a different visitor activation. Very important step to check the Parser/MemberEnter steps.
• -XDverboseCompilePolicy for showing what the compiler is doing. Very important.
• -verbose standard and sometimes annoying (I removed it after a while)
• -XDdev activate dev verbose (Don't know exactly what it means)
• -Xlint:unchecked verify when you messed up with generics ;-)
• -XDstdout very helpful to have compile error in standard out and compiler crashes and exception in error.

So here is my small test class that helped me a lot (Short cycle compile/test/debug).


The going down

This was really, really tough times for me. The issue is that java.lang.Enum does not have a default constructor but harcoded (String name, int ordinal) params. So Attr, TransTypes and even Lower, keep adding/removing this arguments types from the list of type parameters. The ''enum XXX'' and ''abstract enum YYY'' have the name and ordinal parameter added during the "synthetic sugar" phase of the Lower visitor. So, it does not match. I finally solved the issues the following ways:

1. Make some nice good test isDeclaredEnum() and isAbstractEnum() on the Symbol class
2. Allow enum to call super of an abstract enum
3. Found out that only the erasure_field method type has the 2 extra parameters, allowing a nice test (sugar already done or not)

When I finally manage to compile all the classes in one file "AE2.java", I went on the bad class file issue.

The final
I found out that a bad compiled class is a bad class. Logical no!

So, after solving all the issues above, my class was good. The last problem was the signature of abstract enum classes loaded from a XXX.class file. Here again the issue was with the erasure_field is null by default on class load, and in enum he is the only one to contain name and ordinal...

So, I just initialized it on ClassReader with the first well defined type.
Perfect, ''abstract enum'' at last...

Links
The patch and the tests are available here AbstractEnumOnKSL.tgz
Starting to work on the annotations now...

State Machine with abstract enum

Previous blogs

Writing this blog I found out that the subject of state machine with annotations was already covered in:

• Gabriele Carcassi blog
• Strachan's BeanFlow

I will talk about my experience and personal design, but mainly add the point on why "abstract enum" will help a lot.

state machine and code

First of all, since I wrote AADA I'm convince that UML helps you visualize and conceptualize your code and that's were it should stop. Trying to extract to annotate the UML meta-model so that some nice Java code will be generated, is a proof that the target language and framework are missing features.

For all static UML diagrams, it takes less energy to write a Java class in modern IDE than using any UML tool. But for dynamic diagram and especially '''State Diagrams''', UML beats Java (and big time if you use the good GOF pattern).

Most modern Java framework are solving this issue with XML files:

• In Spring Web Flow the flow definition file is an XML that points to fields and methods of your java code (like everything in Spring)
• All vendor workflow
• JBoss JBPM
• Struts, JSF, ...

So with JDK 5, I tried in 2005 to write an event driven state machine with annotations. In general I really prefer to design by events are they are a lot easier to grasp for everyone (business analysist, designers and developers).

So my design is basically based on the fact that each event is typed, and activate a specific object in the model. The '''state machine''' design question was: ''Which method should be activated?''

It is very similar to the previous blogs on the subject.

abstract enum will save all

One of the comment on Gabriele Carcassi blog was: "Actually, BeanFlow uses strings (due to depending on enum fields rather than interfaces) to declare transitions. That's not refactorable. I like this design better." So, the conflict between generic framework (using string) and clean design (using annotations and enum), appeared immediatly.

So, I tried to write with my design the Seminar State Diagram from Agile Modeling, and it looks really good:

public class Seminar {
private SeminarState state;

    private int maxStudents = 10;
private int nbStudents = 0;
private int nbWaitingStudents = 0;

@ActifFromState({null})
public void create() {
    state = SeminarState.proposed;
    }

@ActifFromState({/*all states*/})
public void cancel(CancelEvent e) {
    maxStudents = 10;
    nbStudents = 0;
    nbWaitingStudents = 0;
    state = null;
}

@ActifFromState({SeminarState.proposed})
public void schedule() {
    state = SeminarState.scheduled;
    }

@ActifFromState({SeminarState.scheduled})
public void open() {
    state = SeminarState.openForEnrollement;
    }

@ActifFromState({SeminarState.openForEnrollement})
public void add(NewStudentEvent e) {
    nbStudents++;
    if (nbStudents >= maxStudents) {
        state = SeminarState.full;
        }
}

@ActifFromState({SeminarState.full})
public void addWaiting(NewStudentEvent e) {
    nbWaitingStudents++;
}

@ActifFromState({SeminarState.openForEnrollement})
public void remove(DropStudentEvent e) {
    nbStudents--;
}

@ActifFromState({SeminarState.full})
public void removeWaiting(DropStudentEvent e) {
    nbWaitingStudents--;
    if (nbWaitingStudents == 0) {
        state = SeminarState.openForEnrollement;
        }
}

@ActifFromState({SeminarState.full, SeminarState.openForEnrollement})
public void close() {
    state = SeminarState.closedForEnrolement;
    }

public SeminarState getState() {
    return state;
    }
}

It looks like I can express the state in a more concise way than UML. Of course UML is still graphically more powerful.

The only issue above is that I need the enum SeminarState to extends abstract enum StateDefinition

public abstract enum StateDefinition {
 public Class getModelClass() {...};
 public StateDefinition getSuperState() {...};
 // And many more needed by the framework
}

public enum SeminarState implements StateDefinition {
 proposed, scheduled, openForEnrollement, full, closedForEnrolement;
}

and define in the State manager framework an annotation

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface ActifFromState {
  StateDefinition[] value();
}

Conclusion
It seems to me like most of the XML coding (writing java code in XML) issues, we are seeing today, can be removed if Annotations can have an abstract enum attribute.
I like it...