Make Groovy’s Grape.grab work when running compiled groovy code with java

5. February 2017Michael Lösch Leave a comment

Groovy’s Grape dependency management is pretty handy if you just want to write Groovy scripts and don’t want to use a full-blown dependency manager (gradle, maven). You just annotate @Grab to your Groovy script and it downloads the defined libs (powered by Ivy).

In some constellations, pain starts when the systemclassloader doesn’t know about the loaded classes/libs.

This can be fixed by setting @GrabConfig(systemClassLoader = true) (see e.g. mrhaki). But it’s getting really ugly when you don’t have a suitable ClassLoader (see groovy.grape.GrapeIvy <isValidTargetClassLoaderClass>) like GroovyClassLoader or RootLoader.

You’ll find this situation when running compiled groovy-code with java (e.g. when deploying your groovy code in a fatjar/shadowjar/uberjar). What you actually can do is to instantiate a GroovyClassLoader and call Grape.grab using this one. But you’ll end up in the same situation when you have to call @GrabConfig(systemClassLoader = true), like I said above. Running compiled groovy code with java means your SystemClassLoader is not a suitable one. You just can’t replace existing loaders by a GroovyClassLoader or inject it somehow somewhere.

I had this situation the other day in my open-source project Grip. I’m deploying with gradle and the shadowJar-plugin, so I run the compiled code with java and NOT groovy. Grip has an Interpreter to execute groovy scripts with some extras like scheduling, easy DB access and so on. And I want to provide grab in the scripts to allow users to import whatever they want.

Due to the fact that you can use Grape.grab with your own GroovyClassLoader, the basic mechanism works (dependencies are getting resolved). But what needs to be done is to transfer the information about the downloaded libs to the existing SystemClassLoader.

This is a hack, of course. But to me there’s no other way to deal with it (except to renounce Grape.grab):

After some research, I found a way to add libs to the SystemClassLoader. See ClassPathHacker.java: http://www.javafaq.nu/java-example-code-895.html

Now having ClassPathHacker, I was able to add the resolved dependencies

def groovyLoader = new GroovyClassLoader()

//grab dependencies using a GroovyClassLoader
Grape.grab(classLoader: groovyLoader, dependencies)

//using org.reflections.util.ClasspathHelper to get the URLs
def urls = ClasspathHelper.forClassLoader(groovyLoader).toArray() as URL[]

//and finally adding the URLs to the SystemClassLoader with ClassPathHacker
def cph = new ClassPathHacker()
urls.each { URL url ->
     if (url.file.endsWith('jar')) {
         cph.addFile(url.file)
     }
}

def groovyLoader = new GroovyClassLoader()

//grab dependencies using a GroovyClassLoader

Grape.grab(classLoader: groovyLoader, dependencies)

//using org.reflections.util.ClasspathHelper to get the URLs

def urls = ClasspathHelper.forClassLoader(groovyLoader).toArray() as URL[]

//and finally adding the URLs to the SystemClassLoader with ClassPathHacker

def cph = new ClassPathHacker()

urls.each { URL url ->

if (url.file.endsWith('jar')) {

cph.addFile(url.file)

}

That’s it. Libs loaded by Grape are now known by the SystemClassLoader.

When calling ClasspathHelper.forClassLoader, some errors appeard in the logfile: could not create Vfs.Dir from url (see https://github.com/ronmamo/reflections/issues/80). Those can be avoided by using this Helper https://gist.github.com/nonrational/287ed109bb0852f982e8

Find the whole implementation in package de.metacode.grip.core.classloaderhack in the project https://github.com/codeeraser/grip

Enhance your Groovy DSL in a plugin-ish way

20. August 2015Michael Lösch Leave a comment

The base of a groovy DSL is the Script class or the DelegatingScript. To provide some functions to your DSL, you define them in this class, implement the missingMethod or enhance the metaclass. No matter which way you choose, it blows up your class with rising amount of functions. So after some time you might want to restructure your code. For some other reason you might want to enable other people to enhance the DSL without touching the base DSL class.

The idea is to provide a class that can be instantiated in you DSL with a create-method.

1. Define a marker interface to find the plugin class (you can do this with annotations as well)

Every class implementing this marker interface will be instantiable in your DSL

interface Instantiable {}

1	interface Instantiable {}

2. A class that enhances your DSL

We’ll have a Car class that has only one field describing the color of our car

class Car implements Instantiable {
    def color = "blue"

    def printColor() {
        println color
    }
}

class Car implements Instantiable {

def color = "blue"

def printColor() {

println color

}

3. The actual DSL-backing class

Now we’ll go to the bone. As mentioned above, there are several ways to enhance the DSL class with new functions. In this sample we’ll add new functions by enhancing DSL’s metaClass.

We’ll have two functions to get an instance of Car: newCar and newCarWith.

newCar is calling the Default-Constructor without arguments. newCarWith calls the “Named” Default-Constructor with a map. To achieve this, there are two closures defined in line 3 and 4. First one with two params (actual class and the map) and one with a single param (only the class to instantiate).

Now we need to find the classes to instantiate. I used the reflections-API to find all classes implementing the Instantiable interface (line 6). In this sample only classes of the package de.metacode.dslplugin are scanned. You can scann the whole known classes as well.

Line 7 and 8 is where the magic happens: Two new methods are added to metaClass. Note the curry-call on the newInstance-closures. What this does is called currying (or Schönfinkeln). It reduces a function with n arguments to a function with n-1 arguments. In this case curry generates a new closure without the first parameter Class clazz. So we have one closure without any parameter and another one with a map.

class DSL {
    static {
        def newInstanceWithArgs = { Class clazz, Map args -> clazz.newInstance(args) }
        def newInstance = { Class clazz -> clazz.newInstance() }

        new Reflections("de.metacode.dslplugin").getSubTypesOf(Instantiable).each { Class clazz ->
            DSL.metaClass."new${clazz.simpleName}With" = newInstanceWithArgs.curry(clazz)
            DSL.metaClass."new${clazz.simpleName}" = newInstance.curry(clazz)
        }
    }
}

class DSL {

static {

def newInstanceWithArgs = { Class clazz, Map args -> clazz.newInstance(args) }

def newInstance = { Class clazz -> clazz.newInstance() }

new Reflections("de.metacode.dslplugin").getSubTypesOf(Instantiable).each { Class clazz ->

DSL.metaClass."new${clazz.simpleName}With" = newInstanceWithArgs.curry(clazz)

DSL.metaClass."new${clazz.simpleName}" = newInstance.curry(clazz)

}

4. Writing some DSL-code

Now we can instantiate the Car without params or with named params:

blueCar = newCar() //blue is the default value
blueCar.printColor()

redCar = newCarWith color: 'red'
redCar.printColor()

blueCar = newCar() //blue is the default value

blueCar.printColor()

redCar = newCarWith color: 'red'

redCar.printColor()

The output is

blue
red

blue

red

That’s it

Here you’ll find executable wired-together code:

https://gist.github.com/codeeraser/31d031cd921e168049ac

You’ll might get some issues with reflections-API when running this code via command-line. To get reflections work like one would expect can be pretty annoying (see stackoverflow…). The shortest way is to run the sample in an IDE.

Groovy DSL: How to split up execution of method calls like gradle does

3. May 2015Michael Lösch Leave a comment

This is how you can implement a gradle-ish behavior of executing a designated method before everything else in a dsl script (gradle executes doFirst{} first). Actually, I have no clue how gradle does it. This approach is about manipulating the AST with an implementation of CompilationCustomizer. I’ll describe two variations. First, how to execute the script twice, e.g. to set up an environment and execute the actual code sometimes later. And than how you can re-order the statements before execution. It’ll be helpful if you know about how AST transformations work (if you don’t, see the links at the end of this article).

Let’s say we have a DSL script like this:

println "this is text that will not be printed out in first line!"

doFirst { println "First things first: e.g. setting up environment" }

doStuff { println "doing some stuff now" }

println "That's it!"

println "this is text that will not be printed out in first line!"

doFirst { println "First things first: e.g. setting up environment" }

doStuff { println "doing some stuff now" }

println "That's it!"

First variation:

We’ll handle the DSL using DelegatingScript. The is the delegate for the first run (only doFirst will be executed):

class DoFirstProcessor {
    def doFirst(Closure c) {
        c()
    }
}

class DoFirstProcessor {

def doFirst(Closure c) {

c()

}

For the second run this is doing everything else:

class TheRestProcessor {
    def doStuff(Closure c) {
        c()
    }

    def methodMissing(String name, args) {
        //nothing to do
    }
}

class TheRestProcessor {

def doStuff(Closure c) {

c()

}

def methodMissing(String name, args) {

//nothing to do

}

At this point we’ve got everything we need to execute the DSL. Now comes the AST transformation part. The following CompilerCustomizer removes every statement from the AST except a designated method. Because there can only be one (method), this is called the HighlanderCustomizer 😉

class HighlanderCustomizer extends CompilationCustomizer {
    def methodName

    HighlanderCustomizer(def methodName) {
        super(CompilePhase.SEMANTIC_ANALYSIS)
        this.methodName = methodName
    }

    @Override
    void call(SourceUnit sourceUnit, GeneratorContext generatorContext, ClassNode classNode) throws CompilationFailedException {
        def methods = classNode.getMethods()
        methods.each { MethodNode m ->
            m.code.each { Statement st ->
                if (!(st instanceof BlockStatement)) {
                    return
                }
                def removeStmts = []
                st.statements.each { Statement bst ->
                    if (bst instanceof ExpressionStatement) {
                        def ex = bst.expression
                        if (ex instanceof MethodCallExpression) {
                            if (!ex.methodAsString.equals(methodName)) {
                                removeStmts << bst
                            }
                        } else {
                            removeStmts << bst
                        }
                    } else {
                        removeStmts << bst
                    }
                }
                st.statements.removeAll(removeStmts)
            }
        }
    }
}

class HighlanderCustomizer extends CompilationCustomizer {

def methodName

HighlanderCustomizer(def methodName) {

super(CompilePhase.SEMANTIC_ANALYSIS)

this.methodName = methodName

}

@Override

void call(SourceUnit sourceUnit, GeneratorContext generatorContext, ClassNode classNode) throws CompilationFailedException {

def methods = classNode.getMethods()

methods.each { MethodNode m ->

m.code.each { Statement st ->

if (!(st instanceof BlockStatement)) {

return

}

def removeStmts = []

st.statements.each { Statement bst ->

if (bst instanceof ExpressionStatement) {

def ex = bst.expression

if (ex instanceof MethodCallExpression) {

if (!ex.methodAsString.equals(methodName)) {

removeStmts << bst

}

} else {

removeStmts << bst

}

} else {

removeStmts << bst

}

st.statements.removeAll(removeStmts)

}

Bring it all together:

def cc = new CompilerConfiguration()
cc.addCompilationCustomizers new HighlanderCustomizer("doFirst")
cc.scriptBaseClass = DelegatingScript.class.name

def doFirstShell = new GroovyShell(new Binding(), cc)
def doFirstScript = doFirstShell.parse dsl
doFirstScript.setDelegate new AllStuffProcessor()
doFirstScript.run()

// Next run could be sometimes later. This will handle everything else than doFirst:

cc.compilationCustomizers.clear() // no customizer needed for second run
def shell = new GroovyShell(new Binding(), cc)
def script = shell.parse dsl
script.setDelegate new TheRestProcessor()
script.run()

def cc = new CompilerConfiguration()

cc.addCompilationCustomizers new HighlanderCustomizer("doFirst")

cc.scriptBaseClass = DelegatingScript.class.name

def doFirstShell = new GroovyShell(new Binding(), cc)

def doFirstScript = doFirstShell.parse dsl

doFirstScript.setDelegate new AllStuffProcessor()

doFirstScript.run()

// Next run could be sometimes later. This will handle everything else than doFirst:

cc.compilationCustomizers.clear() // no customizer needed for second run

def shell = new GroovyShell(new Binding(), cc)

def script = shell.parse dsl

script.setDelegate new TheRestProcessor()

script.run()

The console output should look like this:

First things first: e.g. setting up environment
this is text that will not be printed out in first line!
doing some stuff now
That's it!

First things first: e.g. setting up environment

this is text that will not be printed out in first line!

doing some stuff now

That's it!

Second variation:

Now the DSL will not be executed in two steps but rather in one single run. We’ll use a CompilationCustomizer again. This time the customizer finds a designated method call and changes the order of execution.

Our Delegate (this time only one is needed):

class AllStuffProcessor {
    def doFirst(Closure c) {
        c()
    }
    def doStuff(Closure c) {
        c()
    }

    def methodMissing(String name, args) {
        //nothing to do
    }
}

class AllStuffProcessor {

def doFirst(Closure c) {

c()

}

def doStuff(Closure c) {

c()

}

def methodMissing(String name, args) {

//nothing to do

}

The ReOrderCustomizer:

class ReOrderCustomizer extends CompilationCustomizer {
    def methodName

    ReOrderCustomizer(def methodName) {
        super(CompilePhase.SEMANTIC_ANALYSIS)
        this.methodName = methodName
    }

    @Override
    void call(SourceUnit sourceUnit, GeneratorContext generatorContext, ClassNode classNode) throws CompilationFailedException {
        def methods = classNode.getMethods()
        methods.each { MethodNode m ->
            m.code.each { Statement st ->
                if (!(st instanceof BlockStatement)) {
                    return
                }
                def doFirstStmt
                st.statements.each { Statement bst ->
                    if (bst instanceof ExpressionStatement) {
                        def ex = bst.expression
                        if (ex instanceof MethodCallExpression) {
                            if (ex.methodAsString.equals(methodName)) {
                                doFirstStmt = bst
                            }
                        }
                    }
                }
                if (doFirstStmt) {
                    st.statements.remove(doFirstStmt)
                    st.statements.add(0, doFirstStmt)
                }
            }
        }
    }
}

class ReOrderCustomizer extends CompilationCustomizer {

def methodName

ReOrderCustomizer(def methodName) {

super(CompilePhase.SEMANTIC_ANALYSIS)

this.methodName = methodName

}

@Override

void call(SourceUnit sourceUnit, GeneratorContext generatorContext, ClassNode classNode) throws CompilationFailedException {

def methods = classNode.getMethods()

methods.each { MethodNode m ->

m.code.each { Statement st ->

if (!(st instanceof BlockStatement)) {

return

}

def doFirstStmt

st.statements.each { Statement bst ->

if (bst instanceof ExpressionStatement) {

def ex = bst.expression

if (ex instanceof MethodCallExpression) {

if (ex.methodAsString.equals(methodName)) {

doFirstStmt = bst

}

if (doFirstStmt) {

st.statements.remove(doFirstStmt)

st.statements.add(0, doFirstStmt)

}

Bring it all together:

def cc = new CompilerConfiguration()
cc.addCompilationCustomizers new ReOrderCustomizer("doFirst")
cc.scriptBaseClass = DelegatingScript.class.name

def shell = new GroovyShell(new Binding(), cc)
def script = shell.parse dsl
script.setDelegate new AllStuffProcessor()
script.run()

def cc = new CompilerConfiguration()

cc.addCompilationCustomizers new ReOrderCustomizer("doFirst")

cc.scriptBaseClass = DelegatingScript.class.name

def shell = new GroovyShell(new Binding(), cc)

def script = shell.parse dsl

script.setDelegate new AllStuffProcessor()

script.run()

Additionally, one should check that the designated method appears only once in the DSL script.

To understand how the AST looks like, just debug the customizer’s call method.

This article is based on a question I postet on Stackoverflow: http://stackoverflow.com/questions/29967886/groovy-dsl-how-can-i-let-two-delegating-classes-handle-different-parts-of-a-dsl

Links:

Great example of using Groovy to implement a DSL: https://groovy.codeplex.com/wikipage?title=Guillaume%20Laforge’s%20%22Mars%20Rover%22%20tutorial%20on%20Groovy%20DSL’s

About CompilationCustomizer: http://www.jroller.com/melix/entry/customizing_groovy_compilation_process

AST transformation workshop: http://melix.github.io/ast-workshop/

Joe’s series of AST articles: http://joesgroovyblog.blogspot.de

Strange behavior of Firefox when it comes to range requests of JavaScript files

6. January 2015Michael Lösch Leave a comment

At work we have a GWT application which contains a large ( > 10mb) javascript file. In some cases that file is not executed by the browser. It seems like the application is frozen. A colleague and I observed that pretty strange behavior while using Firefox (didn’t recognize this in IE, Crome, Safari,…):

The js file is cached by the browser usually. In some cases the file is not cached completely. The strange thing happens when we want to reload the application. Due to the incomplete caching of that file, Firefox requests the missing part of the file (range request). But that ends up with the frozen-like behavior.

So we tried to isolate the problem especially without using GWT. In the end we found a way to reproduce the problem:

1. The HTML/Javascript file

We created a HTML file that loads large js-Files and reports their state of execution. The javascript method add will be called by the large js-Files (see later on).

ct.html:

<!DOCTYPE html>
<html>
<head>
    <script type="text/javascript">
      var count = 11;
      function initTable() {
        var p = document.getElementById('table');
        var t = document.createElement('table');
        t.setAttribute('border','1');
        for (i = 0; i <= count; i++) {
          var tr = document.createElement('tr');
          tr.setAttribute('id', 'tr-' + i);
          var td = document.createElement('td');
          td.appendChild(document.createTextNode(i));
          tr.appendChild(td);
          t.appendChild(tr);
        }
        p.appendChild(t);
      }
      function add (idx, text) {
        var td = document.createElement("td");
        td.appendChild(document.createTextNode(text));
        document.getElementById('tr-' + idx).appendChild(td);
      }
      function log (text) {
        var e = document.createElement("div");
        e.innerHTML = text;
        document.getElementById('log').appendChild(e);
      }
    </script>
</head>
<body>
  <div id="table"></div>
  <div id="log"></div>
  <script type="text/javascript">
    initTable();
    log(new Date());
    for (i = 0; i <= count; i++) {
      add(i, 'add script');
      var s = document.createElement('script');
      s.setAttribute('src', 'big' + i + '.cache.js');
      s.setAttribute('type', 'text/javascript');
      document.getElementsByTagName('head')[0].appendChild(s);
    }
  </script>
</body>
</html>

<!DOCTYPE html>

<html>

<head>

var count = 11;

function initTable() {

var p = document.getElementById('table');

var t = document.createElement('table');

t.setAttribute('border','1');

for (i = 0; i <= count; i++) {

var tr = document.createElement('tr');

tr.setAttribute('id', 'tr-' + i);

var td = document.createElement('td');

td.appendChild(document.createTextNode(i));

tr.appendChild(td);

t.appendChild(tr);

}

p.appendChild(t);

}

function add (idx, text) {

var td = document.createElement("td");

td.appendChild(document.createTextNode(text));

document.getElementById('tr-' + idx).appendChild(td);

}

function log (text) {

var e = document.createElement("div");

e.innerHTML = text;

document.getElementById('log').appendChild(e);

}

</script>

</head>

<body>

initTable();

log(new Date());

for (i = 0; i <= count; i++) {

add(i, 'add script');

var s = document.createElement('script');

s.setAttribute('src', 'big' + i + '.cache.js');

s.setAttribute('type', 'text/javascript');

document.getElementsByTagName('head')[0].appendChild(s);

}

</script>

</body>

</html>

2. Generating large js files

We wrote a Groovy script that generates files with three lines (named big${i}.cache.js)

The first line contains add calling the add method with the argument “start”.
The second line contains ~30mb spaces
The third line again contains code calling the add method with the argument “end”.

def size = 31450000
for (i in 0..12) {
    def f = new File("/Users/user/apache/htdocs/big${i}.cache.js")
    f.delete()
    f << "add($i, 'start');\n" << " ".multiply(size) << "\nadd($i, 'end');"
}

def size = 31450000

for (i in 0..12) {

def f = new File("/Users/user/apache/htdocs/big${i}.cache.js")

f.delete()

f << "add($i, 'start');\n" << " ".multiply(size) << "\nadd($i, 'end');"

}

3. Bring it together

Copy ct.html and the big.cache.js files into your webservers htdocs.

Clear your browser cache

Navigate to ct.html and wait till the table looks like this (all files are loaded and executed):

0	add script	start	end
1	add script	start	end
2	add script	start	end
3	add script	start	end
4	add script	start	end
5	add script	start	end
6	add script	start	end
7	add script	start	end
8	add script	start	end
9	add script	start	end
10	add script	start	end
11	add script	start	end

Then reload the page.

What you should see is that some scripts are executed (you see start/end columns) and some are not. You can see the not executed ones in Firebugs network sheet. Those are the ones with status “206 Partial Content”:

We reported that problem to Mozilla but got no response yet ( https://bugzilla.mozilla.org/show_bug.cgi?id=1117763 ).

But if you got the solution already, feel free to post it in the comment section 😎

My environment:

Firefox 34.0.5 on a Mac running OsX 10.10.1 (Yosemite)

Read CGLog with Groovy

11. December 2014Michael Lösch Leave a comment

If an app uses Jemalloc to manage space, there’s a security gap if you run Yosemity. Famouse apps using Jemalloc are e.g. Firefox and Thunderbird. So if you want to know you’re affected or not, just take a look at the /tmp dir and watch out for files like CGLog_Thunderbird* or CGLog_Firefox*.

I’m affacted and I wanted to know what mail content was logged. So here’s an example how sweet and easy groovy can handle jobs like that, including some code that cleans up the un-readable chars.

def f = new File("/Users/username/tmp/CGLog.txt")
new File("/tmp/CGLog_Thunderbird_123456").readLines().each { String line ->
    line.eachMatch(~/KeyDown.*char ([0-9]*);/) {
        f << String.valueOf((char) (it[1] as Integer)).replaceAll("[\uF700-\uF72F]", "").replaceAll('\r','\n')
    }
}

def f = new File("/Users/username/tmp/CGLog.txt")

new File("/tmp/CGLog_Thunderbird_123456").readLines().each { String line ->

line.eachMatch(~/KeyDown.*char ([0-9]*);/) {

f << String.valueOf((char) (it[1] as Integer)).replaceAll("[\uF700-\uF72F]", "").replaceAll('\r','\n')

}

Hackserei

It's about software

Widgets

Search

Groovy

Make Groovy’s Grape.grab work when running compiled groovy code with java

Enhance your Groovy DSL in a plugin-ish way

1. Define a marker interface to find the plugin class (you can do this with annotations as well)

2. A class that enhances your DSL

3. The actual DSL-backing class

4. Writing some DSL-code

That’s it

Groovy DSL: How to split up execution of method calls like gradle does

First variation:

Second variation:

Strange behavior of Firefox when it comes to range requests of JavaScript files

1. The HTML/Javascript file

2. Generating large js files

3. Bring it together

Read CGLog with Groovy