Project Control Center

Purpose of the project

Project Control Center

In 2010 I had the idea of creating a to-do list, which would automatically tell the user, when he or she should work on a particular task.

I thought that such a planner would be very convenient and in order to test this hypothesis (by actually using such a planner), I created the system described below.

How it works

From the user point of view the system would work like this:

  1. In a todo list on a mobile device, you enter a list of tasks and projects with priority, effort estimates and dependencies.
  2. In the calendar app on your mobile device you see the times, when you should be working on the different tasks from the to-do list. This scheduling takes into account priorities, effort estimates (entered in the to-do list) and dependencies (when task X must be completed before task Y starts).

Use cases

Use cases

Use cases

Structure of the system

Project Control Center Overview

The system consists of the following parts:

  1. Web front-end created using the Vaadin framework (source code is available here)
  2. Several queues, which run inside an ActiveMQ message broker
  3. Worker application (source code is available here)
  4. Scheduler application (source code is available here)
  5. JavaDB (formerly Derby) database
  6. TaskJuggler III project scheduler

Now let’s see how these part work together in the use cases specified above.

Use case 1: Request invitation

In order to get access to the system, the user had to submit a request for invitation via the web application.

Use case 6: Accept/reject invitation request

In regular intevals the admin was supposed to look into a part of the web application, which showed received invitation requests. The admin could grant or reject user’s request. If the request was granted, he would manually send an e-mail to the user with the password.

Use cases 2 and 3: Grant access to Google Tasks and Google Calendar

The user then would log in into the web application and grant access to his or her

  • Google Tasks and
  • Google Calendar

accounts.

If these operations were successful, the user could now use the system.

Use case 4: Request immediate plan re-calculation

If the user pressed the respective button in the web appliation, following things happened:

  1. The web application sent a message to the queue PCC.WEB.WORKER.
  2. The worker application received that message and extracted the user ID from it.
  3. Then, it read to-do list of the user from Google Tasks and converted it into a TaskJuggler III project file.
  4. Then, it invoked the TaskJuggler III scheduler from the command line, which generated several files with data on when the user should be working on what task.
  5. These data were read and exported to user’s Google Calendar such that he or she could see, when a certain task was supposed to be worked on.

Use case 5: Enable/disable background plan re-calculation

In the web application the user could enable background plan re-calculation. In this case, following things would happen:

  1. Every five minutes, the scheduler application would send a message to the PCC.SCHEDULER.WORKER queue.
  2. Thereafter, the worker application would do steps 2-5 from previous section.

In this way, user’s plan in Google Calendar would be updated every 5 minutes and be virtually always up to date.

Design principles

I designed the Project Control Center application in a way, which should allow

  1. other developers to easily extend and modify the code base,
  2. to test all building blocks individually (unit tests) and
  3. in co-operation with each other (integration tests).

In order to attain these goals, the most of the code of the Project Control Center is designed according to certain principles.

  1. The code base is divided into so-called functional blocks.
  2. Every functional block consists of 2 packages – api and impl.
  3. api is the descriptive part of a functional block.
  4. impl is the implementation part of a functional block.
  5. The api package may only contain
    1. interfaces,
    2. abstract classes and
    3. enums.
  6. All source code elements of an api package must be public.
  7. In the api package, there must be at least one factory interface.
  8. In the impl package there are classes that implement interfaces of the functional block.
  9. All source code elemens of the impl package are package private, except the factory implementation.
  10. If functional block A wants to access the code from functional block B, it does so by passing B’s interface to a dependency injection mechanism (based on Google Guice).
  11. The name of a class is equal to the name of the interface it implements with the prefix Default. If an interface is called SomeFunctionalBlock, then its implementation is called DefaultSomeFunctionalBlock.

Example

I will illustrate these principles on the functional block tj3bookingsparser. It is used to convert certain text files into a form, which can be processed in Java.

The descriptive part is located in the at.silverstrike.pcc.api.tj3bookingsparser package, its implementation – in at.silverstrike.pcc.impl.tj3bookingsparser.

Let’s look at the central interface of this functional block:

public interface Tj3BookingsParser
  extends ModuleWithInjectableDependencies, SingleActivityModule {
    void setInputStream(final InputStream aInputStream);
    void run() throws PccException;
    List getBookings();
}

Just by looking at the interface we can notice several relevant things:

  • It extends the SingleActivityModule interface, which means that this functional block takes several values as inputs (using set methods like setInputStream) and transforms them in the run method into output values (which can be retrieved using get methods like getBookings).
  • It extends the ModuleWithInjectableDependencies interface, which means that this functional block uses other functional blocks and accesses them via Google Guice-based dependency injection mechanism.

In the same package you can find a factory, which creates instances of the bookings parser class.

public interface Tj3BookingsParserFactory
  extends Factory {
}

The interface Factory is necessary in order for all factories in the project to have same structure.

Now let’s look at the implementation of the functional block, DefaultTj3BookingsParser.

class DefaultTj3BookingsParser implements Tj3BookingsParser {
  private List bookings;
  private InputStream inputStream;
  private Injector injector = null;

  public DefaultTj3BookingsParser() {
  }
  public List getBookings() {
    return bookings;
  }
  public void setInputStream(final InputStream aInputStream) {
    this.inputStream = aInputStream;
  }
  @Override
  public void run() throws PccException {
    try {
      final BookingsFile bookingsFile =
        inputStreamToBookingsFile();
      this.bookings = bookingsFileToBookings(bookingsFile);
    } catch (final IOException exception) {
      throw new PccException(exception);
    } catch (final RecognitionException exception) {
      throw new PccException(exception);
    } catch (final NumberFormatException exception) {
      throw new PccException(exception);
    } catch (final ParseException exception) {
      throw new PccException(exception);
    }
 }
  private List bookingsFileToBookings(
     final BookingsFile aBookingsFile) throws ParseException {
     final BookingsFile2BookingsFactory factory = this.injector
       .getInstance(BookingsFile2BookingsFactory.class);
     final BookingsFile2Bookings converter = factory.create();
     converter.setBookingsFile(aBookingsFile);
     converter.setPersistence(
       this.injector.getInstance(Persistence.class));
     converter.run();
     return converter.getTuples();
  }
  private BookingsFile inputStreamToBookingsFile()
    throws IOException, RecognitionException {
    final BookingsLexer lexer = new BookingsLexer(
      new ANTLRInputStream(this.inputStream));
    final CommonTokenStream tokenStream = new CommonTokenStream(
      lexer);
    final BookingsParser parser = new BookingsParser(tokenStream);
    parser.bookingsFile();

    final BookingsFile returnValue = parser.getBookingsFile();
    return returnValue;
  }
  @Override
  public void setInjector(final Injector aInjector) {
    this.injector = aInjector;
  }
}

The logic of the functional block is isolated inside the run, inputStreamToBookingsFile and bookingsFileToBookings methods. There, a text coming from a stream is converted into graph of Java objects. I’ll explain, how it works in one of the following sections.

For now, let’s look at how a functional block can use the functionality of other functional blocks.

First, a reference to a dependency injection mechanism is passed to the functional block in the setInjector method.

@Override
  public void setInjector(final Injector aInjector) {
    this.injector = aInjector;
  }

Then, in the bookingsFileToBookings method, we first retrieve the implementation of a factory object by passing to the injector its interface (BookingsFile2BookingsFactory).

final BookingsFile2BookingsFactory factory = this.injector
 .getInstance(BookingsFile2BookingsFactory.class);

Then, we invoke the create method of the factory and get a reference to the object (converter) we want to use.

final BookingsFile2Bookings converter = factory.create();

Benefits

This design has several advantages.

  • Maintainability: The system is a collection of functional blocks with unified design – if you’ve learned how one functional block interacts with the other, you know how any other does this. I built a complex testing framework for a customer using these principles. In later stages of the project new developers joined the team and could extend and modify the framework on their own after a short introduction. Thus, a simple to understand design (along with a team wiki) helped increase the number of people in a team, who could maintain the code (increased the bus factor).
  • Testability: Each functional block knows only the interfaces (not the details of implementation) of other functional blocks. This means that in order to mock the dependencies of a certain functional block it is sufficient to provide a different dependency injection cotnainer. For example, you can look at the testDefect201109301 test. Here, a dependency injection container is used, which binds the Persistence interface (database access object) to a mock persistence object (not a real database). The same applies to all other functional blocks.

ANTLR parser for TaskJuggler III output files

The TaskJuggler scheduler generates so-called bookings files with information about when certain tasks should be worked on. These files have a rather complex format (see a fragment of such a file below).

task T1 "Doll" {
 start 2011-09-05-14:00-+0000
 end 2011-09-05-16:00-+0000
 scheduling asap
 scheduled
}
[...]
supplement task T1 {
 booking R62 2011-09-05-14:00-+0000 + 2.0h { overtime 2 }
 priority 997
 projectid prj
}
supplement task T2 {
 booking R62 2011-09-05-11:00-+0000 + 3.0h { overtime 2 }
 priority 998
 projectid prj
}

In order to extract information from these files, I built a parser based on the ANTLR framework. First, I defined a grammar – a file, which describes

  1. the structure of the text I want to process and
  2. what to do when a certain element is detected.

Let’s look at a fragment of the grammar:

grammar Bookings;
[...]
@members
{
 private DefaultBookingsFile bookingsFile;
public DefaultBookingsFile getBookingsFile()
 {
 return this.bookingsFile;
 }
}

In this piece of code we declare that the resulting Java class should have a member variable bookingsFile (the root of the object graph with data from the text file) and have a getBookingsFile accessor method.

@header {
package at.silverstrike.pcc.impl.tj3bookingsparser.grammar;
import org.slf4j.Logger;
}
@lexer::header {
package at.silverstrike.pcc.impl.tj3bookingsparser.grammar;
}
bookingsFile
 :
 {
 this.bookingsFile = new DefaultBookingsFile();
 }
 header
 projectIds
 resourceDeclaration
 task+
 (
 suppTask=supplementTask
 {
 if ($suppTask.suppStmt.getTaskId().startsWith("T"))
 {
 this.bookingsFile.addSupplementStatement( $suppTask.suppStmt );
 }
 }
 )*
 supplementResource*
 EOF
 ;

Here we declare bookingsFile to be the root of the grammar tree, which consists of

  1. exactly one header element,
  2. exactly one projectIds element,
  3. exactly one resourceDeclarations element,
  4. one or more task elements and
  5. any (zero to infinity) number of supplementResource elements.

We also specify that whenever we encounter a bookingsFile structure, an instance of DefaultBookingsFile should be created. Whenever we encounter a supplementTask structure, its data should be added to the bookingsFile object using addSupplementStatement method. Full grammar is available here.

The grammar is fed into a code generator, which produces a Java class that transforms a text to an object graph representing the data in that text file. The batch script for generating the Java class is called generateParser.bat.

The result is the class BookingsParser, which is used in the DefaultTj3BookingsParser functional block.

private BookingsFile inputStreamToBookingsFile()
  throws IOException, RecognitionException {
 final BookingsLexer lexer = new BookingsLexer(
    new ANTLRInputStream(this.inputStream));
 final CommonTokenStream tokenStream =
    new CommonTokenStream(lexer);
 final BookingsParser parser = new BookingsParser(tokenStream);
 parser.bookingsFile();
 final BookingsFile returnValue = parser.getBookingsFile();
 return returnValue;
 }

Communication with OAuth

Project control center used

  1. Google Tasks as a data input mechanism and
  2. Google Calendar as a data output mechanism.

In order to read data from Google Tasks and write them to Google Calendar several routines were necessary:

  1. Grant access to Google Tasks and Google Calendar from the web application.
  2. Read data from Google Tasks.
  3. Write data to Google Calendar.

The authentication was implemented using 2 different variants of the OAuth protocol (one for Google Tasks, the other for Google Calendar).

Let’s look at how this is implemented.

Granting access to Google Tasks and Google Calendar

In the web application, there is a form with 2 buttons for granting access Project Control Center access to Google Tasks and Google Calendar. When these buttons are pressed, DefaultUserSettingsPanelController‘s methods initiateGoogleCalendarAuthorization or initiateGoogleTasksAuthorization are called.

class DefaultUserSettingsPanelController
  implements UserSettingsPanelController {
  [...]
  @Override
  public void initiateGoogleCalendarAuthorization() {
  try {
    oauthParameters = new GoogleOAuthParameters();
    oauthParameters.setOAuthConsumerKey("pcchq.com");
    oauthParameters.setScope(SCOPE_CALENDAR);
    oauthParameters.setOAuthCallback(oauthRedirectUri);
    privKey = getPrivateKey();
    oauthHelper =
      new GoogleOAuthHelper(new OAuthRsaSha1Signer(privKey));
    oauthHelper.getUnauthorizedRequestToken(oauthParameters);
    TPTApplication
     .getCurrentApplication()
     .getMainWindow()
     .open(new ExternalResource(oauthHelper
       .createUserAuthorizationUrl(oauthParameters)),
          "_top");
   } catch (final OAuthException exception) {
     LOGGER.error("", exception);
   }
 }
  @Override
  public void initiateGoogleTasksAuthorization() {
  String clientId = CLIENT_ID;
  String scope = SCOPE_TASKS;

  String authorizationUrl =
  new GoogleAuthorizationRequestUrl(clientId, oauthRedirectUri,
    scope).build();
  TPTApplication.getCurrentApplication().getMainWindow()
    .open(new ExternalResource(authorizationUrl), "_top");
  }

Reading from Google Tasks and Google Calendar

The code for reading data from Google is distributed amont several projects.

protected final List<SchedulingObject> importDataFromGoogleTasks(
  final UserData aUserData) {
  final List<SchedulingObject> importedTasks =
    getTasksFromGoogleTasks(aUserData);
  final GoogleCalendarEventImporterFactory factory =
  this.injector.getInstance(
    GoogleCalendarEventImporterFactory.class);
  final GoogleCalendarEventImporter eventImporter =
    factory.create();
  eventImporter.setCalendarScope(this.calendarScope);
  eventImporter.setConsumerKey(this.consumerKey);
  eventImporter.setUser(aUserData);
  eventImporter.setInjector(this.injector);
  try
  {
    eventImporter.run();
  } catch (final PccException exception) {
    LOGGER.error("", exception);
  }
  eventsToDelete = eventImporter.getEventsToDelete();
  eventsToImport = eventImporter.getEventsToImport();
  final List<SchedulingObject> importedEvents =
    convertCalendarEventEntriesToPccEvents(eventsToImport);
  final List<SchedulingObject> returnValue =
    new LinkedList<SchedulingObject>();
  if (importedTasks != null) {
    returnValue.addAll(importedTasks);
  }
  if (importedEvents != null) {
    returnValue.addAll(importedEvents);
  }
  return returnValue;
 }
  1. When the worker receives a recalcalculation request from the queue, it uses the AbstractSchedulingRequestMessageProcessor.importDataFromGoogleTasks method to import the tasks.
  2. For reading Google Tasks data, the DefaultGoogleTasksImporter functional block is used.
  3. In order to be able to calculate the plan, we need to read data not only from Google Tasks, but also from Google Calendar. Google Calendar contains two kinds of tasks, which we need to handle during plan calculation:
    1. Manually scheduled events, which we can’t use for work on tasks.
    2. Task working times, which were inserted by the Project Control Center.
  4. eventImporter in the code snipped above reads both kinds of data. eventsToImport are manually scheduled events, eventsToDelete are task working times.

Writing to Google Calendar

When the plan has been calculated, the working times for each task are exported to Google Calendar. This happens in the AbstractSchedulingRequestMessageProcessor.exportDataToGoogleCalendar method.

protected final void exportDataToGoogleCalendar(
  final UserData aUser, final List<Booking> aBookings) {
    deleteEvents();

    final Exporter2GoogleCalendarFactory factory =
       this.injector.getInstance(
         Exporter2GoogleCalendarFactory.class);
    final Exporter2GoogleCalendar exporter = factory.create();
    exporter.setCalendarScope(this.calendarScope);
    exporter.setConsumerKey(this.consumerKey);
    exporter.setUser(aUser);
    exporter.setBookings(aBookings);
    exporter.setInjector(this.injector);
    exporter.setUser(aUser);
    try {
      exporter.run();
    }
    catch (final PccException exception) {
      LOGGER.error("", exception);
    }
}

Mixing Java and Ruby

In scope of this project I tried to write some functional blocks in Ruby. Ruby (and other functional languages) allows to write more concise source code, which can save time.

But in order to use these languages in serious projects, it is necessary to integrate them into a system in a way, which makes it easy for other functional blocks to interact with them.

I managed to implement this technically in scope of the tj3deadlinesparser functional block. Its interface looks like the interface of any other functional block. But the implementation (class DefaultTj3DeadlinesFileParserFactory, file DefaultTj3DeadlinesFileParser.rb) are different. The interface is written in Java, but its implementation – in JRuby.

Most interesting part of this Java-JRuby integration is the AbstractRubyImplementationFactory class.

public abstract class AbstractRubyImplementationFactory<X>
  implements Factory<X>, ModuleWithInjectableDependencies {
  private static final String RUBY_CONSTRUCTOR = ".new";
  private static final String RUBY_EXTENSION = ".rb";
  private static final String IMPLEMENTATION_PREFIX = "Default";
  private static final String RUBY_DIRECTORY = "rb/";
  private static final Logger LOGGER = LoggerFactory
    .getLogger(AbstractRubyImplementationFactory.class);
  private Injector injector;
  @SuppressWarnings("unchecked")
  private X createRubyImplementation(final String aModule,
    final String aInterfacename) {
    final ScriptEngine jruby = new ScriptEngineManager()
      .getEngineByName("jruby");
    X retVal = null;
    try {
      final EmbeddedFileReader reader = this.injector
       .getInstance(EmbeddedFileReader.class);
      reader.setFilename(RUBY_DIRECTORY + aInterfacename
        + "/" + IMPLEMENTATION_PREFIX + aModule
        + RUBY_EXTENSION);
      reader.run();
      jruby.eval(reader.getFileContents());
      retVal = (X) jruby.eval(IMPLEMENTATION_PREFIX + aModule
        + RUBY_CONSTRUCTOR);
      return retVal;
    } catch (final ScriptException exception) {
      LOGGER.error("", exception);
    } catch (final PccException exception) {
      LOGGER.error("", exception);
    }
    return retVal;
  }
  @Inject
  public final void setInjector(final Injector aInjector) {
    this.injector = aInjector;
  }
  @Override
  public final X create() {
    return createRubyImplementation(getModuleName(),
      getInterfacename());
  }
  protected abstract String getInterfacename();
  protected abstract String getModuleName();
}

The result of my experiments with JRuby and Java – I was faster writing plain old Java code than Java-Ruby mix. Therefore there is only one functional block written in Ruby in the Project Control Center source code.

Measuring code quality

Test coverage report

Test coverage report

In scope of this project I used several code quality measurements in order to

  1. find and fix typical coding errors and
  2. maintain a good-enough level of unit test coverage.

Code quality measurements were done by Maven (mvn site). The results were exported as HTML reports:

  1. The Checkstyle report listed violations of my coding standard.
  2. The Cobertura test coverage report showed the percentage of code covered by automatic tests.
  3. The CPD report showed code duplicates.
  4. The FindBugs and PMD reports list potential defects.

A sample code quality report for one of the Project Control Center modules can be downloaded here.

Source code

  1. pcc
  2. pcc-cdm
  3. pcc-logic
  4. pcc-scheduler
  5. pcc-worker

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s