This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Contribution Guidelines

How to develop new features and coding standards.

1 - Setup & Standards

Coding standards for Lucille and how to set up for local development.

Coding Standards

Local Developer Setup

Prerequisite(s):

  • IntelliJ application installed on machine
  • Java project

Setting up Google Code Formatting Scheme

  • Make sure that Intellij is open
  • Go to the following link: styleguide/intellij-java-google-style.xml at gh-pages · google/styleguide
  • Download the .xml file
  • Open the file in an editor of your choice
  • Navigate to the <option …> tag with name ‘Right Margin’ and edit the value to be 132 (it should default as 100)
  • Save the file
  • In Intellij IDEA, navigate to Settings | Preferences → Code Style → Editor → Java
  • Click on the gear icon on the right panel and drill down to the option Import Scheme and then to Intellij IDEA Code Style XML
  • In the file explorer that opens, navigate to where you stored the aforementioned .xml file we downloaded
  • After selecting the file, you should see a pop-up allowing you to name the scheme; select a name and click ‘Okay’
  • Click ‘Apply’ in the Settings panel
  • Restart the IDE; You can use the ‘Reformat Code’ option to apply the plug-in on your code

Excluding Non-Java Files

Assuming that we don’t want to auto-format non-java files via a directory level ‘Reformat Code’ option, we need to exclude all other files from being reformatted

  • Navigate to Settings | Preferences in Intellij IDEA

  • Navigate to Editor → Code Style

  • Click on the tab on the right window labeled ‘Formatter’

  • In the ‘Do Not Format’ text box, paste the following and click ‘Apply'

    *.{yml,xml,md,json,yaml,jsonl,sql}

  • A restart of Intellij may be required to see changes

This method may prove to be too complicated, especially when new file types are added to the codebase, therefore, consider the following, simpler method instead:

  • When clicking on ‘Reformat Code’ at the directory level, a window will pop up
  • Under the filter sections in the window, select the ‘File Mask(s)’ option and set the value to ‘*.java’
  • This will INCLUDE all .java files in your reformatting

Eclipse Users

Eclipse import conf .xml files

The linked post details some useful information for how Eclipse users can use the same .xml for their code formatting on Eclipse IDE.

2 - Developing New Connectors

The basics of how to develop connectors for Lucille.

Base Classes

Unit Tests

3 - Developing New Stages

The basics of how to develop stages for Lucille.

Introduction

This guide covers the basics of how to develop a new stage for Lucille and what some of its required and optional components are. After reading this, you should be able to start development with a good foundation on how testing works, how configuration is handled, and what features the Stage class affords us.

Prerequisite(s)

  • An up-to-date version of Lucille that has been appropriately installed

  • Please refer to Quick Start Guides - Lucille Local Mode for more info on how to set up Lucille

  • Understanding of Java programming language

  • Understanding of Lucille

Developing a Stage:

Note: Angle brackets (<…>) are used to show placeholder information that is meant to be replaced by you when using the code snippets.

  1. Create a new Java class underneath the following directory in Lucille; name should be in PascalCase
    • lucille/lucille-core/src/main/java/com/kmw/lucille/stage/
  2. The new class should extend the Stage abstract class that can be found at the following directory in Lucille
    • lucille/lucille-core/src/main/java/com/kmw/lucille/core/
    • The class declaration should look like the following public class <StageName> extends Stage {
  3. Create a constructor, your constructor should take in a config variable of type Config
    • The constructor declaration should look similar to this public <StageName>(Config config) {
  4. Next, call super() to reference the protected super constructor from the Stage class
    • We want to provide this constructor with the aforementioned config, but also with the names of any required or optional parameters / parents that we want to make configurable via the config
    • Note that the properties should be in camelCase
    • Example provided below:
super(config, new StageSpec()
  .withRequiredProperties("<requiredProperty>")
  .withOptionalProperties("<optionalProperty>")
  .withRequiredParents("<requiredParent>"));
  1. Define instance variables that correlate to config properties you wish to request from the user (both required and optional)
    • The following code shows examples of common patterns used to extract config parameters; reference the Config code for more methods

config.getConfig("<nameOfProperty>").root().unwrapped(); // for required properties

config.hasPath("<nameOfProperty>") ? config.getInt("<nameOfProperty>") : <defaultValue>;

ConfigUtils.getOrDefault(config, "<nameOfProperty>", <defaultValue>;

  1. Add the abstract method processDocument to your class

    • This method is where we want to make changes to fields on the document and potentially create child documents
    • This method should return null assuming we are not intending to generate child documents
      • Reference the Javadoc in the Stage class for more information on how to support this functionality

  2. Add appropriate comments to explain any important code in the class and add Javadoc before the class declaration

    • Javadoc should explain the behaviour of this Stage and also should list config parameters, their types, whether they are optional, and a short description

Unit Testing:

Lucille uses JUnit as its testing framework, please refer to JUnit best practices when making tests.

  1. Create a new Java class underneath the following directory in Lucille; name should be the same as the Stage’s name, with Test appended to the end

    • lucille/lucille-core/src/test/java/com/kmw/lucille/stage/

  2. Create a new directory underneath the following directory in Lucille: name should same as the testing class’ name

    • lucille/lucille-core/src/test/resources/

    • Underneath this directory create a new file called config.conf; this will be an example config that will be used in our test class; you can create more for further testing

  3. The following code snippet can be used to create a new Stage with the provided config name

StageFactory.of(<StageName>.class).get("<StageTestName>/config.conf");

  1. The following code snippet will process a given Document; reference the Document class for more information s.processDocument(d); // where s is the Stage and d is the Document

The following are standards for testing in Lucille:

  • There should be at least one unit test for a Stage

  • Recommended to aim for 80% code coverage, but not a hard cutoff

  • When testing services, use mocks and spys

  • Tests should test notable exceptions thrown by Stage class code

  • Tests should cover all logical aspects of a Stage’s function

  • Tests should be deterministic

  • Code coverage should not only encapsulate aspects of lucille-core but also modules

Extra Stage Resources:

  • The Stage class has both the start and stop method; both are helpful for when we want to set up or tear down objects

  • Lucille also has a StageUtils class that has some methods that may prove useful in development

4 - Developing New Indexers

The basics of how to develop indexers for Lucille.

Base Classes

Unit Tests