Feeding ElasticSearch with Java Generics and Java Reflection

If you’ve read my blog before, you know how I feel about ElasticSearch (LOVE).

When you have to load as much as data as I have to, things become a bit tedious with several sources of:

A database
A web service
A parsed file
Etc..

You are probably storing the data into object. If so, you would be forced to iterate each object based off its specific fields. Every object will have different fields:

HubspotContact:

Firstname
Lastname
Email

And your Registration object will have:

badgeId
registration_date
Sessions

So, for each object type, you would have to iterate those specific fields, either in separate classes or separate methods.

for(Iterator<HubspotContact> it = registrants.iterator();it.hasNext();) {

HubspotContact registrant = it.next();

get fields…….(getEmail(), getFirstname(), etc….)

}

So, what’s the solution?

Java Generics + Java Reflection = Easy ElasticSearch

Let’s start with your method signature:

public static <T> void insertData(List<T> items, String indexName, String updateIdField)

We want our method to accept any object, so we use Java Generics to help achieve that. Notice the method signature contains <T> which stands for Type. You see <T> again in the parameter list as well, List<T>, indicating we will accept a list of objects of any type. This takes care of the method signature.

Now, using Java Reflection, we can get the fields for the object without knowing them beforehand or having to specifically call their getter methods (getEmail(), getFirstname(), etc…).
Let’s start with iterating the List<T> objects. We use this HashMap for convenience…to gather all fields and values into a data structure that can be manipulated. I could alphabetize, sort, etc…using the Map.

for(Iterator<T> it = items.iterator();it.hasNext();) {

Object o = it.next();

HashMap<String, Object> fieldMap = new HashMap<String, Object>();

Next, we use Java Reflection to get a handle on the fields of the object.

Field[] f = o.getClass().getDeclaredFields();

We then iterate over the each field and store the field names and values in the Map.

for (int i = 0; i < o.getClass().getDeclaredFields().length; i++) {

f[i].setAccessible(true);

String name = f[i].getName();

Object value = null;

try {

value = f[i].get(o);

} catch (IllegalArgumentException | IllegalAccessException e) {

e.printStackTrace();

}

fieldMap.put(name, value);

}

Then, finally, we setup the ElasticSearch IndexRequest…passing in the map we declared earlier.

IndexRequest indexRequest = new IndexRequest(indexName, docType, String.valueOf(fieldMap.get(updateIdField))).source(fieldMap);

Done!

Pros & Cons

Pros:

I can load any simple object without knowing too much about it. Granted I have yet to try this with a complex object, where fields contain lists, maps, etc…it’s still a boost in productivity.

Cons:

All fields of the object will be iterated, and all fields will be passed on to ElasticSearch loading function. You could filter the fields you want, but that would have a negative impact on the convenience of this approach.

Very useful approach that does not call for any special libraries or techniques that are too funky.

Cheers!