Feeding ElasticSearch with Java Generics and Java Reflection

If you’ve read my blog before, you know how I feel about ElasticSearch (LOVE).

When you have to load as much as data as I have to, things become a bit tedious with several sources of:

  • A database
  • A web service
  • A parsed file
  • Etc..

You are probably storing the data into object. If so, you would be forced to iterate each object based off its specific fields. Every object will have different fields:


  • Firstname
  • Lastname
  • Email

And your Registration object will have:

  • badgeId
  • registration_date
  • Sessions

So, for each object type, you would have to iterate those specific fields, either in separate classes or separate methods.

for(Iterator<HubspotContact> it = registrants.iterator();it.hasNext();) {

HubspotContact registrant = it.next();

get fields…….(getEmail(), getFirstname(), etc….)


So, what’s the solution?

Java Generics + Java Reflection = Easy ElasticSearch

Let’s start with your method signature:

public static <T> void insertData(List<T> items, String indexName, String updateIdField)

We want our method to accept any object, so we use Java Generics to help achieve that. Notice the method signature contains <T> which stands for Type. You see <T> again in the parameter list as well, List<T>, indicating we will accept a list of objects of any type. This takes care of the method signature.

Now, using Java Reflection, we can get the fields for the object without knowing them beforehand or having to specifically call their getter methods (getEmail(), getFirstname(), etc…).
Let’s start with iterating the List<T> objects. We use this HashMap for convenience…to gather all fields and values into a data structure that can be manipulated. I could alphabetize, sort, etc…using the Map.

for(Iterator<T> it = items.iterator();it.hasNext();) {

Object o = it.next();

HashMap<String, Object> fieldMap = new HashMap<String, Object>();
Next, we use Java Reflection to get a handle on the fields of the object.
Field[] f = o.getClass().getDeclaredFields();
We then iterate over the each field and store the field names and values in the Map.
for (int i = 0; i < o.getClass().getDeclaredFields().length; i++) {
String name = f[i].getName();
Object value = null;
try {
value = f[i].get(o);
} catch (IllegalArgumentException | IllegalAccessException e) {
fieldMap.put(name, value);
Then, finally, we setup the ElasticSearch IndexRequest…passing in the map we declared earlier.
IndexRequest indexRequest = new IndexRequest(indexName, docType, String.valueOf(fieldMap.get(updateIdField))).source(fieldMap);


Pros & Cons


I can load any simple object without knowing too much about it. Granted I have yet to try this with a complex object, where fields contain lists, maps, etc…it’s still a boost in productivity.


All fields of the object will be iterated, and all fields will be passed on to ElasticSearch loading function. You could filter the fields you want, but that would have a negative impact on the convenience of this approach.

Very useful approach that does not call for any special libraries or techniques that are too funky.


Web Architecture: Setting Up For Success

Once upon a time there was a non-profit organization that wanted to make great improvements to their website. They wanted their website to become more dynamic, interactive and provide tailored content!

Let’s call this non-profit organization ABC! Company ABC wanted to create an account home section for their members when they login to the website. This landing page would be called MyABC and has items like:

  • review & edit profile
  • print member certificate

As you can read, these are basic necessities for a home account. Let’s push this account home further and provide a better user experience for our members.

New Idea! ADMIN users will help to improve the MyABC user experience. We will allow ADMIN users to tag the pages in the website so we can have categorized content. This way content can be tailored for each user. For example, user A specializes in Cancer Research. ADMIN users would tag pages with cancer related content using keywords like “cancer” and “cancer research”. We can then place links to these pages for said user, providing a custom experience.

Even better, by tagging and categorizing these pages, we can then feed that category information into our search engine. In my case, I am using Elasticsearch. We can force users to specific content or just make suggestions. For example:

Here is some sample JSON data about a page on the ABC website:

    "title": "Testing Suggestions",
    "description": "Use this to test...",
    "tags": [

So let’s take a quick inventory so far:

  • JSON data with categories
  • Store JSON data into Elasticsearch
  • use categories to for more precise search

Here’s a screen shot of what we can end up with:


We type the category words we used when storing the data (“test”, “testing”), using JQuery autocomplete, we make an ajax call to Elasticsearch and return the titles of the document(s) matching the keywords!

This figure is a summation of how the data has been stored:


How does this help us in the long run? We get the capability to do the following:

  • Know traffic volume, since we can restrict users to pages available in our search engine, matching to specific keywords!!!
  • Categorized content can appear in the user’s MyABC (account home) page.
  • Send these clicks and page visits to Google Analytics for further analysis.

This was a very high-level write-up. We will cover more details in later posts. The point is to show how categorizing content, whether early or late in the website building process, can earn big gains on the SEO side of things!

Elasticsearch, cut back on your JavaScript

One of my favorite things about Elasticsearch is how it fits into the technology ecosystem. Elasticsearch has proven to be very helpful in not only accomplishing a great search, but also it helps to alleviate some of the work when building a new site/web app.

Take a JavaScript based search for example. My shop uses AngularJS, and we love it. I just wish our front end developer didn’t have to tediously handle data categories for each new set of data we get. That means copy-paste, parsing strings, worrying about commas, escaping characters.

This is done for every piece of data that is different and new.

With Elasticsearch getting a search up and running can be fast and efficient. Instead of parsing strings and escaping characters, we get right into actually categorizing the data.

Say I have some shoe products I’d like to create a search for. First thing is to get my data into a JSON format, Elasticsearch likes to be fed JSON.

About that JSON, it has categories. Elasticsearch likes categories, so much that it has its own name for them. They’re called “Types”.

So we have shoes:

ID Product_Name Type Price
1 Moon Walkers Boot $50
2 Air Jordans Running $100


Here is the first row from the above table in JSON:


“ID”: “1,”,

“Product_Name”: “Moon Walkers”,

“Type”: “Boot”,

“Price”: “$50.00”



Now that the data is stored in Elasticsearch, let’s dig a little deeper as to how Elasticsearch can speed up search creation.

Elasticsearch mostly deals with Types when concerned with data. Instead of a heavy amount of JavaScript, you can send an AJAX request to Elasticsearch for particular type(s). Then, in javascript, your only concern would be filtering the returned text values using a text box!


Ok, I could show a bit more on how to actually do this!

In Elasticsearch, commands are JSON strings. Say we wanted to get all shoes with the type “Boot”. We’d do something like this:

GET /_search


“query”: {

“type” : {

“value” : “Boot”





Imagine a list if checkboxes, of shoe types, “Boot”, “Running”, etc… When a user clicks a check box, we make an ajax call to Elasticsearch, getting data by type. The returned data is then filtered in a JavaScript text box, in our case, AngularJS.




So, your JavaScript and Elasticsearch can work in tandem.

  • Elasticsearch gets your data
  • JavaScript filters the data by text

Rolling out a new search this way is much faster than pure JavaScript on the searching and filtering.