Document Conversion API now open source

opensource

Last Friday I wrote how we implemented Document Conversion in HR Assistant.

Part of the solution was a new project, Document Conversion API

We decided to open source it. Below I copied the details from the previous post about this new project.

Details

It is a Spring boot application, which can run stand alone. I have added also Docker support.

See the ReadMe for more details

Endpoints

Url: /api/convert-to-plain-text
Method: POST
Consumes: File
Produces: JSON
{
“convertedText”: “text”,
“message”: “information or error message”
}

document conversion request

Url: /api/convert-data-to-plain-text
Method: POST
Consumes: JSON
{
“mediaType”: “text”,
“data”: “base64 encoded representation of text/file”
}
Produces: JSON
{
“convertedText”: “text”,
“message”: “information or error message”
}

Swagger

To make it easy to test the api I have added Swagger

swagger

Repository

The Document Conversion API can be found in a public repository on Bitbucket. Feel free to use it or improve it. Better let me know if there are new feature request or add it your self and submit a Pull Request

twittergoogle_plusredditpinterestlinkedintumblrby feather

HR Assistant – Document Conversion but different

Document conversion was one of the top items on our backlog which didn’t make it in the final contest version.

 

What is Watson Document Conversion

On the Document Conversion API documentation page is the following stated about this service

“The IBM Watson™ Document conversion service converts a single HTML, PDF, or Microsoft Word™ document into a normalized HTML, plain text, or a set of JSON-formatted Answer units that can be used with other Watson services. Carefully inspect output to make sure that it contains all elements and metadata required by your or your organization’s security standards.”

 

How we use it.

In HR Assistant we use the Watson Document Conversion Service in the Job Application part.

A job applicant upload a resume in PDF or docx format.The data is stored in the document on Cloudant.

When someone is analysing the data of the job application.

Step1: if there is an attachment, it will be send to the Watson Document Conversion to get the text representation in return.

flow Watson Document Conversion Service

Step 2: combine the text from Document with other text of the job application and send to the various Watson services we use in HR Assistant.

Change of plans

Watson Document Conversion is already deprecated. For stand alone conversion as we do in HR Assistant there is no migration to other Watson services. IBM recommands to use Apache Tika for this sort of operations.

So we have  to refactor the current Document Conversion service in HR Assistant.

 

StHello Document Conversion API

Due to security restrictions in Domino, I created a simple Spring Boot application, with 2 end points. I deployed it via the command line to Bluemix. And secured the endpoints via the build-in API Management.

Endpoints

Url: /api/convert-to-plain-text
Method: POST
Consumes: File
Produces: JSON
{
“convertedText”: “text”,
“message”: “information or error message”
}

File request

Url: /api/convert-data-to-plain-text
Method: POST
Consumes: JSON
{
“mediaType”: “text”,
“data”: “base64 encoded representation of text/file”
}
Produces: JSON
{
“convertedText”: “text”,
“message”: “information or error message”
}

Swagger

To make it easy to test the api I have added Swagger

swagger

Bonus

As a bonus I have added also Docker support, so it can run everywhere. You can find the Instruction in the ReadMe.

The Document Conversion API can be found in a public repository on Bitbucket.Feel free to use it. Better let me know if there are new feature request or add it your self and submit a Pull Request

twittergoogle_plusredditpinterestlinkedintumblrby feather

Quick tip: access the application log files on Bluemix

Last night when I deployed a new version of HR Assistant to Bluemix I ran in some troubles.

When the application was deployed and tried to access it via de Application URL I got a 500 internal error.

Find the log

First step was to open the log page, but was not very explanatory.

Log files on Bluemix

Next step was to open Kibana. There I found more information.

Kibana log

As you can see, I need to consult the error-log-0.xml file for more information

The challenge is where to find this log file.

Solution

I started, in my case, a Terminal window, to login on Bluemix.

Terminal login

Next was to start a SSH session for my app. This can be done by

bx cf ssh HRAssistant

 

After that I was able to open the error-log-0.xml file with the command

cat app/notesdata/domino/workspace/logs/error-log-0.xml

Now it was easy to find the cause of the Internal Error.

 

After I fix the issue, I uploaded a new version, which runs without problems.

twittergoogle_plusredditpinterestlinkedintumblrby feather

HR Assistant – Hello Domino….

While we developed HR Assistant it ran smoothly on a Domino server, but it was not using some strength of Domino, security.

Run on Domino

I made HR Assistant aware of the runtime it runs on, it ‘knows’ when it runs on Bluemix.

If that is the case our own developed login features comes in place, but…

When it runs on Domino it will enforce the ACL, which gave great advantages, by assign groups to the 3 roles of the application, HR, JobApplicant and Administrator. In the end you are elevating the security of the application

Store on Domino

Another feature we wanted to implement, which was on the wish list, was the ability to store the data in the NSF file.

With the great feature of the OpenNTF Domino API, I was able to store the document as JSON in field. Just the way I store the data in the Cloudant database. This feature give the ability to store what ever in a field and it will converted automatically if the data will reach the field limit

To achieve this, I created a new DominoJSONStore which do the same as the CloudantService, store the data in the respective NoSQL store

Choose your NoSQL datastore

I thought it would be nice to be in control of which NoSQL datastore to be used, without the need of the Domino Designer.

I found a great NAPIUtility file created by Sven Hasselbach. More details on his blog and it works very well.

I created a settings page where I load my custom property file, which ‘lives’ in the application.

On this page you can select the dataStorage to use for the application.

After saving the settings page, you need to re-authenticate to have the new settings effective.

Bonus

As you see, there are more settings, like the cloudant username and password. Now I can connect the applications directly to another Cloudant database.

It opens new possibilities. For example by adding the credentials of the various Watson Services. There is no need anymore to store these credentials in the Faces-config.xml.

Be aware

The settings page is not available on the current instance running on Bluemix as the login functionality is not that strict as on Domino. On Domino the settings page is only available for the users with the Administrator role.

Contact

If you want more information about HR Assistant and all the functionalities, let me know.

twittergoogle_plusredditpinterestlinkedintumblrby feather

Quick tip: Make a backup of your Cloudant database

Cloudant

For HR Assistant, we use Cloudant database as storage of the data on the IBM Cloud

Today I was thinking about ‘what if we loose our data as something goes wrong’. It is not likeable, but still possible.

Make a backup

On IBM Cloud, there is documentation about making a backup of your data.

So installed the npm package couchbackup  and run the command as described in the documentation.]

I changed only to store the output as a JSON file.

When I run the command I hit an error about authentication

Cloudant_Backup_Error

Set permissions

I move over to my Cloudant database and opened the permissions page. On that page I selected, temporarily, the _reader checkbox for the Unauthenticated connections.

Moved back to Terminal and return the backup command, this time it ran successful.

After inspecting the backup JSON file, I was really happy with the result.

 

twittergoogle_plusredditpinterestlinkedintumblrby feather

HR Assistant – now available in every language

One of the latest features I had add to HR Assistant is multi language support.

It was obvious we needed this new features, as not the whole world is talking English 😉

One of my goals was to make it as configurable as possible.

Default language

The default language of the application is still English. All the labels are stored in the application it self as a property file if there are no languages configured these property file will be loaded.

On the overview page of the languages is the possibility to download this property file with all the labels to make it easy to translate all the labels before adding to the application

HR Assistant export default language

New language

By clicking the button ‘Create language’, there will be a new language page opened, with all the labels already filled at the bottom of the page.

Give it a meaningful name and code.

HR Assistant language page

The important part is at the bottom, the labels.

You can fill in the labels on the page it self, but just above the labels is upload functionality to import the already translated default property which you can download from the language over page.

By saving the page all the labels will be stored in the database.

Switch language

To switch the application to another language, open the language page of preference and click the ‘Make active’ button. In the background the application will switch to the other language.

By refreshing the browser HR Assistant has changed to the active language.

Export language

On the Language page there is another button, ‘Export labels’. By clicking this button, the user will get an export of all the keys and labels as property file.

Conclusion

In my opinion we have now a highly flexible solution for multi language support in HR Assistant.

If you want to know more of HR Assistant leave me a comment.

twittergoogle_plusredditpinterestlinkedintumblrby feather

HR Assistant – what happened since last year

Last year I won together with Frédéric Dehédin the ICS Developer Contest 2016 with our HR Assistant application, a cognitive solution to help the HR department with of IBM Watson services.

As required the code was open sourced at a public Github repository.

What happened with HR Assistant since that moment?

From public to closed

First thing was to take the code back 😉

In the final stage of the development we were not able to include all the features we wanted upfront. So we skipped some features in favour of stability.

We move over to a closed repository at a GIT server, with the goal to make a real product out of it together with Belsoft Collaboration

Share the knowledge

I had the opportunity to speak at several conferences to share the knowledge I had learned in this contest.

At Sutol and Meet IT I spoke about Cloudant and the similarities with the NSF on Domino.

In 2017 I had the opportunity to share some technical insight of the HR Assistant application at IBM Connect in San Francisco, IBM InterConnect in Las Vegas and last but not least Engage in Antwerp. For slides see Slideshare.

Adding new features

We had a backlog of features we would like to include in the product. The next period I will write blogs about these new features we have added to HR Assistant over time

  • Watson Document Conversion
  • Run it on Domino
  • Multi language support
  • and more….

 

IBM Watson services

 

Contact

If you want to know more about HR Assistant, let me know. I can give a demo.

twittergoogle_plusredditpinterestlinkedintumblrby feather

Domino Version 10 and Beyond – a few toughts

Yesterday  announced the strategic partnership between IBM and HCL to move the Notes and Domino Family forward. In 2018 a brand new version, 10, will be released.

For lots of customers who have invested in Domino applications it is very good news. Less need to move away to other products.

IBM will be organising jams, to define the future of the product. It is a great opportunity to show interest in the product and what you think is most important in the product. Sign up know to keep informed.

My thoughts about the way forward

I am a developer and nowadays develop mostly outside the ICS space, I have some thoughts to get back.

Deprecate Lotusscript

Lotusscript is nice but will not move the product forward. Deprecate is, so the application is still backwards compatible, but stop investing time in a newer version. It makes the product not compatible with other products.

 

Cut the wire

As soon as possible, remove the relation between the Notes Client and the Domino Designer. This relationship slows down the big steps forward the Domino Designer needs.

 

No fat client

Who needs a fat client, in my opinion there is less need of a fat client, like the Notes Client. Most of customers can do their business with a web based solution. So give most love to the Domino Designer and Admin and NOT the Notes Client. Migrate it your self of use the Aveedo or Darwino

 

Make the Designer flexible

Make the the Domino Designer available as flexible as possible. For example available as a Eclipse or IntelliJ plugin. And of course remove the restriction to run the Domino Designer needs Windows.

 

So lots of things to improve, hope to see you at the jam, in person or virtual.

 

Lets jam and make the future bright again.

DominoJam

twittergoogle_plusredditpinterestlinkedintumblrby feather

IBM Champion nominations 2018 are open

It is the time of the year again. Time to thanks your favourite IBM technology expert by nominate them as IBM Champion 2018.

More details on this blog post by the Libby Ingrassia, the overall IBM Champion manager, or watch the video below.

 

The deadline to nominate is November 13th, so start nominating today, https://www.ibm.com/developerworks/champion/?ca=dw-_-IBMChampion-_-dwtv-Featured-_-category

 

IBM Champion logo

twittergoogle_plusredditpinterestlinkedintumblrby feather

Spring boot and Thymeleaf – rapid application development

springboot and thymleaf

At one of my customers I was able to use to build an application with preview functionality of a business critical application.
As usual the business wants this application as soon as possible. So the obvious choice was Spring boot in combination with Thymeleaf.

Spring Boot

Spring boot created spring based applications which can run as a runnable jar, with embedded a Tomcat server. But a Spring boot application can with a few tweaks run on a Websphere Application Server or Websphere Liberty Profile, and in a Docker or on Cloudfoundry (Bluemix or Pivotal).

Google and you will find tons of examples

Thymeleaf

Thymeleaf a templating engine based on Java. It supports layouts and fragments. Fragments are very useful, because you can reuse these fragments in different layouts. The code in one place.

There is an extensive documentation available

Get started

To get started the best way is to head over to the Spring Initializr. You choose your Group, ArtefactId, Package name, packaging and java version.

The big plus of the Spring Initializr is to specify all the additional projects you need for your application. I can reccommand to switch to full version, where you can see which projects you can choose. From SQL, Web, Templating and Cloud projects.

Last thing is to generate the project. It will deliver a downloadable zip file with the correct Java project inside.

Unpack the zip file, open the project in your favourite Java editor, like Eclipse or in my case IntelliJ

Run the code

In IntelliJ under Spring Boot just double click the SpringBoot application. Or in my case I created a Maven run/debug configuration, ‘Spring Boot Run’, with the command line command ‘spring-boot:run’.

And voila, your application is up and running via localhost:8080

 

twittergoogle_plusredditpinterestlinkedintumblrby feather