The Plumbing of IT: Implementing an Integration Strategy

You have a request from your business area to integrate your student information to a new system. You ask your development team to assess it, and they determine that they can clone a batch interface that already has four versions that are running for four other departments. It will take six weeks and require forty hours of development and testing. The department is frustrated. You are frustrated. Why is this so difficult, and what can you do about it?

First, your situation is not unique. There is a solution to the problem. A holistic integration strategy is needed to solve the problem. You may want to introduce new techniques and tools. Using an ESB or iPaaS solution that leverages web services to integrate systems takes a lot less development time than creating and maintaining file-based integrations. This path requires some training of your team (and potentially the receiving department’s team). If you can allow departments to subscribe to a standard service, then you can multiply those savings by not developing it over and over. Think of standard services for employee data, chart of accounts data, and course data.

As you begin to look into modern integration methods and tools, you will find that with some planning and up-front investment, the efficiency, security, and consistency of integration can all be improved for a relatively small investment in time and dollars.

The Problem

The diagram below represents direct integrations between applications. It may be amusing, but it is all too real in some organizations.

As these integrations are developed over time, there may be inconsistencies with the data, and formats for the same data may be different in different integrations. Problems or issues encountered during the development of new integrations may not have been corrected in earlier integrations. Various programming languages may have been used and may not transfer well to newer integrations. The individual integrations may be buried in multiple applications and systems and, therefore, not easily found for troubleshooting and editing.  Most of these legacy integrations are also not real-time or near real-time.

The outcome of this approach is a major amount of technical debt. For example, a change to any common data element may require changes to a variety of integrations, developed in different ways, and using different technologies.

This approach is manageable if your integrations number in the dozens. But scale this to the hundreds, and you have a major headache.

The Solution

Your solution should be a platform that can help manage this complexity and simplify the work of connecting new technologies at your institution. It would look more like this diagram:

In this model, data is extracted from its source system using one code line and distributed to receiving systems as needed. As you move into web services—REST-based services especially—you can break down the integrations to smaller, transaction-level interactions that, in many cases, are able to be processed in near real-time. These methods can also carry file-sized payloads but process them in a more efficient way. In a scenario like this, we can find some critical benefits:

• Efficiency and ease of development and testing
• Processing efficiency
• Access and use management
• Security
• Citizen integrators
• Reusability and standardization

Keep in mind that with some integrations, the point-to-point architecture makes sense. For instance, some administrative system vendors provide pre-built “connectors” that they maintain and update as needed. Some examples of this are integrations with benefit providers and banks. Your integration ecosystem could be a mix of point-to-point and an integration services platform.

Types of Tools

There are several types of tools, which can sometime overlap. Here is a summary of what you might find on the marketplace.

ESB (enterprise service bus). This sounds like an engineering tool, and the early versions of ESBs were patterned from the bus on integrated circuits and computer chips. Traditionally, ESB is an on-premises solution that integrates on-premises applications and can manage and transform complex data for integration. Some ESB products can now be implemented in a hybrid style, with cloud and on-premises components working together to manage traffic.

iPaaS (integration platform as a service). iPaaS is a cloud solution that integrates cloud solutions, though it has elements of an ESB under the hood. It often has pre-built connectors for a set of cloud applications and requires fewer development skills.

ESB and iPaaS are quickly moving to adopt the characteristics of each other. Most ESB solutions can also now manage integrations with cloud applications, and most iPaaS can also now manage on-premises integrations.

How Do I Start?

The great thing about these technologies is that you do not need to replace dozens (or hundreds) of integrations at one time. It is critical to gain the skills and technologies and get comfortable with them before completing your full design and rollout schedule. This change impacts nearly every developer in the organization (unless you have already built an integration team), and it requires them to learn new techniques and design patterns.

Here are some steps you can take to prepare. Details of these steps will follow in a subsequent Tambellini guide.

• Choose a small team.
• Experiment with tools.
• Examine related strategies.
• Inventory your integrations.
• Outline an integration strategy.
• Move cautiously first, then accelerate.

Who Cares?

As noted above, there are many stakeholders on campus who may or may not yet know that they should care about this work.

• IT
• CISO (Chief Information Security Officer)
• Institutional Research
• Data Owners/Stewards

Integration is a difficult topic to get anyone excited about. It is the plumbing of the IT world. However, the potential to optimize, standardize, and streamline integration in any organization is worth investing in. Have you ever lived in a house with bad plumbing? It is not a pleasant experience! By investing in this critical organizational infrastructure and your team, you can create agility in adopting and replacing cloud platforms, standardize the data passing through your organization, and move data in real-time among the institution’s applications.