Darren Wesemann Blog

As a follow on to my Financial Times article here, I’d like to further point out that as many debate how long it takes for the benefits of SOA to be realized, proper context of which benefits is important, because all benefits are not created equal.  The statement I referred to in my article by Larry Ellison here where he claims the benefits of SOA will be realized in 10 to 20 years time, after applications have been rewritten, but the benefit-context he’s likely referring couldn’t be agility because his assumption that all applications need to be rewritten is completely contrary to SOA principles.  A core principle of SOA lies in leveraging prior IT investments by decomposing existing applications (not rewriting them), allowing for flexibility in meeting changing business demand.  The key benefit-context of SOA is agility.  Perhaps Larry is talking to those that expect to see cost savings in the first budget year of an SOA project, which takes much longer.  Still, rewriting entire applications is a bit extreme, and often fatal. 

The proper focus should be on agility because phases of implementation are yield early returns, yielding benefit along the way, and resulting in solution flexibility early on.   SOA will continue to turn applications into collections of loosely coupled services by unlocking buried or hidden components, making them more accessible.  Although SOA is technology (depending upon protocols and technical infrastructure), its primary purpose is in responding rapidly to constantly shifting business needs.  As I indicated in my FT article: “It’s about business.  An agility-based and business-focused mindset is required in order to realize benefits early.  Agility refers to the flexibility in implementing a business process rapidly and perhaps in phases if needed.  In contrast to a rip-and-replace mentality for accommodating a business requirement, SOA endeavors to leave most of the existing applications and infrastructure in place.”  Focus on agility, and benefits will roll in much sooner.

SunGard meets regularly with its customers both individually as well as in groups and forums to gather input and validate direction.  One such counsel is the CSA & Infinity Customer Advisory Board, which is made up of technology leaders from a cross-section of the market we address.  In a recent meeting, I presented a process paradigm we’ve matured as a company through years of our SOA effort.  The discussion that ensued validated the concept as a practical approach to SOA.  The paradigm is a combination of top-down and bottom-up collaboration.  The top-down approach refers to the business-centric domain, which focuses on the flexibility in solution creation and delivery, solving business problems, addressing customer needs and thereby growing revenue.  The bottom-up approach refers to the technical-centric domain, which focuses on common code parts, sharing components, fine tuning architectures and best-of-breed development practices.  One without the other tends to throw the process off-balance, so a mix of the two is ideal, but the structure of how they weave together is a little complex, and (at least in our case) isn’t at all trivial, and I have to admit that we’re still learning too.

Consider the key drivers of the two approaches first.  Most agree that flexibility (a revenue generator) is a stronger and shorter-term benefit than reuse (which is a cost-savings measure).  Most of the early press and hype on SOA gave the distinct impression that SOA benefits favor cost savings enabled by reuse, but recent research is validating that flexibility is more important than cost savings, at least in the short term.

Given that, it’s important to realize that the business domain (top-down) needs to weigh in heavier in the mix, at least as it relates to the prioritization of the SOA activities across the board (see my blog entry from last year here regarding the importance of a business focus on SOA).  This fosters the energy required to accelerate the flexibility benefits early on.  Meaningful reuse takes time anyway, in fact those that realize that sooner rather than later end up less trodden down by missed expectations.  In a delicate balance, developers and architects need to deal with the shared architectures and components that both follow those priorities as well as deal with the lower-level technical aspects the business domain doesn’t need to engage in (stuff like which message caching approach to use, which object-relational mapping to adopt, what debugger to leverage, or which API to chose for a particular task, etc).

SOA strategies that weigh heavier on reuse create problems by focusing on technical design and implementation factors that are actually less likely to produce viable business value than functional design models, (which are the focus of flexibility and change priorities).  Consequently, if the bottom-up focus is in-line with the business priorities, and governs itself within the discipline of the lower-level architecture (while at the same time, not dragging the business folks too deep into the technical details), then the mix works.

I find it fascinating to watch the current battles between application platforms for the SaaS (Software-as-a-Service) paradigm.  I can’t help imagining what the application delivery landscape will look like in 10 years. 

Clearly, SaaS abstracts a number of things away, such as the Operating System (no longer visible, because in a SaaS environment exposes only the service, not the stack underneath), whereas traditional applications make you not only aware of what Operating System is, but the hardware and the application server (expanding from both ends of the OS) are also important to be made aware of for deployment, management and integration purposes.  Abstraction simplifies things, so it’s save to predict more abstraction, more simplification.

Another safe bet is that the platforms upon which SaaS applications are delivered will converge (makes sense, early stage domains typically see lots of convergence).  At some point developers of SaaS-delivered applications should expect to be building from a common foundation, and should be able to be deriving and consuming from common libraries of services.

Although it’s a bit early to predict what framework(s) and/or what provider(s) will win the current SaaS platform battles, it’s interesting to ponder the various players and approaches.  They are:

Pure SaaS plays: these are providers claiming that everything should run in the cloud.  Such as Force.com (Salesforce.com’s platform as a service).  Google has entered the cloud arena as well as long-time providers of cloud compute cycles like Amazon.

Hybrid SaaS/On-premise plays: these are guys providing a solution embracing cloud computing and site-installed applications.  Microsoft endorses this approach, with its Software+Services campaign.  SunGard as well fits into this camp, because Infinity supports service access methods (on-demand and download/install/run scenarios). 

Unlike the pure SaaS method, the hybrid approach deals with the fact that there is usually something local.  In the case of SunGard (who provides software solutions to the Financial Services sector) there is ALWAYS something to integrate with (other applications, data, processes, etc).  Extending and integrating is not going away, therefore the outlook seams to lead me to believe the hybrid approach will be with us long-term, as it’s the only practical method, whereas the pure-SaaS play ignores too many real-world tactics, where the “rubber meets the road”.

Another topic that’s interesting to ponder regarding the future is how services themselves will evolve.  It appears that more and more of the services we rely on are taking up residence outside the enterprise.  If this trend continues, we could cross the line where the majority of what we rely on is serviced somewhere else (companies that outsource heavily can already claim that they are more dependent upon externally executed services than in-house ones).  I guess this is what some refer to as Universal SOA, which seems to be a safe bet for the future, because enterprises are continually becoming more and more comfortable with depending upon external services.

I get a fair amount of email with comments on my blog entries… usually people mention they have company policies that prohibit them from commenting on public blogs, (mostly Tier-1 Banks), so they send their comments via email instead of posting.  Because I comment on SOA in the Financial Services industry, a lot of folks that have something to say generally have to do it privately.  At the risk of losing a regular commenter (I’ll keep the names private to protect the innocent) I would like to make mention of a comment on my blog entry BPM Utilities by themselves versus SOA, SaaS and Content.

The comment makes reference to an article here, written by Kaushal Mashruwala in Financial Express, who makes the point that BPM is a key to SOA enabling business executives and managers to identify with, which is very important because business people don’t understand SOA generally.  The combination of BPM is an enabling fixture in SOA, which allows business management to tangibly understanding what SOA does, in particular it’s benefits.  BPM with SOA empowers business people with the power to control their own destiny in many cases.

I couldn’t agree with Kaushal more, as he puts it: “This role of BPM as the business face of SOA is not just a possibility. It’s happening now.”  Indeed, BPM is the business face of SOA.

Thanks for the heads upon this article; I think it solidifies my early points quite well.  Keep the comments coming, even if they are via email in lieu of posting on the blog-roll

I regularly get to hear various vendor pitches on their SOA offerings.  It never ceases to amaze me how many pitch their offering as a “complete package”.  I usually get the biggest chuckle when a vendor merely dresses up a message involving an app that has existed for year, which magically and all of the sudden has grown SOA-legs simply by saying so (the power of marketing).  Give me a break people!  SOA can’t be bought like a traditional application.

SOA is an architecture, and you can’t buy an architecture, only the artifacts that implement the architecture.  But it’s also a concept that involves specific methodologies, and you can’t package methodologies.  In fact, buying traditional applications in the traditional way is actually abrasive to the whole SOA concept, because the concept involves interacting components/services/processes (chunks) from in-house development or external development (vendor or partner based), i.e. disparate teams.

I’m encouraged when I see traditional software vendors service-enable their historically traditional applications (and new ones of course).  SunGard fits into this category, and seeing that gives me hope for the future of SOA.  There are plenty however, that still don’t get it, and are diluting the whole message and purpose of SOA.  I speak with CIO’s in the financial services industry all the time, and there indeed is considerable confusion resulting from the convoluted vendor messages around SOA today.

It must be tempting to buy an EVERYTHING-SOA, software suites from the likes of Oracle, IBM, and SAP because of the illusion that the SOA task can then be checked off the list of “to-do’s”.   SOA is all about creating flexible, modular environments meaning you can add or remove services, processes, rules, etc to map to a business need.

Begin with a business need, and let the methodology of service orientation implement the need with agility, as opposed to the other way around (i.e. assuming an off-the-shelf SOA environment will do the trick), the world is just too complicated for an EVERYTHING-SOA, it’s a fantasy.

At the Gartner BPM Summit last week it was apparent that a lot of BPM vendors are now making mention of SOA and some of SaaS.  Although I agree there is tremendous power and opportunity in these complementary concepts (see my blog on SOA+BPM+SaaS here), based upon what I saw at the Summit, I’m not sure the BPM industry gets just how this all fits together just yet.

Here’s what I don’t think they get… The codependence of BPM and SOA takes on a rather circular definition: BPM enhances services, but services can also be processes.  SaaS has a similar relationship with both BPM and SOA in that solutions made with processes and services are better off consuming components on demand, while at the same time a hosted environment is more useful if it serves components (in SOA fashion) enabling business processes to be modeled (in BPM fashion) including human-activities (workflow).  With this interdependence comes a powerful extension to the original BPM concept; however the reality is that it requires tools and governance with content, all integrated into a common offering with hooks to further extension. 

The traditional view of BPM is that it exists to manage system tasks as well as human-activities, (and of course the combinations of both).  I had lunch with some folks from a competitor at the Summit last week, who observed that most businesses are not heavily automated so a great deal of business activity involves human interactions, and therefore very little can be put into transactions or services.  Although I agreed with him that there indeed is a class of work that revolves around workflow-like activities, this work has also automation potential at the level that a service may be needed.  Further, when those services (originating from human workflow) are reused, you essentially have the beginnings of an SOA.

Don’t get me wrong, of course a lot of business activity does not include the typical SOA components, such as systems, integration, transactions, data storage, etc. but there most certainly are parts of each business that do include these things, and the human part typically interfaces with them, and the more work the business can offload to the non-human part, the better.  Furthermore, SOA’s are all about breaking up applications into smaller chunks, and occasionally those chunks automate something new and perhaps reusable by other processes.

I got the feeling from many vendors at the Summit that they look at SOA myopically, instead of truly uncovering what SOA can do for a company by enabling it to better manage what’s automated and what’s not automated, and therefore offering an integrated suite.  SOA is not just infrastructure but also involves how a company developers and delivers solutions (whether they are process-based or otherwise).  SOA involves combinations of parts or chunks (components or subsystems) that interact in a loosely couple way, without the overhead and dependencies of older distributed computing approaches (involving heavy state persistence and no clear interface-contract).  The parts that make up the SOA are autonomous with respect to each other and agnostic to the dependencies of other part of the system.  Each part has its own lifecycle and release path yet can participate in the interactions with other part in a composite solution (whether process-oriented or not).  The composite solution governs all interactions by the same interface-contract and data flow.  Eventually the components of the composite solution appear in multiple processes and as businesses achieve higher reuse, the system is more SOA, and if BPM is in the picture cohesively then processes and chunks of processes can also see meaningful reuse.

I think some traditional BPM guys are missing the power of SOA in that autonomous parts can be assembled into solutions that can act like canned applications people are used to, only they have increased agility and greater tolerance for change throughout the life cycle of a system.  The legacy-way of building applications assumes tight coupling and homogeneity across the parts of the system, which create rigidity and large costs to change or enhance.  Coupling SOA with BPM facilitates the consumption of services by processes, and services themselves may be process flows that can be leveraged in other flows. 

Lastly, probably the biggest problem the traditional BPM folks have is they approach the market with merely a utility or infrastructure, and simply saying “SOA” or “SaaS” doesn’t make it a cohesive offering.  Content and governance are key aspects of a unified SOA+BPM+SaaS strategy.  Many of the traditional “utilities” and “infrastructure” have been acquired and merged with larger entities, such is the trend. 

I was speaking with a VC friend this morning regarding the Workday (a company with an ERP SaaS offering) acquisition of Cape Clear (a company with an ESB/SOA offering) and what that means.   I recited what I said above, that utilities and infrastructure ultimately gets snapped up by those that have content (or those that solve specific business problems).  The trend is to move away from complicated utilities and infrastructure (by themselves) and end up with business solutions mapped to business needs, with utilities and infrastructure embedded in a cohesive environment.  Content is king.  Workday says it’s leveraging its Cape Clear purchase as “Integration on Demand,” which is another way of saying “Bring our Business Solution to meet your needs faster with agility”.  I applaud Workday on their acquisition, it confirms what I see as the trend, and it makes me feel great about SunGard’s position with its Infinity strategy, because we have content (hundreds of applications), governance processes and methodologies, and, oh yes of course, utilities and infrastructure.

People used to talk about centralizing everything to make an SOA work (centralized ESB, centralized registry, centralized policy enforcement, etc).  Although centralizing a team to manage an overall SOA effort is still a very popular and successful strategy, the centralization of the supporting technologies isn’t necessary (or practical) in many large organizations.  For example, SunGard used to think in terms of maturity steps leading components towards a complete compliant stack, centralized server architecture).  We’ve learned that by identifying what each component is and certifying certain attributes (but leaving it where it is) is far more useful to emphasize.  This is a federated approach of sorts.

Leaving things where they are, but certifying the definition of key attributes (and organizing their availability to business needs, resulting in composite solutions) is an important step in the federation model.  If you have familiarity with SunGard’s CSA Levels (the centralized approach), the maturity model was recently superseded by a federated approach to attribute definition.  The following structure identifies the method of consumption of a business service, and serves as a certifications federation model for the compliance of any software artifact indexed for consumption:

• Service Interface Certification: complies with standards associated with encapsulation or service interfaces and message protocols (such as WSDL/SOAP, FIX, etc.), replaces Level 1.
• User Authentication Certification: complies with authentication standards, (single sign-on), replaces 1/3 Level 2.
• User Interface Certification: complies with UI standards, (Sphinx), replaces 1/3 Level 2.
• User Authorization Certification: complies with authorization standards, (entitlements), replaces 1/3 Level 2.
• Data Model Certification: complies with one of the standard data models, replaces Level 3.
• Reference Implementation Certification: relies on full reference implementation stack, replaces Level 4.

Another important step in federation enabling SOA is ditching the notion that a centralized service registry contains all your artifacts available to the SOA (at least in terms of how most service registries are defined today).  Instead, the concept of solutions organized through a federation of registries is far more SOA-friendly, and with a federated certifications model, such as the one above, indexing assets across an organization begins to make the federation gel.

SunGard established the concept of a Solution Registry to differentiate the idea from service registries.  Solution Registries manage a federated network of service registries (where low-level service components may be found), and create collections of services as solutions (which are things a business individual could actually make use of).  SunGard’s Infinity offering provides the solutions umbrella linking services to business needs through a federation model of software artifacts wherever they may reside.

ESB’s and other parts of the SOA infrastructure stack are yet another area people have tried to centralize in the past.  At least in large organizations, many are finding that too requires a federated approach to be successful.  As it turns out, mandating a central broker implies that a central support team would have to understand the details of every component and application.  Instead, the delegation model of distributing the knowledge load to others (keeping expertise where it is, with the software) fosters the right kind of ownership that thrives in a successful SOA.

Last week, in a conversation with group of analysts who cover the SOA space, it was clear that some of them see a central ESB as a hindrance in some large companies, as opposed to an enabler because of the fragmentation of the messaging layer.  No two ESB’s look the same, and software intended for reuse often runs into incompatibilities due to the lack of agnosticism between vendors with messaging.  A federated model can ensure the mess is organized, but federation needs to be implemented with a number of other things in order to work well (such as those mentioned above, i.e. registry, certifications, etc).

The SOA Forum surveyed over 500 organizations and concluded that nine out of ten haven’t figured out SOA governance.  I’m quoting it here because this find actually matches my experience with SOA initiatives I’m seeing in financial services IT organizations.  The banks that have advanced their SOA initiatives certainly have some form of governance, but it’s overwhelmingly manual and, for the most part, cover only some areas.  In those organizations, there are some impressive metrics, but they are compiled manually, and quality and policies are implemented casually (or not implemented at all).  The IT organizations I deal with have multi-billion dollar IT budgets and thousands of employees, so formalized and automated governance is going to be key at some point, or it simply won’t scale, and there is a high degree of risk the SOA will fail.

Some more color is needed here though, because SOA governance means different things to different people.  Many view policy enforcement as the sole mechanism for governance, and although a managed policy structure with automation is indeed and very relevant and important, there are other quite necessary functions in the governance realm as well, and few IT shops have implemented them.  For instance, successful SOA projects have established a common support framework mapped to their process including automation supporting the following:

• Requirements capture and management
• Project management
• Ticketing management
• Collaboration between teams around common services
• Registration of services and the certification of the advertised state of each
• Discovery of services for orchestration of composite solutions
• Consumption of services in development, testing and production
• Metering and billing of components in the runtime environment

Clearly, automated policy management can “check” things going in and out of registries and a few other places but a complete approach to governance is rarely seen implemented, (meaning a unified approach to cover the project management activities, runtime metering, bugs recorded, collaborative activities surrounding moving components forward or integration strategies, etc).  The convergence of these things in an automation paradigm coupled with a federated process to ensure teams are on the same page makes up the governance strategy of a successful SOA.

The survey referred to above also mentions that 56% of the respondents indicated that more than half of the software artifacts developed in-house are not reviewed for compliance before moving into production.  This problem isn’t going to be solved with a merely a policy manager.  Unless governance reaches the various functions and activities that touch a component, there will always be ways in which policies can slip through the cracks.  SOA governance needs a very broad, horizontal approach, one that maps process to support infrastructure covering the bulleted items I’ve listed above.

Although the survey asked the respondents for an estimate of how many services are checked when automated policy management is in place (to which the survey reports 88% say more than half of their services are checked, and 50% indicated all are checked), the question I have to ask is: what is “policy management” in the minds of the respondents?  There isn’t a clear answer here, and based upon my interactions with IT organizations in large financial institutions, I tend to believe there are varying ideas out there with respect to the definition of policy enforcement.

Other survey’s I’ve seen report very low percentages regarding satisfaction with governance solutions, but trend upwards for those with automation solutions.  Relatively speaking, it’s safe to say that automation supporting governance processes is key, and the broader the automation covers the more beneficial the governance is to the business.

This might sound shocking coming from a CTO, but SOA isn’t about technology, it’s about facilitating business.  Of course all of IT could be viewed that way.  I go deep into SOA projects in financial institutions on a regular basis and I find many of them initiated and driven out of IT, but with little initiative, support, even awareness from the business of SOA.  Most of the time it boils down to mere education, but whatever the case if an SOA project is purely IT, there will likely be a struggle.

I see metrics reported like “components reused” (don’t get me wrong, that’s a good metric, I report on that metric for SunGard internally as well) but there needs to be business context around it in order for the business to comprehend the value (for example, I include revenue generated, projected, etc. for each component, as well as some subjective attributes like revenue that wouldn’t have otherwise happened without SOA).

Last week I was with a customer whose SOA project is in trouble.  My interpretation of their situation is very simple… IT has little to no involvement from the business in their SOA effort.  They took the approach that the SOA “technology” could be inserted at a layer beneath what the business sees (i.e. the apps and functionality mapped to the business doesn’t change, and the SOA is inserted beneath, abstracted away).  Requirements are written almost exclusively by system analysts and no benefits were exposed to the business.  The project started failing when the business noticed service interruptions and other unexpected problems.  That’s completely the wrong way to go about it.

I routinely state that the process is more important than the technology, and here I’m throwing business direction, support and even leadership into that category as well.  The successful SOA’s have a core team working in a federated way with each business lead and their teams in an organized and collaborative manner, taking direction from the business and exposing value “chunks” little-by-little along the way.

If ever there were four completely independent concepts (in terms of evolution) that are destined to be related (even in the immediately family) and together revolutionize our thinking about how software delivers solutions to meet agile needs business needs, it is SOA, SaaS, BPM and Open Source. 

For a couple of decades people have made attempts at componentizing software for the purposes of reuse and rapid composition of solutions with certain measures of success.  Overall, the componentizing of software into parts that can be reused and assembled into composite solutions is far from an exact science. 

Many years ago I served as CTO for a hardware company that produced parallel processing chips for the super-computing marketplace (supplied to NASA, DoD, various government contractors, etc…).  I still remember battles my electrical engineering staff would have with my software developers regarding the lack of exactness in the componentization of technologies and the assembling into the solution (our product).  Developing hardware takes a very systematic approach (real engineering) involving libraries of components and assembling them together (via VHDL or schematic capture, for instance) into the ultimate solution.  Simulators can tell you if you get what you expected far in advance of production.  The components follow exact interfaces, the libraries are clear and open and the simulator works in conjunction with development environment to provide a complete view of what’s happening.

When the time would come to write the driver and supporting software for the hardware, the battles would ensue.  It became apparent that although I had been developing software for many years, I simply didn’t realize how non-exact (un-engineering-like) software development really is.  A driver and other software required didn’t have the luxury of open libraries to pick from, simulators to run exact assemblies through for precise results, instead the software was mostly not exactly what you expected, until multiple iterations of development and testing.  When the hardware is compiled, you have one shot at getting it right, whereas with software each compile yields an attempt at getting closer to what you want.

After multiple attempts at defining distributed component technologies for software (CORBA, DCOM, EJB, etc).  I think we’re finally getting very close to a far better component-and-assembly world for software solutions with the combined power of SOA, SaaS, BPM and Open Source.  SOA yields better standards than we’ve ever had before (not just in terms of software interfaces including WSDL, BPEL, SCA, etc., but the collaborative governance around how an enterprise functions as well), SaaS for the on-demand environment, BPM for the modeling and assembly of the solutions and Open Source for the openness of libraries and infrastructure. 

Having lived the differences between what the presence and absence of these attributes can have in the development of solutions with technology, I’m convinced these are powerful (and extremely related) attributes to successful composition of solutions from reusable components.