Understanding No-SQL and why is it important for middleware folks to know it!

Understanding No-SQL and why is it important for middleware folks to know it!

Team, this week’s post in my humble opinion is rather weird, I say this because in context of Application Infrastructure or many of the folks on this community who are middleware folks we see ourselves discussing Data!! But I am compelled to write this because this is becoming relevant – even more so when we are constantly being challenged with new architecture and design and middleware and “data centric middleware” are taking centre stage.

With Mobile/Social networks, there has been exponential growth in data --- this not new and we have now known this for several years. The challenge that we face now is how and what to do with this exponential growth? This past decade we have advised our clients to bring data closer to application for performance and scalability. So our clients have listened to us, and either due to necessity or by design made a conscious decision to make that transition. And NOW we are at this cross roads where Middleware meets Data…. So it is relevant!

So now broadly we should use the following technical criteria to distinguish the solutions:

1. Data access and Query Method – Native API, MapReduce
2. Language Support – Java, C++, etc
3. Protocol support for an access and Integration – HTTP / JSON, REST etc.
4. *Important* - Non functional requirements: Security, Scalability and Performance.

In general

1. All of these solution address or claim to address Scalability and are highly scalable distributed database systems
2. In memory access is faster than disk access- this is just Physics so any solution providing in memory access, will have comparable performance outcomes.
3. Where WE should distinguish ourselves ( on Valid technology comparisons) is the ability for us to handle “Elasticity” --- The ability for us to dynamically handle failures, membership changes on instances hosting data, and Non functional requirements: Security, Scalability and Performance.

What is No-SQL?

This movement began around 2009, we have had OG/WXS since 2005-2006, so I am sure the seed are sown around the same time as emergence of In memory data grid or IMDG. So NO SQL I would say is Next Generation Databases mostly associated with terms like:

1. being non-relational,
2. distributed,
3. open-source and horizontally scalable. Etc

The original intention has been to create highly scalable distributed database systems that can handle massive amounts of data without the traditional challenges of a traditional relational database systems (things like table scans, locks etc).

Often more characteristics apply such as:

1. schema-free,
2. easy replication support,
3. simple API,
4. eventually consistent / BASE (not ACID),
5. a huge amount, of data and more…

Broadly this space is classified into the following:

In Memory data Base: This is essentially relational data, store in memory. Example SolidDB, TimesTen.

In Memory data Grid: Provides a set of interconnected java processes that holds the data in memory, thereby acting as shock absorbers to the back end databases. This not only enabled faster data access, as the data is accessed from memory, but also reduces the stress on database. WXS, Coherence, terracotta etc.

Wide Colum Store: As the name suggests DDMS that store data by columns, rather than by row. The advantage of this model is processing or computing over large data of similar items. e.g.: Hadoop, Cassandra, Amazon SimpleDB

Document Store: This is a DDMS which is designed for document oriented or semi structured data. This revolves around the notion of a document, which can be anything, like binary form of PDF, word doc etc, and encoded by an object notification system like JSON, BSON or even XML. e.g.: MongoDB, CouchDB

Key-value or Tuple: General purpose DDMS and like WXS can store data in distributed memory system. As name suggests data is stored as Key value pairs or tuples. One of the most popular DDMS systems and used by many popular names like YouTube, face book, zynga, twitter etc. e.g.; Dynamo, Azure, memchached, Berkley DB

Graph data base: This is an interesting paradigm, as it provides a storage system to store graphs. Which is index-free adjacency. This approach is faster to store associative data sets for example storing the group and its members. SO these systems typically do not require expensive joins. e.g.: InfoGrid, Big data

Object database: This is not new and has been existence since late 1970s. These are also called Object Oriented databases. These never did really take off, and in our context not really an immediate threat. e.g.: Objectivity, Gemstone

The Idea: By caching strategically at many tiers, we are trying to offload processing to various tiers and ONLY dedicating processing in middleware when it is most important or the ‘window shoppers’ now mean business, we will dedicate our cycles to those business meaning clients and service them better with an enhanced experience.

Challenge: I discussed the Design phase, the challenge is to ensure the application design that is modular enough to enable these various tiers of caching and still present a unified front, where the end user is oblivious of the inner working of the application that has its content derived from various layers. An intentional design will enable the content and business logic to be isolated, thus enabling caching at various tiers.

References:

1. http://nosql-database.org/
2. http://cassandra.apache.org/
3. http://hadoop.apache.org/
4. http://www.mongodb.org/
5. http://www.slideshare.net/harrikauhanen/nosql-3376398
6. http://couchdb.apache.org/

Embracing the Challenges of 2012 - Is it Social, OSS or Mobile? - I think it boils down to Access and Large data!

This is my first post of 2012... So Hope you all had a great Holiday season and got much needed break, rest and time off and like me energized to tackle new Challenges of 2012!

I took some time to travel to far off places and with my travels I have observed amazing trends, which I remember discussing in my previous post and it is around surge and ubiquity with mobile technologies. I have seen that in regions that we have termed as “growth Markets” like India, Africa etc, have skipped an entire generation of communication channels ( traditional mail, e-mails) and jumped straight to mobile based communication either due to:

1. Lack of access to computers, traditional phone lines (POTS) or
2. Simple access to these i.e. either at workplace or at home.

However, access to Mobile phones is rampant. For one, they are lot cheaper than traditional computers, and provide significant level of access to cyberspace. In many countries SMS based applications are rampant as not everyone can afford smart phones. Now this trend resonates with one trend we see in developed economies like the EU, US and SA --- same is true with digital natives.

Conclusion: In many ways the world is converging to this universal communication channel, be it the village scholar in Africa or a teenage student in US – They all seem to be humming the same tune.

Why is this relevant:
Businesses all around the world are and will try to capitalize on this converging platform. As this platform access is local with potential global reach.

Here is where we come in:
I was at Lotusphere 2012 last week, and the theme of the conference was “Business made Social” – The entire conference was dedicated to technologies aimed at bringing people together and harnessing the collective knowledge and decision making capability on a single collaborative platform. Lotus brand will call it communities, but the point is to harness the collective power organizational social network to address the new business models. --- This.. I think is an emerging platform where we will do business. Of course some of the trends painted was in my opinion hype and far reaching but there is certain amount of truth in productivity achieved with personal/social network.

Tying it all in:
We see a global trend and then we see new models – some adapting, some leading and yes some dying failing to adapt. So our conversation with our client – as I have always asserted needs to be relevant.

Many clients have said - first it was SOA, now it is cloud, and spotting the trend this year it will be Social-Mobile, are we just coining new fancy terms to combat the same problem? I do not know , but I certainly know that it is an evolution, virtualization came before cloud and is an important component of the Cloud. Cloud, which is not just confined to provisioning and automation, is real. Our role is greater at this time to help our clients understand the distinction and partner with them to make the right choices. I would even argue that resonating with the theme of “evolution” that certain elements and tenets of cloud computing may even be foundation for a robust Social – (and/or) Mobile foundation. And is Social – Mobile strategy different?

I would argue NOT… I would say Mobile is now a channel to be Social --- everything else is ‘supporting technology base’!

As always I will be interested in your thoughts and opinions!

So to Embrace 2012 - MY focus will be on the following areas:

1. Middleware runtime - Focusing back to basics with WebSphere runtime and base WebSphere centric application Infrastructures. This is due to intense competition from OSS.

2. Mobile Platform - I think this is relevant as it creeps into application infrastructure and amalgamates mobility platform, social network and social channels that interact with enterprise hosted application infrastructure.

Lastly,
3. Focus on core and relevant technologies that address our client’s pain points!

Have a Great weekend!


:)
Nitin

Changing Landscape of Middleware

Changing Landscape of Middleware:

Lately I have been engaged with many clients that are maniacally focused on reduced costs by means of reduced footprint. This post, I attempt to discuss some of my thoughts and experiences towards this trend:

1. Growth poses a problem: As business grows so does the resulting infrastructure and primarily the middleware which houses the business logic and in some cases this tier shares the presentation or the store front with business logic. So with growth in business either due to larger client base or new business model implies similar growth in middleware. This consistent growth poses a few challenges some obvious ones include costs – not only of software but also hosting infrastructure and hardware. Some non obvious challenges include manageability of platform i.e. general administration, handling performance and Service level agreement (SLA) and addressing scalability.

2. Consolidation – an Answer? - Many clients have taken an approach of consolidation to address this issue. Now this consolidation can come at many levels including (but not limited to) data center consolidation, IT & middleware virtualization, automation – for installation and configuration. Many clients are even as bold in claiming this effort towards the goal of being ‘cloud’ ready. However, I think that virtualization can only offer so much, at some point to achieve better resource economies clients and leaders have to think about a better design, understanding of business and user behavior and promoting design that not only appeals to the end user but also nudges the user towards a certain desired behavior.

3. Design: I have always advocated dedicating significant amount of time in design phase. While a design phase may not produce a sizable amount of tangible application artifacts, but it does enable a better design pattern for future improvements and upgrades. (More on this in later).

4. Changing Landscape of Middleware: What I am seeing and we have enabling technologies seeping into enterprise infrastructure to make this possible is the Movement of content in the outer tiers. So one way to address the scalability of middleware processing is to not let the request traverse to middleware tiers until absolutely necessary. This can be done in many ways, and one can be creative on how they accomplish this – this is where the application design comes in. Here is an example
1. Many clients push as much content to ultra edge or the public accessible content domain. For example Akamai content network. An example of this type of content is generally static content. This enables faster access to site and catalog content, and has a high user satisfaction rate. This also enables the ‘window shoppers’ from consuming your precious middleware cycles.
2. Caching at edge tier – now not all content can be cached or served from ultra edge, and some content such as domain specific content, some page fragments and jsps etc can be cached at the edge tier (which is usually behind the firewall), this does a great deal in saving processing costs of middleware presentation tier.
3. Caching at middleware tier: Patterns like side cache and in-line database cache are further instrumental in reducing resource usage such as a db lookup, db connection and in-memory access enables faster access to various type of content.

5. The Idea: By caching strategically at many tiers, we are trying to offload processing to various tiers and ONLY dedicating processing in middleware when it is most important or the ‘window shoppers’ now mean business, we will dedicate our cycles to those business meaning clients and service them better with an enhanced experience.

Challenge: I discussed the Design phase, the challenge is to ensure the application design that is modular enough to enable these various tiers of caching and still present a unified front, where the end user is oblivious of the inner working of the application that has its content derived from various layers. An intentional design will enable the content and business logic to be isolated, thus enabling caching at various tiers.

Enabling technologies:

1. Ultra Edge caching – Akamai content network
2. Edge caching – edge caching appliances ( such as IBM XI50, XC10), in memory data grid at the edge (IMDG – such as WebSphere eXtreme Scale (IBM) , Coherence (Oracle)
3. Side Cache and Inline database buffers - such as WebSphere eXtreme Scale (IBM) , Coherence (Oracle)
4. Smart Routing – IBM XI50, IBM AO, F5, Cisco etc.

Thoughts?

J

Nitin

Understanding Mobile Landscape -- Let's Dissect it! AND what is MEAP?

Last week post, I mentioned:

Mobil Space– This is an interesting space, and I think we are rightly focused on this space. From middleware perspective, I think our focus should be on including numerous mobile based services such as a location based service/Mobile payment and mobile gateway services, security.. And more. I will continue to focus on this space, as I think this is current, relevant and area where we still have time to make our mark with respect to Application infrastructure.

This week I would like to focus on Mobile Space, in this post my attempt is to dissect the mobile space and open a dialog that fosters an understanding of mobile landscape and how and where I think we fit in…. and as always I welcome your comments and thoughts:

But before I jump into dissection and deconstruction of various components of enterprise mobile landscape, here are some compelling stats that will force us to think:

1. Over last year alone 3000% of WW mobile traffic ballooned. – Includes Voice and data
2. This space is expected to grow 40X over next 5 years
3. 11% of the world owns a tablet ( now we have 89% of 6+ billion and growing)
4. 90% of the world has access to the mobile networks ( well this amazed me as 90% of world does not have access to clean water and food, but has access to mobile network)
5. 2/3 of us get most of our news on mobile devices
6. 76% take pictures and video on mobile devices
7. in past 2 years mobile ready sites have grown from 150,000 to 3 million
8. 11 billion apps in 2010
9. On commerce front – this year $6.2 billion purchase were made from Mobil device --- compare that to $54.2 billion black Friday sales
10. If mobile users would be nation, it would be largest nation….. And I could go on…

The point : Mobile space is disruptive, engaging, controversial, interactive and international all the same time. Given the reach of mobile devices and recent innovation in this space, has certainly got the attention of business leaders and entrepreneurs alike. So let’s take some time and understand various components.

1. Basics : 3 components :

Device -<-------> Network <-----------> Enterprise
OS ATT/Verizon Design/develop/emulate
Install app Content gateway Host/push updates
Updates/transactions Carrier Services enablement

If we look at core components, we as application infrastructure/middleware folks fit into Enterprise space, however, there are a few things to ponder:
a. With Tight integration between the device manufacturer and service provider there is a set of dependency and new value propositions that is fulfilled by the Network or service provider. So for instance ATT/Verizon/Vodaphone/Orange etc are in better position to provide value add services as they are closest to the clients or end user, and have a larger degree of control. It is therefore conceivable that these service providers will also be in business of mobile application hosting space and other value add services – security etc. – We have seen this with UPS/FEDEX – with our laptop/system repair and parts replacements.
b. So we can see ourselves - middleware folks/vendors working with service providers and helping enterprise to position themselves to take advantage of the new markets created.
c. And if we are innovative with our offerings we can also envision the service providers the ATT, Verizon’s of the world partner with us to better enable the enterprise reach their target audience in Ocean of new consumers and services.

2. Players and Technology : So this newly emerged market also have new and non traditional players that can potentially compete with us or provide technology and services that we have traditionally enabled.

a. Google/Motorola /Apple etc– Are not only in device making business but also provide the software, applications and services on the mobile devices.

b. ATT/Verizon/Vodaphone/Orange etc – Carrier and service provider now also into application hosting and service delivery, since they are closest to the network and client.

c. iPass, ATT Mobile solutions, Mobileron etc are relatively new players that provide the services, ‘cloud’ based hosting and access --- so this gets even interesting as many of these new services are a marriage of “cloud paradigm” and “Mobility platforms”.

d. Technology: Now few things we all should know as middleware professionals are terms like HTML5, DOJO Toolkit, REST, dojox mobile, java scripts, and optimized solutions such as the device and how things like local storage can assist with asynchronous communication with application hosted in enterprise or cloud.

e. Discerning between UI and back end technology – we as technical sellers/consultants should be able to discern between what is on the device that is UI, Type of application – native/hybrid and web based and the back end technology or engine that supports the application. This also means now our sizing, network estimates etc are not traditional conversation.

3. What is MEAP ? - MEAP stands for Mobile Enterprise Application Platform So I think it is important to understand this Device and carrier agnostic framework which is recently adopted term and quite popular on the Mobile street! I also think this framework will allow us to break ice, demonstrate understanding of mobile enterprise space, and above all help discern where we stand. From all my readings MEAP has 5 elements:

a. Defining Development Environment – Application development for enterprise and device app.
b. Connector for Integration with other services – Allows for specific data models.
c. Application Management Infrastructure – Traditional environment
d. Runtime on SmartPhone/Handheld device -
e. Gateway Infrastructure -- This sits between device and back end systems

I think, and would love to hear your thoughts that we fit in at Application Management Infrastructure and Possibly in Gateway infrastructure is we partner with the service providers.

I think this is a good start for discussion on Mobile landscape, and Please add to this discussion.

:)
Nitin

WXS competes with Oracle Coherence

The purpose of this discussion is to compare the IBM WebSphere eXtreme Scale (WXS) product with Oracle Coherence, specifically their architectural differences. While on the surface they may seem interchangeable, digging beneath to understand the fundamental architecture is required to make a better choice and decision

Basically:

I do not focus on feature gap, sooner on later we catch up and one up each other.

I focus on core architecture differences such as

a. Server to Server Communication - WKA - Well Know Address/design based on multicast which is more of an after thought. WKA disables multicast which limits scalability in Coherence architecture.

so while performance should be comparable we all access data from memory, but where we should differ due to architecture is scalability, we should scale easier and painlessly than oracle coherence... which is an important yet subtle distinction.

b. Client to Server Communication : Oracle Coherence requires gateway processes for TCP Extend to work that incurs significant CPU and network latency hops. Also this is more of an after thought.

c. I would ignore any performance claims, because at the end of the day we all access data from memory, --- If someone presents this to me, I would want details and would want to know why? What is so special that gives anyone the X perf advantage?

Coherence one up WXS with C++ integration– which WXS will soon address, however I still think that a common grid between Java/.Net and C/C++ application is utopian, and not practical.

Net-Net : WXS competes on Scalability claims and NOT Performance, Superior Architecture and feature comparison, and Thought leadership and not historical presence.

Thoughts?
:)
Nitin

Datapower Service Cache with with IBM WebSphere XC10, Usability --- aah that is not important! lets focus on release date instead!

Folks it has been a few weeks since I have shared my thoughts…that is because Retail industry is keeping us all busy to get ready for Holiday shopping season! After all the work we put in, I am not sure that many of us will have time to shop… :)

I would like to share the following thoughts, and I do look forward to your thoughts!

I. We should focus on Usability:

This week I would like to focus on Usability, and I have always admired Apple products and as I read on the development of many of Apple products such as iMac, iPod etc they all have one thing in common and that is ease of use… and the results are evident that these products are not only technically sound but have viral adoption pattern…world loves it, and will pay a premium for it! So why not adopt the same design principles in our own products!

The most common complaint that our clients have is that what ever feature they desire it is always slated for next release…. To me that is not leadership! I am compelled to write about many products that would look like if it were to adopt “Steve Jobs” driven – Design principles and product development lifecycle. And while the development lifecycle make take longer ( and hence the cost), a better design speak volume of products, thought leadership and not to mention the savings in support costs which is far greater that initial investment in design.

So I question, and would love you feedback… why are release deadlines so important? That we are willing to release a half baked product and willing to compromise the quality for fixes in future date?

II. Topic in Focus: Datapower Service Cache with with IBM WebSphere XC10

The XC10 and XI52 Integration are compelling. The idea behind the XI52-XC10 integration is to enable faster lookup of cached data and reduced processing at backend tiers, with use of XC10 as a general purpose caching appliance. The XI52-XC10 integration has opened up many possibilities to economize on infrastructure costs and further leverage XC10s caching mechanism to speed up XI52 Authentication process. In simplest terms this integration amalgamates the better of two breeds of appliances. This integration speeds the lookup and processing done by XI50, by extensible and scalable cache enabled by XC10.

Main Points to Note:

1. Address Scalability Challenge - core of XC10 business value.
2. By off-loading the storage/look-up and retrieval of XML documents in XC10, we are also offloading the processing to a tier parallel to XI52/XS40, this does two things:
   -Speeds up the response -- hence improves performance.
   -Prevents requests going to back end 'middleware' tiers -- addressing scalability.
3. XC10 can be scaled as need for data grows - classic WXS/XC10 (collective) feature for Scalability proposition.

The idea behind the XI50-XC10 integration is to enable faster lookup of cached data and reduced processing at backend tiers, with use of XC10 as a general purpose caching appliance.

Have a great thanksgiving weekend!

:)
Nitin

Why Innovate?

So I have been busy and pondering over this questions on Why Innovate?
We have many large corporations that have Innovation somewhere in their values or catchy Slogan. However, true Innovation comes from -- I think, necessity , one has to foster the culture of innovation by allowing failures, and that marks the cornerstone of an 'experienced' and progressive set of values.
I want to explore this with a n example... Amazon once a largest bookseller, came up with with EC2 or Amazon cloud, while technologies powerhouses are still struggling to define themselves in the 'cloud play'.. why is that a book company was able to envision a computing platform while selling books and organizations whose core business was computing could never lead with it? -- Vision.

I think It all boils down to vision and execution... and yes I think failures goes along the way... but what a vision brings is amazing things like iPOD, iPAD and Amazon which is hosting many things from applications to platform for netflix to stream movies.
I still think that with middleware in cloud, the industry is focused on provisioning and automation, and missing the bigger picture.. where there is no focus on utility model, or application design that includes amazing capabilities to linearly scale...speedier access by employing caching technologies, smarter awareness of me the customer --- I do not think we need a lot of money or investment into these insights, we just need the will and reason to innovate!!

Thoughts?

:)
Nitin