Recently I have been heavily using Apache Spark. For those of you who don’t know Spark is a very powerful system for working with data and parallel that is written in the Scala language. Scala is not new, but certainly on the “newer” end of the spectrum. Today new languages are coming out all the time so 12 years is fairly long. What many people find attractive about Scala, at least I do is the fact that it runs on the popular JVM. In fact a developer may be able to write code in Java and interact with code that was written in Scala. The challenge is striking the balance between closeness to Java and still providing whatever it is that the creators of the language hope to achieve.
I am a big believer in Object Oriented software design and development. I’m not saying that every project in the world needs to be written exactly the same, as they all have different requirements. I will say that for enterprise software the level of adaptability is truly best achieved with levels of modularity as well as abstraction. In truth, if you can achieve the principles that an enterprise architect looks at in a technology it may be something to be considered. In the past languages like LISP such as Haskell, were never geared towards the enterprise as their mathematical background and dynamic typing didn’t fit the bill for the type of type safety compilation and code reuse that has been found in other enterprise technologies. In general I like languages that aim for simplicity but at the same time aren’t overly opinonated. I recently read a rant about the Go language for the lack of support of assertions because the language creators felt that people used assertions incorrectly, not like Java chose to avoid pointers because of their inherit nature to cause errors. This is an example of a language being dumbed down or muted, with that said I really like many aspects of Go.
I’ve read a number of articles about so called “veteran” developers who have ditched OOP to embrace some sort of functional language. Complaining that the design principles of OOP aren’t applicable and don’t work. I even read recently that a college professor from Carnegie Mellon removed OOP from the syllabus for freshman. I don’t necessarily think that is too awful, but I do think problem decomposition that one would do when designing an OOP system is not only very helpful but also useful. Not everything easily fits into a “map” or a “reduce”. I can’t speak for all people, but I think that OOP is more natural to the domain than functional. If you truly understand the domain and how to break apart a problem into single scoped entities you will find simplicity and elegance.
With that being said, I think that for parallelism functional programming has always been faster and more efficient. I do however think that there is room to bridge the gap. Scala is a multi-paradigm language, not just functional. I believe it is that aspect that can truly bring something special to the table. Technologies like Spark still require a more or less functional approach. There are layers like Dataframes and graphs that attempt to abstract some of the functional aspects from the developer. What I haven’t yet seen is what Hibernate and other ORM technologies did for SQL, with respect to large scale functional parallelism. I think once we bridge that gap that will be the holy grail for enterprise software. I look forward to seeing how these technologies continue to evolve and mature.
Right now AWS is the leader in cloud-computing, without a question. With DELL’s recent acquisition of EMC, which happens to own roughly 80% of the eminent virtualization company Vmware one can imagine they have their sights set on competing for a piece of the action as well. AWS is so popular that I recently saw companies such a Rackspace who used to be a real competitor to AWS now offers premier managed hosting of AWS. That is the kind of smart attitude shift that companies such as Microsoft have started doing. Now under the tutelage of Satya Nadella Microsoft has been making many wise moves all recognizing that they need to play nicely with other companies and recognize that big bad Redmond isn’t the only company in the ecosystem anymore. With that said most companies I speak to still treat AWS as being synonymous to the cloud, or at least the defacto cloud provider solution. There isn’t anything wrong with that, as long as you recognize what that really means.
Back in the day where companies such as Oracle…actually innovated and dominated the large scale database space. In the realm of commercial databases Oracle is still a big player, of course technologies rooted in open source are becoming more and more common place. Back in the day before ORM’s and when there was no buzz word called the “cloud”, the database was a huge monsterious construct that dominated the stack. Nothing was distributed and hardly clustered. Databases had single masters with some read-only slaves that had stale data. The single master was a bottleneck for writes, you hoped that it was enough because there weren’t the same options that exist today. Lot’s of the database providers would offer their own solutions that were specific to their API. Your DBA and Architects would caution you against vendor lock-in. They would tell you if you implement your system around vendor specific features you will find yourselves unable to break free from their grasp. Now I have always said that you should take advantage of vendor specific features as long as you architect them in a way such that they are more or less lift and shift binding a standard interface with a new vendor implementation so that you can now take advantage of their approach without sacrificing the modularity of your code-base.
This notion of vendor lock-in is not only applicable to databases but in my mind even more important with hosting solutions such as AWS. AWS isn’t a software company, they are a service company. A handful of the services that they offer are actually born and bred in-house, rather most of their services are open source technologies that they have adapted tried and true practices and software solutions and mixed in their automation and high availability in. AWS is different than any other hosting company before it because they provide specialized solutions not just raw hosting. Other companies can and will compete and offer similar if not better products. I anticipate that you will start seeing a great more in specialization. Companies like IBM, who are fairly minimal in their large scale utilization. They do however have a very impressive suite of API’s geared towards machine learning (https://www.ibm.com/watson/developercloud/services-catalog.html) both Google and AWS have some minimal machine learning solution but nothing that compares to the versatile toolkit Watson offers. I’m not telling you that IBM is reliable, performant, or any good at all. I am saying that in addition to their boring Bluemix hosting they are being innovative with their services.
The bottom line is remember that AWS is only a single vendor. Just like all markets, their will be growth and competition. S3 may be a fairly standard key-value system but their API and domain level approaches may be very different from the next leader in the cloud industry. Learn from experience, invest your time in designing your systems to be capable of switching cloud partners without a complete rewrite. It’s okay to use vendor-specific features, as long as you develop them in a way that you can easily accommodate a change.