These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.11 Jul 2017
I see a lot of tools come across my desk each week, and I have to be honest I don’t alway fully get what they are and what they do. There are many reasons why I overlook interesting applications, but the most common reason is because I’m too busy and do not have the time to fully play with a solution. One application I’ve been keeping an eye on as part of my work is Airtable, which I have to be honest, I didn’t get what they were doing, or really I just didn’t notice because I was too busy.
Airtable is part spreadsheet, part database, that operates as a simple, easy to use web application, which with a push of a button, you can publish an API from. You don’t just get an API by default with each Airtable, you get a pretty robust developer portal for your API complete with good looking API documentation. Allowing you to go from an Airtable (spreadsheet / database) to API and documentation–no coding necessary. Trust me. Try it out, anyone can create an Airtable and publish an API that any developer can visit and quickly understand what is going on.
As a developer, API deployment still feels like it can be a lot of work. Then, once I take off my programmers hat, and put on my business user hat, I see that there are some very easy to use solutions like Airtable available to me. Knowing how to code is almost slowing me down when it comes API deployment. Sure, the APIs that Airtable publishes aren’t the perfectly designed, artisanally crafted API I make with my bare hands, but they work just as well as mine. Most importantly, they get business done. No coding necessary. Something that anyone can do without the burden of programming.
Airtable provides me another solution that I can recommend that my readers and clients should consider using when managing their data, which will also allow them to easily deploy an API for developers to build applications against.I also notice that Airtable has a whole API integration part of their platform, which allows you to integrate your Airtables into other APIs–something I will have to write about separately in a future post. I just wanted to make sure and take the time to properly add Airtable to my research, and write a story about them so that they are in my brain, available for recall when people are asking me for easy to use solutions that will help them deploy an API.
I am finally getting the time to invest more into the rest of my API industry guides, which involves deep dives into core areas of my research like API definitions, design, and now deployment. The outline for my API deployment research has begun to come into focus and looks like it will rival my API management research in size.
With this release, I am looking to help onboard some of my less technical readers with API deployment. Not the technical details, but the big picture, so I wanted to start with some simple questions, to help prime the discussion around API development.
- Where? - Where are APIs being deployed. On-premise, and in the clouds. Traditional website hosting, and even containerized and serverless API deployment.
- How? - What technologies are being used to deploy APIs? From using spreadsheets, document and file stores, or the central database. Also thinking smaller with microservices, containes, and serverless.
- Who? - Who will be doing the deployment? Of course, IT and developers groups will be leading the charge, but increasingly business users are leveraging new solutions to play a significant role in how APIs are deployed.
The Role Of API Definitions While not every deployment will be auto-generated using an API definition like OpenAPI, API definitions are increasingly playing a lead role as the contract that doesn’t just deploy an API, but sets the stage for API documentation, testing, monitoring, and a number of other stops along the API lifecycle. I want to make sure to point out in my API deployment research that API definitions aren’t just overlapping with deploying APIs, they are essential to connect API deployments with the rest of the API lifecycle.
Using Open Source Frameworks Early on in this research guide I am focusing on the most common way for developers to deploy an API, using an open source API framework. This is how I deploy my APIs, and there are an increasing number of open source API frameworks available out there, in a variety of programming languages. In this round I am taking the time to highlight at least six separate frameworks in the top programming languages where I am seeing sustained deployment of APIs using a framework. I don’t take a stance on any single API framework, but I do keep an eye on which ones are still active, and enjoying usag bey developers.
Deployment In The Cloud After frameworks, I am making sure to highlight some of the leading approaches to deploying APIs in the cloud, going beyond just a server and framework, and leveraging the next generation of API deployment service providers. I want to make sure that both developers and business users know that there are a growing number of service providers who are willing to assist with deployment, and with some of them, no coding is even necessary. While I still like hand-rolling my APIs using my peferred framework, when it comes to some simpler, more utility APIs, I prefer offloading the heavy lifting to a cloud service, and save me the time getting my hands dirty.
Essential Ingredients for Deployment Whether in the cloud, on-premise, or even on device and even the network, there are some essential ingredients to deploying APIs. In my API deployment guide I wanted to make sure and spend some time focusing on the essential ingredients every API provider will have to think about.
-Compute - The base ingredient for any API, providing the compute under the hood. Whether its baremetal, cloud instances, or serverless, you will need a consistent compute strategy to deploy APIs at any scale. -Storage - Next, I want to make sure my readers are thinking about a comprehensive storage strategy that spans all API operations, and hopefully multiple locations and providers. -DNS - Then I spend some time focusing on the frontline of API deployment–DNS. In todays online environment DNS is more than just addressing for APIs, it is also security. -Encryption - I also make sure encryption is baked in to all API deployment by default in both transit, and storage.
Some Of The Motivations Behind Deploying APIs In previous API deployment guides I usually just listed the services, tools, and other resources I had been aggregating as part of my monitoring of the API space. Slowly I have begun to organize these into a variety of buckets that help speak to many of the motivations I encounter when it comes to deploying APIs. While not a perfect way to look at API deployment, it helps me thinking about the many reasons people are deploying APIs, and craft a narrative, and provide a guide for others to follow, that is potentially aligned with their own motivations.
- Geographic - Thinking about the increasing pressure to deploy APIs in specific geographic regions, leveraging the expansion of the leading cloud providers.
- Virtualization - Considering the fact that not all APIs are meant for production and there is a lot to be learned when it comes to mocking and virtualizing APIs.
- Data - Looking at the simplest of Create, Read, Update, and Delete (CRUD) APIs, and how data is being made more accessible by deploying APIs.
- Database - Also looking at how APIs are beign deployed from relational, noSQL, and other data sources–providing the most common way for APIs to be deployed.
- Spreadsheet - I wanted to make sure and not overlook the ability to deploy APIs directly from a spreadsheet making APIs are within reach of business users.
- Search - Looking at how document and content stores are being indexed and made searchable, browsable, and accessible using APIs.
- Scraping - Another often overlooked way of deploying an API, from the scraped content of other sites–an approach that is alive and well.
- Proxy - Evolving beyond early gateways, using a proxy is still a valid way to deploy an API from existing services.
- Rogue - I also wanted to think more about some of the rogue API deployments I’ve seen out there, where passionate developers reverse engineer mobile apps to deploy a rogue API.
- Microservices - Microservices has provided an interesting motivation for deploying APIs–one that potentially can provide small, very useful and focused API deployments.
- Containers - One of the evolutions in compute that has helped drive the microservices conversation is the containerization of everything, something that compliments the world of APis very well.
- Serverless - Augmenting the microservices and container conversation, serverless is motivating many to think differently about how APIs are being deployed.
- Real Time - Thinking briefly about real time approaches to APIs, something I will be expanding on in future releases, and thinking more about HTTP/2 and evented approaches to API deployment.
- Devices - Considering how APis are beign deployed on device, when it comes to Internet of Things, industrial deployments, as well as even at the network level.
- Marketplaces - Thinking about the role API marketplaces like Mashape (now RapidAPI) play in the decision to deploy APIs, and how other cloud providers like AWS, Google, and Azure will play in this discussion.
- Webhooks - Thinking of API deployment as a two way street. Adding webhooks into the discussion and making sure we are thinking about how webhooks can alleviate the load on APIs, and push data and content to external locations.
- Orchestration - Considering the impact of continous integration and deployment on API deploy specifically, and looking at it through the lens of the API lifecycle.
I feel like API deployment is still all over the place. The mandate for API management was much better articulated by API service providers like Mashery, 3Scale, and Apigee. Nobody has taken the lead when it came to API deployment. Service providers like DreamFactory and Restlet have kicked ass when it comes to not just API management, but making sure API deployment was also part of the puzzle. Newer API service providers like Tyk are also pusing the envelope, but I still don’t have the number of API deployment providers I’d like, when it comes to referring my readers. It isn’t a coincidence that DreamFactory, Restlet, and Tyk are API Evangelist partners, it is because they have the services I want to be able to recommend to my readers.
This is the first time I have felt like my API deployment research has been in any sort of focus. I carved this layer of my research of my API management research some years ago, but I really couldn’t articulate it very well beyond just open source frameworks, and the emerging cloud service providers. After I publish this edition of my API deployment guide I’m going to spend some time in the 17 areas of my research listed above. All these areas are heavily focused on API deployment, but I also think they are all worth looking at individually, so that I can better understand where they also intersect with other areas like management, testing, monitoring, security, and other stops along the API lifecycle.
I saw the news that Google’s Spanner Database is ready for prime time, and I wanted to connect it with a note I took at the Google Analyst Summit a few months back–that gRPC is the heart of the database solution. I’m not intimate with the Spanner architecture, approach, or codebase yet, but the API focus, both gRPC core, and REST APIs for a database platform are very interesting.
My first programming job was in 1987, developing COBOL databases. I’ve watched the database world evolve, contributing to my interest in APIs, and I have to say Google Spanner isn’t something I anticipated. Databases have always been where you start deploying an API, but Spanner feels like something new, where the database and the API are one, and the way the database does everything internally and externally is done via APIs (gRPC).
Now that Spanner Database is ready for prime time, I will invest some more time in standing up an instance of it and get to work playing with what is possible with the REST APIs. I also want to push forward my grPC education by hacking on this side of the database’s interface. Spanner feels like a pretty seismic shift in how we do APIs, and how we do them at scale–when you combine this with the elasticity of the cloud, and the simplicity of RESTful interfaces I think there is a lot of potential.
I'm in the middle of evolving a data schema to be a living breathing API. I just finished generating 130 paths, all with the same names as the schema tables and their fields. It's a natural beginning to any data-centric API. In these situations, it is easy for us to allow the backend system to dictate our approach to API design, rather than considering how the API will actually be used.
I'm taking the Human Service Data Specification (HSDS) schema, and generating the 130 create, read, update, and delete (CRUD) API paths I need for the API. This allows the organizations, location, services, and other details being managed as part of any human service API that will be managed in a very database-driven way. This makes sense to my database administrator brain, but as I sit in a room full of implementors I'm reminded that none of this matters if it isn't serving an actual business objective.
If my API endpoints don't allow a help desk technician properly search for a service, or a website user browse the possibilities to find what they are looking for, my API means nothing. The CRUD is the easy part. Understanding the many different ways my API paths will (or won't) help someone find the services they need or assist a human service organization to better reach their audience is what the API(s) are all about, not just simply reflecting the backend system, and then walking away calling the job done.
I wrote a post the other day sharing my thoughts around GraphQL seeming like we were avoiding the hard work of API design. Shortly after publishing Sashko Stubailo (@stubailo) from Apollo, a GraphQL solution provider, wrote a very thoughtful response to me comments and questions about GraphQL. First I wanted to say that I really dig this approach to responding to other people's blog posts, with a blog post of your own, within your own personal or company domain.
I don't think Sashko has convinced me 100% that GraphQL is the solution we are looking for, but he has convinced me that I should be learning more about it, keeping a closer eye on the technology, and better understand how people are putting it to use.
Regarding my primary question regarding how GraphQL could benefit non-technical folks and end-users--I would say he answered it 50% of the way:
I’m a frontend developer. GraphQL makes my life easy.
It doesn't touch on whether or not non-technical users will be able to reverse engineer and put it to work for them, but that's ok for now. One thing that Sashko touched on for me, that move GraphQL closer to being simple enough for non-technical users, is he helped differentiate GraphQL from SQL:
GraphQL is a query language, like SQL? While GraphQL looks like a query language at first, I think its name might be one of the things that gets people off on the wrong foot. GraphQL is not at all like SQL...So GraphQL is a “query language” just like URLs are the “query language” of REST—it’s a contract that describes how to tell the API server what you’re looking for.
I like the separation from "structured" query language, and moving us to something that augments HTTP and the URL, and doesn't just tunnel into the backend database. This has the potential to move REST forward, not drag the database out front for me--which leaves me more hopeful.
Another area Sashko answered for me was regarding GraphQL seeming like it was too hard:
This is a very real concern whenever a new technology is introduced. Is this going to make stuff more complicated for everyone who isn’t in the know?
Fair enough. Makes me happy to hear this from a service provider who is leading the charge when it comes to GraphQL. His stance seems like it is pragmatic, and aware of the importance that GraphQL needs to be accessible to as wide as possible audience as we can--building on the momentum REST has in this area.
What really pushed away my concern, and got me more interested in paying more attention to GraphQL was when Sashko talked about this just being the beginning:
The most exciting thing to me is that GraphQL has been publicly available for barely more than a year, and already a huge number of people I respect for their technical abilities and design thinking are trying to figure out ways to add it to their architecture.
Ok. Now I want to see where this goes. I've been tracking on GraphQL as a subset of my data API research, but will spend some extra cycles each week keeping an eye who is doing anything interesting with GraphQL. I've added Apollo to my research, and I will work on a roundup of other providers, and any open source tooling I can find out there. I also wanted to thank Sashko for taking the time to answer some of my questions, and respond to my uncertainty around GraphQL. I dig it when API service providers and API providers provide responses to my storytelling on API Evangelist--makes for good conversation in the API community.
Create a PHP Script To Generate An OpenAPI Specification From The ClinicalTrials.Gov Database I Created07 Feb 2016
One of my objectives around importing the ClinicalTrials.gov data, is to create an API. The first step in creating an API, before we ever get programming anything, is to create an OpenAPI Spec for use as a version 1.0 scaffolding for the clinical trials data we now have stored in a MySQL database.
I sure wasn't going to be hand crafting an OpenAPI Spec for this fairly large data set, so I got to work creating a crude PHP script that would do the heavy lifting for me:
This script loops through all the tables in my clinical trials database, and auto generates the necessary JSON schema for the data structure, combined with OpenAPI Spec for describing the API interface for the clinical trials database. I have the result in a single OpenAPI Spec file, but will most likely be breaking up to make it easier to work with:
This OpenAPI Spec for the clinical trials API gives me a base blueprint I can use to generate server side code, client side code, documentation, and other essential building blocks of the API operations I put in place to support accessing the clinical trials API.
I will be added better descriptions for paths, parameters, schema, and other elements of this in the future, with this definition acting as the contract for the clinical trials API.
I came across the Privacy Rights Clearinghouse, while conducting a search that turned up the chronology of data breaches, which provides details on 4,725 data breaches that have been made public since 2005. The website allows you to search for data breaches by type of breach, type of organization, and the year in which it occurred--some very valuable information.
In 2016, as breaches continue to be common place across almost all industries, we are going to need to take the chronology of data breaches up a notch. I would like to see an API be made available for the valuable database. As I do, I write stories about what I'd like to see in the space, and forward the link to key actors, and tell the story to the public at large, in hopes of inciting something to happen.
Making the data breach information available via API, would encourage more storytelling around the events, which could include much more meaningful visualizations using solutions like D3.js. Information about companies, could be included into other business search and discovery tooling, and more push notification networks could be setup that could keep industry experts more informed about what is happening across the sector.
Now that I am on the subject, it would make sense if all the privacy topics, and other resources available via the Privacy Rights Clearinghouse accessible through a simple API interface. If you work with the Privacy Rights Clearinghouse, and would like to talk about making this happen, feel free to reach out. If you are someone who would like to help work on this project, or possibly fund this work, also please let me know.
The type of information the Privacy Rights Clearinghouse is aggregating is only going to become more important in the insecure cyber world we have created, and making it accessible for reading and writing via a simple API, would significantly help the organization make a bigger impact, and educate a larger audiences.
I had a reinder on my task list to check-in on where some of the common database platforms were when it came to APIs. I think it was a Postgres announcement from a while back that put the thought in my notebook, but as an old database guys I tend to check-in regularly on the platforms I have worked most with.
The point of this check-in, is to see how far along each of the database platforms are when it comes to easy API deployment, directly from tables. The three relational database platforms I'm most familiar with are:
- SQL Server - The platform has APIs to manage, and you deploy an OData service, as well as put .NET to work, but nothing really straightward, that would allow any developer to quickly expose simple RESTful API.
- PostgreSQL - I'd say PostgreSQL is furthest along with thier "early draft proposal of an extension to PostgreSQL allowing clients to access the database using HTTP", as they have the most complete information about how to deploy an API.
- MySQL - There was a writeup in InfoQ about MySQL offering a REST API, but from what I can tell it is still in MySQL Labs, without much movement or other stories I could find to show any next steps.
The database that drives my API platform is MySQL running via Amazon RDS. I haven't worked on Postgres for years, and jumped ship on SQL Server a while back (my therapist says I cannot talk about it). I automate the generation of my APIs using Swagger and the Slim framework, then do the finish work like polishing the endpoints to look less like their underlying database, and more like how they will actually be used.
Maybe database platforms shouldn't get into the API game? Leaving API deployment to the API gateway providers like SlashDB and DreamFactory. It just seems like really low hanging fruit for these already used database solutions, to make it dead simple for developers to expose, and craft APIs from existing datasources.
if you are using any database to API solutions for SQL Server, PostgreSQL, or MySQL, please let me know.
I referenced the Recreation Information Database (RIDB), in my story late last year, when I was asking for your help to make sure the Department of Agriculture leads with APIs in their parks and recreation RFP. I'm not exactly sure where it fits in with the RFP, because the RIDB spans multiple agencies.
Here is the description from the RIDB site:
RIDB is a part of the Recreation One Stop (Rec1Stop) project, initiated as a result of a government modernization study conducted in 2004. Rec1Stop provides a user-friendly, web-based resource to citizens, offering a single point of access to information about recreational opportunities nationwide. The web site represents an authoritative source of information and services for millions of visitors to federal lands, historic sites, museums, and other attractions/resources.
When I wrote the post last October, I referenced the PDF for the REST API Requirements for the US Forest Service Recreation Information Database (RIDB), but this week I got an update, complete with fresh links to a preview of a pretty well designed API, complete with documentation developed using Slate.
I haven’t actually hacked on the endpoints, but I took a stroll through the docs, and my first impression was it is well designed, and robust, including resources for organizations involved with the RDB, recreational areas, facilities, campsites, permit entrances, tours, activities, events, media, and links. The RIDB documentation also includes details on errors, pagination, version, and a data dictionary, and the on-boarding was frictionless when looking to get a key.
Everything is in dev mode with the RIDB API, and the team is looking for feedback. I’m not entirely sure they wanted me to publish as story on API Evangelist, but I figure the more people involved the better, as I’m not sure when I’ll get around to actually hacking on the API. I’m happy to see such a quick iteration towards the next generation of the RIDB API, and it makes me hopeful to be covering so many new API efforts out of the federal government in a single day.
The home page of API Evangelist has always been my API 101 page, where any new visitor can land, not know a thing about APIs, read the page, and walk away understanding at least what an API is, and hopefully also a greater understanding of how it impacts their world. From the beginning the API Evangelist home page always had a lean toward API providers, and over time I've added information about consuming APIs, as well as trends in the API space I'm seeing.
In my opinion my 101 content has become too lengthy, and not properly onboarding people to the multiple areas of APIs like providing APIs vs consuming APIs. I think it is time to overhaul my 101 section, and produce some newer, more relevant API 101 content for 2014. To support this, I'm working on a series of 15 API 101 segments, with each one having a story version, as well as a slideshow edition, complete with audio.
I finished the first one, which is my 100K view, API 101 introduction. I'm still working on the slideshow version. Once I'm done I"ll move on to providing APIs, and as I finish each one I'll hang on the homepage, as well as publish out via the blog, and Twitterz.
APIs, also known as Application Programming Interfaces, at their most basic level, allows applications to talk to other applications, but they are so much more than this when you begin to explore the world of APIs further. APIs are slowly becoming part of every aspect of our personal, and business worlds, often even without us knowing. If you surf the web, and use a mobile phone, this explanation of what an API is for you. APIs are affecting your life right now, and with a little more understanding you can learn to identify APIs, and put them to work for you, even if you are not a geek.
There are 861,379,000 websites registered in 2014, making up what we know as the World Wide Web. In 2014 it is easier to build a web than it has at any other pointing in the history of the Internet, allowing any individual, company or organization to publish a website and even point a custom address to it.
In 1995 only the most tech savviest had websites, and by 2000 many businesses realized having a website was the normal way of doing business. In 2010 many of the tech savviest companies had APIs, and by 2015 many businesses are realizing that having an API is the normal way of doing business--imagine what the Internet will look like by 2020.
Websites like Facebook allow us to interact with friends and family, sharing messagings, photos, links, and other essential life bits, establishing a social platform for us to engage in our online lives.
Facebook is a daily destination for users around the world, with much of the interactions via an increasing number of mobile devices that all of us have in our pockets. These mobile devices all use APIs to connect our mobile app to Facebook.com.
Yahoo has long been a staple of the Internet providing users with access to their news, sport, weather, stock, and other important bits of our daily lives. Yahoo was an early player on the web, and continues to find new ways to compete in 2014 via the web and mobile.
Yahoo may not be what it used to be, but the company continues to innovate through acquisitions, and still plays a role as the home page for many users who were first introduced to the World Wide Web through Yahoo search and directories.
YouTube provides us with hours of online entertainment, bringing home user-generated, and sometimes professionally produced videos, ranging from family home videos to music videos, and live concerts. YouTube is quickly replacing the television of the previous century, re-inventing how we are entertained in our homes.
YouTube users often spend time watching videos on the main YouTube website, but also consume a considerable amount of videos using the universal embeddable video player allowing Youtube videos to be embedded on any website around the web, all driven by APIs.
Amazon has made e-commerce part of the mainstream conversation, allowing us to order almost any physical, or digital product we can imagine from the vast warehouses that Amazon posses, as well as those of its affiliates. Amazon found got started with physical products like books, but in 2014 has led the charge in delivering virtual goods such as movies, e-books, and other online products consumers are craving.
Selling you products is just the first part of the conversation when it comes to Amazon, they also want to provide with consumer any possible channel they could possibly choose to purchase their products, including providing infrastructure for web, mobile, and tablet developers, and even producing their own tablet device, and supporting platform.
Twitter allows us to send short, SMS style messages across the public Internet, and follow, or be followed by the people we find ost interesting. Twitter has gone beyond just a messaging platform, or even a social network, and has become the pulse of the online world through its API driven platform.
Everything you know about Twitter was originally developed using the Twitter API, its mobile applications, desktop applications, buttons, widgets, and other analytics and visualizations are all developed using the Twitter public API, which makes the entire Twitter platform available to developers.
Pinterest provides users with a simple, yet addictive way to share photos on the open Internet, and with friends. Photos play an important role in our lives, and Pinterest stepped up as a new way to express ourselves using these valuable photos and images.
Only part of the interactions with Pinterest occur via their website, the majority of engagements occur through sharing on other sites and social networks, and via mobile applications that allow users to quickly share and express themselves with photos.
Reddit is the home page of the Internet, providing users with links to the important stories, products, and conversations of the day. Redditors curate the best links that are submitted through the site, and through user engagements the best of the best floats to the top for everyone to explore.
Reddit demonstrates the real-time network effect that we often associate with the Internet, providing a heartbeat of the best links being viewed, and engaged with across the Internet, making the trillions of web pages, across billions of websites accessible each day by average people.
Bits Of Our Life Across These Sites
All of these websites are built by other people, but as users of these sites we all leave a little bit of ourselves in these locations, each time we visit. Whether its just a comment, photo, all the way up to sustained engagement via our mobile phones, we are consistently leaving ever growing footprint across this digital landscape of web sites, and mobile applications.
It has taken decades to go from just a handful of the first sites on the World Wide Web to the 861,379,000 we have in 2014. It is taken the hard work of web site designers and developers to make these sies a reality, but increasingly through the use of API driven interactions via 3rd party websites and mobile devices, all the information on these sites are being generated by YOU, the user.
Hand Written Websites Using HTML
Iin the early days, all websites were hand coded, with web designers writing all the HTML, and building their own images. Individuals spent numerous hours maintaining each page across vast sites, and often hired large teams help maintain single sites that could span millions of actual web pages.
As the number of websites grew this became even more tedious to maintain, but savvy designers turned developers were finding innovative ways to create, publish, and maintain the growing number of websites. While some websites are still hand-coded, the majority of sites are generated, published, and maintained by an ever evolving set of web development software, and Software as a Service (SaaS) platforms.
Website Content Was Coming From Databases
Eventually the concept of a database driven website came to be, allowing websites to evolve from sites into applications, and all the content across a website was stored in a database. Now your site was more than just publishing content from a database, you also allowed users to interact, purchase, and generate content as well.
Databases allowed websites to evolve, and go beyond just content publishing, potentially making their websites not just about the delivery of valuable content, but you could also encourage users to create content either knowingly or unknowingly, and not just on a single website, but across hundreds or thousands of sites around the world.
Adding An Amazon Product Widget
Along the way it became clear that it wasn't enough to have just your products on your website, or maybe you were a content site and didn't have your own products. Either way, users needed a way to put widgets, and other syndicated content on their site, which would pull products from other sites, which is something Amazon became very, very good at.
API driven content widgets became a great way to publish products onto the growing number of blogs, review, and other sites that were popping up across the Internet. These approaches to content delivery proved to work not just for product owners, but eventually any content could be syndicated across websites using simple, small, embeddable widgets, benefitting an entire network of connected sites.
Add A Delicious Bookmark Widget
As the social evolution of the Internet began, links to valuable content, and the social influence behind these curated groups of links became apparent. It wasn't just about bookmarking your favorite sites, it was about grouping, curated, and sharing of this content across any website that would have you, using embeddable tools.
Curated links often show as spammy advertised links, but can also show relevant links based upon what you are searching for, what your friends are doing, and in other ways, that makes the embedding of links, or groups of links, an important way to share information across the Internet.
Add Twitter Widget
With the social movement in full swing, just posting links to your favorite sites was not enough. Users wanted to be able to communicate in real-time about the sites they were visiting, what they were reading, watching online, and increasingly what people were doing offline. With the introduction of Twitter users wanted to be able to show what what on their mind, not matter where website it was on the internet.
Widgets were now allowing every piece of your life that was generated online to live beyond just the website where you created it, and could travel around from any website, to any other site or application that it could. Embeddable content and tools elevated the World Wide Web beyond just any single site, it was a distributed Internet now.
Content Comes From Many Databases & APIs
If you were a website developer, your website content no longer comes from any single database. You were pulling your content from a variety of databases, and API resources that you may or may not own. Product sites like Amazon, and social networks like Twitter were emerging, equipped with APIs they were generating content, and allowing it to come in from, and be published back to any website on the open Internet.
Having a direct database connection to your content wasn't enough anymore. You needed to have APIs so that trusted partners, as well as potentially the public, can get access to data, content and other information, for use across other websites, anywhere on the Internet. Just as this concept was spreading across the Internet, another thing happened, that would forever change how we interact online.
Then The IPhone Happened
On June 29, 2007 the first edition of the iPhone was release, and slowly over the next 3 years computing would undergo a transformation, building on existing approaches that first were used to distribute content to website, but would also prove to be low cost, efficient ways to distribute content to a growing number of mobile devices.
While the iPhone is just one of many mobile devices being manufactured today, it did set in motion a new way of designing, and developing applications that could run on the web, or mobile devices, anywhere in the world using simple web-based APIs. Mobile application development has brought many changes to the world of compute, but simple API access to valuable data, content, and other resources has been the most significant.
Mobile applications came with the simple knowledge of your location, as well as where you were at online, enabling a new form of communication, called the check-in. A new breed of mobile apps like Foursquare allowed users to see where they were, find friends, as well as places and experiences nearby, making the Internet a more local affair.
From this point forward, the longitude and latitude of each online user would be an option when receiving messages, photos, videos and other life bits from users, changing how we think about developing applications--it was a mobile app world now.
In addition to knowing the location of each application user, each user was potentially given a high quality, internet enable camera on their mobile device. With the ability to take a picture, anywhere, anytime, new applications like Instagram emerged providing users with new features, and social capabilities built around photos.
Photos continue to be one of the most important, and valued life bits that we can generate across the Internet pipes, providing a very basic ingredient for use in online communication. This is something that would prove essential to users of all ages, from teens all the way up to grandparents discovering the potential of web and mobile apps.
As with photos, music plays a central role in our life, and it is no accident that along with the introduction of the iPhone, Apple brought along its successful model for a music platform centered around their portable music player, the iPod. Music is an extremely valuable space when it comes to our life bits, something developers and platforms are looking to capitalize on by delivering the next generation of music apps.
APIs are stimulating our musical tastes beyond just our Pandora or Rdio playlists, by allowing us to interact with friends, watch videos, purchase concert tickets, buy merchandise, and much, much more. The music industry has been slow to change, but ultimately consumer interests are winning out, and the industry is slowly understanding that a mobile experiences is where today's music experience is at.
In an online, mobile world, staying in touch is one of the most important elements of our digital self. Staying in touch with friends, co-workers, and the ability to organize into groups and meet for sport practice, or to protest the government, is the focus of leading messaging apps like WhatsApp.
Companies like WhatsApp are seeing record investment, valuations, and usage by users around the world--showing that messaging online is an important part of the online and mobile experience, and building apps that connect users in real-time, when they they want, and how they want, is a huge opportunity for start-ups.
APIs Are Powering Websites
APIs have been powering websites for almost 15 years now, delivering data, content, and other digital information to the websites we use everyday like Amazon, Twitter, and Pinterest. The average website gets its content from a variety of API driven resources, both from public and private sources, making the aggregation of content a viable business model in 2014.
Using content management systems (CMS) like WordPress, combined with API driven plugins, anyone can be a curator of content, providing a website where users can find stories, photos, videos, and other valuable resources. In 2015 WordPress will also launch its own API, which will give 65 million websites their own API, turning many average site owners, into API providers.
API Are Powering Mobile Apps
The number of mobile phones in our pockets will reach 7.3 billion in 2014--equally the population of the world. Of course not everyone has a cell phone, but it shows the ubiquitous nature of these devices, and shows the size of the opportunity to provide API driven resources like messaging, products, images, videos, and the other life bits we generate, and depend on each day.
Web APIs are well suited for sending and receiving the small, bite-size chunks of data, content, and other digital resources we use through our personal and business worlds. Mobile apps are not just being developed by the latest tech start-ups, mobile apps are becoming the normal mode of operations for companies, organizations, institutions, and government agencies of all shapes and sizes.
APIs Are Powering Buttons, Badges, and Widgets
APIs break up our valuable data and content into small bite-size chunks for transmission to mobile devices, but also so that they can be moved around the web, and published to where we need them. APIs are powering a wide range of buttons, badges, and other widgets that are making any single website, into a globally distributed network of content publishing and creation.
Buttons, badges and widgets are extending likes, follows and favorites from your favorite social network to any other site you may be reading or participating on, providing a bridge between all of the application you use on the web and your mobile devices. Facebook is a big part of the identity of any online user, and being able to log-in, share, and participate on the open Internet using our Facebook profiles has become normal for the average Internet user.
APIs Are Powering Spreadsheets
APIs are quickly finding their way beyond web and mobile applications, and are being used in spreadsheets to provide the data needed for business calculations, analysis and visualizations. A large amount of the worlds data exists in spreadsheets and the next wave of API developers are currently spreadsheet administrators, they just have not been informed that they can get the data they need via APis.
The days of emailing around spreadsheets, having out of date excel documents strewn here and there, is going away. Spreadsheets are connecting to APIs using excel connectors, and online using Google Spreadsheets. Spreadsheets are being constructed to make sense of Tweet streams, aggregate comments or messages, and number crunch the latest industry numbers from the government.
APIs Are Powering Devices
I'm not exaggerating when I say APIs are penetrating almost every aspect of our world. APIs are being used to connect physical devices in our lives like thermostats, smoke detectors, automobiles, clothing, glasses, and just about every object you can imagine, and some you can't.
In 2014 a wave of new start-ups, calling themselves Internet of Thing companies, have emerged to build on the API movement, extending the low-cost, simple approach to delivering resources to the physical world around us. Using APIs, the Internet is moving offline, and wiring up almost every aspect of our personal, public, and professional life.
Not Just For Developers
APIs are not just for developers. If you can use a web page, you can use most APIs. There are a great deal of educational materials (like this), to get you up to speed with APIs. API providers are also working to make it a priority to focus on non-developer users, such as data journalists, analysts, and many other non-programming user types.
Even if an API has a more developer focus to it, often times you can just ask questions of the community, or the API provider and they will help you understand the value an API delivers, and how you can possibly put it to use. One of the characteristics of Software as a Service (SaaS) applications that people are using, is there is an API behind the scenes, something that only the API savviest of users are aware of.
Orchestrate Using APIs With IFTTT And Zapier
A new generation of services have emerged, acknowledging the wealth of API resources available, and the growing number of SaaS applications that have APIs, and are delivering tools that help anyone put APIs to work. Services like IFTTT and Zapier provide very simple recipes that allow anyone to migrate content, data, and other digital resources between online platforms--no code necessary.
These API enabled orchestration platforms allow anyone to automate, migrate, and keep their online worlds in sync. These services are targeting non-developers, but provide such instant value, they are often used by programmers to make their worlds go around. This approach to using online services, is allowing companies, organizations, and even individuals to find new found freedom and control in their digital lives.
Making Companies More Nimble & Agile
Companies are learning that when they make all data, content, and other digital assets available via well designed, simple, and modular APIs over the Internet, they find a new nimbleness and agility in how they operate internally, engage with partners, and even are able to interact with the public at large.
While APIs are technical in nature, their simplicity, and efficiency are meant to make things more accessible, in simple but secure way, using the path of least resistance. Historically valuable digital resources are locked up by software developers and information technology groups, API developers are looking to change this, making resources available to as wide of an audience as possible.
Making Companies More Open & Transparent
Another aspect that is transcending the technology of APIs, is the openness and transparency it can introduce into company, organizational, institutional, and government operations. This isn't always good by default, but when done properly it can let a little sunlight into a potentially closed process, and eliminate some illnesses that occur behind closed doors.
Of course, when it comes to business, sometimes transparency can work against you, but many corporations, and young start-ups find that being closed can hurt not just potential competitors, but also slow your own company in finding success. As we've learned from open source software, openness can actually boost security, privacy and many other concerns, and a true approach to using APIs can bring some of the same effects to organizations of any size.
Allowing Companies To Be More Competitive
The ability to rapidly design, develop, deploy and iterate websites, mobile applications, while also empowering workers to be more effective via API driven spreadsheets and widgets, is giving companies a competitive edge. APIs open up a new way of doing business that the savviest of online businesses are participating in. There is a stunning amount of investment for start-ups looking to disrupt even the most entrenched industries, and companies, but it also doesn't take a start-up to make change, and many large businesses are also taking notice and getting to work on their own API efforts.
In the next five years, it will become evident which companies, and organizations do not have APIs, as they will not be participating at the levels of API driven companies. Large organizations will not be able to keep up in an always on, real-time business environment, something that APIs power across multiple channels including web, mobile, devices, and other ways of doing business online, in this new environment.
Empowering Individuals To Take Control Online
The most important thing APIs are bringing to the table, is the empowerment of the average online individual. APIs are providing access to much of the data and content we are generating online. APIs are how we can get access to the photos of our children on Facebook, and our home videos we've shared on YouTube. APIs are not just about driving business innovation, it is how the average user is taking back their digital self from a growing wave of data hungry start-ups.
API literacy is much like financial literacy. You don't have to understand the inner workings of banking and credit card industry, but you damn sure need to be educated enough to understand who has access to your bank account and credit card statement. With API literacy you don't have to understand how APIs work, but you should only use online services that have an API, and know who has access to your data and content, while also being able to download and walk away with your own valuable information that you have generated via ANY online service.
APIs Are Important, Please Keep Learning
This is just the first of a series of 101 tutorials on the world of APIs. On the home page of API Evangelist you can find other tutorials on providing APIs or consuming APIs, and the research that went into this material. I have other research areas like real-time, and voice APIs, which as I have time will be generating 101 tutorials for--let me know if there are any of them I should prioritize
I will be adding new tutorials as I have time, until then you can tune into the news stream from each area of research on the API Evangelist network, as well as tune into the API Evangelist blog for the latest analysis on the business of APIs, API Voice for the latest on the politics of APis, and the API.Report the latest in news from the APi space.
I’m seeing an increase in the number of API deployment services this year, such as startups like StrongLoop and API Spark. These companies are looking to help all of us deploy APIs from common systems, often without the need for IT or programming resources.
The providers I’m seeing emerge are catering to some of the lowest hanging fruit for deploying APIs. The commonly used, and easiest to access systems, that contain the valuable content, data, and media we need to make accessible via APIs.
The common source for many of these API deploy solutions are:
These common information sources, represent the places where the average person in any size company, organization or government agency will be storing their valuable resources. It makes sense that this new wave of API deployment startups would target these services.
If you consider every system integration option that classic API gateways have delivered, it provides a good reference for finding opportunities for building independent API deployment solutions, that if done right, could be a startup all by itself.
Not all companies can afford a full API gateway solution, and their needs are too small for a gateway. There is an emerging opportunity to help people quickly deploy APIs from common sources like spreadsheet, database, file stores, as well as more unique sources like specialty CMS and CRM systems.
Sometimes I like to look at the expanding API universe as a result of the SOA big bang, where all the tools that were in your SOA toolbox, being liberated, and available as independent services that you can get from an increasing number of new API deployment startups.
I talk about this concept often, but couldn't find any definitive post on APIs opening up a company, organization, or agency to outside ideas. This is something I hear from startups, up to federal government agencies, and from well known business brands, such as Absolut Vodka.
Absolut was one of the keynotes at API Strategy & Practice in Amsterdam, this last march. Eva Sjokvist, from Absolut, walked us through the development of their Absolut Drinks Database API. An amazing story by itself, but one thing she said that stood out to me, which is an interestingly common by-product of the API lifecycle, was the process itself opened up the company, making it more receptive to outside ideas, and collaboration.
I hear this sentiment often from groups who are developing API programs at their companies, and you see shining examples, like from Edmunds.com with their internal API Summit, but rarely do I see a metric to measure API driven, cultural change. APIs don't just open up your company’s assets and resources for access by trusted partners, or potentially the public, establishing a sort of external R&D lab, it has the potential to open up a valuable feedback loop as well, bringing in new ideas that have the potential to change how a group sees the world--evolving internal culture.
The potential for opening up a company, bringing them closer to their partners and customers, or a government agency, opening up healthier dialogue with constituents, is the most important results of an API program—greater than any single application that will get built on an API.
As I continue my research the world of API deployment, I'm trying to distill the services, and tooling I come across, down into what I consider to be a common set of building blocks. My goal with identifying API deployment building blocks is to provide a simple list of what the moving parts are, that enable API providers to successfully deploy their services.
Some of these building blocks overlap with other core areas of my research like design, and management, but I hope this list captures the basic building blocks of what anyone needs to know, to be able to follow the world of API deployment. While this post is meant for a wider audience, beyond just developers, I think it provides a good reminder for developers as well, and can help things come into focus. (I know it does for me!)
Also there is some overlap between some of these building blocks, like API Gateway and API Proxy, both doing very similiar things, but labeled differently. Identifying building blocks for me, can be very difficult, and I'm constantly shifting definitions around, until I find a comfortable fit--so some of these will evolve, especially with the speed at which things are moving in 2014.
|CSV to API - Text files that contain comma separate values or CSVs, is one of the quickest ways to convert existing data to an API. Each row of a CSV can be imported and converted to a record in a database, and easily generate a RESTful interface that represents the data stored in the CSV. CSV to API can be very messy depending on the quality of the data in the CSV, but can be a quick way to breathe new life into old catalogs of data lying around on servers or even desktops. The easiest way to deal with CSV is to import directly into database, than generate API from database, but the process can be done at time of API creation.|
|Database to API - Database to API is definitely the quickest way to generate an API. If you have valuable data, generally in 2013, it will reside in a Microsoft, MySQL, PostgreSQL or other common database platform. Connecting to a database and generate a CRUD, or create, read, updated and delete API on an existing data make sense for a lot of reason. This is the quickest way to open up product catalogs, public directories, blogs, calendars or any other commonly stored data. APIs are rapidly replace database connections, when bundled with common API management techniques, APIs can allow for much more versatile and secure access that can be made public and shared outside the firewall.|
|Framework - There is no reason to hand-craft an API from scratch these days. There are numerous frameworks out their that are designed for rapidly deploying web APIs. Deploying APIs using a framework is only an option when you have the necessary technical and developer talent to be able to understand the setup of environment and follow the design patterns of each framework. When it comes to planning the deployment of an API using a framework, it is best to select one of the common frameworks written in the preferred language of the available developer and IT resources. Frameworks can be used to deploy data APIs from CSVs and databases, content from documents or custom code resources that allow access to more complex objects.|
|API Gateway - API gateways are enterprise quality solutions that are designed to expose API resources. Gateways are meant to provide a complete solution for exposing internal systems and connecting with external platforms. API gateways are often used to proxy and mediate existing API deployments, but may also provide solutions for connecting to other internal systems like databases, FTP, messaging and other common resources. Many public APIs are exposed using frameworks, most enterprise APIs are deployed via API gateways--supporting much larger ideployments.|
|API Proxy - API proxy are common place for taking an existing API interface, running it through an intermediary which allows for translations, transformations and other added services on top of API. An API proxy does not deploy an API, but can take existing resources like SOAP, XML-RPC and transform into more common RESTful APIs with JSON formats. Proxies provide other functions such as service composition, rate limiting, filtering and securing of API endpoints. API gateways are the preffered approach for the enterprise, and the companies that provide services support larger API deployments.|
|API Connector - Contrary to an API proxy, there are API solutions that are proxyless, while just allowing an API to connect or plugin to the advanced API resources. While proxies work in many situations, allowing APIs to be mediated and transformed into required interfaces, API connectors may be preferred in situations where data should not be routed through proxy machines. API connector solutions only connect to existing API implementations are easily integrated with existing API frameworks as well as web servers like Nginx.|
|Hosting - Hosting is all about where you are going to park your API. Usual deployments are on-premise within your company or data center, in a public cloud like Amazon Web Services or a hybrid of the two. Most of the existing service providers in the space support all types of hosting, but some companies, who have the required technical talent host their own API platforms. With HTTP being the transport in which modern web APIs put to use, sharing the same infrastructure as web sites, hosting APIs does not take any additional skills or resources, if you already have a web site or application hosting environment.|
|API Versioning - There are many different approaches to managing different version of web APIs. When embarking on API deployment you will have to make a decision about how each endpoint will be versioned and maintained. Each API service provider offers versioning solutions, but generally it is handled within the API URI or passed as an HTTP header. Versioning is an inevitable part of the API life-cycle and is better to be integrated by design as opposed to waiting until you are forced to make a evolution in your API interface.|
|Documentation - API documentation is an essential building block for all API endpoints. Quality, up to date documentation is essential for on-boarding developers and ensuring they successfully integrate with an API. Document needs to be derived from quality API designs, kept up to date and made accessible to developers via a portal. There are several tools available for automatically generting documentation and even what is called interactive documentation, that allows for developers to make live calls against an API while exploring the documentation. API documentation is part of every API deployment.|
|Code Samples - Second to documentation, code samples in a variety of programming languages is essential to a successful API integration. With quality API design, generating samples that can be used across multiple API resources is possible. Many of the emerging API service providers and the same tools that generate API documentation from JSON definitions can also auto generate code samples that can be used by developers. Generation of code samples in a variety of programming languages is a requirement during API deployment.|
|Scraping - Harvesting or scraping of data from an existing website, content or data source. While we all would like content and data sources to be machine readable, sometimes you have just get your hands dirty and scrape it. While I don't support scraping of content in all scenarios, and business sectors, but in the right situations scraping can provide a perfectly acceptable content or data source for deploying an API.|
|Container - The new virtualization movement, lead by Docket, and support by Amazon, Google, Red Hat, Microsoft, and many more, is providing new ways to package up APIs, and deploy as small, modular, virtualized containers.|
|Github - Github provides a simple, but powerful way to support API deployment, allowing for publsihing of a developer portal, documentation, code libraries, TOS, and all your supporting API business building blocks, that are necessary for API effort. At a minimum Github should be used to manage public code libraries, and engage with API consumers using Github's social features.|
If there are any features, service or tools you depend on when deploying your APIs, please let me know at @kinlane. I'm not trying to create an exhaustive list, I just want to get idea for what is available across the providers, and where the gaps are potentially.
I'm feel like I'm finally getting a handle on the building blocks for API design, deployment, and management, and understanding the overlap in the different areas. I will revisit my design and management building blocks, and evolve my ideas of what my perfect API editor would look like, and how this fits in with API management infrastructure from 3Scale, and even API integration.
Disclosure: 3Scale is an API Evangelist partner.
If you read my blog regularly, you know I am constantly pushing the boundaries of how I see the API space, and sometimes my ramblings can be pretty out there, but API Evangelist is how I work through these thoughts out loud, and hopefully bring them down to a more sane, practical level that everyone can understand.
My crazy vision for the day centers around virtual API stack composition, as beautiful as the Absolut Drinks Database API. Ok, before you can even begin to get up to speed with my crazy rant, you need to be following some of my rants around using virtual cloud containers like we are seeing from docker, AWS and OpenShift, and you need to watch this video from APIStrategy & Practice about Absolut Drink Databse API deployment.
Ok, you up to speed? Are you with me now?
Today, as I was playing around with the deployment of granular API resources using AWS CloudFormations, I was using their CloudFormer tool, that allows me to browse through ALL of my AWS cloud resources (ie. DNS, EC2 Servers, S3 Storage, RDS Databases), and assemble them into a CloudFormation Templates, which is just a JSON definition of this stack I’m going to generate.
Immediately I think of the presentation from Absolut, and how they spent years assembling the image and video parts and pieces that went into the 3500 drinks they wanted available via their API, for developers to use when developing interactive cocktail apps. They had 750 images, and video clips, with a combination of 30K mixing steps, that went into the generation of the 3500 drink combinations. * mind blown *
Now give me this same approach but for composing virtual API stacks, instead of cocktails. With this approach you could define individual API resources such as product API or screen capture API. These are all independent API resources, with open source server and client side code, openly licensed interface via API Commons, and virtual container definitions for both AWS CloudFormations and OpenShift.
Imagine if I have hundreds of high value, individual API resources available to me when composing a virtual stack. I could easily compose exactly the stack my developers need, composed of new and existing API resources. I wouldn’t be restricted to building directly on top of existing data stores or APIs, I could deploy external API deployments that depend on sync to stay up to date, providing the performance levels I demand from my API stack--I could mix virtual API stacks, like I would a cocktail.
Anyhoooo, that is my rant for now. I’ll finish doing the work for deploying AWS CloudFormation and OpenShift containers for my screen capture API, rounding of all the architectural components I outlined in my API operational harness, and then rant some more.
Thanks for reading my rant. I hope it made some sense! ;-)
I’ve been profiling the API management space for almost four years now, and one of the things I keep track of is what some of the common building blocks of API management are. Recently I’ve pushed into other areas like API design, integration and into payment APIs, trying to understand what the common elements providers are using to meet developer needs.
Usually I have to look through the sites of leading companies in the space, like the 38 payment API providers I’m tracking on to find all the building blocks that make up the space, but when it came to cloud computing it was different. While there are several providers in the space, there is but a single undisputed leader—Amazon Cloud Services. I was browsing through AWS yesterday and I noticed their new products & solutions menu, which I think has a pretty telling breakdown of the building blocks of cloud APIs.
Compute & Networking
Compute - Virtual Servers in the Cloud (Amazon EC2)
Auto Scaling - Automatic vertical scaling service (AutoScaling)
Load Balancing - Automatic load balancing service (Elastic Load Balancing)
Virtual Desktops - Virtual Desktops in the Cloud (Amazon WorkSpaces)
On-Premise - Isolated Cloud Resources (Amazon VPC)
DNS - Scalable Domain Name System (Amazon Route 53)
Network - Dedicated Network Connection to AWS (AWS Direct Connect)
Storage & CDN
Storage - Scalable Storage in the Cloud (Amazon S3)
Bulk Storage - Low-Cost Archive Storage in the Cloud (Amazon Glacier)
Storage Volumes - EC2 Block Storage Volumes (Amazon EBS)
Data Portability - Large Volume Data Transfer (AWS Import/Export)
On-Premise Storage - Integrates on-premises IT environments with Cloud storage (AWS Storage Gateway)
Content Delivery Network (CDN) - Global Content Delivery Network (Amazon CloudFront)
Relational Database - Managed Relational Database Service for MySQL, Oracle, SQL Server, and PostgreSQL (Amazon RDS)
NoSQL Database - Fast, Predictable, Highly-scalable NoSQL data store (Amazon DynamoDB)
Data Caching - In-Memory Caching Service (Amazon ElastiCache)
Data Warehouse - Fast, Powerful, Fully Managed, Petabyte-scale Data Warehouse Service (Amazon Redshift)
Hadoop - Hosted Hadoop Framework (Amazon EMR)
Real-Time - Real-Time Data Stream Processing (Amazon Kinesis)
Application Streaming - Low-Latency Application Streaming (Amazon AppStream)
Search - Managed Search Service (Amazon CloudSearch)
Workflow - Workflow service for coordinating application components (Amazon SWF)
Messaging - Message Queue Service (Amazon SQS)
Email - Email Sending Service (Amazon SES)
Push Notifications - Push Notification Service (Amazon SNS)
Payments - API based payment service (Amazon FPS)
Media Transcoding - Easy-to-use scalable media transcoding (Amazon Elastic Transcoder)
Deployment & Management
Console - Web-Based User Interface (AWS Management Console)
Identity and Access - Configurable AWS Access Controls (AWS Identity and Access Management (IAM))
Change Tracking - User Activity and Change Tracking (AWS CloudTrail)
Monitoring - Resource and Application Monitoring (Amazon CloudWatch)
Containers - AWS Application Container (AWS Elastic Beanstalk)
Templates - Templates for AWS Resource Creation (AWS CloudFormation)
DevOps - DevOps Application Management Services (AWS OpsWorks)
Security - Ops Application Management Services (AWS OpsWorks)Security - Hardware-based Key Storage for Regulatory Compliance (AWS CloudHSM)
The reason I look through at these spaces in this way, is to better understand the common services that API providers are, that are really making developers lives easier. Through assembling a list of the common building blocks, it allows me look at the raw ingredients that makes things work, and not get hunt up with just companies and their products.
There is a lot to be learned from API pioneers like Amazon, and I think this list of building blocks provides a lot of insight into what API driven resources the are truly making the Internet operate in 2014.
A question I get regularly at API Evangelist is around the need for spreadsheet to API, and database to API services. Every couple weeks I get DMs and emails from someone who ask what tools are available, and that they were thinking about developing a solution. Right now I have three separate people asking this, and rather than just reply in email I figured I’d write a piece to respond, based upon my experience.
First, let me state that there is a need for simple spreadsheet to API solution. Much of the worlds data is locked up in spreadsheets, and in control of data stewards who don't access to API deployment resources. However, I'm not sure there is a lot of money to be made in providing this as a service, i feels like something that should just be default for all spreadsheet operations—you know like Google Spreadsheet.
Google Spreadsheet allows XML and JSON access to spreadsheets by default. in addition to the full blown Google Spreadsheet API. The only problem is with Google Spreadsheet you have some size limitations that make it a solution for only the simplest of data sources. I have written several posts on deploying APIs from Google Docs for both public and private spreadsheets, and have received numerous requests for a cloud version of this functionality, that would allow anyone to quickly deploy APIs from Google Spreadsheet sources.
I think the fact that Google Spreadsheet is free to use, and has API access to is spreadsheets, make this a real tough space to make money in, even if you do smooth out the process. To further convince people to not reinvent the wheel, there is already the “Wordpress” for data management, called CKAN, which is, "a powerful data management system that makes data accessible – by providing tools to streamline publishing, sharing, finding and using data.” CKAN is open source, and a tool that city, state, county and federal governments around the globe are downloading and deploying in growing numbers.
Even with the availability of Google Spreadsheet and its API, and data portal solutions like CKAN, this is still a problem I try to tackle from time to time, when I get the same itch as the people who have triggered this post. I’ve created what I call my OnGithub solutions, which allows for data management via Github, and API deployment using simple AWS hosted API framework.
- API.OnGithub.com - My attempt to provide the API layer on top of data.ongithub.com, separating the data management from the API deployment. This application runs as simple PHP API framework, deployed on Linux on AWS.
While this work was interesting, it still didn’t solve the problem of allowing the average data steward to easily manage their data that is warehoused in spreadsheets, and deploy APIs with no technical experience required. However it was a fun exercise to see what is possible for developing open tooling around this problem.
When it actually comes to deploying services that address data management I’ve watched providers like InfoChimps try and end up pivoting, and others such as Kasabi and Emergent One try, then go out of business. There are successful providers like SlashDB, Junar, and Socrata, who focus on providing open data platforms, whether it originates from a database or a spreadsheet. In 2014 I predict we will continue to see new startups emerge that promise to solve this problem, but will run into walls when it comes to monetization. There is money to be made in delivering APIs from spreadsheets, there just won’t be VC level money to be made when it comes to converting spreadsheets to APIs, but if you want to provide some open tooling and provide professional services, I think you can make an acceptable living.
The moral of this story is that if you want to do a startup, get funding and offer a data solution, you better demonstrate you have something really new and innovative, otherwise don’t bother, the space is saturated. If you want to develop some open tools, or use existing open tools and help organizations tackle their open data problems, I think there is plenty of opportunity to make both significant change, but also make some money offering sensible consulting services.
Before I got to work on the development of the FAFSA API I needed to select what my storage architecture will be for the API. When I'm developing an API around smaller sets of data I've been using JSON as the datastore, but in this case I think I will go with a MySQL backend.
This prototype of the FAFSA API will be running on an AWS EC2 linux instance, so the MySQL backend will run using AWS RDS. I created a new database and then using my JSON list of FAFSA fields I generated a MySQL table script.
Each FAFSA form can have well over 100 fields, so making sure they are small, precise datatypes is pretty critical. I will keep adjusting this as I am building the FAFSA API.
SlashDB aka /db, has recently been added to the Amazon Marketplace, providing a complete database to API solution as an Amazon Machine Image (AMI).
Companies can use the /db Amazon image to automatically generate REST APIs from common relational databases like Microsoft SQL Server, Oracle, MySQL, PostGreSQL, which includes Amazon RDS.
/db charges based upon the number of databases you launch and the number of users that are using the API for the database, and you will have to pay for the regular charges for any Amazon EC2 instances.
I like the idea of building API solutions and launching as Amazon Machine Images. I think ready-to-go platform solutions like this for AWS, Google, Heroku and other top cloud platforms are good for potential API providers.
I have a long, winding history of database administration in my past. I've been managing databases since my first job working on school district databases in the State of Oregon in 1988. So I've been that database administrator you are asking for data access from, and I've personally managed and interacted with too many of these DBAs to keep track.
I was discussing strategies for getting access to your organizations central database today, and realized I don't have enough of these data acquisition stories on API Evangelist.
This data access quest will start with a burning need to build a website, mobile app or provide access to a 3rd party partner to some data. You know that this data is in your organizations database, all you need is access to make your project come to life.
In many organizations this can be easier said than done. There could be numerous reasons why this will be difficult ranging from technical to political, and be fuly prepared for it to often be political, masked as technical.
Database access bottlenecks can range from security concerns to cranky, unfriendly DBAs or possibly just that the central database system has been in operation for many years and cannot handle large number of additional requests.
Unwinding the politics of database access can often be more time consuming than the technical side of database access. In many organizations your time is better spent just getting what you need and moving on. Make it only about access, and not looking to change the legacy process.
This legacy is why the most common way of getting at your organizations data will be a data dump to a network or FTP location, in a CSV, XML or JSON format. If your database administrator offers this to you, take it. You can make use of this, design the application you need need, without the headache of battling to change things or getting direct access.
A data dump is a great for DBA's to give you the data you requested, but offload the processing and responsibility to someone else. So take it, if it will meet your minimal requirements.
Once you get your data dump, build your applications datastore, API and site or application. Then at a later date, you can take your blueprint and sell it back to your database administrator, IT or further up the management chain, on how things can be done better.
While a data dump may not be optimal, it might be all you can get for now. A great deal of education of IT and database folks is needed, to help them understand that APIs are not all publicly available like Twitter, and they actually provide much granular level security and access than traditional IT access levels. But we have many years of education to occur before we will change the tides.
So take that data dump, build your own API and application and make evolving IT a separate battle from your project goals.
I've been organizing much of my research around APIs into groupings that I call "stacks". The term allows me to loosely bundle common API resources into meaningful "stacks" for my readers to learn about.
I'm adding a new project to my list of 30+ stacks, that is intended to bring together the most commonly used API resources, into a single, meaningful stack of resources any web or mobile developer can quickly put to use.
So far I have compiled the following APIs in 29 separate groups:
- Amazon EC2
- Google AppEngine
- Amazon S3
- Rackspace Cloud Files
- Amazon RDS
- Amazon SimpleDB
- Amazon Route 53
- Rackspace Cloud DNS
- DNS Made Easy
- Amazon SES
- Rackspace Email
- AT&T SMS
- AT&T SMS
- Push Notifications
- Urban Airship
- AT&T SMS
- Facebook Chat
- Google Talk
- Google Directions
- Google Distance Matrix
- Google Geocoding
- Google Latitude
- Google Drive
- Echo Nest
- Delicious Pinboard
- Businesses / Places
- Google Places
- Google Payments
- Google Real-time
- URL Shortener
- Google URL Shortener
This is just a start. I will publish a full stack, complete with logos, descriptions and links. For now I'm just flushing out my thoughts regarding what are some of the top resources that are currently available for developers.
I will be making another pass over the APIs I track on in the coming weeks, as well as add to the list each week as part of my monitoring.
If you see anything missing, that should be in there...let me know!
I feel we have done a good job explaining what is an API, why people need APIs, and providing services to manage APIs, but we are falling short on delivering information, tools and services for deploying APIs.
First I want to acknowledge that many companies have established IT resources, in the form of technology talent, and internal systems they can put to use when deploying APIs. This post isn’t for you. This is about individual problem owners looking to deploy APIs, who most likely have little or no access to traditional IT resources for a variety of reasons.
You want to deploy an API, because you havplease some data, content or other resources you need to make accessible to partners or the public, allowing for integration into other systems or be used in the development of web or mobile applications or possibly even data analysis and visualizations.
As much as I love APIs, I suggest everyone consider your overall objective behind launching an API. You may not need an API, you may just need to make your information machine readable and available on the open Internet. Technically, this isn’t an API, but publishing your data or content in a machine readable format like XML or JSON can accomplish many of the same goals of an API, without the overhead of an API.
Many tools you use every day like Microsoft Office, Google Apps, Salesforce provide tools for publishing data and content in machine readable formats like CSV, XML or JSON, much like publishing or printing as a PDF. Not everyone needs an API, but everyone needs to understand the how-to, and the benefits of making machine readable a default in all business, organization, government and even individual processes.
PDF made data and content portable, for humans. But a PDF does very little good for other machines, websites and mobile apps. CSV, XML and JSON make data and content portable, while also making them machine readable. Please consider this simple evolution, while planning your API.
Machine readable will go far to increase efficiencies, interoperability, transparency and innovation within many existing government, organizational and business operations. By providing data, content and other resources in CSV, XML and JSON they can be easily used across mobile, web and data implementations. However in other situations there will be a higher need for more advanced interaction with data and content in the form of querying, filtering, transformations as well as the need to secure resources, limiting who has access and to what scale. This is where we start moving beyond just machine readable, and into a truly needing an API.
When it comes to API deployment I would put users into two distinct groups:
- Technical - You have skills and resources to architect, design, develop and deploy infrastructure, code to meet your API needs
- Non-Technical - You do not have the resources to assemble and execute an API strategy, making a cloud service an optimal route
APIs are about making critical assets and resources available to technical and non-technical groups. APIs are often about circumventing some of the ways technology bottlenecks can slow us down. It is important that there are not just a wealth of cloud services for non-technical folks to deploy APIs, but also a buffet of openly licensed, downloadable tools that anyone can put to use when deploying an API.
You have some technical resources and the know how (or willingness to learn), and you want an API. Ideally there would be open source tools available for you to download and implement:
- CSV to API
- Database to API
- Developer Portal
- Key Management
- App Management
These tools would allow you to assemble your own API strategy in the language and platform of your choice. There should be educational materials helping you understand how to implement individually or collectively as a single platform.
You do not have the technical resources or time to implement your own API Strategy. You need a dead simple, software as a service implementation. You can see examples of this with the latest breed of API service providers, Emergent One, API Spark, and API Engine.
We are just seeing this new breed of API service providers rise to meet the demand of not just API management, but now also API deployment. Making API deployment accessible to anyone, not just developers or companies with necessary resources. This will make the benefits of APIs accessible to the everyday problem owners, who often are restricted by IT bottlenecks or entire lack of IT resources.
Making data and content from a spreadsheet or existing database, machine readable by default or deployed as an API, allowing for more sophisticated interactions, should be as easy as deploying Wordpress for your blog. You should be able to go to a service provider and get exactly the hosting, deployment and management options you desire, or you should be able to download and install on your own infrastructure, giving you the control you desire over storage, hosting and access.
I’m adding a new project area to API Evangelist called I Want An API, where I’ll be working to help define this space, and bring together other people to assist in developing the resources and tools we will all need to easily launch an API.
I spent time this week looking at 20, of what I’m calling API reciprocity providers, who are providing a new generation of what is historically known as ETL in the enterprise, to connect, transfer, transform and push data and content between the cloud services we are increasingly growing dependent on.
With more and more of our lives existing in the cloud and via mobile devices, the need to migrate data and content between services will only grow more urgent. While ETL has all the necessary tools to accomplish the job, the cloud democratized IT resources, and the same will occur to ETL, making these tools accessible by the masses.
There are quite a few ETL solutions, but I feel there are 3 solutions that are starting to make a migration towards an easier to understand and implement vision of ETL:
These providers are more robust, and provide much of the classic ETL tools the enterprise is used to, but also have the new emphasis on API driven services. But there are 10 new service providers I’m calling reciprocity platforms, that demonstrate the potential with offering very simple tasks, triggers and actions that can provide interaction between two or more API services:
I consider reciprocity an evolution of ETL, because of three significant approaches:
- Simplicity - Simple, meaningful connections with transfer and tranformations that are meaningful to end users, not just a wide array of ETL building blocks an IT architect has to implement
- API - Reciprocity platforms expose meaningful connections users have the cloud services they depend on. While you can still migrate from databases or file locations as with classic ETL, reciprocity platforms focus on APIs, while maintaining the value for end-users as well as the originating or target platforms
- Value - Reciprocity focus on not just transmitting data and content, but identifying the value of the payload itself and the relationships, and emotions in play between users and the platforms they depend on
This new generation of ETL providers began the migration online with Yahoo Pipes. Which resonated with the alpha developers looking to harvest, migrate, merge, mashup and push data from RSS, XML, JSON and other popular API sources--except Yahoo lacked the simplicity necessary for wider audience appeal.
While I feel the 10 reciprocity providers isted above represent this new wave, there are six others incumbents trying to solve the same problem:
While studying the approach of these 20 reciprocity providers, it can be tough to identify a set of common identifiers to refer to the value created. Each provider has their own approach and potentially identifying terminology. For my understanding, I wanted to try and establish a common way to describe how reciprocity providers are redefining ETL. While imperfect, it will give me a common language to use, while also being a constant work in progress.
For most reciprocity providers, it starts with some ecompassing wrapper in the form of an assembly which describes the overall recipe, formula or wrapper that contains all the moving ETL parts.
Within this assembly, you can execute on workflows, usually in a single flow, but with some of the providers you can daisy chain together multiple (or endless) workflows to create a complex series of processes.
Each workflow has a defining trigger which determines the criteria that will start the workflow such as new RSS post or new tweet, and with each trigger comes a resulting action which is the target of the workflow, publishing the RSS post to a syndicated blog or adds the tweet to a Google Spreadsheet or Evernote, or any other combination of trigger and action a user desires.
Triggers and actions represent the emotional connections that are the underpinnings of ETL’s evolution into a more meaningful, reciprocation of value that is emerging in the clouds. These new providers are connecting to the classic lineup of ETL interfaces to get things done:
- Web Service
While also providing the opportunity for development of open connectors to connect to any custom database, file, messages and web services. But these connectors are not described in boring IT terms, they are wrapped in the emotion and meaning derived from the cloud service--which could have different meanings for different users. This is where one part of the promise of reciprocity comes into play, by empowering average problem owners and every day users to define and execute against these types of API driven agreements.
All of these actions, tasks, formulas, jobs or other types of process require the ability to plan, execute and audit the processes, with providers offering:
- History / Logging
With data being the lifeblood of much of these efforts, of course we will see “big data” specific tools as well:
- Data Quality
- Big Data
While many reciprocity providers are offering interoperability between two specific services, moving data and resource from point a to b, others are bringing in classic ETL transformations:
After the trigger and before the action, there is also an opportunity for other things to happen, with providers offering:
During trigger, action or transformation there are plenty of opportunities for custom scripting and transofrmations, with several approaches to custom programming:
- Custom Scripts
- Command Line
In some cases the reciprocity provider also provides a key value store allowing the storage of user specified data extracted from trigger or action connections or during the transformation process. Introducing a kind of memory store during the reciprocal cycle.
With the migration of critical resources, many of the leading providers are offering tools for testing the process before live execution:
With any number of tasks or jobs in motion, users will need to understand whether the whole apparatus is working, with platforms offering tools for:
While there are a couple providers offering completely open source solutions, there are also several providing OEM or white label solutions, which allow you to deploy a reciprocity platform for your partners, clients or other situations that would require it to branded in a custom way.
One area that will continue to push ETL into this new category of reciprocity providers is security. Connectors will often use OAuth, respecting a users already established relationship with platform either on the trigger or action sides, ensureing their existing relationship is upheld. Beyond this providers are offering SSL to provide secure transmissions, but in the near future we will see other layers emerge to keep agreements in tact, private and maintain the value of not just the payload but the relationships between platforms, users and reciprocity providers.
Even though reciprocity providers focus on the migration of resources in this new API driven, cloud-based world, several of them still offer dual solutions for deploying solutions in both environments:
There is not one approach either in the cloud, or on premise that will work for everyone and all their needs. Some data will be perfectly find moving around the cloud, while others will require a more sensitive on-premise approach. It will be up to problem owners to decide.
Many of this new breed of providers are in beta and pricing isn’t available. A handful have begun to apply cloud based pricing models, but most are still trying to understand the value of this new service and what market will bear. So far I’m seeing pricing based upon:
Much like IaaS, PaaS SaaS and now BaaS, reciprocity providers will have a lot of education and communication with end users before they’ll fully understand what they can charge for their services--forcing them to continue to define and differentiate themselves in 2013.
One of the most important evolutionary areas, I’m only seeing with one or two providers, is a marketplace where reciprocity platform users can browse and search for assemblies, connectors and tasks that are created by 3rd party providers for specific reciprocity approaches. A marketplace will prove to be how reciprocity platforms serve the long tail and niches that will exist within the next generation of ETL. Marketplaces will provide a way for developers to build solutions that meet specific needs, allowing them to monetize their skills and domain expertise, while also bringing in revenue to platform owners.
I understand this is all a lot of information. If you are still ready this, you most likely either already understand this space, or like me, feel it is an important area to understand and help educate people about. Just like with API service providers and BaaS, I will continue to write about my research here, while providing more refined materials as Github repos for each research area.
Let me know anything I'm missing or your opinions on the concept of API reciprocity.
I was checking out the updates to the AWS Reference Architecture, where they provide blueprints for how you can use AWS. In this version AWS provides an e-commerce architecture reference--providing a system overview, a detailed architectural diagram, and a list of the AWS services used in the architectural approach.
The AWS e-commerce architecture reference provides three separate areas:
- Web Frontend - This is a reference architecture for the web frontend of an e-commerce site. It makes use of Route 53, CloudFront, Elastic Beanstalk, S3, ElastiCache, DynamoDB and CloudSearch
- Checkout Pipeline - This is a reference architecture for a secure and highly available checkout pipeline service for an e-commerce site. It uses the Virtual Private Cloud, the Simple Workflow Service, Elastic Beanstalk, the Relational Database Service, and the Simple Email Service
- Marketing & Recommendations Service - This is a reference architecture for a marketing and recommendation service for an e-commerce site. It uses Elastic MapReduce, S3, Elastic Beanstalk, the Relational Database Service, DynamoDB, and the Simple Email Service
I will add API architecture references as a building block for API owners to consider when planning and managing an API. If you can provide clear blueprints for developers to follow, you can increase the chances developers will be successful with integration of an API into their application or systems.
I’ve been processing a conversation over at Branch, that was triggered by a story in TechCrunch by Sarah Perez(@sarahintampa) called, “StackMob Ratchets Up The Competition: Makes API Calls Free, Launches A Marketplace For Third-Party Mobile Services”.
There are several layers to the convSteersation, the part I’m focusing on is about how BaaS providers are structuring their pricing, in which Sarah Perez writes:
StackMob is now making API calls free. This latter news will impact the competitive landscape, which includes startups like Parse, Kinvey and others, all of which have traditionally charged developers as API calls increase alongside an app’s popularity.
Making the argument that:
Today, developers have to have an understanding of how many API calls they make, and if an app becomes really successful, developers are penalized for that with increased pricing.
Sarah quotes StackMob CEO Ty Amell, saying that:
“this isn’t really all that fair, explaining that it doesn’t matter to StackMob how many API calls the app makes – the cost to them doesn’t really go up. “It doesn’t matter to us whether someone’s making a million API calls or 20 million API calls – the cost is fairly manageable on our parts. So we felt like this was a blocker to innovation on the developer’s side,” he explains. “And we feel like API calls are starting to become a commodity, where it’s really how can you provide value beyond that?”
The conversation immediately begins on Twitter, with:
Steve Gershnik(@StackMobSteve) of Stackmob - @stackmob doesn't charge for 'active users' either. No success tax for you.
Joe Chernov (@jchernov) of Kinvey - but you charge for features. In the end, everyone charges for something. It's business!
After this, the conversation moves to Branch. I’m not including full comments, because there is a lot of banter back and forth, discussing various claims BaaS providers make, that I feel are irrelevant to the more important, pricing story.
The conversation continues:
Joe Chernov (@jchernov) of Kinvey:
If a developer prefers to pay by number of users, our model may work for them; if someone would rather buy features, then StackMob may make more sense. It's all good. Everyone has their own twist on pricing. Vendors should strive to highlight their own value without misrepresenting others'. Fair? Steve Gershnik (StackMob):
We don't charge per user. We don't charge per API call. We think those are outdated and archaic pricing models.
Miko Matsumura (@mikojava) of Kii:
I respect what @StackMob is doing but in looking at their pricing page, isn't it just segmenting everyone into a free tier and a "custom" Enterprise tier?
Steve Willmott(@njyx) of 3Scale:
Which model is best in our experience depends a lot on what type of market you're serving - individual developers tend to like by the drop because (start cheaply and scale). Large companies tend to like predictability so they can budget properly.
Box.net is a storage company - yet, for all their enterprise accounts storage is unlimited. Variable scared their customers. Instead they charge by number of seats/users - something their enterprise customers CAN predict.
Thinking more abstractly: If a developer is successful, there will be some sort of increase in volume. Unless the PAAS service is truly magical that means extra cost for the PAAS provider.
You can call this a success tax, but in my view what matters is not if there is more cost to the developer but if the unit cost goes down over time. I.e. As I succeed, I'm happy to pay you a bit more as long as you are not getting an increasing %age of the pie (and ideally if the %age is decreasing). If that %age increases I'll most likely ditch you for someone who offers me better terms at that point.
Our pricing model is per API calls as well, we do it like that because it is clear for developers (and for our business model), also allows to create apps and prototypes that, in case they are not successful, the developers won't be charged. according to @jchernov
Only Stackmob, Kinvey, Kii and Quickblox are represented in the conversation on Branch. So I wanted to see how each of the 20 BaaS providers I’m tracking on, structure their pricing:
- APIOMAT - Beta, can't find
- Appcelerator - Events, Notifications, API Calls, Storage, Emails, Support
- Applicasa - Beta, can't find
- Buddy - API Calls
- CloudMine - User
- CloudyRec - App, API Calls, File Storage
- FeedHenry - Can't Find
- Flurry - Beta, can't find
- Geoloqi - API Calls
- iKnode - API Calls, Data Storage
- Kii - API Calls, Data Storage, Push Notifications
- Kinvey - Users, API Calls
- MobDB - API Calls, Push Notifications, Data Storage
- Parse - API Calls, Push Notifications, Data Storage
- Proxomo - API Calls
- Quickblox - API Calls
- ScottyApp - Apps, Users, Database Size, Emails
- Sencha.io - Can't Find, Beta
- Stackmob - Features
- Urban Airship - Push Messages
I couldn’t find pricing for five of the BaaS providers, because they are in beta, but pricing by API call is definitely the preferred way for the rest of them, with 11 of the 15 using this approach. The rest of them charge based upon a mix of number of users, number of apps, data and file storage, push messages, and email messages. With Stackmob going with pricing based upon feature, add-on or premium services, which can also be from 3rd party services via the StackMob marketplace.
Personally I don’t think there is one approach to API or BaaS pricing. Seems to me, that different models will work for different industries, companies and even at the application level. But I wanted to document this conversation for my larger research project around the BaaS space--maybe I will have more opinions after I dive in deeper.
Let me know what you think about BaaS pricing.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.