Rating: 8.0/10.
Designing Web APIs: Building APIs That Developers Love by Brenda Jin, Saurabh Sahni, Amir Shevat
A fairly short book about best practices for designing web APIs to be easy to use and maintainable, written by three engineers who worked on the Slack API integration, which has been a successful marketplace for apps.
Chapter 1. APIs are useful for a company to solve a problem for another company, so they don’t have to build it themselves, and there’s often a strong business case for this. One challenge is to make it a standard test of time, since changing an API is expensive. An API might start out as internal use only and eventually evolve to be available to external users, in other cases it is designed for external use from the beginning.
Chapter 2: REST principles involve using POST to create, GET to retrieve, and PUT or PATCH to update. All URLs should contain nouns and not verbs. However, this is not the most suitable for more complex queries that are not directly changing the state of an object. The RPC style, while still operating over HTTP, uses verbs like “conversations.archive”, this style is more suitable when the actions are more complex and not simply CRUD operations. GraphQL only has one endpoint and uses a complex query language, it is more efficient as it avoids sending the full response payload: when only a few fields are needed, it only sends back what is requested, and it does type checking and offers other conveniences.
Webhooks are good for real-time communications; the server calls the endpoint on the client app, which is more efficient than the client polling the server. However, when developing webhooks, you need to handle failures, retry, and the security is more complex. Other alternatives include websockets for bi-directional communication and HTTP streaming for unidirectional communication; complexities arise when handling connection, reconnection, and buffering.
Chapter 3: API Security. The most basic type of security is Authorization: Basic, for simple username and password combinations, but it’s not the most secure. In OAuth flow, an application (eg: TripAdvisor) asks the API provider (eg: Facebook), to perform an action on behalf of the user (eg: making a post). In this flow, the API provider grants an access token to the application after the user authorizes it. OAuth scopes need to be granular enough so as not to grant the client application more permissions than are needed for the task. The access token should have a short lifespan to limit the exposure if it is compromised, and instead, issue a refresh token that can be exchanged for a new access token, this process also requires the client secret. The best practice is to have a UI to revoke access tokens and secrets that are known to be compromised. The OAuth API provider needs to carefully follow best practices to prevent various attacks, such as MITM attacks, clickjacking, phishing using deceptive application names, etc.
Webhook security requires that a webhook should contain a verification token to ensure that the sender is who he claims to be, and not someone else, since the endpoint is publicly accessible, or even better for the sender to sign it, as this method prevents any possible replay attacks or forgery. This is particularly important if the token is intercepted during transit. A more secure alternative, which does not require the client-side developer to implement any best practices, is to send a thin payload that merely notifies that something has changed, and subsequently, the client makes a request to retrieve the actual data.
Chapter 4: API Design Best Practices. Return errors that have as much detail as possible to make debugging easier: these should include both a machine-readable error code and a human-readable message, and document these possible errors. Build tooling to track which endpoints and parameters are used, along with their error rates. It’s a good idea to have a beta program for a few customers to use first, to catch any design mistakes before they are rolled out. Versioning is a good strategy to introduce breaking changes; maintaining backward compatibility can be difficult and require a lot of extra engineering work (like translation layers) but is mission-critical in many cases.
Chapter 5. When designing an API, first write down agree on what the API should accomplish before going into too much technical detail. Then, decide on which API paradigm and scopes are needed, then write a detailed API spec, should include all the endpoints, inputs and outputs with examples, required scopes, and possible errors it may throw, then get feedback from other stakeholders (possibly from outside the company).
Chapter 6. Scaling APIs can be done in various ways, such as caching, indexing, and various distributed system designs that are outside the scope of this book. Many times, scaling issues can be addressed by changing the API design, for example, a frequent cause of heavy load is when clients are forced to poll to get real-time updates; changing the design to have a webhook will lessen this load.
Avoid endpoints that return a lot of results that are expensive to compute in the backend, and oftentimes, the client doesn’t actually want them. Consolidate similar endpoints, like slack groups versus channels, so that the client doesn’t need to make multiple calls to get this information. Support bulk operations in the endpoint to avoid multiple calls and add filter options to the request.
Pagination can be done in two ways. Offset pagination is simple; you just pass an offset and it easily translates to a UI with multiple pages, allowing you to jump to the middle, but the disadvantage is that it might miss elements or return duplicates if the underlying data is changing while the fetch happens. Another method, cursor-based pagination, is more performant: it works by returning a cursor, usually an ID or a timestamp, which the client has to pass back on every request, but unlike offset pagination, you cannot jump to the middle.
Rate limiting should be done to protect the infrastructure against bad actors. This can be done by user, application, or by IP for unauthenticated users. One algorithm to do this is the token bucket algorithm: each request consumes a token and the bucket fills up at a fixed rate; if the token bucket is empty, then the request is denied. This allows some burst of traffic for some period of time. When rate limiting, it’s good to indicate helpful error codes and either have an API or response headers to communicate the status of the limit, and document how this rate limiting works. Consider releasing a client-side SDK that implements the best practices like pagination, validation, rate limiting, and exponential backoff.
Chapter 7: API inconsistency often arises when the requirements change, causing a new endpoint to be implemented differently from the previous one, this is often hard to avoid. Automated testing can be used to define a schema and ensure that your response always conforms to this schema, but it’s hard to ensure that the input matches the schema. When the API is externally facing, any minor change can cause integrations to break (even if it doesn’t change documented behavior). It’s important to identify who is using every endpoint that is about to change and give them at least a few months’ notice via email in advance of changing it; a deprecation timeline that is too short will hurt your trust with clients. Versioning is best specified in the routing URL rather than a query parameter; use semantic versioning so that adding things results in a minor version bump, whereas removing or changing functionality leads to a major version. Be careful with balancing between multiple deprecated versions because they have high engineering and infrastructure costs and they slow down future development.
Chapter 8: Business and professional users are likely to pay more for your API, but they are more sensitive to API changes compared to hobbyists and hackers. Define the attributes of your developers, the platforms and tools they are going to use, and the common use cases they will have for your API. Also define a funnel: how developers first become aware of your product, try it out as an example, build a product, and launch it to production, and measure the conversion rate at each step of the funnel. Different strategies should be used to improve each step of the funnel, eg: a hackathon might get people to try using a product, but it won’t help with pushing it to production users.
Chapter 9. Key documentation includes a getting started guide where the aim is to reach hello world or equivalent as quickly as possible. The reference documentation should be as complete as possible, and tutorials should focus on the most complex and common tasks. Examples should be included in several languages that your developers are likely to use, with a drop-down to switch between them. Consider making an SDK or possibly a framework to provide a higher level of abstraction for developers than an API, as this will make common use cases easier. Community events, webinars, conferences, and forums are good ways for people to learn from each other in a community-driven way.
Chapter 10: Developer programs can either be deep, targeting a few developers and providing a lot of support, or broad, where you reach a multitude of developers. A partner engineer can work individually with your large clients and support their unique needs. Partner with a beta program to have your API integrations ready and launched for the public: to do this, reach out to them and work closely with them during the beta period before the public launch. Broad programs reach a large number of developers and these include activities such as hackathons, speaking at conferences, and making online videos.
