On 2024-09-26 I attended the AWS UK Community Summit in Manchester.

Of course it rained. Everyone warned me it would! I’m used to it living in Glasgow. It turns out Manchester isn’t even in the top 20 rainiest UK cities. I wonder why it gets such a wet reputation.

My picks from the schedule were mostly from the Architects track, with some Builders and Leaders content too.

  • Damien Jones: Building and Automating Serverless Auto Scaling Data Pipelines in AWS
  • David Boyne: Avoiding the big ball of mud with serverless event driven architectures
  • Sheen Brisals: How do you measure your enterprise’s readiness for serverless?
  • Yan Cui: Money-saving tips for the frugal serverless developer
  • Vadym Kazulkin: Detect operational anomalies in Serverless applications with ML-based Amazon DevOps Guru
  • Sarah Hamilton: Operating Quality Event-Driven Serverless Architectures

Full schedule:

Serverless: in 5 out of 6 talk titles, it looks like an unofficial theme of the event! It’s a broad term that refers to managed services that allow you to build robust solutions at a higher level of abstraction. You think about events and workflows rather than servers, storage, and requests. That stuff is obviously still important, but for most use cases there is no need to stray from the default behavior that services such as S3 and Lambda provide. I’m enthusiastic about serverless, because the less generic infrastructure I have to manage, the more I can focus on delivering value.

Sarah’s talk was my favorite because she changed my mind and settled a doubt. It was a secondary point in her argument, but Sarah made me see the light on AWS’s CloudFormation Development Kit (CDK).

A complex CloudFormation template may describe resources with patterns of interdependencies. Writing this all out by hand may be tedious and validating its correctness can be worse. If you’re tempted to parse the template into an object model to check its properties programmatically, you may as well go further and use a whole object model to generate the template. That’s what the CDK is: a programmatic object model for CloudFormation resources that also renders templates.

AWS maintains the CDK so we trust the correctness of the object model and the template renderer. Then what remains for us to test the correctness of our templates is to test properties of an object model.

Sarah’s revelation is what made this report take so long to write. After the conference I put her lesson into action. Last week I speedran a course in TypeScript and CDK to solve a problem for which bare CloudFormation offers ugly solutions. I hope to share more about that later.

The rest of the article shares my main takeaways and dumps links or resources for further research.

Damien Jones, a.k.a. the Amazon Web Shark, kicked us off with Building and Automating Serverless Auto Scaling Data Pipelines in AWS.

“Big Data” is often described by four V’s: Volume, Variety, Velocity, and Veracity. Damien added a 5th V: Value: Is it important or useful? Does it have a monetary value?

Damien told us the pros and cons of the building blocks he uses for his pipelines: S3, Lambda, Athena, Glue, Step Functions, and EventBridge.

I hadn’t seen Glue in action before. I like its dashboards that present the data quality checks.

Damien’s recorded demo of his pipelines was slick. I prefer recorded demos because controlling the screen doesn’t distract from the communication. He overlayed a timer, highlights, and blurring to guide the audience’s attention to the right place.

David Boyne followed with Avoiding the big ball of mud with serverless event driven architectures.

The Project Paradox: Why do we take the biggest decisions at a time when we have the least knowledge?

Without conscious design, every project inevitably graduates toward a complex, unpleasant mess.

Event-Driven Architecture (EDA) and Domain-Driven Design (DDD) give us patterns to control the complexity, but we need better tools to make them more discoverable.

Links and resources to learn more:

Sheen Brisals asked How do you measure your enterprise’s readiness for serverless?

Serverless is not magic. You need to understand your use case, skills, and requirements. You need to assess its efficiency, cost, and trustworthiness.

Afraid of vendor lock-in? AWS is not your vendor but a partner.

I asked Sheen what he thought about David Heinemeier Hansson publicly quitting AWS for bare-metal servers because it’s cheaper, because he doesn’t need instant scaling. Sheen said that you need to look at the total cost of ownership and innovation.

Yan Cui took us up to lunch with Money-saving tips for the frugal serverless developer.

Yan gave a rapid fire of common costly mistakes and how to solve them. After each, a refrain: “I don’t mind spending money, but I want value!” All components need a purpose and a return on investment.

Favorite 1: CloudWatch Logs ingestion is expensive. If you use Lambda extensions and a telemetry API to bypass CloudWatch logs, CloudWatch Logs may still charge you for ingesting the default messages that every Lambda execution writes. Block all writes using an IAM policy.

Favorite 2: Avoid using the RequestResponse (synchronous) calls for Lambda-to-Lambda, because you will pay twice for the execution time. For secondary tasks that aren’t in the critical path, use Event (asynchronous) invocation.

I asked Yan whether I can avoid paying for Lambda to wait for a slow API response by using a Step Function, which waits for free. If the API was a webhook, then maybe a Step Function would make sense, but if it was a synchronous HTTP request/response, then Lambda or something needs to keep the connection open to receive the response.

More resources:

Vadym Kazulkin showed how to Detect operational anomalies in Serverless applications with ML-based Amazon DevOps Guru.

We learned what problems the Amazon DevOps Guru solves and saw it in action.

DevOps Guru is a time-saver over standard CloudWatch. It takes the drudgery out of making alarms for every operational metric. It applies machine learning to Config, CloudWatch, X-Ray, and CloudTrail to provide faster operational insights.

It categorizes notifications as “reactive” and “proactive”. Its proactive mode overlaps with Security Hub. Whereas Security Hub checks security configurations such as public access and encryption, DevOps Guru adds value by checking operational configurations such as read capacity or detailed monitoring.

Vadym generated capacity notifications using a load testing tool called hey, a replacement for Apache Bench, or like a rapid-fire curl.

Sarah Hamilton told a story about Operating Quality Event-Driven Serverless Architectures.

She opened with an analogy, a restaurant called Sync’n’Serve that struggles to coordinate its chefs, waiters, and customers as its popularity grows. I like talks that start with stories.

Sarah shared her opinions on testing, with reference to the “square of balance” popularized by Sheen’s book. The square describes a trade off among testing, observing, recovering, and delivering.

Sarah prefers unit tests over integration tests and end-to-end tests. The further you get from each component, the more it costs to perform a test because of its complexity, and the less it tells you about exactly where a problem occurs and who is accountable. Unit tests of course can’t catch everything, so she prefers to cover only the critical paths using the more complex test types.

Speedy reversal of a bad change is worth more than complete test coverage. Bugs and errors are inevitable, so we need to be ready to detect (observe) them and handle (recover from) them, and we need to keep delivering to stay relevant in the market.

Thanks to all the Summit team for delivering an exciting and satisfying full day of AWS learning and networking. Joe Park, Edmund Craske, Robin Ford, Phil Horn, Tom Misiukanis, and the rest who I’ve yet to meet, you all deserve to be proud of what you did for the community this year.

Looking forward to more, bigger, and better next year!