SREcon17 Europe/Middle East/Africa

• In a distributed service, every component is a frontend to another one down the stack. How can it deal with backend failures so that the service as a whole does not go down?  
• In a distributed service, every component is a backend for another one up the stack. How can it be scaled and managed, avoiding overload and under-use?  
• In a distributed service, latency is often the biggest uncertainty. How can it be kept predictable?  
• In a distributed service, availability, processing, and latency costs contributions are hard to assign. When things (inevitably) go wrong, what components are to blame? When they work, where are the biggest opportunities for improvement?

• Managing SSH keys: The bad parts


• Replacing SSH keys with ephemeral certificates: how & why


• Discussion of an implementation of a CA for SSH certificates


• Call for participation, showing github source

• logging myths, and dangerous fallacies


• being deliberate about operational and engineering needs


• better troubleshooting using structured and canonical logs


• getting more benefits and performance out of notoriously slow static analyzers


• writing tests with performance in mind

• Web Performance: CDNs, POPs, RUM steering


• Multi-vendor strategy for redundancy & performance


• Tools for cross-vendor consistency, vendor fail-out, global site monitoring & status boards

• How do you build an SRE team for a product which isn't serving real users yet?


• How do you build a cohesive team and structure out of many disparate teams? (Networking, SRE, software development)


• How do you build oncall discipline in a team which largely hasn't been oncall before?

Many companies are adopting SRE. SREcon is a fabulous way to learn about how others in the industry are implementing and approaching SRE. We learn about some companies through talks, and others through networking. This BoF has two aims: a) share our companies successes and challenges in implementing SRE and b) consider if the industry would benefit from having a survey and white paper (similar to the State of DevOps survey and white paper).

• Consistent naming schemas for services, environments
  
• Maintaining lightweight, easy-to-change production configuration abstraction layers
  
• Release early, often
  
• Canary everything by sharding into more A/B environments than you'd think you’d need
  
• Encourage backwards compatibility in all APIs
  
• Validate and test all configuration before changing global state
  

• our journey from a firefighting to a proactive work organization


• good and bad organizational decisions we made


• impacts on the morale of the team

• An analysis a set of cognitive biases, with illustrated examples (e.g. anchoring/priming, substitition/availability, loss aversion, etc)


• Introduction of Kahneman/Tversky's "System 1/System 2" hypothesis (i.e. that our mental architecture is divided quite sharply into two modes of thinking/being)


• Description of the on-call experience for SREs


• Relation of this to previous cognitive biases; assertion that on-call is actually about using humans' infinite "jump out of the system" ability, otherwise the software could just fix itself


• Description of techniques to move an engineer from System 1 to System 2 thinking (which is what you actually want)


• Call to action for self-healing software

• What psychological tricks might affect their next on-call shift


• What to do about them


• Why on-call sucks (no, really) and why there may be a future without it

This presentation replaces the talk by Dinah McNutt, who is unable to attend. Laura Nolan will revisit her SRE 101 content from yesterday; if you missed the session due to the meeting room being at capacity, this is your opportunity to attend.

The purpose of an SRE team is to keep its services up, reliable, performant and efficient. How do effective SRE teams do this?

We'll run through an overview of key SRE competencies: monitoring and alerting, incident response, disaster recovery, performance and efficiency, change management and capacity planning.

We'll also look at the habits of successful SRE teams and some common pitfalls.

1. Challenges with debugging in production


2. Various approaches that are used in the industry


3. Examples from Bing and Cortana incidents and steady state problems to illustrate the techniques


4. Service design ideas that make them easier to debug

• The earliest jobs in computers (~40s). What were they, why did they exist, and who held them?


• What different and distinct things happened in the 50s, 60s, 70s, 80s, 90s, and noughties?


• What was the difference between the keypunch operator and the computer operator?


• What was unglamorous about programming in the 40s, such that women were allowed to do it?


• What factors led to women being crowded out of programming as a profession?


• Why did system administration come into being as a job family, and why was it minimally viable to separate that from programming?

• 6 Ways a Culture of Communication Strengthens Your Team’s Resiliency
  Jaime Woo, Shopify
• Dynamic Documentation in 5 minutes
  Daniel Lawrence, LinkedIn
• Resource management and isolation, the non-shiny way
  Luiz Viana, Demonware
• Collecting metrics with Snap - the open telemetry framework
  Guy Fighel, SignifAI
• Decentralized Data
  Jason Koppe, Indeed

• If you need to explain your product/service/etc. to users: Chances are, the most satisfying, enjoyable, and rewarding part of your job is engineering work—creating a tool, fixing a problem, redesigning infrastructure, etc. It *isn't* answering the same questions from users over and over. → Get solid documentation in place to free up engineer time.


• If your team internal documentation is a mess: It might be hard to find the docs you need when you get paged, hard to identify current content, or some information might just be flat-out missing. → A TW can help you whip a documentation rat's nest into shape, and give you the tools to maintain your docs easily moving forward.


• If you want to make your work more visible (so other people can leverage it and learn from it): → A TW can help you get that information out there!

• How to measure the quality of APIs you provide and consume.


• How to interpret the telemetry data that is emitted from the systems you are running.


• How to aggregate metrics from single nodes to service-level views.

• improve availability and reduce risks,


• save time and money,


• improve operability,


• and enable us to move fast for the long term


• we should build and run Intercom using the smallest sensible set of core infrastructure components.


• we are cautious about adding new technologies to the mix


• we’d often rather consider using one of our existing/established technology components and write (and maintain) more software ourselves, rather than taken on the overhead of learning and maintaining expertise in a new / more powerful technology.


• where tools, systems or workflows are required to support building/managing Intercom, especially in areas that could be deemed “undifferentiated heavy lifting”, we’d rather not write any software or operate any systems ourselves at all, and instead use world-class 3rd-party services.

• The challenges of monitoring a high volume, anycast, edge network across 100+ locations


• The architecture we chose to maximise the reliability of our monitoring


• Why Prometheus excels as the new industry standard for modern monitoring


• Approaches reducing alert noise and alert fatigue


• Triaging alerts into a ticket system
• Analysing past alert data for continuous improvement


• The pain points we endured


• Effecting change across engineering teams

• Engineers that do not understand your utopian visions (nobody understands me)!


• Everyone is rational, no one is right


• “This too shall pass” i.e. the team that knows you will eventually sort out the rough edges in your tooling or find another job, and can safely ignore you until then