Distributed Systems - what should be considered

Typical Architecture

• Load balancer with multiple backend replicas
• Server with multiple backends
• Server tree

The CAP Principle

• Consistency
• Availability
• Partition Tolerance

Operational Requirements

• Configuration
• Startup and shutdown
• Queue draining
• Software upgrades
• Backups and restores
• Redundancy
• Replicated databases
• Hot swaps
• Toggles for individual features
• Graceful degradation
• Access controls and rate limits
• Data import controls
• Monitoring
• Auditing
• Debug instrumentation
• Exception collection
• Documentation for Operations

A platform may be described along three axes

• Level of service abstraction: IaaS, PaaS, SaaS
• Type of machine: Physical, virtual, or process container
• Level of resource sharing: Shared or private

Common Strategies

• Default to Virtual
• Make a Cost-Based Decision
• Leverage Provider Expertise
• Get Started Quickly
• Implement Ephemeral Computing
• Use the Cloud for Overflow Capacity
• Leverage Superior Infrastructure
• Develop an In-House Service Provider
• Contract for an On-Premises, Externally Run Service
• Implement a Bare Metal Cloud

Load Balancer Types

• DNS Round Robin
• Layer 3 and 4 Load Balancers
• Layer 7 Load Balancer

Load Balancing Methods

• Round Robin (RR)
• Weighted RR
• Least Loaded (LL)
• Least Loaded with Slow Start
• Utilization Limit
• Latency
• Cascade

General Strategy

1. Identify Bottlenecks
2. Reengineer Components
3. Measure Results
4. Be Proactive

developed by Abbott, Keeven, and Fisher

• x: Horizontal Duplication (also known as horizontal scaling or scaling out.)
• y: Functional or Service Splits
• z: Lookup-Oriented Split

Caching

• Cache Effectiveness
• Cache Placement
• Cache Persistence
• Cache Replacement Algorithms
• Cache Entry Invalidation
• Cache Size

key features for Resiliency

• Everything Malfunctions Eventually
• Resiliency through Spare Capacity
• Failure Domains
• Software Failures
• Physical Failures
• Overload Failures
• Human Error

Service Life Cycle

• Service Launch
• Emergency Tasks
• Nonemergency Tasks
• Upgrades
• Decommissioning
• Project Work

Team Member Day Types

1. Project-Focused Days
2. Oncall Days
3. Ticket Duty Days

Three Ways of DevOps

1. Workflow
   • Ensure each step is done in a repeatable way
   • Never pass defects to the next step
   • Ensure no local optimizations degrade global performance
   • Increase the flow of work
2. Improve Feedback
   • Understand and respond to all customers, internal and external
   • Shorten feedback loops
   • Amplify all feedback
   • Embed knowledge where it is needed
3. ContinualExperimentationand Learning
   • Rituals are created that reward risk taking
   • Management allocates time for projects that improve the system
   • Faults are introduced into the system to increase resilience
   • You try “crazy” or audacious things

Common Technical DevOps Practices

• Same Development and Operations Toolchain
• Consistent Software Development Life Cycle (SDLC)
• Managed Configuration and Automation
• Infrastructure as Code
• Automated Provisioning and Deployment
• Artifact-Scripted Database Changes
• Automated Build and Release
• Release Vehicle Packaging
• Abstracted Administration

delivery platform should consider

• Confidence
• Reduced Risk
• Shorter Interval from Keyboard to Production
• Less Wait Time
• Less Rework
• Improved Execution
• A Culture of Continuous Improvement
• Improved Job Satisfaction

reasons to use flag flips

• Rapid Development
• Gradual Introduction of New Features
• Finely Timed Release Dates
• Dynamic Roll Backs
• Bug Isolation
• A-BTesting
• One Percent Testing
• Differentiated Services

factors should be taken into consideration when deciding whether to pause continuous delivery

• Build Health
• Test Comprehensiveness
• Test Reproducibility
• Production Health
• Schedule Permission
• Oncall Schedule
• Manual Stop
• Push Conflicts
• Intentional Delays
• Resource Contention