AWS essentials

Introduction

  • Account basic overview

    • AWS free tier (for learning) different services, different free periods
    • Create AWS account: email / pass / unique account name / credit card info -> access to AWS console
  • Navigate AWS console

    • One click navigations (shortcuts): go to arrow, drag and drop what you want to the navbar
    • AWS services: user search bar
    • Resource groups (tag/label): manage production groups
    • System alerts: (know if your resources were affected by outage) list of issues, status, regions…
    • Check / switch regions
    • Support
    • Documentation

Note

  • Create billing alarm (dashboard/billing)
  • email: billing preferences -> receive free tier usage alert -> enabled
  • on cloudwatch: (left side) uder alarm -> billing -> create an alarm -> select metric (>=1$), use SNS (Simple Notificaction service)

Identity and Access Management (IAM)

  • Users / accounts /services / groups / roles / policies (you get root automatically, admin for everything)
  • Best practices (settings, configuration and architecture for high security, accesibility and availability)
    • delete root access keys (use IAM keys)
    • activate MFA (Multi-Factor Authentication)
    • create individual IAM users (by default no access)
    • use groups to assign permissions (manage accounts permissions easily)
    • apply on IAM password policy
  • Users and policies
    • create users
      • programatic access (key + secret key, no password)
      • AWS management console access (password, less secure)
    • attach existent policy (e.g. AWS S3 full access) -> easy, but bad practice: use groups!
  • Groups and policies
    • on a group several users get access to the same policies
    • a user can be part of several groups
  • IAM roles
    • services get roles: e.g. “role for EC2 to grant it access to S3”
    • set on “Roles” on IAM dashboard -> create role (to AWS service, e.g. EC2), them add pòlicy nad name it
    • “Deny” always overrides “Allow”

Network and connectivity

  • VPC (private cloud which allows access to services)
    • A private subsection af AWS
    • Management console -> “Network & content delivery” -> VPC
graph TD

A(Internet)
B[Cable/Modem]
C[Router/Switch]
D[Firewall]
E[Device 1]
F[Device 2]

G(Internet)
H[Internet Gateway]
I[Route Table]
J[Network Access Control List - NACL]
K[EC2 instance 1]
L[EC2 instance 2]

subgraph Home
  A
  B
  C
  D
  E
  F
end

subgraph VPC
  G
  H
  I
  J
  K
  L
end

A-->B;
B-->C;
C-->D;
D-->E;
D-->F;

G-->H;
H-->I;
I-->J;
J-->K;
J-->L;

Note

  • Both EC2 instance may be on the same subnet
  • You get a default VPC when you create an Amazon AWS account
  • Internet Gateways (IGW)

    • Hardware and software to connect to the Internet
    • Horizontally scaled
    • Redundance and highly availale VPC
    • Allows communication between VPC instances and the Internet
  • Route Tables (RTs)

    • VPC -> Route tables -> has routes
    • Routes can still exist if there us bo IGW (shows “black hole”)
    • Default “not local” address: 0.0.0.0/0
    • If there is no route, it doesn’t matter that we have no IGW
  • Network Access Control List

    • Default NACL on default VPC: 6 default subnets (1 per availability zone)
    • Stateless (input definition doesn’t imply output)
    • Security -> “Network ACL”
    • Rules
      • Types
        • Inbound (Internet -> Subnet)
        • Outbound (Subnet -> Internet)
      • Default: “DENY everything”
      • Matching from top/down, overrides eveything else
      • Be careful, you may cut all communication here!
  • Subnets

    • Sub-network: public/private
    • Default: subnets go to main route table
  • Availability zones (VPC specific)

    • Add subnets to availability zones
    • Utilize several for high availability (redundancy) + fault tolerance

Computer Services (EC2)

  • Elastic Cloud Compute (EC2)
    • Basics: server on cloud, scalable
    • Purchase options
      • On demand (most expensive, most efficient)
      • Reserved (period of time: pricing discount)
      • Spot (“bid for instance”, if it fits your price you got it while it is available)
      • Free tier
    • Tier variations
      • Instance type
        • general
        • compute optimized
        • accelerated computing
        • memory optimized
        • storage optimized
      • EBS oriented
      • AIM type (image = OS + extra modules)
      • Data transfer
      • Region

❗ You may compare AWS structure to a local computer structure

AWS Local PC
AMI OS
Instance Type Processor
EBS (Elastic Block Storage) Local storage
IP Addressing Network Adapter
Security Groups Firewall
RAM Memory
  • Amazon Machine Images (AMI)

    • Template to deploy EC2 instances (e.g. Linux + Apache)
    • Prevent human error
    • Categories
      • Community
      • AWS marketplace images
      • Custom
  • Instance Types (“CPU of your instance”)

    • Family (category on what they are optimized to do)
    • Type (subcategory)
    • CPU (number of virtual CPUs)
    • Memory (ammount of RAM)
    • Instance storage (memory type - harddrive: SSD, classic)
    • EBS optimized available
    • Network performance (based on transfer rate)
  • Elastic Block Storage (EBS)

    • “Block level storage for use on EC2 instances”
    • On the same availability zone as EC2 instance (consider it its hard drive)
    • IOPS (Input-output Operations per second)
    • EC2 -> instance -> select AMI -> delete storage on termination?
    • Root (EC2 must have one) + additional (you may swap it between EC2 instances, like a pendrive)
    • Snapshots (image/backup/duplicate)
  • Security groups

    • Similar to NACL: allow/deny traffic on instance level (virtual firewall)
    • Stateful (input implies output)

Note
Elastic Load Balancer (ELB) sends traffic to either subnet 1 or 2 depending on the use rates/load
If you are using an ELB, the rules on the 2 security groups must be the same

graph TD

A[Internet]
B[Gateway]
C[ELB]
D[EC2 - 1]
E[EC2 - 2]

subgraph Sec group 1
  D
end

subgraph Sec group 2
  E
end

A-->B;
B-->C;
C--NACL-->D;
C--NACL-->E;
  • IP adressing

    • Providing a public IP Address to an EC2 instance (address on the network)
    • Default: all EC2 have private IP address (communicate each other inside VPC)
    • After selecting instance type -> “auto-assign public address?”
  • Launching and using EC2 instances

    • Launch

      1. select AMI
      2. select instance type
      3. configure instance details
      4. add storage
      5. add tag (give instance)
      6. confugure/assign security group
      7. review launch
      8. create & download a key pair
    • Connect (Linux SSH)

      1. select instance
      2. under actions, choose connect
      3. follow terminal instructions
        3.1. open terminal to access command line
        3.2. navigate to directory where you donwnloaded the key pair
        3.3. run this command on thekey pair to change its permissions chmod essentialkp.pom
        3.4. run the example command provided by the AWS management console
    • Connecting to an EC2 instance with a windows PC

      • Use PuTTY Putty-gen application to convert key files
      • You may use RDP (Remote Desktop, GUI, for Windows instances)

Storage services

  • Basics

    • S3 (simple Storage Service): online bulk storage service
    • Bucket: root level folder in S3. They have regions, closer to you = fastest
    • Folder: a bucket subfolder
    • Object: file in a bucket
    • Pricing: by storage or by requests (operations: get, put, copy…)
  • Buckets and objects

    • Bucket names must be unique, lowercase, with number, hypens, and avoid IP address format
    • Use GUI: Upload/create folder
    • Properties (bucket, level and object level)
  • Storage classes

    • “classification assigned to object” (from top availability and price to lower rates)
      • Standard
      • Indifferent timing
      • Standard infrequent access
      • One-time standard infrequent access (One-time IA)
      • Glacier
      • Glacier deep archive
      • Reduced Redundancy (not recommended)
    • Classes have different cost, availability, durability (% over a year file is not lost) and frequency of access (availability: % over a year file will be accesible)
    • Each object has been assigned a class when added to bucket
    • You can change an object’s storage class
    • Settings / changing storage class
      • Default: standard
      • Change
        • Select another type during upload
        • using object lifecycle policies (glacier = only this method, takes 2 days for effect)
        • manual change (allow multipart for big files)
  • Object lifecycles

    • Rules that automate the migration of an object storage class to a different one, or its deletion based on time intervals
    • Located at bucket level, but you can decide which elements you want to change
graph LR
A[Standard]
B((30 days))
C[Infrequent]
D((60 days))
E[Glacier]

A --> B;
B --> C;
C --> D;
D --> E;
  • Permissions
    • Allow you granular control over who can view, access and use specific buckets of object
    • 2 level
      • Bucket
        • List (bucket name)
        • Upload/delete (objects)
        • Permissions (add, edit, view)
      • Object
        • open/download
        • view permissions
        • edit permissions
    • S3 permissions: “share object ith the world”
      1. on object (grantee (everyone), check (open/download)
      2. under actions: “make public”
      3. link under properties is now live
      4. remove access (delete permissions, remove bucket policy that provided public access)
  • Object versioning
    • “Feature which keeps track of and stores all versions of an object so that you can access and use an older version if you like”
    • ON/OFF, once ON you may only suspend (previous versions will still exist)
    • Set only on bucket level, applies to all objects in bucket

Database Services

  • RDS & Dynamo basics
    • RDS: SQL relational

      • very structured (tables)
      • engines available: Aurora, MySQL, MariaDB, PosgresSQL, Oracle SQL, MSSQL
      • pricing depends on:
        • on demand/reserved
        • instace type
        • RDS engine
        • DB storage
        • Data transfer to RDS
    • Dynamo DB

      • json like documents
      • no alternative engines
      • pricing depends on:
        • processed by capacity
        • indexed data storage
        • DynamoDB storage
        • reserved capacity
        • data transfer
    • Provisioning RDS (MySQL)

      • needs private subnet group which contains EC2 subnets (navigate to subnet groups and “create”, complete form)
graph TD
A[EC2 - 1]
B[EC2 - 2]
C[Route table]
D[RDS]

A --- C;
B --- C;
C --> D;
  • Launching a RDS DB

    1. select engine
    2. Specify DB details (instance specs)
    3. Configure advanced settings (network security (private subnet groups, do not create publicly accessible), DB options, backup, monitoring, maintenance)
    4. Launch
  • Connect to MySQL RDS DB

    • Download, install 3rd party app (MySQL workbench) and open app and setup connection:
      • Name it
      • Standard TCP/IP over SSH
      • SSH hostname (public IP address of EC2 to tunnel to)
      • SSH username (default user for EC2)
      • SSH Keyfile (.pem from EC2)
      • Hostname (writer cluster endpoint from RDS console)
      • Port: 3306 (important! check it!)
      • Username & pass word are stored in the keychain (they were used when you created the DB)
      • Click on “Test connection”, and if it successfull, the “Connect”

Monitoring, Alerts and Notifications

  • Simple Notification Service (SNS)
    • SNS automates sending emails and notifications (PUSH) according to events in AWS
      • topic: “group tag”
      • publisher (producer, endpoints)
      • subscribers (consumer, e.g. SQS Queues)
      • pricing: depends on number of publishers, notification deliveries, data transfer
    • Important data
      • max subscriptions per topic: 12.500.000
      • max topics per account: 10.000
      • can trigger EC2, S3, lambda
    • Using SNS
      • create a topic (name, display name for SMS, create)
      • add subscriptions
        1. select topic
        2. create subscription
        3. select protocol
        4. enter endpoint (e.g. email address)
        5. create subscription
      • publishing to topic
        1. click on “publish to topic”
        2. enter a subject and message
        3. click on “publish message”

Management tools

  • CloudWatch (what happened?)
    • “service which allows you to monitor services in AWS account”
    • Dashboard with metrics from EC2, S3, billing… Set threshold to trigger alarms
    • Pricing (depends on regions)
      • per dashboard
      • detailed monitoring EC2 instance
      • custom metrics
      • CloudWatch API requests
      • CloudWatch logs
      • CloundWatch events/custom events
    • Metrics and alarms
      • Dashboard: create -> add widget -> explore metrics -> set timer -> create
      • Alarms: create -> choose category -> explore metrics -> name + description + threshold + action (e.g. connect to SNS topic :smile:) + period statistics -> create
  • CloudTrail (who did it?)
    • service which tracks actions taken while using your AWS account (governance, compliance)
    • logs if actions happens (stored in S3 bucket)
    • pricing
      • management events (save logs in bucket)
      • data events
      • using charges (e.g. encryption)
    • create trail -> name it -> all regions? encryption? -> select events and S3 bucket for logs (create if non-existent)

Load balancing, Elasticity and Scalability

  • Elastic Load Balancer (ELB)
    • “evenly distributes traffic betwwen multiple EC2 instances”
    • takes into account availability zones and instance health
    • security groups must have the same configuration in order to be able to balance them equally
    • pricing: per hour, per GB of data transferred throuh it
    • creating an ELB (application ELB, other one is legacy)
      • Basic configuration (name, scheme internet-facing, IP address type, protocol, port, availability zones)
      • configure security settings
      • configure routing (target group with name, protocol, port)
      • configure health checks (protocol and thresholds: 5 OKs healthy, 2 KOs unhealthy)
      • register targets (where you want to serve traffic)
      • add tags, review and create
  • Autoscaling
    • “Automates the process of adding/removing instances as demand increases/decreases”
    • Autoscaling groups: collections of EC2 (can contain instances from different subnets)
    • Components
      • Launch configuration: EC2 template used when autoscaling needs to add an additional server to scaling group
      • Autoscaling group: rules + settings that govern when EC2 server is added/removed (min: 2 EC2s)
    • Pricing: free, but you need to pay for the resources provisioned
    • Using Autoscaling
      • Create a launch configuration
        1. Select AMI
        2. select instance type
        3. create launch configuration (name, public IP, optional bash script)
        4. select add storage type
        5. configure security group (check ports)
        6. Review and create
      • Create an auto-scaling group
        1. Create autoscaling groups using the launch configuration
        2. Configure details (name, number of instances, VPC and subnets you want to autoscale, Advanced: ELB + health checks)
        3. Configure policies (min, max, execute)
        4. Configure notifications (SNS topic)
        5. Configure add
        6. Review and create
graph TD
A[User]
B[ELB]
C[EC2-1]
D[EC2-2]
E[EC2-3 - added on demand]

subgraph Autoscaling group
  C
  D
  E
end

A --- B;
B --- C;
B --> D;
B --> E;
  • Route 53 (Domain and DNS - Domain Name System)
    • “Configure and manage web domains of websites and apps on AWS”
      • register domain names
      • Domain Name System (DNS) service
      • Health checking
    • Pricing
      • hosted zones
      • traffic flow (per policy)
      • latency based routing
      • geo DNS queries
      • helth checks
      • register transfer to a domain
    • Using Route 53 (A-records or address records)
      • domain registration
        1. search and select domain name available
        2. fill out contact details page
        3. review details and purchase
        4. complete and wait for “domain registration process to complete”
      • hosted ones and record sets
        1. navigate to Hosted zones and select the domain name you just registered
        2. create 2 type A record sets for your domain to route the ELB
        3. create record set, and done
    • Cloudfront
      • “AWS service which replicates data, video and apps around the world to reduce latency (speed up distribution)”
      • Set up S3 objects distribution
        1. upload data to S3 bucket
        2. go to CloudFront console -> distribution
        3. choose delivery method (e.g. web)
        4. create distribution -> origin settings -> choose S3 bucket
        5. default cache behaviour settings (forward requests, HTTP access, cache behaviour (Edge location))
        6. distribution settings: price, class, alternate domain names, distribution state
        7. Choose “Create Distribution”
        8. Ensure alternate domain names are configured in Route53 if they are set in CloudFront (alias field = ELB)

Serverless compute: lambda

graph LR
A[User]
B[Internet]
C[Route 53]
D[Lambda]

subgraph AWS
  D
end

A --> B;
B --> C;
C --> D;
  • Basics

    • “Lambda is SERVERLESS computing, forget about EC2. It executes your code and autoscales alone. You pay for the code running.”
    • Supports different runtimes, such as Java, NodeJS, Go, C#…
    • Pricing (min: 100ms)
      • requests (to execute your code)
      • duration (length of time it takes the code to execute)
      • accessing data from other AWS services/resources (trigger = CloudWatch alarm to monitor EC2 instances, so if they are shut down, they can be rebooted)
  • Test

    • How to create a lambda function
      1. select a lambda blueprint that fits your needs, or “author from scratch”
      2. configure the function (name, runtime, role)
      3. lambda function code
      4. lambda execution role (permissions to interact with other AWS services)
      5. advanced settings (ammount of memory, run on VPC, encrypt variables with AWS KMS (Key Management Service), tags, debugging and error handling, concurrency (limit scalability, audit & compliance with CloudTrail))
    • How to execute (test) a lambda function
      1. select function and click on “test”
      2. enter a “test event” (if required)
      3. click “Save”
      4. click “Test” and review the result