docs: explain all the triggers, ACM and ALB implementations, fix headings

Author: Soxasora

docs/dev/custom-domains.md

@@ -1,17 +1,16 @@
 # The Hitchhiker's Guide to Custom Domains
 Lets territory owners attach a domain to their territories.
 TODO: change the title
-TODO: add links to various documentations
 
 Index:
 TODO
 
 # Middleware
 Every time we hit a custom domain, middleware checks if it's allowed via a cached list of `ACTIVE` domains, coupled with their `subName`.
 If it's allowed, we redirect and rewrite to give custom domains a seamless territory-centered SN experience.
-###### Main middleware
+##### Main middleware
 Referral cookies and security headers gets applied the same way as before on SN, with the exception of being their own functions, so that now we can apply them also to the customDomainMiddleware resulting response.
-###### customDomainMiddleware
+##### customDomainMiddleware
 A `x-stacker-news-subname` header with the `subName` is injected into the request headers to give the SN code awareness of the territory attached to a custom domain.
 
 Since SN has several paths that depends on the `sub` parameter or the `~subName/` paths, it manipulates the URL to always stay on the right territory:
@@ -43,7 +42,7 @@ domain is PENDING
   IMPLICIT: DNS is OK
   2. Certificate Management
 	  KO: critical AWS error, schedule up to 3 retries
-		  every step can throw an error
+          every step can throw an error
 
         CONDITION: certificate is not issued
         2a. issue certificate
@@ -71,7 +70,7 @@ domain is PENDING
 It uses the `Resolver` class from `node:dns/promises` to resolve CNAME records on a domain.
 
 If the CNAME record is correct, it logs a `DomainVerificationAttempt` tied with the `DomainVerificationRecord`, having status `VERIFIED`. This resulting status is shared with the connected `DomainVerificationRecord` thanks to a trigger.
-###### dnsmasq
+##### dnsmasq
 In local, **dnsmasq** is used as a DNS server to mock records for the domain verification job.
 To have a dedicated IP for the `node:dns` Resolver, the `worker` container is part of a dedicated docker network that gives dnsmasq the `172.30.0.2` IP address.
 
@@ -85,81 +84,106 @@ The domain verification job also handles critical AWS operations, such as:
 - certificate polling
 - certificate attachment to ELB
 
-###### Certificate issuance
+##### Certificate issuance
 After DNS checks, if we don't have a certificate already, we request ACM a new certificate for the domain.
 ACM will return a `certificateArn`, which is the unique ID of an ACM certificate, that is immediately used to check its status. These informations are then stored in the `DomainCertificate` table.
 
 If we couldn't request a certificate, check its status or store it in the DB, it throws an error so that pgboss can retry the job.
 
-###### Certificate validation values
+##### Certificate validation values
 ACM needs to verify domain ownership in order to validate the certificate, in this case we use the DNS method.
 
 We ask ACM for the DNS records so that we can store them as a `DomainVerificationRecord` and present them to the user. Finally, we re-schedule the job so that the user can adjust their DNS configuration.
 
 If we couldn't get validation values or store them in the DB, it throws an error so that pgboss can retry the job.
 
-###### Certificate validation polling
+##### Certificate validation polling
 We asked ACM for a certificate, got its validation values and presented them to the user. Now we need to poll ACM to know if the verification was successful.
 
 Since we're directly checking the certificate status, we also update DomainCertificate on our DB with the new status.
 
 AWS timings are unpredictable, if the verification returns a negative result, we re-schedule the job to repeat this step.
-
 And If we couldn't contact ACM, it throws an error so that pgboss can retry the job.
 
-###### Certificate attachment to the ALB listener
+##### Certificate attachment to the ALB listener
 This is the last step regarding AWS in our domain verification job, it attaches a completely verified ACM certificate to our load balancer listener.
 
 The ALB listener is the gatekeeper of the application load balancer (ALB), it determines how incoming requests should be routed to the target server.
 
 In the case of Stacker News, the domain points directly at the load balancer listener, this means that we can both direct the user to point their `CNAME` record to `stacker.news` and that we can serve their ACM certificate directly from the load balancer.
 
-- how jobs are sent
-- on errors use the received job id to see if we couldn't verify a domain after 3 attempts, we put it on hold and delete any jobs left
-
 ### End of the job
 When we finish a step in the domain verification job, and the resulting status is still `PENDING`, we re-schedule a job using `sendDebounced` by pgboss.
 
-Since we use a singletonKey to avoid same-domain concurrent jobs, and you can't schedule another job if one is already running, sendDebounced will try to schedule a job when it can, e.g. when the job finishes or after 30 seconds.
+Since we use a `singletonKey` to avoid same-domain concurrent jobs, and you can't schedule another job if one is already running, `sendDebounced` will try to schedule a job when it can, e.g. when the job finishes or after 30 seconds.
 
 ### Error handling
 If something throws an error, we catch it to log the attempt and then re-throw it to let pgboss retry up to 3 times.
 Using the `jobId` that we pass with each job, we can know if we're reaching 3 retries using pgboss' `getJobById`. And if we did reach 3 retries, we put the domain on `HOLD`, stopping and deleting future jobs tied to this domain.
 
 ### Domain Verification logger
-We need to be able to track where, when and what went wrong during domain verification. To do this, every step of the job calls logAttempt
-###### logAttempt
+We need to be able to track where, when and what went wrong during domain verification. To do this, every step of the job calls `logAttempt`
+##### logAttempt
 This is a simple function that logs a message returned by a domain verification step in the DB.
 Some steps, like DNS and SSL verification, calls `logAttempt` by also passing the interested record in `DomainVerificationRecord`, triggering a synchronization of `status` by the result of a step.
 
 # AWS
 ### ACM certificates
-TODOs
-- on domain or domain certificate removal detach and delete certificate
-- implementation
-- implementation thinking
+We don't expect territory owners to set up their own SSL certificates but we expect their custom domain to have SSL to reach Stacker News.
+With ACM we can request a certificate for a domain and serve it via the Application Load Balancer, effectively giving the custom domain a SSL certificate.
+
+The implemented functions are non-destructive to the original Stacker News configuration:
+- Request Certificate
+- Describe Certificate
+- Get Certificate Status
+- Delete Certificate
+
+We request a certificate intentionally asking for DNS validation as it's the most reliable method of verification, and also fits nicely with the CNAME record we ask the user to insert in their DNS configuration.
+
+Describe Certificate is crucial to get SSL validation values that the user needs to put in their domain's DNS configuration; also to get the **status** of a certificate.
+
+We can only delete a certificate if we have the `certificateArn` (unique ID). In fact, when a domain gets deleted, we trigger a job that takes the `certificateArn` as parameter before we lose it forever, trying up to 3 times if ACM gives an error.
+
+In local, Localstack provides bare-minimum ACM operations and OK responses.
 
 ### AWS Application Load Balancer
-TODO
+The Application Load Balancer distributes the incoming requests across Stacker News servers. We can attach up to **25** ACM certificates (per default quota).
+
+After creating and verifying an ACM certificate, the next step is to attach this certificate to our ALB Listener, so that the load balancer can serve the right certificate for the right domain.
+
+Since in local we don't have the possibility to use Localstack to mock the ALB, there's a new class called `MockELBv2`: it provides bare minimum response mocks for attach/detach operations.
+
+As the ALB is really important to reach stacker.news, we only implemented Attach/Detach certificate functions that takes a specific `certificateArn` (unique ID). This way we can't possibly mess with the default ALB configuration.
 
 # plpgSQL functions and triggers
 ### Clear Long Held Domains
 Every midnight, the `clearLongHeldDomains` job gets executed to remove domains that have been on `HOLD` for more than 30 days.
 
+A domain removal also means the certificate removal, which triggers **Ask ACM to delete certificate**.
+
 ### Update `DomainVerificationRecord` status
 The `DomainVerification` job logs every step into `DomainVerificationAttempt`, when it comes to steps that involves DNS records like the `CNAME` record or ACM validation records, a connection between `DomainVerificationAttempt` and `DomainVerificationRecord` gets established.
 
 If the result of a DNS verification on the `CNAME` record is `VERIFIED`, it triggers a field `status` update to the related `DomainVerificationRecord`, keeping the record **statuses** in sync with the `DomainVerification` job results.
 
 
 ### HOLD domain on territory STOP
-TODO
+Let's say the territory owner doesn't renew their territory, and they have a custom domain attached to it. We can't let the custom domain access Stacker News as the domain can be transferred or out of original owner's control.
+
+A territory stop triggers a function that puts the custom domain on `HOLD`, effectively stopping the custom domain functionality.
+
+If the territory owner comes back and renews, they have to repeat the Domain Verification process just to make sure that everything is alright. The verification values will be the same and the certificate hasn't been deleted, so it should just take 30 seconds.
 
 ### Clear domain on territory takeover
-TODO
+If a new territory owner comes up, we delete every trace of the custom domain. This also deletes its certificates, verification attempts, DNS records and customizations.
+
+The main reason is safety: as we don't delete this stuff when a territory gets stopped, in hope that the original territory owner renews it, it's best to delete everything - above all, validation values. This will also trigger **Ask ACM to delete certificate**.
 
 ### Ask ACM to delete certificate
-TODO
+Whenever a domain or domain certificate gets deleted, we run a job called `deleteCertificateExternal`.
+It detaches the ACM certificate from our ALB listener and then deletes the ACM certificate from ACM.
+
+It's a necessary step to ensure that we don't waste AWS resources and also provide safety regarding the custom domain access to Stacker News.
 
 # Neat stuff