Workflow Diagrams

Introduction

The whole workflow is composed of 3 layers:
  • One layer is responsible for delivering trunk to distro and handling the transaction to it. The delivery can be one or multiple components in one shot.
  • The two other layers are the 2 different workflow to deliver a work. Either a MP to a project (we handle direct commit to trunk as well that way) or a feature branch that matured for a while. In any case, this is created as a ticket entry for the first base layer.

Putting this in images:

_images/sequence-workflows.png

Components:

We can achieve all those functionalities with mainly 7 components:
  • Projects manager
  • Landing manager - Ticket manager (they do share most of their code)
  • Branch listener
  • Branch/Source builder
  • ppa assigner
  • Image builder
  • Test runner

Delivery system

Case 1: delivering a ticket to trunk, (no pre-build package), success

_images/sequence-ticket-to-trunk-success.png

Note: in case of failure in any step, the Projects manager will comment on the MP and reject it.

If direct push to trunk -> only set an error on the component and don’t enable any other landing to it.

Note 2: while monitoring the ppa or tests, we still can get signal from Projects manager to Landing manager telling “ignore this arch” or “ignore this step”. This impacts the landing manager on its view and can unblock/bypass some steps

Case 2: delivering a ticket to trunk, merge fail

_images/sequence-ticket-to-trunk-merge-fail.png

Case 3: delivering a ticket to trunk, build fail

_images/sequence-ticket-to-trunk-build-fail.png

Case 4: delivering a ticket to trunk, tests fail

_images/sequence-ticket-to-trunk-tests-fail.png

Case 5: delivering a ticket to trunk, blocked in UNAPPROVED/NEW

_images/sequence-ticket-to-trunk-blocked-unapproved.png

Case 6: delivering a ticket to trunk, blocked in proposed

Note: the package is in the archive at this moment. There is no way to backout the change, so the merge needs to go in one way or another. As the component is blocked in proposed, there is no advantage of unblocking the queue as blocked in proposed means that further unrelated landings will still be blocked in proposed. It will continue blocking potentially other unrelated packages (if a transition happens). The only way then is to unblock the package.

variant a: fix in the same component itself

_images/sequence-ticket-to-trunk-blocked-in-proposed-a.png

variant b: fix outside the CI system: in another component or by a direct upload to that component

_images/sequence-ticket-to-trunk-blocked-in-proposed-b.png

Workflow I: direct MP/trivial commit to one component (low-cost ticket)

Reminder: this is the case of a quick bug fix/feature (< 2 commits). The main case is manual ticket landing.

Workflow I A: direct commit to trunk or MP, all infos set

_images/sequence-direct-commit-to-trunk.png

Workflow I B: direct MP, not following project rules (no commit message/enough approver)

_images/sequence-direct-mp.png

Workflow II: opening a ticket

Notes: ticket manager and landing manager shares most of their code. Only the order and some interface changes.

Workflow II A: feature branch/transitions/fix involving multiple components (opening a first class ticket)

_images/sequence-ticket-mutiple-components.png

Components versus number of instances:

Please note that all services are NOT running at the same time. This diagram is just to show what is spawn by what and what takes care of one or multiple components.

Case delivering to trunk:
  • one ticket A with mir, libunity-mir, platform-api
  • one ticket B with libunity-mir only
  • one ticket C with mir
  • one ticket D with unity8
  • one ticket E with unity-scope-home

First landing: Ticket A, D and E:

_images/sequence-processes-first-landing.png

Second landing: Ticket B (starts as soon as A is treated) and D (starts as soon as D is treated) “Treated” corresponds to “ticket removed from the queue in the different “Delivery system” cases.

_images/sequence-processes-second-landing.png

Workflow I

Note that all those processes are always running (and they only files ticket)

_images/sequence-processes-workflow1.png

Workflow II

Feature involving mir, libunity-mir, and platform-api

_images/sequence-processes-workflow2.png

Sources

Below are the sources processed at http://bramp.github.io/js-sequence-diagrams/ to produce the sequence diagrams above.

Case 1

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state
Note over Landing manager:report of successful build
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager->Test runner: test against latest proposed image
Test runner->Landing manager: results
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager–>archive: copy source from ppa to archive and monitor the transition
Note over Landing manager: Push merged branch
Note right of Landing manager: MP is going to be closed
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager: update final status
Note over Projects manager: ticket closed and removed from queue

Case 2

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package can’t merge to trunk
Branch/Source builder->Landing manager: report failure
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager:update failure
Note over Projects manager: ticket removed from the queue

Case 3

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state and FAILED
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager: update failure
Note over Projects manager: ticket removed from the queue

Case 4

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state
Note over Landing manager:report of successful build
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager->Test runner: test against latest proposed image and FAILED
Test runner->Landing manager: failure results
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager: update failure
Note over Projects manager: ticket removed from the queue

Case 5

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state
Note over Landing manager:report of successful build
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager->Test runner: test against latest proposed image
Test runner->Landing manager: results
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager–>archive: copy source from ppa to archive and monitor the transition
Landing manager->archive: seeing it’s in NEW/UNAPPROVED
Landing manager->Projects manager: update status
Note left of Landing manager: no-one looking at the NEW/UNAPPROVED package
Note over Landing manager: warning for anormal wait
Landing manager->Projects manager: update arnomal wait
Note left of Landing manager: if package rejected -> reject ticket
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager: update failure to land
Note over Projects manager: ticket removed from the queue

Case 6

variant a

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state
Note over Landing manager:report of successful build
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager->Test runner: test against latest proposed image
Test runner->Landing manager: results
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager–>archive: copy source from ppa to archive and monitor the transition
Landing manager->Projects manager: update status
Note over Landing manager: warning for anormal wait
Landing manager->Projects manager: update arnomal wait
Note right of Projects manager: ticket updated with a new commit in a MP or direct push to trunk
Projects manager->Landing manager: new req. for component X
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state
Note over Landing manager:report of successful build
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager->Test runner: test against latest proposed image
Test runner->Landing manager: results
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager–>archive: copy source from ppa to archive and monitor the transition
Note over Landing manager: Push merged branch
Note right of Landing manager: MP is going to be closed
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager: update final status
Note over Projects manager: ticket closed and removed from queue

variant b

Title: Trunk delivery
participant Projects manager
participant Landing manager
participant Branch/Source builder
participant ppa assigner
participant ppa
participant Test runner
participant archive
Note over Projects manager: new ticket in the queue
Projects manager->Landing manager: Assign branch responsability get metadata (like integration tests to run)
Landing manager->ppa assigner:ask for a clean available ppa
ppa assigner->Landing manager: give ppa infos
Landing manager->Branch/Source builder: send branch and ppa infos
Note over Branch/Source builder: prepare source package
Branch/Source builder->ppa:push source
Branch/Source builder->Landing manager: give prepared branch and version infos
Landing manager->Projects manager:update status
Projects manager->Landing manager:get possible overrides
Landing manager–>ppa: monitoring build state
Note over Landing manager:report of successful build
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager->Test runner: test against latest proposed image
Test runner->Landing manager: results
Landing manager->Projects manager: update status
Projects manager->Landing manager:get possible overrides
Landing manager–>archive: copy source from ppa to archive and monitor the transition
Landing manager->Projects manager: update status
Note over Landing manager: warning for anormal wait
Landing manager->Projects manager: update arnomal wait
Note over Landing manager: migration finally happened or newer | version in the archive (direct upload)
Note over Landing manager: Push merged branch
Note right of Landing manager: MP is going to be closed
Landing manager->ppa assigner: deprovision that ppa
Landing manager->Projects manager: update final status
Note over Projects manager: ticket closed and removed from queue

Workflow I A

Title: Direct MP or commit to trunk
participant Projects manager
participant Branch listener
participant launchpad MP API
participant launchpad bugs API
Projects manager->Branch listener: Assign branch responsability and rules (like 2 approvers)
Note left of Branch listener:CASE OF DIRECT PUSH TO TRUNK
Branch listener->Branch listener: get infos from unreleased commit message, and eventual bug reports mentioned/attached
Note left of Branch listener:CASE OF MP
Branch listener->launchpad MP API: pool for existing top approved MP
launchpad MP API->Branch listener: grab MP infos and possible attached bugs
Note left of Branch listener:END CASES
Branch listener->launchpad bugs API: grab bugs infos (like title)
launchpad bugs API->Branch listener: get those infos
Branch listener->Projects manager: open a new low-cost ticket on the component associated

Workflow I B

Title: Direct failing MP to trunk
participant Projects manager
participant Branch listener
participant launchpad MP API
Projects manager->Branch listener: Assign branch responsability and rules (like 2 approvers)
Branch listener->launchpad MP API: pool for existing top approved MP
launchpad MP API->Branch listener: grab MP infos and possible attached bugs
Note left of Branch listener: check that MP didn’t follow the rules
Branch listener->launchpad MP API:comment and set back to needs review

Workflow II A

Title: Ticket handling
participant Projects manager
participant Branch listener
participant ppa assigner
participant Ticket manager
participant Branch/Source builder (branch)
participant Branch/Source builder (trunk + branch)
participant ppa (branch)
participant ppa (trunk + branch)
participant Test runner
Note over Projects manager: new ticket opened, specifying components
Projects manager->Ticket manager: Assign branch responsibility get metadata (like integration tests to run)
Note over Ticket manager: create branches (if not existing)
Ticket manager->ppa assigner:ask for two clean available ppas
ppa assigner->Ticket manager: give ppas infos
Ticket manager->Branch listener: Set up branches to watch
Ticket manager->Projects manager:update status
Note left of Ticket manager: REPEAT ON ALL REQUESTS
Note over Branch listener: See workflow I A: see other diagram same apply but on diverged branch
Branch listener->Ticket manager: send incoming request
Ticket manager->Branch/Source builder (branch): send branch info
Note over Branch/Source builder (branch): prepare source package
Branch/Source builder (branch)->ppa (branch):push source
Branch/Source builder (branch)->Ticket manager: give ack back
Ticket manager->Projects manager:update status
Projects manager->Ticket manager:get possible overrides
Ticket manager–>ppa (branch): monitoring build state
Note over Ticket manager:report of successful build
Ticket manager->Projects manager: update status
Projects manager->Ticket manager:get possible overrides
Note over Ticket manager: Push merged branch
Note right of Ticket manager: MP is going to be closed
Note left of Ticket manager: END REPEAT
Note left of Ticket manager: (in parallel for all requests)
Ticket manager->Branch/Source builder (trunk + branch): send branch info
Note over Branch/Source builder (trunk + branch): prepare source package (merged with trunk)
Branch/Source builder (trunk + branch)->ppa (trunk + branch):push source
Branch/Source builder (trunk + branch)->Ticket manager: give ack back
Ticket manager->Projects manager:update status
Ticket manager–>ppa (trunk + branch): monitoring build state
Note over Ticket manager:report of successful build
Ticket manager->Projects manager: update status
Note left of Ticket manager: END REPEAT
Note over Ticket manager: EVERYTIME NOTHING IS BUILDING AND: * A NEW COMPONENT LANDED * OR A NEW AVAILABLE IMAGE * OR PROJECTS MANAGER CONFIG CHANGED IMAGE#
Ticket manager->Image builder: send request for a new image
Image builder–>ppa (branch): build IMAGE A (branch with fixed image num)
Image builder–>ppa (branch): build IMAGE B (branch with latest image if different num)
Image builder–>ppa (trunk + branch): build IMAGE C (trunk merged with latest image num)
Image builder->Ticket manager: report and publish isos
Ticket manager->Projects manager: push infos
Ticket manager->Test runner: test against available images
Test runner->Ticket manager: results
Ticket manager->Projects manager: update status
Note left of Ticket manager: END TESTING
Ticket manager->Projects manager: request for more additional constraints (design review, packaging changes)
Projects manager->Ticket manager: additional ack by release team or design or core-dev
Note over Ticket manager: LANDING button only available if IMAGE 3 built, req. fulfill and tests pass Then go to landing manager case.
Ticket manager->ppa assigner: deprovision the (branch) ppa