What is a Docker Plugin?

A Docker plugin is an executable that follows the docker-{plugin_name} naming convention. It can be
placed in the ~/.docker/plugins directory for the current user or in one of the following global directories:

• /usr/local/lib/docker/cli-plugins
• /usr/local/libexec/docker/cli-plugins
• /usr/lib/docker/cli-plugins
• /usr/libexec/docker/cli-plugins

Why do I need Buildx?

The primary reason for using the Buildx plugin is to enable multi-platform builds and parallel building
of multiple targets with the bake command. You can find more information about bake here.

What’s up with Moby?

Moby is the open-source base project of Docker. While it is not recommended as a developer tool, it is
more suitable for tinkering with Docker internals. Despite that it’s the tool of my choice since it is easier to adopt on Fedora
compared to Docker CE. This is mainly because some upcoming fixes, designed to address updates more promptly, are implemented in Moby
first. Fedora aims to provide the latest Linux packages faster than other distributions. The only drawback I have
observed so far is that all plugins are disabled by default in Moby, unlike Docker CE.

How to install Buildx

1. Download the Buildx binary into your user’s Docker plugins directory (Alternatively, you can download
   it to one of the global directories mentioned above):

   1 2	wget https://github.com/docker/buildx/releases/download/v0.10.5/buildx-v0.10.5.linux-amd64 \ -O ~/.docker/cli-plugins/docker-buildx

2. Make it executable :

   1	chmod +x ~/.docker/cli-plugins/docker-buildx

3. Check that the plugin is enabled:

   1	docker info

First build

1. Create builder:

   1	docker buildx create --name multiarch --driver docker-container --use

2. Login to your registry in my case ECR:

   1 2 3

   $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email 2> /dev/null) || aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login \ --username AWS \ --password-stdin $AWS_ACC.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com

3. Build and push to repo. Remember that multi-architecture build make sense only for pushing to a repository. Then when you pull the image back the one applicable to your host machine will be pulled.

   1	docker buildx build --platform linux/amd64,linux/arm64 --push -t "$AWS_ACC.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$REPO:$TAG" -

AWS Event Bridge and AWS Fargate: A match made in heaven

Running a command in container on demand is a powerful feature of AWS Fargate. This opens a whole world of possibilities when combined with aws event bridge. Using an EventRule you can match existing aws events or custom events you intend to send to event bridge and set an AWS Fargate task as a target.

A practical Example

To see this in practice we’ll use an example in aws CDK:

First we create a base class for our triggered constructs that is responsible for:

• Creating the event target fargate task given taskDefinition, securityGroups and containerOverrides . Task definition will let you set memory & cpu to be used, the image, the command to be run and logging. Security Groups define access . Container overrides give you the option to override the configuration and the environment of container, we ‘ll come back to this hidden gem later. Once the ecs task is created it’s assigned as a target to the provided rule.
• Configuring aws log driver. That’s useful for viewing execution output in AWS cloudwath.
• Discovering/ maintaining a default ecs cluster of the vpc(for the task to be run in) if we don’t provide a specific cluster to use

Then we create a subclass that simplifies construct creation responsible for constructing task definitions.
We could also have a scheduled task subclass that would create a cron like event rule internally.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

export abstract class TriggeredTaskBase extends Construct {
  public readonly cluster: ICluster;

  public readonly desiredTaskCount: number;

  public readonly eventRule: Rule;

  constructor(scope: Construct, id: string, props: TriggeredTaskBaseProps) {
    super(scope, id);

    this.cluster = props.cluster || this.getDefaultCluster(this, props.vpc);
    if (props.desiredTaskCount !== undefined && props.desiredTaskCount < 1) {
      throw new Error('You must specify a desiredTaskCount greater than 0');
    }
    this.desiredTaskCount = props.desiredTaskCount || 1;

    // An EventRule that describes the event trigger
    this.eventRule = props.rule;
    Tags.of(this).add('Module', getModuleName(__filename));
  }

  /**
   * Create an ECS task using the task definition provided 
   * and  adds it as a target to the event rule.
   *
   * @param ecsTaskProps properties of the ecs task
   */
  protected addTaskDefinitionToEventTarget({
    taskDefinition,
    securityGroups,
    containerOverrides,
  }: {
    taskDefinition: TaskDefinition;
    securityGroups?: ISecurityGroup[];
    containerOverrides?: ContainerOverride[];
  }): EcsTask {
    // Use the EcsTask as the target of the EventRule
    const eventRuleTarget = new EcsTask({
      cluster: this.cluster,
      taskDefinition,
      taskCount: this.desiredTaskCount,
      containerOverrides,
      securityGroups,
    });

    this.eventRule.addTarget(eventRuleTarget);

    return eventRuleTarget;
  }

  /**
   * Returns the default cluster.
   */
  protected getDefaultCluster(scope: Construct, vpc?: IVpc): Cluster {
    // magic string to avoid collision with user-defined constructs
    const DEFAULT_CLUSTER_ID = `EcsDefaultCluster${vpc ? vpc.node.id : ''}`;
    const stack = Stack.of(scope);
    return (
      (stack.node.tryFindChild(DEFAULT_CLUSTER_ID) as Cluster) ||
      new Cluster(stack, DEFAULT_CLUSTER_ID, { vpc })
    );
  }
  protected createAWSLogDriver(prefix: string): AwsLogDriver {
    return new AwsLogDriver({ streamPrefix: prefix });
  }
}

export class TriggeredFargateTask extends TriggeredTaskBase {
  // Create a Task Definition for the container to start
  public readonly taskDefinition: TaskDefinition;
  public readonly eventTarget: EcsTask;
  constructor(scope: Construct, id: string, props: TriggeredFargateTaskProps) {
    super(scope, id, props);
    if (props.triggeredFargateTaskDefinitionOptions && props.triggeredFargateTaskImageOptions) {
      throw new Error(
        'You must specify either a triggeredFargateTaskDefinitionOptions or triggeredFargateTaskOptions, not both.'
      );
    } else if (props.triggeredFargateTaskDefinitionOptions) {
      this.taskDefinition = props.triggeredFargateTaskDefinitionOptions.taskDefinition;
    } else if (props.triggeredFargateTaskImageOptions) {
      const taskImageOptions = props.triggeredFargateTaskImageOptions;
      this.taskDefinition = new FargateTaskDefinition(this, 'TriggeredTaskDef', {
        taskRole: props.taskRole,
        memoryLimitMiB: taskImageOptions.memoryLimitMiB || 512,
        cpu: taskImageOptions.cpu || 256,
      });
      const { logDriver, ...imageOptions } = taskImageOptions;
      this.taskDefinition.addContainer(props.containerName, {
        ...imageOptions,
        logging: logDriver !== undefined ? logDriver : this.createAWSLogDriver(this.node.id),
      });
    } else {
      throw new Error('You must specify one of: taskDefinition or image');
    }

    this.eventTarget = this.addTaskDefinitionToEventTarget({
      taskDefinition: this.taskDefinition,
      securityGroups: props.securityGroups,
      containerOverrides: props.containerOverrides,
    });
  }
}

Until now, it all seems straight forward. We can now create our task. In our example we run a database migration task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

new TriggeredFargateTask(this, `my-proj ManagementTask`, {
     cluster,
     taskRole: imageOptions.taskRole,
     containerName,
     securityGroups,
     triggeredFargateTaskImageOptions: {
       ...imageOptions,
       cpu: 2048,
       memoryLimitMiB: 4096,
       command: ['sh', '-c', 'python manage.py migrate'],
     },
     rule: new Rule(this, `my-proj Management Trigger`, {
       eventPattern: {
         detail: {
           subject: ['execute'],
         },
         detailType: ['Management'],
         source: [`com.corp.env.my-proj.manage`],
       },
     }),
   });

The one task to rule them all

If you are familiar with django(the above example demonstrates executing django management commands) you will know that there is a plethora of commands that you would want to run and creating a task for each command would require considerable amount of effort. The other issue is that some commands accept parameters which would be impractical to be hardcoded in task definition. Facing these issues I embarked on a journey to create the one task to rule them all.

Container Overrides to the rescue

The concept was simple. I wanted to provide parameters to the fargate task dynamically when triggering the event bridge event. Although it seemed a common use case, most of the solutions I found where suggesting setting up an sqs queue that would accept messages containing the parameters consumed at execution time, which seemed like an overkill to me. The only way I have found to pass data to the container without involving extra resources was to set environment variables from event bridge event payload via container overrides(using input transformer behavior). I actually found this by digging in some github tickets rather than finding some reference solution, neither the docs where clear on how to use the constructs to reach to a solution. Here is our revised example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

new TriggeredFargateTask(this, `my-proj ManagementTask`, {
     cluster,
     taskRole: imageOptions.taskRole,
     containerName,
     securityGroups,
     triggeredFargateTaskImageOptions: {
       ...imageOptions,
       cpu: 2048,
       memoryLimitMiB: 4096,
       command: ['sh', '-c', 'python manage.py $MANAGEMENT_COMMAND'],
     },
     rule: new Rule(this, `my-proj Management Trigger`, {
       eventPattern: {
         detail: {
           subject: ['execute'],
         },
         detailType: ['Management'],
         source: [`com.corp.env.my-proj.manage`],
       },
     }),
     containerOverrides: [
       {
         containerName,
         environment: [
           {
             name: 'MANAGEMENT_COMMAND',
             value: EventField.fromPath('$.detail.command'),
           },
         ],
       },
     ],
   });

Notice that the command changed to :

command: ['sh', '-c', 'python manage.py $MANAGEMENT_COMMAND'],

and we added the following override:

1
2
3
4
5
6
7
8
9
10
11

containerOverrides: [
  {
    containerName,
    environment: [
      {
        name: 'MANAGEMENT_COMMAND',
        value: EventField.fromPath('$.detail.command'),
      },
    ],
  },
],

The above override uses EventField.fromPath('$.detail.command') to transform the value of detail.command from the event to the value of the environment variable in the container override. Now detail.command attribute of our event can hold any arbitrary command including any parameters we want (the migration to migrate to for example) and will pass it to be executed via the $MANAGEMENT_COMMAND environment variable. This allows us to dynamically specify what we expect to run in the container.
A sample of triggering the above using event bridge event via cdk would be the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

const eventDetails = [
  {
    Detail: JSON.stringify({ subject: 'execute', command:'showmigrations myapp'}),
    DetailType: 'Management',
    Resources: [],
    Source: `com.corp.env.my-proj.manage`,
    Time: new Date(),
  }
];
eventBridge
  .putEvents(eventDetails)
  .promise()
  .then((data) =>
    console.log('Successfully triggered django management event details:\n', JSON.stringify(data, null, 4))
  )

The above would trigger showing migrations of myapp. To see the output you can check cloudwatch . You would normally accept the command as command line argument or as an argument of a function/method depending on if you are creating a CLI tool or a Library. (typically you would build a library that is used by your cli tool as well)

The above demonstrates how you can be flexible when emitting orchestration commands as events but could also be used in a choreographic scenario(system event for example) where the consuming target gets some configuration about execution from an event passed as environment variable.

Want to use an android app on your computer? Waydroid has you covered I have also had a look in anbox in the past but it seemed to be a lot of trouble to setup and sort of unstable to use(at leas on machine). Installing wydroid is straight forward while using it seems flowless to me. I have managed to get it working on my thinkpad x260 I would like to get it working on my thinkpad t14 gen3 as well but that’s nothing to blame waydroid for since my t14 has general display issues.

Installation Steps

Here are the steps I followed to set it up :

1. Make sure you’re using kernel version 5.15 or higher

2. Add the Copr repository:
   sudo dnf copr enable aleasto/waydroid

3. Update sepolicy if using fedora35 (I didn’t need to run this step)
   sudo dnf update

4. Install waydroid
   sudo dnf install waydroid

5. Initialize waydroid. To include google services flag don’t forget the gapps flag wen initializing waydroid:

   1	sudo waydroid init -c https://ota.waydro.id/system -v https://ota.waydro.id/vendor -s GAPPS

Google services registration

1. Register the output of the following command at
   https://www.google.com/android/uncertified/ :

   1	echo 'ANDROID_RUNTIME_ROOT=/apex/com.android.runtime sqlite3 /data/data/com.google.android.gsf/databases/gservices.db "select * from main where name = \"android_id\";"'| sudo waydroid shell

2. To mitigate changing registration number add a cron job:

   1 2	sudo su crontab -e

   add the following line:

   1	*/5 * * * * echo 'ANDROID_RUNTIME_ROOT=/apex/com.android.runtime sqlite3 /data/data/com.google.android.gsf/databases/gservices.db "update main set value=\"<DEVICE_ID>\" where name = \"android_id\";"'| /usr/bin/waydroid shell

   replace <DEVICE_ID> with the device id you have registered in previous step.

Use

You may now start waydroid using the desktop menu entry. It will take some time for your device to register so if you receive any google service errors reporting your device as unregistered try again later.

Sometimes you might need to perform a firmware upgrade of some hardware on your machine (Intel Management Engine, your finger print reader, your ssd disk etc).
This can be performed via fwupdmgr fwupd command:

1
2
3
4

fwupdmgr get-devices # Get all devices that support firmware updates
fwupdmgr refresh # Refresh metadata from remote server use --force if you want to refresh again without 3 hours passing
fwupdmgr get-updates # Gets the list of updates for connected hardware
fwupdmgr update # Updates all specified devices to latest firmware version, or all devices if unspecified

Keeping the snippet here for future reference hope it helps somebody.

According to python docs map build in function applies a given function to every item of an iterable, yielding the results. map returns an object that is an iterable lazily executing the provided transformation function to each element of input iterable. From the above we understand that map works pretty much like an Generator.

Why is this awesome ?

Well If you were to perform an expensive transformation on a huge iterable just to utilize a sub-part of it, lazily transforming what’s needed is the optimal approach. Let’s have a look at the following example :

1
2
3
4
5
6
7
8
9
10

def isEven(val):
    print(F'processing {val}' )
    return (val % 2) == 0

huge_range = range(1,10**24+1)

if any(map(isEven, huge_range)):
  print('Some even found')
else:
  print('No even found')

The above will have the following result:

1
2
3

processing 1
processing 2
Some even found

MyCompiler

As we can see any build in function iterates a given iterable until any truthy value is found and returns true(otherwise it returns false). The only case that our iterable would have all its elements transformed is if no even value was included in it. In our case the second element was even, so we skipped 10^24-2 transformations.

How about Javascript?

In javascript the build in method for mapping is the one of Array.prototype.map. This method is on array prototype (applicable to just arrays) and returns a new array. Since an array is returned transformations get executed all at once. How about lodash helpers? Same goes with lodash supporting just arrays. As a result it’s obvious that if we want to get something similar to python we need to implement our own functions that can work lazily using generators.

The map function

1
2
3
4
5

function* map<a, b>(a: Iterable<a>, f:(a:a) => b){
  for (const value of a) {
    yield f(value)
  }
}

The some function (similar to any in python)

As with map function we don’t have a build in or lodash function that accepts arbitrary iterables so we resort to the following:

1
2
3
4
5
6
7
8

function some<a>(iterable:Iterable<a>) {
  for (const el of iterable){
    if (el)) {
      return true
    }
  }
  return false
}

The range

Since 10^24 size array is huge and we won’t be able to actually have such an array (it’s bigger than 2^32), we ‘ll go ahead and make a generator just for demo purposes:

1
2
3
4

function* range(start:number, end:number,step: number= 1) {
  let x = start - step;
  while(x < end - step) yield x += step;
}

Putting it all together

After putting the above together we achieve the same results as with python build in functions.

1
2
3
4
5
6
7
8
9
10
11
12

function isEven(a:number){
  console.log(`processing ${a}`)
  return a%2 === 0
}

const hugeRange = range(1, Math.pow(10,24)+1);

if (some(map(hugeRange, isEven))){
  console.log('Some even found')
}else{
  console.log('No even found')
}

The above will have the following result:

1
2
3

processing 1
processing 2
Some even found

Playground Link

How about existing javascript libraries?

The only javascript library that I know of, giving similar behavior is rxjs , through it’s map and every (similar to python’s all) operators. Here is an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

import { range } from 'rxjs/observable/range';
import { every, map } from 'rxjs/operators';

function isOdd(a: number) {
  console.log(`processing ${a}`);
  return a % 2 === 1;
}

range(1, Math.pow(10, 24) + 1)
  .pipe(
    map(isOdd),
    every((x) => x)
  )
  .subscribe((allOdd) => {
    if (!allOdd) {
      console.log('Some even found');
    } else {
      console.log('All Odd');
    }
  });

The above will have the following result:

1
2
3

processing 1
processing 2
Some even found

Stackblitz

While I have been dockerizing a variety of apps I realized that python has its own special set of gotchas. Therefore I decided to write this post to list a few tips that will save time to first-timers.

PYTHONBUFFERRED: Missing standard output

One of the first gotchas you might notice is that if you are missing PYTHONBUFFERED configuration stdout/stderr messages may not be preset in terminal output in a timely manner or several lines may be entirely missing on application crash. Depending on you use-case, you may set PYTHONBUFFERED to a non-empty string so that output will be unbuffered. The above issues are solved afterwards.

Slow builds: The curse of alpine

If you try to build your app in alpine you ‘ll notice slow builds, by magnitude slower than installing apps on your local machine. This is because Standard PyPI wheels don’t work on Alpine.

Most Linux distributions use the GNU version (glibc) of the standard C library that is required by pretty much every C program, including Python. I contrast Alpine Linux uses musl and since pipy wheels are compiled using giblic they are not supported on Alpine.

Although most Python packages include binary wheels that significantly decrease install time on all alpine set-ups you ‘ll need need to compile all the C code in every Python package that you use.

To my experience the most convenient distro base image for python is debian slim. I t keeps your image small whilst not sacrificing on distributed wheel support, hence giving faster build times.

Optimal pipenv dependency setup

In first instance, using pipenv virtual environment seemed quite unstable. Issues ranged form not setting current directory in python path - to crashing shells. I am sure all the above might have already been fixed, and I already knew a few workarounds that could mitigate them. Another thing to consider is optimal image size, global setup of dependencies in an isolated image would hardly cause any issue whilst it can keep image size to the minimum. In order to install to the parent system rather than creating a virtual environment, the --system flag needs to be used. Another thing we need to take care of is to ensure lock file entries are exactly what’s installed using --ignore-pipfile. To ensure lock file is up-to date we can use --deploy which will fail the command if any dependency is outdated.

Considering the above the ideal command for most use cases I dealt with is:

1

pipenv install --system --deploy --ignore-pipfile

I am sure this is just a few of the things I could list here but I ll keep this page updated.