One of the advantages of using a Task Queue is the ability to easily parallelise work. If we are building up a backlog of work, we can just add more workers and that way, scale easily.
First, let's try to run two worker processes at the same time. They will both get messages from the queue, but how exactly? Let's see.
You need three consoles open. Two will run the (receive) funtion in the
kt-worker.lfe module. As such, these consoles will be two consumers.
$ make repl-no-deps
Starting an LFE REPL ...
Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] ...
LFE Shell V6.2 (abort with ^G)
> (kt-worker:receive)
[*] Waiting for messages. To exit press CTRL+C
$ make repl-no-deps
Starting an LFE REPL ...
Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] ...
LFE Shell V6.2 (abort with ^G)
> (kt-worker:receive)
[*] Waiting for messages. To exit press CTRL+C
In the third one we'll publish new tasks. Once you've started the consumers you can publish a few messages:
> (kt-new-task:send)
[x] Sent message '"Hello, world!"'
ok
> (kt-new-task:send '())
[x] Sent message '"Hello, world!"'
ok
> (kt-new-task:send "Here's a third message")
[x] Sent message '"Here's a third message"'
ok
> (kt-new-task:send "Here's a fourth message")
[x] Sent message '"Here's a fourth message"'
ok
> (kt-new-task:send "Here's a fifth message")
[x] Sent message '"Here's a fifth message"'
ok
>
Let's see what is delivered to our workers. Here's the output we see in the first terminal:
[x] Received: <<"Hello, world!">>
[x] Done.
[x] Received: <<"Here's a third message">>
[x] Done.
[x] Received: <<"Here's a fifth message">>
[x] Done.
And this is the output in the second terminal:
[x] Received: <<"Hello, world!">>
[x] Done.
[x] Received: <<"Here's a fourth message">>
[x] Done.
By default, RabbitMQ will send each message to the next consumer, in sequence. On average every consumer will get the same number of messages. This way of distributing messages is called round-robin. Try this out with three or more workers.
Now let's simulate the complex tasks by inserting dots into our messages. Head back to the "sending" terminal window:
> (kt-new-task:send "Message 1 ..........")
[x] Sent message '"Message 1 .........."'
ok
> (kt-new-task:send "Message 2 .........")
[x] Sent message '"Message 2 ........."'
ok
> (kt-new-task:send "Message 3 ........")
[x] Sent message '"Message 3 ........"'
ok
> (kt-new-task:send "Message 4 .......")
[x] Sent message '"Message 4 ......."'
ok
> (kt-new-task:send "Message 5 ......")
[x] Sent message '"Message 5 ......"'
ok
>
Output in the other terminals should be this:
[x] Received: <<"Message 1 ..........">>
[x] Done.
[x] Received: <<"Message 3 ........">>
[x] Done.
[x] Received: <<"Message 5 ......">>
[x] Done.
and this:
[x] Received: <<"Message 2 .........">>
[x] Done.
[x] Received: <<"Message 4 .......">>
[x] Done.
But the thing to notice here is that when sending these messgaes and checking
the output in the terminals, you should be seeing delays due to the complexity
simulation (timeouts on the (receive ...))