A Queue in Core

Executive summary

There has been on and off discussion I’ve seen about the benefits of adding a Queue system to TYPO3. I agree with them in general, so I decided to do some research on what would be involved and what the options are.

I surveyed 13 PHP libraries sourced through the PHP grapevine. I then made a specific attempt to make use of two of them: Symfony Messenger and Enqueue. My throw-away noodling-about code can be found here:

https://github.com/Crell/queue-test

Based on that research, I see three options for how to proceed.

Enqueue

Enqueue is a dedicated and robust Queue library. It is built as a Queue from the ground up. It has widespread use, although it is in maintenance-only mode at this point. (Arguably it is feature complete, so that is not a major drawback.) It exposes its queue-ness front-and-center, for better or worse. On the upside it is the most robust queue-centric option available, and has fairly good documentation (although there are still a few gaps). On the downside, some of its design decisions are quite odd, and undesirable. In particular:

  • Although it includes a Doctrine DBAL driver for a backend, its mechanism for hooking it up is rather clunky, undocumented, and frankly bizarre. There would be some contortions to connect it to our existing Doctrine connection, although those would be hidden away from extension developers.
  • Its messages are bare strings, not classed objects. Presumably that is helpful if you want cross-language support, but as that is not relevant for TYPO3 that is all down-side, and a major downside at that.

Enqueue has the most complete and robust set of support for different queue backends, and many other queue systems bridge in whole or in part to Enqueue’s available backends (called “Transports”) to avoid duplication.

Symfony Messenger

Symfony Messenger is first and foremost a low-end Command Bus. I say low-end here because it lacks most of the built-in functionality and robustness of tools like Broadway, Prooph, Ecotone. Its closest competition in the PHP space would be Tactician from the PHP League. However, it has plugins available that allow command messages to be deferred via a queue backend (which it also calls Transports). However, I find its design for doing so quite clunky and suboptimal.

Messenger is widely used within Symfony applications, and while it can be used on its own I do not have any data on how often it is used outside of Symfony.

Messenger’s documentation for use within the Symfony framework is fairly good. Its documentation for using outside of Symfony is decidedly poor. I was able to figure out how to run it thanks to help from Symfony’s documentation lead, and once you understand how it works it’s fairly tractable, just not at all self-evident.

The queue portion of Messenger includes a Doctrine transport as well as a number of others, although not as many as Enqueue does. It also has Enqueue bridges available separately to use Enqueue’s transports in some cases.

  • Messenger’s setup is, in my view, over-abstracted. However, most of that would be/can be insulated from extension developers.
  • Its worker script is not as robust as Enqueue’s. We would potentially need to do more work here to build a CLI tool for background running, although the level of effort is non-zero in either case.
  • On the plus side, Messenger uses objects throughout its design, so Messages are classed value objects. That is a much nicer developer experience.

Using Messenger would have the additional result of being available as a command bus, which is a useful architectural style on its own. Unlike the more robust libraries, it doesn’t ask you to redesign your whole application around it. Making a command bus available may help with on-going efforts to better separate out code within the system.

Effectively, whether we would use Messenger as a command bus and getting queue support as a nice bonus or use it as a queue system and get a command bus along the way for free is a matter of perspective, but we would get both, even if not best-of-breed of either one.

Rolling our own

Writing our own queue system is always an option. It is not, I would argue, a good option in this case, as the problem space has a lot more gotchas than are readily apparent. At bare minimum, if we took this approach we should use the Transports from Enqueue rather than trying to write our own drivers for RabbitMQ, SQS, etc. This would also take the longest.

On the upside, we could of course purpose-build the exact API developer experience we want. On the downside, we’d have to argue about what API and developer experience we want, which would no doubt take additional time.

Recommendation

The state of PHP’s queue support is sadly rather poor at this point. Efforts to form a PHP-FIG working group to improve it were unsuccessful, mainly due to lack of interest from the major players. The unfortunate reality is that there are no great options at the moment.

Given that, I believe the least-bad option would be to integrate the Symfony Messenger component into TYPO3. We already use a number of Symfony components, which reduces the additional dependency weight. It also gives us the benefit of both a command bus and a queue backend, even if not the best version of either.

The out-of-the-box configuration should use Doctrine DBAL as a transport, although a way to switch it to some other service should be well-documented.

Additionally, cron should, by default, try to run queued tasks so that administrators do not need to take any extra steps to have the queue “just work” for low-end use cases. However, a simple toggle should disable that behavior. A provided alternative persistent worker script CLI command should be the recommended alternative to use if possible.

Background

For the purposes of this discussion, a Queue system is a library or subsystem that would allow either extensions or core code to send instructions to a separate process that will take a potentially-asynchronous action with no return value (that is, no Promise object). The use cases are for actions that need to be taken but their result is not necessary for handling the current request, or cases where some process has an extremely large number of similar subtasks that can be handled asynchronously from the current request. The canonical example is emailing a large number of subscribers when a particular piece of data is updated, but there are numerous others.

In the typical case, messages are enqueued by the system to some queue storage server. A separate, persistent process then listens for items in the queue (the mechanism may vary) and processes them in FIFO order. Importantly, running multiple of these “workers” in parallel is both possible and mundane to allow for easy horizontal scaling. However, more low-end implementations can simulate that process with a relatively frequent cron task. (Some frameworks recommend running cron every minute as a form of pseudo-queue, but that is a generally bad idea. If tasks need that frequent an update, using a proper persistent queue runner is greatly preferred.)

Many dedicated queue servers exist, both those that can be installed locally (RabbitMQ, ZeroMQ, etc.) and 3rd party services (Amazon SQS, IronMQ, etc.). It’s also possible to use many general purpose datastores as a queue store through polling mechanisms (Any SQL database, MongoDB, Redis, etc.), and some have specific queue-supporting features. In short, there are ample options we would need to support and abstract over.

In particular, since not everyone will want or need to install a dedicated server for queue tasks we must have a default of storing queued items in the main SQL database and allowing tasks to be processed from a cron job if no other worker is running. That is, from an extension author point of view it “just works” out of the box, and as needed can be cleanly separated to a persistent worker command and/or a dedicated server without modifying extension code.

Command and Event busses

Command Buses, Event buses, and CQRS Command/Request buses are a separate but often related topic to queues. They already provide a clean separation between the requesting side and the processing side of a request, and communicate with a defined message object of some sort. Inserting a delay into that connection via a queue server is an obvious add-on, and many such libraries do exactly that. As a result, many of the more robust queue options in PHP today are actually add-ons to Command buses, some of them more hacky than others.

Command bus libraries, however, have their own complexity independent of the queue system. They may have considerable additional machinery on the assumption that they’ll be used for CQRS, EventSourcing, or other such architectural designs. If those designs are of benefit on their own, that’s great, and the ability to defer message handling in some cases is icing on the cake. If not, then the extra machinery is mostly dead weight to make debugging harder.

I considered a few Command bus libraries for this analysis, but the more robust ones I determined to be out of scope at this time. Adopting a high-end CQRS or EventSourcing library would have its benefits, and queue support would be one among many, but that is a far larger discussion with much deeper implications, including a host of backward compatibility questions.

Raw research notes

The following libraries are listed in roughly the order in which I looked into them. This list was primarily collected through the tried-and-true research method of “asking for suggestions on Twitter.”

Lines marked with a @ are informational, or neutral.
Lines marked with (+) are positive
Lines marked with (-) are negative

Queue-Interop

@ GitHub - queue-interop/queue-interop: Promoting the interoperability of message queue objects.
@ MIT license
@ Highly engineered version of what a PSR would be if it had been done in FIG
@ Based on the Java equivalent (JMS)
(+) 76 Packagist dependents, nearly 12 million installs (+)
(-) Most of the dependents major seem to all be parts of Enqueue, from the same author
(-) Many of the dependencies are forks of Enqueue. Hm.
(-) No stable release. Latest tag is 1.0 alpha 2, but author says to use 0.8.1?
(+) 0.8.1 branch lists PHP 8 support.
(-) Author says it’s unmtaintained: Still maintained? · Issue #39 · queue-interop/queue-interop · GitHub

Enqueue

@ https://php-enqueue.github.io/
@ MIT license
(+) 7 million Packagist installs (Meaning over half of queue-interop’s installs are Enqueue)
@ Based on the Queue-Interop project; same author, it’s the reference implementation
@ Has a Giter chat room
(+) Seems robust
(-) Only supports string message bodies. (Does not auto-serialize more complex values.)
(+) Supports lots of different backends, including Doctrine DBAL
(-) Some of the backends are reporting failed tests, according to GitHub
(+) Has had recent updates
(+) PHP 7.3+, including PHP 8.0+
(+) Existing bridges for Laravel, Symfony, Magento2, and others
(-) Author says it is in maintenance mode only

PHP-Message-Queue

@ GitHub - bozerkins/php-message-queue: a simple message stack on php using files
@ MIT license
(-) Clearly abandoned; no stable releases, no code changes in 4 years.

Symfony Messenger component

@ The Messenger Component (Symfony Docs)
@ MIT license
@ 19.1 million installs on Packagist (what percentage of those are Symfony apps vs not is unclear)
(+) Well-maintained by a known entity
(-) Documentation mostly assumes you’re using Symfony, not just the component
(+) Supports many transports, including doctrine DBAL with Postgres additions, Redis, SQS, in-mem, etc.
(+) Extremely flexible. Which could be a downside because that means a lot of stuff to configure
(-) Wiring it all up for non-Symfony use is likely to be a not-small task because of all the abstraction that assumes you have FrameworkBundle’s magic. Documentation for using outside of Symfony is almost non-existent.
@ Uses an “envelope” around each message to carry metadata from one middleware to another, if necessary.
@ The 5.4 version has 5 Symfony dependencies for production that aren’t always needed (eg, doctrine-messenger, amqp-messenger, deprecation-contracts, etc.). 11 more for dev. The 6.0 version fixes this and is down to just psr/log, which is good.
(-) Currently has unnecessary soft-dependencies on the EventDispatcher component rather than PSR-14. I reported it and it’s been fixed in 6.1, and there are workarounds possible for earlier versions.

Swarrot

@ GitHub - swarrot/swarrot: A lib to consume message from any Broker
@* MIT license
(+) PHP 8-friendly. Appears to be maintained.
(-) Only handles the consumer side, not the producer side. We need both.

Castor

@ GitHub - castor-labs/queue: A simple queue abstraction for your PHP projects
@ MIT license
(+) Small and easy to setup
(-) Very little usage. Probably just the author. :slight_smile:
(-) Only supports strings as messages.
(-) Very basic

Tactician

@ https://tactician.thephpleague.com/
@ MIT license
(+) Managed the PHP League, a known entity with high standards.
(+) 5.8 million Packagist installs
(+) PHP 7.4 and up
@ Mainly a message bus, with a queue middleware plugin.
(-) The queue plugin is… 6 years old and has no stable releases. Latest is 0.6.0 from 6 years old.
@ Has a 3rd party bridge to link to Enqueue
@ Lots of configuration options, for good or ill.
(-) Ross Tuck (Tactician maintainer) recommends against using queue plugin, and message buses generally for queue interfaces. (cf: https://twitter.com/rosstuck/status/1511083883160293377)

Bernard

@ GitHub - bernardphp/bernard: Bernard is a multi-backend PHP library for creating background jobs for later processing.
@ MIT license
@ Dedicated queue library
@ 1.2 million downloads on Packagist
(+) Latest commit is less than a month ago, handling PHP 8.1 support.
(-) Latest actual release is 0.13.0, 3 years ago.
(+) Can route messages to different queues based on many factors, including class name.
(+) Drivers available for many backends, including Doctrine DBAL, Redis, AppEngine, SQS, etc.
@ Has Symfony CLI command integration for making quick CLI runners.
@ Has its own serializer and envelope system.
@ Has existing bridge libraries for Symfony, Laravel, and Silex(!)

PHP-FPM-Queue

@ GitHub - makasim/php-fpm-queue: Use php-fpm as a simple built-in async queue
@ MIT license
@ 106 installs on Packagist
@ More of a hack to use PHP-FPM as a pseudo-queue in memory
(-) Chews up the FPM worker pool
@ Uses the Queue-Interop interfaces
(-) No stable release; latest tag is 0.1.2 from 2018
(-) More of a “because we can” demo than a real system.

Laravel Illuminate Queue

@ GitHub - illuminate/queue: [READ ONLY] Subtree split of the Illuminate Queue component (see laravel/framework)
@ MIT License
@ 13.9 million installs (unclear if that includes Laravel full framework installs or not)
(+) Well-maintained by a known entity
(+) Requires PHP 8.0.2+
(-) No standalone docs, only in the main Laravel framework docs.
(-) Includes drivers for SQS, Redis, Beanstalkd, but NOT Doctrine DBAL. (There is a synchronous driver for testing.) There is a “database” driver, but that’s for Laravel’s DB API. We’d have to write one for Doctrine.
(-) Has 8 dependencies on the rest of Laravel, plus 2 3rd-party (Symfony Process and Ramesey/UUID). That includes its own Container, among other things. It’s really not intended for stand-alone use.
(-) Uses arrays for the payload, not an object.
(-) Seems to use lots of magic traits.

Yii2 Queue

@ GitHub - yiisoft/yii2-queue: Yii2 Queue Extension. Supports DB, Redis, RabbitMQ, Beanstalk and Gearman
@ BDS-3 license
@ 3.9 million downloads on Packagist
(+) Recent activity suggests it’s still maintained.
(+) Uses objects for messages, not arrays.
(-) Master branch build is failing, though.
(-) Requires superclosure. What?
(-) PHP version 5.5 and up. That’s… old.
(-) PHPUnit 4.4. What?

PHP Resque

@ GitHub - resque/php-resque: An implementation of Resque in PHP.
@ MIT license
@ 415,000 Packagist downloads
(-) Listed as unmaintained as of 2020
(-) Redis only
(-) PHP 5.3+

Ecotone

@ GitHub - ecotoneframework/ecotone: Ecotone Framework is Service Bus Implementation. It enables message driven architecture and DDD, CQRS, Event Sourcing PHP
@ 48,9000 Packagist installs
@ MIT license
(+) Requires PHP 8.0+
(+) Only two notable dependencies, ramsey/uuid and friendsofphp/proxy-manager-lts.
(+) Very extensive documentation with tutorials
(-) Mainly an EventSourcing/CQRS tool, not a queue system specifically
@ Driven/configured through PHP Attributes
@ Its Doctrine/queue functionality appears to make use of Enqueue? Or at least Enqueue’s connection handling.

Excluded EventSourcing tools

These tools provide EventSourcing/CQRS/CommandBus functionality for PHP, which can often have a queue-based component to them. I did not evaluate them at this time, as a CQRS bus is a much larger topic worthy of its own consideration. In short, “out of scope.”

Prooph: https://getprooph.org/

Broadway: GitHub - broadway/broadway: Infrastructure and testing helpers for creating CQRS and event sourced applications.

Symfony experiments

The Symfony docs are characteristically awful when it comes to using Messenger outside of Symfony itself. Or for how the system is put together. In fact, the graphics for the architecture are actively misleading.

How it actually works is that a message is wrapped into an Envelope along with “Stamps” (markers). Then it’s passed through a series of middleware steps, which are entirely arbitrary and can terminate the chain at any time by just returning, rather than calling the next step in the middleware chain. Middleware steps that want to avoid being run multiple times must include a stamp on the envelope after they’re done to avoid running processing a message multiple times. (More on that in a moment.)

One of the middlewares is HandleMessageMiddleware, which is usually last. Its job is to unwrap the message and delegate it to a handler (callable). The handler is actually derived using a HandlersLocator, which is pluggable. Common versions of it include a k/v map on the class name and magic derivation (eg, Foo is handled by FooHandler, etc.). Although it’s usually last, it doesn’t have to be and will dutifully call other middleware if there are any.

Another middleware is the SendMessageMiddleware, which will “send” the message to a queue backend (or, really, anything that implements a SenderInterface) and then, importantly, terminate the pipeline. It also has a Locator to map to the right Sender, which is swappable. The most common one maps a Message class name to a container service ID. Each queue backend (Redis, Doctrine, RabbitMQ, etc.) has its own Sender implementation.

A Receiver is a class that pulls messages out of a queue and puts them back into the bus from the start. Usually, a Sender and Receiver are implemented together in a single class, called a Transport. A Transport is just a Sender + Receiver.

The SendMessageMiddleware adds a stamp to the Envelope before it sends it to the Sender that says “I already did this.” That means when the message is passed through the bus a second time, SendMessageMiddlware will ignore it and let it continue on to the next middleware, which is usually (but doesn’t have to be) HandleMessageMiddleware.

To reiterate, any middleware that are listed before SendMessageMiddleware must account for the fact that any run-immediately message will go through the pipeline once, but any deferred-to-a-queue message will run through the first half of the pipeline twice (anything before SendMessageMiddleware). It’s up to the middleware what “account for” means, but usually it means adding its own custom “I’ve seen this” stamp.

Configuring how to map messages to handlers and messages to transports/senders/queues is mostly the job of the various Locators, which can be configured via injected DI properties or whatever else. This is where the TYPO3 glue will go, most likely.

Once values are in the queue, something needs to pull them back out to re-insert into the pipeline. That is the Worker class, which takes a list of Receivers to check (again, usually a combined Transport object) and the bus to pull from, plus some other details. Its only job is to pull stuff out of the corresponding queue and toss it into the bus again for a second round of processing. It’s designed to run in an infinite loop as a CLI command.

In particular, the Doctrine Transport can self-initialize the tables it needs, including having multiple queues in one table or separate tables. However, the mechanism to do so is entirely undocumented and non-obvious. It also involves a number of layers of indirection to wrap a connection into a Messenger connection wrapper, then into a Transport, which is then registered as a container service, and then message classes may be mapped to it. It is ugly, but once setup is entirely insulated from most end-users.

Overall, Messenger is not a queue system. It’s an over-abstracted message bus with a queue hack bolted to the side. It works, but it’s definitely not purpose built as a queue. (A purpose-built queue system, I would argue, would be built the other way around; assume all messages go to a queue and architect around that, with one of the queues just happening to “execute immediately.”)

In its defense, Messenger does offer a number of queue-related features via stamps and extra middlewares, if properly configured. For instance, when sending a message you can include a stamp to delay running it for some period of time, or have a separate queue for messages that have failed to get re-tried or analyzed as “these failed, why?” There are also various other middleware included that may or may not be useful.

The Worker, Sender, and Receiver classes currently optionally depend on the Symfony EventDispatcher rather than PSR-14. There’s no architectural reason they have to, and it was most likely an accidental oversight. I reported it and it has already been fixed in 6.1, but likely won’t be backported to 5.4/6.0. There is a workaround available, however. (cf: https://github.com/symfony/symfony/issues/45963 and https://github.com/symfony/symfony/pull/45967) The 5.4 version also has a number of additional dependencies that became optional in 6.0, mainly all of the supported transports.

Thanks to Symfony’s Ryan Weaver for his help in explaining how all of this works.

Enqueue experiments

The queue-interop project was a non-FIG attempt by the author to, essentially, port Java’s JSR 914 (its queue spec) to PHP. Enqueue is the reference implementation. Queue-interop never really caught on, and the maintainer now considers it abandoned. However, some of its transports (queue backends) have bridges to other queue systems, including Symfony Messenger. At this point, we should view queue-interop as a historical artifact that happens to have a lot of transports built for it, but that’s about it.

Enqueue’s documentation is more complete than Symfony’s, though still not ideal. It’s also built as a queue, specifically. That means many queue-related features – such as pub/sub for multiple receivers from one message, delays, TTL, etc. – are first-class citizens in Enqueue’s API whereas they’re sideways supported in Messenger via stamps. (Whether or not TYPO3 would use them often enough for that to matter is debatable.)

What is missing in the documentation is knowledge of some less-used parts of Doctrine. You can have Enqueue’s Doctrine transport make its own Doctrine connection, or use Doctrine’s ManagerRegistry interface… which Enqueue provides no documentation for using, and neither does Doctrine, and in fact Doctrine has no full implementation of that interface at all, just an abstract class. From what I’ve been able to determine, ManagerRegistry is an ancient alternative to having a DI Container that has never been jettisoned; why Enqueue is using that, rather than something that leverages PSR-11 I do not know, but it will make integrating with TYPO3/Doctrine DBAL more challenging.

Enqueue also has a first-class API for processing a single message and then continuing, in addition to a separate persistent runner. (Symfony may be able to do this, but it’s either buried or undocumented or both.) It also has a built-in “run only for a certain amount of time” flag, whereas Symfony needs a Symfony-specific event that gets tossed into the event dispatcher as a way to sneak a message into it. Enqueue definitely has the stronger runner here.

On the downside, Enqueue is much less maintained than Symfony Messenger. According to the author it’s in maintenance mode only, and while he’ll accept PRs from others he isn’t actively developing it. (That may or may not be necessary; it is a stable library at this point, so it’s unclear what new features would even be added.)

Enqueue also has no built-in “just run it now anyway” mechanism, and it’s unclear how to write one. That is arguably by design, since it’s a queue, not a message bus, but it is a factor to consider.

The other major downside is that its message body payload only supports strings. It cannot do its own serialization, so you have to serialize messages to strings first, and then manually decode them from strings in the consumer. This is a major usability flaw, in my view, and makes the otherwise reasonable (if somewhat convoluted in places) architecture much less reasonable. The serialization/deserialization really should be insulated from the user.

The likely reason for that is to support producers and consumers in different languages, so PHP’s serialized format would not work. However, there are ample ways to make that more portable if necessary, which Symfony Messenger does. This is also not a benefit that is relevant for TYPO3 so we get no value from this trade-off.

Discuss

4 Likes

Big general “yes” to queues from me! Would be perfect for delayed reference index updates.

Symfony Messenger is a much-used system, so +1 for easy onboarding. Otherwise no opinion on the choice of library.

Thank you for this great elaboration. The days I got hold of this feature at CraftCMS and I was thrilled. So in general definitely +1.

Since we use a lot from symfony for the components I would stay there for the queue as well.

For comparison: CraftCMS uses the Yii2 queue.

Thanks for sharing the differences between the available options.

I think a Queue system is a really good addition for TYPO3 v12. I do think we need a Doctrine DBAL adapter (combined with a poor-mans cronjob or something along the lines), but I think we would need to provide our own DB adapter and I wouldn’t depend on a solution provided by the framework already. So as long as the solution is flexible to allow DB adapters, we’re good on that side.

Two major angles I do have on this topic are:

  1. We should first collect use-cases, which TYPO3 Core could use and demonstrate. Of course, the “send an email to a user” (= notifier) is a good case in general, however we don’t currently have a “mass send out an email” feature in TYPO3. An example such as the Reference Index Updater as mentioned by @mabolek already exists via the CLI Task. The only thing I can imagine is when a full workspace gets published or deleted (that could run into a memory_limit issue), or when importing or exporting data using EXT:impexp. However, the key here is probably not the choice of queue framework, but rather the UX (how the user gets notified when his/her import is done). So yes, let’s go with a queue, but also define use-cases (let’s say 3-5) that we implement along the way until v12 LTS to demonstrate when the queue system should be used rather than a scheduled tasks. One could be to execute the frontend after a page content gets modified.

  2. Although you explicitly state that a Command / Event Bus isn’t really “in scope”, I guess that would be highly beneficial to consider such use-cases as well? If we treat our “DataHandler DataMap/CommandMap” array as real commands, we could benefit from better validation, abstraction, extensibility and consistency in the DB?

All in all it boils down to the use case in our application framework to choose the technology, but more on the “use cases” we have, if we want a queuing system as a “job queue” or as part of a “command bus”. @ohader already has built something to prepare for CQRS (a.k.a. DataHandler v2) somewhere on GitHub, and I think @susanne.moog has built a job queue for v11 somewhere as extension. maybe they could add them somewhere in this thread?

Thanks again for the massive write-up.

2 Likes

A few potential queue use cases off the top of my head that TYPO3 could use:

  • Search reindexing. When a page or pages are updated, don’t reindex them immediately. Toss the ID into a queue and let it get reindexed as the queue gets to it. Benefit: Reindexing never takes up time in the UI, even if reindexing the entire site.
  • Cache warming. Similar to the previous point, if a series of pages (or anything else for that matter) need to be cache warmed, especially after they’ve been cleared, that is better done in a queue. Potentially, if the cache is of a type that tolerates stale data for a short time, the cache entry needn’t even be cleared in advance; just let the existing cache entry continue to be served until the queue worker updates it. (Would only work for things like a page cache, but still potentially useful in some cases.)
  • Mass notification. Core doesn’t have a mass-email feature, but extensions can certainly provide it by hooking onto the “page saved” event (or something else, take your pick). A queue would be the correct way to handle that, even if only notifying a relatively small number of people (eg, a dozen editors or so).
  • Immediate batch tasks. This is a place where Drupal had a queue but never used it, to its detriment. If running some high-volume task where you do want it to run immediately, but doing it in a single process is too much (eg, publishing or unpublishing a hundred pages at once, enabling a dozen extensions, etc.), you want a batch process system. While there are many ways to implement such a system, I believe the best is to first populate a queue, then have a separate UI page that polls a process that just works through the queue, showing some kind of progress meter. By implementing it this way, any queue could be “run immediately” through the UI this way if desired. So, for instance, if you wanted to publish 100 pages right now, you could toss them into a queue and the batch page would just sit there and ping a dedicated page to make the handler run. When the batch is done, your queue is done. If you don’t care when it happens and want to get on with your work, let the queue run “later” (which could also be very immediate, depending on details), and it will happen in the background. The processing code for the specific task is then identical.
  • CLI Batch processor. There’s little if any difference between a CLI command that runs through a bunch of pending tasks and a CLI-running worker that is waiting for a bunch of tasks and runs through them. The only meaningful difference is a flag that says “terminate the process when the queue is empty” vs “block and wait for more items when the queue is empty.”
  • Import-export. Building on the items above, any sort of import/export would ideally be done via a queue, and then could be run in the background or via a UI batch runner as the situation requires. Again, the backend code is identical.

With regards to a command bus, on the last few Symfony projects I’ve worked I’ve come to the conclusion that basically any side effect (data write) I want to happen that does not affect my ability to complete the request and might have a cardinality of >1 should go though the Messenger bus. (Even if the cardinality is =1, I often do anyway to avoid code duplication.) Basically, it’s either part of my immediate DB transaction or it’s a Message.

Even if I don’t hook up the bus to a queue at all and let everything run immediately, it gives me a built-in place that I can easily do so without any code changes. It also forces me into a message-passing mindset, which is beneficial overall for code separation. I do believe that having a command bus available would be beneficial for the overall goal of further decoupling code and breaking up old mega-classes, even if it’s not passed through a queue.

That’s another reason why I favor Symfony over Enqueue, in that we get both tools in one, and can leverage either or both as the situation dictates.

Note: This is for a Command bus, not an Event or Request bus. While Messenger has documentation for how to configure it as an Event bus or a CQRS setup, I am not as convinced that is useful at this time. It might be in the future, but Event Sourcing and CQRS are far, far deeper changes to the system that I do think are mostly out of scope at this point. That’s why I didn’t look deeply into the other libraries that are built more as a Event Sourcing or CQRS backbone that happen to have a queue on the side. That would be an entire major version’s worth of work to retrofit into TYPO3 all on its own, and I’m not yet convinced that’s even the right path forward.

I do, however, feel confident that a Command bus and readily-available queue system have immediate benefits that would not require rebuilding major parts of the system around them all at once. (It also means we could experiment with it as an Event Source or CQRS system with some careful configuration if we wanted, without committing to anything.)

1 Like

@benni I’m curious why you don’t think using the provided Doctrine adapter would be good. TYPO3 has some extensions to Doctrine connections but they should be backward compatible; I don’t see a compelling reason to make extra work for ourselves if there is already a driver read-made.

An example of a specific use case and how it would fan-out, using a command bus queue:

Toggling a page from unpublished to published. First, implement it as a Command, with a Handler. The Command is as simple as:

class PublishPage
{
  public function __construct(public readonly int $uid) {}
}

The handler for it is a service, call it PublishPageHandler, with injected dependencies (as appropriate). It takes a PublishPage instance as an argument, and does the relatively simple work of loading the page object, updating it, and saving it back.

By default, clicking the “publish” button triggers that command, and it happens immediately. If 100 pages are to be published at once, you can (through configuration or a Stamp that we custom define or something) instead queue 100 PublishPage commands. Depending on your queue setup, those could run immediately in-process, a few seconds later in a waiting queue worker, or “whever cron next runs”. Or the UI could flip to a batch processing page with the queue to process, chew through them all immediately, then return to the page listing screen.

While there is some work to make all of those viable, the core logic of “send message” and “here’s how to process the message to update the DB” is identical in all cases. That gives both core devs and site admins a lot of flexibility for how they want to handle their behavior and what set of trade-offs are appropriate. It also necessitates using clean services and messages, which helps with keeping code cleanly factored.

Regardless of which Lib/Option it will make:

TYPO3 needs this. Ever since working with other frameworks or even back then with Magento, I miss a simple native and ready out of the box queue solution for TYPO3.

1 Like

Quick note on that:

ext-hooky/SendCommand.php at 5a073256e088fbcfaff39592f4c535cb3c3b72b5 · waldhacker/ext-hooky · GitHub - It’s a very rudimentary implementation based on enqueue/dbal just because I needed “some kind of” queue. :wink:

All that goes back to 2016…

Thanks to some kind soul on Twitter (https://twitter.com/WapplerSystems/status/1516416879161716737), it looks like someone already has an experimental extension for this: GitHub - WapplerSystems/messenger: Integrates Symfony Messenger into TYPO3

I’ve only glanced at it a bit; I’d probably want to do it a bit differently for core, but there’s no doubt quite a bit of code that could be lifted from there. Or… could be, if the author puts a license on it. :frowning: (Without an explicit license, it’s illegal to use.)

Edit: License issue quickly resolved.

First, thanks for all your footwork to put the options on the table for us, really appreciated.

In general, I think a queue system in the TYPO3 Core can be beneficial. As an extension developer of the Crawler. I see some of a use case, besides the ones listed already.

The Crawler currently having a “queue” which is simply a DB table, this could fairly easy be migrated to a Core Queue, which would ease other extension developers to listen to the same queue if actions are needed/wanted on given events/queue entries. Of course, this is indirectly possible today with PSR-14, if the events exists.

The use cases already listed with Search Indexing and Caching is already hitting some purpose of the Crawler, a lot of installations with Indexed Search also have the Crawler installed for indexing. So, having a shared queue, might ease some complexity present in the implementation of today. The Crawler also ensures that the caching is kept warm, when pages are edited, they are instantly crawled and the cache is shortly after up to date again.

I don’t know how exactly the implementation would look like, but for the crawler there are some information that are important to the queue entry, but that should be possible to wrap in a payload for later use, like the Crawler Configuration for a given page.

I also just found out about dfau/ghost - Packagist based on bernard

Long ago I made a backport from the flow jobqueue package: TYPO3 Extension Repository
Personally I would opt for a solution where the core defines some interfaces as an abstraction layer but the real implementation is done by using a stable existing queue/bus system.