Understand SPE processing queue
This article explains details about SPE asynchronous requests processing queue, the processing task lifecycle and its handling states.
When SPE receives an asynchronous API request, a task is created and put in a queue. Size of the queue is defined by server.n_task_limit
setting in SPE configuration file, default value is 1000
tasks.
Tasks in queue are then handled according to their priority (see Task prioritization section in REST API documentation and also Understanding SPE processing priority article); handling tasks with identical priority follows the FIFO principle.
Tasks are picked up from the queue and handled by workers (which allocate instances of technologies for processing – see Speech Engine and technologies, instances, workers… explained for more details).
Number of tasks which can be handled simultaneously is defined by server.n_workers
for audio files processing and server.n_realtime_workers
for realtime streams processing settings in SPE configuration file. This is by default set automatically, based on your hardware and software configuration – see How to configure Speech Engine workers article.
The picture below demonstrates the queue processing (for the sake of simplicity, technologies assignments to workers are not shown):
- non-colored pending tasks are waiting in the queue
(task state in the response toGET /pending/{ID}
request says “waiting“) - colored pending tasks are being handled by processing workers
(task state in the response toGET /pending/{ID}
request says “running“) - SPE is configured with 3 standard and 2 realtime workers, i.e. 5 tasks are handled simultaneously
(one task has higher priority, i.e. gets handled even though it’s placed farther in the queue)
When processing is done, the task state changes to “finished” and the final result is available for pickup for a limited time only.
The time limit is defined by server.finished_task_timeout
setting in SPE configuration file, default value is 60
seconds. The limit exists so that SPE does not hoard the results, consuming memory…
Picking up the result is possible:
- using
GET /pending/{ID}
request
(which responds with HTTP status code 303, redirecting to/done/{ID}
) - using
GET /done/{ID}
request
ⓘ Depending on the used development framework (and/or its configuration), you may not get the HTTP status code 303 response because the framework handles the redirect internally.
This happens for example in Microsoft .NET, since the HttpWebRequest class has the AllowAutoRedirect property set to true
by default. Similarly, the WebAPI fetch() method or the XmlHttpRequest follow redirects automatically by default.
After picking up the final result by successful GET /done/{ID}
call, or after the time limit has passed, the result is removed from SPE and cannot be retrieved anymore. Calls to /pending/{ID}
or /done/{ID}
return HTTP status code 404.