LXRT and hard real time in user space.

We provide hard real time services in user space, also for normal, i.e. non root, users. We think that it will not be as good a performer as kernel space hard real time task modules, but a few microseconds more latency can be acceptable for many applications, especially on most recent almost Ghz CPUs. Many users will be glad with it for itself. At the very least, it will be useful in easing development and many other things: training, teaching and so on. It is wholly along the basic LXRT concept so we have seen it as an extension to LXRT.

To get hard real time in user space you need a fully preemptable kernel. The question, within RTAI "philosophy", is how to get full preemption with minimum changes, possibly none, to the kernel source.

The solution calls for a compromise. We propose to accept that a hard real time process does no Linux context kernel operations leading to a task switch. In that sense, it is better to speak of a "user space kernel module", and we will use the two terms interchangeably.

The approach is similar to what one of us did when he was using QNX: he always mated a hard real time tasks with a buddy for any I/O operation that could lead to excessive delays. In fact, even within such a good and fully preemptable kernel, I/Os could lead to deadlines misses under heavy hard real time I/O load from many hard real time tasks. Many examples in this distribution, i.e.: clocks, latency calibration and sound, show you a clear picture of how easy it is to use kernel services by mating to a buddy server process, without any problem. So, at least on the base of our modest experience, that is not an unbearable constraint. Since RTAI has many good intertask services, we do not see any problem in using the same approach again, especially in view of with what Pierre has done, is doing and will do, to make it the "informed" way.

It is nonetheless possible that such a constraint will be somewhat lifted as development proceed. Moreover the user space approach does not forbid you to do it in kernel space, if it is eventually needed. In fact it is not seen as a complete alternative to doing it in the kernel, but simply as a way of giving you more opportunities, at least during the development phase. However the more we use it the more we tend to avoid kernel space, as far as possible.

Taking into account that the present solution is somewhat still at the beginning of its development, we see a lot of space for making it better. For sure it is simpler than the exhausting search of safe scattered kernel pre- emption points many experts are looking at.

How is that possible?

We think that what you'll in RTAI-LXRT shows that it can work, even if it can be improved. The idea is to keep soft interrupts disabled for hard real time user space modules. This way, kernel module hard real time tasks and hard real time interrupts can preempt user space modules, but user space modules cannot be preempted neither by Linux hard interrupt nor by Linux processes. Linux hardware interrupt are pended as usual for service when RTAI's real time tasks (both in the kernel and user space) are idle.

How does it work?

Hard real time user space modules are just normal Linux processes that mate to a special buddy hard real time kernel task module, as done under LXRT already. They must be POSIX real time Linux processes locked into memory using SCHED_FIFO. Thus their memory must be pre grown to its maximum extension and completely locked in memory. See Linux man pages for mlockall/ munlockall. To distinguish them from usual LXRT firm real time processes the user simply calls rt_make_hard_real_time(), whereas by using rt_make_soft_real_time() he/ she can return to standard Linux task switching. Note that some of the required features, e.g POSIX real time under Linux, require root permission. However by using the function rt_allow_nonroot_hrt() you are allowed to: make a process POSIX real time, lock the memory and do IO operations, as a normal non root user. It is nonetheless necessary that the superuser "insmod"s the required modules (rtai, rtai_sched and lxrt).

The call to rt_make_hard_real_time allows to take a normal process out from the Linux running queue by calling schedule() after having queued the task to a bottom handler. When the bottom handler runs, the task is scheduled as a hard real time module by lxrt_schedule(), and Linux will not know of it, while having it still in its process list.

Lxrt_schedule() is also set as the signal function to be called when returning to the Linux context from a hard real time kernel space schedule, thus ensuring preemption in any case.

Lxrt_schedule() clear the soft interrupt flags and mimics the Linux schedule() function, with scheduling policy SCHED_FIFO, even from within interrupts.

To return to soft real time, rt_make_soft_real_time() does the opposite.

What it currently does:

There are some (not so) simple test processes that runs periodically and on which scheduling latency is measured. No doubt that it does something different as by running the same tasks under the same load with plain LXRT the latency goes as high as Linux 10 ms tick, compared to a 10/20 microseconds under user space modules (preliminary rough measures) and load. Note that within this new context it is likely that you can use also Linux pthreads both for soft and hard real time. In fact pthreads are normal user processes in disguise, Xavier made a choice, i.e. pthreads as cloned processes, that is good also for LXRT. Other examples show interacting tasks at work, while the sound task gives an idea of IOs from user space.

The experience gathered so far indicates that, despite the availability of more processing power, under SMP the latency for the same background load can be double/tripled with respect to UP. That is likely due to cache trashing caused by process switches and seems not to depend on the RTAI MP scheduler you are using. So it makes a larger jitter difference, with respect to working in kernel space, using hard real time processes under SMP than under UP. In fact under UP the jitter is roughly the same weather you are using user or kernel space modules.

What it currently does not very quickly:

Lxrt_schedule() can schedule in and out plain Linux processes, but to do it safely that must happen within Linux idle tasks. Clearly when one tests under heavy load the starting and ending of hard real time mode can be somewhat sluggish. In any case problems are just in starting and ending, once user space modules are in place they are fine. The matter has been somewhat improved by forcing the scheduling weight of the idle task, just four lines added/modified within the kernel. We know that there can be other ways of doing it, but all what we could conceive is likely to require heavy kernel modifications. Once more we recall that all our "philosophy" is to deplete the kernel with the slightest changes possible to it, better if none.

Note that within lxrt.c we trapped the kernel sys call and interrupt enabling to be sure that they are not called within hard real time user space modules. Pierre has conceived the same thing as possible to be done directly in rtai.c, as it can already trap all the reserved Linux traps, but no alternative handler has been implemented yet.

The new additions to lxrt:

• changed rtai_lxrt_handler to avoid ret_from_intr if returning from within a hard real time process;

• added macros my_switch_to(prev,next,last), loaddebug and function switch_to, all copied from Linux;

• added lxrt_schedule to schedule hard real time user space tasks among themselves and to and from the Linux context, with soft flags disabled (cli()), a lot of new data needed are found just above it, the name should self explain them;

• added function lxrt_do_steal to be run from the bh timer to schedule a new hard real time process;

• added the pointer rthal_enint to save the trapped trap rtahl.enint in order to diagnose enable from within rt user space modules;

• added lxrt_enint to actually do the above trapping;

• added lxrt_sigfun to lxrt_schedule when getting back to Linux from the rtai schedulers;

• added steal_from_linux to make a Linux process a user space hard real time module;

• added give_back_to_linux to return a user space module to the Linux processes;

• added linux_syscall_handler to save the trapped Linux sys handler;

• added lxrt_linux_syscall_handler to diagnose calls to sys from hard real time processes;

• print_to_screen to allow a safe printing of diagnosting messages from within user space modules working in hard real time mode.

User functions:

• print_to_screen(const char *format, ...): to safely print information and diagnostic messages in hard real time user space modules;

• void rt_make_soft_real_time(void): to return a hard real time user space process to soft Linux POSIX real time;

• void rt_make_hard_real_time(void): to make a soft Linux POSIX real time process a hard real time LXRT process;

• rt_allow_nonroot_hrt(void): to allow a non root user the make a process Linux POSIX real time, lock process memory in ram and carry out IO operations from user space.

Tests:

There is a wealth of examples to show extended lxrt operations, both in soft and hard real time mode. They can be useful also in giving you some clues for your applications.

Tests list:

• single task (directory one);
• two tasks (directory two);
• many tasks (directory many);
• many tasks (directory forked);
• many pthreads (directory threads);
• latency calibration (directory latency_calibration);
• sound test (directory sound);
• digital clock with semaphores (directory sem_clock);
• digital clock with messages (directory msg_clock).
• task resumed from an interrupt handler (directory resumefromintr).
• press test (directory pressa);
• resume in user space directly from the timer interrupt handler(directory resumefromint).

The possibility of using pthread_create to generate Linux processes is very useful since it allows a task layout that is close to the structure of modules. That could make it easier the translation to kernel modules for maximum performances. Also to be remarked is the possibility of resuming user space modules directly from interrupt handlers, see example reseumefromint.

If you want to check the jitter while one of the clocks or the sound example are running, you should enter the latency_calibration directory under another screen and type "./rt_process 1 &" followed by "./check". Try it varying Linux load. Be carefull, you must end it before closing the clocks/sound tests, see a more detailed comment within README in latency_calibration directory.

Have a look at the README files in each directory for more information.

It is important to remark that what is found under this directory can be used for any application but it is intended mainly for development work. It will be soon ported to lxrt-informed for a safer production use. Thus it is remarked that you must install a SIGINT handler if you want to safely terminate your LXRT processes, cleaning up any RTAI resource they use, after Ctrl-C. Some examples show how it can be done. We remind once more that what you find in directory lxrt is the final development version, the related production version is in lxrt-informed. It may happen that under this directory you can find features not yet ported in lxrt-informed. It will likely be so for a very short time. So take care of abnormal terminations yourself or wait for help from lxrt-informed.

---