c – 使用Ubuntu(Linux)主机的Docker守护程序/容器实时调度

在开始之前,我有两个想法是否应该在SuperUser或Stackoverflow中提出这个问题 – 如果它位于错误的位置,请提前道歉.

我有一个docker容器(包含C/C++可执行代码),它执行音频/视频处理.因此,我想测试使用RT调度约束运行容器的好处.在网上搜索,我遇到了各种各样的信息,但我正在努力将所有部分放在一起.

系统环境：

>主持人：Ubuntu(股票)Zesty 17.04(无RT内核补丁,内核：4.10.0-35-genric)
> Docker版本：17.05.0-ce
> Docker Images操作系统：Ubuntu Zesty 17.04.

在嵌套在docker镜像/容器中的可执行文件中,执行以下代码以将调度程序从“SCHED_OTHER”更改为“SCHED_FIFO”(请参阅docs)：

    struct sched_param sched = {};

    const int nMin = sched_get_priority_min(SCHED_FIFO);
    const int nMax = sched_get_priority_max(SCHED_FIFO);

    const int nHlf = (nMax - nMin) / 2;
    const int nPriority = nMin + nHlf + 1;

    sched.sched_priority = boost::algorithm::clamp(nPriority, nMin, nMax);

    if (sched_setscheduler(0, SCHED_FIFO, &sched) < 0)
        std::cerr << "SETSCHEDULER Failed - err = " << strerror(errno) << std::endl;
    else
        std::cout << "Priority set to \"" << sched.sched_priority << "\"" << std::endl;

我一直在阅读使用实时调度程序的各种Docker文档.一个有趣的page州,

Verify that CONfig_RT_GROUP_SCHED is enabled in the Linux kernel by running zcat /proc/config.gz | grep CONfig_RT_GROUP_SCHED or by checking for the existence of the file /sys/fs/cgroup/cpu.rt_runtime_us. For guidance on configuring the kernel realtime scheduler, consult the documentation for your operating system.

根据上述建议,股票Ubuntu Zesty 17.04 OS似乎未通过这些检查.

第一个问题：我不能使用RT调度程序吗？什么是’CONfig_RT_GROUP_SCHED’？令我困惑的一件事是,从2010年到2012年,网上有一些关于使用RT补丁修补内核的旧帖子.从那以后,似乎Linux内核中有一些与软RT相关的工作.

报价here引发了我的疑问：

From kernel version 2.6.18 onward, however, Linux is gradually becoming equipped with real-time capabilities, most of which are derived from the former realtime-preempt patches developed by Ingo Molnar, Thomas Gleixner, Steven Rostedt, and others. Until the patches have been completely merged into the mainline kernel (this is expected to be around kernel version 2.6.30), they must be installed to achieve the best real-time performance. These patches are named:

继续……

阅读了其他信息后,我注意到设置ulimits非常重要.我改变了/etc/security/limits.conf：

#*               soft    core            0
#root            hard    core            100000
#*               hard    RSS             10000

# NEW ADDITION
gavin            hard    rtprio          99

第二个问题：可能上面需要启用docker守护进程来运行RT？看起来守护进程是通过systemd控制的.

我继续进行调查,在同一个Docker文档页面上看到以下代码段：

To run containers using the realtime scheduler, run the Docker daemon with the –cpu-rt-runtime flag set to the maximum number of microseconds reserved for realtime tasks per runtime period. For instance, with the default period of 10000 microseconds (1 second), setting –cpu-rt-runtime=95000 ensures that containers using the realtime scheduler can run for 95000 microseconds for every 10000-microsecond period, leaving at least 5000 microseconds available for non-realtime tasks. To make this configuration permanent on systems which use systemd, see Control and configure Docker with systemd.

在this page之后,我发现守护进程有两个参数值得关注：

  --cpu-rt-period int                     Limit the cpu real-time period in microseconds
  --cpu-rt-runtime int                    Limit the cpu real-time runtime in microseconds

同一页面表明docker守护进程参数可以通过’/etc/docker/daemon.json’指定,所以我试过：

{
    "cpu-rt-period": 92500,
    "cpu-rt-runtime": 100000
}

注意：文档未将上述选项指定为“Linux上允许的配置选项”.我想我会试一试.

重启时Docker守护进程输出：

-- Logs begin at Wed 2017-10-04 09:58:38 BST, end at Wed 2017-10-04 10:01:32 BST. --
Oct 04 09:58:47 gavin systemd[1]: Starting Docker Application Container Engine...
Oct 04 09:58:47 gavin dockerd[1501]: time="2017-10-04T09:58:47.885882588+01:00" level=info msg="libcontainerd: new containerd process, pid: 1531"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.053986072+01:00" level=warning msg="Failed to rename /var/lib/docker/tmp for background deletion: %!s(<nil>).
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.161303803+01:00" level=info msg="[graphdriver] using prior storage driver: aufs"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.303409053+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304002725+01:00" level=warning msg="Your kernel does not support swap memory limit"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304078792+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304201239+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.305534113+01:00" level=info msg="Loading containers: start."
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.730193030+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemo
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.784938130+01:00" level=info msg="Loading containers: done."
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.888035017+01:00" level=info msg="Daemon has completed initialization"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.888104120+01:00" level=info msg="Docker daemon" commit=89658be graphdriver=aufs version=17.05.0-ce
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.903280645+01:00" level=info msg="API listen on /var/run/docker.sock"
Oct 04 09:58:48 gavin systemd[1]: Started Docker Application Container Engine.

特定的兴趣点：

Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304078792+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304201239+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"

鉴于我之前的发现,这并不奇怪.

最后一个问题：当这最终工作时,我如何能够确定我的容器是否真正运行了RT调度？ “顶级”这样的人会满意吗？

编辑：我运行了一个内核诊断script,我通过moby在github上运行了found.这是输出：

warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-4.10.0-35-generic ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONfig_NAMESPACES: enabled
- CONfig_NET_NS: enabled
- CONfig_PID_NS: enabled
- CONfig_IPC_NS: enabled
- CONfig_UTS_NS: enabled
- CONfig_CGROUPS: enabled
- CONfig_CGROUP_cpuACCT: enabled
- CONfig_CGROUP_DEVICE: enabled
- CONfig_CGROUP_FREEZER: enabled
- CONfig_CGROUP_SCHED: enabled
- CONfig_cpuSETS: enabled
- CONfig_MEMCG: enabled
- CONfig_KEYS: enabled
- CONfig_VETH: enabled (as module)
- CONfig_BRIDGE: enabled (as module)
- CONfig_BRIDGE_NETFILTER: enabled (as module)
- CONfig_NF_NAT_IPV4: enabled (as module)
- CONfig_IP_NF_FILTER: enabled (as module)
- CONfig_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONfig_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONfig_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONfig_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONfig_IP_NF_NAT: enabled (as module)
- CONfig_NF_NAT: enabled (as module)
- CONfig_NF_NAT_NEEDED: enabled
- CONfig_POSIX_MQUEUE: enabled

Optional Features:
- CONfig_USER_NS: enabled
- CONfig_SECCOMP: enabled
- CONfig_CGROUP_PIDS: enabled
- CONfig_MEMCG_SWAP: enabled
- CONfig_MEMCG_SWAP_ENABLED: missing
    (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
- CONfig_LEGACY_VSYSCALL_EMULATE: enabled
- CONfig_BLK_CGROUP: enabled
- CONfig_BLK_DEV_THRottLING: enabled
- CONfig_IOSCHED_CFQ: enabled
- CONfig_CFQ_GROUP_IOSCHED: enabled
- CONfig_CGROUP_PERF: enabled
- CONfig_CGROUP_HUGETLB: enabled
- CONfig_NET_CLS_CGROUP: enabled (as module)
- CONfig_CGROUP_NET_PRIO: enabled
- CONfig_CFS_BANDWIDTH: enabled
- CONfig_FAIR_GROUP_SCHED: enabled
- CONfig_RT_GROUP_SCHED: missing
- CONfig_IP_VS: enabled (as module)
- CONfig_IP_VS_NFCT: enabled
- CONfig_IP_VS_RR: enabled (as module)
- CONfig_EXT4_FS: enabled
- CONfig_EXT4_FS_POSIX_ACL: enabled
- CONfig_EXT4_FS_Security: enabled
- Network Drivers:
  - "overlay":
    - CONfig_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONfig_CRYPTO: enabled
      - CONfig_CRYPTO_AEAD: enabled
      - CONfig_CRYPTO_GCM: enabled (as module)
      - CONfig_CRYPTO_SEQIV: enabled
      - CONfig_CRYPTO_GHASH: enabled (as module)
      - CONfig_XFRM: enabled
      - CONfig_XFRM_USER: enabled (as module)
      - CONfig_XFRM_ALGO: enabled (as module)
      - CONfig_INET_ESP: enabled (as module)
      - CONfig_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONfig_IPVLAN: enabled (as module)
  - "macvlan":
    - CONfig_MACVLAN: enabled (as module)
    - CONfig_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONfig_NF_NAT_FTP: enabled (as module)
    - CONfig_NF_CONNTRACK_FTP: enabled (as module)
    - CONfig_NF_NAT_TFTP: enabled (as module)
    - CONfig_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONfig_AUFS_FS: enabled (as module)
  - "btrfs":
    - CONfig_BTRFS_FS: enabled (as module)
    - CONfig_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONfig_BLK_DEV_DM: enabled
    - CONfig_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONfig_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

意义重点：

- CONfig_RT_GROUP_SCHED: missing

解决方法:

在容器中进行RT调度有两种选择：

>添加SYS_NICE功能

docker run –cap-add SYS_NICE …
>使用特权模式和–privileged标志

docker run –privileged …

特权模式被认为是不安全的,因此选项1最好只添加您需要的功能.

如果以root用户身份运行(Docker容器的默认设置),则可能还必须在sysctl中启用实时调度：

sysctl -w kernel.sched_rt_runtime_us=-1

要使其永久化(更新您的图像)：

echo 'kernel.sched_rt_runtime_us=-1' > /etc/sysctl.conf

https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

c – 使用Ubuntu(Linux)主机的Docker守护程序/容器实时调度

相关推荐