如何设置docker限制容器内存容器中Java应用的内存限制
点击联系发帖人
时间：2016-08-22 02:07
如何控制Docker容器的内存，硬盘，CPU等参数_百度知道docker高级应用之cpu与内存资源限制 - 推酷
docker高级应用之cpu与内存资源限制
之前介绍过如何使用ovs的qos技术对docker容器做网络资源限制，本次介绍如何使用docker本身的参数进行cpu与内存资源限制。
默认docker本身提供在创建容器的时候，进行内存、swap、cpu限制，但使用起来就得有些技巧需要注意，具体如下：
1、内存限制
默认docker内存限制可以使用-m参数进行限制，但如果仅-m参数，不添加其他，那么内存限制不会限制死，比如你-m 256m内存，那么容器里程序可以跑到256m*2=512m后才会被oom给杀死，
原因是源码里memory.memsw.limit_in_bytes&值是被设置成我们指定的内存参数的两倍。
源码地址是
// By default, MemorySwap is set to twice the size of RAM.
// If you want to omit MemorySwap, set it to `-1'.
if d.c.MemorySwap != -1 {
if err := writeFile(dir, &memory.memsw.limit_in_bytes&, strconv.FormatInt(d.c.Memory*2, 10)); err != nil {
return err
当然以上的内容都是有人发过的，不是我原创，地址是
下面是介绍一下我的测试过程
先限制内存为100m
docker run --restart always -d
--name='11' docker.:5000/centos6-base:5.0 /usr/bin/supervisord
然后登陆容器后使用stress进行内存测试
PS：怕大家不知道如何安装stress，我也简单介绍一下
rpm -Uvh http://pkgs.repoforge.org/stress/stress-1.0.2-1.el7.rf.x86_64.rpm
这一个命令就安装完成了
使用stress进行测试
stress --vm 1 --vm-bytes 190M --vm-hang 0
这个命令是跑190mM的内存
下面是跑的结果
10:11:35 # stress --vm 1 --vm-bytes 190M --vm-hang 0
stress: info: [282] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [282] (420) &-- worker 283 got signal 9
stress: WARN: [282] (422) now reaping child worker processes
stress: FAIL: [282] (456) failed run completed in 1s
可以看到程序死了，想查看程序死的原因可以去/var/log/messages里查看
May 22 10:11:06 ip-10-10-125-11 docker:
10:11:06 http: multiple response.WriteHeader calls
May 22 10:11:45 ip-10-10-125-11 kernel: stress invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 22 10:11:45 ip-10-10-125-11 kernel: stress cpuset=docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope mems_allowed=0-1
May 22 10:11:45 ip-10-10-125-11 kernel: CPU: 7 PID: 37242 Comm: stress Not tainted 3.10.0-229.el7.x86_64 #1
May 22 10:11:45 ip-10-10-125-11 kernel: Hardware name: Dell Inc. PowerEdge R720/01XT2D, BIOS 2.0.19 08/29/2013
May 22 10:11:45 ip-10-10-125-11 kernel: ffff d4767f ffff ffffffff81604b0a
May 22 10:11:45 ip-10-10-125-11 kernel: ffff ffffffff815ffaaf 0046 ffff
May 22 10:11:45 ip-10-10-125-11 kernel: ffff ffff ffffffff81159fb6 ffffa8
May 22 10:11:45 ip-10-10-125-11 kernel: Call Trace:
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81604b0a&] dump_stack+0x19/0x1b
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff815ffaaf&] dump_header+0x8e/0x214
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81159fb6&] ? find_lock_task_mm+0x56/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8115a44e&] oom_kill_process+0x24e/0x3b0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8107bdde&] ? has_capability_noaudit+0x1e/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bd5ae&] __mem_cgroup_try_charge+0xb9e/0xbe0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bdd80&] ? mem_cgroup_charge_common+0xc0/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bdd19&] mem_cgroup_charge_common+0x59/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bf96a&] mem_cgroup_cache_charge+0x8a/0xb0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] __add_to_page_cache_locked+0x52/0x260
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] add_to_page_cache_lru+0x37/0xb0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] mpage_readpages+0xb5/0x160
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffffa020b310&] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffffa020b310&] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffffa0208f0c&] ext4_readpages+0x3c/0x40 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8116333c&] __do_page_cache_readahead+0x1cc/0x250
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] ra_submit+0x21/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81158f1d&] filemap_fault+0x11d/0x430
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8117e63e&] __do_fault+0x7e/0x520
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] handle_mm_fault+0x3d5/0xd70
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] ? mmap_region+0x186/0x610
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8160fe06&] __do_page_fault+0x156/0x540
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81189b25&] ? do_mmap_pgoff+0x305/0x3c0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81173f89&] ? vm_mmap_pgoff+0xb9/0xe0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8161020a&] do_page_fault+0x1a/0x70
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] page_fault+0x28/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: Task in /system.slice/docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope killed as a result of limit of /system.slice/docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope
May 22 10:11:45 ip-10-10-125-11 kernel: memory: usage 101144kB, limit 102400kB, failcnt 10934
May 22 10:11:45 ip-10-10-125-11 kernel: memory+swap: usage 204800kB, limit 204800kB, failcnt 37
May 22 10:11:45 ip-10-10-125-11 kernel: kmem: usage 0kB, limit 0991kB, failcnt 0
May 22 10:11:45 ip-10-10-125-11 kernel: Memory cgroup stats for /system.slice/docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope: cache:80KB rss:101064KB rss_huge:90112KB mapped_file:0KB swap:103656KB inactive_anon:72136KB active_anon:28928KB inactive_file:56KB active_file:24KB unevictable:0KB
May 22 10:11:45 ip-10-10-125-11 kernel: [ pid ]
tgid total_vm
rss nr_ptes swapents oom_score_adj name
May 22 10:11:45 ip-10-10-125-11 kernel: [34515]
0 supervisord
May 22 10:11:45 ip-10-10-125-11 kernel: [34531]
May 22 10:11:45 ip-10-10-125-11 kernel: [34542]
May 22 10:11:45 ip-10-10-125-11 kernel: [34544]
0 rsyslogd
May 22 10:11:45 ip-10-10-125-11 kernel: [35344]
May 22 10:11:45 ip-10-10-125-11 kernel: [36682]
May 22 10:11:45 ip-10-10-125-11 kernel: [37241]
May 22 10:11:45 ip-10-10-125-11 kernel: [37242]
May 22 10:11:45 ip-10-10-125-11 kernel: Memory cgroup out of memory: Kill process 37242 (stress) score 945 or sacrifice child
May 22 10:11:45 ip-10-10-125-11 kernel: Killed process 37242 (stress) total-vm:201136kB, anon-rss:94772kB, file-rss:0kB
显示内存使用超过2倍限制的内存，所以进行oom kill掉stress的进程，但容器不会死，仍然能正常运行。
但如果你想写死内存，可以使用
-m 100m --memory-swap=100m
这样就直接限制死，只要程序内存使用超过100m，那么就会被oom给kill。
2、cpu限制
目前cpu限制可以使用绑定到具体的线程，或者是在绑定线程基础上对线程资源权重分配。
绑定线程可以使用参数--cpuset-cpus=7（注意1.6版本才是--cpuset-cpus，1.6版本以下是--cpuset）
分配线程资源使用可以使用参数-c
具体的cpu如何使用与测试，参考上面国外朋友写的文章，很全面。
我这里注意介绍一下我这里平台cpu在容器里是如何使用？
我这里平台是直接把docker弄成iaas平台，然后在这个iaas基础上跑paas；其他的docker云基本都是在openstack的iaas或者其他的iaas里跑docker做paas。
内存使用我是直接限制为256m、512m、1g，cpu默认都是0.2c。
c的概念是cpu线程，比如你top的时候，使用1命令可以直接看到系统里有多少线程，以16个线程为例，我保留2个c给宿主机系统使用，另外14个c给docker使用，一个c有5个容器共同使用，最少可以使用20%资源，也就是0.2c，最多可以使用100%资源，也就是1c。
这样好处是线程不忙的时候，容器可以独享100%资源，资源紧张的时候，也就保证自己有20%的资源使用，另外同一台宿主机的容器即使某个资源使用多，也不好太影响其他c的容器。
我这里测试环境出现过某个项目，在进行压力测试的时候，由于没有对容器做资源限制，导致测试中把宿主机压测挂了，导致同台宿主机所有容器都无法访问。
错误如下图
讲了一大堆内存与cpu资源限制，基本别人都多多少少的结束过，我在介绍一下我独有的。
3、模块化进行cpu、内存、网络资源限制
可以使用先是脚本进行一键化的配置下面功能容器：
1、持久化独立ip；
2、可以做系统描述符配置；
3、可以限制内存使用资源；
4、可以限制网络使用资源；
5、可以自动绑定cpu线程；
6、可以出现故障时自动重启容器；
7、支持容器使用密码或者key验证；
下面是创建容器的过程
1、创建密码验证的
内存是256m，网络资源1m（bit），使用密码验证，文件描述符是65535
[root@ip-10-10-125-8 docker_code]# python create_docker_container_multi.py test1 docker.:5000/centos6-base:5.0 /usr/bin/supervisord 256m 1 passwd &535
{'Container_id': '522b5ac5ebba6f8b43d737dd7290cfad4ef789555', 'Container_gateway_ip': u'172.16.0.14', 'Memory_limit': '256m', 'Cpu_share': '3', 'Container_status': 'running', 'Physics_ip': '10.10.125.8', 'Container_create': '-10:47:41', 'Container_name': 'test1', 'Cpu_core': '14', 'Network_limit': '1', 'Container_gateway_bridge': 'ovs2', 'Container_ip': u'172.16.2.164/16'}
[root@ip-10-10-125-8 docker_code]# ssh 172.16.2.164
root@172.16.2.164's password:
Last login: Fri Mar 13 13:53:37 2015 from 172.17.42.1
root@522b5a918383:~
10:48:56 # ifconfig eth1
Link encap:Ethernet
HWaddr 86:44:D4:F5:5C:01
inet addr:172.16.2.164
Bcast:0.0.0.0
Mask:255.255.0.0
inet6 addr: fe80::8444:d4ff:fef5:5c01/64 Scope:Link
UP BROADCAST RUNNING MULTICAST
RX packets:96 errors:0 dropped:0 overruns:0 frame:0
TX packets:75 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1 KiB)
TX bytes:1 KiB)
root@522b5a918383:~
10:48:59 # ulimit -a
core file size (blocks, -c) unlimited
data seg size
(kbytes, -d) unlimited
scheduling priority
(blocks, -f) unlimited
pending signals
(-i) 128596
max locked memory
(kbytes, -l) 64
max memory size
(kbytes, -m) unlimited
open files
(-n) 65535
(512 bytes, -p) 8
POSIX message queues
(bytes, -q) 819200
real-time priority
stack size
(kbytes, -s) 8192
(seconds, -t) unlimited
max user processes
(-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks
(-x) unlimited
内存的话，使用docker stats模块查看
[root@ip-10-10-125-8 docker_code]# docker stats test1
MEM USAGE/LIMIT
30.7 MiB/256 MiB
可以看到就是256m的内存限制
网络的话，我通过wget下载iso文件来测试
root@522b5a918383:~
10:50:20 # wget -S -O /dev/null http://mirror./centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso
-- 10:51:04--
http://mirror./centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso
Resolving mirror.... 202.118.1.64, 2001:da8:9000::64
Connecting to mirror.|202.118.1.64|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Tue, 09 Jun :48 GMT
Content-Type: application/octet-stream
Content-Length:
Last-Modified: Wed, 01 Apr :50 GMT
Connection: keep-alive
Accept-Ranges: bytes
(4.0G) [application/octet-stream]
Saving to: “/dev/null”
eta 9h 54m
可以看到下载速度是1m（bit）/8=117k（byte）。
在测试一下持久化固定ip，重启容器
[root@ip-10-10-125-8 docker_code]# docker restart test1
[root@ip-10-10-125-8 docker_code]# docker exec test1 ifconfig
Link encap:Local Loopback
inet addr:127.0.0.1
Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b)
TX bytes:0 (0.0 b)
可以看到重启后容器ip没了，但我有另外的一个脚本，会在发现容器没有ip后自动的把ip以及之前的配置重新给予（在crontab里每分钟运行一次）。
[root@ip-10-10-125-8 docker_code]# docker exec test1 ifconfig
Link encap:Ethernet
HWaddr B6:6D:BC:CC:CD:69
inet addr:172.16.2.164
Bcast:0.0.0.0
Mask:255.255.0.0
inet6 addr: fe80::b46d:bcff:fecc:cd69/64 Scope:Link
UP BROADCAST RUNNING MULTICAST
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:648 (648.0 b)
TX bytes:690 (690.0 b)
Link encap:Local Loopback
inet addr:127.0.0.1
Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b)
TX bytes:0 (0.0 b)
[root@ip-10-10-125-8 docker_code]#
可以看到ip重新赋予了，实现了持久化固定ip功能。
下面是创建过程中使用的脚本.大家参考就行，没必要直接复制使用，每个人的需求都是不一样的。
[root@ip-10-10-125-8 docker_code]# cat create_docker_container_multi.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
#author:Deng Lei
import sys
import etcd
import time
import socket, struct, fcntl
from docker import Client
import subprocess
from multiprocessing import cpu_count
#get local host ip
def get_local_ip(iface = 'em1'):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sockfd = sock.fileno()
SIOCGIFADDR = 0x8915
ifreq = struct.pack('16sH14s', iface, socket.AF_INET, '\x00'*14)
res = fcntl.ioctl(sockfd, SIOCGIFADDR, ifreq)
return None
ip = struct.unpack('16sH2x4s8x', res)[2]
return socket.inet_ntoa(ip)
def docker_container_all():
docker_container=docker_client.containers(all=True)
container_name=[]
container_stop_name=[]
for i in docker_container:
container_name.append(i['Names'])
for b in container_name:
for c in b:
container_stop_name.append(c[1:])
return container_stop_name
def get_now_cpu_core():
#get recycle cpu use in etcd
recycle_cpu_key='%s%s/resource_limit/cpu_limit/now_cpu_core_avaiable'%(docker_etcd_key,local_ip)
all_recycle_cpu_result=eval(etcd_client.read(recycle_cpu_key).value)
except etcd.EtcdKeyNotFound:
all_recycle_cpu_result=''
if len(all_recycle_cpu_result) & 0:
now_core={}
now_core['now_core']=all_recycle_cpu_result[0].split('-')[0]
now_core['now_core_share']=all_recycle_cpu_result[0].split('-')[1]
all_recycle_cpu_result.remove(all_recycle_cpu_result[0])
etcd_client.write(recycle_cpu_key,all_recycle_cpu_result)
return now_core
#get local core use in etcd
now_core={'count':cpu_core,'now_core':cpu_core-1,'core_share_limit':cpu_core_share_limit,'now_core_share':-1}
now_core_use=eval(etcd_client.read('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip)).value)
except KeyError:
now_core_use=now_core
etcd_client.write('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip),now_core)
time.sleep(1)
except etcd.EtcdKeyNotFound:
now_core_use=now_core
etcd_client.write('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip),now_core)
time.sleep(1)
if int(now_core_use['now_core']) == int((cpu_core_retain - 1)):
msg={'status':1,'resource':'cpu','result':'the avaiable cpu core is less than retain:%s,i can not create new container!'%cpu_core_retain}
sys.exit(1)
if int(now_core_use['now_core_share']+1) &= int(now_core_use['core_share_limit']):
now_core_use['now_core']=now_core_use['now_core']-1
if int(now_core_use['now_core']) == int((cpu_core_retain - 1)):
msg={'status':1,'resource':'cpu','result':'the avaiable cpu core is less than retain:%s,i can not create new container!'%cpu_core_retain}
sys.exit(1)
now_core_use['now_core_share']=0
now_core_use['now_core_share']=now_core_use['now_core_share']+1
etcd_client.write('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip),now_core_use)
return now_core_use
def get_container_ip():
#get avaiable container ip
recycle_ip_key='%scontainer/now_ip_avaiable'%docker_etcd_key
all_recycle_ip_result=eval(etcd_client.read(recycle_ip_key).value)
except etcd.EtcdKeyNotFound:
all_recycle_ip_result=''
if len(all_recycle_ip_result) & 0:
now_recycle_ip=all_recycle_ip_result[0]
all_recycle_ip_result.remove(all_recycle_ip_result[0])
etcd_client.write(recycle_ip_key,all_recycle_ip_result)
return now_recycle_ip
#get new container ip
now_container_ip=etcd_client.read('%scontainer/now_ip'%(docker_etcd_key)).value
except KeyError:
now_container_ip=default_container_ip
except etcd.EtcdKeyNotFound:
now_container_ip=default_container_ip
ip=now_container_ip.split('/')[0]
netmask=now_container_ip.split('/')[1]
if int(ip.split('.')[-1]) &254:
new_container_ip='.'.join(ip.split('.')[0:3])+'.'+str(int(ip.split('.')[-1])+1)+'/'+netmask
elif int(ip.split('.')[-1]) &=254 and int(ip.split('.')[-2]) &254:
if int(ip.split('.')[-1]) ==254:
last_ip='1'
last_ip=int(ip.split('.')[-1])+1
new_container_ip='.'.join(ip.split('.')[0:2])+'.'+str(int(ip.split('.')[-2])+1)+'.'+last_ip+'/'+netmask
elif int(ip.split('.')[-2]) &=254 and int(ip.split('.')[-3])&254:
if int(ip.split('.')[-1]) == 254:
last_ip='1'
last_ip=int(ip.split('.')[-1])+1
new_container_ip='.'.join(ip.split('.')[0:2])+'.'+str(int(ip.split('.')[2])+1)+'.'+last_ip+'/'+netmask
elif int(ip.split('.')[-2]) ==255 and int(ip.split('.')[-1])==254:
print 'now ip had more than ip pool!'
sys.exit(1)
etcd_client.write('%scontainer/now_ip'%docker_etcd_key,new_container_ip)
return new_container_ip
if __name__ == &__main__&:
bridge='ovs2'
default_physics_ip='172.16.0.1/16'
default_container_ip='172.16.1.1/16'
local_ip=get_local_ip('ovs1')
if local_ip is None:
local_ip=get_local_ip('em1')
docker_etcd_key='/app/docker/'
container=sys.argv[1]
if container == &-h& or container == &-help&:
print 'Usage: container_name container_image command'
sys.exit(1)
except IndexError:
print 'print input container name!'
sys.exit(1)
images=sys.argv[2]
except IndexError:
print 'print input images!'
sys.exit(1)
daemon_program=sys.argv[3]
except IndexError:
print 'please input run image commmand!'
sys.exit(1)
memory_limit=sys.argv[4]
except IndexError:
print 'please input memory limit where unit = b, k, m or g!'
sys.exit(1)
network_limit=sys.argv[5]
except IndexError:
print 'please input network limit where unit = M!'
sys.exit(1)
ssh_method=sys.argv[6]
except IndexError:
print 'please input container ssh auth method by key or passwd!'
sys.exit(1)
ssh_auth=sys.argv[7]
except IndexError:
print 'please input container ssh auth passwd or key_context!'
sys.exit(1)
ulimit=sys.argv[8]
except IndexError:
ulimit=65535
etcd_client=etcd.Client(host='127.0.0.1', port=4001)
cpu_core=cpu_count()
cpu_core_retain=2
cpu_core_share_limit=5
local_dir=sys.path[0]
#get local core use in etcd
now_core=get_now_cpu_core()
docker_client = Client(base_url='unix://var/run/docker.sock', version='1.15', timeout=10)
docker_container_all_name=docker_container_all()
#check input container exist
if container in docker_container_all_name:
print 'The container %s is exist!'%container
sys.exit(1)
#check pipework software exist
check_pipework_status=((subprocess.Popen(&which pipework &&&/dev/null && echo 0 || echo 1&,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_pipework_status != &0&:
print 'I can not find pipework in this host!'
sys.exit(1)
#check ovs bridge
check_bridge_status=((subprocess.Popen(&ovs-vsctl
list-br |grep %s &&&/dev/null && echo 0 || echo 1&%bridge,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_bridge_status !=&0&:
subprocess.Popen(&ovs-vsctl add-br %s&%bridge,shell=True,stdout=subprocess.PIPE)
#check default
physics ip
default_physics_ip=etcd_client.read('%sphysics/now_ip'%(docker_etcd_key)).value
except KeyError:
default_physics_ip=default_physics_ip
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,default_physics_ip)
time.sleep(1)
except etcd.EtcdKeyNotFound:
default_physics_ip=default_physics_ip
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,default_physics_ip)
time.sleep(1)
#check physics ip
new_physics_ip=etcd_client.read('%sphysics/%s'%(docker_etcd_key,local_ip)).value
except KeyError:
now_physics_ip=etcd_client.read('%sphysics/now_ip'%(docker_etcd_key)).value
ip=now_physics_ip.split('/')[0]
netmask=now_physics_ip.split('/')[1]
if int(ip.split('.')[-1]) &254:
new_physics_ip='.'.join(ip.split('.')[0:3])+'.'+str(int(ip.split('.')[-1])+1)+'/'+netmask
elif int(ip.split('.')[-1]) &=254:
print 'physics gateway ip more than set up ip pool!'
sys.exit(1)
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,new_physics_ip)
etcd_client.write('%sphysics/%s'%(docker_etcd_key,local_ip),new_physics_ip)
except etcd.EtcdKeyNotFound:
now_physics_ip=etcd_client.read('%sphysics/now_ip'%(docker_etcd_key)).value
ip=now_physics_ip.split('/')[0]
netmask=now_physics_ip.split('/')[1]
if int(ip.split('.')[-1]) &254:
new_physics_ip='.'.join(ip.split('.')[0:3])+'.'+str(int(ip.split('.')[-1])+1)+'/'+netmask
elif int(ip.split('.')[-1]) &=254:
print 'physics gateway ip more than set up ip pool!'
sys.exit(1)
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,new_physics_ip)
etcd_client.write('%sphysics/%s'%(docker_etcd_key,local_ip),new_physics_ip)
#set physics ip
#run_command=&awk -v value=`ifconfig %s|grep inet|grep -v inet6|awk '{print $2}'|wc -l` 'BEGIN{if(value&1) {system('/sbin/ifconfig %s %s')}}'&%(bridge,bridge,new_physics_ip)
set_physics_status=((subprocess.Popen(&/sbin/ifconfig %s %s && echo 0 || echo 1&%(bridge,new_physics_ip),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if set_physics_status !=&0&:
print 'config local physics ip:%s is fail!'%new_physics_ip
sys.exit(1)
docker_version=docker_client.version()['Version'][0:3]
if float(docker_version) &= 1.6 :
cpu_tag='--cpuset-cpus'
container_ulimit=&--ulimit nofile=%s:%s&%(ulimit,ulimit)
cpu_tag='--cpuset'
container_ulimit=''
#create new container
get_container_id=((subprocess.Popen(&docker run --restart always %s -d -m %s --memory-swap=%s %s=%s --net='none' --name='%s' %s %s&%(container_ulimit,memory_limit,memory_limit,cpu_tag,now_core['now_core'],container,images,daemon_program),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
check_container_status=((subprocess.Popen(&docker inspect %s|grep Id &&&/dev/null && echo 0 || echo 1&%(container),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_container_status !=&0&:
print 'Create container %s is fail!'%container
sys.exit(1)
#get container ip
new_container_ip=get_container_ip()
#check container network
new_physics_ip=new_physics_ip.split('/')[0]
set_container_status=((subprocess.Popen(&`which pipework` %s %s %s@%s
&&&/dev/null && echo 0 || echo 1&%(bridge,container,new_container_ip,new_physics_ip),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if set_container_status != &0&:
print 'config container:%s network is fail!'%container
sys.exit(1)
#set container network limit
set_container_network_limit=((subprocess.Popen(&/bin/bash %s/modify_docker_container_network_limit.sh limit %s %s
&&&/dev/null && echo 0 || echo 1&%(local_dir,container,network_limit),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if set_container_network_limit != &0&:
print 'config container:%s network limit:%s is fail!'%(container,network_limit)
sys.exit(1)
#write container network limit to etcd
etcd_client.write('%s%s/resource_limit/network_limit/%s'%(docker_etcd_key,local_ip,container),network_limit)
#modify container ssh auth
if ssh_method == 'key':
cmd='docker exec %s sh -c &echo \'%s\' &&/root/.ssh/authorized_keys && echo 0 || echo 1&'%(container,ssh_auth)
elif ssh_method == 'passwd':
cmd='docker exec %s sh -c &echo &%s& | passwd --stdin root &&&/dev/null && echo 0 || echo 1&'%(container,ssh_auth)
check_ssh_status=((subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_ssh_status != &0&:
print 'config container:%s ssh auth method:%s is fail!'%(container,ssh_method)
delete_cmd='docker rm $(docker stop %s)'%container
delete_ssh_status=((subprocess.Popen(delete_cmd,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
sys.exit(1)
#write container info to etcd
key='%s%s/container/%s'%(docker_etcd_key,local_ip,container)
value={'Physics_ip':'%s'%local_ip,'Container_id':get_container_id,'Container_name':container,'Container_ip':'%s'%new_container_ip,'Container_gateway_bridge':bridge,'Container_gateway_ip':new_physics_ip,'Container_create':time.strftime('%Y.%m.%d-%T'),'Container_status':'running','Cpu_core':now_core['now_core'],'Cpu_share':now_core['now_core_share'],'Memory_limit':memory_limit,'Network_limit':network_limit}
etcd_client.write(key,value)
#print 'create container:%s is success.follow is container info.'%container
print value
这个脚本是创建容器并进行资源限制的
下面脚本是进行网络资源限制，上面脚本会调用做网络限制
[root@ip-10-10-125-8 docker_code]# cat modify_docker_container_network_limit.sh
#!/bin/bash
#filename:modify_docker_container_network_limit.sh
#author:Deng Lei
container=$2
if [ -z $1 ] || [ -z $2 ]; then
echo &Usage: operation container_name limit(default:5m)&
echo &Example1: I want limit 5m in the container:test&
echo &The command is: bash `basename $0` limit test 5&
echo &Example2: I want delete network limit in the container:test&
echo &The command is: bash `basename $0` ulimit test&
if [ -z $3 ];then
limit='5m'
if [ `docker inspect --format &{{.State.Pid}}& $container &&&/dev/null && echo 0 || echo 1` -eq 1 ];then
echo &no this container:$container&
ovs_prefix='veth1pl'
container_id=`docker inspect --format &{{.State.Pid}}& $container`
device_name=`echo ${ovs_prefix}${container_id}`
docker_etcd_key='/app/docker/'
local_ip=`ifconfig ovs1|grep inet|awk '{print $2}'|head -n1`
if [ $op == 'limit' ];then
for v in $device_ do
ovs-vsctl set interface $v ingress_policing_rate=$((limit*1000))
ovs-vsctl set interface $v ingress_policing_burst=$((limit*100))
ovs-vsctl set port $v qos=@newqos -- --id=@newqos create qos type=linux-htb queues=0=@q0 other-config:max-rate=$((limit*1000000)) -- --id=@q0 create queue other-config:min-rate=$((limit*1000000)) other-config:max-rate=$((limit*1000000)) &&&/dev/null && echo '0' || echo '1'
#save result to etcd
`which etcdctl` set ${docker_etcd_key}${local_ip}/resource_limit/network_limit/${container} $limit &&&/dev/null
elif [ $op == 'ulimit' ];then
for v in $device_ do
ovs-vsctl set interface $v ingress_policing_rate=0
ovs-vsctl set interface $v ingress_policing_burst=0
ovs-vsctl clear Port $v qos &&&/dev/null && echo '0' || echo '1'
`which etcdctl` rm ${docker_etcd_key}${local_ip}/resource_limit/network_limit/${container} &&&/dev/null
已发表评论数()
请填写推刊名
描述不能大于100个字符!
权限设置： 公开
仅自己可见
正文不准确
标题不准确
排版有问题
主题不准确
没有分页内容
图片无法显示
视频无法显示
与原文不一致}
docker高级应用之cpu与内存资源限制
之前介绍过如何使用ovs的qos技术对docker容器做网络资源限制，本次介绍如何使用docker本身的参数进行cpu与内存资源限制。
默认docker本身提供在创建容器的时候，进行内存、swap、cpu限制，但使用起来就得有些技巧需要注意，具体如下：
1、内存限制
默认docker内存限制可以使用-m参数进行限制，但如果仅-m参数，不添加其他，那么内存限制不会限制死，比如你-m 256m内存，那么容器里程序可以跑到256m*2=512m后才会被oom给杀死，
原因是源码里memory.memsw.limit_in_bytes&值是被设置成我们指定的内存参数的两倍。
源码地址是
// By default, MemorySwap is set to twice the size of RAM.
// If you want to omit MemorySwap, set it to `-1'.
if d.c.MemorySwap != -1 {
if err := writeFile(dir, &memory.memsw.limit_in_bytes&, strconv.FormatInt(d.c.Memory*2, 10)); err != nil {
return err
当然以上的内容都是有人发过的，不是我原创，地址是
下面是介绍一下我的测试过程
先限制内存为100m
docker run --restart always -d
--name='11' docker.:5000/centos6-base:5.0 /usr/bin/supervisord
然后登陆容器后使用stress进行内存测试
PS：怕大家不知道如何安装stress，我也简单介绍一下
rpm -Uvh http://pkgs.repoforge.org/stress/stress-1.0.2-1.el7.rf.x86_64.rpm
这一个命令就安装完成了
使用stress进行测试
stress --vm 1 --vm-bytes 190M --vm-hang 0
这个命令是跑190mM的内存
下面是跑的结果
10:11:35 # stress --vm 1 --vm-bytes 190M --vm-hang 0
stress: info: [282] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [282] (420) &-- worker 283 got signal 9
stress: WARN: [282] (422) now reaping child worker processes
stress: FAIL: [282] (456) failed run completed in 1s
可以看到程序死了，想查看程序死的原因可以去/var/log/messages里查看
May 22 10:11:06 ip-10-10-125-11 docker:
10:11:06 http: multiple response.WriteHeader calls
May 22 10:11:45 ip-10-10-125-11 kernel: stress invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
May 22 10:11:45 ip-10-10-125-11 kernel: stress cpuset=docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope mems_allowed=0-1
May 22 10:11:45 ip-10-10-125-11 kernel: CPU: 7 PID: 37242 Comm: stress Not tainted 3.10.0-229.el7.x86_64 #1
May 22 10:11:45 ip-10-10-125-11 kernel: Hardware name: Dell Inc. PowerEdge R720/01XT2D, BIOS 2.0.19 08/29/2013
May 22 10:11:45 ip-10-10-125-11 kernel: ffff d4767f ffff ffffffff81604b0a
May 22 10:11:45 ip-10-10-125-11 kernel: ffff ffffffff815ffaaf 0046 ffff
May 22 10:11:45 ip-10-10-125-11 kernel: ffff ffff ffffffff81159fb6 ffffa8
May 22 10:11:45 ip-10-10-125-11 kernel: Call Trace:
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81604b0a&] dump_stack+0x19/0x1b
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff815ffaaf&] dump_header+0x8e/0x214
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81159fb6&] ? find_lock_task_mm+0x56/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8115a44e&] oom_kill_process+0x24e/0x3b0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8107bdde&] ? has_capability_noaudit+0x1e/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bd5ae&] __mem_cgroup_try_charge+0xb9e/0xbe0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bdd80&] ? mem_cgroup_charge_common+0xc0/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bdd19&] mem_cgroup_charge_common+0x59/0xc0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff811bf96a&] mem_cgroup_cache_charge+0x8a/0xb0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] __add_to_page_cache_locked+0x52/0x260
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] add_to_page_cache_lru+0x37/0xb0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] mpage_readpages+0xb5/0x160
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffffa020b310&] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffffa020b310&] ? _ext4_get_block+0x1b0/0x1b0 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffffa0208f0c&] ext4_readpages+0x3c/0x40 [ext4]
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8116333c&] __do_page_cache_readahead+0x1cc/0x250
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] ra_submit+0x21/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81158f1d&] filemap_fault+0x11d/0x430
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8117e63e&] __do_fault+0x7e/0x520
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] handle_mm_fault+0x3d5/0xd70
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] ? mmap_region+0x186/0x610
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8160fe06&] __do_page_fault+0x156/0x540
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81189b25&] ? do_mmap_pgoff+0x305/0x3c0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff81173f89&] ? vm_mmap_pgoff+0xb9/0xe0
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff8161020a&] do_page_fault+0x1a/0x70
May 22 10:11:45 ip-10-10-125-11 kernel: [&ffffffff&] page_fault+0x28/0x30
May 22 10:11:45 ip-10-10-125-11 kernel: Task in /system.slice/docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope killed as a result of limit of /system.slice/docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope
May 22 10:11:45 ip-10-10-125-11 kernel: memory: usage 101144kB, limit 102400kB, failcnt 10934
May 22 10:11:45 ip-10-10-125-11 kernel: memory+swap: usage 204800kB, limit 204800kB, failcnt 37
May 22 10:11:45 ip-10-10-125-11 kernel: kmem: usage 0kB, limit 0991kB, failcnt 0
May 22 10:11:45 ip-10-10-125-11 kernel: Memory cgroup stats for /system.slice/docker-6ae152aa189fa2e9c414ae70addc55a2fe495b93ed5.scope: cache:80KB rss:101064KB rss_huge:90112KB mapped_file:0KB swap:103656KB inactive_anon:72136KB active_anon:28928KB inactive_file:56KB active_file:24KB unevictable:0KB
May 22 10:11:45 ip-10-10-125-11 kernel: [ pid ]
tgid total_vm
rss nr_ptes swapents oom_score_adj name
May 22 10:11:45 ip-10-10-125-11 kernel: [34515]
0 supervisord
May 22 10:11:45 ip-10-10-125-11 kernel: [34531]
May 22 10:11:45 ip-10-10-125-11 kernel: [34542]
May 22 10:11:45 ip-10-10-125-11 kernel: [34544]
0 rsyslogd
May 22 10:11:45 ip-10-10-125-11 kernel: [35344]
May 22 10:11:45 ip-10-10-125-11 kernel: [36682]
May 22 10:11:45 ip-10-10-125-11 kernel: [37241]
May 22 10:11:45 ip-10-10-125-11 kernel: [37242]
May 22 10:11:45 ip-10-10-125-11 kernel: Memory cgroup out of memory: Kill process 37242 (stress) score 945 or sacrifice child
May 22 10:11:45 ip-10-10-125-11 kernel: Killed process 37242 (stress) total-vm:201136kB, anon-rss:94772kB, file-rss:0kB
显示内存使用超过2倍限制的内存，所以进行oom kill掉stress的进程，但容器不会死，仍然能正常运行。
但如果你想写死内存，可以使用
-m 100m --memory-swap=100m
这样就直接限制死，只要程序内存使用超过100m，那么就会被oom给kill。
2、cpu限制
目前cpu限制可以使用绑定到具体的线程，或者是在绑定线程基础上对线程资源权重分配。
绑定线程可以使用参数--cpuset-cpus=7（注意1.6版本才是--cpuset-cpus，1.6版本以下是--cpuset）
分配线程资源使用可以使用参数-c
具体的cpu如何使用与测试，参考上面国外朋友写的文章，很全面。
我这里注意介绍一下我这里平台cpu在容器里是如何使用？
我这里平台是直接把docker弄成iaas平台，然后在这个iaas基础上跑paas；其他的docker云基本都是在openstack的iaas或者其他的iaas里跑docker做paas。
内存使用我是直接限制为256m、512m、1g，cpu默认都是0.2c。
c的概念是cpu线程，比如你top的时候，使用1命令可以直接看到系统里有多少线程，以16个线程为例，我保留2个c给宿主机系统使用，另外14个c给docker使用，一个c有5个容器共同使用，最少可以使用20%资源，也就是0.2c，最多可以使用100%资源，也就是1c。
这样好处是线程不忙的时候，容器可以独享100%资源，资源紧张的时候，也就保证自己有20%的资源使用，另外同一台宿主机的容器即使某个资源使用多，也不好太影响其他c的容器。
我这里测试环境出现过某个项目，在进行压力测试的时候，由于没有对容器做资源限制，导致测试中把宿主机压测挂了，导致同台宿主机所有容器都无法访问。
错误如下图
讲了一大堆内存与cpu资源限制，基本别人都多多少少的结束过，我在介绍一下我独有的。
3、模块化进行cpu、内存、网络资源限制
可以使用先是脚本进行一键化的配置下面功能容器：
1、持久化独立ip；
2、可以做系统描述符配置；
3、可以限制内存使用资源；
4、可以限制网络使用资源；
5、可以自动绑定cpu线程；
6、可以出现故障时自动重启容器；
7、支持容器使用密码或者key验证；
下面是创建容器的过程
1、创建密码验证的
内存是256m，网络资源1m（bit），使用密码验证，文件描述符是65535
[root@ip-10-10-125-8 docker_code]# python create_docker_container_multi.py test1 docker.:5000/centos6-base:5.0 /usr/bin/supervisord 256m 1 passwd &535
{'Container_id': '522b5ac5ebba6f8b43d737dd7290cfad4ef789555', 'Container_gateway_ip': u'172.16.0.14', 'Memory_limit': '256m', 'Cpu_share': '3', 'Container_status': 'running', 'Physics_ip': '10.10.125.8', 'Container_create': '-10:47:41', 'Container_name': 'test1', 'Cpu_core': '14', 'Network_limit': '1', 'Container_gateway_bridge': 'ovs2', 'Container_ip': u'172.16.2.164/16'}
[root@ip-10-10-125-8 docker_code]# ssh 172.16.2.164
root@172.16.2.164's password:
Last login: Fri Mar 13 13:53:37 2015 from 172.17.42.1
root@522b5a918383:~
10:48:56 # ifconfig eth1
Link encap:Ethernet
HWaddr 86:44:D4:F5:5C:01
inet addr:172.16.2.164
Bcast:0.0.0.0
Mask:255.255.0.0
inet6 addr: fe80::8444:d4ff:fef5:5c01/64 Scope:Link
UP BROADCAST RUNNING MULTICAST
RX packets:96 errors:0 dropped:0 overruns:0 frame:0
TX packets:75 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1 KiB)
TX bytes:1 KiB)
root@522b5a918383:~
10:48:59 # ulimit -a
core file size (blocks, -c) unlimited
data seg size
(kbytes, -d) unlimited
scheduling priority
(blocks, -f) unlimited
pending signals
(-i) 128596
max locked memory
(kbytes, -l) 64
max memory size
(kbytes, -m) unlimited
open files
(-n) 65535
(512 bytes, -p) 8
POSIX message queues
(bytes, -q) 819200
real-time priority
stack size
(kbytes, -s) 8192
(seconds, -t) unlimited
max user processes
(-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks
(-x) unlimited
内存的话，使用docker stats模块查看
[root@ip-10-10-125-8 docker_code]# docker stats test1
MEM USAGE/LIMIT
30.7 MiB/256 MiB
可以看到就是256m的内存限制
网络的话，我通过wget下载iso文件来测试
root@522b5a918383:~
10:50:20 # wget -S -O /dev/null http://mirror./centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso
-- 10:51:04--
http://mirror./centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso
Resolving mirror.... 202.118.1.64, 2001:da8:9000::64
Connecting to mirror.|202.118.1.64|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Tue, 09 Jun :48 GMT
Content-Type: application/octet-stream
Content-Length:
Last-Modified: Wed, 01 Apr :50 GMT
Connection: keep-alive
Accept-Ranges: bytes
(4.0G) [application/octet-stream]
Saving to: “/dev/null”
eta 9h 54m
可以看到下载速度是1m（bit）/8=117k（byte）。
在测试一下持久化固定ip，重启容器
[root@ip-10-10-125-8 docker_code]# docker restart test1
[root@ip-10-10-125-8 docker_code]# docker exec test1 ifconfig
Link encap:Local Loopback
inet addr:127.0.0.1
Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b)
TX bytes:0 (0.0 b)
可以看到重启后容器ip没了，但我有另外的一个脚本，会在发现容器没有ip后自动的把ip以及之前的配置重新给予（在crontab里每分钟运行一次）。
[root@ip-10-10-125-8 docker_code]# docker exec test1 ifconfig
Link encap:Ethernet
HWaddr B6:6D:BC:CC:CD:69
inet addr:172.16.2.164
Bcast:0.0.0.0
Mask:255.255.0.0
inet6 addr: fe80::b46d:bcff:fecc:cd69/64 Scope:Link
UP BROADCAST RUNNING MULTICAST
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:648 (648.0 b)
TX bytes:690 (690.0 b)
Link encap:Local Loopback
inet addr:127.0.0.1
Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b)
TX bytes:0 (0.0 b)
[root@ip-10-10-125-8 docker_code]#
可以看到ip重新赋予了，实现了持久化固定ip功能。
下面是创建过程中使用的脚本.大家参考就行，没必要直接复制使用，每个人的需求都是不一样的。
[root@ip-10-10-125-8 docker_code]# cat create_docker_container_multi.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
#author:Deng Lei
import sys
import etcd
import time
import socket, struct, fcntl
from docker import Client
import subprocess
from multiprocessing import cpu_count
#get local host ip
def get_local_ip(iface = 'em1'):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sockfd = sock.fileno()
SIOCGIFADDR = 0x8915
ifreq = struct.pack('16sH14s', iface, socket.AF_INET, '\x00'*14)
res = fcntl.ioctl(sockfd, SIOCGIFADDR, ifreq)
return None
ip = struct.unpack('16sH2x4s8x', res)[2]
return socket.inet_ntoa(ip)
def docker_container_all():
docker_container=docker_client.containers(all=True)
container_name=[]
container_stop_name=[]
for i in docker_container:
container_name.append(i['Names'])
for b in container_name:
for c in b:
container_stop_name.append(c[1:])
return container_stop_name
def get_now_cpu_core():
#get recycle cpu use in etcd
recycle_cpu_key='%s%s/resource_limit/cpu_limit/now_cpu_core_avaiable'%(docker_etcd_key,local_ip)
all_recycle_cpu_result=eval(etcd_client.read(recycle_cpu_key).value)
except etcd.EtcdKeyNotFound:
all_recycle_cpu_result=''
if len(all_recycle_cpu_result) & 0:
now_core={}
now_core['now_core']=all_recycle_cpu_result[0].split('-')[0]
now_core['now_core_share']=all_recycle_cpu_result[0].split('-')[1]
all_recycle_cpu_result.remove(all_recycle_cpu_result[0])
etcd_client.write(recycle_cpu_key,all_recycle_cpu_result)
return now_core
#get local core use in etcd
now_core={'count':cpu_core,'now_core':cpu_core-1,'core_share_limit':cpu_core_share_limit,'now_core_share':-1}
now_core_use=eval(etcd_client.read('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip)).value)
except KeyError:
now_core_use=now_core
etcd_client.write('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip),now_core)
time.sleep(1)
except etcd.EtcdKeyNotFound:
now_core_use=now_core
etcd_client.write('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip),now_core)
time.sleep(1)
if int(now_core_use['now_core']) == int((cpu_core_retain - 1)):
msg={'status':1,'resource':'cpu','result':'the avaiable cpu core is less than retain:%s,i can not create new container!'%cpu_core_retain}
sys.exit(1)
if int(now_core_use['now_core_share']+1) &= int(now_core_use['core_share_limit']):
now_core_use['now_core']=now_core_use['now_core']-1
if int(now_core_use['now_core']) == int((cpu_core_retain - 1)):
msg={'status':1,'resource':'cpu','result':'the avaiable cpu core is less than retain:%s,i can not create new container!'%cpu_core_retain}
sys.exit(1)
now_core_use['now_core_share']=0
now_core_use['now_core_share']=now_core_use['now_core_share']+1
etcd_client.write('%s%s/resource_limit/cpu_limit/now_cpu_core'%(docker_etcd_key,local_ip),now_core_use)
return now_core_use
def get_container_ip():
#get avaiable container ip
recycle_ip_key='%scontainer/now_ip_avaiable'%docker_etcd_key
all_recycle_ip_result=eval(etcd_client.read(recycle_ip_key).value)
except etcd.EtcdKeyNotFound:
all_recycle_ip_result=''
if len(all_recycle_ip_result) & 0:
now_recycle_ip=all_recycle_ip_result[0]
all_recycle_ip_result.remove(all_recycle_ip_result[0])
etcd_client.write(recycle_ip_key,all_recycle_ip_result)
return now_recycle_ip
#get new container ip
now_container_ip=etcd_client.read('%scontainer/now_ip'%(docker_etcd_key)).value
except KeyError:
now_container_ip=default_container_ip
except etcd.EtcdKeyNotFound:
now_container_ip=default_container_ip
ip=now_container_ip.split('/')[0]
netmask=now_container_ip.split('/')[1]
if int(ip.split('.')[-1]) &254:
new_container_ip='.'.join(ip.split('.')[0:3])+'.'+str(int(ip.split('.')[-1])+1)+'/'+netmask
elif int(ip.split('.')[-1]) &=254 and int(ip.split('.')[-2]) &254:
if int(ip.split('.')[-1]) ==254:
last_ip='1'
last_ip=int(ip.split('.')[-1])+1
new_container_ip='.'.join(ip.split('.')[0:2])+'.'+str(int(ip.split('.')[-2])+1)+'.'+last_ip+'/'+netmask
elif int(ip.split('.')[-2]) &=254 and int(ip.split('.')[-3])&254:
if int(ip.split('.')[-1]) == 254:
last_ip='1'
last_ip=int(ip.split('.')[-1])+1
new_container_ip='.'.join(ip.split('.')[0:2])+'.'+str(int(ip.split('.')[2])+1)+'.'+last_ip+'/'+netmask
elif int(ip.split('.')[-2]) ==255 and int(ip.split('.')[-1])==254:
print 'now ip had more than ip pool!'
sys.exit(1)
etcd_client.write('%scontainer/now_ip'%docker_etcd_key,new_container_ip)
return new_container_ip
if __name__ == &__main__&:
bridge='ovs2'
default_physics_ip='172.16.0.1/16'
default_container_ip='172.16.1.1/16'
local_ip=get_local_ip('ovs1')
if local_ip is None:
local_ip=get_local_ip('em1')
docker_etcd_key='/app/docker/'
container=sys.argv[1]
if container == &-h& or container == &-help&:
print 'Usage: container_name container_image command'
sys.exit(1)
except IndexError:
print 'print input container name!'
sys.exit(1)
images=sys.argv[2]
except IndexError:
print 'print input images!'
sys.exit(1)
daemon_program=sys.argv[3]
except IndexError:
print 'please input run image commmand!'
sys.exit(1)
memory_limit=sys.argv[4]
except IndexError:
print 'please input memory limit where unit = b, k, m or g!'
sys.exit(1)
network_limit=sys.argv[5]
except IndexError:
print 'please input network limit where unit = M!'
sys.exit(1)
ssh_method=sys.argv[6]
except IndexError:
print 'please input container ssh auth method by key or passwd!'
sys.exit(1)
ssh_auth=sys.argv[7]
except IndexError:
print 'please input container ssh auth passwd or key_context!'
sys.exit(1)
ulimit=sys.argv[8]
except IndexError:
ulimit=65535
etcd_client=etcd.Client(host='127.0.0.1', port=4001)
cpu_core=cpu_count()
cpu_core_retain=2
cpu_core_share_limit=5
local_dir=sys.path[0]
#get local core use in etcd
now_core=get_now_cpu_core()
docker_client = Client(base_url='unix://var/run/docker.sock', version='1.15', timeout=10)
docker_container_all_name=docker_container_all()
#check input container exist
if container in docker_container_all_name:
print 'The container %s is exist!'%container
sys.exit(1)
#check pipework software exist
check_pipework_status=((subprocess.Popen(&which pipework &&&/dev/null && echo 0 || echo 1&,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_pipework_status != &0&:
print 'I can not find pipework in this host!'
sys.exit(1)
#check ovs bridge
check_bridge_status=((subprocess.Popen(&ovs-vsctl
list-br |grep %s &&&/dev/null && echo 0 || echo 1&%bridge,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_bridge_status !=&0&:
subprocess.Popen(&ovs-vsctl add-br %s&%bridge,shell=True,stdout=subprocess.PIPE)
#check default
physics ip
default_physics_ip=etcd_client.read('%sphysics/now_ip'%(docker_etcd_key)).value
except KeyError:
default_physics_ip=default_physics_ip
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,default_physics_ip)
time.sleep(1)
except etcd.EtcdKeyNotFound:
default_physics_ip=default_physics_ip
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,default_physics_ip)
time.sleep(1)
#check physics ip
new_physics_ip=etcd_client.read('%sphysics/%s'%(docker_etcd_key,local_ip)).value
except KeyError:
now_physics_ip=etcd_client.read('%sphysics/now_ip'%(docker_etcd_key)).value
ip=now_physics_ip.split('/')[0]
netmask=now_physics_ip.split('/')[1]
if int(ip.split('.')[-1]) &254:
new_physics_ip='.'.join(ip.split('.')[0:3])+'.'+str(int(ip.split('.')[-1])+1)+'/'+netmask
elif int(ip.split('.')[-1]) &=254:
print 'physics gateway ip more than set up ip pool!'
sys.exit(1)
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,new_physics_ip)
etcd_client.write('%sphysics/%s'%(docker_etcd_key,local_ip),new_physics_ip)
except etcd.EtcdKeyNotFound:
now_physics_ip=etcd_client.read('%sphysics/now_ip'%(docker_etcd_key)).value
ip=now_physics_ip.split('/')[0]
netmask=now_physics_ip.split('/')[1]
if int(ip.split('.')[-1]) &254:
new_physics_ip='.'.join(ip.split('.')[0:3])+'.'+str(int(ip.split('.')[-1])+1)+'/'+netmask
elif int(ip.split('.')[-1]) &=254:
print 'physics gateway ip more than set up ip pool!'
sys.exit(1)
etcd_client.write('%sphysics/now_ip'%docker_etcd_key,new_physics_ip)
etcd_client.write('%sphysics/%s'%(docker_etcd_key,local_ip),new_physics_ip)
#set physics ip
#run_command=&awk -v value=`ifconfig %s|grep inet|grep -v inet6|awk '{print $2}'|wc -l` 'BEGIN{if(value&1) {system('/sbin/ifconfig %s %s')}}'&%(bridge,bridge,new_physics_ip)
set_physics_status=((subprocess.Popen(&/sbin/ifconfig %s %s && echo 0 || echo 1&%(bridge,new_physics_ip),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if set_physics_status !=&0&:
print 'config local physics ip:%s is fail!'%new_physics_ip
sys.exit(1)
docker_version=docker_client.version()['Version'][0:3]
if float(docker_version) &= 1.6 :
cpu_tag='--cpuset-cpus'
container_ulimit=&--ulimit nofile=%s:%s&%(ulimit,ulimit)
cpu_tag='--cpuset'
container_ulimit=''
#create new container
get_container_id=((subprocess.Popen(&docker run --restart always %s -d -m %s --memory-swap=%s %s=%s --net='none' --name='%s' %s %s&%(container_ulimit,memory_limit,memory_limit,cpu_tag,now_core['now_core'],container,images,daemon_program),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
check_container_status=((subprocess.Popen(&docker inspect %s|grep Id &&&/dev/null && echo 0 || echo 1&%(container),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_container_status !=&0&:
print 'Create container %s is fail!'%container
sys.exit(1)
#get container ip
new_container_ip=get_container_ip()
#check container network
new_physics_ip=new_physics_ip.split('/')[0]
set_container_status=((subprocess.Popen(&`which pipework` %s %s %s@%s
&&&/dev/null && echo 0 || echo 1&%(bridge,container,new_container_ip,new_physics_ip),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if set_container_status != &0&:
print 'config container:%s network is fail!'%container
sys.exit(1)
#set container network limit
set_container_network_limit=((subprocess.Popen(&/bin/bash %s/modify_docker_container_network_limit.sh limit %s %s
&&&/dev/null && echo 0 || echo 1&%(local_dir,container,network_limit),shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if set_container_network_limit != &0&:
print 'config container:%s network limit:%s is fail!'%(container,network_limit)
sys.exit(1)
#write container network limit to etcd
etcd_client.write('%s%s/resource_limit/network_limit/%s'%(docker_etcd_key,local_ip,container),network_limit)
#modify container ssh auth
if ssh_method == 'key':
cmd='docker exec %s sh -c &echo \'%s\' &&/root/.ssh/authorized_keys && echo 0 || echo 1&'%(container,ssh_auth)
elif ssh_method == 'passwd':
cmd='docker exec %s sh -c &echo &%s& | passwd --stdin root &&&/dev/null && echo 0 || echo 1&'%(container,ssh_auth)
check_ssh_status=((subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
if check_ssh_status != &0&:
print 'config container:%s ssh auth method:%s is fail!'%(container,ssh_method)
delete_cmd='docker rm $(docker stop %s)'%container
delete_ssh_status=((subprocess.Popen(delete_cmd,shell=True,stdout=subprocess.PIPE)).stdout.readlines()[0]).strip('\n')
sys.exit(1)
#write container info to etcd
key='%s%s/container/%s'%(docker_etcd_key,local_ip,container)
value={'Physics_ip':'%s'%local_ip,'Container_id':get_container_id,'Container_name':container,'Container_ip':'%s'%new_container_ip,'Container_gateway_bridge':bridge,'Container_gateway_ip':new_physics_ip,'Container_create':time.strftime('%Y.%m.%d-%T'),'Container_status':'running','Cpu_core':now_core['now_core'],'Cpu_share':now_core['now_core_share'],'Memory_limit':memory_limit,'Network_limit':network_limit}
etcd_client.write(key,value)
#print 'create container:%s is success.follow is container info.'%container
print value
这个脚本是创建容器并进行资源限制的
下面脚本是进行网络资源限制，上面脚本会调用做网络限制
[root@ip-10-10-125-8 docker_code]# cat modify_docker_container_network_limit.sh
#!/bin/bash
#filename:modify_docker_container_network_limit.sh
#author:Deng Lei
container=$2
if [ -z $1 ] || [ -z $2 ]; then
echo &Usage: operation container_name limit(default:5m)&
echo &Example1: I want limit 5m in the container:test&
echo &The command is: bash `basename $0` limit test 5&
echo &Example2: I want delete network limit in the container:test&
echo &The command is: bash `basename $0` ulimit test&
if [ -z $3 ];then
limit='5m'
if [ `docker inspect --format &{{.State.Pid}}& $container &&&/dev/null && echo 0 || echo 1` -eq 1 ];then
echo &no this container:$container&
ovs_prefix='veth1pl'
container_id=`docker inspect --format &{{.State.Pid}}& $container`
device_name=`echo ${ovs_prefix}${container_id}`
docker_etcd_key='/app/docker/'
local_ip=`ifconfig ovs1|grep inet|awk '{print $2}'|head -n1`
if [ $op == 'limit' ];then
for v in $device_ do
ovs-vsctl set interface $v ingress_policing_rate=$((limit*1000))
ovs-vsctl set interface $v ingress_policing_burst=$((limit*100))
ovs-vsctl set port $v qos=@newqos -- --id=@newqos create qos type=linux-htb queues=0=@q0 other-config:max-rate=$((limit*1000000)) -- --id=@q0 create queue other-config:min-rate=$((limit*1000000)) other-config:max-rate=$((limit*1000000)) &&&/dev/null && echo '0' || echo '1'
#save result to etcd
`which etcdctl` set ${docker_etcd_key}${local_ip}/resource_limit/network_limit/${container} $limit &&&/dev/null
elif [ $op == 'ulimit' ];then
for v in $device_ do
ovs-vsctl set interface $v ingress_policing_rate=0
ovs-vsctl set interface $v ingress_policing_burst=0
ovs-vsctl clear Port $v qos &&&/dev/null && echo '0' || echo '1'
`which etcdctl` rm ${docker_etcd_key}${local_ip}/resource_limit/network_limit/${container} &&&/dev/null
已发表评论数()
请填写推刊名
描述不能大于100个字符!
权限设置： 公开
仅自己可见
正文不准确
标题不准确
排版有问题
主题不准确
没有分页内容
图片无法显示
视频无法显示
与原文不一致}
我要回帖
更多关于 docker容器内应用日志 的文章
更多推荐
- ·QQ挂机平台是指QQ在后台运行吗？
- ·月考成绩被多算了十二分，找老师说，老师也没管，还被发了奖状补考57分怎么求老师改成绩办？
- ·熟悉的说一下，哪个农村物流管理系统软件有哪些软件好用？
- ·轨道倾斜位移测量传感器用什么倾斜传感器比较好？
- ·wps表格中的宏被禁止使用怎么办l宏运行时错误13类型不匹配？
- ·降价为千元机的旗舰机旗舰，荣耀5c和5x哪个更优秀
- ·电子称充电器和手机(小灵通手机改)充电器一样吗
- ·谁知道在东平去台前的前四后八车主的手机号，积极急急急积极急急急积极急急急
- ·看追啊手机怎么od追注册码
- ·note4 能推动 vsd5和血色围城 吗， 还是vsd3 好些 ， 不要推荐其他的， 我只买了这2个
- ·mewmark拉绳位移传感器分析中的Dn怎么得到
- ·奔腾b70官网没我油没电启动正常
- ·大Q1雷霆战机官网版手机Root死了机咋办
- ·这个手机壳男生用粉色手机壳可以吗，会不会有斜眼
- ·如何清洗空调过滤网进气管有无过濾网
- ·为什么obs说话不停地重复提几乎是重复的问题会失效
- ·我想在美团解绑手机号上下订单，外卖送过来，可是为什么我填写手机号的时候它却说凭借美团解绑手机号密码去本
- ·换机精灵下载系统内存不够有安装内存卡怎么解决？
- ·佳能mg7580 拆 打印头的打印头即小车怎么拆下来？要更换
- ·苹果a1524是什么型号iphone型号/A1524多少钱
- ·如何设置docker限制容器内存容器中Java应用的内存限制
- ·怎么取消空调emo紧急停止开关ems开关
- ·谁有锁怎么看手机有没有锁的software,给我一个！
- ·手机被偷了，有四位字母的屏幕密码，内部信息被暴露的英国退欧可能性有多大大呀，担心中，求解答。
- ·加工中心刀库种类刀具无放下动作怎么回事
- ·电脑机箱重启按钮路由器断网重启后正常，就直接进入这…。 是系统坏了还是硬盘？ 退出了
- ·天猫网极能三菱电机空调摇控器电池
- ·亲们大家好！我现在急需柯尼卡美能达官方网7235复印机电容，帮忙提供，或提供信息。合十感谢哦
- ·包头市麻池北新村2016麻池镇麻池村张德的手机号
- ·冀tpa210车有保险 tpa公司吗
- ·为什么说深度学习是金融创新产品的新引擎
- ·开通了邮储手机银行，就是在支付宝看银行卡余额里可以查询邮储银行卡的余额了，要
- ·农村合作医疗保险可以补交吗补交多久可以用？我还有四十天生孩子才想起因为之前有别的保险所以农村合作医疗保险可以补交吗不给投了，现
- ·我在石家庄暂住证如何办理没有暂住证，但是有在石家庄暂住证如何办理工作的证明，还有购房合同，租房合同，可以直接上石家庄暂住证如何办理牌照吗？
- ·买卖双方都直接或间接控制的受第三方控制是什么意思
