openstack swift container server的进程经常莫名其妙进入 D Ds等状态
记录一下这个时候 storage.error的log 便于分析
一种情形是下面这种log
Jun 3 16:46:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012 broker.put_object(obj, timestamp, int(req.headers['x-size']),#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012 return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun 3 16:51:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012 broker.put_object(obj, timestamp, int(req.headers['x-size']),#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012 return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun 3 16:56:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012 broker.put_object(obj, timestamp, int(req.headers['x-size']),#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012 return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun 3 17:01:03 host207 container-server: ERROR __call__ error with PUT /sda5/100256/AUTH_3ed1d3f47b4e4794a59b7217bf136eee/atest1/data4M : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 498, in __call__#012 res = method(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 2217, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 837, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/container/server.py", line 271, in PUT#012 broker.put_object(obj, timestamp, int(req.headers['x-size']),#012 File "/usr/lib/python2.7/site-packages/swift/common/swob.py", line 219, in __getitem__#012 return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_SIZE' (txn: tx1b48300503fc454d82e07-0055498a24)
Jun 3 17:59:27 host207 object-replicator: Lockup detected.. killing live coros.
Jun 3 18:23:19 host207 container-server: SIGTERM received
Jun 3 18:23:19 host207 account-server: SIGTERM received
Jun 3 18:23:19 host207 object-server: SIGTERM received