最近接手的是个python脚本,通过crontab定期运行,用facebook提供的api抓取数据,处理后再插入到数据库.
许多人反映这个服务不稳定,抓取的数据不全.
我在测试的过程中发现,这个脚本开发的时间已经过了很久了,facebook提供的接口数据已经有所调整. 所以会有数据
格式不对的问题;还有就是在跑了个小时后,会抛出 ‘MySQL server has gone away’ 的异常..
网上找到的类似案例如下:
http://stackoverflow.com/questions/8689649/solving-mysql-server-has-gone-away-errors
http://stackoverflow.com/questions/7942154/mysql-error-2006-mysql-server-has-gone-away
我观察到操作数据库的类库是torndb,已经过去这么久,会不会这个bug在官方也有反应并修复了呢.
果不其然..
1:
2: def _ensure_connected(self):
3: # Mysql by default closes client connections that are idle for
4: # 8 hours, but the client library does not report this fact until
5: # you try to perform a query and it fails. Protect against this
6: # case by preemptively closing and reopening the connection
7: # if it has been idle for too long (7 hours by default).
8: if (self._db is None or
9: (time.time() - self._last_use_time > self.max_idle_time)):
10: self.reconnect()
11: self._last_use_time = time.time()