gitlab错误修复

502错误修复

服务器断电关机，再开机时一般会导致 gitlab 502错误

$ gitlab-ctl status
run: gitaly: (pid 1072) 689s; run: log: (pid 1071) 689s
run: gitlab-monitor: (pid 1101) 689s; run: log: (pid 1090) 689s
run: gitlab-workhorse: (pid 1092) 689s; run: log: (pid 1078) 689s
run: logrotate: (pid 1115) 689s; run: log: (pid 1097) 689s
run: nginx: (pid 1091) 689s; run: log: (pid 1077) 689s
run: node-exporter: (pid 1111) 689s; run: log: (pid 1096) 689s
run: postgres-exporter: (pid 1113) 689s; run: log: (pid 1098) 689s
run: postgresql: (pid 1114) 689s; run: log: (pid 1095) 689s
run: prometheus: (pid 14267) 0s; run: log: (pid 1089) 689s
run: redis: (pid 1110) 689s; run: log: (pid 1094) 689s
run: redis-exporter: (pid 1075) 689s; run: log: (pid 1074) 689s
run: sidekiq: (pid 1076) 689s; run: log: (pid 1073) 689s
run: unicorn: (pid 14244) 1s; run: log: (pid 1079) 689s
# 尝试重启
$ gitlab-ctl restart
ok: run: gitaly: (pid 14454) 0s
ok: run: gitlab-monitor: (pid 14465) 1s
ok: run: gitlab-workhorse: (pid 14469) 0s
ok: run: logrotate: (pid 14482) 1s
ok: run: nginx: (pid 14508) 0s
ok: run: node-exporter: (pid 14514) 0s
ok: run: postgres-exporter: (pid 14519) 1s
ok: run: postgresql: (pid 14532) 0s
ok: run: prometheus: (pid 14544) 0s
ok: run: redis: (pid 14549) 0s
ok: run: redis-exporter: (pid 14558) 0s
ok: run: sidekiq: (pid 14575) 0s
ok: run: unicorn: (pid 14587) 0s

# 按照官网的建议
# 如果所有服务都正常（没有down掉）还是502错误的话，重启sidekiq
$ gitlab-ctl restart sidekiq

# 或者
$ gitlab-ctl hup unicorn 

500错误修复

记录一次500错误，是由于强制关掉宿主机导致的，用502错误的修复方法无效果。

# 通过status知道prometheus和redis两个服务不能正常启动
$ gitlab-ctl status
***
down: prometheu
down: redis
***

# 查看实时日志，发现prometheu(leveldb数据库存在问题要修复)和redis(dump.rdb存在问题要修复)在打印错误日志
$ gitlab-ctl tail

# 网上有修复leveldb的方法，但是太麻烦，python的版本太旧，修复leveldb存在困难
# 考虑到prometheu(监控告警时序数据，项目中没用到)和redis(缓存数据，丢了可以重新生成)的数据不重要，所以决定把损坏的数据库文件做删除处理。

# 1. 停止所有服务
$ gitlab-ctl stop

# 2. 移除prometheu的leveldb数据库
$ cd /var/opt/gitlab/prometheus
$ ls
data  prometheus.yml
$ mv data data_old
# 3. 移除redis持久化数据库
$ cd /var/opt/gitlab/redis
$ ls
dump.rdb  redis.conf  redis.socket  temp-56637.rdb
$ mv dump.rdb mump.rdb.bak
$ mv temp-56637.rdb temp-56637.rdb.bak
$ ls
$ mump.rdb.bak  redis.conf  redis.socket  temp-56637.rdb.bak

# 4. 重新启动服务
$ gitlab-ctl start
$ gitlab-ctl restart sidekiq
$ gitlab-ctl hup unicorn
# 5. 浏览器打开Gitlab正常，仓库数据没有丢失。