appnode面板以及httpd、mysql自动关闭(已解决,防火墙问题)

魅影 2018-7-20 1129

2018.7.20号晚上9点左右,网站不能访问,ssh查看top后appnode面板以及网站相关服务各种重启。

然后我重启服务器,都不能恢复,尝试手动启动也没用,还是自动关闭。

附件有top截图以及云服务器面板截图。

希望appnode官方给一个解决办法。

上传的附件:
  • 1.png (大小:43.60K,下载次数:275)
  • 2.png (大小:38.56K,下载次数:272)
  • 3.png (大小:38.32K,下载次数:270)
  • 4.png (大小:30.01K,下载次数:281)
最新回复 (14)
  • 虚竹 2018-7-20
    引用 2

    给出以下命令的输出结果(敏感信息自己过滤):

    rpm -q appnode-ccenter
    tail -n 50 /var/log/appnode/ccenter.error.log
    grep mysql /var/log/messages
  • 魅影 2018-7-20
    引用 3
    appnode-ccenter-1.12.7-1.el6.x86_64

    Existing lock /var/run/yum.pid: another copy is running as pid 2083.
    Another app is currently holding the yum lock; waiting for it to exit...
      The other application is: yum
        Memory :  19 M RSS (211 MB VSZ)
        Started: Fri Jul 20 20:31:23 2018 - 00:01 ago
        State  : Running, pid: 2083
    Existing lock /var/run/yum.pid: another copy is running as pid 2091.
    Another app is currently holding the yum lock; waiting for it to exit...
      The other application is: yum
        Memory :  20 M RSS (214 MB VSZ)
        Started: Fri Jul 20 20:31:24 2018 - 00:02 ago
        State  : Sleeping, pid: 2091
    Existing lock /var/run/yum.pid: another copy is running as pid 2082.
    Another app is currently holding the yum lock; waiting for it to exit...
      The other application is: yum
        Memory :  19 M RSS (211 MB VSZ)
        Started: Fri Jul 20 20:40:19 2018 - 00:01 ago
        State  : Running, pid: 2082
    Existing lock /var/run/yum.pid: another copy is running as pid 2102.
    Another app is currently holding the yum lock; waiting for it to exit...
      The other application is: yum
        Memory :  71 M RSS (270 MB VSZ)
        StarteExisting lock /var/run/yum.pid: another copy is running as pid 2107.
    Another app is currently holding the yum lock; waiting for it to exit...
      The other application is: yum
        Memory :  78 M RSS (277 MB VSZ)
        Started: Fri Jul 20 21:15:57 2018 - 00:03 ago
        State  : Running, pid: 2107

    第三个mysql的日志没有返回结果。
  • 虚竹 2018-7-20
    引用 4

    看日志像是你的系统中有其它yum在运行,先把appnode全部停止:

    appnode ccenter stop
    appnode agent stop

    然后查看下yum命令是否能正常输出:

    yum makecache
  • 魅影 2018-7-20
    引用 5

    好像是可以的。已经停止掉面板。。

    上传的附件:
    • 1.png (大小:37.22K,下载次数:277)
  • 虚竹 2018-7-20
    引用 6

    yum makecache 命令执行结束后,再启动appnode,并且查看日志:

    appnode ccenter start
    appnode agent start
    tail -n 100 /var/log/appnode/ccenter.error.log
  • 魅影 2018-7-20
    引用 7

    虚竹大神请看。

    上传的附件:
    • 1.png (大小:53.09K,下载次数:304)
  • 虚竹 2018-7-20
    引用 8

    你的输出看起来不太对劲啊,tail -n 100是输出最后100行,怎么你输出的还不满100行。
    如果没有敏感数据,直接把 /var/log/appnode/ccenter.error.log 文件上传上来看看。

    另外,看下日志 /var/log/messages 是否有 appnode 被意外中止的日志:

    tail -n 100 /var/log/messages
  • 魅影 2018-7-20
    引用 9
    好像日志还是和之前一样,这个是面板崩溃了嘛?
  • 魅影 2018-7-20
    引用 10
    [root@nosnLt141874 ~]# tail -n 100 /var/log/messages
    Jul 20 21:36:47 nosnLt141874 kernel: mptspi 0000:00:10.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
    Jul 20 21:36:47 nosnLt141874 kernel: mptbase: ioc0: Initiating bringup
    Jul 20 21:36:47 nosnLt141874 kernel: ioc0: LSI53C1030 B0: Capabilities={Initiator}
    Jul 20 21:36:47 nosnLt141874 kernel: Refined TSC clocksource calibration: 2593.993 MHz.
    Jul 20 21:36:47 nosnLt141874 kernel: Switching to clocksource tsc
    Jul 20 21:36:47 nosnLt141874 kernel: scsi0 : ioc0: LSI53C1030 B0, FwRev=01032920h, Ports=1, MaxQ=128, IRQ=17
    Jul 20 21:36:47 nosnLt141874 kernel: scsi 0:0:0:0: Direct-Access     VMware,  VMware Virtual S 1.0  PQ: 0 ANSI: 2
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:0: Beginning Domain Validation
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:0: Domain Validation skipping write tests
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:0: Ending Domain Validation
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
    Jul 20 21:36:47 nosnLt141874 kernel: scsi 0:0:1:0: Direct-Access     VMware,  VMware Virtual S 1.0  PQ: 0 ANSI: 2
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:1: Beginning Domain Validation
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:1: Domain Validation skipping write tests
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:1: Ending Domain Validation
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
    Jul 20 21:36:47 nosnLt141874 kernel: scsi 0:0:2:0: CD-ROM            VMware,  Virtual CD-ROM   1.0  PQ: 0 ANSI: 2
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:2: Beginning Domain Validation
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:2: Domain Validation skipping write tests
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:2: Ending Domain Validation
    Jul 20 21:36:47 nosnLt141874 kernel: scsi target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
    Jul 20 21:36:47 nosnLt141874 kernel: scsi1 : ata_piix
    Jul 20 21:36:47 nosnLt141874 kernel: scsi2 : ata_piix
    Jul 20 21:36:47 nosnLt141874 kernel: ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0x1060 irq 14
    Jul 20 21:36:47 nosnLt141874 kernel: ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0x1068 irq 15
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] 83886080 512-byte logical blocks: (42.9 GB/40.0 GiB)
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Write Protect is off
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Cache data unavailable
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Write Protect is off
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Cache data unavailable
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Assuming drive cache: write through
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Cache data unavailable
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
    Jul 20 21:36:47 nosnLt141874 kernel: sda: sda1
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Cache data unavailable
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Assuming drive cache: write through
    Jul 20 21:36:47 nosnLt141874 kernel: sdb: sdb1
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Cache data unavailable
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: [sda] Attached SCSI disk
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Cache data unavailable
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Assuming drive cache: write through
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: [sdb] Attached SCSI disk
    Jul 20 21:36:47 nosnLt141874 kernel: sr0: scsi3-mmc drive: 1x/1x writer dvd-ram cd/rw xa/form2 cdda tray
    Jul 20 21:36:47 nosnLt141874 kernel: Uniform CD-ROM driver Revision: 3.20
    Jul 20 21:36:47 nosnLt141874 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
    Jul 20 21:36:47 nosnLt141874 kernel: dracut: Mounted root filesystem /dev/sda1
    Jul 20 21:36:47 nosnLt141874 kernel: SELinux:  Disabled at runtime.
    Jul 20 21:36:47 nosnLt141874 kernel: type=1404 audit(1532093795.480:2): selinux=0 auid=4294967295 ses=4294967295
    Jul 20 21:36:47 nosnLt141874 kernel: dracut: 
    Jul 20 21:36:47 nosnLt141874 kernel: dracut: Switching root
    Jul 20 21:36:47 nosnLt141874 kernel: udev: starting version 147
    Jul 20 21:36:47 nosnLt141874 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
    Jul 20 21:36:47 nosnLt141874 kernel: sd 0:0:1:0: Attached scsi generic sg1 type 0
    Jul 20 21:36:47 nosnLt141874 kernel: sr 0:0:2:0: Attached scsi generic sg2 type 5
    Jul 20 21:36:47 nosnLt141874 kernel: e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
    Jul 20 21:36:47 nosnLt141874 kernel: e1000: Copyright (c) 1999-2006 Intel Corporation.
    Jul 20 21:36:47 nosnLt141874 kernel: e1000 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
    Jul 20 21:36:47 nosnLt141874 kernel: e1000 0000:02:00.0: eth0: (PCI:66MHz:32-bit) 00:0c:29:f2:a6:cc
    Jul 20 21:36:47 nosnLt141874 kernel: e1000 0000:02:00.0: eth0: Intel(R) PRO/1000 Network Connection
    Jul 20 21:36:47 nosnLt141874 kernel: e1000 0000:02:01.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
    Jul 20 21:36:47 nosnLt141874 kernel: e1000 0000:02:01.0: eth1: (PCI:66MHz:32-bit) 00:0c:29:f2:a6:d6
    Jul 20 21:36:47 nosnLt141874 kernel: e1000 0000:02:01.0: eth1: Intel(R) PRO/1000 Network Connection
    Jul 20 21:36:47 nosnLt141874 kernel: EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
    Jul 20 21:36:47 nosnLt141874 kernel: NET: Registered protocol family 10
    Jul 20 21:36:47 nosnLt141874 kernel: lo: Disabled Privacy Extensions
    Jul 20 21:36:47 nosnLt141874 kernel: e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    Jul 20 21:36:47 nosnLt141874 kernel: e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    Jul 20 21:36:47 nosnLt141874 kernel: type=1305 audit(1532093807.556:3): audit_pid=1145 old=0 auid=4294967295 ses=4294967295 res=1
    Jul 20 21:36:48 nosnLt141874 ntpd[1246]: ntpd 4.2.6p5@1.2349-o Mon Feb  6 07:22:46 UTC 2017 (1)
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: proto: precision = 0.102 usec
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: 0.0.0.0 c01d 0d kern kernel time sync enabled
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: line 16 column 28 syntax error, unexpected T_Iburst, expecting T_EOC
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: syntax error in /etc/ntp.conf line 16, column 28
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: line 17 column 28 syntax error, unexpected T_Iburst, expecting T_EOC
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: syntax error in /etc/ntp.conf line 17, column 28
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: line 18 column 28 syntax error, unexpected T_Iburst, expecting T_EOC
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: syntax error in /etc/ntp.conf line 18, column 28
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: line 19 column 28 syntax error, unexpected T_Iburst, expecting T_EOC
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: syntax error in /etc/ntp.conf line 19, column 28
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: Listen and drop on 1 v6wildcard :: UDP 123
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: Listen normally on 2 lo 127.0.0.1 UDP 123
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: Listen normally on 3 eth0 43.226.40.153 UDP 123
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: Listen normally on 4 eth1 10.226.40.153 UDP 123
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: Listening on routing socket on fd #21 for interface updates
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: 0.0.0.0 c016 06 restart
    Jul 20 21:36:48 nosnLt141874 ntpd[1247]: 0.0.0.0 c012 02 freq_set kernel -6.130 PPM
    Jul 20 21:36:50 nosnLt141874 udevd[459]: can not read '/etc/udev/rules.d/70-persistent-net.rules'
    Jul 20 21:36:55 nosnLt141874 ntpd[1247]: 0.0.0.0 c61c 0c clock_step +0.299290 s
    Jul 20 21:36:55 nosnLt141874 ntpd[1247]: 0.0.0.0 c614 04 freq_mode
    Jul 20 21:36:56 nosnLt141874 ntpd[1247]: 0.0.0.0 c618 08 no_sys_peer
    Jul 20 21:37:05 nosnLt141874 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
    Jul 20 21:37:05 nosnLt141874 kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
    Jul 20 21:52:05 nosnLt141874 ntpd[1247]: 0.0.0.0 c612 02 freq_set kernel 27.491 PPM
    Jul 20 21:52:05 nosnLt141874 ntpd[1247]: 0.0.0.0 c615 05 clock_sync
    Jul 20 22:57:36 nosnLt141874 ntpd[1247]: 0.0.0.0 0613 03 spike_detect +0.350474 s
  • 魅影 2018-7-20
    引用 11

    ccenter.error.log
    日志和我发的是一样的。

    上传的附件:
  • 虚竹 2018-7-20
    引用 12
    提交工单吧,我具体看看日志,从你的描述上看应该是系统在kill进程。
  • 魅影 2018-7-20
    引用 13
    提交工单直接提交appnode面板工单是吗?
  • 虚竹 2018-7-20
    引用 14

    你的防火墙规则只允许22端口通行:

    # Firewall configuration written by system-config-firewall
    # Manual customization of this file is not recommended.
    *filter
    :Blacklist -
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    -A INPUT -j Blacklist
    -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    -A INPUT -i lo -j ACCEPT
    -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
    -A INPUT -j REJECT --reject-with icmp-host-prohibited
    -A FORWARD -j REJECT --reject-with icmp-host-prohibited
    COMMIT


    停止防火墙后,面板可访问:

    service iptables stop


    至于你说的一直重启,我观察了5分钟并没有出现,要怎么让它出现?

  • 魅影 2018-7-20
    引用 15
    感谢大佬。
返回
发新帖