zoukankan      html  css  js  c++  java
  • hadoop集群格式化namenode报错(dfs.namenode.shared.edits.dir )

    hadoop集群格式化namenode报错
    ************************************************************/
    20/08/05 10:34:11 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
    20/08/05 10:34:11 INFO namenode.NameNode: createNameNode [-format]
    20/08/05 10:34:12 ERROR namenode.NameNode: Failed to start namenode.
    java.io.IOException: Multiple shared edits directories are not yet supported
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceEditsDirs(FSNam esystem.java:1429)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceEditsDirs(FSNam esystem.java:1414)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:972)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:14 34)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
    20/08/05 10:34:12 INFO util.ExitUtil: Exiting with status 1
    20/08/05 10:34:12 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at DM00/172.25.5.36
    ************************************************************/

    查看hdfs-site.xml文件,发现

    配置dfs.namenode.shared.edits.dir的value中不同的JN的地址和端口之间用“,”隔开的,应该用“;”
    dfs.namenode.shared.edits.dir 这是NameNode读写JNs组的uri。通过这个uri,NameNodes可以读写edit log内容。
    URI的格式"qjournal://host1:port1;host2:port2;host3:port3/journalId"。
    这里的host1、host2、host3指的是Journal Node的地址,这里必须是奇数个,至少3个;
    其中journalId是集群的唯一标识符,对于多个联邦命名空间,也使用同一个journalId。
    配置如下

    <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/mycluster</value>
    </property>

    https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

    https://www.cnblogs.com/followyourdream/p/6884037.html

    https://www.cnblogs.com/xiqing/p/9645619.html

    列出的所有端口都是TCP。

    在下表中,每个端口的“ 访问要求”列通常是“内部”或“外部”。在此上下文中,“内部”表示端口仅用于组件之间的通信; “外部”表示该端口可用于内部或外部通信。
    ComponentServiceQualifierPortAccess RequirementConfigurationComment

    Hadoop HDFS

    DataNode

    50010

    External

    dfs.datanode.address
    

    DataNode HTTP server port

    DataNode

    Secure

    1004

    External

    dfs.datanode.address
    

    DataNode

    50075

    External

    dfs.datanode.http.address
    

    DataNode

    50475

    External

    dfs.datanode.https.address
    

    DataNode

    Secure

    1006

    External

    dfs.datanode.http.address
    

    DataNode

    50020

    External

    dfs.datanode.ipc.address
    

    NameNode

    8020

    External

    fs.default.name
    
    or
    fs.defaultFS
    
    fs.default.name
    
    is deprecated (but still works)

    NameNode

    8022

    External

    dfs.namenode. servicerpc-address
    

    Optional port used by HDFS daemons to avoid sharing the RPC port used by clients (8020). Cloudera recommends using port 8022.

    NameNode

    50070

    External

    dfs.http.address
    
    or
    dfs.namenode.http-address
    
    dfs.http.address
    
    is deprecated (but still works)

    NameNode

    Secure

    50470

    External

    dfs.https.address
    
    or
    dfs.namenode.https-address
    
    dfs.https.address
    
    is deprecated (but still works)

    Secondary NameNode

    50090

    Internal

    dfs.secondary.http.address
    
    or
    dfs.namenode. secondary.http-address
    
    dfs.secondary.http.address
    
    is deprecated (but still works)

    Secondary NameNode

    Secure

    50495

    Internal

    dfs.secondary.https.address
    

    JournalNode

    8485

    Internal

    dfs.namenode.shared.edits.dir
    

    JournalNode

    8480

    Internal

    dfs.journalnode. http-address
    

    JournalNode

    8481

    Internal

    dfs.journalnode. https-address
    

    Failover Controller

    8019

    Internal

    Used for NameNode HA

    NFS gateway

    2049

    External

    nfs port (nfs3.server.port

    NFS gateway

    4242

    External

    mountd port (nfs3.mountd.port

    NFS gateway

    111

    External

    portmapper
    
    or
    rpcbind
    
    port
      NFS gateway   50079 External
    nfs.http.port
    
    CDH 5.4.0 and higher. The NFS gateway daemon uses this port to serve metrics. The port is configurable on versions 5.10 and higher.
      NFS gateway Secure 50579 External
    nfs.https.port
    
    CDH 5.4.0 and higher. The NFS gateway daemon uses this port to serve metrics. The port is configurable on versions 5.10 and higher.

    HttpFS

    14000

    External    

    HttpFS

    14001

    External    

    Hadoop YARN (MRv2)

    ResourceManager

    8032

    External

    yarn. resourcemanager.address
    

    ResourceManager

    8030

    Internal

    yarn. resourcemanager.scheduler.address
    

    ResourceManager

    8031

    Internal

    yarn. resourcemanager.resource-tracker. address
    

    ResourceManager

    8033

    External

    yarn. resourcemanager.admin.address
    

    ResourceManager

    8088

    External

    yarn. resourcemanager.webapp.address
    

    ResourceManager

    8090

    External

    yarn. resourcemanager.webapp.https.address
    

    NodeManager

    8040

    Internal

    yarn. nodemanager.localizer. address
    

    NodeManager

    8041

    Internal

    yarn. nodemanager. address
    

    NodeManager

    8042

    External

    yarn. nodemanager.webapp.address
    

    NodeManager

    8044

    External

    yarn. nodemanager.webapp.https.address
    

    JobHistory Server

    10020

    Internal

    mapreduce. jobhistory.address
    

    JobHistory Server

    10033

    Internal

    mapreduce.jobhistory.admin. address
    

    Shuffle HTTP

    13562

    Internal

    mapreduce.shuffle.port

    JobHistory Server

    19888

    External

    mapreduce. jobhistory.webapp.address
    

    JobHistory Server

    19890

    External

    mapreduce. jobhistory.webapp.https.address
    
     

    ApplicationMaster

       

    External

     

    The ApplicationMaster serves an HTTP service using an ephemeral port that cannot be restricted. This port is never accessed directly from outside the cluster by clients. All requests to the ApplicationMaster web server is routed using the YARN ResourceManager (proxy service). Locking down access to ephemeral port ranges within the cluster's network might restrict your access to the ApplicationMaster UI and its logs, along with the ability to look at running applications.

    Flume

    Flume Agent

    41414

    External

       

    Hadoop KMS

    Key Management Server

    16000

    External

    kms_http_port
    

    CDH 5.2.1 and higher. Applies to both Java KeyStore KMS and Key Trustee KMS.

     

    Key Management Server

    16001

    Localhost

    kms_admin_port
    

    CDH 5.2.1 and higher. Applies to both Java KeyStore KMS and Key Trustee KMS.

    HBase

    Master

    60000

    External

    hbase.master. port
    

    IPC

    Master

    60010

    External

    hbase.master. info.port
    

    HTTP

    RegionServer

    60020

    External

    hbase. regionserver. port
    

    IPC

    RegionServer

    60030

    External

    hbase. regionserver.info.port
    

    HTTP

    HQuorumPeer

    2181

    Internal

    hbase. zookeeper. property.clientPort
    

    HBase-managed ZooKeeper mode

    HQuorumPeer

    2888

    Internal

    hbase. zookeeper. peerport
    

    HBase-managed ZooKeeper mode

    HQuorumPeer

    3888

    Internal

    hbase. zookeeper.leaderport
    

    HBase-managed ZooKeeper mode

    REST

    Non- Cloudera Manager - managed

    8080

    External

    hbase.rest.port
    

    The default REST port in HBase is 8080. Because this is a commonly used port, Cloudera Manager sets the default to 20550 instead.

    REST

    Cloudera Manager - managed

    20550

    External

    hbase.rest.port
    

    The default REST port in HBase is 8080. Because this is a commonly used port, Cloudera Manager sets the default to 20550 instead.

    REST UI

     

    8085

    External

     

    Thrift Server

    Thrift Server

    9090

    External

    Pass -p <port> on CLI

    Thrift Server

     

    9095

    External

     

    Avro server

    9090

    External

    Pass --port <port> on CLI

      hbase-solr-indexer Lily Indexer 11060 External    

    Hive

    Metastore

    9083

    External

    HiveServer2

    10000

    External

    hive. server2. thrift.port
    

    The Beeline command interpreter requires that you specify this port on the command line.

    If you use Oracle database, you must manually reserve this port. For more information, see Reserving Ports for HiveServer 2.

    HiveServer2 Web User Interface (UI)

    10002

    External

    hive. server2. webui.port
    
    in
    hive-site.xml
    
     

    WebHCat Server

    50111

    External

    templeton.port
    

    Hue

    Server

    8888

    External

    Kafka

    Broker

    TCP Port

    9092

    External/Internal

    port

    The primary communication port used by producers and consumers; also used for inter-broker communication.

     

    Broker

    TLS/SSL Port

    9093

    External/Internal

    ssl_port

    A secured communication port used by producers and consumers; also used for inter-broker communication.

     

    Broker

    JMX Port

    9393

    Internal

    jmx_port

    Internal use only. Used for administration via JMX.

     

    MirrorMaker

    JMX Port

    9394

    Internal

    jmx_port

    Internal use only. Used to administer the producer and consumer of the MirrorMaker.

     

    Broker

    HTTP Metric Report Port

    24042

    Internal

    kafka.http.metrics.port

    Internal use only. This is the port via which the HTTP metric reporter listens. It is used to retrieve metrics through HTTP instead of JMX.

    Kudu

    Master

     

    7051

    External

     

    Kudu Master RPC port

     

    Master

     

    8051

    External

     

    Kudu Master HTTP server port

     

    TabletServer

     

    7050

    External

     

    Kudu TabletServer RPC port

     

    TabletServer

     

    8050

    External

     

    Kudu TabletServer HTTP server port

    Oozie

    Oozie Server

    11000

    External

    OOZIE_HTTP_PORT
    
    in
    oozie-env.sh
    

    HTTP

     

    Oozie Server

    SSL

    11443

    External

     

    HTTPS

    Sentry

    Sentry Server

    8038

    External

    sentry.service. server.rpc-port
    
     

    Sentry Server

    51000

    External

    sentry.service. web.port
    

    Spark

    Default Master RPC port

     

    7077

    External

       
     

    Default Worker RPC port

     

    7078

    External    
     

    Default Master web UI port

     

    18080

    External

       
     

    Default Worker web UI port

     

    18081

    External    
     

    History Server

    18088

    External

    history.port
    
     
      Shuffle service   7337 Internal    

    Sqoop

    Metastore

    16000

    External

    sqoop. metastore.server.port
    

    Sqoop 2

    Sqoop 2 server

     

    8005

    Localhost

    SQOOP_ADMIN_PORTenvironment variable

     
     

    Sqoop 2 server

    12000

    External

     

    Sqoop 2

    12001

    External

    Admin port

    ZooKeeper

    Server (with CDH 5 or Cloudera Manager 5)

    2181

    External

    clientPort

    Client port

    Server (with CDH 5 only)

    2888

    Internal

    X in server.N =host:X:Y
    

    Peer

    Server (with CDH 5 only)

    3888

    Internal

    X in server.N =host:X:Y
    

    Peer

    Server (with CDH 5 and Cloudera Manager 5)

    3181

    Internal

    X in server.N =host:X:Y
    

    Peer

    Server (with CDH 5 and Cloudera Manager 5)

    4181

    Internal

    X in server.N =host:X:Y
    

    Peer

    ZooKeeper JMX port

    9010

    Internal

    ZooKeeper will also use another randomly selected port for RMI. To allow Cloudera Manager to monitor ZooKeeper, you must do oneof the following:
    • Open up all ports when the connection originates from the Cloudera Manager Server
    • Do the following:
      1. Open a non-ephemeral port (such as 9011) in the firewall.
      2. Install Oracle Java 7u4 JDK or higher.
      3. Add the port configuration to the advanced configuration snippet, for example:
        -Dcom.sun.management.jmxremote.rmi.port=9011
        
      4. Restart ZooKeeper.
  • 相关阅读:
    strongswan--HA
    RFC8221 -- 密码算法实现要求和使用指南
    strongswan--ikev2软件架构
    strongswan--ike sa状态机
    strongswan--ipsec.conf配置文件分析
    strongswan--linux内核ipsec policy类型
    strongswan--配置Charon-systemd问题解决
    strongswan——IKE建立过程
    GCC入门
    MPLS-TP OAM各个层次
  • 原文地址:https://www.cnblogs.com/songyuejie/p/13439292.html
Copyright © 2011-2022 走看看