General Questions

Expand all | Collapse all

Webconsole is not able to detect nodes of gridgrain cluster

  • 1.  Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-14-2019 07:24 PM

    Gridgrain webconsole is not able to detect running active nodes, Gridgrain frontend webconsole, Gridgrain backend webconsole & Gridgrain webagent is configured as per documentation --> https://apacheignite-tools.readme.io/docs/kubernetes-installation

    But in logs agent is failing with below errors;

    [2019-11-13 14:11:51,386][INFO ][main][AgentLauncher] Starting Apache Ignite Web Console Agent... [2019-11-13 14:11:51,997][WARN ][main][AgentLauncher] Failed to find properties file: default.properties

    Web Console Agent configuration : User's security tokens : ********************************69a3 URI to Ignite node REST server : http://ignite-dataservice.ocpapps.cn-sdp-labocp.local:8080 URI to Ignite Console server : http://gridgain-console.ocpapps.cn-sdp-labocp.local Path to properties file : default.properties Path to JDBC drivers folder : ./jdbc-drivers Demo mode : enabled

    [2019-11-13 14:11:53,611][INFO ][main][WebSocketRouter] Starting Web Console Agent... [2019-11-13 14:11:53,694][INFO ][Connect thread][WebSocketRouter] Connecting to server: ws://gridgain-console.ocpapps.cn-sdp-labocp.local [2019-11-13 14:11:56,493][ERROR][HttpClient@3110a6ad-15][WebSocketRouter]** Failed to establish websocket connection with server: ws://gridgain-console.ocpapps.cn-sdp-labocp.local**

    Logs from Webconsole Backend:

    WARNING: Unknown module: jdk.internal.jvmstat specified to --add-exports WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/opt/gridgain-web-console-server/libs/spring-core-4.3.23.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [2019-11-13T11:15:19,335][INFO ][exchange-worker-#28][msg] Components activation performed in 1546 ms. [2019-11-13T11:18:05,168][INFO ][main][MigrationFromMongo] MongoDB URL was not specified. Migration disabled. [2019-11-13T11:18:37,519][INFO ][main][Application] Full logs is available in /opt/gridgain-web-console-server/work/log [2019-11-13T11:18:37,540][INFO ][main][Application] GridGain Web Console started on TCP port 3000 in 426.89 seconds

    After moving gridgrain-restapi libraries (depended jars files) from /optional directory of gridgrain to /libs directory , Webagent is able to establish connection with frontend via REST-API HTTP connection in response failed with error 403 which means -"The HTTP 403 Forbidden client error status response code indicates that the server understood the request but refuses to authorize it"

    Logs from frontendwebconsole container:-

    gridgain-console.ocpapps.cn-sdp-labocp.local 10.130.2.1 - - [14/Nov/2019:13:24:02 +0000] "GET /browsers?demoMode=false HTTP/1.1" 403 339 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "-" gridgain-console.ocpapps.cn-sdp-labocp.local 10.131.2.1 - - [14/Nov/2019:13:24:05 +0000] "GET /browsers?demoMode=false HTTP/1.1" 403 339 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "-" gridgain-console.ocpapps.cn-sdp-labocp.local 10.131.2.1 - - [14/Nov/2019:13:24:08 +0000] "GET /browsers?demoMode=false HTTP/1.1" 403 339 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "-" gridgain-console.ocpapps.cn-sdp-labocp.local 10.131.0.1 - - [14/Nov/2019:13:24:09 +0000] "GET /agents HTTP/1.1" 403 354 "-" "Jetty/9.4.18.v20190429" "-" gridgain-console.ocpapps.cn-sdp-labocp.local 10.131.2.1 - - [14/Nov/2019:13:24:09 +0000] "GET /browsers?demoMode=false HTTP/1.1" 403 339 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/5

    I believe this authorisation is done based on token passsed via webagent to connect to frontend webconsole, which we have passsed in configuration properties of webagent



    ------------------------------
    ankit gupta
    BigData consultant

    ------------------------------


  • 2.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-15-2019 01:17 AM
    Hello!

    I guess this means that the token is incorrect. Maybe you set the token and then re-deployed the Web Console so that it forgot all previously assigned tokens.

    I also advise you to download our newest Web Console, which is Java based and not depends on Mongo: Download GridGain Software - GridGain Systems (scroll a bit to reveal Web Console downloads).

    Regards,

    ------------------------------
    Ilya Kasnacheev
    Community Support Specialist
    GridGain
    ------------------------------



  • 3.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-15-2019 01:43 AM

    Hi IIya ,

    Thanks for your response. We are running Gridgrain 2 node cluster on Openshift Container based platform (Kubernetes enviornment). we are able to use native feature of gridgain like cache & persistence storage of table.

    We are able to run webconsole  frontend & backend also webagent  as a POD. But webagent is failing to establish websocket connection with server (URI to Ignite Console server - webconsole frontend).

    i tried to re-deploy the webconsole services of gridgrain and passed the same security token which i got from profile tab after successful registration. But still the same error.

    [14/Nov/2019:13:24:09 +0000] "GET /agents HTTP/1.1" 403 354 "-" "Jetty/9.4.18.v20190429" "-" gridgain-console.ocpapps.cn-sdp-labocp.local 10.131.2.1 - - [14/Nov/2019:13:24:09 +0000] "GET /browsers?demoMode=false HTTP/1.1" 403 339 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/5

    Gridgain image details taken from https://hub.docker.com/r/gridgain/enterprise

     -- Gridgain/enterprise:latest    version -( GridGain 8.7.6)

    Gridgain webconsole:-

     Agent gridgain-web-agent: v2019.10.01

    backend web-console-backend v2019.10.00

    frontend web-console-frontend:2019.10.00



    ------------------------------
    ankit gupta
    BigData consultant
    Mr.
    09665929944
    ------------------------------



  • 4.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-15-2019 03:33 AM
    Hello!

    Could you please try to use the following instruction for your deployment: 
    https://www.gridgain.com/docs/web-console/latest/deploying-web-console-openshift

    Please take a special attention to the "Create a ConfigMap for Web Console Frontend" section, it contains an additional configuration for /agents location if Web Agent is deployed on OpenShift Cluster:
          location /agents {
            proxy_pass http://backend-endpoint;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            # Use this setting if you plan to run Web Agent in a container
            proxy_set_header Origin http://backend-endpoint;
            # Use this setting if you plan to run Web Agent as a standalone application
            # proxy_set_header Origin $scheme://$http_host;
          }​


    Also I recommend to use the latest version of Web Console (2019.11.00) and Web Agent (2019.11.00):
    gridgain/gridgain-web-console-frontend:2019.11.00
    gridgain/gridgain-web-console-backend:2019.11.00
    gridgain/gridgain-web-agent:2019.11.00

    Thank you!
    Ilya.



    ------------------------------
    Ilya Murchenko
    Cloud Engineer
    GridGain
    ------------------------------



  • 5.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-17-2019 11:37 PM

    Hi IIya, 
    I followed your instruction & deployed & re-deployed Gridgain webconsole & Webagent with latest version of image (2019.11.00).

    But still getting same error as i was getting earlier "[WebSocketRouter]** Failed to establish websocket connection with server".

    Can you review the configuration/service files which i have used to deployed the Gridgrain services.



    ------------------------------
    ankit gupta
    BigData consultant
    Mr.
    09665929944
    ------------------------------



  • 6.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-18-2019 12:41 AM
    Hello!

    As I can see in your configuration files Web Agent is deployed to a different project (namespace) than the Web Console, am I right?

    In this case you need to use the fully qualified domain name(s) (FQDN) when setting the NODE_URI and SERVER_URI paths as described in the instruction: 
    Note: If Web Console and/or the GridGain or Ignite cluster are deployed to a different project (namespace) than the Web Agent, you will need to use the fully qualified domain name(s) (FQDN) when setting the NODE_URI and SERVER_URI paths. A fully qualified domain name typically follows the pattern of <internal_service_name>.<namespace>.svc.cluster.local. Using the above examples, the FQDN would look like web-console-frontend.gridgain-web-console.svc.cluster.local. For more information on networking configuration, please see the OpenShift Networking Documentation.

    Could you please try to set FQDN for Frontend service as a SERVER_URI i.e.: 
    http://web-console-frontend.dataservice.svc.cluster.local

    My setup:
    1. Web Console Frontend and Backend PODs are deployed to "dataservice" project.
    2. Web Agent POD is deployed to "gridgain-web-agent" project.
    3. SERVER_URI parameter is set to "http://web-console-frontend.dataservice.svc.cluster.local"

    Logs:
    [2019-11-18 08:32:09,926][INFO ][main][AgentLauncher] Starting Apache Ignite Web Console Agent...
    [2019-11-18 08:32:10,323][WARN ][main][AgentLauncher] Failed to find properties file: default.properties
    Web Console Agent configuration :
    User's security tokens          : ********************************ace5
    URI to Ignite node REST server  : http://172.25.4.183:8080
    URI to Ignite Console server    : http://web-console-frontend.dataservice.svc.cluster.local
    Path to properties file         : default.properties
    Path to JDBC drivers folder     : /opt/ignite/drivers
    Demo mode                       : enabled
    [2019-11-18 08:32:11,756][INFO ][main][WebSocketRouter] Starting Web Console Agent...
    [2019-11-18 08:32:11,780][INFO ][Connect thread][WebSocketRouter] Connecting to server: ws://web-console-frontend.dataservice.svc.cluster.local
    [2019-11-18 08:32:14,180][INFO ][HttpClient@3110a6ad-11][WebSocketRouter] Successfully completes handshake with server


    Thank you!
    Ilya.


    ------------------------------
    Ilya Murchenko
    Cloud Engineer
    GridGain
    ------------------------------



  • 7.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-18-2019 03:25 AM

    Hi IIya,
    Webagent & Webconsole(Backend & Frontend) both are running inside same namespace project 'dataservice' . I see that in webagen deployment config file namespace was not mentioned, So i re-deployed the webagent  configuration with same namespace 'dataservice'. But still getting same error. 

    =================================
    Webagent logs:-

    [2019-11-18 11:13:14,636][INFO ][main][AgentLauncher] Starting Apache Ignite Web Console Agent...
    [2019-11-18 11:13:14,995][WARN ][main][AgentLauncher] Failed to find properties file: default.properties
    Web Console Agent configuration :
    User's security tokens : ********************************c023
    URI to Ignite node REST server : http://ignite-dataservice.ocpapps.cn-sdp-labocp.local:8080
    URI to Ignite Console server : http://web-console-frontend-dataservice.ocpapps.cn-sdp-labocp.local
    Path to properties file : default.properties
    Path to JDBC drivers folder : /opt/ignite/drivers
    Demo mode : enabled
    [2019-11-18 11:13:15,605][INFO ][main][WebSocketRouter] Starting Web Console Agent...
    [2019-11-18 11:13:15,620][INFO ][Connect thread][WebSocketRouter] Connecting to server: ws://web-console-frontend-dataservice.ocpapps.cn-sdp-labocp.local
    [2019-11-18 11:13:16,983][ERROR][HttpClient@2550ba19-15][WebSocketRouter] Failed to establish websocket connection with server: ws://web-console-frontend-dataservice.ocpapps.cn-sdp-labocp.local

    ==========================

    Is there any specific configuration / setup need to be done to bring webagent in running status apart from steps which are mentioned in "https://www.gridgain.com/docs/web-console/latest/deploying-web-console-openshift?_ga=2.255413192.1587515088.1573921276-749559361.1567487455"

    Web agent, webconsole frontend, webconsole backend pod is running in same namespace, & i can access the frontend console using expose routes URL " http://web-console-frontend-dataservice.ocpapps.cn-sdp-labocp.local"



    ------------------------------
    ankit gupta
    BigData consultant
    Fujitsu
    ------------------------------



  • 8.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 11-18-2019 03:47 AM
    Hello,

    Could you please try to set a FQDN of Frontend service (i.e. http://web-console-frontend.dataservice.ocpapps.cn-sdp-labocp.local" (<internal_service_name>.<namespace>.svc.cluster.local)depends on your OpehShift's configuration) instead of exposed route URL? 

    Thank you!
    Ilya.

    ------------------------------
    Ilya Murchenko
    Cloud Engineer
    GridGain
    ------------------------------



  • 9.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 12-02-2019 08:04 PM

    Now able gridgain webconsole is up, i can see the cluster (nodes) in console and can run sql queries on top native Gridgain database.

    issue was incorrect URI to ignite console server in webagent configuration.

    But when I'm trying to import schema from webconsole using "Import From Database" option. Nothing is coming up on screen to connect to database(ignite auto schema import utility window) , when i checked the logs of webagent i see below error;

    =================================================================================

    [2019-12-02 04:51:47,184][ERROR][HttpClient@6650eb6d-11][WebSocketRouter] Failed to process message: WebSocketRequest [reqId=339a8606-b4d7-4d69-9489-b0b38ac59151, evtType=schemaImport:drivers, payload={}][2019-12-02 04:51:47,184][ERROR][HttpClient@6650eb6d-11][WebSocketRouter] Failed to process message: WebSocketRequest [reqId=339a8606-b4d7-4d69-9489-b0b38ac59151, evtType=schemaImport:drivers, payload={}]java.lang.IllegalStateException: JDBC drivers folder not specified at org.apache.ignite.console.agent.handlers.DatabaseHandler.collectJdbcDrivers(DatabaseHandler.java:148) at org.apache.ignite.console.agent.handlers.WebSocketRouter.onMessage0(WebSocketRouter.java:376) at org.apache.ignite.console.agent.handlers.WebSocketRouter.lambda$onMessage$1(WebSocketRouter.java:356) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Thread.java:748)

    =================================================================================

    So jdbc driver manually need to copied at below location ? to make this import from database work.

    /opt/ignite/ignite-web-console-agent/jdbc-drivers $ pwd
    /opt/ignite/ignite-web-console-agent/jdbc-drivers

    is there any work around to manually import database configuration in gridgain webconsole?



    ------------------------------
    ankit gupta
    BigData consultant
    Fujitsu
    ------------------------------



  • 10.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 12-04-2019 01:10 PM
    Hello!

    You need to put JDBC driver for your database into jdbc-drivers/ subdirectory in your Web Agent directory, and restart Web Agent.

    Please see https://apacheignite-tools.readme.io/docs/getting-started#section-configuration

    Regards,

    ------------------------------
    Ilya Kasnacheev
    Community Support Specialist
    GridGain
    ------------------------------



  • 11.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 12-04-2019 07:26 PM

    Hi llya... Thanks for your response.

    Official gridgain docker image of webagent comes with root user privileges .. so i cannot copy anything on default jdbc path.. (Actually it does't exist, so we cannot create as container is running on root access).

    as per documentation of gridgain- jdbc-driver path is Path to JDBC drivers folder : /opt/ignite/ignite-web-console-agent/jdbc-drivers , which can be configured in webagent deployment yaml configuration file, to override the default jdbc path.

    When i modified the base image to include required JDBC jar files for postgressql to copied at path /opt/ignite/ignite-web-console-agent/jdbc-drivers   it worked like charm.

    I believe Official webagent gridgain image need to modify to build with non-root user access, & default jdbc driver path need to change.



    ------------------------------
    ankit gupta
    BigData consultant
    Fujitsu
    ------------------------------



  • 12.  RE: Webconsole is not able to detect nodes of gridgrain cluster

    Posted 12-04-2019 11:53 PM
    Hello!

    Thank you for this input, we will improve our documentation.

    Regards,

    ------------------------------
    Ilya Kasnacheev
    Community Support Specialist
    GridGain
    ------------------------------



Would you attend a July Meetup?


Announcements