How to get all the metrics via Metrics Servlet in JSON format (as in Web UI) from Apache Spark

Refresh

December 2018

Views

1.5k time

2

The apache Spark web UI gives a ton of useful information about the workers, driver, storage etc. But the Metrics Servlet gives only a few information. For eg, I get the following from /metrics/json :

{
  "version":"3.0.0",
  "gauges":{
    "local-1431683138663.<driver>.BlockManager.disk.diskSpaceUsed_MB":{
      "value":0
    },
    "local-1431683138663.<driver>.BlockManager.memory.maxMem_MB":{
      "value":3179
    },
    "local-1431683138663.<driver>.BlockManager.memory.memUsed_MB":{
      "value":169
    },
    "local-1431683138663.<driver>.BlockManager.memory.remainingMem_MB":{
      "value":3009
    },
    "local-1431683138663.<driver>.DAGScheduler.job.activeJobs":{
      "value":1
    },
    "local-1431683138663.<driver>.DAGScheduler.job.allJobs":{
      "value":1
    },
    "local-1431683138663.<driver>.DAGScheduler.stage.failedStages":{
      "value":0
    },
    "local-1431683138663.<driver>.DAGScheduler.stage.runningStages":{
      "value":1
    },
    "local-1431683138663.<driver>.DAGScheduler.stage.waitingStages":{
      "value":0
    }
  },
  "counters":{
  },
  "histograms":{
  },
  "meters":{
  },
  "timers":{
  }
}

How do I get other metrics like storage etc.?

1 answers

0

Все данные теперь доступны в Спарк мониторинга REST API, начиная с версии 1.4 и выше. Благодаря Apache Spark!

Ссылка: http://spark.apache.org/docs/latest/monitoring.html#rest-api