Thursday 6 December 2018

Spark App error: BindException: Can't assign requested address: Service 'sparkDriver'

Problem:


When you run your Spark Application or Spark Application unit tests, we may get the following error:


java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.


Solution:

To solve this problem, please do the following steps:
  • Know your host name
         $ hostname
            my_hostname
  • Execute the following command 
         $ sudo hostname -s 127.0.0.1
  • Update "/etc/hosts" file with the following new entry
          $vi /etc/hosts
          127.0.0.1      my_hostname
  • Finally export the following environment variable
          export SPARK_LOCAL_IP="127.0.0.1" 

Thats it.

It solved my problem in running my Spark application unit tests.

No comments:

Post a Comment