Security in distributed Computing

Distributed computing is an awesome approach to distribute workload of huge tasks and easily outsource them, if needed. It makes computing tasks scalable and cheap, as cloud computing is involved. Computing time can be rent, which is mostly cheaper compared to buying the necessary hardware. However, outsourcing into foreign networks comes with the advantages and drawbacks of public infrastructure. Public networks cannot be trusted, therefore, traffic should be encrypted and connections should be authorized, which sounds easier than it is when using Python frameworks such as Dispy, Celery or Twisted.

Although implemented in the core of the frameworks I used, security is optional, sometimes flawed (e.g asynchronous cryptography in Dispy which uses the same private as well as public key on client as well as on server side, otherwise it will not work) and often with lack of good documentation.

The main priority of official tutorials is to make it work, period – so developers test the shown code and use it, without bothering about further security steps. Tutorials showing working code with all needed encryption and authorization steps are rare. Often framework developers are showing the needed parts separated, but not the complete setup. So developers have to spend a lot of time puzzling all needed steps together. Security should be implemented by default, which unfortunately is not often the case in official framework tutorials. It was therefore not easy to find documented literature to implement the frameworks in a secure way. Sometimes, the frameworks did not even offer complete secure solutions. Let us have look at a framework called Twisted in combination with a JSON-RPC server and how to secure it. There is still room for improvement, but I hope this blog-post will help developers hardening and securing their software a little bit more.

The example code can be found here.

Twisted based RPC

Twisted is a Python framework for event-driven networking to implement asynchronous network programs. It is used in a lot of common projects like Tor2web, TwitchTV or Scrapy. I will show how to secure Twisted with an RPC (Remote Procedure Call) Server. XML-RPC is already implemented in Twisted, but we will not use XML in our example. XML is a very powerful data transport container format, which has a lot of functions won’t be needed for a simple RPC server. We will use JSON instead to reduce the possible attack surface. There are a few JSON-RPC implementations based on Twisted out there, but most of them are not Python 3 compatible. We will use the txJSON-RPC library, which implements the JSON-RPC protocol and is also compatible with Python 2 as well as Python 3. It supports communication through HTTP and TCP (Netstring). For encryption we are using TLS on the transport layer and client certificates for authentication. The way we use it can be adopted to most other Twisted based applications as well.

The most common authentication mechanism found seems to be using Twisted with password based authentication protocols like basic authentication or digest authentication (defined in RFC 2617). There are a few issues with basic authentication as its security highly depends on password complexity and an encrypted connection. The password is sent in clear text through the connection. However, it is not hardened against brute-force attacks.

Digest authentication does not send the password in clear text. As it is also password based, its security highly depends on password complexity, as well. For client – server connections there is often no need to use password based authentication at all. Instead authentication through certificates can be used to have highly secured authentications.

The following tutorial uses certificate based authentication combined with TLS encrypted connections. Therefore, we have to generate certificate authorities as well as key pairs. The keys will then be signed with the generated certificate authorities as shown below:

First we have to create our own certificate authority for our clients with an aes256 encrypted key:

openssl genrsa -aes256 -out clientCA.key 4096

Then, we generate CA’s certificate (you can change the days parameter to another value, if you need the CA to be valid for another time frame):

openssl req -x509 -new -nodes -key clientCA.key -sha512 -days 712 -out clientCA.crt

Now we create a client key. In order to avoid typing the password each time you run the server, you can leave out the “-aes256” paramter. However, keep in mind that the keyfile will not be encrypted then:

openssl genrsa -aes256 -out client.key 4096

Create a signing request:

openssl req -new -key client.key -out client.csr

And sign the request with our new CA:

openssl x509 -req -in client.csr -CA clientCA.crt -CAkey clientCA.key -CAcreateserial -out client.crt -days 356 -sha512

We have to create a signed key pair for our server with another CA. You should not use clients’ CA for it otherwise clients could use their certificate as trusted server certificate against other clients. Therefore we build another CA (leave out the -aes256 part if you won’t encrypt your keyfile):

openssl genrsa -aes256 -out serverCA.key 4096

And then generate CA’s certificate:

openssl req -x509 -new -nodes -key serverCA.key -sha512 -days 128 -out serverCA.crt

Generate new private key:

openssl genrsa -aes256 -out server.key 4096

Create a signing request:

openssl req -new -key server.key -out server.csr

And sign the request:

openssl x509 -req -in server.csr -CA serverCA.crt -CAkey serverCA.key -CAcreateserial -out server.crt -days 500 -sha512

OK, now we have built our own CA for client certificates as well as for server certificates, also we created signed keys for a server and a client. We can add additional clients the same way, if needed.

To ensure that no one steals the keys, you should change the rights of the key-files to be read-only for its owners on the final system with:

chmod 400 <NAME>.key

where <NAME> stands for server, client, clientCA and serverCA.

In case you want to harden the solution, I would recommend to use a dedicated system user to execute the scripts from client as well as from server with:

sudo useradd -r <USERNAME>

Additionally, you should change the ownership of the files to match users the software will be run as:

chown <USERNAME>:<GROUPNAME> <NAME>.key

We now have generated all keys. So let’s have a look at how to build a JSON-RPC server and its client.

Source code for the server and client can be found here. You still have to change the paths to your certificates, keys and CA’s to match with your system.

In RPC_Client.py you have to change the variable value of “privKey” at line 59 to match with the path of your client’s private key, called “client.key”. Then at line 61, change the path value of “certificate” to match with your client’s certificate called “client.crt” and finally, at line 63, change the path value of “accepted_ca” to match with servers certificate authority called “serverCA.crt”.

In RPC_Server.py we have to do nearly the same. Change the variable value of “privKey” at line 34 to match with path of your server’s private key, called “server.key”.

Then at line 36, change the path value of “certificate” to match with your client’s certificate called “server.crt” and finally, at line 39 change the value of “accepted_ca” to match with client’s certificate authority called “clientCA.crt”.

For using the server with Python3 you have to install txJSON-RPC which will install Twisted as well:

sudo python3 -m pip install --upgrade txJSON-RPC

For using the server with Python2 instead, you can use:

sudo python2 -m pip install --upgrade txJSON-RPC

You can start now the server with “python3 RPC_Server.py” and start the client with “python3 RPC_Client.py” to test the JSON-RPC server and client. You can use “sudo -u <USERNAME> ” as prefix to execute the scripts as a different user.  The Code is fully Python2  as well as Python3 compatible.

How does it work?

The server is made out of RPC_Server.py and RPC_functions.py. The server itself is set up in RPC_Server.py and offers functions defined in RPC_functions.py to JSON-RPC clients.

If further functions are needed, they can be defined in there. Clients only have to know their names and what arguments have to be passed. You can see how JSON-RPC works on the official PyPi page.

At first RPC_Server.py checks in line 69, if it is run as root and if so, it closes the Python script because called RPC functions would be executed as root otherwise.

At line 32 to 41 it gathers all needed information for connection and authentication. The “port” variable can be used to define a port, keep in mind, that ports below 1024 cannot be used as normal user. “sslMethod” defines the encryption method to be used. Twisted uses OpenSSL as back-end, so you can define each method defined in OpenSSL.SSL.

Here we are using the most recent TLS (as of today 2017/04/01) to encrypt our connections, i.e. TLSv1_2_METHOD. In line 44 we define our JSON-RPC Server as resource, which is imported from RPC_functions.py.

To encrypt our connection we need an sslContext, defined in line 47. It gets the previously defined private key and its certificate as well as the defined sslMethod.

The part to add the certificate based authentication is in line 49 to 55. Here we define, that clients should be authenticated and the connection will fail if the client cannot offer a valid certificate. We also define our own verifyCallback function (defined in line 13 to 28) as callback for certificate checks. We use it to get further information if a certificate check fails. We add our generated CA certificate in line 55 to only trust and accept certificates which are signed from our clientCA. We add the JSON-RPC resource and the sslContext to use for our Twisted reactor in line 59 and finally start our server with reactor.run().

Our client is defined in RPC_Client.py it verifies that the script is not executed as root as well. At line 170 to 173 it configures JSON-RPC’s server IP and its port. “debugmode” is set as a flag if the Twisted logging infrastructure should be used. Line 179 creates a Proxy which will act as connection for accessing the JSON-RPC server. It uses its own SSL context factory, defined in line 54 to 80. Within that factory we define our private Key, certificate and CA’s certificate as already explained. We also define the sslMethod to use SSL.TLSv1_2_METHOD. Similar to the server, we define that the server should only be accepted with a valid Certificate signed by serverCA (connections will fail if it’s not) and give our own verifyCallback (line 20 to 35) to use as callback function for certificate checks. Additionally, we add a password callback function “password_cb” defined in line 40 to 48. This function will be called from OpenSSL to get a passphrase if client’s keyfile is encrypted.

The function asks the user to type in the password. The callback is defined so the password is asked only once instead for each JSON-RPC call. It saves the password at run time in keypw defined in line 16. As an alternative, a keyfile without a pass-phrase can be used.

Back to the main function, in line 188 and 194 are the function calls for the JSON-RPC server. Therefore a callRemote function is defined in line 108 to 146. Its arguments are:

  • The proxy connection
  • The name of the function defined in RPC_functions.py without leading “jsonrpc_”
  • The callback function if everything is calculated well (printValue from line 150 to 151 for instance)
  • A callback function if something went wrong (printError from line 162 to 163).
    “callRemote” is a helper function and it’s main part is to call the “timeout” function (defined in line 85 to 102) which then calls the real JSON-RPC call and therefore makes sure that the call does not take longer than the time in seconds defined in timeout_duration. If it does, the function will be closed.

It is not the simplest way to do the JSON-RPC Calls, which in their simplest conduction seem like this:

d = proxy.callRemote('echo', 'test')
d.addCallbacks(printValue, printError)

But the callRemote function and their timeout are very handy during bigger projects with Twisted based JSON-RPC, because new JSON-RPC calls only need one line and getting closed if they hang up. Finally, in line 200 we start the reactor loop for our client.

Conclusion

When compared to solutions with basic or digest authentication, the implementation of certificate based authentication is less complex and it offers a higher degree of security. So I would recommend using certificate based authentication together with TLS encryption.

It’s a shame that official documentation does not show how to build secure code out of the box. It is only shown like a site note of separated TLS documentation, that you could check server’s and client’s identity with certificates. Developers ask often for a secure solution, but they have to invest a lot of time in researching to get an accurate solution. I hope I was able to help you getting started with Twisted and using it in a secure way.

 

Feel free to share the newsShare on Facebook
Facebook
Share on Google+
Google+
Tweet about this on Twitter
Twitter
Share on LinkedIn
Linkedin

Published by

Niklas Abel

IT Security Consultant