Primordial Echo

Increasing security in Erlang and Elixir SSL applications

2015-09-18T00:00:00-00:00

With the recent outbreak of major security flaws in the OpenSSL library (CRIME, FREAK, POODLE, logjam, BEAST etc), the need to properly configure our application’s SSL layer (whichever purpose that might be serving) is bigger than ever.

SSL in Erlang is a very particular kind of beast. An SSL socket will behave almost exactly the same way as a gen_tcp socket. Almost. But we’re not here to talk about that.

Conventional operation of an ssl-based application in Erlang is relatively simple. You can start different SSL sockets, each with its own set of certificates, ciphers and a battery of other SSL-specific options. Naturally, these options can only be set at either the listen leven (ssl:listen/2) or the accept level (ssl:ssl_accept/4), are usually exposed by libraries that use the ssl application through an initialization option and essentially control, among other things, the security guarantees that this specicic library/socket will be providing.

Since the part of code where you define this configuration is different for every application that uses SSL, I’ll present the options in a generic manner and then provide 2 application-specific examples (Cowboy and RabbitMQ). In general, SSL needs can be stricter or looser depending on what kind clients the application expects (users or machines). The options below will guarantee you an SSL setup that’s balanced between security and compatibility.

Generic configuration

{versions, ['tlsv1.2', 'tlsv1.1', 'tlsv1']}

The versions tuple defines the SSL/TLS versions that the server supports. Since the POODLE attack, SSLv3 has mostly been deprecated in favor of TLSv1 and newer revisions. By strictly defining TLS in your configuration, you completely avoid SSLv3 exposure.

{dhfile, "dh-params.pem"}

The Ephemeral Diffie-Helman key exchange is a very effective way of ensuring Forward Secrecy by exchanging a set of keys that never hit the wire. Since the DH key is effectively signed by the private key, it needs to be at least as strong as the private key. In addition, the default DH groups that most of the OpenSSL installations have are only a handful (since they are distributed with the OpenSSL package that has been built for the operating system it’s running on) and hence predictable (not to mention, 1024 bits only).

In order to escape this situation, first we need to generate a fresh, strong DH group, store it in a file and then use the option above, to force our SSL application to use the new DH group. Fortunately, OpenSSL provides us with a tool to do that. Simply run

openssl dhparam -out dh-params.pem 2048

to generate a new DH group file of 2048 bits and store it in dh-params.pem.

{secure_renegotiate, true}

SSL parameter renegotiation is a feature that allows a client and a server to renegotiate the parameters of the SSL connection on the fly. RFC 5746 defines a more secure way of doing this. By enabling secure renegotiation, you drop support for the insecure renegotiation, prone to MitM attacks.

{reuse_sessions, true}

A performance optimization setting, it allows clients to reuse pre-existing sessions, instead of initializing new ones. Read more about it here.

{honor_cipher_order, true}

An important security setting, it forces the cipher to be set based on the server-specified order instead of the client-specified order, hence enforcing the (usually more properly configured) security ordering of the server administrator.

{ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
        "ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384", "ECDHE-ECDSA-DES-CBC3-SHA",
        "ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384","ECDH-ECDSA-AES256-SHA384",
        "ECDH-RSA-AES256-SHA384","DHE-DSS-AES256-GCM-SHA384","DHE-DSS-AES256-SHA256",
        "AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
        "ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256","ECDHE-RSA-AES128-SHA256",
        "ECDH-ECDSA-AES128-GCM-SHA256","ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
        "ECDH-RSA-AES128-SHA256","DHE-DSS-AES128-GCM-SHA256","DHE-DSS-AES128-SHA256",
        "AES128-GCM-SHA256","AES128-SHA256","ECDHE-ECDSA-AES256-SHA",
        "ECDHE-RSA-AES256-SHA","DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA",
        "ECDH-RSA-AES256-SHA","AES256-SHA","ECDHE-ECDSA-AES128-SHA",
        "ECDHE-RSA-AES128-SHA","DHE-DSS-AES128-SHA","ECDH-ECDSA-AES128-SHA",
        "ECDH-RSA-AES128-SHA","AES128-SHA"]}

This is the single most important configuration option of an Erlang SSL application. Ciphers (and their ordering) define the way the client and server encrypt information over the wire, from the initial Diffie-Helman key exchange, the session key encryption algorithm and the message digest algorithm. Selecting a good cipher suite is critical for the application’s data security, confidentiality and performance. The cipher list above offers:

A good balance between compatibility with older browsers. It can get stricter for Machine-To-Machine scenarios.
Perfect Forward Secrecy.
No old/insecure encryption and HMAC algorithms

Most of it was copied from Mozilla’s Server Side TLS article and then modified and it has been tested in production with Erlang 18.1 but should work on the 17.x series as well, given a relatively modern OpenSSL installation (1.0.2d or newer).

All of the above options result in getting an A rating from SSLLabs, which is a good compromise between security and compatibility.

Cowboy configuration

In Cowboy, you can enable these options as part of the https listener initialization routine:

{ok, CowboyPid} = cowboy:start_https(
    cowboy_https_receiver,
    100,
    [
        {port, 443},
        {cacertfile,"/path/to/testca/cacert.pem"},
        {certfile,"/path/to/server/cert.pem"},
        {keyfile,"/path/to/server/key.pem"},
        {versions, ['tlsv1.2', 'tlsv1.1', 'tlsv1']},
        {dhfile, "/path/to/testca/dh-params.pem"},
        {ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
            "ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384", "ECDHE-ECDSA-DES-CBC3-SHA",
            "ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384","ECDH-ECDSA-AES256-SHA384",
            "ECDH-RSA-AES256-SHA384","DHE-DSS-AES256-GCM-SHA384","DHE-DSS-AES256-SHA256",
            "AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
            "ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256","ECDHE-RSA-AES128-SHA256",
            "ECDH-ECDSA-AES128-GCM-SHA256","ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
            "ECDH-RSA-AES128-SHA256","DHE-DSS-AES128-GCM-SHA256","DHE-DSS-AES128-SHA256",
            "AES128-GCM-SHA256","AES128-SHA256","ECDHE-ECDSA-AES256-SHA",
            "ECDHE-RSA-AES256-SHA","DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA",
            "ECDH-RSA-AES256-SHA","AES256-SHA","ECDHE-ECDSA-AES128-SHA",
            "ECDHE-RSA-AES128-SHA","DHE-DSS-AES128-SHA","ECDH-ECDSA-AES128-SHA",
            "ECDH-RSA-AES128-SHA","AES128-SHA"]},
        {secure_renegotiate, true},
        {reuse_sessions, true},
        {honor_cipher_order, true},
        {max_connections, infinity}
    ],
    [{max_keepalive, 1024}, {env, [{dispatch, Dispatch}]}]).

RabbitMQ configuration

In RabitMQ, you can enable these options in the rabbitmq.config file:

[
  {rabbit, [
     {ssl_listeners, [5671]},
     {ssl_options, [{cacertfile,"/path/to/testca/cacert.pem"},
                    {certfile,"/path/to/server/cert.pem"},
                    {keyfile,"/path/to/server/key.pem"},
                    {verify,verify_peer},
                    {fail_if_no_peer_cert,false},
                    {versions, ['tlsv1.2', 'tlsv1.1', 'tlsv1']},
                    {dhfile, "/path/to/testca/dh-params.pem"},
                    {ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
                        "ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384", "ECDHE-ECDSA-DES-CBC3-SHA",
                        "ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384","ECDH-ECDSA-AES256-SHA384",
                        "ECDH-RSA-AES256-SHA384","DHE-DSS-AES256-GCM-SHA384","DHE-DSS-AES256-SHA256",
                        "AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
                        "ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256","ECDHE-RSA-AES128-SHA256",
                        "ECDH-ECDSA-AES128-GCM-SHA256","ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
                        "ECDH-RSA-AES128-SHA256","DHE-DSS-AES128-GCM-SHA256","DHE-DSS-AES128-SHA256",
                        "AES128-GCM-SHA256","AES128-SHA256","ECDHE-ECDSA-AES256-SHA",
                        "ECDHE-RSA-AES256-SHA","DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA",
                        "ECDH-RSA-AES256-SHA","AES256-SHA","ECDHE-ECDSA-AES128-SHA",
                        "ECDHE-RSA-AES128-SHA","DHE-DSS-AES128-SHA","ECDH-ECDSA-AES128-SHA",
                        "ECDH-RSA-AES128-SHA","AES128-SHA"]},
                    {secure_renegotiate, true},
                    {reuse_sessions, true},
                    {honor_cipher_order, true},
                    {max_connections, infinity}]}
   ]}
].

Phoenix configuration

The popular Elixir web framework Phoenix uses Cowboy under the scenes. In order to configure it to use the above options, we simply have to specify them in the appropriate configuration file (such as config/prod.exs):

use Mix.Config

config :phoenix, SSLApp.Endpoint,
  https: [port: 443,
          host: 'sslapp.com',
          cacertfile: '/path/to/testca/cacert.pem',
          certfile: '/path/to/server/cert.pem',
          keyfile: '/path/to/server/key.pem',
          versions: [:'tlsv1.2', :'tlsv1.1', :'tlsv1'],
          dhfile: '/path/to/testca/dh-params.pem',
          ciphers: ['ECDHE-ECDSA-AES256-GCM-SHA384','ECDHE-RSA-AES256-GCM-SHA384',
                        'ECDHE-ECDSA-AES256-SHA384','ECDHE-RSA-AES256-SHA384', 'ECDHE-ECDSA-DES-CBC3-SHA',
                        'ECDH-ECDSA-AES256-GCM-SHA384','ECDH-RSA-AES256-GCM-SHA384','ECDH-ECDSA-AES256-SHA384',
                        'ECDH-RSA-AES256-SHA384','DHE-DSS-AES256-GCM-SHA384','DHE-DSS-AES256-SHA256',
                        'AES256-GCM-SHA384','AES256-SHA256','ECDHE-ECDSA-AES128-GCM-SHA256',
                        'ECDHE-RSA-AES128-GCM-SHA256','ECDHE-ECDSA-AES128-SHA256','ECDHE-RSA-AES128-SHA256',
                        'ECDH-ECDSA-AES128-GCM-SHA256','ECDH-RSA-AES128-GCM-SHA256','ECDH-ECDSA-AES128-SHA256',
                        'ECDH-RSA-AES128-SHA256','DHE-DSS-AES128-GCM-SHA256','DHE-DSS-AES128-SHA256',
                        'AES128-GCM-SHA256','AES128-SHA256','ECDHE-ECDSA-AES256-SHA',
                        'ECDHE-RSA-AES256-SHA','DHE-DSS-AES256-SHA','ECDH-ECDSA-AES256-SHA',
                        'ECDH-RSA-AES256-SHA','AES256-SHA','ECDHE-ECDSA-AES128-SHA',
                        'ECDHE-RSA-AES128-SHA','DHE-DSS-AES128-SHA','ECDH-ECDSA-AES128-SHA',
                        'ECDH-RSA-AES128-SHA','AES128-SHA'],
          secure_renegotiate: true,
          reuse_sessions: true,
          honor_cipher_order: true,
          max_connections: :infinity]

Increasing security in Erlang and Elixir SSL applications was originally published by Panagiotis "PJ" Papadomitsos at Primordial Echo on September 18, 2015.

Optimizing your Linux server for memory-based NoSQL databases (Part 1)

2012-07-04T00:00:00-00:00

Hey! This article was originally written for BugSense’s blog and was published on 2012-07-03. You can find the original post here.

So you have been listening to the hype of NoSQL databases for some time now and how they can make you web applications run much faster and be more adaptive and welcoming to horizontal scaling and you’d like to try it too and see how it plays out for you. What you should know though, is that apart from setting up and configuring your selected flavor of NoSQL, be it something less complex, like Memcache or Redis, to more enterprise schemes like Cassandra or HBase, you will need eventually to optimize the server(s) hosting it to make the most out of your investment.

Word of the wise: Please read this before even thinking about dumping your RDMBS in favor of a NoSQL backend. A NoSQL database is not a replacement for traditional relational databases, and it never will be.

Optimizing a Linux server is a humongous topic that touches multiple layers of the application stack and is not an exact science. In this article series we’ll mostly care about optimizing for the more lightweight bunch of NoSQL databases - the bunch that does not rely on a VM (such as Cassandra, HBase or Couch) but runs on native code (such as Memcache, Redis and Mongo) and we will begin by rebuilding the most essential component of any Linux server: the kernel.

Disclaimer: Proceed on your own risk. I assume you already know how to download, extract and build a vanilla kernel - if you don’t, the internet is swamped with such articles and GIF. Failing to properly build and install the kernel may render your server unbootable, erase your 9gag posts and force your country into asking financial aid by the IMF.

So, on a vanilla 3.x kernel (3.4.4 as of the day this article was written) the following options are your best friends for the aforementioned workloads:

CONFIG_SLUB

Chris Lameter’s kernel object caching system. Much more efficient in managing kernel memory allocations than the old SLAB, offers per-CPU slab queues and enhanced diagnostics via the slabinfo tool. It’s selected by default in recent kernels.

CONFIG_JUMP_LABEL

An in-kernel branching optimization that alters branching on the fly for specific cases. Makes the kernel faster. ‘nuff said :-)

CONFIG_NUMA and friends

Useful for recent manycore servers, enables NUMA awareness in the kernel, improves cache coherency and memory locality on supported hardware.

CONFIG_SPARSEMEM_VMEMMAP

A sparse memory optimization option for pfn_to_page and page_to_pfn functions.

CONFIG_TRANSPARENT_HUGEPAGE, CONFIG_COMPACTION, CONFIG_MIGRATION and CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS

Whew, that’s a lot of config options. These are probably some of the most important kernel options you can set for this kind of workload. What they essentially enable in-kernel is the ability to allocate larger memory pages than the 4KB default, speeding up memory allocation for memory-hungry processes. In addition to that, they also allow memory page compaction and migration to satisfy these huge page requests, further reducing memory fragmentation.

CONFIG_KSM

An important mechanism that actually deduplicates memory pages flagged with MADV_MERGEABLE, providing extreme memory savings.

CONFIG_ZRAM

Provides a memory-based block device. Data written in that block device will be compressed and stored only in-memory. Useful for temporary storage space (such us mounting under /tmp). As this feature is in the staging area, please build it as a module and proceed with caution.

CONFIG_ZCACHE, CONFIG_ZSMALLOC and CONFIG_CLEANCACHE

A memory page compression framework that transparently compresses clean and swap pages in-memory providing effortless performance improvements for memory-based workloads. CleanCache uses the ZCache framework as a transcendent memory provider to swap-in clean pages in tmem, further reducing I/O in high-memory environments. As this feature is in the staging area, please proceed with caution. It goes of course without saying that your setup should always be running on x86_64 at least. Additionally, a number of nice-to-have options that does not necessarily pertain to our workloads are:

CONFIG_TASK_IO_ACCOUNTING

Extremely important, allows you to monitor the server’s disk activity per process via awesome tools like iotop.

CONFIG_PERF_EVENTS

Kernel performance counter hooks to use with tools like perf. Critical for in-depth performance monitoring.

CONFIG_PROFILING

Performance profiling hooks used by tools such as OProfile. Equally important to PERF_EVENTS.

HAVE_BPF_JIT

Just in time rule compiler for pcap-based userland tools that use the Berkeley Packet Filter (such as tcpdump & friends). Should speed up complex rules considerably! Enable it via /proc/sys/net/core/bpf_jit_enable.

Select these options with your favorite config method, save, run make or run it through your favorite .deb/.rpm packager to pack it for massive deployment and prepare for glory!

I have to note here that there are many other promising technologies available that may aid in building more efficient infrastructure schemes for memory-based workloads (such as RAMSter that offers swap clustering) but they are still in need of further testing to even consider building a production kernel with them. I invite you to try however, and if you do, please let us know how it went!

Of course, simply recompiling the kernel is not sufficient enough to even say that we have finished optimizing, but it’s a start! In the next article we’ll deal with system & scheduler tuning and how it can help us make the most of our setup, be it virtualized or physical!

Stay tuned!

Optimizing your Linux server for memory-based NoSQL databases (Part 1) was originally published by Panagiotis "PJ" Papadomitsos at Primordial Echo on July 04, 2012.

Attaching local disks to a Citrix XenServer VM

2011-10-18T00:00:00-00:00

So this story goes like this: All my precious media files are kept inside two enterprise level hard disks (think of WD RE4 and such) buried inside a few external USB cases. However, after setting up an Ubuntu VM in PV mode in my Citrix XenServer testing system, I got the urge to attach the aforementioned hard disk to the VM I had just installed and share it over my local network through Samba (and later, iSCSI).

Disclaimer: As this story involves procedures related to tampering with hard disk drives, always keep an up-to-date backup of your data before trying anything! I will not be responsible for any data loss resulting from poor safety measures!

This would be an extremely easy thing to do if I wanted USB speeds, as XenServer supports attaching external USB storage to VMs out-of-the-box (for backup purposes). But USB external storage maxes out at ~ 30 MB/s and my hard disk has a read/write throughput of ~80 MB/s, so I popped the server case open, connected it through the actual SATA interface and started trying to figure a way to make XenServer believe that this was actually a USB disk and not a locally attached drive.

What we already know is that XenServer uses the well-praised Linux 2.6 kernel right under the Xen hypervisor which abstracts USB as well as SATA (and newer libata-based PATA) connected hard disks with the a SCSI disk device naming scheme of sdxy, where x is a letter assigned to the device in boot time (depending on the device-scanning order) and y stands for the legacy (not EFI) partition number(s) the drive contains. XenServer also hosts a full-blown Linux distribution (think Redhat), so this naming scheme (any much, much more) is read by the udev daemon at boot time and the daemon creates the appropriate device nodes in the /dev directory.

Now, after some digging, I found out that the Xen hypervisor (the XAPI backend to be precise) actually utilized the udev-created device nodes in the /dev tree to attach physically connected devices to the VMs.

This is done by simply creating a symlink of the device in question (i.e. sda, sdb, sdc etc.) inside the /dev/xapi/block directory! So what I did was to modify the server’s /etc/udev/rules.d/50-udev.rules and add these lines (sdb was the detected device name of my drive):

ACTION=="add", KERNEL=="sdb", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"
ACTION=="remove", KERNEL=="sdb", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"

Hey! You can repeat the above rules for more than one hard drive, replacing the relevant kernel device names.

The code above adds 2 actions to udev:

When the device is detected, add a symlink to /dev/xapi/block/(kernel name given to the device) and run a script (this was copied from the USB mass storage device detection rules)
When the device is removed (that is, if the disk is detached by hot-unplugging it-this is a whole procedure, don’t try this at home if you don’t do your research first), run the same script as in “add”

After rebooting the server, the hard disk was immediately recognized, symlinked to the /dev/xapi/block directory and was available through the xe command or XenCenter, waiting to be attached to a VM! As a nice extra, in Citrix XenServer, the USB-attached mass storage device is not emulated as a drive attached to an emulated USB controller. Rather, it is connected through an emulated SCSI controller and led by a near-native speed PVOPS driver on Linux guests, effectively removing any bandwidth limit posed by emulating USB without loosing the hot-plugging ability!

In the end, attaching the disk was a breeze and the transfer rate of the drive prove the effort’s worth as I got more than 50 MB/s throughput, doing network file copy over CIFS and GbE.

That’s it! Feedback is always welcome :)

Attaching local disks to a Citrix XenServer VM was originally published by Panagiotis "PJ" Papadomitsos at Primordial Echo on October 18, 2011.