First version

README.md

@@ -1,5 +1,141 @@
+CycleCloud Redis Sample Cluster
+===================================
 
-# Contributing
+The CycleCloud Redis sample demonstrates configuring and launching a basic
+Redis ( https://redis.io/ ) cluster from the CycleCloud GUI.
+
+
+Pre-Requisites
+--------------
+
+This sample requires the following:
+
+  1. CycleCloud must be installed and running.
+
+     a. If this is not the case, see the CycleCloud QuickStart Guide for
+        assistance.
+
+  2. The CycleCloud CLI must be installed and configured for use.
+
+  3. You must have access to log in to CycleCloud.
+
+  4. You must have access to upload data and launch instances in your chosen
+     Cloud Provider account.
+
+  5. You must have access to a configured CycleCloud "Locker" for Project Storage
+     (Cluster-Init and Chef).
+
+  6. Optional: To use the `cyclecloud project upload <locker>` command, you must
+     have a Pogo configuration file set up with write-access to your locker.
+
+     a. You may use your preferred tool to interact with your storage "Locker"
+        instead.
+
+
+**NOTE:**
+::
+   
+  The instructions in this guide assume the use of Amazon Web Services for the Cloud Provider account.
+
+
+Usage
+=====
+
+A. Configuring the Project
+--------------------------
+
+The first step is to configure the project for use with your storage locker:
+
+  1. Open a terminal session with the CycleCloud CLI enabled.
+
+  2. Switch to the Redis-Cluster sample directory.
+
+  3. Run ``cyclecloud project add_target my_locker`` (assuming the locker is named "my_locker").
+     The locker name will generally be the same as the cloud provider you created when configuring
+     CycleCloud. The expected output looks like this:::
+
+       $ cyclecloud project add_target my_locker
+       Name: redis-cluster
+       Version: 1.0.0
+       Targets:
+          my_locker: {'default': 'true', 'is_locker': 'true'}
+
+     NOTE: You may call add_target as many times as needed to add additional target lockers.
+
+       
+B. Deploying the Project
+------------------------
+
+To upload the project (including any local changes) to your target locker, run the
+`cyclecloud project upload` command from the project directory.  The expected output looks like
+this:::
+
+    $ cyclecloud project upload
+    Sync completed!
+
+*IMPORTANT*
+
+For the upload to succeed, you must have a valid Pogo configuration for your target Locker.
+
+
+C. Importing the Cluster Template
+---------------------------------
+
+To import the cluster:
+
+  1. Open a terminal session with the CycleCloud CLI enabled.
+
+  2. Switch to the Redis-Cluster sample directory.
+
+  3. Run ``cyclecloud import_template redis-cluster -f ./templates/redis-cluster.txt``.  The
+     expected output looks like this:::
+
+       $ cyclecloud import_template redis-cluster -f ./redis-cluster.txt
+       Importing template redis-cluster....
+       ----------------------
+       Redis : *template*
+       ----------------------
+       Keypair: $keypair
+       Cluster nodes:
+           proxy: off
+       Total nodes: 1
+
+
+D. Creating a Redis Cluster
+-------------------------------
+
+  1. Log in to your CycleCloud from your browser.
+
+  2. Click the **"Clusters"** to navigate to the CycleCloud "Clusters" page, if
+     you are not already there.
+
+  3. Click the **"+"** button in the "Clusters" frame to create a new cluster.
+
+  4. In the cluster creation page, click on the **Redis-Cluster** cluster icon.
+
+  5. At a minimum, select the Cloud Provider Credentials to use and enter a Name
+     for the cluster.
+
+  6. Click the **"Start"** button.
+
+
+E. Starting and Stopping the Redis Cluster
+----------------------------------------------
+
+  1. Select the newly created Redis cluster from the **Clusters**
+     frame on the CycleCloud "Clusters" page
+
+  2. To start the cluster, click the **Start** link in the cluster status
+     frame.
+     
+  3. Later, to stop a started cluster, click the **Terminate** link in the
+     cluster status frame.
+     
+
+
+  
+Contributing
+============
 
 This project welcomes contributions and suggestions.  Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
@@ -12,3 +148,5 @@ provided by the bot. You will only need to do this once across all repos using o
 This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+

project.ini

@@ -0,0 +1,10 @@
+[project]
+name = redis
+version = 1.2.0
+
+
+[spec proxy]
+run_list = recipe[redis-cluster::client]
+
+[spec server]
+run_list = role[redis_server]

specs/default/chef/roles/redis_server.rb

@@ -0,0 +1,9 @@
+name "redis_server"
+description "Redis Server role"
+run_list("recipe[redis-cluster::server]")
+
+default_attributes(
+  "cyclecloud" => { "discoverable" => true },
+  "redis" => { "role" => "server",
+               "ready" => true }
+)

specs/default/chef/site-cookbooks/redis-cluster/attributes/default.rb

@@ -0,0 +1,23 @@
+# Redis Cluster for clients to use
+default['redis']['cluster_name'] = node['cyclecloud']['cluster']['id']
+
+# Client configuration file (alternately use: jetpack config redis.hostlist)
+default['redis']['hostlist'] = []
+default['redis']['hostfile'] = File.join(node['cyclecloud']['home'], "config", "redis_hosts.txt")
+
+
+# Server settings
+default['redis']['home'] = "/mnt/scratch/redis"
+default['redis']['version'] = '3.2.6'
+default['redis']['source'] = "redis-3.2.6.tar.gz"
+
+
+default['redis']['cluster_size'] = 1
+default['redis']['base_port'] = 7000
+default['redis']['server_slots'] = 4
+default['redis']['replicas'] = nil
+default['redis']['mb_per_slot'] = 4000
+default['redis']['cluster-node-timeout'] = 5000
+
+default['redis']['sysctl']['vm']['overcommit_memory'] = 1
+default['redis']['sysctl']['net']['core']['somaxconn'] = 512

specs/default/chef/site-cookbooks/redis-cluster/metadata.rb

@@ -0,0 +1,9 @@
+name             "redis-cluster"
+maintainer       "CycleComputing LLC"
+maintainer_email "support@cyclecomputing.com"
+license          "All Rights Reserved"
+description      "Installs/Configures redis cluster"
+version          "1.0.0"
+
+depends 'line'
+depends 'sysctl'

specs/default/chef/site-cookbooks/redis-cluster/recipes/client.rb

@@ -0,0 +1,19 @@
+# The basic Redis Client recipe simply makes the server hostlist available
+# to client applications via jetpack or via a conf file.
+
+redis_cluster = node['redis']['cluster_name']
+servers = cluster.search(:clusterUID => redis_cluster).select {|n| not n['redis'].nil? and n['redis']['ready'] == true}.map  do |n|
+  "#{n['cyclecloud']['instance']['ipv4']}:#{n['redis']['base_port']}"
+end
+servers.sort!
+Chef::Log.info "Redis cluster: #{servers.inspect}"
+
+
+# Clients should use the node['redis']['hostlist'] to connect
+node.set['redis']['hostlist'] = servers
+
+# create a redis cluster host list that can be used by client applications
+file node['redis']['hostfile'] do
+  content servers.join("\n")
+end
+

specs/default/chef/site-cookbooks/redis-cluster/recipes/default.rb

@@ -0,0 +1,31 @@
+
+directory node['redis']['home'] do
+  mode "0755"
+  recursive true
+end
+
+if node['platform_family'] == 'rhel'
+  ['redis', 'redis-trib'].each {|p| package p}
+else
+
+  # the redis version on ubuntu is ancient, this one is more up-to-date
+  apt_repository 'redis-server' do
+    uri 'ppa:chris-lea/redis-server'
+    distribution node['lsb']['codename']
+  end
+  
+  ['redis-server', 'ruby'].each {|p| package p}
+
+  gem_package 'redis'
+
+  # sadly, this file is not available in a debian package
+  cookbook_file '/usr/bin/redis-trib' do
+    # using http, classy!
+    source 'redis-trib.rb'
+    mode '0755'
+  end
+  
+end
+
+
+

specs/default/chef/site-cookbooks/redis-cluster/recipes/server.rb

@@ -0,0 +1,147 @@
+include_recipe 'redis-cluster::default'
+include_recipe 'redis-cluster::server_install'
+
+# disable the default redis service
+['redis', 'redis-sentinel'].each do |s|
+  service s do
+    action [:stop, :disable]
+  end
+end
+
+# IMPORTANT:
+# If running through STunnel - set NODE_IP to loopback (127.0.0.1)
+# redis-cluster will report back the ips of the cluster members
+# See: https://github.com/xetorthio/jedis/issues/943
+
+
+# Default: 0, 1 or 2 replicas depending on cluster size
+if node['redis']['replicas'].nil?
+  node.set['redis']['replicas'] = case node['redis']['cluster_size'].to_i
+                                  when 0..1
+                                    0
+                                  when 2
+                                    1
+                                  else
+                                    2
+                                  end
+end
+node.set['redis']['replicas'] = [0, node['redis']['replicas'].to_i].max
+Chef::Log.info "Joining Redis cluster [size: #{node['redis']['cluster_size']}, replicas: #{node['redis']['replicas']}]..."
+
+
+# TODO: get list from search
+Chef::Log.info "Creating #{node['redis']['server_slots']} server slots..."
+for i in 0..(node['redis']['server_slots'] - 1)
+    server_slot_port = node['redis']['base_port'].to_i + i
+    server_slot_home = "#{node['redis']['home']}/#{server_slot_port}"
+    server_conf_file = "redis_#{server_slot_port}.conf"
+    server_address = "#{node['cyclecloud']['instance']['ipv4']}:#{server_slot_port}"
+
+    Chef::Log.info "Creating server slot #{i} at #{server_slot_home}..."
+    directory server_slot_home do
+      owner 'root'
+      group 'root'
+      mode '0755'
+      recursive true
+      action :create
+    end
+
+    template "#{server_slot_home}/#{server_conf_file}" do
+        source "redis.conf.erb"
+        mode "0644"
+        owner 'root'
+        group 'root'
+        variables(:port => server_slot_port)
+    end
+
+
+    log "Checking Redis server: #{server_address} in #{server_slot_home}" do level :info end
+
+    log "Redis server: #{server_address} already running" do
+        level :info
+        only_if "ps aux | grep 'redis.*:#{server_slot_port}' | grep -q -v grep"
+    end
+
+    if node["redis"]["version"] < "3.2"
+      protected_mode_arg = ""
+    else
+      protected_mode_arg = "--protected-mode no"
+    end
+
+
+    execute "start_redis_#{server_slot_port}" do
+        command "redis-server #{server_conf_file} #{protected_mode_arg} --loglevel verbose > ./redis_#{server_slot_port}.log 2>&1 &"
+        cwd server_slot_home
+        action :nothing
+    end
+
+    execute "clean_redis_#{server_slot_port}" do
+        command "rm -f *.aof nodes.#{server_slot_port}.conf redis_#{server_slot_port}.out redis_#{server_slot_port}.err"
+        cwd server_slot_home
+        notifies :run, "execute[start_redis_#{server_slot_port}]", :immediately
+
+        not_if "ps aux | grep 'redis.*:#{server_slot_port}' | grep -q -v grep"
+    end
+
+    # It appears that sometimes redis comes up without the correct config for protected mode...
+    # TBD: does this mean the other configs aren't loaded?
+	 execute "disable_protected_mode_#{server_slot_port}" do
+	   command "redis-cli -c -h 127.0.0.1 -p #{server_slot_port} config set protected-mode no >> ./redis_#{server_slot_port}.log 2>&1 &"
+	   cwd server_slot_home
+	   action :run
+	   not_if node["redis"]["version"] < "3.2"
+	 end
+
+end
+node.set['redis']['ready'] = true
+node.set['cyclecloud']['discoverable'] = true
+
+if node["redis"]["servers"].nil?
+  servers = nil
+  timeout = 60 * 5
+  omega = Time.now.to_i + timeout
+  while Time.now.to_i < omega do
+    servers = cluster.search.select {|n| not n['redis'].nil? and n['redis']['ready'] == true}.map  do |n|
+      n[:cyclecloud][:instance][:ipv4]
+    end
+    if servers.length >= node["redis"]["cluster_size"]
+      break
+    end
+    Chef::Log.info "Waiting on Redis cluster: so far - #{servers.inspect}"
+    sleep 10
+  end
+end
+
+if servers.length < node["redis"]["cluster_size"]
+  raise Exception, "Redis cluster timed out!"
+end
+servers.sort!
+Chef::Log.info "Redis cluster: #{servers.inspect}"
+node.set['redis']['servers'] = servers
+
+# first ip will initialize
+if node['cyclecloud']['instance']['ipv4'] == servers[0]
+  cluster_replica_list = ""
+  for server in servers
+    for i in 0..(node['redis']['server_slots'] - 1)
+      server_slot_port = node['redis']['base_port'] + i
+      server_address = "#{server}:#{server_slot_port}"
+      cluster_replica_list = cluster_replica_list + "#{server_address} "
+    end
+  end
+
+  execute "Create Redis cluster #{cluster_replica_list}" do
+      command "yes yes | redis-trib create --replicas #{node['redis']['replicas']} #{cluster_replica_list} | tee /dev/stderr | ( ! grep -q 'error\|CLUSTERDOWN' )"
+      cwd node['redis']['home']
+      # Verify that the "myself" line is not the only line
+      not_if "redis-cli -h #{node['cyclecloud']['instance']['ipv4']} -p #{node['redis']['base_port']} cluster nodes | grep -q -v 'myself'"
+  end
+
+  # Sadly, redis-cli isn't reliable about setting the error code
+  execute "Test redis status or fail" do
+    command "redis-cli -c -h #{node['cyclecloud']['instance']['ipv4']}  -p #{server_slot_port} set status_test up | tee /dev/stderr | ( ! grep -q 'error\|CLUSTERDOWN' )"
+  end
+
+end
+
+include_recipe 'redis-cluster::client'

specs/default/chef/site-cookbooks/redis-cluster/recipes/server_install.rb

@@ -0,0 +1,26 @@
+include_recipe 'redis-cluster::default'
+
+log "Configuring kernel for high memory redis application..." do level :info end
+include_recipe 'sysctl::default'
+
+execute "disable transparent_hugepage" do
+  command "echo never > /tmp/transparent_hugepage.enabled; mv /tmp/transparent_hugepage.enabled /sys/kernel/mm/transparent_hugepage/enabled"
+  only_if "grep -qv never /sys/kernel/mm/transparent_hugepage/enabled"
+end
+
+# rc.local is deprecated in centos, find something better
+include_recipe 'line'
+append_if_no_line "disable transparent_hugepage via rc.local" do
+  path "/etc/rc.local"
+  line "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
+end
+
+sysctl_param 'vm.overcommit_memory' do
+    value node['redis']['sysctl']['vm']['overcommit_memory']
+end
+
+sysctl_param 'net.core.somaxconn' do
+    value node['redis']['sysctl']['net']['core']['somaxconn']
+end
+
+

specs/default/chef/site-cookbooks/redis-cluster/templates/default/redis.conf.erb

@@ -0,0 +1,9 @@
+port <%= @port %>
+cluster-enabled yes
+cluster-config-file nodes.<%= @port %>.conf
+cluster-node-timeout <%= node['redis']['cluster-node-timeout'] %>
+appendonly no
+save ""
+<% if node[:redis][:version] > '3.2' -%>
+protected-mode no
+<% end -%>

specs/server/cluster-init/tests/test_servers.py

@@ -0,0 +1,23 @@
+import unittest
+import subprocess
+import jetpack.config
+
+
+def do_ping(port):
+    '''The function actually asks zookeeper if all of the kafka brokers are registered'''
+    p = subprocess.Popen(['redis-cli', '-p', port, 'ping'],
+                         stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = p.communicate()
+    return (p.returncode, stdout, stderr)
+
+
+class TestRedisCluster(unittest.TestCase):
+
+    def test_ping(self):
+        slots = int(jetpack.config.get('redis.server_slots'))
+        starting_port = 7000
+        for slot in range(0, slots):
+            port = starting_port + slot
+            returncode, stdout, stderr = do_ping(str(port))
+            self.assertTrue(returncode == 0, msg="Redis server not listening on port %s"
+                            % str(port))

templates/redis.txt

@@ -0,0 +1,184 @@
+######################################
+## Redis-Cluster Configuration File ##
+######################################
+
+[cluster Redis]
+FormLayout = selectionpanel
+IconUrl = https://github.com/CycleCloud/redis/raw/master/icon.png
+Category = Infrastructure
+
+Autoscale = false
+
+    [[node defaults]]
+    Credentials = $Credentials    
+    ImageName = $ImageName
+    SubnetId = $SubnetId
+    Region = $Region
+    KeyPairLocation = $KeyPairLocation
+
+    AdditionalClusterInitSpecs = $DefaultClusterInitSpecs
+    
+    [[[cluster-init cyclecloud/redis:default:$ProjectVersion]]]
+
+    [[node proxy]]
+    IsReturnProxy = $ReturnProxy
+    MachineType = $ProxyMachineType
+
+       [[[configuration]]]
+       run_list = recipe[cganglia::server]
+       cyclecloud.discoverable = true
+
+       [[[cluster-init cyclecloud/redis:proxy:$ProjectVersion]]]
+
+       [[[network-interface eth0]]]
+       AssociatePublicIpAddress = $UsePublicNetwork
+
+       [[[input-endpoint ganglia]]]
+       PrivatePort = 8652
+       PublicPort = 8652        
+
+
+    [[nodearray server]]
+    MachineType = $ServerMachineType
+    InitialCount = $RedisClusterSize
+
+    AdditionalClusterInitSpecs = $ServerClusterInitSpecs
+    
+
+       [[[configuration]]]
+       run_list = recipe[cganglia::client]
+       cyclecloud.discoverable = true
+       redis.cluster_size = $RedisClusterSize
+
+       [[[cluster-init cyclecloud/redis:server:$ProjectVersion]]]
+
+       [[[network-interface]]]
+       AssociatePublicIpAddress = false
+
+
+
+[parameters About]
+Order = 1
+
+    [[parameters About Redis Cluster]]
+
+        [[[parameter RedisCluster]]]
+        HideLabel = true
+        Config.Plugin = pico.widget.HtmlTemplateWidget
+        Config.Template := "<table><tr><td><img src='https://github.com/CycleCloud/redis/raw/master/icon.png' width='192' height='192'></td></tr><tr><td><p>Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. See the <a href=\"https://redis.io/\" target=\"_blank\">Redis project site</a>for an overview.  This cluster type is configured to use <a href=\"https://redis.io/topics/cluster-tutorial\" target=\"_blank\">Redis Cluster</a> to create a scalable cache.</p></td></tr></table>"
+
+        [[[parameter Readme]]]
+        HideLabel = true
+        Config.Plugin = pico.widget.HtmlTemplateWidget
+        Config.Template := "Follow the instructions in the <a href=\"https://github.com/CycleCloud/redis/\" target=\"_blank\">README</a> for details on instructions on extending and configuring the Project for your environment."
+
+
+[parameters Required Settings]
+Order = 10
+
+    [[parameters Virtual Machines ]]
+    Description = "The cluster, in this case, has two roles: the scheduler master-node with shared filer and the execute hosts. Configure which VM types to use based on the requirements of your application."
+    Order = 20
+
+
+        [[[parameter Region]]]
+        Label = Region
+        Description = Deployment Location
+        ParameterType = Cloud.Region
+        DefaultValue = westus2
+
+        [[[parameter ProxyMachineType]]]
+        Label = Proxy VM Type
+        Description = The VM type for proxy node
+        ParameterType = Cloud.MachineType
+        DefaultValue = Standard_D2_v2
+
+        [[[parameter ServerMachineType]]]
+        Label = Server VM Type
+        Description = The VM type for the Redis Servers
+        ParameterType = Cloud.MachineType
+        DefaultValue = Standard_DS4_v2
+
+
+    [[parameters Networking]]
+    Order = 40
+
+        [[[parameter SubnetId]]]
+        Label = Subnet ID
+        Description = Subnet Resource Path (ResourceGroup/VirtualNetwork/Subnet)
+        ParameterType = Azure.Subnet
+
+
+[parameters Advanced Settings]
+Order = 20
+
+    [[parameters Azure Settings]]
+    Order = 10 
+
+        [[[parameter Credentials]]]
+        Description = The credentials for the cloud provider
+        ParameterType = Cloud.Credentials
+
+
+    [[parameters Redis]]
+    Description = Adjust the Redis configuration options.  (Data redunancy and fault tolerance requires more than 1 server (max: 10).)
+    Order = 20
+
+        [[[parameter RedisClusterSize]]]
+        Label = Cluster Size
+        Description = The number of Redis Servers to start initially.
+        DefaultValue = 1
+        Config.Plugin = pico.form.NumberTextBox
+        Config.MinValue = 1
+        Config.MaxValue = 10
+        Config.IntegerOnly = true
+
+        [[[parameter ProjectVersion]]]
+        Label = Project Version
+        DefaultValue = 1.2.0
+        Description = Select the Redis project spec version
+        Hidden = true
+
+    [[parameters Software]]
+    Description = "Specify the scheduling software, and base OS installed on all nodes, and optionally the cluster-init and chef versions from your Locker."
+    Order = 10
+
+        [[[parameter ImageName]]]
+        Label = Base OS
+        ParameterType = Cloud.Image
+        Config.OS = linux
+        DefaultValue = cycle.image.centos7
+
+        [[[parameter DefaultClusterInitSpecs]]]
+        Label = Default Cluster-Init Specs
+        Description = Cluster init specs to apply to all nodes
+        ParameterType = Cloud.ClusterInitSpecs
+
+        [[[parameter ServerClusterInitSpecs]]]
+        Label = Server Cluster-Init Specs
+        DefaultValue = =undefined
+        Description = Cluster init specs to apply to the master node
+        ParameterType = Cloud.ClusterInitSpecs
+
+
+    [[parameters Advanced Networking]]
+    Description = Advanced networking settings
+
+        [[[parameter ReturnProxy]]]
+        Label = Return Proxy
+        DefaultValue = false
+        ParameterType = Boolean
+        Config.Label = Use SSH tunnel to connect to CycleCloud (required if direct access is blocked)
+
+        [[[parameter UsePublicNetwork]]]
+        Label = Public Head Node
+        DefaultValue = true
+        ParameterType = Boolean
+        Config.Label = Access master node from the Internet
+
+        [[[parameter KeyPairLocation]]]
+        Label = Keypair Path
+        Description = The path to the private SSH key to use for the `cyclecloud` user on linux nodes.
+        DefaultValue = ~/.ssh/cyclecloud.pem
+        Required = True
+